Another point to consider:
When implementing this we must consider that the Librem5 might have forks of important libraries with patches that are either:

• not upstream yet since they're still in review
• not in the version that is currently in Debian testing Therefore we should cater for a way to build packages on or own until we get the correct version from Debian.

This is a non-negligible amount of work.

Well, we can already sync packages with Debian unstable or experimental, but those have to be recompiled in PureOS - otherwise we can not ensure they actually work with the software that is in PureOS (that's based on testing).
Also, pulling packages from non-testing suites is an explicit task that is currently done manually.

I am aware that we can recompile.

This issue is explicitly about ability to track custom inclusion *without* recompilation, to stay close to Debian.

No, it is not always needed to recompile - Debian packages are *never* recompiled when moving from unstable to testing.

To clarify - and address the comments from @heather.ellsworth above:

This issue is *not* about handling ABI or API deviations. Exactly because for such "we can not ensure they actually work with the software that is in PureOS".

This is about packages that in Debian _would_ auto-migrate to testing had it not been for an RC bug keeping it out: It is about our ability to overrule Debian in their choice of what to auto-migrate to testing.

No, it is not always needed to recompile - Debian packages are *never* recompiled when moving from unstable to testing.

That is a false assumption. If we sync (binary) packages from Debian after they migrated to testing, we can be almost 100% sure that they work with all the other packages we synced from testing. If we sync from unstable however, it means that the package is compiled against a set of packages that are also only in Debian unstable, e.g. new GTK+ versions, a more recent libc, using a GCC not yet available in testing, etc.
Therefore, the only way that we can be sure that packages from unstable work in a PureOS testing environment is by recompiling them in PureOS.

Will binary syncs from unstable work in PureOS based on testing most of the time? Probably yes. But that isn't good enough - especially if we automatically sync things from other suites we need to be sure that the packages work 100%, nor in 90% of all cases causing us manual work and annoying debugging later.

Since recompiling unstable packages in PureOS is recommended, is it just as easy as adding a source site to a list and then that site is polled for updates? I am assuming that these packages are auto compiled by the build system?

@heather.ellsworth Jup, this feature would be rather straightforward to implement.
The tough bit implementation-wise would be to make Synchrotron selectively only sync source packages. At the moment, there is only a global switch for whether binary packages should be synced or not.

In any case: Do you have any concrete examples on what packages would actually use this feature? While I do think it's generally useful, I don't think we would actually use it very often.

(seems I forgot to actually post this yesterday)

Well I suppose this feature would be used in cases where packages have changes applied that are not in available in Debian. In this case, we would need to pull the source, apply whatever patches, auto compile against PureOS, and include it into PureOS.

A concrete example would be if some package depends on newer patched libraries (like Guido mentioned).

(this relates to the original topic of recompiling packages not about having to rebuild because we carry patches)

It is a continous delivery antipattern to recompile artifacts (debs in our case) since you basically lose most of the knowledge about what works and what not if you don't reproduce the original build environment by 100%.

Reproducible builds will help here a _lot_ and I can see points in favour of rebuilding (like Debian does not yet enforce source only uploads) but we might rather want to help getting Debian there (same for reproducible builds).
So IMHO Jonas has a point here that we should not recompile for the fun of it (having bit identical debs make reproducing bugs and comparing with Debian much simpler.

@guido Packages recompiled in different environments do produce different behavior though. A lot of packages have Ubuntu/Tanglu/PureOS specific configurations that get enabled when (re)built on a particular system.
Also, lots of software enables different codepaths depending on which version of a particular shared library it was compiled against. Additionally, unless there is a symbols file in a Debian package, the Debian package will always depend on at least the shared library version it was compiled against, meaning that you can not selectively pull a few package from a particular suite and expect the result to work.

We actually did a field experiment for this in Tanglu a few years back, to maybe be able to rebuild less stuff and save build resources, and the result was a real mess (fortunately Britney caught most of that before it migrated).
So, mixing binary packages from one suite with selective binary packages from another suite may work, but it's not at all guaranteed and not something we can rely on (meaning it's either rebuild things, or do binary syncs and have someone continuously look for breakage and for transitions to migrate (the latter of which we already have issues with today))

This is also related to https://tracker.pureos.net/T594

Thanks for mentioning, @guido - I had missed T594 now linked here.

Syncing from unstable is just flipping a switch. But we need to know if we *want* to do that. Unstable won't get many package changes during the freeze period either.
Autosyncing with experimental is a terrible idea, because experimental packages may just get deleted without ever reaching unstable. There is also a lot of broken stuff in there which we really don't want in PureOS (experimental is really just a playground with zero QA happening).
So, stuff from experimental should only ever be synced manually and if there is a good reason for doing so.

@jeremiah.foster are you asking to *generally* switch from tracking testing to track unstable (or experimental), or are you - like this issue was initially intended to be about - asking about singling out packages to be tracked exceptionally from unstable (or experimental), where packages otherwise generally continue to track testing?

From the phone perspective we we'd like to pull a rather fixed set of packages from e.g. Debian experimental during the freeze automatically (assuming Debian's GNOME team ends up putting the packages there) but after all it's more of an example of a general pattern: grab packages a, b, c from repository x (ideally including required dependencies not present in the target distribution).

Since you insist, @jeremiah.foster, on expanding/redefining the scope of this issue to be about switching *GENERALLY*, I cannot help and therefore unsubscribe from participation here:

I see absolutely no benefit in a generally unstable system for phones!

Comments that were removed can be discussed in a separate issue: https://tracker.pureos.net/T722

Scope of this issue now relaxed to include source-identical but rebuilt packages.

For the more scope introduced by @jeremiah.foster, please see T722.

For the more narrow original scope of auto-migrating non-rebuilt binary packages from unstable or experimental, please see T724.