Problem #

Today I saw a failing CI builds due to an HTTP 403 error. The logs showed it was apt-get install failing to fetch git=1:2.11.0-3+deb9u6, which no longer existed in the repo (it had been replaced with 1:2.11.0-3+deb9u7), hence the URL it was trying to fetch the package from was returning an error.

What’s going on? Well today I learned about Debian package versioning.

Learning #

The versioning scheme used here is [epoch:]upstream_version[-debian_revision]¹.

upstream_version #

Packages in Debian are maintained by maintainers, who publish a version of a package to the repository for a distribution. Where this package is not written specifically to be a Debian package, it is an upstream package. Debian package maintainers publish a Debian version (a .deb file) of this upstream_version of the package, which is what you get when you install using apt-get or similar.

debian_revision #

In some cases (e.g. security patches), there will be updates to the Debian version of the package that are based on the same upstream version. These updates to the Debian version (but not the upstream version) are the Debian revisions of the package, and have their own version number (e.g. 3-deb9u7): this is the debian_revision.

It is convention to restart the debian_revision at 1 each time the upstream_version is increased, so the first Debian release of git 2.11.1 would be 2.11.1-1, regardless of what the latest debian_revision of 2.11.0 was.

epoch #

The epoch identifies the versionning scheme. It changes when the upstream package’s versionning scheme changes, or when there are mistakes in versions of old packages.

Solution #

Back to the issue of the failing builds: what had happened was the Debian version of git available to the package manager had been updated from 3-deb9u6 to 3-deb9u7 - a security update. This broke the build because the build script was looking for git=1:2.11.0-3+deb9u6.

What’s the fix?

1. Get alerted to this change periodically by failing builds, update the build scripts to reference the new version each time.

   → Annoying but keeps you informed about what’s happening with your dependencies. I don’t know how often Debian packages get updated, but I can see this happening quite frequently at scale.

2. Un-pin the version.

   → Pinning dependencies is a “best practice” for a reason - it ensures your code and environment it executes in is repeatable which gives you confidence you won’t see behaviour in other environments (e.g. production) that you didn’t see in testing. To put it differently, if dependency versions are not pinned, you’re potentially executing different software each time to re-run the build process.

3. Change the pinned version, making it less specific.

   → For example, reducing the pin to just the upstream version. This transparently ensures you get the latest security patches from new Debian revisions each time, but doesn’t break your build each time a the Debian revision is updated. However, unless you track the Debian revision release dates by some other means, you won’t know when your dependencies have changed. Although they’re based on the same upstream version of the package, it’s still possible that a change to the Debian revision can impact your usage.

So what’s the best answer? #2 is a bad idea, and #1 is an easy fix. But could #3 be a reasonable option?

Honestly, I don’t know. On the one hand, there seems little downside of automatically getting the new Debian revision of a package each time you install and not having to spend the 5 minutes or so putting up a pull request and getting it approved for this change. However, it is just a few minutes, and getting notified something has changed can be valuable, e.g. if issues are found later on.

In the end, due to not having enough information about #3 and needing a fix quickly, #1 was chosen. Arriving at this solution made for an interesting discussion, one I expect will happen again the next time a build breaks.

Footnotes: