Skip to content

[master] Update "deb" packaging to use "dh_installsystemd"#686

Merged
thaJeztah merged 1 commit intodocker:masterfrom
tianon:deb-dh_installsystemd
May 17, 2022
Merged

[master] Update "deb" packaging to use "dh_installsystemd"#686
thaJeztah merged 1 commit intodocker:masterfrom
tianon:deb-dh_installsystemd

Conversation

@tianon
Copy link
Contributor

@tianon tianon commented May 7, 2022

This also removes the now-obsolete upstart script and updates the "debhelper compat level" to 13.

(See also https://lists.debian.org/debian-devel-announce/2020/02/msg00000.html)

Opened as a draft because it's still a WIP -- in theory, this should help with issues like moby/moby#41792 (by triggering debhelper to generate the appropriate systemctl commands in our postrm).

If this is something we want to actually pursue/merge, we probably want to revert the change to .github/workflows/ci.yml (which I included just so I could make sure I was successfully building on all the things), and we'll need to update deb/common/docker-ce.maintscript to include the correct "next" version number (see https://manpages.debian.org/bullseye/debhelper/dh_installdeb.1.en.html#package~8 and https://manpages.debian.org/bullseye/dpkg/dpkg-maintscript-helper.1.en.html#Removing_a_conffile / https://manpages.debian.org/bullseye/dpkg/dpkg-maintscript-helper.1.en.html#prior-version).

For example, for a conffile removed in version 2.0-1 of a package, prior-version should be set to 2.0-1~. This will cause the conffile to be removed even if the user rebuilt the previous version 1.0-1 as 1.0-1local1. Or a package switching a path from a symlink (shipped in version 1.0-1) to a directory (shipped in version 2.0-1), but only performing the actual switch in the maintainer scripts in version 3.0-1, should set prior-version to 3.0-1~.

I have not done any testing yet on the actual end result (I was already stretching to squeeze in doing commits periodically to get it to build successfully 🙈).

cc @thaJeztah

@thaJeztah
Copy link
Member

Some failures on raspbian buster (could that be "too old"?)

dwz: debian/docker-ce/usr/bin/docker-init: .debug_info section not present
dwz: debian/docker-ce/usr/bin/docker-proxy: .debug_info section not present
dwz: debian/docker-ce/usr/bin/dockerd: .debug_info section not present
dwz: Too few files for multifile optimization
dh_dwz: warning: No dwz multifile created, but not explicitly requested either so ignoring it.
dh_dwz: warning: Common issues include no debug information at all (missing -g) and
dh_dwz: warning: compressed debug information (#931891).
dwz: debian/docker-ce-cli/usr/bin/docker: .debug_info section not present
dh_dwz: error: dwz -- debian/docker-ce-cli/usr/bin/docker returned exit code 1
make: *** [debian/rules:129: binary] Error 25
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
Makefile:76: recipe for target 'raspbian-buster' failed
make[1]: *** [raspbian-buster] Error 2
make[1]: Leaving directory '/home/ubuntu/workspace/docker-ce-packaging_PR-686/deb'
Makefile:92: recipe for target 'raspbian-buster' failed
make: *** [raspbian-buster] Error 2
script returned exit code 2

Let me kick CI again, just in case it was a glitch

@@ -0,0 +1 @@
engine/contrib/init/systemd/docker.service No newline at end of file
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this work? 🤔 Can the file contain the name of "actual file to look for"? Or is this just a "placeholder"? (trying to understand what this does)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh sorry, this is a symlink! We could probably replace the other init scripts with symlinks too (if they're unchanged from what's in moby/moby).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh! LOL

Didn't expect it to show up as a regular file in github's UI 🤦‍♂️😂

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeahhhhh, I really wish symlinks were more obviously labelled in diffs, but at least it shows something! 😄

ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y curl devscripts equivs git
ARG SUITE
RUN apt-get update && apt-get install -y curl devscripts equivs git debhelper/${SUITE}-backports dwz/${SUITE}-backports
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's the dwz/ ? Is that something defined by deb_helper ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah, I should've explained better -- dwz is a dependency used by debhelper, and when we pull in the newer debhelper, we also need the newer dwz (the failure noted in #686 (comment) is related -- it needs the same change, but Raspbian has no backports so uh, this gets complicated and might force this PR to be reworked / abandoned for now 🙈).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could go to debhelper-compat version 12 instead, and then Buster would no longer need backports, but then we run into bugs with dh_dwz (which frankly, even if it works correctly, won't be doing anything), so we'd have to also add override_dh_dwz that's a no-op. It's probably worth updating this to that so you can see that (perhaps in a separate commit so you can see both?).

@tianon tianon force-pushed the deb-dh_installsystemd branch from 7626726 to 2398dd7 Compare May 9, 2022 17:29
This also removes the now-obsolete upstart script and updates the "debhelper compat level" to 12 (which necessitates adding "override_dh_dwz" -- safe to remove once we can update to debhelper-compat 13+).

(See also https://lists.debian.org/debian-devel-announce/2020/02/msg00000.html)

Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
@tianon tianon force-pushed the deb-dh_installsystemd branch from 2398dd7 to f9ac2f6 Compare May 13, 2022 21:27
@tianon
Copy link
Contributor Author

tianon commented May 13, 2022

Latest push removes the TODO in deb/common/docker-ce.maintscript because after looking at the implementation, we're safe to just use an in-between version number like 5:21~ (knowing that we might still have 5:20.xx releases but that the next release will likely be 5:22.xx and we will not ever have an actual 5:21.xx).

Also, in case I didn't say it explicitly, this is not suitable to backport to the 20.xx branches as-is (we'll have to revisit that TODO if we do so).

@tianon tianon marked this pull request as ready for review May 13, 2022 21:29
@tianon
Copy link
Contributor Author

tianon commented May 13, 2022

Now the only part of this I'm not sure about is the expanded GitHub Actions matrix (which I'm happy to trim back down). 👍

@thaJeztah
Copy link
Member

Now the only part of this I'm not sure about is the expanded GitHub Actions matrix (which I'm happy to trim back down

Probably fine for now; we created that action to do some initial testing. Those jobs run in parallel, so shouldn't make CI longer to run (and eventually we want to just have CI run on actions - no support for non-x86 though yet)

Copy link
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah thaJeztah merged commit 24e75f9 into docker:master May 17, 2022
@tianon tianon deleted the deb-dh_installsystemd branch May 17, 2022 16:33
@thaJeztah thaJeztah changed the title Update "deb" packaging to use "dh_installsystemd" [master] Update "deb" packaging to use "dh_installsystemd" May 19, 2022
@thaJeztah
Copy link
Member

Hmm... I suspect this change may have introduced an issue in our "verify" step in the internal pipeline (which we should still upstream to this repository 😞)

Details
+ dpkg --ignore-depends=containerd.io,iptables,libdevmapper,libdevmapper1.02.1 --force-depends -i ./packaging/deb/debbuild/ubuntu-bionic/docker-buildx-plugin_0.8.2~ubuntu-bionic_arm64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-ce-cli_22.06.0~beta.0~3-0~ubuntu-bionic_arm64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-ce-rootless-extras_22.06.0~beta.0~3-0~ubuntu-bionic_arm64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-ce_22.06.0~beta.0~3-0~ubuntu-bionic_arm64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-compose-plugin_2.5.1~ubuntu-bionic_arm64.deb
Selecting previously unselected package docker-buildx-plugin.
(Reading database ... 6938 files and directories currently installed.)
Preparing to unpack .../docker-buildx-plugin_0.8.2~ubuntu-bionic_arm64.deb ...
Unpacking docker-buildx-plugin (0.8.2~ubuntu-bionic) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../docker-ce-cli_22.06.0~beta.0~3-0~ubuntu-bionic_arm64.deb ...
Unpacking docker-ce-cli (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce-rootless-extras.
Preparing to unpack .../docker-ce-rootless-extras_22.06.0~beta.0~3-0~ubuntu-bionic_arm64.deb ...
Unpacking docker-ce-rootless-extras (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../docker-ce_22.06.0~beta.0~3-0~ubuntu-bionic_arm64.deb ...
Unpacking docker-ce (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-compose-plugin.
Preparing to unpack .../docker-compose-plugin_2.5.1~ubuntu-bionic_arm64.deb ...
Unpacking docker-compose-plugin (2.5.1~ubuntu-bionic) ...
Setting up docker-buildx-plugin (0.8.2~ubuntu-bionic) ...
Setting up docker-ce-cli (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
Setting up docker-ce (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
invoke-rc.d: syntax error: unknown option "--skip-systemd-native"
dpkg: error processing package docker-ce (--install):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Setting up docker-compose-plugin (2.5.1~ubuntu-bionic) ...
Setting up docker-ce-rootless-extras (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
dpkg: docker-ce-rootless-extras: dependency problems, but configuring anyway as you requested:
 docker-ce-rootless-extras depends on dbus-user-session; however:
  Package dbus-user-session is not installed.
docker-ce-rootless-extras depends on dbus-user-session; however:
  Package dbus-user-session is not installed.

Possibly because that install check is using offline packages (before they're published)

That dependency was already added long time ago (and is already in 20.10); #571

Previously we did things manually, and possibly the automated bits don't take some of this into account?

@thaJeztah
Copy link
Member

So, I'm likely able to fix it in our verify script, but want to understand "what changed" (and if it's indeed to be expected, so just a missing thing in the verify script).

I'm doing a build of ubuntu bionic (18.04) before/after this change, and try to find the differences in the packages before/after (metadata, post install scripts, etc).

@thaJeztah
Copy link
Member

Can't find any differences, or at least, can't find anything that stands out 🤔

Did a compare between dpkg-deb -I output, and extracted the .deb packages for both, and I don't see an immediate difference between the two (other than different checksums, docker compose 2.5.0 -> 2.5.1, and the containerd >= 1.6.4 change)

cd before; for f in *.deb; do dpkg-deb -I $f >> $f.txt; done; cd ../
cd after; for f in *.deb; do dpkg-deb -I $f >> $f.txt; done; cd ../


cd before; for f in *.deb; do mkdir "${f%.*}"; dpkg-deb -R $f "${f%.*}"; done; cd ../
cd after; for f in *.deb; do mkdir "${f%.*}"; dpkg-deb -R $f "${f%.*}"; done; cd ../

@thaJeztah
Copy link
Member

For future reference; this is what the same (on a test-build of docker 22.04) looked like

Details
+ dpkg --ignore-depends=containerd.io,iptables,libdevmapper,libdevmapper1.02.1 --force-depends -i ./packaging/deb/debbuild/ubuntu-bionic/docker-ce-rootless-extras_22.04.0~beta.0~3-0~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-compose-plugin_2.3.4~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-buildx-plugin_0.8.1~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-scan-plugin_0.17.0~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-ce-cli_22.04.0~beta.0~3-0~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-ce_22.04.0~beta.0~3-0~ubuntu-bionic_amd64.deb
Selecting previously unselected package docker-ce-rootless-extras.
(Reading database ... 6945 files and directories currently installed.)
Preparing to unpack .../docker-ce-rootless-extras_22.04.0~beta.0~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:22.04.0~beta.0~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-compose-plugin.
Preparing to unpack .../docker-compose-plugin_2.3.4~ubuntu-bionic_amd64.deb ...
Unpacking docker-compose-plugin (2.3.4~ubuntu-bionic) ...
Selecting previously unselected package docker-buildx-plugin.
Preparing to unpack .../docker-buildx-plugin_0.8.1~ubuntu-bionic_amd64.deb ...
Unpacking docker-buildx-plugin (0.8.1~ubuntu-bionic) ...
Selecting previously unselected package docker-scan-plugin.
Preparing to unpack .../docker-scan-plugin_0.17.0~ubuntu-bionic_amd64.deb ...
Unpacking docker-scan-plugin (0.17.0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../docker-ce-cli_22.04.0~beta.0~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce-cli (5:22.04.0~beta.0~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../docker-ce_22.04.0~beta.0~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce (5:22.04.0~beta.0~3-0~ubuntu-bionic) ...
Setting up docker-compose-plugin (2.3.4~ubuntu-bionic) ...
Setting up docker-buildx-plugin (0.8.1~ubuntu-bionic) ...
Setting up docker-scan-plugin (0.17.0~ubuntu-bionic) ...
Setting up docker-ce-cli (5:22.04.0~beta.0~3-0~ubuntu-bionic) ...
Setting up docker-ce (5:22.04.0~beta.0~3-0~ubuntu-bionic) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up docker-ce-rootless-extras (5:22.04.0~beta.0~3-0~ubuntu-bionic) ...
dpkg: docker-ce-rootless-extras: dependency problems, but configuring anyway as you requested:
 docker-ce-rootless-extras depends on dbus-user-session; however:
  Package dbus-user-session is not installed.

+ apt-get -y install --no-install-recommends --no-upgrade --fix-broken ./packaging/deb/debbuild/ubuntu-bionic/docker-ce-rootless-extras_22.04.0~beta.0~3-0~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-compose-plugin_2.3.4~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-buildx-plugin_0.8.1~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-scan-plugin_0.17.0~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-ce-cli_22.04.0~beta.0~3-0~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-ce_22.04.0~beta.0~3-0~ubuntu-bionic_amd64.deb
Reading package lists...
Building dependency tree...
Reading state information...
Correcting dependencies... Done
Skipping docker-ce, it is already installed and upgrade is not set.
Skipping docker-ce-rootless-extras, it is already installed and upgrade is not set.
Skipping docker-scan-plugin, it is already installed and upgrade is not set.
Skipping docker-buildx-plugin, it is already installed and upgrade is not set.
Skipping docker-compose-plugin, it is already installed and upgrade is not set.
Skipping docker-ce-cli, it is already installed and upgrade is not set.
The following additional packages will be installed:
  containerd.io dbus dbus-user-session iptables libapparmor1 libargon2-0
  libcap2 libcryptsetup12 libdevmapper1.02.1 libidn11 libip4tc0 libip6tc0
  libiptc0 libjson-c3 libkmod2 libmnl0 libnetfilter-conntrack3 libnfnetlink0
  libpam-systemd libxtables12 multiarch-support systemd systemd-sysv
Suggested packages:
  kmod systemd-container policykit-1
Recommended packages:
  dmsetup networkd-dispatcher libnss-systemd
The following NEW packages will be installed:
  containerd.io dbus dbus-user-session iptables libapparmor1 libargon2-0
  libcap2 libcryptsetup12 libdevmapper1.02.1 libidn11 libip4tc0 libip6tc0
  libiptc0 libjson-c3 libkmod2 libmnl0 libnetfilter-conntrack3 libnfnetlink0
  libpam-systemd libxtables12 multiarch-support systemd systemd-sysv
0 upgraded, 23 newly installed, 0 to remove and 1 not upgraded.

And the new one (failure)

Details
+ dpkg --ignore-depends=containerd.io,iptables,libdevmapper,libdevmapper1.02.1 --force-depends -i ./packaging/deb/debbuild/ubuntu-bionic/docker-buildx-plugin_0.8.2~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-ce_22.06.0~beta.0~3-0~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-compose-plugin_2.5.1~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-scan-plugin_0.17.0~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-ce-rootless-extras_22.06.0~beta.0~3-0~ubuntu-bionic_amd64.deb ./packaging/deb/debbuild/ubuntu-bionic/docker-ce-cli_22.06.0~beta.0~3-0~ubuntu-bionic_amd64.deb
Selecting previously unselected package docker-buildx-plugin.
(Reading database ... 6945 files and directories currently installed.)
Preparing to unpack .../docker-buildx-plugin_0.8.2~ubuntu-bionic_amd64.deb ...
Unpacking docker-buildx-plugin (0.8.2~ubuntu-bionic) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../docker-ce_22.06.0~beta.0~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-compose-plugin.
Preparing to unpack .../docker-compose-plugin_2.5.1~ubuntu-bionic_amd64.deb ...
Unpacking docker-compose-plugin (2.5.1~ubuntu-bionic) ...
Selecting previously unselected package docker-scan-plugin.
Preparing to unpack .../docker-scan-plugin_0.17.0~ubuntu-bionic_amd64.deb ...
Unpacking docker-scan-plugin (0.17.0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce-rootless-extras.
Preparing to unpack .../docker-ce-rootless-extras_22.06.0~beta.0~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../docker-ce-cli_22.06.0~beta.0~3-0~ubuntu-bionic_amd64.deb ...
Unpacking docker-ce-cli (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
Setting up docker-buildx-plugin (0.8.2~ubuntu-bionic) ...
Setting up docker-compose-plugin (2.5.1~ubuntu-bionic) ...
Setting up docker-scan-plugin (0.17.0~ubuntu-bionic) ...
Setting up docker-ce-cli (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
Setting up docker-ce (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
invoke-rc.d: syntax error: unknown option "--skip-systemd-native"
dpkg: error processing package docker-ce (--install):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Setting up docker-ce-rootless-extras (5:22.06.0~beta.0~3-0~ubuntu-bionic) ...
dpkg: docker-ce-rootless-extras: dependency problems, but configuring anyway as you requested:
 docker-ce-rootless-extras depends on dbus-user-session; however:
  Package dbus-user-session is not installed.

Errors were encountered while processing:
 docker-ce
make: *** [Makefile:116: verify] Error 1
script returned exit code 2

OOOOOOH, and I now see the actual failure;

invoke-rc.d: syntax error: unknown option "--skip-systemd-native"
dpkg: error processing package docker-ce (--install):
 installed docker-ce package post-installation script subprocess returned error exit status 1
unknown option "--skip-systemd-native"

@thaJeztah
Copy link
Member

thaJeztah commented May 19, 2022

And looks like others ran into that issue; sipwise/rtpengine#848

Have you tried with init-system-helpers from Bionic backports?

And this one https://lintian.debian.org/tags/skip-systemd-native-flag-missing-pre-depends

@thaJeztah
Copy link
Member

Seems related to https://man7.org/linux/man-pages/man7/debhelper.7.html

Retroactively removed in debhelper/13.5:
(...)
This change makes dh_installinit inject a
misc:Pre-Depends for init-system-helpers (>= 1.54~).
Please ensure that the package lists
${misc:Pre-Depends} in its Pre-Depends field before
upgrading to compat 12.

@tianon
Copy link
Contributor Author

tianon commented May 19, 2022

For continuity here, copying #699 (comment):

To unpack this a little bit more, I think to be able to reconsider what I was trying to do in #686 we'd really need to either drop support for bionic (not great as it's still very popular with users) or make it work somehow with compat level 10.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants