Migrate SecureDrop servers to Ubuntu 16.04 (Xenial)

[eloquence]

Ubuntu Trusty is reaching EOL in April 2019. We should upgrade SecureDrop servers to Ubuntu Xenial (16.04) before then. This will also unblock some blocked issues.

(Please keep discussion about moving to 18.04 out of scope of this issue. We will consider the best path to 18.04, but will not immediately go from 14.04 to 18.04.)

Initially this epic captures preliminary only work; we will update it as we discover more work. The preliminary work must only impact the development environment and must not have production consequences.

Tasks:

• Timeboxed install attempt ([xenial] Perform timeboxed install attempt of SecureDrop against Ubuntu 16.04 #3207)
• Timeboxed upgrade attempt ([xenial] Perform timeboxed upgrade attempt of SecureDrop from Ubuntu 14.04 to 16.04 #3491)
• Add Xenial staging VMs ([xenial] Add staging VMs for Ubuntu 16.04 #3206)
• Enable installation on Xenial staging VMs ([xenial] Enable installation on Xenial staging VMs #3825)
• Add host firewall support for _apt user introduced in 16.04 ([xenial] Add host firewall configuration change to permit _apt user to fetch packages #3952)
• Update AppArmor rules to work on Trusty, Xenial ([xenial] Update AppArmor rules to work on Trusty, Xenial #3962)
• Investigate if PAM common-auth customizations are still needed ([xenial] Investigate if PAM common-auth customizations are still needed #3963)
• Support branched Trusty/Xenial packaging build logic ([xenial] Support branched Trusty/Xenial packaging build logic #3961)
• Support running CI staging tests against both 14.04 and 16.04 ([xenial] Support running CI staging tests against both 14.04 and 16.04 #3966)
• Investigate/fix apt sources under xenial when upgrading ([xenial] Investigate/fix apt sources under xenial when upgrading  #4017)
• Display advisory in Journalist Interface for instances running 14.04 ([xenial] Display advisory in Journalist Interface for instances running 14.04 (Trusty) #4027)
• Resolve i18n related test failures [xenial] resolve i18n related test failures #4036
• Fix 2FA functionality in app code [xenial] fix 2FA functionality in app code #4037
• Fix Unknown status message: u'IMPORT_OK' related test failures - [xenial] fix Unknown status message: u'IMPORT_OK' related test failures #4038
• Fix errors in functional tests - [xenial] fix errors in functional tests #4039
• Add package/version information to securedrop-admin logs command ([xenial] Add package/version information to securedrop-admin logs command  #3967)

In current sprint

• Verify Trusty backup -> Xenial recovery story ([xenial] Verify Trusty backup -> Xenial recovery story #3960)
• Test upgrade path: Ubuntu 14.04.5 → 14.04.5 with all updates → 16.04 ([xenial] Test upgrade path: Ubuntu 14.04.5 → 14.04.5 with all updates → 16.04 #3965)
• Journalist replies are not displayed in source interface ([xenial] Journalist replies are not displayed in source interface #4013)
• Handle import_ok exception during upgrade ([xenial] Handle import_ok exception during upgrade #4108)
• Ensure TestInfra test completeness on Trusty, Xenial ([xenial] Ensure TestInfra test completeness on Trusty, Xenial #3964)
• Document the upgrade process ([xenial] Document upgrade process #4057)
• Reinstate LTS upgrade notification ([xenial] Reinstate LTS upgrade notification #4104)
• Update the install guide to account for the use of Xenial ([xenial] Update the install guide to account for the use of Xenial instead of Trusty as the server base OS #4103)

Stretch goals

• Implement nightly CI upgrade test from 14.04 to 16.04 ([xenial] Implement nightly CI upgrade test from 14.04 to 16.04 #3969)

[conorsch]

Config tests are passing against Xenial. Will run through a few repeated runs on both Trusty and Xenial to make sure there aren't any test flakes. On the subject of flakes, over in #3206 (comment) I noted:

OSSEC (the no-messages-containing "ERROR: Incorrectly formated message from" check is failing; that's bad, and could take a while to debug).

Pleased to report that was a false alarm! The corrupted messages were caused by my running a separate app-staging VM concurrently, for use with developing the SecureDrop Workstation. Since the app-staging IPv4 addresses are hard-coded in staging, the machines were fighting, and OSSEC was correctly reporting bad messages due to inconsistent key exchange. Powering off the secondary app-staging machine and rebuilding resolved all OSSEC-related errors i nthe config tests.

[heartsucker]

Hey, so this came up when chatting with @redshiftzero recently, and I figure I'll leave my thoughts here.

We had a timeboxed upgrade attempt from Trusty to Xenial, but I think we should not do this on production boxes. I have a few times upgraded laptops from one Debian version to the immediate next using apt dist-upgrade and fiddling with the sources.list{,.d} entries. This has worked in as much as I had a functional operating system and most things "just worked" as they did after the upgrade and next reboot.

Where things didn't work in cases where a piece of software was included as the default in version N-1 but not in version N. I mean this specifically in the case of Jessie to Wheezy when the OS went from init to systemd. This was fine because there was compatibility with all the packages, but it made debugging a lot harder because searches like wheezy upstart tor or whatever just had crap results.

It's also possible to do a bunch of manual fiddling to get the system in to the next state by installing, removing, updating alternatives, and updating the default configs, but this is a lot of work. I hesitate to suggest we do this because one problem we have with SD is that right now it's treated like a package/service when in reality it's more an appliance. Which is to say, we don't have full control or even introspection into the base OS, so if things get a little out of whack, we just don't know.

I say we should tell admins that they need to do a full reinstall and give them one (two?) releases of SD in which to do that in where we support both Trusty and Xenial. We already have the backup / restore script that does the magic of replacing the files, so this not so burdensome.

My biggest concern isn't that we botch the Trusty to Xenial upgrade, and in fact I bet we could totally nail it. The concern is that it's complex and that knowledge will be lost quickly (people forget, and likely only 2-3 people on the team will really fully understand it), and then future change will have to remember that systems could be fresh Xenials or upgrade Xenials when considering all ops / app related problems.

Another advantage of just mandating that we do reinstalls is that it's less engineering effort on our side, and this deadline feels very close. Given this constraint, it seems like a safer choice even if we disregard the long term complexities I mentioned above.

Also, I'm acknowledging that I haven't been paying super close attention to this ticket/epic and we may be too far down the line to change this now. Or we may have addressed these concerns already, and what I'm saying here is out of date.