ssh.service failures on azure stemcells

[s4heid]

Occassionally, it happens that cloud-init reports issues regarding the startup of ssh.service. In the cloud-init log you would see the following error:

$ cloud-init status --long
status: done
extended_status: degraded done
boot_status_code: enabled-by-generator
last_update: Thu, 01 Jan 1970 00:00:27 +0000
detail: DataSourceAzure [seed=/dev/sr0]
errors: []
recoverable_errors:
WARNING:
	- Unhandled non-multipart (text/x-not-multipart) userdata: 'b'{"server":{"name":"agent'...'
	- 'ssh_pwauth' configuration may not be applied. Cloud-init was unable to restart SSH daemon due to error: 'Unexpected error while running command. Command: ['systemctl', 'restart', 'ssh', '--job-mode=ignore-dependencies'] Exit code: 1 Reason: - Stdout:  Stderr: Job for ssh.service failed because the control process exited with error code.         See "systemctl status ssh.service" and "journalctl -xeu ssh.service" for details.'

The journal log shows that the host keys could not be loaded, causing ssh.service to fail with exit status 1.

systemd[1]: Starting OpenBSD Secure Shell server...
░░ Subject: A start job for unit ssh.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit ssh.service has begun execution.
░░ 
░░ The job identifier is 307.
sshd[928]: Unable to load host key: /etc/ssh/ssh_host_rsa_key
sshd[928]: Unable to load host key: /etc/ssh/ssh_host_ecdsa_key
sshd[928]: Unable to load host key: /etc/ssh/ssh_host_ed25519_key
sshd[928]: sshd: no hostkeys available -- exiting.
systemd[1]: ssh.service: Control process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ An ExecStartPre= process belonging to unit ssh.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Nov 06 10:12:01 fb126bb7-ad45-4af4-a70e-8da723c21826 systemd[1]: ssh.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit ssh.service has entered the 'failed' state with result 'exit-code'.
systemd[1]: Failed to start OpenBSD Secure Shell server.
░░ Subject: A start job for unit ssh.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit ssh.service has finished with a failure.

Looking at the serial console of the VM, we are pointed to a file in cloud-init, cc_set_passwords.py:

[  OK  ] Started Run anacron jobs.
[   12.661141] cloud-init[912]: Cloud-init v. 25.2-0ubuntu1~22.04.1 running 'modules:config' at Mon, 17 Nov 2025 16:54:26 +0000. Up 12.58 seconds.
[  OK  ] Stopped OpenBSD Secure Shell server.
         Starting OpenBSD Secure Shell server...
[  OK  ] Stopped OpenBSD Secure Shell server.
         Starting OpenBSD Secure Shell server...
[FAILED] Failed to start OpenBSD Secure Shell server.
See 'systemctl status ssh.service' for details.
[  OK  ] Started Dispatcher daemon for systemd-networkd.
[   12.721602] cloud-init[912]: 2025-11-17 16:54:26,851 - cc_set_passwords.py[WARNING]: 'ssh_pwauth' configuration may not be applied. Cloud-init was unable to restart SSH daemon due to error: 'Unexpected error while running command.
[   12.729634] cloud-init[912]: Command: ['systemctl', 'restart', 'ssh', '--job-mode=ignore-dependencies']
[   12.733487] cloud-init[912]: Exit code: 1
[   12.735251] cloud-init[912]: Reason: -
[   12.736812] cloud-init[912]: Stdout:
[   12.738550] cloud-init[912]: Stderr: Job for ssh.service failed because the control process exited with error code.
[   12.743876] cloud-init[912]:         See "systemctl status ssh.service" and "journalctl -xeu ssh.service" for details.'
[  OK  ] Finished Cloud-init: Config Stage.
[  OK  ] Stopped OpenBSD Secure Shell server.
         Starting OpenBSD Secure Shell server...
[FAILED] Failed to start OpenBSD Secure Shell server.
See 'systemctl status ssh.service' for details.
[  OK  ] Stopped OpenBSD Secure Shell server.
         Starting OpenBSD Secure Shell server...
[FAILED] Failed to start OpenBSD Secure Shell server.
See 'systemctl status ssh.service' for details.
[  OK  ] Stopped OpenBSD Secure Shell server.
[FAILED] Failed to start OpenBSD Secure Shell server.

It looks like this is a recoverable error, as the ssh.service is active and running on the VM.

$ systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2025-11-17 16:50:28 UTC; 1 week 0 days ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 4066 (sshd)
      Tasks: 1 (limit: 9431)
     Memory: 3.9M
     CGroup: /system.slice/ssh.service
             └─4066 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

However, it appears that, besides cloud-init, another process might also be attempting to manage the SSH service at the same time, which occasionally results in these errors.

[rkoster]

Might be related to the first boot script which reconfigure ssh: https://github.com/cloudfoundry/bosh-linux-stemcell-builder/blob/ubuntu-noble/stemcell_builder/stages/base_ubuntu_firstboot/assets/root/firstboot.sh

[s4heid]

How I understand it, the firstboot script regenerates the host keys (triggered by rc.local). According to the logs, it looks like cloud-init’s config stage runs the ssh and set-passwords modules declared in cloud.cfg concurrently. Those modules call systemctl restart ssh --job-mode=ignore-dependencies in cloud-init's cc_set_passwords.py. Because the host keys have just been removed and have not yet been re-created, each restart attempt fails, as shown in the journal log (and you would see multiple attempts in the full log). If cloud-init waited until the host keys were regenerated, it might resolve the issue.

So yes, @rkoster, I agree that this is likely related to the firstboot script.