Fix disk partitioning race condition and using partition number 0

[chewi]

I've admittedly lost the output of the race condition triggering, but this is what was going on underneath.

$ partx --add --nr 1 --verbose /dev/nvme0n1
partition: none, disk: /dev/nvme0n1, lower: 1, upper: 1
/dev/nvme0n1: partition table type 'gpt' detected
range recount: max partno=9, lower=1, upper=1
partx: /dev/nvme0n1: adding partition #1 failed: Device or resource busy
partx: /dev/nvme0n1: error adding partition 1

We started reliably seeing this in Flatcar after some batch updates. We don't know exactly which update triggered it, but it was probably systemd. While we could have looked into systemd's changes, I strongly felt that this code was always potentially racy. There was never anything stopping the kernel picking up the partition changes before partx had a chance to run.

This change therefore allows partx to fail and then checks that added/updated partitions have the right start sector and size and that deleted partitions are absent once udev has settled.

On first submitting this change, Gemini highlighted that I had broken the feature that allows you to specify partition number 0 to get the next available slot. On testing this, I found that this was already broken since c2cc56c. Passing 0 to partx causes it to try and add all the partitions, which will almost always fail because the kernel will usually already know about at least some of them.

If anything, my initial change had improved the situation by ignoring the partx failure, but I have now fixed the issue properly. This changes getRealStartAndSize() to also determine and return the resulting partition numbers so that subsequent operations use these instead of 0.

sgdisk does support --new=0, but it has no way to report which partition number it actually used.

Following that, I was able to drop the restriction that prevented users from deleting a partition while creating partition number 0. We previously disallowed this because we didn't resolve the partition numbers to their final values, making it impossible to determine whether they would actually exist in the end.

The error message that was shown mentioned partitions having a start or size of 0 rather than the number. It's not clear why this was.

New cases have been added to tests/positive/partitions/complex-mb.go to cover all this. The creation of the new-auto3 partition triggers the race condition on Fedora 44. I don't know why just this one does, but at least that confirms the fix works.

[gemini-code-assist]

Code Review

This pull request modifies the disk partitioning logic to validate partition starts and sizes against sysfs after partitioning, and updates partition number types from uint64 to int. However, several critical issues were identified in the review. Changing getRealStartAndSize to return a map instead of a slice introduces non-deterministic ordering and causes partition overwrites when multiple partitions use number 0. Additionally, the new validation logic, partx commands, and deletion checks do not account for partition 0 (which represents the next available partition and does not exist in sysfs or /dev), leading to potential runtime failures. Reverting the map changes to slices and skipping partition 0 in sysfs/partx operations is recommended.

[chewi]

I see that 3 FCOS Kola tests have failed. Flatcar's own Kola tests have passed. I would run the FCOS Kola tests myself, but CI doesn't seem to expose the build. I'd prefer not to get tangled up with the FCOS build process. Do you have any suggestions on how I can work through this?

[chewi]

I've managed to fix 2 of the 3 failures. The remaining one is ext.config.multipath.custom-partition. Help would still be appreciated.

[chewi]

I've realised that FCOS hasn't been hitting this race condition because partx was only run when the disk has partitions in use. This does happen with Flatcar because it sets up dm-verity early and, more recently, it also mounts a second stage initrd from the /usr partition. Now that we ignore partx failures, I've decided to run it unconditionally just in case.

Regarding the failing test, I ended up building FCOS myself. It actually worked with these changes, although I was running it manually rather than with Kola, so maybe there's a subtle difference there.

One partx invocation was failing due to the race condition, and that was being flagged up after booting was complete, so I've downgraded the failure from error to notice to hide that and also in the hope that it fixes the test. I'm doubtful though because CI said it was hitting the emergency shell. Let's find out.

[chewi]

Okay, it's easier to run that test with Kola though I expected. When I run it manually, coreos-boot-disk is a symlink to vda. Under Kola, it appears to be dm-0, which totally confuses things because it starts looking for dm-0p4. I have no idea why.

[chewi]

Right, I now understand this is multipath. For my own reference:

kola qemuexec --qemu-multipath --add-ignition autologin --kargs rd.multipath=default

I'll try to fix up the code to work with device mapper.

[chewi]

Aaaah, it's finally passed. 😌 The device mapper bits needed very different handling, and even some of the existing code wasn't handling it properly, so I got Copilot to help me out a bit here.

[prestist]

From my perspective this LGTM, thank you for finding and fixing that hidden bug!

From my perspective, I believe it to be okay to update the stable specs in this case, as its removing of validation that was bug'd. However I want to double check with @travier in case I am missing something.

[prestist]

LGTM, thank you for your work on this :)

@travier LMK if you see any reason not to merge.

[travier]

Very light review but looks sane to me. Thanks