[scos10] `ostree.sync` kola test failed

[HuijingHei]

See coreos/coreos-assembler#4026 (comment)

Run ostree.sync failed on SCOS-10, seems stuck in systemd-shutdown and failed to reboot:

[  OK  ] Reached target reboot.target - System Reboot.
[  129.531632] systemd-shutdown[1]: Syncing filesystems and block devices - timed out, issuing SIGKILL to PID 4377.
[  129.537918] systemd-journald[1457]: Failed to send WATCHDOG=1 notification message: Connection refused
[  132.064914] systemd-shutdown[1]: Waiting for process: 4259 (dd), 4284 ((sd-sync)), 4243 (dd), 4377 ((sd-sync)), 4246 (dd)
[  137.077884] systemd-shutdown[1]: Waiting for process: 4377 ((sd-sync)), 4284 ((sd-sync)), 4243 (dd), 4259 (dd), 4246 (dd)

Seems it is not always failed, the failed rate is high with 4/5 (totally 5 times).

[aaradhak]

@HuijingHei We came across this issue as we were running through the list of issues in denyllist during our pipeline health sync yesterday, wanted to check if there is an update on this?

[HuijingHei]

Sorry that I was blocked by other things like confidential working on RHCOS, I will try to work on this soon.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[HuijingHei]

/remove-lifecycle stale

We need to do some research to find the root cause.

[aaradhak]

Hey @HuijingHei , we came across this issue during our health sync meeting, just wanted to check if you had the chance to look into this .

[HuijingHei]

I will put it to my plate, thank you for the reminder!

[HuijingHei]

Guess this is not related to ostree, might be systemd issue, will try to see if can reproduce on rhel.

[HuijingHei]

/label jira

[openshift-ci]

@HuijingHei: The label(s) /label jira cannot be applied. These labels are supported: acknowledge-critical-fixes-only, platform/aws, platform/azure, platform/baremetal, platform/google, platform/libvirt, platform/openstack, ga, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, px-approved, docs-approved, qe-approved, ux-approved, no-qe, downstream-change-needed, rebase/manual, cluster-config-api-changed, run-integration-tests, approved, backport-risk-assessed, bugzilla/valid-bug, cherry-pick-approved, jira/valid-bug, ok-to-test, stability-fix-approved, staff-eng-approved. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

Details

In response to this:

/label jira

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.