DAOS-18552 pool: Fix a PS start-stop race#17564
Merged
daltonbohning merged 1 commit intomasterfrom Mar 12, 2026
Merged
Conversation
|
Ticket title is 'rebuild/container_rf.py:RbldContRfTest.test_rebuild_with_container_rf - pool create failed: DER_BUSY(-1012): Device or resource busy' |
8d0122c to
4ccf463
Compare
The following race happened during a pool create operation, triggered by
abnormally slow VMs:
ds_rsvc_start
start
pool_svc_alloc_cb
ds_pool_lookup: OK
....VM slowness causes start timeout, which triggers stop....
ds_pool_stop
pool->sp_stopping = 1
ds_pool_svc_stop: none
insert
wait for ds_pool references: hang
This patch is a quick fix that prevents ds_rsvc_start from inserting a
PS to the hash table if the ds_pool is stopping, so that ds_pool_stop
won't hang. Manual testing shows that such a pool create operation will
now retry and succeed transparently.
Signed-off-by: Li Wei <liwei@hpe.com>
4ccf463 to
0ec1e95
Compare
kccain
approved these changes
Feb 18, 2026
| rc = rsvc_class(class)->sc_insert(svc); | ||
| if (rc != 0) { | ||
| D_DEBUG(DB_MD, "%s: sc_insert: " DF_RC "\n", svc->s_name, DP_RC(rc)); | ||
| goto err_svc_started; |
Contributor
There was a problem hiding this comment.
unusual goto, though I see the reasoning (do not duplicate stop call line of code here, and do not generate inaccurate D_DEBUG log when it is sc_insert that failed rather than d_hash_rec_insert)
Contributor
Author
There was a problem hiding this comment.
Yeah, it's a bit unusual, though I occasionally use this method. If I revise this PR next time, I'll change this to just duplicate the stop call.
liuxuezhao
approved these changes
Mar 4, 2026
Contributor
|
Removing gatekeeper until merge approval is granted |
Contributor
Author
|
Looks like this PR is now approved for 2.8. |
liw
added a commit
that referenced
this pull request
Mar 13, 2026
The following race happened during a pool create operation, triggered by
abnormally slow VMs:
ds_rsvc_start
start
pool_svc_alloc_cb
ds_pool_lookup: OK
....VM slowness causes start timeout, which triggers stop....
ds_pool_stop
pool->sp_stopping = 1
ds_pool_svc_stop: none
insert
wait for ds_pool references: hang
This patch is a quick fix that prevents ds_rsvc_start from inserting a
PS to the hash table if the ds_pool is stopping, so that ds_pool_stop
won't hang. Manual testing shows that such a pool create operation will
now retry and succeed transparently.
Signed-off-by: Li Wei <liwei@hpe.com>
daltonbohning
pushed a commit
that referenced
this pull request
Mar 18, 2026
The following race happened during a pool create operation, triggered by
abnormally slow VMs:
ds_rsvc_start
start
pool_svc_alloc_cb
ds_pool_lookup: OK
....VM slowness causes start timeout, which triggers stop....
ds_pool_stop
pool->sp_stopping = 1
ds_pool_svc_stop: none
insert
wait for ds_pool references: hang
This patch is a quick fix that prevents ds_rsvc_start from inserting a
PS to the hash table if the ds_pool is stopping, so that ds_pool_stop
won't hang. Manual testing shows that such a pool create operation will
now retry and succeed transparently.
Signed-off-by: Li Wei <liwei@hpe.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The following race happened during a pool create operation, triggered by abnormally slow VMs:
This patch is a quick fix that prevents ds_rsvc_start from inserting a PS to the hash table if the ds_pool is stopping, so that ds_pool_stop won't hang. Manual testing shows that such a pool create operation will now retry and succeed transparently.
Steps for the author:
After all prior steps are complete: