feat: prune excess defunct membership entries - #10231
Merged
ReubenBond merged 10 commits intoJun 17, 2026
Merged
Conversation
Add a configurable limit for defunct membership table entries and move cleanup into MembershipTableCleanupAgent so the first active silo asynchronously removes excess dead entries on membership changes. Preserve the nullable option semantics so null disables threshold-based cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Simplify defunct membership cleanup comparison helpers by storing the MembershipEntry directly in the priority wrapper and inlining the dead-status check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Inline the defunct cleanup cutoff calculation into the cleanup flow and simplify logging to avoid carrying counters used only for log messages. Rename the shutdown helper to clarify that it signals shutdown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Calculate threshold cleanup cutoffs from the membership update snapshot instead of reading all membership table entries, and remove the special int.MaxValue guard by using a non-overflowing tracked-entry count. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the no-op health-check participant registration from MembershipTableCleanupAgent and move defunct entry priority comparison into DefunctSiloEntryPriority using comparison operators. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Treat membership updates as notifications and use IMembershipManager.CurrentSnapshot for first-active ownership and defunct cleanup cutoff calculations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoid calling CleanupDefunctSiloEntries on every membership change by checking for expired non-active entries or elapsed DefunctSiloExpiration since the last cleanup call, while preserving threshold cleanup behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the local MaxDefunctSiloEntries negative-value guard from MembershipTableCleanupAgent since option validation is handled by SiloClusteringValidator. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds threshold-based pruning of defunct silo membership entries to prevent membership tables from growing excessively between age-based cleanup runs. It introduces a new ClusterMembershipOptions.MaxDefunctSiloEntries setting and updates MembershipTableCleanupAgent to perform cleanup opportunistically in response to membership updates (owned by the first active silo by SiloAddress sort order), with validation and test coverage.
Changes:
- Add
ClusterMembershipOptions.MaxDefunctSiloEntries(nullable) and validate it is>= 0(ornullto disable). - Rework
MembershipTableCleanupAgentto trigger cleanup on membership updates (expiration-based and/or threshold-based) and gate ownership to the first active silo. - Add/update unit tests and test infrastructure hooks for cleanup invocation.
Show a summary per file
| File | Description |
|---|---|
| test/Orleans.Core.Tests/SiloBuilderTests.cs | Adds config validation test for negative MaxDefunctSiloEntries. |
| test/Orleans.Core.Tests/Membership/MembershipTableCleanupAgentTests.cs | Adds tests for non-owner silo behavior and threshold cleanup; refactors cleanup agent test setup. |
| test/Orleans.Core.Tests/Membership/InMemoryMembershipTable.cs | Adds a callback hook for CleanupDefunctSiloEntries to support new tests. |
| src/Orleans.Runtime/MembershipService/MembershipTableCleanupAgent.cs | Replaces timer-based cleanup with membership-update-driven cleanup and adds threshold-based pruning logic. |
| src/Orleans.Runtime/Hosting/DefaultSiloServices.cs | Removes MembershipTableCleanupAgent health check registration (since it no longer implements it). |
| src/Orleans.Runtime/Configuration/Validators/SiloClusteringValidator.cs | Validates MaxDefunctSiloEntries is non-negative when specified. |
| src/Orleans.Core/Configuration/Options/ClusterMembershipOptions.cs | Adds the new option and updates docs for cleanup behavior. |
| src/api/Orleans.Core/Orleans.Core.cs | Updates public API surface to include MaxDefunctSiloEntries. |
Copilot's findings
- Files reviewed: 8/8 changed files
- Comments generated: 3
Wait for the cleanup task to complete in the enabled cleanup-agent scenario instead of racing a fixed timeout, while preserving the bounded wait for disabled behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use DefunctSiloCleanupPeriod, not DefunctSiloExpiration, as the elapsed-time throttle between expiration cleanup calls while preserving eager first cleanup behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large numbers of dead silo entries can cause membership tables to grow before age-based cleanup removes them.
This adds a nullable maximum retained-defunct-entry option and has MembershipTableCleanupAgent perform threshold-based asynchronous cleanup on membership changes. Cleanup is owned by the first active silo by natural SiloAddress sort order, reuses the existing membership table cleanup API, and can be disabled by setting the option to null.
Microsoft Reviewers: Open in CodeFlow