Makes user compaction metadata independent.#3578
Merged
Merged
Conversation
Before this change user initiated compactions stored config in a single place in zookeeper and had a single counter in each tablet. This system made it impossible for concurrent user initiated compactions to run on the same table with different configuration. This commit adds compaction config storage and tablet metadata that has 1:1 correspondence with the fate operation driving a user initiated compaction. This allows each user initiated compaction to be independent. This new system also avoid some odd race conditions with tablet metadata updates that existed with old per tablet compaction counter. Instead of a counter each tablet now conceptually has a set of the fate transaction ids that have completed a compaction. See the new documention in TableOperations.compact() for an example. fixes apache#3517
Contributor
Author
|
This commit also implements compaction cancellation and fixes a few misc compaction bugs in the elasticity branch. |
Contributor
Author
|
All tests in CompactionIT are passing with these changes. |
DomGarguilo
reviewed
Jul 12, 2023
…tadataSchema.java Co-authored-by: Dom G. <domgarguilo@apache.org>
…Operations.java Co-authored-by: Dom G. <domgarguilo@apache.org>
Contributor
Author
|
Merged this because I have another PR that builds on this to remove code from the tserver and I need to get that change in so I can start making changes to improve the tablet server code. If anyone was looking at this feel free to comment still, I can address any comments in follow on commits or issues. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this change user initiated compactions stored config in a single place in zookeeper and had a single counter in each tablet. This system made it impossible for concurrent user initiated compactions to run on the same table with different configuration. This commit adds compaction config storage and tablet metadata that has 1:1 correspondence with the fate operation driving a user initiated compaction. This allows each user initiated compaction to be independent. This new system also avoids some odd race conditions with tablet metadata updates that existed with old per tablet compaction counter. Instead of a counter each tablet now conceptually has a set of the fate transaction ids that have completed a compaction.
See the new documention in TableOperations.compact() for an example.
fixes #3517