[SPARK-31197][CORE] Shutdown executor once we are done decommissioning#29211
Closed
holdenk wants to merge 20 commits into
Closed
[SPARK-31197][CORE] Shutdown executor once we are done decommissioning#29211holdenk wants to merge 20 commits into
holdenk wants to merge 20 commits into
Conversation
Loading
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Exit the executor when it has been asked to decommission and there is nothing left for it to do.
This is a rebase of #28817
Why are the changes needed?
If we want to use decommissioning in Spark's own scale down we should terminate the executor once finished.
Furthermore, in graceful shutdown it makes sense to release resources we no longer need if we've been asked to shutdown by the cluster manager instead of always holding the resources as long as possible.
Does this PR introduce any user-facing change?
The decommissioned executors will exit and the end of decommissioning. This is sort of a user facing change, however decommissioning hasn't been in any releases yet.
How was this patch tested?
I changed the unit test to not send the executor exit message and still wait on the executor exited message.