Skip to content

Fix ConfigNode leader transfer retries during removal - #18348

Merged
CRZbulabula merged 1 commit into
apache:masterfrom
CRZbulabula:yongzao/fix-config-node-leader-transfer-retry
Jul 29, 2026
Merged

Fix ConfigNode leader transfer retries during removal#18348
CRZbulabula merged 1 commit into
apache:masterfrom
CRZbulabula:yongzao/fix-config-node-leader-transfer-retry

Conversation

@CRZbulabula

Copy link
Copy Markdown
Contributor

Description

This change makes removing the current ConfigNode leader resilient to transient Ratis leadership transfer failures.

Previously, the removal pre-check selected a single running follower and failed immediately when transferLeader returned an error. A transient election timeout could therefore abort the removal even when another healthy ConfigNode was available. The code also redirected to the requested transfer target without verifying which ConfigNode was actually elected.

This PR:

  • tries running ConfigNode followers in deterministic node ID order
  • retries only transient RatisRequestFailedException failures
  • verifies the actual ConfigNode leader after every transfer attempt
  • redirects the removal request to the actual elected leader
  • bounds additional attempts by the remaining ConfigNode RPC timeout budget
  • adds unit coverage for fallback, ambiguous transfer results, exhausted candidates, and non-retriable errors
  • updates the ConfigNode removal integration test to explicitly remove the current leader

This fixes V2-1059.

Tests

  • mvn test -pl iotdb-core/confignode -Dtest=NodeManagerTest
  • mvn test-compile -DskipTests
  • mvn test-compile -DskipTests -P with-zh-locale
  • mvn verify -DskipUTs -Dit.test=IoTDBRemoveConfigNodeNormalIT -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false -pl integration-test -am -PClusterIT -P with-integration-tests

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves ConfigNode removal robustness by making leader-transfer handling resilient to transient Ratis transfer failures and by redirecting the request to the actual elected leader after transfer attempts.

Changes:

  • Enhance NodeManager.transferLeader to try all running followers in deterministic ConfigNode ID order, retrying only transient RatisRequestFailedExceptions and confirming the actual leader after each attempt.
  • Add unit tests covering transient failures, ambiguous transfer outcomes (transfer throws but leader changes), exhausted candidates, and non-retriable exceptions.
  • Update the ConfigNode removal integration-test framework to explicitly remove the current ConfigNode leader.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/node/NodeManager.java Adds deterministic multi-candidate leader-transfer attempts with transient-failure retry and post-transfer leader verification/redirect.
iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/manager/node/NodeManagerTest.java New unit tests validating retry/fallback behavior and correct redirection semantics.
iotdb-core/confignode/src/main/i18n/en/org/apache/iotdb/confignode/i18n/ManagerMessages.java Adds i18n log templates for new leader-transfer retry/verification warnings.
iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ManagerMessages.java Chinese i18n equivalents for the new log templates.
integration-test/src/test/java/org/apache/iotdb/confignode/it/removeconfignode/IoTDBRemoveConfigNodeITFramework.java Adjusts removal target selection to remove the current leader based on the environment’s leader wrapper and SHOW CONFIGNODES.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@CRZbulabula
CRZbulabula merged commit 847cc6d into apache:master Jul 29, 2026
41 of 42 checks passed
@CRZbulabula
CRZbulabula deleted the yongzao/fix-config-node-leader-transfer-retry branch July 29, 2026 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants