Fixed multiple configNode bugs by Caideyipi · Pull Request #17609 · apache/iotdb

Caideyipi · 2026-05-06T08:57:51Z

Description

In Simple mode, a successful response will be returned stating "Metadata has taken effect, but WAL has not been flushed to disk." Configuration changes will be lost after a restart.
In the Simple mode, some files have the same sorting key, posing a risk of log replay in disorder.
The return TSStatus of ConsensusManager.write() is not checked, and the exception is only a warn followed by proceeding to the success branch. This means that if consensus writing is rejected, redirected, or the state machine returns a failure, the caller will still receive ACCEPT, while the local heartbeat cache/metrics have already been created in advance, leading to a divergence between metadata and runtime states.
The value returned by getRegionGroupCount() increases by 1 every time it is counted, and this value is used for capacity expansion judgment, display, and maxRegionGroupNum calculation, which directly affects the region scaling decision.
When taking a snapshot, the regionMaintainTaskList lock is not acquired before traversing. Concurrent offer/poll may throw ConcurrentModificationException or put the maintenance queue snapshot into an inconsistent state.
If the pipe listener fails to process the snapshot, the entire takeSnapshot() method will return false; however, ConfigRegionStateMachine combines and absorbs the failures of loadSnapshot and the pipe listener, which can amplify auxiliary link failures into consensus snapshot failures, and even mask the real failure of snapshot loading.

This PR has:

Key changed/added classes (or packages if there are too many classes) in this PR

codecov · 2026-05-06T10:01:18Z

Codecov Report

❌ Patch coverage is 44.40154% with 144 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.33%. Comparing base (a8497be) to head (a12b657).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
...che/iotdb/confignode/manager/node/NodeManager.java	42.74%	75 Missing ⚠️
...nsensus/statemachine/ConfigRegionStateMachine.java	44.91%	65 Missing ⚠️
...onfignode/persistence/partition/PartitionInfo.java	71.42%	2 Missing ⚠️
...de/procedure/impl/node/AddConfigNodeProcedure.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #17609      +/-   ##
============================================
- Coverage     43.40%   43.33%   -0.08%     
  Complexity      374      374              
============================================
  Files          5366     5364       -2     
  Lines        382885   383319     +434     
  Branches      49796    49880      +84     
============================================
- Hits         166187   166104      -83     
- Misses       216698   217215     +517

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

CRZbulabula

Overall

This PR fixes 6 real correctness bugs in ConfigNode (WAL ordering, log sorting, unchecked consensus writes, off-by-one, concurrent snapshot, pipe-listener failure masking). Changes are well-structured, rollback logic is carefully designed, and test coverage for core scenarios is solid.

Inline comments below cover a few items worth attention before merge.

CRZbulabula · 2026-05-07T09:22:31Z

+      if (persistStatus.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+        return persistStatus;
+      }
+    }


WAL replay idempotency requirement

With the new write-ahead ordering (persist → execute), if persistPlanForSimpleConsensus succeeds at line 125 but executor.executeNonQueryPlan fails at line 133, the plan is already persisted in WAL and will be replayed on restart.

This is correct WAL semantics, but it introduces a hard requirement: all ConfigPhysicalPlan implementations must be idempotent under replay. For example, a RegisterDataNodePlan replayed against an already-registered node must not fail or double-register.

Is this idempotency property currently guaranteed across all ConfigPhysicalPlan types? If not, this could cause issues on crash recovery in edge cases.

Addressed in cddb4b1. A plan whose execution fails is now removed from the SimpleConsensus WAL by truncating and force-syncing back to the pre-write offset, so normal rollback leaves nothing to replay. ConfigRegionStateMachineTest.testFailedSimpleConsensusWriteRollsBackPersistedPlan verifies the WAL contains no replayable plan. For the registration-related plans touched here, NodeInfoTest.testRegistrationPlansAreIdempotentForWalReplay also executes RegisterDataNodePlan, ApplyConfigNodePlan, and UpdateVersionInfoPlan twice and verifies stable state. If truncation itself fails, the status and operator log explicitly warn that replay remains possible.

CRZbulabula · 2026-05-07T09:22:31Z

      ByteBuffer buffer = plan.serializeToByteBuffer();
      buffer.position(buffer.limit());
      simpleLogWriter.write(buffer);
+      simpleLogWriter.force();


force() on every write makes scheduled flush redundant

Now that every write is synchronously forced here, the scheduled flushWALForSimpleConsensus thread (lines 447-455) becomes redundant — it calls simpleLogWriter.force() periodically, but there's nothing left to flush.

This is harmless (double-force is idempotent) but worth either:

Removing the scheduled thread to avoid confusion, or

Adding a brief comment explaining the intentional belt-and-suspenders approach.

For ConfigNode metadata operations (low write frequency), the per-write force() is the right durability choice.

Addressed. The scheduled flush executor and flushWALForSimpleConsensus method were removed; each persisted write is synchronously forced, and the writer is now closed from stop().

CRZbulabula · 2026-05-07T09:22:31Z

+          "Persist current ConfigPhysicalPlan for ConfigNode SimpleConsensus mode failed", e);
+      return new TSStatus(TSStatusCode.EXECUTE_STATEMENT_ERROR.getStatusCode())
+          .setMessage(
+              "Persist ConfigNode SimpleConsensus log failed: " + String.valueOf(e.getMessage()));


Nit: String.valueOf(e.getMessage()) is redundant — string concatenation with + already handles null by converting to "null". Can simplify to:

"Persist ConfigNode SimpleConsensus log failed: " + e.getMessage()

Addressed in the rewritten persistence error path: the message now directly concatenates e.getMessage().

CRZbulabula · 2026-05-07T09:22:31Z

+      final List<RegionMaintainTask> copiedRegionMaintainTaskList;
+      synchronized (regionMaintainTaskList) {
+        copiedRegionMaintainTaskList = new ArrayList<>(regionMaintainTaskList);
+      }


Good fix — synchronized copy before iterating. One minor observation: this copies the entire list under the lock. If regionMaintainTaskList can grow very large, this could cause a brief pause. In practice, for ConfigNode this list is typically small, so this is fine. Just noting for future awareness.

Thanks for noting this. The snapshot path intentionally copies under the lock to obtain a consistent list, then performs serialization after releasing the lock. As noted, this list is expected to remain small for ConfigNode.

CRZbulabula

PTAL.

CRZbulabula · 2026-05-07T10:08:17Z

              }
            });
-    return result.getAndIncrement();
+    return result.get();


Is this a bug?

Yes, this was an off-by-one side effect: getAndIncrement() returned the correct count but unnecessarily advanced the local counter before returning. It now uses get(), and PartitionInfoTest.testRegionGroupCount covers both schema and data region counts.

# Conflicts: # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/statemachine/ConfigRegionStateMachine.java # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/node/AddConfigNodeProcedure.java

luoluoyuyu

Review summary

Fixes several ConfigNode correctness issues including WAL persist-before-execute and rollback on execute failure. Before merge, document or test that all ConfigPhysicalPlan types are idempotent under WAL replay when persist succeeds and execute fails.

Also resolve CI failures in this PR.

luoluoyuyu · 2026-05-27T08:03:56Z

  protected TSStatus write(ConfigPhysicalPlan plan) {
+    SimpleConsensusPersistResult persistResult = null;
+    if (ConsensusFactory.SIMPLE_CONSENSUS.equals(CONF.getConfigNodeConsensusProtocolClass())) {
+      persistResult = persistPlanForSimpleConsensus(plan);


With persist before execute, a successful persistPlanForSimpleConsensus followed by a failed executeNonQueryPlan leaves the plan in WAL and it will be replayed on restart. Every ConfigPhysicalPlan executed on replay must be idempotent. Please add an audit or tests for the plan types touched by this PR.

Addressed. The failed-write test audits the persist-success/execute-failure path and verifies rollback leaves no replayable WAL entry. I also added replay idempotency coverage for the plan types directly touched by the registration changes: RegisterDataNodePlan, ApplyConfigNodePlan, and UpdateVersionInfoPlan.

luoluoyuyu · 2026-05-27T08:03:56Z

+
    TSStatus result;
    try {
      result = executor.executeNonQueryPlan(plan);


rollbackFailedPlanForSimpleConsensus on execute failure is the right safety net. Ensure rollback failure paths are logged clearly enough for operators when WAL and in-memory state may diverge.

Addressed. Rollback failure logging now includes the plan type, WAL file, truncate offset, and pre-write end index. The returned status also explicitly says that the persisted plan may replay after restart, so both operators and callers can see the possible WAL/in-memory divergence.

# Conflicts: # iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/path/PartialPath.java

Caideyipi · 2026-06-03T03:47:06Z

Addressed the review feedback and pushed the updates.

Changes:

Added WAL rollback/replay coverage: failed SimpleConsensus writes now assert that the rolled-back WAL contains no replayable plan.
Added idempotency coverage for replay-relevant NodeInfo plans touched by this PR: RegisterDataNodePlan, ApplyConfigNodePlan, and UpdateVersionInfoPlan.
Improved SimpleConsensus rollback failure logging with plan type, log file, truncate offset, and endIndexBeforeWrite; the returned status now also notes that the persisted plan may replay after restart if rollback fails.
Removed newly introduced TODO comments; todo-check is now passing.
Merged latest master and resolved the PartialPath conflict by keeping master's implementation, so the PR is mergeable again.

Verification:

mvn -pl iotdb-core/confignode spotless:apply
mvn -pl iotdb-core/node-commons spotless:apply
mvn -pl iotdb-core/confignode '-Dtest=ConfigRegionStateMachineTest,NodeInfoTest' test
git diff --check origin/master...HEAD

Current CI note: the Simple (17) failures were checkout/network failures on self-hosted runners (gnutls_handshake before tests started). I reran the failed jobs. todo-check has passed; the remaining jobs are still pending.

sonarqubecloud · 2026-06-03T03:47:57Z

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

# Conflicts: # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/consensus/statemachine/ConfigRegionStateMachine.java # iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/executor/ConfigPlanExecutor.java

Caideyipi · 2026-07-30T07:20:54Z

Verified that the review changes already present in cddb4b1c962 remain intact after the latest master merge, and replied to every inline thread without resolving them.

Current verification:

ConfigRegionStateMachineTest: 4 passed
NodeInfoTest: 2 passed
PartitionInfoTest: 5 passed

The current Windows DataNode UT failure is unrelated to this PR: the runner hit Java heap OOM, followed by a timing-sensitive L1PriorityQueueTest.testPollBlocked failure; 4,420 tests ran and the PR changes do not touch that code. The Sonar Reliability-C issues are also reported in files outside this PR's 11-file diff after the master merge, so there is no additional #17609 code change to apply for those checks.

sonarqubecloud · 2026-07-31T04:21:39Z

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Fixed multiple config bugs

2047d3a

CRZbulabula reviewed May 7, 2026

View reviewed changes

CRZbulabula requested changes May 7, 2026

View reviewed changes

Caideyipi added 4 commits May 22, 2026 14:02

fix

1d9fe36

fix

1b86d98

Update ConfigRegionStateMachine.java

2e4c652

This comment was marked as outdated.

Sign in to view

luoluoyuyu suggested changes May 27, 2026

View reviewed changes

Caideyipi added 2 commits June 3, 2026 11:30

Address ConfigNode review feedback

cddb4b1

Merge remote-tracking branch 'origin/master' into config-bug

65ae336

# Conflicts: # iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/path/PartialPath.java

Caideyipi added 5 commits June 29, 2026 14:55

Fix compile

c39aed2

Merge branch 'master' of https://github.com/apache/iotdb into config-bug

82cbdb2

Compile fix

bfc80d6

Merge master into config-bug

f4b0fa4

Fix NodeManager Sonar issues

a12b657

Uh oh!

Conversation

Caideyipi commented May 6, 2026

Description

Key changed/added classes (or packages if there are too many classes) in this PR

Uh oh!

codecov Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

CRZbulabula left a comment

Choose a reason for hiding this comment

Overall

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CRZbulabula left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

luoluoyuyu left a comment

Choose a reason for hiding this comment

Review summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Caideyipi commented Jun 3, 2026

Uh oh!

sonarqubecloud Bot commented Jun 3, 2026

Quality Gate passed

Uh oh!

Caideyipi commented Jul 30, 2026

Uh oh!

sonarqubecloud Bot commented Jul 31, 2026

Quality Gate failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented May 6, 2026 •

edited

Loading