SessionPool blocks indefinitely after a server outage and never recovers 服务端断开后 SessionPool 永久阻塞且无法自愈

[CritasWang]

Summary / 概述

After the IoTDB server becomes unreachable, SessionPool stops raising errors and blocks callers
indefinitely. A SessionPool that survived an outage cannot be used again, and it does not recover even
after the server comes back — the process has to be restarted.

IoTDB 服务端不可达后，SessionPool 会停止抛出异常并让调用方永久阻塞。经历过一次断连的连接池无法继续
使用，且服务端恢复后也无法自愈，只能重启进程。

Reported by a user on Windows with IoTDB 2.0.10.
由 Windows + IoTDB 2.0.10 环境的用户反馈。

Reproduce / 复现步骤

1. Open a SessionPool against a running IoTDB (default PoolSize = 8).
   连接一个正常运行的 IoTDB 并 Open()（默认 PoolSize = 8）。
2. Stop the DataNode service to simulate an outage.
   停止 DataNode 服务以模拟异常。
3. Call InsertAlignedRecordAsync (or any write) in a loop.
   循环调用 InsertAlignedRecordAsync（或任意写入接口）。

Observed / 实际现象:

• The first ~8 calls throw SessionPoolDepletedException as expected.
  前约 8 次调用会按预期抛出 SessionPoolDepletedException。
• From the 9th call on, the await never completes. No exception, no catch, no timeout.
  从第 9 次调用起 await 永不返回：不抛异常、不进 catch、也不超时。
• IsOpen() keeps returning true throughout.
  全程 IsOpen() 始终返回 true。

Root cause / 根因

1. Pool wait timeout: milliseconds written, seconds read / 池等待超时的单位错误

SessionPool.Open() assigns a millisecond value:

_clients.Timeout = _timeout * 5;   // _timeout is SetConnectionTimeoutInMs, default 500

but ConcurrentClientQueue.Take() reads it back as seconds:

timeout = !Monitor.Wait(ClientQueue, TimeSpan.FromSeconds(Timeout));

500 * 5 = 2500 is therefore interpreted as 2500 seconds ≈ 41 minutes. Callers using the legacy
positional constructor with timeout: 10000 get 50000 seconds ≈ 13.9 hours. That is why the block
looks permanent.

Open() 按毫秒赋值，Take() 却按秒解释，500 * 5 = 2500 被当作 2500 秒 ≈ 41 分钟；旧的位置参数
构造函数传 timeout: 10000 时更是 13.9 小时——所以看起来像永久卡死。

2. Pool slots are lost on every failed reconnection / 重连失败导致槽位永久丢失

In ExecuteClientOperationAsync, when reconnection fails the dead client is discarded and never returned:

catch (ReconnectionFailedException reconnectEx)
{
    shouldReturnClient = false;   // slot is gone for good
    throw new SessionPoolDepletedException(...);
}

Nothing replenishes the pool, so capacity shrinks by one per failure. After PoolSize failures the queue
is empty and every subsequent Take() waits on a queue that nobody will ever feed — combined with defect
(1), that wait is effectively unbounded. Because all connections have been discarded, the pool cannot
recover even after the server is reachable again.

重连失败时失效连接被丢弃且不归还，池也没有补充机制，因此每失败一次容量就减一。PoolSize 次失败后队列
为空，之后每次 Take() 都在等一个永远不会有人投喂的队列——叠加缺陷 (1) 后等待时间实际上是无限的。
由于所有连接都已被丢弃，服务端恢复后连接池也无法自愈。

3. IsOpen() is widely mistaken for a health check / IsOpen() 被普遍误解为健康检查

IsOpen() is !_isClose, and _isClose only flips on explicit Open() / Close(). There is no
heartbeat, so a server going down never changes it. This is working as designed and matches the Java
client, but the name invites the following guard, which silently prevents the pool from ever being
rebuilt:

if (_pool != null && _pool.IsOpen()) return;   // short-circuits forever

IsOpen() 即 !_isClose，只有显式 Open() / Close() 会改变它；客户端无心跳，服务端断开不会回写。
这是符合设计的（与 Java 客户端一致），但命名容易引起误用，上面这种守卫会导致连接池永远无法被重建。

Expected / 期望行为

• An operation against a dead server fails within a bounded, configurable time.
  对已断开服务端的操作应在有界且可配置的时间内失败。
• Pool capacity stays at PoolSize; failed connections are rebuilt rather than leaked.
  池容量应保持为 PoolSize，失效连接应被重建而不是泄漏。
• Once the server is reachable again, the pool recovers without Close() + Open().
  服务端恢复后，连接池应无需 Close() + Open() 即可自行恢复。
• The lifecycle-only semantics of IsOpen() are documented.
  IsOpen() 仅表示生命周期的语义应在文档中说明。

Workaround before the fix / 修复前的规避方案

• Do not use IsOpen() as a connectivity check.
  不要用 IsOpen() 判断连通性。
• Wrap operations in your own timeout (Task.WhenAny + Task.Delay) so business threads are not pinned.
  用 Task.WhenAny + Task.Delay 自行加超时，避免业务线程被挂死。
• On SessionPoolDepletedException, explicitly Close() and re-Open() the pool.
  捕获 SessionPoolDepletedException 后显式 Close() 并重新 Open()。

Environment / 环境

• Client: Apache.IoTDB C# SDK, main @ 8214361
• Server: IoTDB 2.0.10, Windows