stopping pool via systemd, causes backtrace, leaves pool in enabled state

[calestyo]

Hey.

I've noted the following behaviour on 11.2.3, with java 21.0.11 while debugging some stuff for my Prometheus exporter...

I have a running pool, and stop the systemd service, which causes:

2026-07-14T05:40:35.678539+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:35+02:00 (c-core0-AAZSrLxVP-A) [lcg-lrz-dcache0.grid.lrz.de/[2001:4ca0:0:201:a:0:0:0]:11111] Connection to core0 (lcg-lrz-dcache0.grid.lrz.de/[2001:4ca0:0:201:a:0:0:0]:11111) interrupted. Reason: java.lang.InterruptedException
2026-07-14T05:40:35.678931+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:35+02:00 (c-core0-AAZSrLxVP-A) [lcg-lrz-dcache0.grid.lrz.de/[2001:4ca0:0:201:a:0:0:0]:11111] Sleeping 13 seconds
2026-07-14T05:40:35.679041+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:35+02:00 (c-core2-AAZSrLxUxsg) [lcg-lrz-dc51.grid.lrz.de/[2001:4ca0:0:201:a:0:2:33]:11111] Connection to core2 (lcg-lrz-dc51.grid.lrz.de/[2001:4ca0:0:201:a:0:2:33]:11111) interrupted. Reason: java.lang.InterruptedException
2026-07-14T05:40:35.679167+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:35+02:00 (c-core2-AAZSrLxUxsg) [lcg-lrz-dc51.grid.lrz.de/[2001:4ca0:0:201:a:0:2:33]:11111] Sleeping 7 seconds
2026-07-14T05:40:35.679605+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:35+02:00 (c-core1-AAZSrLxVKHE) [lcg-lrz-dc50.grid.lrz.de/[2001:4ca0:0:201:a:0:2:32]:11111] Connection to core1 (lcg-lrz-dc50.grid.lrz.de/[2001:4ca0:0:201:a:0:2:32]:11111) interrupted. Reason: java.lang.InterruptedException
2026-07-14T05:40:35.679800+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:35+02:00 (c-core1-AAZSrLxVKHE) [lcg-lrz-dc50.grid.lrz.de/[2001:4ca0:0:201:a:0:2:32]:11111] Sleeping 16 seconds
2026-07-14T05:40:35.715768+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:35+02:00 (lcg-lrz-dc03_11) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server,dead): Shutdown

here, the systemctl stop still hangs... i.e. the process isn't dead yet....

2026-07-14T05:40:40.686332+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:40+02:00 (System) [] Still waiting for c-core0-AAZSrLxVP-A to shut down.
2026-07-14T05:40:40.686579+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:40+02:00 (System) [] Still waiting for c-core2-AAZSrLxUxsg to shut down.
2026-07-14T05:40:40.686668+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:40+02:00 (System) [] Still waiting for c-core1-AAZSrLxVKHE to shut down.
2026-07-14T05:40:43.678815+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:43+02:00 (System) [] Killed c-core2-AAZSrLxUxsg in 8007 ms
2026-07-14T05:40:45.687931+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:45+02:00 (System) [] Thread Group "c-core0-AAZSrLxVP-A-threads":
2026-07-14T05:40:45.688224+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: Thread: TunnelConnector-core0 [A--] (5) TIMED_WAITING
2026-07-14T05:40:45.688313+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.sleep0(Native Method)
2026-07-14T05:40:45.688386+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.sleep(Thread.java:509)
2026-07-14T05:40:45.688481+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.network.LocationManagerConnector.run(LocationManagerConnector.java:125)
2026-07-14T05:40:45.688583+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellNucleus.lambda$wrapLoggingContext$2(CellNucleus.java:725)
2026-07-14T05:40:45.688686+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.run(Thread.java:1583)
2026-07-14T05:40:45.688803+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:45+02:00 (System) [] Thread Group "c-core1-AAZSrLxVKHE-threads":
2026-07-14T05:40:45.688874+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: Thread: TunnelConnector-core1 [A--] (5) TIMED_WAITING
2026-07-14T05:40:45.688938+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.sleep0(Native Method)
2026-07-14T05:40:45.689003+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.sleep(Thread.java:509)
2026-07-14T05:40:45.689068+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.network.LocationManagerConnector.run(LocationManagerConnector.java:125)
2026-07-14T05:40:45.689128+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellNucleus.lambda$wrapLoggingContext$2(CellNucleus.java:725)
2026-07-14T05:40:45.689220+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.run(Thread.java:1583)
2026-07-14T05:40:45.689283+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:45+02:00 (System) [] Thread Group "Killer-Thread-Group":
2026-07-14T05:40:45.689356+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: Thread: killer-0 [A--] (5) WAITING
2026-07-14T05:40:45.689421+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/jdk.internal.misc.Unsafe.park(Native Method)
2026-07-14T05:40:45.689470+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
2026-07-14T05:40:45.689517+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
2026-07-14T05:40:45.689571+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
2026-07-14T05:40:45.689620+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
2026-07-14T05:40:45.689665+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1746)
2026-07-14T05:40:45.689720+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
2026-07-14T05:40:45.689777+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1070)
2026-07-14T05:40:45.689820+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
2026-07-14T05:40:45.689866+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2026-07-14T05:40:45.689910+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.run(Thread.java:1583)
2026-07-14T05:40:45.689945+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: Thread: killer-1 [A--] (5) RUNNABLE
2026-07-14T05:40:45.689995+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.getStackTrace(Thread.java:2451)
2026-07-14T05:40:45.690039+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellGlue.listThreadGroup(CellGlue.java:300)
2026-07-14T05:40:45.690096+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellGlue.listKillerThreadGroup(CellGlue.java:288)
2026-07-14T05:40:45.690150+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellNucleus.listKillerThreadGroup(CellNucleus.java:632)
2026-07-14T05:40:45.690215+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.SystemCell.shutdownCells(SystemCell.java:282)
2026-07-14T05:40:45.690269+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.SystemCell.shutdownSystem(SystemCell.java:220)
2026-07-14T05:40:45.690318+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.SystemCell.stopped(SystemCell.java:112)
2026-07-14T05:40:45.690380+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellAdapter.postRemoval(CellAdapter.java:730)
2026-07-14T05:40:45.690428+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellNucleus.shutdown(CellNucleus.java:1061)
2026-07-14T05:40:45.690477+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellGlue.lambda$doKill$3(CellGlue.java:423)
2026-07-14T05:40:45.690522+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
2026-07-14T05:40:45.690581+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2026-07-14T05:40:45.690638+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2026-07-14T05:40:45.690673+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.run(Thread.java:1583)
2026-07-14T05:40:45.690715+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: Thread: killer-3 [A--] (5) WAITING
2026-07-14T05:40:45.690763+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Object.wait0(Native Method)
2026-07-14T05:40:45.690812+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Object.wait(Object.java:366)
2026-07-14T05:40:45.690859+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.join(Thread.java:2079)
2026-07-14T05:40:45.690904+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.join(Thread.java:2155)
2026-07-14T05:40:45.690961+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     com.google.common.util.concurrent.Uninterruptibles.joinUninterruptibly(Uninterruptibles.java:167)
2026-07-14T05:40:45.691009+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.network.LocationManagerConnector.stopped(LocationManagerConnector.java:155)
2026-07-14T05:40:45.691056+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellAdapter.postRemoval(CellAdapter.java:730)
2026-07-14T05:40:45.691112+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellNucleus.shutdown(CellNucleus.java:1061)
2026-07-14T05:40:45.691157+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellGlue.lambda$doKill$3(CellGlue.java:423)
2026-07-14T05:40:45.691204+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
2026-07-14T05:40:45.691244+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2026-07-14T05:40:45.691279+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2026-07-14T05:40:45.691322+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.run(Thread.java:1583)
2026-07-14T05:40:45.691365+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: Thread: killer-4 [A--] (5) TIMED_WAITING
2026-07-14T05:40:45.691417+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/jdk.internal.misc.Unsafe.park(Native Method)
2026-07-14T05:40:45.691461+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410)
2026-07-14T05:40:45.691510+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452)
2026-07-14T05:40:45.691566+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.SynchronousQueue$Transferer.xferLifo(SynchronousQueue.java:194)
2026-07-14T05:40:45.691624+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.SynchronousQueue.xfer(SynchronousQueue.java:235)
2026-07-14T05:40:45.691682+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:338)
2026-07-14T05:40:45.691722+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
2026-07-14T05:40:45.691769+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
2026-07-14T05:40:45.691813+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2026-07-14T05:40:45.691852+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.run(Thread.java:1583)
2026-07-14T05:40:45.691888+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: Thread: killer-5 [A--] (5) WAITING
2026-07-14T05:40:45.691922+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Object.wait0(Native Method)
2026-07-14T05:40:45.691953+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Object.wait(Object.java:366)
2026-07-14T05:40:45.691983+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.join(Thread.java:2079)
2026-07-14T05:40:45.692013+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.join(Thread.java:2155)
2026-07-14T05:40:45.692046+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     com.google.common.util.concurrent.Uninterruptibles.joinUninterruptibly(Uninterruptibles.java:167)
2026-07-14T05:40:45.692084+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.network.LocationManagerConnector.stopped(LocationManagerConnector.java:155)
2026-07-14T05:40:45.692115+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellAdapter.postRemoval(CellAdapter.java:730)
2026-07-14T05:40:45.692145+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellNucleus.shutdown(CellNucleus.java:1061)
2026-07-14T05:40:45.692179+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     dmg.cells.nucleus.CellGlue.lambda$doKill$3(CellGlue.java:423)
2026-07-14T05:40:45.692217+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
2026-07-14T05:40:45.692247+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2026-07-14T05:40:45.692277+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2026-07-14T05:40:45.692310+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]:     java.base/java.lang.Thread.run(Thread.java:1583)
2026-07-14T05:40:45.696935+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: 2026-07-14 05:40:45+02:00 (System) [] Unexpected exception during shutdown.
2026-07-14T05:40:45.696973+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: java.util.NoSuchElementException: No such cell: c-core0-AAZSrLxVP-A-AAZSrLxV-2A
2026-07-14T05:40:45.697009+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at dmg.cells.nucleus.CellGlue.kill(CellGlue.java:249)
2026-07-14T05:40:45.697048+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at dmg.cells.nucleus.CellNucleus.kill(CellNucleus.java:617)
2026-07-14T05:40:45.697082+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180)
2026-07-14T05:40:45.697118+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
2026-07-14T05:40:45.697154+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
2026-07-14T05:40:45.697194+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
2026-07-14T05:40:45.697232+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
2026-07-14T05:40:45.697271+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
2026-07-14T05:40:45.697316+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2026-07-14T05:40:45.697362+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
2026-07-14T05:40:45.697397+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at dmg.cells.nucleus.SystemCell.shutdownCells(SystemCell.java:251)
2026-07-14T05:40:45.697437+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at dmg.cells.nucleus.SystemCell.shutdownSystem(SystemCell.java:223)
2026-07-14T05:40:45.697480+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at dmg.cells.nucleus.SystemCell.stopped(SystemCell.java:112)
2026-07-14T05:40:45.697519+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at dmg.cells.nucleus.CellAdapter.postRemoval(CellAdapter.java:730)
2026-07-14T05:40:45.697566+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at dmg.cells.nucleus.CellNucleus.shutdown(CellNucleus.java:1061)
2026-07-14T05:40:45.697606+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at dmg.cells.nucleus.CellGlue.lambda$doKill$3(CellGlue.java:423)
2026-07-14T05:40:45.697640+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
2026-07-14T05:40:45.697675+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2026-07-14T05:40:45.697710+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2026-07-14T05:40:45.697748+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[13407]: #011at java.base/java.lang.Thread.run(Thread.java:1583)

now it's dead and systemct stop returns.

Here I start it again:

2026-07-14T05:41:04.428621+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[2004998]: 2026-07-14 05:41:04+02:00 (lcg-lrz-dc03_11) [] Pool mode changed to disabled(fetch,store,stage,p2p-client,p2p-server): Awaiting initialization
2026-07-14T05:41:04.437165+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[2004998]: 2026-07-14 05:41:04+02:00 (lcg-lrz-dc03_11) [] Pool mode changed to disabled(store,stage,p2p-client,loading): Loading...
2026-07-14T05:41:04.438758+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[2004998]: 2026-07-14 05:41:04+02:00 (lcg-lrz-dc03_11) [] Reading inventory from Inotify[Checksum[IoStatistics[data=/srv/dcache/pools/11/data;meta=/srv/dcache/pools/11/meta]]].
2026-07-14T05:41:04.452530+02:00 lcg-lrz-dc03 dcache@pool_lcg-lrz-dc03_11[2004998]: 2026-07-14 05:41:04+02:00 (lcg-lrz-dc03_11) [] Pool mode changed to enabled

The main reason I report this is in particular, that the mode changed to disabled(fetch,store,stage,p2p-client,p2p-server,dead): Shutdown never seems to reach the rest of dCache... in particular my PoolManager still shows e.g.:

lcg-lrz-dc03_11  (enabled=true;active=1343991;rdOnly=false;links=0;pgroups=1;hsm=[];mode=enabled)

but with only the ever growing active times indicating something's fishy.

Cheers,
Chris.

[calestyo]

Actually I just noticed that this might cause even more problems, though this is perhaps rather a separate issue:

When I manually disable a pool like via pool disable -strict or so... then I still can get it's data via e.g.:

curl -s http://localhost:3880/api/v1/pools/lcg-lrz-dc44_1/usage | jq '.["poolData"]["detailsData"]["poolMode"]'

But the above "disappeared" pool simply vanishes from the REST:

$ curl -s http://localhost:3880/api/v1/pools/lcg-lrz-dc44_0/usage | jq 
{}

Which means one cannot really use the REST as a check whether a pool is down, because it doesn't reliably include it`