hooks enabled: client times out and disconnects prematurely during long afterLease

[evakhoni]

during a long afterHook which is set to eventually timeout, the client disconnects prematurely. with default timeout of 120s, client kicked out after ~30secs

[02/17/26 18:55:53] INFO     Running afterLease hook (Ctrl+C to skip)... [jumpstarter_cli.shell]
                    INFO     Starting hook subprocess... [exporter:afterLease]
                    INFO     Creating PTY pair... [exporter:afterLease]
                    INFO     PTY created: parent_fd=16, child_fd=17 [exporter:afterLease]
                    INFO     Executing inline script (interpreter: /bin/sh) [exporter:afterLease]
                    INFO     Spawning subprocess with command: ['/bin/sh', '-c', 'echo "start posthook long sleep"\nsleep 999\necho "done postlease"\n'] [exporter:afterLease]
                    INFO     Subprocess spawned with PID 2707870 [exporter:afterLease]
                    INFO     Starting PTY output reader and process waiter (timeout=120) [exporter:afterLease]
                    INFO     Task group created, starting tasks... [exporter:afterLease]
                    INFO     Waiting for subprocess to complete... [exporter:afterLease]
                    INFO     start posthook long sleep [exporter:afterLease]
[02/17/26 18:55:54] INFO     Status changed: LEASE_READY -> AFTER_LEASE_HOOK (version=4) [jumpstarter.client.status_monitor]
[02/17/26 18:56:23] WARNING  Timeout waiting for afterLease hook to complete [jumpstarter_cli.shell]
                    INFO     Releasing Lease 019c6c86-7740-7e17-a169-9dfce9fc2a86 [jumpstarter.client.lease]  

# the exporter is still in afterLease at this point

[ambient-code]

Investigation: Issue #243 - Client disconnects prematurely during long afterLease hook

Root Cause

The premature ~30s disconnect during a long afterLease hook is caused by a chain of timeouts on the client side:

1. EndSession RPC timeout (10s): In shell.py line 365, the EndSession call is wrapped in anyio.move_on_after(10), but the RPC itself also has a timeout=5.0 in core.py line 223. The 5s gRPC deadline fires first, but connection errors are treated as success (return True), so EndSession typically succeeds.

2. StatusMonitor stops polling on UNAVAILABLE: After EndSession succeeds, the shell waits for AVAILABLE or AFTER_LEASE_HOOK_FAILED via monitor.wait_for_any_of(..., timeout=300.0) (line 370-371). However, during a long-running afterLease hook, the exporter's session may become unstable or the gRPC connection may hit keepalive timeouts. When the StatusMonitor._poll_loop() encounters UNAVAILABLE, it permanently marks _connection_lost = True and stops (line 390-396).

3. gRPC keepalive defaults: The default gRPC keepalive settings can cause the connection to be terminated by the transport layer after ~20-30s of perceived inactivity, particularly if the afterLease hook is running a subprocess that doesn't generate gRPC-level traffic.

4. wait_for_any_of returns None on connection_lost: Once _connection_lost is set, wait_for_any_of() returns None immediately (line 274, 290). The shell sees result is None and monitor.connection_lost is True, logs "Connection lost, skipping afterLease hook wait" (line 400), and proceeds to release the lease and exit.

The 300s timeout at line 372 is generous enough for any hook, but it's never reached because the status monitor gives up on the first UNAVAILABLE error ~30s in.

Why the exporter remains in afterLease state

The exporter continues running the afterLease hook in a CancelScope(shield=True) (hooks.py line 517), so the hook completes server-side. But the client has already disconnected and released the lease. The exporter eventually finishes the hook and transitions to AVAILABLE, but by then the client is long gone.

Proposed Fix

1. StatusMonitor: Don't give up on UNAVAILABLE during afterLease hook. When the monitor knows an afterLease hook is running (current_status == AFTER_LEASE_HOOK), treat UNAVAILABLE as transient and retry with backoff, similar to DEADLINE_EXCEEDED handling. The hook's configured timeout should be the true deadline, not gRPC transport issues.

2. Core: Increase GetStatus RPC timeout during hook execution. The 5s per-RPC timeout in GetStatus calls is too aggressive for a system under load during hook execution. Consider increasing to 15-30s when a hook is known to be running.

3. Shell: Use hook timeout as the afterLease wait deadline. Instead of hardcoding 300s, the client should query the exporter for the hook's configured timeout and use that as the wait deadline. At minimum, the shell should not exit just because the status monitor lost connection -- it should attempt direct GetStatus probes before giving up.

Files to Modify

• /workspace/repos/jumpstarter/python/packages/jumpstarter/jumpstarter/client/status_monitor.py - Retry UNAVAILABLE during hook states
• /workspace/repos/jumpstarter/python/packages/jumpstarter/jumpstarter/client/core.py - Increase GetStatus timeout during hooks
• /workspace/repos/jumpstarter/python/packages/jumpstarter-cli/jumpstarter_cli/shell.py - Fallback to direct GetStatus probes when monitor reports connection_lost during afterLease