Skip to content

fix(teardown): wait for the session's process tree before worktree teardown (#183) - #184

Merged
pbean merged 3 commits into
mainfrom
fix/teardown-process-tree-183
Jul 18, 2026
Merged

fix(teardown): wait for the session's process tree before worktree teardown (#183)#184
pbean merged 3 commits into
mainfrom
fix/teardown-process-tree-183

Conversation

@pbean

@pbean pbean commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Closes #183. Follow-up to #139/#182: that PR made a lost teardown race degrade instead of crash; this one removes the race itself.

Mechanism

A process spawned inside a session (backgrounded pytest, a setsid/double-forked runner) escapes the pane's process group, so tmux kill-window's SIGHUP never reaches it — and the #157 verified kill only harvested pane pids when the window outlived teardown_grace_s. A session that ended cleanly (window dead on the first probe) never had any pids read at all, leaving stragglers free to write into the worktree while the engine merged and deleted it.

Fix

  • GenericAdapter.kill — when teardown_grace_s > 0, harvest the pane pids plus their transitive descendants (each stamped with a pid-reuse identity) before the first kill strike, while the tree is provably alive. After the window dies, _reap_straggler_tree terminates → polls → force-kills identity-confirmed survivors within the same grace deadline (one budget, two phases; terminate first so a mid-write straggler can flush). The wedged-window escalation also force-kills harvested descendants, with the same None-identity refusal as the clean path. grace = 0 remains the legacy single strike, byte-for-byte.
  • ProcessHost.descendants(pid) — new seam capability: transitive descendants as a dict[int, float | None] (pid → pid-reuse identity captured DURING enumeration, so there is no post-hoc stamp to race a reuse); a member whose identity can't be confirmed is OMITTED rather than returned unstamped. Never raises, {} degrade. Linux uses a one-pass /proc ppid map + BFS, taking ppid and starttime from the SAME stat read (the core stays psutil-free); macOS/win32 go through the existing lazy _psutil(), revalidating each child via is_running() before stamping it — create_time() is only construction-bound on the WINDOWS branch of psutil's _get_ident, so on macOS it is a call-time read that would otherwise return a recycled pid's identity.
  • opencode_http._kill_process — mirrored pre-signal harvest + reap bounded by kill_wait_s (kept as its own copy per the adapter's teardown convention; these sessions have no tmux window, so the process-tree wait is the only teardown signal there).
  • A misconfigured BMAD_LOOP_PROCESS_HOST override still strikes the window once before the loud ProcessHostError re-raise, so a config error can never leak a live window.

Accepted limitations (documented at the harvest site): a double-fork orphan that reparented before the harvest, or a process detached in the harvest→kill TOCTOU window, is unfindable by construction; macOS without the non-linux extra degrades to today's behavior.

Verification

  • New E2E (test_e2e_detached_writer_reaped_before_worktree_teardown) automates the issue's acceptance sketch: a fake CLI setsid-detaches a writer and ends cleanly; the writer is reaped within the grace, the worktree removes cleanly, and no worktree-teardown-degraded fires. (The existing session-end journaled 2h19 after session_timeout_min fires when the session is wedged inside a tool call #157 fake child shares the pane pgid and dies via SIGHUP — it cannot reproduce this bug.)
  • Kill-path units extended: clean-end straggler reap (terminate precedes force-kill), None-identity never signalled at all on either branch (no terminate, no poll, no force-kill), reused-pid skip, no-pids backend degrade, bad-host-override strike; test_kill_returns_on_first_dead_probe_without_escalating's pane_pid_reads == 0 assertion is deliberately inverted to == 1 — the old assertion encoded the bug.
  • descendants seam: real grandchild /proc enumeration + monkeypatched-psutil never-raise contract, plus a psutil pid-rollover unit (create_time() returning the recycled generation while is_running() reports False — that pid must be omitted, verified to fail without the fix); opencode mirror: real Popen setsid descendant reaped, None-identity refusal unit.
  • Full suite: 2446 passed, 1 skipped (win32-only); full trunk check clean.

Summary by CodeRabbit

  • Bug Fixes
    • Improved teardown to more reliably harvest and reap detached/descendant processes within the grace window, including wedged escalation cases.
    • Added identity-verified reaping so unverified/unconfirmable PIDs are never signaled, reducing risks from PID reuse.
    • Enhanced failure handling so the tmux window is struck before errors surface, improving consistency across teardown paths.
    • Strengthened end-to-end behavior to confirm detached processes are reaped before worktree merge/removal.
  • Documentation
    • Clarified that the teardown grace period bounds both window termination and the subsequent descendant-process reap.

…ardown (#183)

Harvest pane pids plus their descendant tree (with pid-reuse identities)
before the first kill strike, then reap identity-confirmed stragglers
within teardown_grace_s once the window dies — a setsid/double-fork
survivor can no longer race the worktree merge+removal. Adds the
ProcessHost.descendants seam (/proc walk on Linux, psutil elsewhere,
never raises) and mirrors the reap in opencode_http's _kill_process.
@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3ac8f92b-3289-426b-819e-5acf8b9ed931

📥 Commits

Reviewing files that changed from the base of the PR and between e3be915 and 3128eab.

📒 Files selected for processing (3)
  • src/bmad_loop/adapters/generic.py
  • src/bmad_loop/process_host.py
  • tests/test_process_host.py
🚧 Files skipped from review as they are similar to previous changes (3)
  • tests/test_process_host.py
  • src/bmad_loop/process_host.py
  • src/bmad_loop/adapters/generic.py

Walkthrough

Process-host implementations now discover descendant trees, while the GenericTmuxAdapter and OpenCode adapter harvest identities before teardown and reap confirmed survivors within bounded grace periods. Tests and documentation cover PID reuse, uncertain identities, detached processes, and worktree teardown ordering.

Changes

Process-tree teardown

Layer / File(s) Summary
Process-tree discovery seam
src/bmad_loop/process_host.py, tests/test_process_host.py
Adds non-raising descendant discovery through Linux /proc or psutil, with transitive-tree and failure-path coverage.
Generic tmux teardown
src/bmad_loop/adapters/generic.py, tests/test_generic_tmux.py
Harvests pane descendants before window termination and reaps or force-kills only identity-confirmed survivors.
OpenCode process teardown
src/bmad_loop/adapters/opencode_http.py, tests/test_opencode_http.py
Snapshots descendant identities before signaling and applies bounded terminate-then-force-kill reaping.
Integration verification and teardown contract
docs/FEATURES.md, tests/test_stories_e2e.py
Documents the shared teardown budget and verifies detached processes are reaped before worktree removal.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Session as Session teardown
  participant Host as ProcessHost
  participant Window as tmux window
  participant Tree as Descendant processes
  Session->>Host: Harvest pane PIDs and identities
  Session->>Window: Kill window
  Window-->>Session: Window exits or grace expires
  Session->>Host: Check harvested survivors
  Host->>Tree: Terminate identity-confirmed survivors
  Host->>Tree: Force-kill remaining confirmed survivors
  Session-->>Session: Proceed to worktree teardown
Loading

Possibly related PRs

Poem

I’m a rabbit who guards every PID,
With identity stamps tucked under my lid.
Windows may vanish, descendants may flee,
Confirmed little stragglers are reaped carefully.
The worktree stays tidy, the journal stays bright—
Hop, hop, teardown done right!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 30.43% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: waiting on the session process tree during teardown.
Linked Issues check ✅ Passed The PR implements the process-tree harvesting, identity checks, grace-period reaping, and opencode_http parity required by #183.
Out of Scope Changes check ✅ Passed The changes stay centered on teardown race handling, process-host support, tests, and docs, with no unrelated scope visible.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/teardown-process-tree-183

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/bmad_loop/adapters/generic.py`:
- Around line 789-808: The straggler-reap flow must never signal descendants
whose identity is None. In src/bmad_loop/adapters/generic.py lines 789-808,
filter unconfirmed entries before terminate, polling, and forced escalation,
using only confirmed identities for all signaling decisions. In
tests/test_generic_tmux.py lines 314-331, update coverage to assert that an
unconfirmed PID receives no signal.

In `@src/bmad_loop/adapters/opencode_http.py`:
- Line 994: Update the process-host handling around get_process_host() to catch
ProcessHostError, perform the existing legacy Popen root strike before
re-raising the error, and ensure teardown still propagates the failure. Add the
equivalent regression test for this invalid process-host override scenario,
matching the coverage already present for the tmux path.

In `@src/bmad_loop/process_host.py`:
- Around line 243-283: Make descendant discovery return a PID-to-identity
snapshot rather than bare PIDs: update _linux_descendants and
_psutil_descendants to capture each descendant’s identity while enumerating
ancestry, preserving the never-raise behavior. In src/bmad_loop/process_host.py
lines 243-283, define and propagate the mapping; in
src/bmad_loop/adapters/generic.py lines 724-728, consume the captured identities
without reacquiring them. In tests/test_process_host.py lines 191-229, add
coverage for enumerating generation A followed by PID reuse as generation B.

In `@tests/test_generic_tmux.py`:
- Around line 314-331: The test
test_kill_never_force_kills_identity_none_straggler currently expects
termination of an unconfirmed PID. Update its assertions so PID 200 receives
neither terminate nor force-kill, while preserving the existing lifecycle
outcome checks and recording it as unconfirmed if that behavior is exposed.

In `@tests/test_opencode_http.py`:
- Around line 532-548: Update test_kill_process_reaps_detached_descendant to
avoid invoking the external setsid utility: replace the shell-based
detached-child setup with a Python subprocess launched using
start_new_session=True, while preserving PID-file creation, child liveness, and
the existing descendant-reaping assertions.

In `@tests/test_process_host.py`:
- Around line 191-229: Add a test case for _psutil_descendants that simulates
PID reuse between enumeration and identity validation: return a process from the
initial child enumeration representing generation A, then make the subsequent
identity lookup represent generation B for the same PID. Assert the resulting
snapshot excludes that PID and does not authenticate the reused generation.

In `@tests/test_stories_e2e.py`:
- Around line 213-245: Gate the E2E test containing DETACHED_WRITER_FAKE_CLI to
run only on Linux, skipping it on macOS and other non-Linux platforms. Update
the existing platform-skip logic in the relevant test rather than changing the
fake CLI.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9223180f-de11-4a82-9dfe-96df8748e64e

📥 Commits

Reviewing files that changed from the base of the PR and between e7378b3 and 18cdb99.

📒 Files selected for processing (8)
  • docs/FEATURES.md
  • src/bmad_loop/adapters/generic.py
  • src/bmad_loop/adapters/opencode_http.py
  • src/bmad_loop/process_host.py
  • tests/test_generic_tmux.py
  • tests/test_opencode_http.py
  • tests/test_process_host.py
  • tests/test_stories_e2e.py

Comment thread src/bmad_loop/adapters/generic.py Outdated
Comment thread src/bmad_loop/adapters/opencode_http.py Outdated
Comment thread src/bmad_loop/process_host.py Outdated
Comment thread tests/test_generic_tmux.py Outdated
Comment thread tests/test_opencode_http.py Outdated
Comment thread tests/test_process_host.py Outdated
Comment thread tests/test_stories_e2e.py
…ities (PR #184 review)

- ProcessHost.descendants now returns a pid -> identity snapshot captured
  during enumeration (same /proc stat read on Linux, same psutil Process
  object elsewhere); a member whose identity can't be captured is omitted,
  closing the enumerate-then-stamp pid-reuse window.
- The straggler reaps (tmux + opencode mirror) no longer signal identity-None
  members at all: no SIGTERM, no poll burn, no force-kill — observation-only,
  reported via the unreaped breadcrumb.
- opencode _kill_process now mirrors the tmux strike-before-reraise doctrine
  for a bogus BMAD_LOOP_PROCESS_HOST override (+ regression test).
- Tests: pid-rollover omission coverage, portable start_new_session=True
  detached child (no setsid(1) — macOS lacks it), stories E2E gated to Linux
  (the fake CLIs are bash + GNU coreutils).
@pbean

pbean commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator Author

@CodeRabbit review

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/bmad_loop/adapters/generic.py (1)

758-763: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Identity-guard refreshed pane PIDs before force-killing.

A pane can exit after window_pane_pids(), allowing its PID to be reused before Line 762. Use the harvested identity and alive_and_ours() instead of force-killing the bare PID.

🔒 Proposed fix
         for pid in repane:
+            identity = tree.get(pid)
+            if identity is None or not host.alive_and_ours(pid, identity):
+                continue
             try:
                 host.force_kill(pid)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bmad_loop/adapters/generic.py` around lines 758 - 763, Update the
escalation loop around window_pane_pids and force_kill to retain each harvested
process identity, revalidate it with alive_and_ours() immediately before
termination, and only force-kill when the identity still belongs to the expected
task. Preserve the existing lifecycle note and already-gone exception handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/bmad_loop/process_host.py`:
- Around line 294-305: Update the psutil child handling in the descendant
collection function to read create_time(), then verify the same child Process
with is_running() before recording its PID and generation; omit the child when
validation returns false or raises, while preserving the existing never-raise
behavior. Update the rollover test fake so create_time() returns generation B
and is_running() returns False, covering reuse without relying on an exception.

---

Outside diff comments:
In `@src/bmad_loop/adapters/generic.py`:
- Around line 758-763: Update the escalation loop around window_pane_pids and
force_kill to retain each harvested process identity, revalidate it with
alive_and_ours() immediately before termination, and only force-kill when the
identity still belongs to the expected task. Preserve the existing lifecycle
note and already-gone exception handling.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c36043f2-67c2-4e38-ba05-5bbcd91a0489

📥 Commits

Reviewing files that changed from the base of the PR and between 18cdb99 and e3be915.

📒 Files selected for processing (7)
  • src/bmad_loop/adapters/generic.py
  • src/bmad_loop/adapters/opencode_http.py
  • src/bmad_loop/process_host.py
  • tests/test_generic_tmux.py
  • tests/test_opencode_http.py
  • tests/test_process_host.py
  • tests/test_stories_e2e.py
🚧 Files skipped from review as they are similar to previous changes (3)
  • tests/test_stories_e2e.py
  • src/bmad_loop/adapters/opencode_http.py
  • tests/test_generic_tmux.py

Comment thread src/bmad_loop/process_host.py
…g (PR #184 review)

`_psutil_descendants` stamped each enumerated child with `create_time()` and
the docstring claimed that was reuse-safe because "psutil binds identity at
construction". That is false on macOS.

Per `psutil.Process._get_ident` (7.2.2), only the WINDOWS branch pre-populates
the epoch cache (`self._create_time = create_time(fast_only=True)`). The
`LINUX or NETBSD or OSX` branch binds a *monotonic* starttime and leaves
`self._create_time` unset, and `create_time()` itself never consults
`_raise_if_pid_reused()`. So on macOS our stamp was the first call — a raw
call-time kernel read. A pid reaped and reused between `children()` and the
stamp handed back the NEWCOMER's identity, which teardown would then
authenticate successfully and SIGTERM/SIGKILL: precisely the invariant this
branch exists to enforce, defeated by a comment asserting the opposite.

Read the identity, revalidate via `is_running()` (construction-bound ident vs.
a fresh `Process`), then record. Order matters — validating first reopens the
same TOCTOU gap. An unconfirmable member is omitted, never stamped. Linux never
reaches this path (psutil-free `/proc` walk); Windows was already safe via the
`_get_ident` cache.

Both consumers ride the seam (`generic.py`, `opencode_http.py`), so neither
adapter changes. The rollover test's exception-only fake passed against the
buggy code; it now also models the shape that actually bites — `create_time()`
returning generation B while `is_running()` reports False — and fails without
the fix.
@pbean

pbean commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator Author

@CodeRabbit review

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@pbean

pbean commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator Author

@CodeRabbit review

@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Session teardown does not wait for the session's process tree — stragglers race the worktree removal (#139 bug 1)

1 participant