chore(tests): share the --json purity helper and de-hardcode the command count

[pbean]

Enabling work for #196. No --json flag is added here — this only touches the
lines the three follow-up PRs (#203 decisions --list, #204 clean/cleanup,
#205 validate) would all otherwise conflict on.

The shared purity helper moves to conftest.py

_machine_json lived in tests/test_cli.py, where test_probe.py and
test_cleanup.py could not reach it. It is now machine_json in
tests/conftest.py — the module test_cleanup.py already imports from — with
two keyword-only parameters, both defaulting to the strict form:

• rc — validate --json (Adopt --json on validate (pure-document contract) #205) will legitimately exit 1 while still
  owing the caller a complete document. What the helper asserts is stdout
  purity, not success.
• err_contains — see below.

Parsing the whole of stdout is still the purity assertion, and the docstring
still says why.

err_contains, and a wrong premise in the plan

The plan called for test_probe.py to adopt the helper on the grounds that it
"re-implements the same assertions inline". It does not — it asserts the
opposite on stderr:

assert "ok:" in err  # the human trailer moved to stderr

That is deliberate. probe-adapter --json routes its ok: trailer to stderr
precisely so stdout stays the document alone (cli.py:1945-1947), which
machine.py documents as legitimate ("with only a confirmation on stderr").
So err == "" is simply false for probe.

Rather than relax the default, err_contains opts a call site into a
different assertion. Default None keeps err == "" for status, list
and diagnose; probe passes err_contains="ok:", which still fails if that
trailer ever moves back to stdout. Same shape as rc: strict by default,
explicit at the one site that differs.

The sorted-keys check moves to the renderer

_assert_keys_sorted needs the raw string — only object_pairs_hook sees
key order, and machine_json returns a parsed dict that has already lost it.
Handing it a re-serialized json.dumps(doc, sort_keys=True) would sort the
keys on the way in and assert nothing at all; the docstring now says so, since
that is the obvious wrong repair.

It is now test_render_json_sorts_keys_at_every_depth, asserting against
probe.render_json's own return value. That is where the property lives — the
CLI just prints what the renderer returns — and it is strictly stronger:
dropping sort_keys=True from render_json fails it on a nested object
(['SessionStart', 'Stop', 'SessionEnd', 'PreCompact']), which the old
stdout-based check only reached at the top level. Verified by making that edit
and watching it fail, then reverting.

Docs

machine.py and docs/FEATURES.md no longer hardcode "all four" commands —
three more are about to take the flag. The #195 reference and the
no-exception meaning are unchanged. machine.py now also says the contract is
the flag's meaning, not a style the first four happen to share.

README.md:257 ("preflights all four upstream skills") is left alone — that is
DEV_BASE_SKILLS, unrelated.

Verification

• uv run pytest -q — 2548 passed, 1 skipped
• trunk check — clean (it reformatted one argument list in test_probe.py)
• Regression-proved the new sorted-keys test as described above

Closes nothing; unblocks #203, #204, #205 under #196.

Summary by CodeRabbit

• Documentation

  • Clarified that every CLI command supporting --json follows the same machine-readable output contract.
  • Documented the current supported commands: status, list, diagnose, and probe-adapter.
  • Specified that output is either one complete valid JSON document or empty, with no appended text or markdown.

• Tests

  • Expanded coverage to verify clean JSON output, expected error handling, and consistent key ordering at every level.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5b3cb118-00be-48bc-a015-c9fc93c0e0aa

📥 Commits

Reviewing files that changed from the base of the PR and between a1f16a5 and 7c0e4f0.

📒 Files selected for processing (5)

• docs/FEATURES.md
• src/bmad_loop/machine.py
• tests/conftest.py
• tests/test_cli.py
• tests/test_probe.py

---

Walkthrough

The --json contract now applies to every supporting CLI command, and shared test infrastructure validates pure JSON stdout, expected exit codes, stderr behavior, and recursively sorted keys.

Changes

JSON contract and test coverage

Layer / File(s)	Summary
Generalize the JSON contract docs/FEATURES.md, src/bmad_loop/machine.py	Documentation now applies the machine-readable output rules to every command accepting --json and removes the former fenced-output exception.
Centralize JSON output testing tests/conftest.py, tests/test_cli.py, tests/test_probe.py	A shared machine_json helper replaces duplicated parsing logic and supports purity, exit-code, stderr, and nested key-order assertions.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• bmad-code-org/bmad-loop#193: Adds and validates stable status --json output covered by this contract.
• bmad-code-org/bmad-loop#197: Standardizes the pure JSON stdout and stderr-only error contract.
• bmad-code-org/bmad-loop#201: Covers unified JSON behavior for diagnose and probe-adapter.

Suggested reviewers: dracic

Poem

I’m a rabbit with JSON in my paws,
Pure stdout follows all the laws.
Keys hop sorted, deep and wide,
Errors scurry off stderr’s side.
One clear document—nothing more!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: shared JSON test helper and removing the hardcoded command count.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch chore/json-contract-prep-196

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}