Skip to content

feat: CPU-spike fixes (Phase 1) + Center Stage / digital PTZ (Phase 2)#100

Merged
TCVinNYC merged 24 commits into
mainfrom
feat/cpu-spikes-and-center-stage
Jun 24, 2026
Merged

feat: CPU-spike fixes (Phase 1) + Center Stage / digital PTZ (Phase 2)#100
TCVinNYC merged 24 commits into
mainfrom
feat/cpu-spikes-and-center-stage

Conversation

@TCVinNYC

@TCVinNYC TCVinNYC commented Jun 24, 2026

Copy link
Copy Markdown
Member

Executed via subagent-driven development from docs/superpowers/plans/2026-06-24-cpu-spikes-and-center-stage.md. Each task was TDD'd and passed an individual spec+quality review; a final whole-branch review (opus) returned READY with no Critical/Important findings.

Phase 1 — CPU spikes / latency (behaviour-affecting → validate on real cameras)

  • Ego-motion decimation — run the per-frame optical flow every Nth inference frame (ptz.ego_comp_interval, default 3) and reuse a decayed estimate between. The real macOS lever (OpenCV GCD can't be thread-capped).
  • Stage spreading — never run the detector and the pose pass on the same frame (tracking.stage_spread, default on), so a heavy detect tick and a heavy pose tick don't stack into a 200 ms frame.
  • Governor cost amortization + hysteresis — the CPU governor now amortizes each stage's cost by how often it actually runs (was over-counting), with a 0.70–0.90 hysteresis band so the tier can't flap.
  • System-CPU-aware governor — the supervisor samples process CPU% (~1 Hz) and pushes it to workers, so several "individually fine" cameras back off collectively when the machine is hot.

Phase 2 — Center Stage (new feature)

  • DigitalPTZBackend — a PTZBackend that integrates the controller's velocity commands into a crop window instead of driving motors. Reuses the entire control loop + framing math.
  • Factory + config — backend="digital", plus ptz.vcam_out / digital_output_w / digital_output_h.
  • VirtualCamSink — optional pyvirtualcam output (graceful no-op when the package/driver is absent).
  • Output stage — applies the digital crop to the preview push and feeds the virtual cam; zero-impact passthrough for non-digital cameras; never raises into the capture thread.
  • UI — digital backend choice + "Virtual camera output" toggle in the PTZ panel.

This fills a validated market gap: every competitor (OBSBOT/PTZOptics/AVer/Insta360/EMEET) is hardware-locked — there's no vendor-agnostic software that adds center-stage to any existing webcam.

Tests

  • Full suite 1047 passing, ruff check + format clean. New: tests/test_cpu_governor.py, tests/test_digital_ptz.py, tests/test_vcam.py, tests/test_flags.py (+ UI tests).

Validation before merge (your policy)

  • CPU: with stream + face + detection + pose on, confirm frame-time variance (30–200 ms) and CPU bursts (60–300 %/s) tighten; watch the Services-panel quality reason + per-stage ms.
  • Center Stage: set a plain USB webcam to the digital backend, enable "Virtual camera output", confirm it auto-frames a walking subject and appears as a virtual camera in Zoom/OBS.

Deferred follow-ups (from final review, non-blocking)

  • Hysteresis state-transition regression test; minor low↔balanced event flap at ratio≈1.10; optionally gate the vcam send on a digital backend; AUTOPTZ_* env parity for the new flags.

🤖 Generated with Claude Code

TCVinNYC and others added 14 commits June 24, 2026 11:36
@TCVinNYC @claude
perf(engine): decimate ego-motion optical flow to every Nth frame
28a5b1c 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
perf(engine): never run detect and pose on the same frame (stage spread)
1b93277 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
perf(engine): amortize governor cost by stage cadence + add hysteresis
61e1e0f 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
perf(engine): make the CPU governor system-CPU-aware
f15d311 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
feat(ptz): DigitalPTZBackend — velocity→crop window for Center Stage
f72ab8a 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
feat(ptz): register digital backend in the factory + config schema
181775d 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
feat(pipeline): optional virtual-camera output sink (pyvirtualcam)
ff443e5 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
feat(engine): apply digital crop to the framed output + virtual-cam send
ed4ed4e 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
feat(ui): expose Center Stage (digital PTZ + virtual-cam output)
741d114 
- Add "digital" to the PTZ backend combo (Center Stage — software
  auto-framing for non-PTZ cameras, output as a virtual camera).
- Add self._vcam_out QCheckBox ("Virtual camera output") mirroring
  _auto_zoom: constructed in the PTZ section, connected to _schedule,
  loaded from pz.get("vcam_out", False), saved in _push.
- Tests (TestCenterStageUI): backend combo contains "digital";
  _vcam_out round-trips False→True into _push cfg dict.
- CHANGELOG: Added Center Stage entry under [Unreleased].

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
refactor(engine): name CPU-sample interval constant + clarify cadence…
e1fb03c 
…-gate comments

- supervisor.py: extract _CPU_SAMPLE_INTERVAL_S = 1.0 near _PUMP_INTERVAL_S /
  _HEALTH_SCAN_INTERVAL_S and use it in the tick() throttle guard (was a bare
  literal 1.0)
- camera_worker.py (_update_ego_motion): add sentence-case comment explaining
  why the ego_comp_interval getattr sentinel is 1 (serialized-config fallback
  to legacy every-frame; normal Pydantic path always supplies default 3)
- camera_worker.py (_maybe_track): add sentence-case comment at the frame/detector
  early-return guards noting the stale _detected_this_tick flag is intentional —
  no inference ran, so there are no detections to pose anyway

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
fix(tracking): don't let appearance ReID drift a named-identity target
c86f7d7 
Bug: when a target was selected by identity (name) and then lost, the
appearance-only ReID recovery (`_maybe_reid_recover`) re-bound the lock to
whatever visible track best matched the body template — so the camera drifted
onto the next visually-similar person, then the next unknown track.

A named-identity target now re-binds via ReID only to a track that
face-recognition has confirmed as the SAME identity (`_reid_rebind_allows`);
otherwise it stays searching until the real person's face is seen again. A
manual/clicked target (no identity) keeps the appearance-based re-bind it relies on.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
fix(tracking): coast on sudden box collapse instead of chasing an occ…
bb658fa 
…luded target

Bug: when the target was covered by something/someone the detector returned only
the visible part (e.g. the legs); the tracker kept that shrinking box as a
confident, non-lost track, so the PTZ chased it down to the last-known partial
position (box + aim dot drifting onto the legs).

The drive loop now tracks a slowly-updated "healthy" target-height reference and
coasts (track_active=False) when the box height suddenly collapses below it — the
occlusion signature — while a gradual shrink (the subject walking away) keeps the
reference in step and is NOT flagged, so normal following is unaffected. The
camera resumes when the subject reappears at full size.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
perf(reid): run OSNet ReID on the platform GPU (mps/cuda) instead of …
b7107cf 
…always CPU

ReID was pinned to device="cpu" — a major per-frame cost on Macs that also stalls
the inference thread through the GIL (the appearance pass holds it during the
torch/numpy work). It now auto-selects the Apple GPU (mps) or CUDA when available
and falls back to CPU if GPU init fails. AUTOPTZ_REID_DEVICE overrides the choice
(e.g. =cpu if an OSNet op misbehaves on mps).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
docs(changelog): note ReID-on-GPU + the two target-lock fixes
4a18c5d 
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC TCVinNYC mentioned this pull request Jun 24, 2026
Closed
TCVinNYC and others added 10 commits June 24, 2026 14:46
@TCVinNYC
Merge branch 'fix/target-lock-and-perf' into feat/cpu-spikes-and-cent…
e3a7af1 
…er-stage
@TCVinNYC @claude
feat(ui): Center Stage toggle + move raw PTZ backend selector to Adva…
65d8678 
…nced

Replaces the "select the digital backend" step with a single user-facing
**Center Stage** checkbox in the PTZ panel: on → the camera uses the digital
auto-framing backend (crop-follow) and the raw transport picker is greyed.
The protocol selector (Backend / Address) moves into a collapsed PTZ → Advanced
group, since most users use Center Stage or the auto-probe. Center Stage maps to
backend="digital" on save and is reflected on load.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
perf(inference): make CoreML compute units tunable (AUTOPTZ_COREML_UN…
47c74d8 
…ITS)

On Intel + AMD Macs (e.g. iMac Pro Xeon + Vega 56) CoreML may silently run on the
CPU instead of the discrete GPU. The compute-unit target is now configurable
(default ALL) so it can be verified/forced: run `--bench` with CPUOnly and again
with ALL/CPUAndGPU and compare to see whether the Vega is actually helping.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC
Merge branch 'fix/target-lock-and-perf' into feat/cpu-spikes-and-cent…
06cc60f 
…er-stage
@TCVinNYC @claude
fix(tracking): stop the camera ping-pong from stale ego-motion subtra…
cb8624e 
…ction

Root cause (confirmed by a multi-agent investigation + adversarial review): the
ego-motion decimation reuses a decayed estimate between optical-flow ticks, but
_estimate_aim_velocity ran every tick and unconditionally SUBTRACTED that
stale/decayed ego from the per-frame aim velocity. During a pan-follow that
produced a sawtooth phantom world-velocity which the controller's predictive lead
chased — flipping the command sign frame to frame (the "ping-pong" that only
appears when the camera AND the subject move).

Fix: gate the subtraction on a fresh measurement. _update_ego_motion sets
_ego_fresh True only on a tick it actually re-measures (False on decimated,
disabled, and error ticks); _estimate_aim_velocity zero-subtracts ego when not
fresh. Decimation (the CPU saving) stays; the phantom is gone.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
fix(ui): Center Stage frames the target directly (it never zoomed bef…
66f1bf6 
…ore)

Center Stage drove the digital crop through the physical PTZ velocity controller,
whose conservative auto-zoom never sent a zoom command — so the crop stayed at the
full frame (and with no zoom there was no room to pan either). Reproduced: a real
controller step produced pan commands but zoom=0.000 forever, crop = full frame.

Replace it with a direct auto-framer (DigitalFramer): each frame compute the crop
straight from the selected target's bounding box — centre on the subject, size so
they fill ~62% of the crop, match the output aspect, clamp inside the frame, and
EMA-smooth. With no target the crop eases back to the whole frame. The crop now
tracks and zooms onto the subject as expected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
fix(engine): rebuild the PTZ backend live when the transport changes
4365612 
Toggling Center Stage flips ptz.backend to "digital", but update_config only
pushed PTZ *tuning* to the existing controller — it never rebuilt the backend.
On a webcam (no PTZ hardware) no backend exists at startup, so toggling Center
Stage on left _ptz_backend = None and _framed_output saw no DigitalPTZBackend:
Center Stage did nothing until an app restart.

update_config now detects a backend/address change and calls _rebuild_ptz_backend
(tear down + rebuild from the new config), so switching to/from the digital
backend takes effect live.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
fix(engine): robust Center Stage target resolution + diagnostic
9a86d86 
_current_digital_target now falls back to the maintained trusted target box when
the live track for the current id isn't in the latest _last_tracks (track-id
churn / identity re-bind timing) — so the crop holds instead of snapping to the
full frame. Adds a throttled center-stage diagnostic log (target found + crop
dims) to make 'not cropping' debuggable from the Logs panel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
feat(center-stage): force a visible crop + live tightness via Framing…
dd940de 
… dropdown

The framer was sizing the crop purely from subject fill, so a close subject that
already fills the sensor produced a crop == the whole frame (diagnostic showed
crop=1920x1080) — no visible Center Stage effect. Now the crop is constrained to
a window (max_frac of the frame), so it always pans/zooms like real Center Stage,
and the headroom lift keeps the head while cropping the torso/desk below.

Crop tightness now follows the live 'Framing' dropdown (tracking.framing):
face 2.0x / head_shoulders 1.6x / upper_body 1.35x (default) / full_body 1.06x —
dial the shot with no restart. Framer gains max_frac; tests updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC @claude
fix(center-stage): release the virtual camera when output is turned off
06d9f3e 
_push_frame only ever opened the VirtualCamSink; toggling 'Virtual camera
output' off left the device open on a frozen frame (it only closed at worker
stop). Now it closes and clears _vcam when vcam_out is disabled, so the virtual
camera disconnects from Zoom/OBS immediately. (pre-merge audit nit, PR #100)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TCVinNYC TCVinNYC merged commit 9c036aa into main Jun 24, 2026
5 of 6 checks passed
@TCVinNYC TCVinNYC deleted the feat/cpu-spikes-and-center-stage branch June 24, 2026 23:07
@TCVinNYC TCVinNYC mentioned this pull request Jun 24, 2026
Merged
TCVinNYC added a commit that referenced this pull request Jun 24, 2026
@TCVinNYC @claude
release: 2.2.0-rc3 (#102)
6149c47 
Cuts **2.2.0-rc3** off `main` (which now includes #100).

## What's in this RC (since rc2)
- **Center Stage** — software auto-framing / digital PTZ for non-PTZ
cameras: one toggle that crops, zooms, and pans to keep the selected
target framed (head-and-shoulders, holds steady through track
re-association), with **Framing**-dropdown tightness (Face / Head &
shoulders / Upper body / Full body) and an optional virtual-camera
output that disconnects cleanly when off. Raw PTZ transport selector
moved to PTZ → Advanced.
- **CPU-spike fixes** — OpenCV thread cap, ReID on GPU (`mps`/CUDA,
`AUTOPTZ_REID_DEVICE` override), `AUTOPTZ_COREML_UNITS` knob to verify
the GPU path, preview rate cap, ego-motion freshness gate (ping-pong
fix).
- **Tracking fixes** — named-target no longer drifts to the wrong person
(identity-gated ReID rebind); no longer chases an occluded subject onto
the visible remnant (occlusion coast).
- Removed the dead `tracking.face_confirm` toggle; consolidated the
unified-pose flag.

## Release mechanics
- `__version__` 2.2.0-rc2 → **2.2.0-rc3** (single source;
`pyproject.toml` derives it).
- CHANGELOG `[Unreleased]` rolled into `[2.2.0-rc3]`.
- Tag `v2.2.0-rc3` will be created on the squash-merge commit.

Pre-merge adversarial audit of #100 ran clean (perf/threading 0
findings; engine fixes positively verified). Full suite green locally
(1075) and on CI.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@TCVinNYC