Skip to content

feat: unified in-process worker (replace Docker containers)#5

Merged
TytaniumDev merged 2 commits intomainfrom
feature/worker-visibility-frontend
Feb 7, 2026
Merged

feat: unified in-process worker (replace Docker containers)#5
TytaniumDev merged 2 commits intomainfrom
feature/worker-visibility-frontend

Conversation

@TytaniumDev
Copy link
Owner

Summary

  • Replace Docker containers with direct child process execution for forge simulations and in-process log condensing/GCS upload
  • Add dynamic parallelism that scales based on available CPU cores and memory instead of a fixed parallelism count
  • Make run_sim.sh configurable via environment variables (FORGE_PATH, DECKS_DIR, LOGS_DIR, RUN_DECKS_DIR) so it works in both standalone container and local/unified execution modes
  • Add new modules: condenser.ts (log parsing/condensing), gcs-client.ts (GCS upload), types.ts (shared types)
  • Add unified worker Docker setup (unified-worker/) with Dockerfile and docker-compose for containerized deployment

Test plan

  • Run the worker locally with FORGE_PATH pointed at a local Forge installation and verify simulations execute
  • Verify log condensing produces valid condensed.json and analyze-payload.json
  • Verify GCS upload works with valid credentials
  • Test dynamic parallelism calculation on machines with varying CPU/memory
  • Build and run the unified-worker Docker container end-to-end
  • Verify run_sim.sh still works in the standalone forge-sim container

🤖 Generated with Claude Code

Replace forge-sim and misc-runner Docker containers with direct child
process execution and in-process log condensing/GCS upload. This
eliminates Docker as a runtime dependency, adds dynamic parallelism
based on system resources, and makes run_sim.sh configurable via
environment variables for both container and local execution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@TytaniumDev
Copy link
Owner Author

Code review

Found 1 issue:

  1. Parallel forge runs share the same RUN_DECKS_DIR (defaults to $HOME/.forge/decks/commander in run_sim.sh). Each run does `rm -rf` and copies decks there, so when parallelism > 1 they race and overwrite each other's decks; Forge may run with the wrong decks or see removed files. Fix: set RUN_DECKS_DIR per run to a unique directory (e.g. under the job dir) and pass it in the env from `runForgeSim`.

local-worker/src/worker.ts (runForgeSim env)

forge-simulation-engine/run_sim.sh (RUN_DECKS_DIR default and rm/cp) and L94-L101

Generated with Claude Code

- If this code review was useful, please react with a thumbs up. Otherwise, react with a thumbs down.

- run_sim.sh: symlink decks from DECKS_DIR to RUN_DECKS_DIR instead of copy
- Orchestrator: GET /api/decks/:id/content (worker auth) for deck content
- Orchestrator: store deckIds on job; GET /api/jobs/:id returns decks and/or deckIds for workers
- Worker: DECK_CACHE_DIR, on-demand fetch on cache miss, processJob uses cache when job has deckIds
- Fix orchestrator-service/.gitignore: /jobs/ so app/api/jobs routes are not ignored

Co-authored-by: Cursor <cursoragent@cursor.com>
@TytaniumDev TytaniumDev merged commit e9c9888 into main Feb 7, 2026
2 checks passed
@TytaniumDev TytaniumDev deleted the feature/worker-visibility-frontend branch February 7, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant