Skip to content

fix: keep standard scoped scans out of ranking - #71

Open
createsomethingtoday wants to merge 2 commits into
openai:mainfrom
createsomethingtoday:agent/standard-scan-scope-inventory
Open

fix: keep standard scoped scans out of ranking#71
createsomethingtoday wants to merge 2 commits into
openai:mainfrom
createsomethingtoday:agent/standard-scan-scope-inventory

Conversation

@createsomethingtoday

@createsomethingtoday createsomethingtoday commented Jul 29, 2026

Copy link
Copy Markdown

What

  • replace the scoped standard-scan make-repo-rank-input bootstrap with a deterministic make-scope-inventory command
  • make the TypeScript SDK materialize that inventory before Codex starts, so setup cannot be deferred until after model spending
  • inventory every first-party regular file in the requested scopes, including tests, examples, README files, binaries, and generated files, while excluding .git, node_modules, and symlinks
  • use JSONL path records so unusual filenames remain data-safe
  • let candidate normalization consume the new inventory while retaining the legacy newline-delimited scope option for existing deep/diff workflows
  • align the standard-scan prompt, capability profile, skill, and artifact documentation with the no-ranking contract
  • when the workbench permits exactly one target kind, bind that authoritative kind during finalization and remove incompatible model-authored target coordinates

Why

The scoped-path SDK prompt instructed a standard scan to call make-repo-rank-input, even though the standard-scan skill explicitly says not to rank or filter files. That helper deliberately excludes tests, README files, binaries, and other non-source-like files.

A first report-only pilot against a 28-file authentication package spent 5.34M input tokens and reached its $5.049 cost limit while still in preflight. It produced a 20-row rank_input.jsonl, zero worklist rows, and no contract, coverage, report, or findings artifacts. The eight omitted files were the package README and its test files. This is one concrete failure mode behind the cost-without-findings behavior discussed in #25.

A second capped run using the initial version of this PR proved that changing only the prompt/helper contract was insufficient: the model could still defer the inventory step. It reached $5.078 after 9 minutes with 5.48M input tokens, no inventory, and no worklist.

An explicitly uncapped run then reached the natural terminal after 63m47s and an estimated $56.302 (71.39M input tokens, 68.94M cached, 320k output). It demonstrated the scanner's substantive value by producing a threat model, 55 candidate-ledger rows, and 48 partial finding drafts. It also exposed two deterministic integration defects:

  1. the filesystem inventory included five bootstrapped node_modules/.bin wrappers in addition to the 28 requested first-party files;
  2. finalization rejected a model-authored git_worktree kind even though the clean workbench target permitted only git_revision: scan.target.kind: must match the workbench target.

This revision fixes both defects. The uncapped findings remain partial/unsealed and application remediation is intentionally outside this PR.

Validation

  • pnpm run format
  • pnpm run types
  • pnpm test (378 passed, 3 skipped, 0 failed)
  • pnpm run build
  • pnpm pack --pack-destination ../../dist
  • pnpm run check:package ../../dist/openai-codex-security-0.1.1.tgz (179 entries; public import, CLI, and 95 bundled plugin files validated)
  • pnpm run test:package

Regression coverage proves that:

  • the SDK writes the complete standard inventory before the Codex turn starts;
  • README, binary, source, and test files remain in scope while .git, node_modules, and symlinks do not;
  • candidates located in test files are accepted by normalization; and
  • a sole workbench-allowed target kind replaces an incompatible model-authored draft during finalization.

@createsomethingtoday
createsomethingtoday marked this pull request as ready for review July 29, 2026 17:16
@createsomethingtoday

Copy link
Copy Markdown
Author

Post-fix recovery validation completed without another model turn. I preserved the failed original scan, registered a parent-linked recovery scan at the same repository revision and scope, copied the unsealed artifacts, and completed it through this branch. The corrected workbench sealed the contract successfully: validate_scan_contract.py reports status: valid; the workbench reports complete, 48 findings with derived IDs, and both Markdown and SARIF reports. A relative content digest confirmed the original artifact set and its preservation copy are identical. Separately, the corrected inventory helper emits exactly the 28 tracked scoped files and matches git ls-files with no node_modules or .git rows. The sealed coverage remains truthfully partial with four deferred questions, so this validates the integration/finalization fixes rather than claiming a complete security review.

@createsomethingtoday

Copy link
Copy Markdown
Author

Fresh natural-terminal validation now confirms this branch on the ordinary live path, not only recovery. A standard scoped scan against a separate 36-row authorization-package inventory completed after 32m44s. The model-authored unsealed manifest again used git_worktree; the workbench allowed only git_revision; this branch replaced the kind and incompatible snapshot coordinate, sealed successfully, generated Markdown/SARIF, and validate_scan_contract.py reports status: valid. Workbench readback is complete with complete coverage and 8 findings. The deterministic inventory was present before the Codex turn. Target revision remained clean and package TypeScript plus all 24 tests passed afterward. This directly validates both the pre-model inventory setup and singleton target-kind binding fixes in a new scan.

mldangelo-oai added a commit that referenced this pull request Jul 29, 2026
Preserve and harden the scoped-inventory design from external PR #71. Keep standard workflow instructions, explicitly selected dependency scopes, both attested inventories, and authoritative target finalization consistent with the released 0.1.3 base.

Co-authored-by: Create Something <micah@createsomething.io>
@createsomethingtoday

Copy link
Copy Markdown
Author

The extended exact-commit validation run reached completion after 56 minutes and surfaced two source-validated findings in the target project, but final save failed because immutable diff targets did not carry the required snapshotDigest. I opened focused follow-up PR #112 with a regression and the completion binding fix. The partial findings were retained and are being remediated before a fresh sealed run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant