[codex] require strategy evidence risk metrics#188
Conversation
Co-Authored-By: Codex <noreply@openai.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c2db9ecb5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if not isinstance(risk, dict): | ||
| issues.append("risk must be an object") | ||
| else: | ||
| metrics = risk.get("metrics") |
There was a problem hiding this comment.
Adding the stricter risk checks only to this standalone script leaves the documented promotion path unprotected: README.md:53 tells operators to run quant-lifecycle evidence, and src/quant_platform_kit/strategy_lifecycle/cli.py:163-167 delegates to strategy_lifecycle.evidence_gate.validate_evidence_package_file, whose validator never reads risk at all (evidence_gate.py:91-138). I verified that a live_candidate package with no risk.metrics, risk.benchmark, risk.cost_stress, or risk.oos is still reported valid through that CLI, so production promotion checks can bypass the new minimum risk-evidence standard unless this validation is wired into the lifecycle validator as well.
Useful? React with 👍 / 👎.
| def _is_number(value: Any) -> bool: | ||
| return isinstance(value, (int, float)) and not isinstance(value, bool) |
There was a problem hiding this comment.
When evidence packages are emitted by Python tooling, json.dumps/json.loads can round-trip NaN and Infinity by default, and this predicate accepts those floats as valid numbers for the new risk metrics, benchmark values, and cost-stress values. I verified validate_payload returns no issues for risk.metrics.sharpe_ratio = NaN, risk.metrics.win_rate = NaN, and risk.benchmark.alpha = Infinity; NaN also bypasses the win-rate bounds because both comparisons are false, so the audit gate can approve metrics that strict JSON/schema consumers cannot parse or that downstream risk review cannot interpret.
Useful? React with 👍 / 👎.
| elif not _is_number(value): | ||
| issues.append(f"risk.metrics.{field} must be a number") |
There was a problem hiding this comment.
Because profit_factor is part of REQUIRED_RISK_METRICS, this branch rejects it unless it is numeric, but the repo's own metric calculator returns None when a window has no gross loss (src/quant_platform_kit/strategy_lifecycle/performance_metrics.py:101) and the contract explicitly models profit_factor: float | None (contracts.py:37). In that no-losing-trades scenario, otherwise complete risk evidence generated from the existing lifecycle metrics cannot pass this new validator, so the field should allow a null/undefined value or encode the no-loss case consistently.
Useful? React with 👍 / 👎.
Summary
Validation
python3 -m json.tool schemas/strategy-evidence-package.schema.json >/dev/nullpython3 -m pytest -q tests/test_strategy_evidence_package_validator.py tests/test_validate_strategy_evidence_package.pypython3 -m ruff check .PYTHONPATH=src python3 -m pytest -q tests