feat: enable evals in 64 agentic workflows - #46774
Merged
Merged
Conversation
Add eval questions to 64 workflows that previously had none, covering A/B advisors, CI tools, daily reporters, code analysis, security scanners, and more. All workflows recompiled successfully. Workflows updated: ab-testing-advisor, ace-editor, agent-performance-analyzer, agent-persona-explorer, agentic-token-audit, agentic-token-optimizer, agentic-token-trend-audit, api-consumption-report, approach-validator, archie, artifacts-summary, audit-workflows, avenger, bot-detection, brave, ci-doctor, claude-code-user-docs-review, cli-version-checker, code-simplifier, copilot-agent-analysis, copilot-centralization-drilldown, copilot-centralization-optimizer, copilot-opt, copilot-pr-merged-report, copilot-pr-nlp-analysis, copilot-pr-prompt-analysis, copilot-session-insights, craft, daily-agent-of-the-day-blog-writer, daily-ambient-context-optimizer, daily-architecture-diagram, daily-astrostylelite-markdown-spellcheck, daily-aw-cross-repo-compile-check, daily-cache-strategy-analyzer, daily-caveman-optimizer, daily-cli-performance, daily-cli-tools-tester, daily-code-metrics, daily-community-attribution, daily-compiler-threat-spec-optimizer, daily-elixir-credo-snippet-audit, daily-evals-report, daily-experiment-report, daily-fact, daily-file-diet, daily-firewall-report, daily-geo-optimizer, daily-hippo-learn, daily-mcp-concurrency-analysis, daily-model-inventory, daily-observability-report, daily-performance-summary, daily-regulatory, daily-reliability-review, daily-repo-chronicle, daily-safe-output-integrator, daily-safe-output-optimizer, daily-safe-outputs-conformance, daily-secrets-analysis, daily-security-observability, daily-security-red-team, daily-sentrux-report, hourly-ci-cleaner, issue-arborist Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot created this pull request from a session on behalf of
pelikhan
July 20, 2026 08:47
View session
pelikhan
marked this pull request as ready for review
July 20, 2026 08:57
Contributor
There was a problem hiding this comment.
Pull request overview
Adds binary task and artifact evaluations to 64 agentic workflows and regenerates affected compiled workflows.
Changes:
- Adds core-task and outcome evaluation questions.
- Covers CI, reporting, security, analysis, and slash-command workflows.
- Regenerates 10 lock workflows with evaluation and state-persistence jobs.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/issue-arborist.md |
Adds issue-relationship evals. |
.github/workflows/hourly-ci-cleaner.md |
Adds CI remediation evals. |
.github/workflows/daily-sentrux-report.md |
Adds quality-report evals. |
.github/workflows/daily-sentrux-report.lock.yml |
Compiles Sentrux eval jobs. |
.github/workflows/daily-security-red-team.md |
Adds security-scan evals. |
.github/workflows/daily-security-observability.md |
Adds observability evals. |
.github/workflows/daily-secrets-analysis.md |
Adds secrets-analysis evals. |
.github/workflows/daily-safe-outputs-conformance.md |
Adds conformance evals. |
.github/workflows/daily-safe-output-optimizer.md |
Adds safe-output optimization evals. |
.github/workflows/daily-safe-output-integrator.md |
Adds coverage-integration evals. |
.github/workflows/daily-repo-chronicle.md |
Adds chronicle evals. |
.github/workflows/daily-reliability-review.md |
Adds reliability evals. |
.github/workflows/daily-regulatory.md |
Adds regulatory-report evals. |
.github/workflows/daily-regulatory.lock.yml |
Compiles regulatory eval jobs. |
.github/workflows/daily-performance-summary.md |
Adds performance-summary evals. |
.github/workflows/daily-observability-report.md |
Adds telemetry-report evals. |
.github/workflows/daily-model-inventory.md |
Adds model-inventory evals. |
.github/workflows/daily-mcp-concurrency-analysis.md |
Adds concurrency evals. |
.github/workflows/daily-hippo-learn.md |
Adds memory-learning evals. |
.github/workflows/daily-hippo-learn.lock.yml |
Compiles Hippo eval jobs. |
.github/workflows/daily-geo-optimizer.md |
Adds GEO audit evals. |
.github/workflows/daily-firewall-report.md |
Adds firewall-report evals. |
.github/workflows/daily-firewall-report.lock.yml |
Compiles firewall eval jobs. |
.github/workflows/daily-file-diet.md |
Adds large-file evals. |
.github/workflows/daily-fact.md |
Adds verse-publication evaluation. |
.github/workflows/daily-experiment-report.md |
Adds experiment-report evals. |
.github/workflows/daily-evals-report.md |
Adds eval-adoption evals. |
.github/workflows/daily-elixir-credo-snippet-audit.md |
Adds Credo audit evals. |
.github/workflows/daily-compiler-threat-spec-optimizer.md |
Adds threat-coverage evals. |
.github/workflows/daily-community-attribution.md |
Adds attribution evals. |
.github/workflows/daily-code-metrics.md |
Adds code-metrics evals. |
.github/workflows/daily-cli-tools-tester.md |
Adds CLI testing evals. |
.github/workflows/daily-cli-performance.md |
Adds benchmark evals. |
.github/workflows/daily-caveman-optimizer.md |
Adds instruction-optimization evals. |
.github/workflows/daily-cache-strategy-analyzer.md |
Adds cache-analysis evals. |
.github/workflows/daily-aw-cross-repo-compile-check.md |
Adds compatibility evals. |
.github/workflows/daily-astrostylelite-markdown-spellcheck.md |
Adds spellcheck evals. |
.github/workflows/daily-architecture-diagram.md |
Adds architecture evals. |
.github/workflows/daily-ambient-context-optimizer.md |
Adds context-optimization evals. |
.github/workflows/daily-agent-of-the-day-blog-writer.md |
Adds blog-generation evals. |
.github/workflows/craft.md |
Adds workflow-crafting evals. |
.github/workflows/copilot-session-insights.md |
Adds session-analysis evals. |
.github/workflows/copilot-pr-prompt-analysis.md |
Adds prompt-analysis evals. |
.github/workflows/copilot-pr-nlp-analysis.md |
Adds conversation-analysis evals. |
.github/workflows/copilot-pr-merged-report.md |
Adds merged-PR evals. |
.github/workflows/copilot-opt.md |
Adds Copilot optimization evals. |
.github/workflows/copilot-centralization-optimizer.md |
Adds centralization evals. |
.github/workflows/copilot-centralization-optimizer.lock.yml |
Compiles centralization eval jobs. |
.github/workflows/copilot-centralization-drilldown.md |
Adds drilldown evals. |
.github/workflows/copilot-centralization-drilldown.lock.yml |
Compiles drilldown eval jobs. |
.github/workflows/copilot-agent-analysis.md |
Adds agent-usage evals. |
.github/workflows/code-simplifier.md |
Adds simplification evals. |
.github/workflows/code-simplifier.lock.yml |
Compiles simplifier eval jobs. |
.github/workflows/cli-version-checker.md |
Adds version-update evals. |
.github/workflows/claude-code-user-docs-review.md |
Adds documentation-review evals. |
.github/workflows/ci-doctor.md |
Adds CI diagnosis evals. |
.github/workflows/brave.md |
Adds web-search evals. |
.github/workflows/bot-detection.md |
Adds suspicious-activity evals. |
.github/workflows/avenger.md |
Adds CI-fixer evals. |
.github/workflows/audit-workflows.md |
Adds workflow-audit evals. |
.github/workflows/artifacts-summary.md |
Adds artifact-summary evals. |
.github/workflows/archie.md |
Adds diagram-response evals. |
.github/workflows/approach-validator.md |
Adds approach-validation evals. |
.github/workflows/api-consumption-report.md |
Adds API-usage evals. |
.github/workflows/agentic-token-trend-audit.md |
Adds AIC trend evals. |
.github/workflows/agentic-token-optimizer.md |
Adds AIC optimization evals. |
.github/workflows/agentic-token-audit.md |
Adds AIC audit evals. |
.github/workflows/agentic-token-audit.lock.yml |
Compiles AIC audit eval jobs. |
.github/workflows/agent-persona-explorer.md |
Adds persona-analysis evals. |
.github/workflows/agent-persona-explorer.lock.yml |
Compiles persona eval jobs. |
.github/workflows/agent-performance-analyzer.md |
Adds performance-analysis evals. |
.github/workflows/ace-editor.md |
Adds ACE-link evals. |
.github/workflows/ab-testing-advisor.md |
Adds experiment-advisor evals. |
.github/workflows/ab-testing-advisor.lock.yml |
Compiles advisor eval jobs. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 88/128 changed files
- Comments generated: 13
- Review effort level: Medium
Comment on lines
+77
to
+81
| evals: | ||
| - id: aic_usage_analyzed | ||
| question: Did the agent audit AI Credit (AIC) usage across agentic workflows for the target period? | ||
| - id: report_or_discussion_created | ||
| question: Was a report or discussion created with historical trend data and per-workflow AIC breakdowns? |
Comment on lines
+112
to
+116
| evals: | ||
| - id: high_aic_workflow_identified | ||
| question: Did the agent identify a high-AIC workflow and audit its recent runs? | ||
| - id: recommendations_produced | ||
| question: Were efficiency improvement recommendations produced, including any inline sub-agent refactor suggestions? |
Comment on lines
+209
to
+213
| evals: | ||
| - id: ci_failure_investigated | ||
| question: Did the agent investigate failed CI workflows or PR check failures? | ||
| - id: diagnostic_issue_created_or_noop | ||
| question: Was a diagnostic issue created with root cause analysis, or was noop used when no actionable failures were found? |
Comment on lines
+71
to
+73
| evals: | ||
| - id: verse_posted | ||
| question: Did the agent post a daily poetic verse about the gh-aw project to a discussion thread? |
Comment on lines
+84
to
+85
| - id: entry_published | ||
| question: Was the blog entry published as a discussion or committed to the repository? |
Comment on lines
+287
to
+288
| - id: updates_proposed_or_noop | ||
| question: Were updates to the builtin model alias mapping proposed, or was noop used when no changes were needed? |
Comment on lines
+44
to
+45
| - id: test_results_reported | ||
| question: Were the test results reported with any issues or anomalies identified? |
Comment on lines
+74
to
+75
| - id: diagram_created_or_noop | ||
| question: Was an ASCII architecture diagram created or updated, or was noop used when no significant changes were detected? |
Comment on lines
+164
to
+165
| - id: recommendations_produced | ||
| question: Were recommendations produced for prompt, skill, and agent changes to reduce ambient context size? |
Comment on lines
+305
to
+306
| - id: recommendations_produced | ||
| question: Were recommendations produced for new workflows, tools, shared prompts, or other automations? |
Contributor
|
🎉 This pull request is included in a new release. Release: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
108 of 259 workflows now have evals (up from 44). Adds
evals:sections to 64 workflows that were previously unevaluated, covering two outcome dimensions per workflow: whether the core task executed, and whether the expected artifact (issue, PR, discussion, report) was produced or noop was correctly invoked.Workflows updated
avenger,ci-doctor,hourly-ci-cleanerdaily-code-metrics,daily-firewall-report,daily-performance-summary,daily-observability-report,daily-reliability-review,daily-repo-chronicle,daily-sentrux-report,daily-security-red-team,daily-security-observability,daily-evals-report,daily-experiment-report, and ~10 moredaily-safe-output-integrator,daily-safe-output-optimizer,daily-safe-outputs-conformance,schema-feature-coveragecopilot-opt,copilot-agent-analysis,copilot-session-insights,copilot-pr-merged-report,copilot-pr-nlp-analysis,copilot-pr-prompt-analysisagentic-token-audit,agentic-token-optimizer,agentic-token-trend-audit,api-consumption-reportarchie,brave,ace-editor,craftcode-simplifier,semantic-function-refactor(via daily batches),audit-workflows,artifacts-summarydaily-community-attribution,daily-regulatory,uk-ai-operational-resilience,security-complianceEval shape
Each entry follows the existing convention — a binary
questionanswerable from run output: