Summary
- Run mode: dry-run
- Status: ✅ (exit code 0)
- Note: Suite execution skipped —
OPENROUTER_API_KEY not set; no benchmark results available
Key Findings
-
Add OPENROUTER_API_KEY secret to enable benchmark mode — The optimizer ran in dry-run only because OPENROUTER_API_KEY was absent. Without real model runs, skill quality regressions go undetected. Expected impact: converts the workflow from a lint-only check to a full graded evaluation, catching skill degradation before it ships.
-
Expand the agentic-workflows skill router with explicit fallback guidance — The SKILL.md lists 40+ prompt files to load lazily but provides no decision tree or fallback when the task type is ambiguous. Agents must guess which .github/aw/*.md file to load. Expected impact: reduces missed or wrong skill loads, improving first-turn accuracy for new workflow types.
-
Add a skill-coverage map linking each .github/skills/ entry to the workflows that consume it — There are 35+ skill directories but no registry showing which workflows invoke which skills. Stale or orphaned skills accumulate silently. Expected impact: makes it easy to audit usage, deprecate unused skills, and ensure new workflows are covered.
Evidence from Artifact
summary.json (/tmp/gh-aw/agent/skill-optimizer-results/summary.json):
{
"repository": "github/gh-aw",
"run_mode": "dry-run",
"run_status": 0,
"run_url": "https://github.com/github/gh-aw/actions/runs/29799428993"
}run.log excerpt:
dry-run: Docker available but OPENROUTER_API_KEY not set; skipping suite execution
.github/skills/agentic-workflows/SKILL.md (lines 1–50): The router lists 40+ .github/aw/*.md files with no branching logic or fallback — an agent must infer the correct file from task wording alone.
.github/skills/ directory listing: 35 skill directories present with no cross-reference registry or usage manifest.
Recommendations
-
Enable benchmark mode: Add OPENROUTER_API_KEY as a repository secret (or environment secret scoped to the skill-optimizer workflow). Update the workflow YAML to pass it to the optimizer job so real model evaluations run on every scheduled trigger.
-
Add a decision-tree fallback to the agentic-workflows router: In .github/skills/agentic-workflows/SKILL.md, add a short decision table (task keyword → prompt file) and a catch-all fallback instruction (e.g., "if no match, load .github/aw/create-agentic-workflow.md and ask one clarifying question").
-
Create a skill registry / coverage map: Add a .github/skills/REGISTRY.md (or a skills.yml manifest) that maps each skill directory to the workflows, job steps, or system-prompt sections that reference it. Include a lint step (e.g., in Makefile) that flags skills with zero references.
Generated by ⚡ Daily Skill Optimizer Improvements · 15.6 AIC · ⌖ 5.67 AIC · ⊞ 4.4K · ◷
Summary
OPENROUTER_API_KEYnot set; no benchmark results availableKey Findings
Add OPENROUTER_API_KEY secret to enable benchmark mode — The optimizer ran in dry-run only because
OPENROUTER_API_KEYwas absent. Without real model runs, skill quality regressions go undetected. Expected impact: converts the workflow from a lint-only check to a full graded evaluation, catching skill degradation before it ships.Expand the
agentic-workflowsskill router with explicit fallback guidance — TheSKILL.mdlists 40+ prompt files to load lazily but provides no decision tree or fallback when the task type is ambiguous. Agents must guess which.github/aw/*.mdfile to load. Expected impact: reduces missed or wrong skill loads, improving first-turn accuracy for new workflow types.Add a skill-coverage map linking each
.github/skills/entry to the workflows that consume it — There are 35+ skill directories but no registry showing which workflows invoke which skills. Stale or orphaned skills accumulate silently. Expected impact: makes it easy to audit usage, deprecate unused skills, and ensure new workflows are covered.Evidence from Artifact
summary.json(/tmp/gh-aw/agent/skill-optimizer-results/summary.json):{ "repository": "github/gh-aw", "run_mode": "dry-run", "run_status": 0, "run_url": "https://github.com/github/gh-aw/actions/runs/29799428993" }run.logexcerpt:.github/skills/agentic-workflows/SKILL.md(lines 1–50): The router lists 40+.github/aw/*.mdfiles with no branching logic or fallback — an agent must infer the correct file from task wording alone..github/skills/directory listing: 35 skill directories present with no cross-reference registry or usage manifest.Recommendations
Enable benchmark mode: Add
OPENROUTER_API_KEYas a repository secret (or environment secret scoped to theskill-optimizerworkflow). Update the workflow YAML to pass it to the optimizer job so real model evaluations run on every scheduled trigger.Add a decision-tree fallback to the agentic-workflows router: In
.github/skills/agentic-workflows/SKILL.md, add a short decision table (task keyword → prompt file) and a catch-all fallback instruction (e.g., "if no match, load.github/aw/create-agentic-workflow.mdand ask one clarifying question").Create a skill registry / coverage map: Add a
.github/skills/REGISTRY.md(or askills.ymlmanifest) that maps each skill directory to the workflows, job steps, or system-prompt sections that reference it. Include a lint step (e.g., inMakefile) that flags skills with zero references.