Add skill-validator structural checks for skills and agents#126814
Conversation
Run skill-validator check (from dotnet/skills nightly release) on PRs that touch .github/skills/ or .github/agents/. The validator is an AOT-compiled binary that performs static analysis -- no .NET runtime or secrets required. Key design choices: - Path-filtered triggers to avoid firing on unrelated PRs - Sparse checkout for minimal clone - Separate invocations for skills and agents (the tool requires exactly one of --plugin, --skills, or --agents per call) - Concurrency group with cancel-in-progress to avoid wasting runners - Output captured to step summary for easy diagnosis - Nightly binary intentionally used (not pinned) so validation improvements from dotnet/skills are picked up automatically Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
As I understand it, this is the "future proof" way to validate -- as the validator there is improved ("vally"?) we pick it up for free. |
There was a problem hiding this comment.
Pull request overview
Adds a GitHub Actions workflow to validate the structure of repo-local skills and agents using the skill-validator tool, so malformed frontmatter/missing fields and size warnings are caught during PR validation.
Changes:
- Introduces a new path-filtered workflow for
.github/skills/**and.github/agents/**. - Downloads the
skill-validatorLinux x64 nightly release and runs separate checks for skills and agents. - Writes a truncated (200 lines) failure log into the GitHub Actions step summary for faster diagnosis.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/skill-validation.yml |
New CI workflow to download and run skill-validator check for skills and agents with sparse checkout and summarized output. |
Copilot's findings
- Files reviewed: 1/1 changed files
- Comments generated: 2
Local test resultsRan Pre-existing issues found and fixed in this PRSkills check (
Agents check (
Fixes applied
Current stateBoth checks now pass clean: The validator also reports useful warnings (not errors) about token counts -- several skills are in the "comprehensive" range (>5K tokens) where the SkillsBench research suggests diminishing returns. These are informational and don't block the check. |
- api-proposal: remove deep file reference to copilot-instructions.md - update-os-coverage: remove deep file references to copilot-instructions.md and os-onboarding.md (3 occurrences) - issue-triage: remove parent-directory traversal reference to area-owners.md - agentic-workflows agent: add missing 'name' field in frontmatter All references changed from markdown links to plain text mentions. The agent still has access to these files via repo context. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
@ViktorHofer I'm trimming some lines from one of the skills to get under 500. Honestly this seems a bit arbitrary. I just collapsed bullets using semicolons. I thought what AI cared about was tokens! Can you expose |
Condense verbose multi-line bullet points into single lines in the Anti-Patterns, Tips, and Related Skills sections. No content removed, just tightened formatting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Actions workflow to automatically run skill-validator check on changes to repo-local Copilot skills and agents, helping catch structural issues early.
Changes:
- Introduces a path-filtered Actions workflow to validate
.github/skills/**and.github/agents/**using theskill-validatornightly binary. - Adjusts several skill markdown files to align with expected structure/formatting.
- Adds required frontmatter metadata (
name) to theagentic-workflowsagent definition.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/skill-validation.yml |
New CI workflow that downloads and runs skill-validator check for skills and agents with step-summary output. |
.github/skills/update-os-coverage/SKILL.md |
Minor formatting/link presentation adjustments in the skill guidance text. |
.github/skills/issue-triage/SKILL.md |
Formatting/wording updates; clarifies anti-pattern guidance and restructures some reference content. |
.github/skills/api-proposal/SKILL.md |
Updates prerequisite reference to the repo build/test instructions. |
.github/agents/agentic-workflows.agent.md |
Adds name to YAML frontmatter for validator compatibility. |
Copilot's findings
Comments suppressed due to low confidence (1)
.github/skills/api-proposal/SKILL.md:140
- In this section the file path to the build/test instructions is referenced inconsistently: the blockquote points to
.github/copilot-instructions.md, but the next sentence refers tocopilot-instructions.mdwithout the.github/path. Please make these references consistent so readers can reliably locate the document (ideally use the same path/link in both places).
> **Prerequisite:** Follow the build and test workflow in `.github/copilot-instructions.md` — complete the baseline build, configure the environment, and use the component-specific workflow for the target library. All build and test steps below assume the baseline build has already succeeded.
**Step 1: Build and test**
Build the src and test projects, then run all tests for the target library using the workflow described in `copilot-instructions.md`. All tests must pass with zero failures.
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Move existence check before chmod so the custom error message is emitted instead of a cryptic bash -e failure - Add missing comma in issue-triage SKILL.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Tagging subscribers to this area: @dotnet/runtime-infrastructure |
🤖 Copilot Code Review — PR #126814Note This review was generated by Copilot. Holistic AssessmentMotivation: Well-justified. Adding CI validation for skills/agents markdown files is a sensible quality gate, and the structural fixes (missing Approach: Sound. The workflow follows good practices — minimal permissions ( Summary: ✅ LGTM. The workflow is well-designed, the markdown fixes are correct, and the issue-triage condensing preserves semantic content. Two minor observations below are advisory only. Detailed Findings✅ Workflow Design — Well-structured CI gateThe ✅ Markdown Link Fixes — Correct structural changesReplacing deep relative markdown links ( ✅ Agent Frontmatter Fix — Missing
|
|
Agreed. I'm not sure where the line check is coming from. LLM is telling me the following: Feel free to submit a PR to skill-validator to change/remove the line metric. |
|
Ok. Can you sign off? |
Follow-up of #126814 ### Motivation - The repo local skills differ from plugin skills: - the later must not traverse outside of their folder, as it would break their isolated acquisition - the former are alredy part of the repo. If we prevent them to traverse valuable files outside of their folder (docs, scripts, etc.) - we force the agents to do at least one extra full turn to discover the full path. **tl;dr;:** reintroducing full relative paths in skills - this is **good** pattern in repo local skills. Also adding flag that is allowing this. cc: @danmoseley --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
### Context Let's use the skill-validator to check for basic problems of our skills and agents This is inspired and sourced from similar integrations: * github/awesome-copilot#1195 * dotnet/runtime#126814 ### Changes Made - Added pipeline using the skill-validator - Fixed the current findings (renames of folders and agent file) ### Testing - skill-validator is the test harness for the skills
<!-- Please let the below note in for people that find this PR --> > [!NOTE] > Are you waiting for the changes in this PR to be merged? > It would be very helpful if you could [test the resulting artifacts](https://github.com/dotnet/maui/wiki/Testing-PR-Builds) from this PR and let us know in a comment if this change resolves your issue. Thank you! ### Description of Change Adopting skills-validator This is inspired and sourced from similar integrations: * github/awesome-copilot#1195 * dotnet/runtime#126814 * dotnet/msbuild#13537 One important difference is that in this case we run as well in the LLM judging mode cc: @PureWeen - for awareness and for help with iterating on this :)
Summary
Add a GitHub Actions workflow that runs
skill-validator check(from dotnet/skills nightly release) on PRs that touch.github/skills/or.github/agents/. The validator is an AOT-compiled binary that performs structural static analysis -- no .NET runtime or secrets required.dotnet/runtime currently has 11 skills and 2 agents with no validation. This catches structural issues (malformed frontmatter, missing fields, skill size warnings) before merge.
Design choices
.github/skills/**,.github/agents/**, or the workflow itself. With runtime's high PR volume, 99%+ of PRs won't trigger this.skill-validator checkrequires exactly one of--plugin,--skills, or--agentsper call. Both run regardless of individual failures to give complete diagnostics.Relation to dotnet/arcade-skills#19
This is the runtime equivalent of dotnet/arcade-skills#19, adapted for runtime's
.github/skills/layout (repo-local skills) rather than arcade-skills'plugins/layout (marketplace format). Key learnings from that PR were applied: curl retries, extraction verification, PIPESTATUS for exit code through tee, and step summary output.Testing
This workflow won't fire in this PR's CI because the workflow doesn't exist on
mainyet. After merge, any PR touching.github/skills/**or.github/agents/**will trigger it. Can also be tested viaworkflow_dispatch.cc @lewing @ViktorHofer