📚 Documentation Noob Test Report - 2026-03-06

[github-actions[bot]]

Summary

• Date of test: 2026-03-06
• Pages visited:
  • /gh-aw/ (Home page)
  • /gh-aw/setup/quick-start/ (Quick Start)
  • /gh-aw/setup/cli/ (CLI Commands)
  • /gh-aw/setup/creating-workflows/ (Creating Agentic Workflows)
  • /gh-aw/introduction/overview/ (About Workflows)
  • /gh-aw/introduction/how-they-work/ (How They Work)
  • /gh-aw/reference/auth/ (Authentication)
  • /gh-aw/reference/glossary/ (Glossary)
  • /gh-aw/examples/issue-pr-events/triage-analysis/ (Examples)
  • /gh-aw/examples/comment-triggered/ (Examples)
• Overall impression: The documentation has a lot of great content and video demos, but a brand-new user without GitHub Actions knowledge will hit several confusing walls—especially around the "compile" step, the mysterious create.md URL, and the promise that their first run will fail.

---

🔴 Critical Issues Found

1. The create.md URL in "Creating Workflows" is unexplained

Page: /gh-aw/setup/creating-workflows/

The guide instructs users to paste the following prompt into their AI agent:

Create a workflow for GitHub Agentic Workflows using https://github.com/github/gh-aw/main/create.md

The purpose of the workflow is...

A complete beginner has no idea:

• What create.md is or why it's at a raw GitHub URL
• Why the AI agent needs to "use" a URL in a chat message
• What happens if they can't access that URL
• Why they're pasting a URL into a chat instead of a command

Suggestion: Add a brief callout like:

What is create.md? This is a prompt template that teaches the AI agent how to create gh-aw workflows correctly. The AI will fetch and follow it automatically when you include the URL.

---

2. "First run will surely fail" — but no recovery path

Page: /gh-aw/setup/creating-workflows/

The guide includes this TIP:

On the first run in a new repository, the workflow will surely fail because the secrets are not configured. The agentic workflow should detect the missing tokens and create an issue with instructions on how to configure them.

For a beginner, being told "your first run will fail" with no explicit next step is alarming and confusing. What does "should detect" mean? Is that guaranteed? Where does the issue appear? What if it doesn't create an issue?

Suggestion: Either move the secret setup step before the first run, or add a more explicit "What to do when it fails" section with a direct link to the secrets setup guide.

---

3. Examples pages don't include actual runnable workflow code

Pages: /gh-aw/examples/issue-pr-events/triage-analysis/, /gh-aw/examples/comment-triggered/, etc.

The Examples section describes patterns and links to external repositories, but doesn't include copy-paste-ready workflow files. A beginner expecting complete, working examples to copy and adapt will be disappointed—they'll need to navigate to an external GitHub repo, understand the file structure there, and copy files manually.

Suggestion: Include at least one complete, copy-pasteable workflow (frontmatter + markdown body) per example page, even if it's simplified.

---

🟡 Confusing Areas

4. "Frontmatter" used before it's defined

Page: Home page (/gh-aw/)

The home page uses "frontmatter" in the example code comment (# Frontmatter) without explanation. For a user who has never seen YAML configuration blocks in markdown files, this is mysterious. The term is properly defined in the Glossary and Introduction, but a first-time visitor lands on the home page first.

Suggestion: Add a tooltip, popover, or inline link on the very first use of "frontmatter" on the home page. A brief parenthetical like "(the YAML configuration at the top)" would help.

---

5. When to run gh aw compile is unclear

Pages: /gh-aw/setup/quick-start/ (Step 4), /gh-aw/introduction/overview/

Step 4 of Quick Start says: "If you have changed the frontmatter, regenerate the workflow YAML..." This implies you only need to compile if frontmatter changed. But:

• What if you changed the markdown instructions (the natural language body)? Do you need to compile? (Answer: no, but this isn't stated)
• What exactly counts as "frontmatter"? A beginner editing the "What to include" section might not know this is the markdown body and not frontmatter

Suggestion: Add a clear rule of thumb: "Only re-run gh aw compile when you change the YAML configuration (the section between --- markers). Editing the markdown instructions below does not require recompilation."

---

6. Lock file concept introduced without context

Pages: Home page, Overview, Quick Start

The .lock.yml file is mentioned frequently but a beginner doesn't understand why two files are needed or what to do with each. The Glossary defines it well, but it's not prominently linked from the Quick Start.

Suggestion: Add a brief "Two files, one workflow" explanation box early in the Quick Start or Introduction:

You'll always have two files: my-workflow.md (you edit this) and my-workflow.lock.yml (auto-generated, commit both). Never edit the .lock.yml directly.

---

7. Navigation complexity overwhelming for beginners

The sidebar has 8+ major sections including "Design Patterns" with 15 items (CentralRepoOps, ChatOps, DailyOps, DataOps, DispatchOps, IssueOps, LabelOps, MultiRepoOps...). A newcomer scanning the sidebar sees an enormous amount of terminology before they've even run their first workflow.

Suggestion: Consider a progressive disclosure approach—either collapse advanced sections by default, or add a "🌱 New? Start here" badge/section that guides users through: Introduction → Quick Start → Creating Workflows, before surfacing advanced content.

---

8. Two entry points on home page without clear differentiation

The home page has two primary CTAs:

• "Quick Start with CLI" → installs CLI and adds a pre-built workflow
• "Creating Workflows" → authors a new custom workflow with AI

These serve very different use cases (install a template vs. create from scratch) but the names don't make this obvious. A beginner might click "Creating Workflows" when they just want to get something working first.

Suggestion: Add a short subtitle under each CTA:

• Quick Start with CLI: "Try a pre-built workflow in 10 minutes"
• Creating Workflows: "Build your own custom automation"

---

9. GitHub Actions knowledge assumed in Quick Start

The Quick Start guide mentions "triggers," "permissions," and "GitHub Actions" without a link to the excellent GitHub Actions Primer that lives in Guides. A complete beginner who has never used GitHub Actions may not know what these terms mean.

Suggestion: Add a callout near the top of Quick Start: "New to GitHub Actions? Check out our GitHub Actions Primer for background context."

---

10. PAT permissions are specific but discovery path is indirect

Page: /gh-aw/reference/auth/

The auth page does a good job with the pre-filled PAT link. However, the Quick Start wizard step that says "Set up the required secret" links to the auth reference page, which requires users to scroll to find the right secret for their chosen engine. Users who don't choose Copilot need to find the Claude or Codex section.

Suggestion: The add-wizard command should surface the direct auth page anchor for the chosen engine (e.g., linking directly to #anthropic_api_key when Claude is selected). Or, on the auth page, add a quick-picker at the top.

---

🟢 What Worked Well

1. Estimated time on guides — "⏱️ Estimated time: 10 minutes" is very reassuring and sets correct expectations
2. Video tutorials embedded — Videos for install-and-add-workflow and create-workflow-on-github are excellent
3. Multiple installation methods — Covering CLI extension, standalone installer, and GitHub Actions setup-cli action is comprehensive and practical
4. Most common commands table — The table at the top of the CLI page is the right pattern; beginners immediately see the 6 most-used commands
5. Glossary with definitions — The tip at the top of the Glossary ("Click any glossary link to understand unfamiliar concepts") is helpful
6. Pre-filled PAT creation link — The link that pre-fills token name, description and permissions is a nice UX touch in the auth page
7. Clear prerequisites section in Quick Start — AI account, GitHub repo, GitHub Actions enabled, GitHub CLI, and OS requirements are all listed
8. GitHub Actions Primer guide — Excellent resource for absolute beginners; just needs to be linked more prominently from Quick Start
9. TIP callouts — Good use of callouts throughout the docs for common issues and shortcuts
10. Home page example workflow — Showing a complete minimal workflow right on the home page is very effective

---

Recommendations

Quick Wins (High impact, low effort)

1. Add "What is create.md?" explanation in Creating Workflows guide (~2 sentences)
2. Add gh aw compile usage rule in Quick Start: explain when it's needed vs not needed
3. Link to GitHub Actions Primer from Quick Start intro callout
4. Add subtitles to home page CTAs to clarify the two paths

Medium Term

5. Add at least one complete, runnable workflow per Examples category page (not just links to external repos)
6. Add "Two files" explanation box in Quick Start / Introduction for .md vs .lock.yml
7. Improve first-run failure guidance in Creating Workflows — make it a step, not a TIP; include explicit "what to do next" link

Longer Term

8. Progressive sidebar navigation — Collapse or badge advanced sections; add "Start Here" guidance for new users
9. Consider a dedicated "New User Journey" page that walks through: What is this? → Install → Add first workflow → Customize → Create your own
10. Home page CTA differentiation — Make it visually clearer which path is for "first-time users" vs "creating custom automation"

---

This report was generated by an automated documentation testing workflow simulating a first-time user experience.

AI generated by Documentation Noob Tester · history

• expires on Mar 7, 2026, 8:01 AM UTC

[github-actions[bot]]

🤖 BEEP BOOP — The smoke test agent has landed! 🚀

I've just finished poking around your repository and making sure everything still works. Consider this my official "I was here" stamp. The workflow is humming, the build compiles, and GitHub is still... GitHub.

drops mic, drives off in a self-driving test runner 🎤🚗💨

📰 BREAKING: Report filed by Smoke Copilot · ◷

[github-actions[bot]]

💥 WHOOSH!

ZAP! The smoke test agent swoops in from the digital aether! 🦸

POW! All systems checked. BAM! Build succeeded. WHAM! Playwright confirmed GitHub is alive and well!

🔥 SMOKE TEST AGENT WAS HERE 🔥
Run 22754642684 — Claude engine — all nominal!

...and with a dramatic cape-swirl, the agent vanishes back into the workflow pipeline! 💨

💥 [THE END] — Illustrated by Smoke Claude · ◷

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-07T08:01:38.787Z.

Closed by Workflow