[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-06-18

[github-actions[bot]]

Summary

• Date: 2026-06-18
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/ (Quick Start)
  • https://github.github.com/gh-aw/setup/cli/ (CLI Commands)
• Overall impression: The home page gives a strong first impression with clear CTAs, but the Quick Start guide has some rough patches that would trip up an absolute beginner — particularly around token setup and monitoring workflow runs.

---

🔴 Critical Issues Found

1. Token setup instructions are beginner-hostile

In Step 2 of the Quick Start, setting up COPILOT_GITHUB_TOKEN involves:

gh secret set COPILOT_GITHUB_TOKEN < /path/to/token.txt

A complete beginner doesn't know what /path/to/token.txt means. How do I make this file? Where do I put it? The more natural form a beginner would understand is:

echo "ghp_YourTokenHere" | gh secret set COPILOT_GITHUB_TOKEN

The distinction between COPILOT_GITHUB_TOKEN (a new fine-grained PAT you must create) and the automatic GITHUB_TOKEN (the built-in one) is buried in a NOTE callout after the main step description. As a beginner, I missed this on first read and thought the setup was done.

📎 [confusing-token-setup.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/5f4d945c1160c725de06d83b18120fa1e5da1a75d7ad6b3ade4cf1eae6c8dfca.png?raw=true

2. Step 3 gives no guidance on monitoring

Step 3 ("Wait for the workflow to complete") says:

An automated workflow run can take 2-3 minutes.

That's it. As a beginner, I have no idea:

• Where do I look? GitHub Actions tab? A link would help.
• How do I check progress? Is there a gh aw status command I should run?
• What does success look like? (A new issue is created — but this is only described after the instruction)
• What if it fails? No error recovery hint is mentioned.

📎 [step3-no-monitoring.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/18c28334e6ac4cecda20cb5ecd976ec23107db143d9764eca4d0def36927b826.png?raw=true

---

🟡 Confusing Areas

3. The add-wizard command format is unexplained upfront

The Quick Start tells you to run:

gh aw add-wizard githubnext/agentics/daily-repo-status

The format <owner>/<repo>/<workflow-name> is explained after the command, not before. A beginner will stare at githubnext/agentics/daily-repo-status wondering what it means. Putting a brief format explanation before the command would reduce confusion.

4. "frontmatter" used without definition

Step 4 says: "If you changed the frontmatter (the configuration block between the --- markers at the top of the file)"

The parenthetical is helpful, but "frontmatter" is a technical term. For a beginner, a consistent inline definition or a glossary link the first time the term appears would help.

5. "Public Preview" warning is easy to miss

The home page has this note:

i Note: GitHub Agentic Workflows is in Public Preview and may change significantly.

It's rendered as a plain blockquote. Given that the software may change significantly, this seems important enough to be a prominent notice/callout rather than an easily-scrolled-past paragraph.

6. "What's next?" sends users to a blog post

The Quick Start's final section, "What's next?", links to "Peli's Agent Factory" — which is a personal blog post. This is an unusual "next step" for technical documentation. Beginner readers expect to be guided toward the next documentation page (like "Creating Workflows" or "How Agentic Workflows Work"), not a blog entry.

7. CLI Commands page: three install commands, unclear when to use each

The page lists:

• gh extension install github/gh-aw (standard)
• Standalone installer script (for auth issues)
• GitHub Actions setup action (for CI use)

A beginner following Quick Start doesn't know which one applies to them. A simple decision-tree or callout ("use this if X, use that if Y") would clarify.

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/29ef24e5e6cfa32e76e74991e3cfc80a4cfd2381c51c6d939d990e76b251f0fd.png?raw=true

📎 [quick-start.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/bff99d65957703af2a831c39abc1da9b25ab9ea76817faf1336fe05027a13d8d.png?raw=true

---

🟢 What Worked Well

• Estimated time (⏱️ 10 minutes) on the Quick Start is a great touch — sets expectations immediately
• The CLI Commands table is excellent: a clean summary of all commands with descriptions is exactly what a beginner needs before diving deep
• Multiple AI engine support is clearly communicated throughout — beginner doesn't feel locked in
• Hero section on home page is clear with two strong CTAs: "Quick Start with CLI" and "Creating Workflows"
• Prerequisite list in Quick Start is well-structured with all needed tools listed
• Video embedded in Quick Start is a nice addition for visual learners

---

Recommendations

Quick wins 🚀

1. Step 3 – Add a monitoring command: Add gh aw status or a link to the GitHub Actions runs page so beginners know how to track progress.
2. Token setup – Improve the secret command: Change < /path/to/token.txt to a echo "..." | gh secret set example for beginner friendliness.
3. Explain add-wizard format before the command, not after.
4. "What's next?" – Link to documentation pages first, then the blog post.

Medium-term improvements 📋

5. Add a "First Run Troubleshooting" mini-section to Step 3 with 2-3 common errors and how to fix them.
6. Define "frontmatter" with a glossary entry and link on first use.
7. Upgrade "Public Preview" notice to a more prominent callout/banner.
8. CLI Commands page: Add example output for the most common commands (gh aw status, gh aw list) so beginners know what to expect.

Longer-term 🗓️

9. Add a "Beginner's Cheat Sheet" — a single page with the top 5 commands, what they do, and when to use them.
10. Interactive tutorial or demo video walking through the full Quick Start end-to-end.

---

Screenshots

📎 [home.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/29ef24e5e6cfa32e76e74991e3cfc80a4cfd2381c51c6d939d990e76b251f0fd.png?raw=true

📎 [quick-start.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/bff99d65957703af2a831c39abc1da9b25ab9ea76817faf1336fe05027a13d8d.png?raw=true

📎 [confusing-token-setup.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/5f4d945c1160c725de06d83b18120fa1e5da1a75d7ad6b3ade4cf1eae6c8dfca.png?raw=true

📎 [step3-no-monitoring.png] — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/18c28334e6ac4cecda20cb5ecd976ec23107db143d9764eca4d0def36927b826.png?raw=true

---

References: §27738455642

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · ◷

• expires on Jun 18, 2026, 9:30 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-06-19T05:30:37.183Z.

Closed by Workflow