UX Analysis Report – 2026-03-18: Documentation Accuracy & Onboarding Clarity

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (engines.md, getting-started-mcp.md, common-issues.md)
• 1 CLI command (gh aw logs)
• 1 workflow message configuration (ci-doctor.md)
• 1 validation file (engine_validation.go)

Overall Quality: Predominantly professional with well-structured examples, actionable error messages, and thorough CLI documentation.

Key Finding: A misleading inline comment in the primary engines reference doc (engines.md) conflates Copilot defaults with Claude model names, creating potential confusion for new enterprise users configuring their first workflow.

---

Quality Highlights ✅

1. Engine Validation Error Messages — pkg/workflow/engine_validation.go

• What works well: Error messages include the problem, valid alternatives, a corrected example snippet, and a docs URL — all in one message. This is exactly the pattern that reduces friction in developer tools.
• Reference (line 72–78): "inline engine definition references unknown runtime.id: %s. Known runtime IDs are: %s.\n\nDid you mean: %s?\n\nExample:…\n\nSee: %s" — exemplary enterprise error messaging.

2. CLI logs Help Text — pkg/cli/logs_command.go

• What works well: Organized into logical groups ("Basic usage", "Date filtering", "Content filtering", "Output options"), each with realistic, copy-paste examples. The WorkflowIDExplanation insertion is a reusable pattern that prevents duplication.

3. Getting Started with MCP Guide — docs/src/content/docs/guides/getting-started-mcp.md

• What works well: Clear 5-minute quick-start with progressive disclosure — from a minimal example to advanced custom servers. The Troubleshooting section covers the most common failure modes concisely, avoiding wall-of-text syndrome.

---

Improvement Opportunities 💡

High Priority

Opportunity 1: Misleading Default Model Comment in engines.md

• File: docs/src/content/docs/reference/engines.md
• Current State (line 31):

  engine:
    id: copilot
    model: gpt-5                          # defaults to claude-sonnet-4

• Issue: The comment # defaults to claude-sonnet-4 is embedded in an example block for the Copilot engine, but claude-sonnet-4 is an Anthropic/Claude model. This creates a direct contradiction — users configuring Copilot will see a Claude model name as the stated default, leading to invalid configurations or loss of trust in the documentation.
• User Impact: Enterprise users copying this example to set up their first workflow will be confused by a Claude model name appearing as the default for Copilot. They may attempt to use this value for non-Claude engines, get a validation error, and lose confidence in the reference docs.
• Suggested Change: Replace the inaccurate comment with the correct Copilot default model.
• Design Principle: Trust and Reliability — documentation must be accurate; a wrong default in a reference example undermines confidence in the entire page.

Medium Priority

Opportunity 2: Sparse "Getting Help" Section in common-issues.md

• File: docs/src/content/docs/troubleshooting/common-issues.md
• Current State (lines 607–609):

  ## Getting Help
  
  Review [reference docs]..., search [existing issues]..., or create an issue. See [Error Reference] and [Frontmatter Reference].
  

• Issue: This is the final section of the primary troubleshooting reference. Users who reach it have already exhausted all prior options. A two-sentence paragraph with four links provides no triage hierarchy — users cannot tell whether to search docs, open an issue, or ask for real-time help. The section also lacks the gh aw audit / agent-assisted debugging path that is highlighted elsewhere.
• User Impact: Enterprise users who are stuck see a minimal dead-end instead of a clear escalation ladder, increasing resolution time and support volume.
• Suggested Change: Expand to a short tiered list with clear decision criteria (self-service → community → file issue).
• Design Principle: Efficiency and Productivity — the help section should reduce cognitive load by telling users which resource to try first.

---

Files Reviewed

Documentation

• docs/src/content/docs/reference/engines.md — ⚠️ Needs Minor Work (inaccurate default model comment)
• docs/src/content/docs/guides/getting-started-mcp.md — ✅ Professional
• docs/src/content/docs/troubleshooting/common-issues.md — ⚠️ Needs Minor Work (sparse "Getting Help" section)

CLI Commands

• gh aw logs (pkg/cli/logs_command.go) — ✅ Professional

Workflow Messages

• .github/workflows/ci-doctor.md — ✅ Professional (playful medical theme is consistent and intentional)

Validation Code

• pkg/workflow/engine_validation.go — ✅ Professional

Metrics

• Files Analyzed: 6
• Quality Distribution:
  • ✅ Professional: 4
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

---

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

---

Task 1: Fix Misleading Default Model Comment — docs/src/content/docs/reference/engines.md

File to Modify: docs/src/content/docs/reference/engines.md

Current Experience

Line 31 of the extended configuration example block reads:

engine:
  id: copilot
  model: gpt-5                          # defaults to claude-sonnet-4

Quality Issue

Design Principle: Trust and Reliability

The comment # defaults to claude-sonnet-4 is factually wrong in the context of the Copilot engine block. claude-sonnet-4 is a Claude/Anthropic model. A user who has never used gh-aw before will read this reference first, copy the example, and immediately get an incorrect mental model of which model the Copilot engine uses. If they then file a bug or ask for help with model selection, this comment is the likely root cause.

Proposed Improvement

Replace the inaccurate comment with the correct Copilot default (verifiable from the AGENTS.md which states defaults to claude-sonnet-4 for the engine — but that default only applies to the overall engine selection, not to the model field within a Copilot block).

Before (line 31):

model: gpt-5                          # defaults to claude-sonnet-4

After:

model: gpt-5                          # defaults to the engine's recommended model

If there is a known correct Copilot default (e.g., gpt-5), the comment should name it specifically. If the default varies by account tier, the comment should say so rather than citing a Claude model name.

Why This Matters

• User Impact: Prevents "where did this Claude model name come from?" confusion for Copilot users
• Quality Factor: Documentation accuracy — the rejig docs #1 driver of user trust
• Frequency: This is the primary engines.md reference; every user configuring a non-default engine reads this block

Success Criteria

• Changes made to docs/src/content/docs/reference/engines.md only
• The model field comment no longer references a Claude model name in a Copilot example
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/reference/engines.md
• No changes to other files required
• Can be completed independently

---

Task 2: Improve the "Getting Help" Section — docs/src/content/docs/troubleshooting/common-issues.md

File to Modify: docs/src/content/docs/troubleshooting/common-issues.md

Current Experience

Lines 607–609 — the final section of the primary troubleshooting doc:

## Getting Help

Review [reference docs]..., search [existing issues]..., or create an issue. See [Error Reference] and [Frontmatter Reference].

Quality Issue

Design Principle: Efficiency and Productivity

Users who reach this section have worked through the entire troubleshooting guide without finding a solution. They need a clear escalation path, not an undifferentiated list of four links. The section also omits the agent-assisted debugging flow documented elsewhere in the same file (lines 532–549).

Proposed Improvement

Replace the single-sentence paragraph with a tiered help structure that maps user situations to specific resources.

Before:

## Getting Help

Review [reference docs](/gh-aw/reference/workflow-structure/), search [existing issues](https://github.com/github/gh-aw/issues), or create an issue. See [Error Reference](/gh-aw/troubleshooting/errors/) and [Frontmatter Reference](/gh-aw/reference/frontmatter/).

After:

## Getting Help

If the sections above haven't resolved your issue, try these resources in order:

1. **Search the reference docs** — the [Frontmatter Reference](/gh-aw/reference/frontmatter/) and [Error Reference](/gh-aw/troubleshooting/errors/) cover configuration options and known error codes in detail.

2. **Ask an agent** — paste your run URL into Copilot Chat for automated diagnosis:
   ```text
   /agent agentic-workflows debug https://github.com/OWNER/REPO/actions/runs/RUN_ID

3. Search existing issues — your issue may already be tracked or resolved at github.com/github/gh-aw/issues.

4. Open a new issue — include the output of gh aw audit (run-id) and your workflow frontmatter (redact any secrets).

**Why This Matters**
- **User Impact**: Transforms a dead-end into a clear escalation ladder, reducing time-to-resolution
- **Quality Factor**: Efficiency — users know exactly what to try next
- **Frequency**: Every user who exhausts the troubleshooting guide lands here

**Success Criteria**
- [ ] Changes made to `docs/src/content/docs/troubleshooting/common-issues.md` only
- [ ] Section provides a tiered escalation path (self-service → agent → community → file issue)
- [ ] Agent-assisted debugging path is included and consistent with lines 532–549
- [ ] Quality rating improves from ⚠️ to ✅

**Scope Constraint**
- **Single file only**: `docs/src/content/docs/troubleshooting/common-issues.md`
- No changes to other files required
- Can be completed independently

---

**References:** [§23242671568](https://github.com/github/gh-aw/actions/runs/23242671568)


<!-- gh-aw-tracker-id: delight-daily -->


> AI generated by [Delight](https://github.com/github/gh-aw/actions/runs/23242671568) · [history](https://github.com/search?q=repo%3Agithub%2Fgh-aw+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Fdelight%22&type=discussions)
> - [x] expires <!-- gh-aw-expires: 2026-03-25T11:42:04.293Z --> on Mar 25, 2026, 11:42 AM UTC

<!-- gh-aw-workflow-id: delight -->
<!-- gh-aw-workflow-call-id: github/gh-aw/delight -->

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #21773.