[syntax-error-quality] Error Message Quality: YAML Syntax Errors Need Improvement

[github-actions]

📊 Error Message Quality Analysis

Analysis Date: 2025-07-23
Test Cases: 3
Average Score: 66.7/100
Status: ⚠️ Needs Improvement

---

Executive Summary

Static analysis of the compiler error pipeline was used to reconstruct the error messages produced for three representative error types. YAML syntax errors score critically low (44/100), falling below the per-test threshold of 55, due to raw parser output being surfaced without plain-English translation or fix guidance. Engine validation errors (74/100) and schema constraint errors (82/100) are much stronger but have minor consistency issues. The average score of 66.7/100 falls below the 70/100 threshold, and one test case is in the critical range, triggering this issue.

Key Findings:

• ✅ Strengths: Schema errors (Test 3) have excellent source context rendering; engine errors have "Did you mean" suggestions and documentation links; schema constraint messages are translated to plain English
• ⚠️ Weaknesses: YAML parse errors surface raw goccy jargon without translation; engine errors are attributed to file:1:1 instead of the actual engine: field location; engine errors leak the internal "failed to generate YAML" prefix
• ❌ Critical Issue: YAML syntax errors provide no actionable fix guidance — no "Correct usage:" example, no suggestion, no translated message

---

Test Case Results

Test Case 1: Invalid YAML Syntax — Score: 44/100 ❌ Critical

Test Configuration

Workflow: dependabot-burner.md (22 lines, simple)
Error Type: Category A — Frontmatter YAML syntax error
Error Introduced: Line 2: on weekly (missing colon — should be on: weekly)

Reconstructed Compiler Output

The compiler calls yaml.Unmarshal() in pkg/parser/frontmatter_content.go, which fails and is passed to FormatYAMLError(). The goccy formatted output is then wrapped as-is via formatCompilerError(markdownPath, "error", err.Error(), err):

test-1-err.md:1:1: error: failed to parse frontmatter:
[2:9] string was used where mapping is expected
>  2 | on weekly
         ^
```

*(The goccy source pointer is present, but the outer position is stuck at 1:1 and the message is raw parser jargon.)*

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 12/25 | Poor |
| Actionability | 8/25 | Poor |
| Context | 13/20 | Acceptable |
| Examples | 2/15 | Critical |
| Consistency | 9/15 | Acceptable |
| **Total** | **44/100** | **Critical ❌** |

#### Strengths
- ✅ `goccy/go-yaml` renders the problematic source line with a `^` pointer
- ✅ The source line `on weekly` is visible so a developer can spot the issue

#### Weaknesses
- ❌ "string was used where mapping is expected" is raw YAML parser jargon — opaque to developers unfamiliar with the YAML spec
- ❌ No suggestion of the correct syntax (`on: weekly`)
- ❌ No "Did you mean:" hint or "Correct usage:" example
- ❌ Outer file position is `1:1` — imprecise since `goccy` already knows the exact line
- ❌ No documentation link
- ❌ "failed to parse frontmatter:" prefix is accurate but adds no actionable context

#### Improvement Suggestions

1. **Translate common goccy YAML error messages to plain English** (highest impact):
   ```
   Current:  "string was used where mapping is expected"
   Better:   "Missing ':' after key 'on' — expected a key: value pair"
   ```

2. **Add "Correct usage:" example for common YAML mistakes**:
   ```
   Correct usage:
     on: weekly
   ```

3. **Propagate goccy's line/column into the outer `formatCompilerError` call** so `file:line:col` is accurate:
   ```go
   // In pkg/parser/frontmatter_content.go, extract line from goccy error
   // and call formatCompilerErrorWithPosition(markdownPath, actualLine, actualCol, ...)
   ```

4. **Shared "Did you mean" pattern** already used in engine validation — apply the same pattern for field names

</details>

<details>
<summary><b>Test Case 2: Invalid Engine Name</b> — Score: 74/100 ✅ Good</summary>

#### Test Configuration

**Workflow**: `issue-triage-agent.md` (88 lines, medium)
**Error Type**: Category B — Configuration error
**Error Introduced**: `engine: copiilot` (typo — should be `engine: copilot`)

#### Reconstructed Compiler Output

Engine validation in `pkg/workflow/engine_validation.go` produces an excellent message, but it's wrapped by `generateAndValidateYAML()` with a noisy prefix:

```
test-2-err.md:1:1: error: failed to generate YAML: invalid engine: copiilot.
Valid engines are: copilot, claude, codex, custom.

Did you mean: copilot?

Example:
  engine: copilot

See: https://github.com/github/gh-aw/blob/main/README.md#engines
```

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 18/25 | Good |
| Actionability | 23/25 | Excellent |
| Context | 9/20 | Poor |
| Examples | 14/15 | Excellent |
| Consistency | 10/15 | Acceptable |
| **Total** | **74/100** | **Good ✅** |

#### Strengths
- ✅ "Did you mean: copilot?" is highly actionable
- ✅ Lists all valid engines
- ✅ Shows correct usage example
- ✅ Provides documentation link

#### Weaknesses
- ⚠️ Position is `1:1` — should point to the `engine:` field line
- ⚠️ "failed to generate YAML: " prefix leaks internal implementation detail; confusing to users
- ⚠️ No source context lines rendered (the `Context` field is empty for engine errors)

#### Improvement Suggestions

1. **Remove "failed to generate YAML:" prefix** from engine validation errors — it's internal plumbing:
   ```go
   // In generateAndValidateYAML(), detect engine errors and return them with
   // formatCompilerErrorWithPosition(markdownPath, engineLine, 8, "error", err.Error(), nil)
   // rather than wrapping with "failed to generate YAML: ..."
   ```

2. **Locate the `engine:` field in frontmatter** and use its line number for precise `file:line:col` attribution

</details>

<details>
<summary><b>Test Case 3: Negative Timeout</b> — Score: 82/100 ✅ Good</summary>

#### Test Configuration

**Workflow**: `research.md` (70 lines, medium-complex)
**Error Type**: Category C — Semantic/schema constraint error
**Error Introduced**: `timeout-minutes: -10` (negative value — minimum is 1)

#### Reconstructed Compiler Output

Schema validation in `pkg/parser/schema_compiler.go` locates the field precisely and translates the constraint:

```
test-3-err.md:3:1: error: at '/timeout-minutes' (line 3, column 1): must be at least 1 (got -10). Example: timeout-minutes: 10
  |
1 | ---
2 | description: Performs web research on any topic...
3 | timeout-minutes: -10
  | ^~~~~~~~~~~~~~~~~~~~
4 | strict: true
5 | on:
  |

Evaluation Scores

Dimension	Score	Rating
Clarity	22/25	Excellent
Actionability	18/25	Good
Context	19/20	Excellent
Examples	10/15	Good
Consistency	13/15	Excellent
Total	82/100	Good ✅

Strengths

• ✅ Rust-like source rendering with ^~~~ visual pointer
• ✅ Precise file:line:col location
• ✅ Constraint translated to plain English: "must be at least 1 (got -10)"
• ✅ JSON path shown: at '/timeout-minutes'
• ✅ Example value provided

Weaknesses

• ⚠️ No documentation link
• ⚠️ No explanation of why the minimum exists (GitHub Actions limitation)
• ⚠️ Example only shows one valid value — could show a range or explain valid range

---

Overall Statistics

Metric	Value
Tests Run	3
Average Score	66.7/100
Excellent (85+)	0
Good (70-84)	2
Acceptable (55-69)	0
Critical (<55)	1

Quality Assessment: ❌ Needs Improvement — Average score 66.7 is below the threshold of 70, and Test Case 1 (44/100) is below the per-test critical threshold of 55.

---

Priority Improvement Recommendations

🔴 High Priority — YAML Syntax Error Translation

Problem: Raw goccy/go-yaml error messages are exposed to users without translation or fix guidance.

Solution: Add a translation map in pkg/parser/yaml_error.go for the most common goccy messages:

var yamlErrorTranslations = map[string]string{
    "string was used where mapping is expected":     "Missing ':' after key — expected 'key: value'",
    "mapping values are not allowed in this context": "Unexpected ':' — check indentation or key syntax",
    "did not find expected key":                     "Incorrect indentation or missing key",
    "found a tab character that violates indentation": "Use spaces for indentation, not tabs",
    "block sequence entries are not allowed":        "Unexpected list item '-' — check indentation",
}

Then after translating, append a "Correct usage:" example derived from the field name if detectable.

🔴 High Priority — Propagate Precise Location for YAML Errors

Problem: formatCompilerError is called with line:1, col:1 even though goccy already returns the exact error position.

Solution: In pkg/parser/frontmatter_content.go, extract line/column from the goccy error and call formatCompilerErrorWithPosition:

// Extract goccy error position
if yamlErr, ok := err.(*yaml.SyntaxError); ok {
    line, col := yamlErr.GetToken().Position.Line, yamlErr.GetToken().Position.Column
    // Adjust for frontmatter offset, then call formatCompilerErrorWithPosition
}

🟡 Medium Priority — Remove Internal Prefix from Engine Errors

Problem: Engine validation errors are wrapped with "failed to generate YAML: " in generateAndValidateYAML(), leaking an internal step name.

Solution: In pkg/workflow/compiler.go:351, detect when the underlying error is from engine validation and return it directly (or with a cleaner prefix):

// Before: return "", formatCompilerError(markdownPath, "error",
//   fmt.Sprintf("failed to generate YAML: %v", err), err)
// After: return "", err  // engine validation already produces a well-formatted error

🟡 Medium Priority — Precise Location for Engine Errors

Problem: Engine field errors use file:1:1 instead of the actual engine: line number.

Solution: After frontmatter is parsed, locate the engine key in the YAML source and pass the correct line to validateEngine's error formatter. The LocateJSONPathInYAMLWithAdditionalProperties utility already exists for this purpose.

🟢 Low Priority — Documentation Links for Schema Errors

Add documentation links to schema constraint errors (similar to engine validation), pointing to the workflow reference documentation.

---

Implementation Guide

Files to change (in priority order):

File	Change	Priority
pkg/parser/yaml_error.go	Add yamlErrorTranslations map; call translation in FormatYAMLError()	🔴 High
pkg/parser/frontmatter_content.go	Extract goccy error position; call formatCompilerErrorWithPosition	🔴 High
pkg/workflow/compiler.go:351	Return engine errors directly instead of wrapping with "failed to generate YAML"	🟡 Medium
pkg/workflow/engine_validation.go	Accept frontmatterContent string; locate engine: field for precise position	🟡 Medium
pkg/parser/schema_compiler.go	Add documentation links to constraint error messages	🟢 Low

---

References:

• §22409568160

Generated by Daily Syntax Error Quality Check

• expires on Feb 28, 2026, 6:22 PM UTC