[prompt-analysis] 🤖 Copilot PR Prompt Analysis - 2026-07-22

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (rolling window, 1000 PRs sampled)
Total PRs: 1000 | Merged: 760 (76.2%) | Closed: 237 (23.5%) | Open: 3 (0.3%)

---

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Bug Fix	769	592	77%
Testing	137	105	77%
Documentation	52	39	75%
Feature Addition	20	14	70%
Refactoring	16	8	50%
Other/Uncategorized	6	2	33%

---

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 194 words
• Most common keywords: workflow, actions, engine, model, copilot
• Prompts are direct and imperative — they describe a concrete problem or change
• No [WIP] prefix: 79% success rate (746/947)

Example successful prompts:

1. PR Document bounded optimization for repetitive work #47335: "Large repetitive workloads can exhaust agent budgets before making durable progress. Optimization guidance should favor..." → Merged
2. PR Pin docker/build-push-action to commit SHA (CodeQL alert #625) #47334: "`docker/build-push-action` was referenced by a mutable tag (`v7.2.0`), leaving the workflow vulnerable to supply-chain attacks..." → Merged
3. PR Fix literal \n in CLI-mode safeoutputs string args not becoming newlines #47330: "When safeoutputs is called via shell (i.e. `cli-proxy: true`), string arguments like `--body "line1\nline2"` pass a literal..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 196 words (slightly longer, suggesting over-scoped work)
• PRs with [WIP] prefix: only 26% success rate (14/53)
• PRs with task checklists (- [ ]): only 12% success rate (5/42) — indicating the agent acknowledged scope uncertainty
• PRs with vague planning language ("I will get started on it and keep this PR's description up to date") consistently fail

Example unsuccessful prompts:

1. PR [WIP] Normalize report formatting for non-compliant workflows #47329 [WIP]: "Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date..." → Closed
2. PR [WIP] Fix workflow failure in fuzz job on main #47298 [WIP]: "- [x] Reproduce and inspect CI failure context for run #29916480693 - [ ] Identify root cause in fuzz-related code..." → Closed
3. PR Add compiler warning for unbounded list_code_scanning_alerts prompt usage #47265: "Workflows that call `list_code_scanning_alerts` without `state: open` and `severity: critical,high` produce oversized MCP..." → Closed

View full keyword frequency data

Top keywords in merged PR bodies:
github (3280), workflow (1804), sous (1430), chef (1430), actions (1084), runs (994), copilot (895), engine (684), model (674)

Top keywords in closed PR bodies:
github (981), workflow (583), copilot (312), workflows (263), actions (255), runs (240), model (234), sous (231), chef (231)

The vocabulary is largely the same — domain-specific terms dominate in both groups — confirming that topic alone doesn't predict success. The structural and framing differences matter more.

---

Key Insights

• [WIP] prefix is a strong failure signal: PRs with [WIP] in the title have only a 26% merge rate vs 79% for others. These represent exploratory or uncertain work that agents couldn't complete confidently.
• Task checklists correlate with failure: PRs containing - [ ] style checklists have only a 12% success rate — likely because they signal the agent is mid-flight or tracking uncertain scope rather than delivering a complete change.
• Refactoring prompts underperform: At 50% success, refactoring tasks struggle compared to targeted bug fixes (77%) or test additions (77%). Refactors tend to be broader and harder to scope cleanly.
• Prompt length is not a differentiator: Merged (194 words) and closed (196 words) prompts are nearly identical in length — verbosity alone neither helps nor hurts.

---

Recommendations

1. DO: Write prompts that describe a specific, bounded problem — include the failing behavior, the expected behavior, and ideally the affected file or component.
2. DO: Prefer bug fix and testing prompts — they have the highest success rates (77%) and are well-scoped by nature.
3. DO: For documentation and feature work, follow the same pattern: one clear deliverable per PR.
4. AVOID: Starting prompts with [WIP] or using task checklists — these signal the agent is uncertain about scope and the PR is unlikely to complete successfully.
5. AVOID: Broad refactoring prompts without a clear, measurable outcome. Refactoring has a 50% success rate — break it into smaller, targeted changes.

---

Historical Trends (Last 7 Days)

Date	PRs	Merged	Success Rate
2026-07-22	1000	760	76.2%
2026-07-21	1000	764	77%
2026-07-20	1000	755	76%
2026-07-19	1000	762	77%
2026-07-18	1000	757	77%
2026-07-17	1000	756	77%
2026-07-16	1000	768	77%

Trend: Success rate has been stable at ~77% over the past week. Today's slight dip to 76.2% is within normal variance and not a concern. The rolling 30-day dataset is mature and patterns are consistent.

---

References:

• §29957215497

Generated by Copilot PR Prompt Analysis (Run: 29957215497)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · sonnet46 26 AIC · ⌖ 10.5 AIC · ⊞ 7.6K · ◷

• expires on Jul 23, 2026, 1:06 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-07-23T21:06:47.461Z.

Closed by Workflow