[Due for payment 2026-01-15] [Due for payment 2025-11-11] [Due for payment 2025-11-07] Improvements to the AI reviewer

[mountiny]

This is a tracking issue for the AI reviewer improvements in the App repo.

Parent issue https://github.com/Expensify/Expensify/issues/521398

Issue Owner

Current Issue Owner: @adamgrzybowski

[kacper-mikolajczak]

As discussed, I am starting with improving quality of currently used checks by fixing errors that reviewer is currently running into. I've analysed 86 recent runs of ai reviewer, to get the bigger picture on how to approach fixing them. Here's an actual summary divided into Error Report and Analysis phases.

Error Frequency Report

Error Frequency Report

Summary

• Total errors: 185
• Unique errors: 59

Error Breakdown

Count	Error Message
39	``Error creating inline comment: Validation Failed: {"resource":"PullRequestReviewComment","code":"custom","field":"pull_request_review_thread.line","message":"could not be resolved"} - https://docs.github.com/rest/pulls/comments#create-a-review-comment-for-a-pull-request`
29	Command contains backticks (`) for command substitution
12	This command uses shell operators that require approval for safety
11	accepts at most 1 arg(s), received 2
9	This command requires approval
9	<tool_use_error>File does not exist.</tool_use_error>
8	could not determine current branch: failed to run git: not on any branch
6	fatal: bad revision 'HEAD~1'
3	Command contains output redirection (>) which could write to arbitrary files
3	``<tool_use_error>InputValidationError: Glob failed due to the following issue:`
2	no pull requests found for branch "main"
2	``Unknown JSON field: "changed_files"`
2	File content (30428 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
2	Command contains process substitution <()
2	Claude requested permissions to write to /tmp/review_comment.txt, but you haven't granted it yet.
2	Claude requested permissions to write to /tmp/pr_comment.md, but you haven't granted it yet.
2	Claude requested permissions to use WebFetch, but you haven't granted it yet.
1	``unknown shorthand flag: 'U' in -U0`
1	``unknown flag: --unified`
1	rg: unrecognized file type: tsx
1	``fatal: ambiguous argument 'HEAD~10..HEAD': unknown revision or path not in the working tree.`
1	``Unknown JSON field: "headSha"`
1	``Unknown JSON field: "headBranchName"`
1	``Unknown JSON field: "headBranch"`
1	``This Bash command contains multiple operations. The following parts require approval: gh api repos/Expensify/App/pulls/72236/comments --method POST , --field body='### ❌ PERF-4 (docs)`
1	``This Bash command contains multiple operations. The following parts require approval: echo "## HelpDot Documentation Review`
1	``This Bash command contains multiple operations. The following part requires approval: jq -r '.files[].path'
1	This Bash command contains multiple operations. The following part requires approval: grep -E '^diff --git.*docs/'
1	``This Bash command contains multiple operations. The following part requires approval: grep -E '.(md
1	``This Bash command contains multiple operations. The following part requires approval: grep -E '.(md
1	``This Bash command contains multiple operations. The following part requires approval: grep -E '.(md
1	``This Bash command contains multiple operations. The following part requires approval: grep -E '.(md
1	``This Bash command contains multiple operations. The following part requires approval: grep -E '.(md
1	``This Bash command contains multiple operations. The following part requires approval: grep -E ".(md
1	``This Bash command contains multiple operations. The following part requires approval: grep -E ".(md)$"
1	This Bash command contains multiple operations. The following part requires approval: gh show HEAD:src/libs/actions/App.ts
1	This Bash command contains multiple operations. The following part requires approval: gh --help
1	This Bash command contains multiple operations. The following part requires approval: env
1	``MCP error -32602: MCP error -32602: Invalid arguments for tool create_inline_comment: [`
1	File content (89352 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	File content (89273 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	File content (72378 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	File content (46504 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	File content (46406 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	File content (34717 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	File content (32221 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	File content (31715 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	File content (30397 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	File content (25770 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	File content (25502 tokens) exceeds maximum allowed tokens (25000). Please use offset and limit parameters to read specific portions of the file, or use the GrepTool to search for specific content.
1	Command contains process substitution >()
1	Claude requested permissions to write to /tmp/review_comment.md, but you haven't granted it yet.
1	Claude requested permissions to write to /tmp/pr_review_comment.txt, but you haven't granted it yet.
1	Claude requested permissions to write to /tmp/pr_review.md, but you haven't granted it yet.
1	Claude requested permissions to write to /tmp/perf6_comment.json, but you haven't granted it yet.
1	Claude requested permissions to write to /tmp/perf4_comment.txt, but you haven't granted it yet.
1	Claude requested permissions to use WebSearch, but you haven't granted it yet.
1	``<tool_use_error>InputValidationError: TodoWrite failed due to the following issue:`
1	``<tool_use_error>InputValidationError: Bash failed due to the following issue:`

Analysis of Report

🔥 Critical (Fix These Now)

1. Inline Comments Failing (39 errors - 21%)

Problem: AI tries to comment on lines that don't exist anymore on PRs from forks

Fix:

TBD

2. Command Safety Triggers (62 errors - 33%)

Problem: AI generates bash commands that hit sandbox restrictions

Fix:

• Update AI system prompt with explicit rules:

  BASH COMMAND RULES (CRITICAL):
  - ✅ Use $(command) for substitution
  - ❌ NEVER use `command` (backticks)
  - ✅ Use && to chain commands
  - ❌ Avoid |, >, <, << unless absolutely necessary
  - ✅ Prefer native tools (grep, read_file) over bash
  
  BAD:  files=`ls *.md`
  GOOD: files=$(ls *.md)
  
  BAD:  gh pr view | grep "state"
  GOOD: gh pr view --json state
  

3. Git Context Broken (14 errors - 7.5%)

Problem: Running in detached HEAD or shallow clones

Fix:

• Fix GitHub Actions checkout in .github/workflows/claude-review.yml:

  - uses: actions/checkout@v4
    with:
      fetch-depth: 0  # Full history, not shallow
      ref: ${{ github.head_ref }}  # Check out the actual PR branch
      # For PR from forks:
      repository: ${{ github.event.pull_request.head.repo.full_name }}

• Add git context validation at workflow start:

  # In workflow or AI prompt
  if ! git rev-parse --abbrev-ref HEAD 2>/dev/null; then
    echo "Warning: Detached HEAD state"
    export CURRENT_BRANCH="${{ github.head_ref }}"
  fi
  
  # Ensure we have parent commits
  if ! git rev-parse HEAD~1 2>/dev/null; then
    echo "Warning: Insufficient git history"
    git fetch --depth=10  # Fetch more history
  fi

• Provide fallback context to AI:

  Git Context:
  - Current ref: ${{ github.sha }}
  - Base ref: ${{ github.event.pull_request.base.sha }}
  - Changed files: [use GitHub API instead of git commands]
  

---

⚠️ Medium Priority

4. Tool Parameter Misuse (11 errors - 6%)

Problem: AI passing wrong number of args to tools

Fix:

• Improve tool documentation in tool definitions:

  // Bad docs:
  grep(pattern, path?)
  
  // Good docs with examples:
  /**
   * Search for pattern in files
   * @param pattern - Regex pattern (required)
   * @param options.path - Single file or directory (optional)
   * 
   * Examples:
   *   grep("TODO", {path: "src/file.ts"})      ✓
   *   grep("ERROR")                             ✓
   *   grep("TODO", "src/", "lib/")              ✗ Only one path allowed
   */

• Add parameter validation with helpful errors:

  if (args.length > maxArgs) {
    throw new Error(
      `Tool accepts at most ${maxArgs} arg(s), received ${args.length}.\n` +
      `Example usage: ${exampleUsage}`
    );
  }

• AI prompt additions:

  When you get "accepts at most 1 arg" error:
  - Check if you're passing multiple arguments
  - Most tools take a single object/string, not multiple params
  - Look for array parameters if you need to pass multiple values
  

5. File Too Large (15 errors - 8%)

Problem: AI tries to read 89k token files in one go

Fix:

• Teach AI the chunking pattern in system prompt:

  LARGE FILE HANDLING:
  
  1. Check file size first (via list_dir or file metadata)
  
  2. If file > 25k tokens:
     Option A: Use grep to find specific content
       grep("MyFunction", {path: "large_file.ts"})
     
     Option B: Read in chunks
       read_file("large_file.ts", {offset: 0, limit: 1000})
       // Read imports and first part
       
       read_file("large_file.ts", {offset: 5000, limit: 1000})
       // Read specific section
  
  3. NEVER try to read entire file >25k tokens
  

• Auto-suggest chunking in error message:

  if (fileTokens > 25000) {
    throw new Error(
      `File is ${fileTokens} tokens. Try:\n` +
      `1. grep("search_term", {path: "${path}"})\n` +
      `2. read_file("${path}", {offset: 0, limit: 5000})\n` +
      `3. Search for specific functions/classes instead`
    );
  }

6. Permission Denied (8 errors - 4%)

Problem: Trying to write to /tmp or use WebFetch without permission

Fix:

• Pre-grant permissions in workflow:

  - name: Run Claude Review
    env:
      AUTO_GRANT_PERMISSIONS: "write,network"
      # Or specifically:
      ALLOWED_WRITE_PATHS: "/tmp"
      ALLOW_WEB_FETCH: "true"

• Update AI prompt to avoid temp files:

  FILE HANDLING:
  - ❌ Don't write temp files: echo "$content" > /tmp/file.txt
  - ✅ Use variables: CONTENT="..." then use $CONTENT
  - ✅ Pipe directly: echo "$content" | gh pr comment
  
  WEB FETCH:
  - If you need external docs, request permission first
  - Or provide docs in context instead
  

7. File Not Found (9 errors - 5%)

Problem: AI tries to access files that don't exist

Fix:

• Better error recovery in AI prompt:

  If you get "File does not exist":
  1. Use file_search to find the correct path
  2. Use list_dir to see what files are available
  3. Check if the file is in the PR's changed files
  4. Maybe the file was renamed or deleted in this PR
  

🎯 Quick Wins (Biggest Impact)

Fix these → eliminate 60% of errors:

1. Backticks → $(...) - Add to AI prompt, auto-sanitize commands (15%)
2. Line validation - Add try/catch + fallback to PR comments (21%)
3. Git checkout config - Add fetch-depth: 0 to workflow (7.5%)
4. File size checks - Return token counts in listings, teach chunking (8%)
5. Pre-grant permissions - Add env vars to workflow (4%)
6. Tool param validation - Better error messages with examples (6%)

Next steps

I will continue working on evaluating fix suggestions and their implementation.

[kacper-mikolajczak]

Posted PR for the second most frequent issue #72921. Would it be worth adding sub-tickets for those error fixes?