[AI Reviewer] Fix file content exceeds maximum allowed tokens

[kacper-mikolajczak]

Problem

The AI reviewer is trying to ingest big files which results in tool usage error:

File content (89479 tokens) exceeds maximum allowed tokens (25000).

Initial error report findings

• Total occurrences: 13
• Affected files: big ones like src/CONST/index.ts (89k tokens), src/libs/actions/Policy/Policy.ts (70k tokens)
• Root cause: Agents were trying to read entire files, exceeding the 25k token read limit

[kacper-mikolajczak]

Update for Fix file content exceeds maximum allowed tokens after merging #72950

Based on yesterday error report, there were two instances of this error after fix was merged, so it did not meet the expectations.

However, even it did not eliminate the error, it looks like it's not critical.

Current behaviour is as follows:

1. Agent tries to ingest entire big file and fails
2. Agent proceeds to use Grep alike methods to chunk the file down
3. Agent continues to work on chunked contents

Proofs:

• https://github.com/Expensify/App/actions/runs/19022590262
• https://github.com/Expensify/App/actions/runs/19039893909

Conclusion:

As agent is able to crawl back from initial error, I would not mark this issue as critical. As a next step we might look into a way to avoid failures entirely or if nothing innovative comes to mind, we could revert the PR because current behaviour seems to be a default one picked up by Claude (to be determined).

[kacper-mikolajczak]

Parent issue: #72802

CC @AndrewGable @mountiny

[kacper-mikolajczak]

As @adamgrzybowski is going to be OOO for near future, @mateuuszzzzz is going to replace him. Mateusz is going to start with working on this ticket, @mountiny could you please re-assign this issue? Thanks <3

[mateuuszzzzz]

Hey, the fix works pretty well overall. I’d just tweak the threshold we pass to the model for deciding whether to read the file directly or use grep. Using 5,000 lines as the cutoff feels pretty arbitrary, and being under the maximum token limit isn’t necessarily guaranteed for files with fewer than 5,000 lines. I presume that's why model can still experience File content (X tokens) exceeds maximum allowed tokens (25000). As a possible improvement, we could base this threshold on the estimated number of tokens in the file rather than the number of lines. cc: @AndrewGable @mountiny

[mountiny]

Sounds good to me

[kacper-mikolajczak]

Hi! What's the status for this issue?

[mateuuszzzzz]

Hi, I’m not actively focusing on this at the moment. However, I have a more general concern that such errors (especially cascading ones) may lead to a decline in the reviewer’s accuracy. I’m currently investigating this and evaluating how the model performs in these scenarios, including its ability to recover. This should help determine whether it’s worth investing time in fixing such issues.

[AndrewGable]

I think this is fixed, let's reopen if it happens again.