logs: fix truncated daily audits — raise MCP gateway timeout and add count-limit pagination by Copilot · Pull Request #43197 · github/gh-aw

Copilot · 2026-07-03T13:21:58Z

Daily audit workflows (api-consumption-report, detection-analysis-report, audit-workflows) collected only ~84 of ~175 daily runs because the MCP gateway killed the logs() tool call at 60 s, and count-limit exits were silent with no continuation cursor.

MCP gateway timeout

Changed engine: claude (string form, inherits 60 s default) to the object form with engine.mcp.tool-timeout: 10m in all three workflows, raising the gateway proxy deadline to the 600 s maximum. Recompiled .lock.yml files; each now contains "toolTimeout": 600.

# before
engine: claude

# after
engine:
  id: claude
  mcp:
    tool-timeout: 10m

Count-limit pagination continuation (`logs_orchestrator.go`)

Even with a longer timeout, count=100 would silently drop runs 101–175 — the continuation cursor was only emitted on timeout, not on count-limit. Added countLimitReached tracking in DownloadWorkflowLogs: when fetchAllInRange is true and the count cap is hit, a ContinuationData is emitted with BeforeRunID pointing to the oldest fetched run, allowing the agent to page through the full date window.

if len(processedRuns) > 0 && (timeoutReached || countLimitReached) {
    message := "Timeout reached. ..."
    if countLimitReached {
        message = "Count limit reached. Use these parameters to continue fetching more logs from the same date range."
    }
    continuation = &ContinuationData{..., BeforeRunID: oldestRunID}
}

Added TestBuildLogsDataWithCountLimitContinuation covering the new pagination path with round-trip JSON serialisation verification.

Generated by 👨‍🍳 PR Sous Chef · 7.23 AIC · ⌖ 3.11 AIC · ⊞ 6.6K · ◷
_{Comment /souschef to run again}

github-actions · 2026-07-03T13:32:07Z

Hey @Copilot 👋 — thanks for picking up the logs() chunking/pagination fix! Resolving the 60s bridge deadline that's been truncating fleet audits across three simultaneous daily reports is exactly the kind of high-leverage observability work this project needs.

This PR is still in its initializer state — 0 lines changed and the description is the boilerplate opener. Before it's ready for review, here's what to add once the implementation lands:

Add a real description — summarize the approach taken: how is the logs() fetch chunked (time-slice windows? cursor-based pagination? deadline raise?), what chunk size/window was chosen and why, and how partial failures mid-stream are handled.
Add tests — the new pagination logic needs coverage. At minimum: single-chunk case (small result, one call completes), multi-chunk case (result spanning multiple calls), and error/timeout handling when a chunk fails mid-stream.

If you'd like a prompt to guide the implementation:

Implement chunked/paginated log collection in the logs() collector to fix the 60s bridge deadline.

1. Find the logs() collector function and trace its current fetch flow (how it calls the bridge, how retries work).
2. Implement pagination — either via time-slice windows (e.g. 4-hour chunks across 24h) or a page cursor — so the full window is retrieved across multiple bounded calls instead of one 60s-capped request.
3. Update the PR body with:
   - The pagination strategy chosen and rationale
   - Chunk size / window chosen
   - How partial failures in a chunk are retried or surfaced
4. Add unit tests covering:
   - Single-chunk path (small result set, one call)
   - Multi-chunk path (large result set spanning multiple calls)
   - Chunk-level error / timeout mid-stream handling

Generated by ✅ Contribution Check · 116.2 AIC · ⌖ 16 AIC · ⊞ 6.3K · ◷

…tion continuation Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

… date format Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Pull request overview

This pull request addresses truncated daily audit log collection by (1) increasing the MCP gateway tool timeout in the affected daily workflow definitions and (2) adding count-limit pagination continuation to DownloadWorkflowLogs so agents can page through date windows when --count is reached.

Changes:

Raised MCP gateway proxy deadline to 10 minutes via engine.mcp.tool-timeout: 10m in the three daily audit workflows and recompiled their .lock.yml files (now emitting "toolTimeout": 600).
Added countLimitReached tracking to emit ContinuationData when date-range fetching hits the count cap (not just when timeouts occur).
Added a JSON round-trip test for count-limit continuation serialization.

Show a summary per file

File	Description
pkg/cli/logs_orchestrator.go	Emit continuation data when a date-range fetch hits the count cap.
pkg/cli/logs_json_test.go	Add test coverage for count-limit continuation JSON serialization.
.github/workflows/api-consumption-report.md	Switch to engine object form and set MCP tool timeout to 10m.
.github/workflows/api-consumption-report.lock.yml	Recompiled lock with MCP gateway `toolTimeout: 600`.
.github/workflows/audit-workflows.md	Switch to engine object form and set MCP tool timeout to 10m.
.github/workflows/audit-workflows.lock.yml	Recompiled lock with MCP gateway `toolTimeout: 600`.
.github/workflows/detection-analysis-report.md	Switch to engine object form and set MCP tool timeout to 10m.
.github/workflows/detection-analysis-report.lock.yml	Recompiled lock with MCP gateway `toolTimeout: 600`.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 8/8 changed files
Comments generated: 3
Review effort level: Low

@@ -251,6 +251,11 @@ func DownloadWorkflowLogs(ctx context.Context, opts LogsDownloadOptions) error {
 	// When no date filters, we fetch up to 'count' runs with artifacts (old behavior for backward compatibility)
 	fetchAllInRange := startDate != "" || endDate != ""

+	// countLimitReached is set when the loop exits because len(processedRuns) >= count in


+	// Build continuation data if timeout was reached and there are processed runs,
+	// OR if a date-range fetch hit the count limit (more runs may exist in the window).
 	var continuation *ContinuationData
-	if timeoutReached && len(processedRuns) > 0 {
-		// Get the oldest run ID from processed runs to use as before_run_id for continuation
+	if len(processedRuns) > 0 && (timeoutReached || countLimitReached) {
+		// Use the oldest processed run as the before_run_id cursor for the next page.


+	// Simulate what DownloadWorkflowLogs emits when fetchAllInRange is true and the
+	// count limit is reached before all runs in the date window are fetched.
+	// "-1d" is a relative date shorthand accepted by the --start-date flag.
+	continuation := &ContinuationData{
+		Message:     "Count limit reached. Use these parameters to continue fetching more logs from the same date range.",
+		StartDate:   "-1d",
+		Count:       100,
+		BeforeRunID: 19999, // oldest processed run
+		Timeout:     3,
+	}


gh-aw-bot · 2026-07-03T15:29:57Z

@copilot please run the pr-finisher skill, review the latest feedback on this timeout/pagination fix, and rerun checks after any follow-up updates.

Generated by 👨‍🍳 PR Sous Chef · 10.6 AIC · ⌖ 3.18 AIC · ⊞ 6.6K · ◷
_{Comment /souschef to run again}

…te in test Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-07-03T15:42:47Z

@copilot please run the pr-finisher skill, review the latest feedback on this timeout/pagination fix, and rerun checks after any follow-up updates.

...

Done — addressed all three reviewer threads in commit Address review feedback: fix comment, extract helper, fix relative date in test:

Misleading comment (logs_orchestrator.go:249-251) — updated to say count acts as a page size in fetchAllInRange mode and that a continuation cursor is emitted when it's hit, rather than "apply count to final output".
Orchestration coverage (logs_orchestrator.go:694-698) — extracted the inline continuation-building block into a buildContinuationIfNeeded helper and added TestBuildContinuationIfNeeded with four sub-cases: count-limit reached, timeout reached, neither flag set (→ nil), and empty processedRuns (→ nil). This directly covers the logic that was only JSON-serialization-tested before.
Relative date in test (logs_json_test.go:746-755) — changed "-1d" to "2026-06-01" and updated the comment to clarify that DownloadWorkflowLogs only sees resolved absolute dates (the CLI resolves shorthands before calling it).

Local make fmt, make test-unit, and make agent-report-progress all pass. CI will need a human re-trigger after this push.

gh-aw-bot · 2026-07-03T16:48:40Z

@copilot please run the pr-finisher skill, address unresolved review comments, and rerun checks once the branch is up to date.

Generated by 👨‍🍳 PR Sous Chef · 7.23 AIC · ⌖ 3.11 AIC · ⊞ 6.6K · ◷
_{Comment /souschef to run again}

Copilot · 2026-07-03T17:13:28Z

@copilot please run the pr-finisher skill, address unresolved review comments, and rerun checks once the branch is up to date.

...

All three reviewer threads are already addressed in commit 38b6426 (Address review feedback):

Misleading fetchAllInRange comments (logs_orchestrator.go:289-291) — updated to describe count as a page size with continuation cursor emission.
Orchestration coverage gap — buildContinuationIfNeeded helper extracted; TestBuildContinuationIfNeeded added in logs_orchestrator_unit_test.go with four sub-cases (count-limit, timeout, neither → nil, empty runs → nil).
Relative date in test — "-1d" replaced with "2026-06-01" in TestBuildLogsDataWithCountLimitContinuation.

Local make fmt, make test-unit, and make agent-report-progress all pass. All review threads are outdated (addressed). The one CI failure (build job) is a pre-existing docs link-validation error in docs/reference/glossary.md (invalid hash #text-sanitization on the safe-outputs page) — none of my commits touch docs/. A human re-trigger is needed to get fresh CI on HEAD (38b6426).

github-actions · 2026-07-03T18:51:12Z

PR Triage

Field	Value
Category	bug (MCP timeout / pagination fix)
Risk	Medium
Score	62 (impact 30 + urgency 20 + quality 12)
Action	`fast_track`

Breakdown: Raises MCP gateway timeout to 10 min and adds count-limit pagination for daily audit workflows. Directly fixes audit coverage dropping from 175 to 84 runs. 195 add/35 del across audit workflow files. Well-described PR.

Next: Human review of workflow timeout changes and pagination cursor logic.

Generated by 🔧 PR Triage Agent · 86.6 AIC · ⌖ 10.9 AIC · ⊞ 5.5K · ◷

github-actions · 2026-07-07T10:08:58Z

🎉 This pull request is included in a new release.

Release: v0.82.3

Initial plan

831be37

Copilot AI assigned Copilot and pelikhan Jul 3, 2026

Copilot started work on behalf of pelikhan July 3, 2026 13:22 View session

Copilot AI linked an issue Jul 3, 2026 that may be closed by this pull request

[deep-report] Chunk/paginate the logs() collector to beat the 60s bridge deadline (fixes truncated audits) #42994

Closed

github-actions Bot mentioned this pull request Jul 3, 2026

[Contribution Check Report] Contribution Check — 2026-07-03 #43068

Closed

Copilot AI and others added 2 commits July 3, 2026 13:41

Fix truncated audits: add MCP tool-timeout 10m and count-limit pagina…

0376d49

…tion continuation Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Simplify continuation block to if-else; add test comment for relative…

6003b0b

… date format Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Chunk and paginate logs collector to meet deadline~~ logs: fix truncated daily audits — raise MCP gateway timeout and add count-limit pagination Jul 3, 2026

Copilot finished work on behalf of pelikhan July 3, 2026 13:49

Copilot AI requested a review from pelikhan July 3, 2026 13:49

pelikhan marked this pull request as ready for review July 3, 2026 14:48

Copilot AI review requested due to automatic review settings July 3, 2026 14:48

Copilot started reviewing on behalf of pelikhan July 3, 2026 14:49 View session

Copilot AI reviewed Jul 3, 2026

View reviewed changes

Copilot started work on behalf of gh-aw-bot July 3, 2026 15:30 View session

Address review feedback: fix comment, extract helper, fix relative da…

38b6426

…te in test Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot July 3, 2026 15:43

Copilot AI requested a review from gh-aw-bot July 3, 2026 15:43

Copilot started work on behalf of gh-aw-bot July 3, 2026 17:04 View session

Copilot finished work on behalf of gh-aw-bot July 3, 2026 17:13

github-actions Bot added pr-action:fast_track pr-agent:copilot-swe-agent pr-priority:medium pr-risk:medium labels Jul 3, 2026

github-actions Bot added the pr-type:bug label Jul 3, 2026

This was referenced Jul 3, 2026

[PR Triage Report] PR Triage Report — 2026-07-03 (Run #28677894072) #43257

Closed

[Outcome Report] Outcome Report — 2026-07-04 #43274

Closed

[PR Triage Report] PR Triage Report — 2026-07-04 (Run #28689979672) #43275

Closed

pelikhan merged commit 4a85f7e into main Jul 4, 2026
29 of 30 checks passed

pelikhan deleted the copilot/deep-report-chunk-paginate-logs branch July 4, 2026 03:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

logs: fix truncated daily audits — raise MCP gateway timeout and add count-limit pagination#43197

logs: fix truncated daily audits — raise MCP gateway timeout and add count-limit pagination#43197
pelikhan merged 4 commits into
mainfrom
copilot/deep-report-chunk-paginate-logs

Copilot AI commented Jul 3, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

gh-aw-bot commented Jul 3, 2026

Uh oh!

Copilot AI commented Jul 3, 2026

Uh oh!

gh-aw-bot commented Jul 3, 2026

Uh oh!

Copilot AI commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Copilot AI commented Jul 3, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MCP gateway timeout

Count-limit pagination continuation (logs_orchestrator.go)

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

gh-aw-bot commented Jul 3, 2026

Uh oh!

Copilot AI commented Jul 3, 2026

Uh oh!

gh-aw-bot commented Jul 3, 2026

Uh oh!

Copilot AI commented Jul 3, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

PR Triage

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jul 3, 2026 •

edited by github-actions Bot

Loading

Count-limit pagination continuation (`logs_orchestrator.go`)