fix: handle index_not_found in token resolver to avoid spurious checkin errors

[ycombinator]

What is the problem this PR solves?

When .fleet-actions does not yet exist (e.g. a new project that has never had an explicit agent action), a fleet-server restart or a new fleet-server instance being spun up clears the in-memory LRU ack-token cache. The subsequent cache-miss triggers a search on .fleet-actions via TokenResolver.Resolve → dl.FindSeqNoByDocID. ErrIndexNotFound was not handled in this code path and propagated all the way up to the agent check-in handler, logging resolveSeqNo: elastic fail 404: index_not_found_exception: no such index [.fleet-actions] on every check-in.

How does this PR solve the problem?

Treats ErrIndexNotFound the same as ErrNotFound in TokenResolver.Resolve, so the caller (resolveSeqNo in handleCheckin.go) falls back to agent.ActionSeqNo gracefully. This is consistent with how every other read path in fleet-server already handles a missing .fleet-actions index (e.g. findActionsHits, DeleteExpiredForIndex).

How to test this PR locally

Start fleet-server against a cluster where .fleet-actions does not exist, enroll an agent, and verify check-ins succeed without resolveSeqNo errors in the logs.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

🤖 Generated with Claude Code

[github-actions]

TL;DR

The failing :gcloud: Cloud e2e FIPS Test job exited non-zero, but the attached log artifact only contains cleanup (terraform destroy) and does not include the actual failing command/test output. Immediate next action is to rerun this job and capture the full mage test:cloudE2ERun output (or JUnit artifact) to identify the concrete failing test/assertion.

Remediation

• Re-run the :gcloud: Cloud e2e FIPS Test step and collect full logs for mage test:cloudE2ERun (the failure happened before cleanup).
• Ensure the step uploads JUnit/test artifacts from the cloud E2E run so the first failing test case is visible in CI.
• If the rerun fails the same way, share the first error block before cleanup and we can map it directly to the responsible test/code path.

Investigation details

Root Cause

From the provided artifact, this is currently infrastructure/observability gap in CI logs, not a confirmed code regression:

• The failing log file starts in teardown state and shows only deployment destroy output, then generic process exit.
• .buildkite/scripts/cloud_e2e_test.sh preserves the failing status from an earlier command and always runs cleanup via trap:
  • .buildkite/scripts/cloud_e2e_test.sh:17-28 (cleanup() captures $?, runs mage test:cloudE2EDown, then exit $r)
  • .buildkite/scripts/cloud_e2e_test.sh:48 runs mage test:cloudE2ERun (likely failing command, but its error output is missing from the artifact)

Because the artifact lacks the upstream error lines, there is not enough evidence to attribute this failure to the PR code changes.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16037
• Job/step: :gcloud: Cloud e2e FIPS Test
• Key log excerpt (from /tmp/gh-aw/buildkite-logs/fleet-server-gcloud-cloud-e2e-fips-test.txt):
  • Destroy complete! Resources: 1 destroyed.
  • 🚨 Error: The command exited with status 1
  • user command error: exit status 1

Verification

• Not run locally (cloud E2E/FIPS environment is CI-managed and this detective workflow is read-only).

Follow-up

• I also checked for existing flaky-test labeled issues matching this failure mode in elastic/fleet-server and found none tied to Cloud E2E FIPS from the available query.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[github-actions]

@Mergifyio backport 9.5 9.4 8.19

[mergify]

backport 9.5 9.4 8.19

✅ Backports have been created

Details

• #7559 [9.5](backport #7548) fix: handle index_not_found in token resolver to avoid spurious checkin errors has been created for branch 9.5
• #7560 [9.4](backport #7548) fix: handle index_not_found in token resolver to avoid spurious checkin errors has been created for branch 9.4
• #7561 [8.19](backport #7548) fix: handle index_not_found in token resolver to avoid spurious checkin errors has been created for branch 8.19