fix(purchases): close double-buy holes in idempotency, atomic claim, and partial-success labelling - #1037
Conversation
…and partial-success labelling Three interlocking money-path defects that could cause DUPLICATE real cloud commitment purchases: #1012 - idempotency identity was tied to the mutable execution ID, so retry (fresh UUID) and multi-account fan-out (per-account UUID) derived a DIFFERENT provider token than the original attempt, defeating provider-level dedupe. - Add a stable idempotency_key column (migration 000066), generated once at first creation, INSERT-only (never rewritten by ON CONFLICT). - Derive the per-rec token from idempotencyLineageKey(exec) (the key, with an ExecutionID fallback for legacy NULL rows) instead of exec.ExecutionID. - Copy the key verbatim onto retry successors; seed each multi-account row's key from rootKey+accountID (deterministic, not uuid.New()). #1013 - the atomic row-claim that the sync approve path has was missing on the SQS execute_purchase and cron paths, so at-least-once SQS redelivery and overlapping cron ticks could double-execute the same row. - Add claimAndExecute: CAS [approved,pending,notified] -> running; only the winner proceeds, a lost CAS is a benign ack/skip. - Route handleExecutePurchase and ProcessScheduledPurchases through it; add a positive-allowlist guard on the cron loop (M2). #1014 - multi-account partial success was joined into an opaque error string that finalizeExecution classified as "failed", and an SQS handler returning it invited redelivery -> re-execution -> double-buy. - executeMultiAccount returns a typed *multiAccountPartialError (>=1 account committed) vs errAllAccountsFailed (nothing committed). - finalizeExecution maps the partial sentinel to partially_completed, never failed; the root row is now always saved with its aggregate status (H2), fixing the stranded-"running" root the new claim would otherwise leave. - SQS/cron callers ack (no redeliver) when >=1 account committed. Regression tests replicate the real scenarios and fail on pre-fix code: retry/fan-out reuse the same token; a redelivered SQS message executes the cloud purchase exactly once; a multi-account partial run is acked and recorded partially_completed (not failed). Closes #1012 Closes #1013 Closes #1014
|
Warning Review limit reached
More reviews will be available in 7 minutes and 55 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (14)
📝 WalkthroughWalkthroughThis PR resolves a critical double-buy risk in purchase execution retries and multi-account fan-out by introducing stable idempotency lineage (persisted ChangesPurchase execution idempotency and atomic claiming
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
internal/config/store_postgres.go (1)
1251-1283:⚠️ Potential issue | 🟠 Major | ⚡ Quick win
scanExecutionRowscontract is now stricter thanGetPlannedExecutionsprojection.Line 1256 scans
executed_by_user_id,executed_at,pre_approval_skip_reason, andidempotency_key, but theGetPlannedExecutionsSELECT (Line 993-Line 999) still stops atapproval_token_expires_at. This can fail at runtime withfailed to scan executionon planned-purchase reads.🛠️ Suggested fix
func (s *PostgresStore) GetPlannedExecutions(ctx context.Context, statuses []string, limit int) ([]PurchaseExecution, error) { @@ query := ` SELECT plan_id, execution_id, status, step_number, scheduled_date, notification_sent, approval_token, recommendations, total_upfront_cost, estimated_savings, completed_at, error, expires_at, cloud_account_id, source, approved_by, cancelled_by, capacity_percent, created_by_user_id, retry_execution_id, retry_attempt_n, - approval_token_expires_at + approval_token_expires_at, + executed_by_user_id, executed_at, pre_approval_skip_reason, + idempotency_key FROM purchase_executions WHERE status = ANY($1) ORDER BY scheduled_date ASC NULLS LAST, id ASC LIMIT $2 `🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/config/store_postgres.go` around lines 1251 - 1283, The scanExecutionRows scanning code now expects columns executed_by_user_id, executed_at, pre_approval_skip_reason and idempotency_key but the GetPlannedExecutions SELECT projection stops at approval_token_expires_at; update the GetPlannedExecutions SELECT to include executed_by_user_id, executed_at, pre_approval_skip_reason and idempotency_key (matching types/nullability) so rows.Scan in scanExecutionRows can map columns correctly (or alternatively remove those extra scans from scanExecutionRows if they are not needed) — locate the SELECT in GetPlannedExecutions and align its projected columns with the rows.Scan call in scanExecutionRows.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/config/store_postgres_pgxmock_test.go`:
- Around line 449-450: The test sets up both legacy (NULL) and non-NULL
idempotency_key scan paths but never asserts the resulting Exec object's
IdempotencyKey, so add explicit assertions: after the NULL-path scan (where the
diff passes nil for idempotency_key) assert exec.IdempotencyKey == nil (or empty
as your type expects), and after the non-NULL-path scan assert
exec.IdempotencyKey is non-nil and equals the original idempotency key value
used in the test; update the assertions around the scan calls that populate exec
(look for the variable name exec and the idempotency_key setup in the test) to
validate both outcomes.
---
Outside diff comments:
In `@internal/config/store_postgres.go`:
- Around line 1251-1283: The scanExecutionRows scanning code now expects columns
executed_by_user_id, executed_at, pre_approval_skip_reason and idempotency_key
but the GetPlannedExecutions SELECT projection stops at
approval_token_expires_at; update the GetPlannedExecutions SELECT to include
executed_by_user_id, executed_at, pre_approval_skip_reason and idempotency_key
(matching types/nullability) so rows.Scan in scanExecutionRows can map columns
correctly (or alternatively remove those extra scans from scanExecutionRows if
they are not needed) — locate the SELECT in GetPlannedExecutions and align its
projected columns with the rows.Scan call in scanExecutionRows.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 31780e21-0af8-45ff-bf82-0ce5f0403d30
📒 Files selected for processing (14)
internal/api/handler_purchases.gointernal/config/store_postgres.gointernal/config/store_postgres_pgxmock_test.gointernal/config/types.gointernal/database/postgres/migrations/000066_purchase_executions_idempotency_key.down.sqlinternal/database/postgres/migrations/000066_purchase_executions_idempotency_key.up.sqlinternal/purchase/coverage_extra_test.gointernal/purchase/execution.gointernal/purchase/execution_test.gointernal/purchase/manager.gointernal/purchase/manager_test.gointernal/purchase/messages.gointernal/purchase/messages_test.gointernal/purchase/money_path_regression_test.go
…gocyclo Adding idempotency_key handling pushed scanExecutionRows to cyclomatic complexity 11, one over the budget of 10. Pull the nullable-timestamp mapping into applyExecutionNullableTimes, a behavior-preserving helper, bringing scanExecutionRows back under the limit. No logic change.
…test The pre-commit gofmt hook runs against the PR merge ref and flagged unaligned struct literal fields in reshape_crossfamily_test.go (carried over from an earlier merge). Re-run gofmt to unblock CI. Whitespace only.
GetPlannedExecutions was the only execution-reading query whose SELECT still stopped at approval_token_expires_at (22 columns) after migration 000066 added idempotency_key. Every other query and the shared scanExecutionRows Scan target expect 26 columns, so the planned-purchase list path (handler_purchases.go) would fail at runtime with "failed to scan execution". Add the four trailing columns (executed_by_user_id, executed_at, pre_approval_skip_reason, idempotency_key) to align the projection with the scan. Add TestPGXMock_GetPlannedExecutions_ProjectsAllScanColumns, which fails on the pre-fix projection (the regexp query matcher does not match idempotency_key) and passes once the columns are present, mirroring the real scan-failure path. Also assert IdempotencyKey scan outcomes in the GetExecutionByID NULL and non-NULL tests so a scan-order regression on that field cannot slip through.
|
Addressed CR review findings (pass 1) plus the two CI gates:
@coderabbitai review |
|
Kicking off pass 2 now. 🧠 Learnings used✅ Action performedReview finished.
|
sem <- struct{}{} in FanOutWithConcurrency was unconditional, so a
large fan-out on a cancelled/expired context would block indefinitely
at the semaphore boundary rather than draining quickly.
Replace with a select that immediately records ctx.Err() on the
unstarted result slot when the context is done, then continues the
launch loop. The wg counter is still incremented before the select so
wg.Wait() remains correct; wg.Done() is called in the ctx.Done branch
to balance it.
Regression test: TestFanOut_ContextCancelled_BlockedSemaphore -- with
maxConcurrency=1 and a cancel fired while the first goroutine holds the
slot, the second item gets context.Canceled immediately instead of
blocking until the first goroutine releases (fails on pre-fix code).
Closes #1069 (partial).
…cs (05-M1) applySuppressionIndex and filterRecsByResolvedConfigs both wrote out := recs[:0] which aliases the caller's backing array. Any caller that holds a reference to the original slice would see elements silently overwritten on the first appends into out. Replace with make([]config.RecommendationRecord, 0, len(recs)) so the output is always an independent allocation. Regression tests: TestApplySuppressionIndex_DoesNotMutateCallerSlice and TestFilterRecsByResolvedConfigs_DoesNotMutateCallerSlice snapshot the original IDs, call the functions, and assert the original slice is unchanged (fail on pre-fix code). Closes #1069 (partial).
…/M4/N1/N2/N3)
05-M4 (commitmentopts): add Warn log when provider probe data exists but
the requested service is absent, so operators see misconfigured service
names or probe gaps without blocking the plan save.
05-L1 (audit gap marker): prefix recordHistoryAuditGap's exec.Error note
with historyAuditGapPrefix ("history_write_failed") + commitment ID so
log queries can filter on the prefix programmatically instead of grepping
free-form prose.
05-L2 (getAWSAccountID): change signature from (string) to (string, error).
Returns an error on all failure paths instead of the "unknown" sentinel;
the sole caller logs a Warn and proceeds with an empty account ID, so
the caller decides the fallback rather than the STS helper.
Existing tests updated to assert on (id, err) pairs.
05-L3 (normalizePurchaseSource): add explanatory comment documenting the
"proceed untagged" decision and why failing the rec over a tag-only field
would be a worse outcome than an absent tag.
05-N1 (reaper _ = now): wire `now` into the sweep log line (RFC3339
timestamp) so the injected clock is actually used and the `_ = now`
suppressors can be removed.
05-N2 (aggregator contract): add a comment before the
aggregatePurchaseOutcomes call documenting the single-threaded-aggregator
invariant so it is visible at the call site, not just on the function.
05-N3 (sweep-threshold doc): expand staleApprovedThreshold's godoc to
cross-reference DefaultReapAfter in reaper.go and explain why the two
thresholds differ.
Closes #1069 (partial).
FOLD-1037 scope expansion: per-finding summaryThe following FOLD-1037 findings from 16-remaining-findings-plan.md have been implemented in 3 additional commits (81ff5e3, c6cb85b, d74e045). All regression tests confirmed to fail on pre-fix code. 05-H3 (High) -- fanout semaphore ignores ctx cancellation (internal/execution/fanout.go:70-72): replaced unconditional sem<- with select+ctx.Done(); test: TestFanOut_ContextCancelled_BlockedSemaphore 05-M1 (Medium) -- recs[:0] slice aliasing in scheduler filters (scheduler.go:1144, scheduler_overrides.go:55): allocate make([]T,0,len(recs)) instead; tests: TestApplySuppressionIndex_DoesNotMutateCallerSlice, TestFilterRecsByResolvedConfigs_DoesNotMutateCallerSlice 05-M4 (Medium) -- commitmentopts silent permissive fallback on unknown service (service.go:163): add Warn log when provider known but service absent in probe data 05-L1 (Low) -- unqueryable audit-gap marker (recordHistoryAuditGap): prefix note with 'history_write_failed:' + commitment ID for CloudWatch filter support 05-L2 (Low) -- getAWSAccountID returns 'unknown' sentinel: changed to (string, error); caller logs Warn and proceeds with empty accountID; tests updated 05-L3 (Low) -- normalizePurchaseSource: added comment documenting 'proceed untagged' decision; behavior unchanged 05-N1 (Nit) -- dead _ = now in reaper: wired now into sweep log line (RFC3339), removed both suppressors 05-N2 (Nit) -- missing single-threaded aggregator contract: added comment before aggregatePurchaseOutcomes call 05-N3 (Nit) -- two stuck-execution thresholds undocumented: expanded staleApprovedThreshold godoc to cross-reference DefaultReapAfter Tracked separately (NOT implemented here, by design):
Closes #1069 @coderabbitai review |
|
Tip For best results, initiate chat on the files or code changes.
Triggering pass 3 now. [review] 🧠 Learnings used |
|
Addressed CR pass-1 findings (both stale -- resolved in FOLD commits):
Build, vet, and @coderabbitai review |
|
🧠 Learnings used✅ Action performedReview finished.
|
Summary
Three interlocking money-path defects that could cause DUPLICATE real cloud commitment purchases (AWS RI/SP, Azure reservation, GCP CUD). They share the same files, so they land together. Source review:
docs/code-review/05-purchase-execution.md(findings C1/H1, C2/C3/M2, H2/M3).The single-account synchronous approve path was the only one with the "claim the row atomically before touching the cloud" guard; the other two executor entry points lacked it, and idempotency identity was tied to a per-attempt mutable key.
What changed
#1012 — idempotency identity independent of the execution ID
exec.ExecutionID, which retry regenerates (fresh UUID per successor) and multi-account fan-out regenerates (fresh UUID per account) — so a re-drive of a "failed"-but-landed execution derived a different token and the provider guard couldn't match → double-buy.idempotency_keycolumn (migration 000066, nullable), generated once at first creation, INSERT-only (never rewritten byON CONFLICT, lockstep with the existingcreated_by_user_id/retry_attempt_npattern).idempotencyLineageKey(exec)(the key, with anExecutionIDfallback for legacy NULL rows).rootKey + ":" + accountID(deterministic, replacesuuid.New()).#1013 — single atomic claim-then-execute for all entry points
claimAndExecute: CAS["approved","pending","notified"] -> "running"via the existingTransitionExecutionStatus; only the winner proceeds, a lost CAS is a benign ack/skip (mirrors the reaper's race model).handleExecutePurchase(SQS, at-least-once redelivery) andProcessScheduledPurchases(overlapping cron ticks) now funnel through it.pending/notifiedproceed).#1014 — typed multi-account result; partial success is not "failed"
executeMultiAccountreturns a typed*multiAccountPartialError(≥1 account committed) vserrAllAccountsFailed(nothing committed), instead of an opaque joined string.finalizeExecutionmaps the partial sentinel topartially_completed, neverfailed.!wasMultiAccountsave-skip, and prevents the new claim from stranding the root inrunning).How each fix was verified
Regression tests in
internal/purchase/money_path_regression_test.goreplicate the real failing scenarios and were confirmed to fail on pre-fix code and pass after:TestRetryReusesIdempotencyToken/TestMultiAccountSeedsStablePerAccountKey— retry + fan-out reuse the identical provider token / per-account key (fail pre-fix: tokens differ).TestLegacyRowFallsBackToExecutionID— legacy NULL-key rows keep working.TestSQSRedeliveryDoesNotDoubleExecute— two deliveries of the same message run the cloud purchase exactly once (fails pre-fix: runs twice).TestMultiAccountPartialSuccessIsAcked— a partial multi-account run is acked (handler returns nil) and the root is recordedpartially_completed(fails pre-fix: handler errors → redeliver).Build/vet/test:
go build ./...,go vet ./internal/purchase/... ./internal/api/... ./internal/config/..., andgo test ./internal/purchase/... ./internal/api/... ./internal/config/...all pass (2439 tests across 6 packages after fold).go test -race ./internal/purchase/...clean.DB-backed integration tests were exercised via
pgxmock(column scan order updated for the new column); no live Postgres was required.Closes #1012
Closes #1013
Closes #1014
Scope expanded
Additional findings from the FOLD-1037 review pass (
docs/code-review/16-remaining-findings-plan.md) have been folded into this PR. All touch files already modified by the original commits. Regression tests confirm pre-fix failures.05-H3 — fanout semaphore ignores context cancellation (
internal/execution/fanout.go)sem <- struct{}{}in the goroutine-launch loop blocked unconditionally even when the context was already cancelled. A large fan-out after a deadline expiry could hang indefinitely. Fixed with aselectthat recordsctx.Err()on the result slot and continues.Regression test:
TestFanOut_ContextCancelled_BlockedSemaphore(maxConcurrency=1, second item getscontext.Canceledimmediately after cancel).05-M1 — recs[:0] slice aliasing (
internal/scheduler/scheduler.go,scheduler_overrides.go)applySuppressionIndexandfilterRecsByResolvedConfigswroteout := recs[:0], aliasing the caller's backing array. Replaced withmake([]config.RecommendationRecord, 0, len(recs)).Regression tests:
TestApplySuppressionIndex_DoesNotMutateCallerSlice,TestFilterRecsByResolvedConfigs_DoesNotMutateCallerSlice.05-M4 — commitmentopts silent permissive fallback (
internal/commitmentopts/service.go)Validatereturnedtruewithout logging when the provider was known but the service was absent in probe data. Now logs at Warn so operators can detect probe gaps or misconfigured service names.05-L1 — unqueryable audit-gap marker (
internal/purchase/execution.go)recordHistoryAuditGapnow prefixes the note withhistory_write_failed:+ commitment ID, making these entries filterable in CloudWatch Insights.05-L2 — getAWSAccountID sentinel string (
internal/purchase/execution.go)getAWSAccountIDnow returns("", error)on all failure paths instead of the"unknown"sentinel; the caller decides the fallback. Existing tests updated.05-L3 — normalizePurchaseSource decision documented (
internal/purchase/execution.go)Added explanatory comment: the purchase proceeds untagged on invalid source (by design) because failing the rec over a tag-only field is a worse outcome.
05-N1 — dead
_ = nowin reaper (internal/purchase/reaper.go)Wired
nowinto the sweep log line (RFC3339 timestamp) so the injected-clock parameter is actually used and the_ = nowsuppressors are removed.05-N2 — single-threaded aggregator contract (
internal/purchase/execution.go)Added comment before the
aggregatePurchaseOutcomescall documenting that the aggregation is intentionally serial.05-N3 — sweep-threshold discrepancy documented (
internal/purchase/manager.go)Expanded
staleApprovedThresholdgodoc to cross-referenceDefaultReapAfterinreaper.goand explain the 15m vs 10m difference.Also closes #1069
Tracked separately (not implemented here):
updatePlanProgressnon-atomic read-modify-write (05-L4); needs FOR UPDATE or atomic DB incrementGetPendingExecutionsTxdead code chore (superseded by fix(purchases): close double-buy holes in idempotency, atomic claim, and partial-success labelling #1037's per-row CAS design, which is intentional and sound)Summary by CodeRabbit
Release Notes