fix(execution): recover from panics inside FanOutWithConcurrency goroutines (closes #669)

[cristim]

Summary

Defence-in-depth for internal/execution/FanOutWithConcurrency. Before this change, any panic inside fn (nil deref, type assertion failure, slice OOB in a future refactor) propagated up the unrecovered goroutine and crashed the entire Lambda process. Consequences:

• Purchase execution row stays at approved with no transition to failed (the aggregator that would record the failure never runs).
• Lambda invocation terminates abnormally; CloudWatch may not flush the final log lines.
• User sees a generic Lambda invocation error rather than the actual panic detail.

The fan-out helper now installs a deferred recover() in every goroutine that:

1. Converts the panic into a structured Err on the per-item Result slot.
2. Logs the goroutine stack at Error level for post-mortem.
3. Lets the parent aggregator process the failure exactly like a normal fn-returned error.

Why now?

Current call sites in internal/purchase/execution.go (per-account executeForAccount + per-rec processPurchaseRecommendations) have no obvious panic source today, so this is cheap insurance, not a bug fix. But the failure mode is catastrophic (whole-Lambda crash + stranded execution at approved), and the cost is ~10 lines with zero perf impact. Found during the concurrency-audit pass on #667 + #632; filed as #669.

What changed

• internal/execution/fanout.go: added a deferred recover() inside the goroutine launched by FanOutWithConcurrency. Captures the panic value + stack via runtime.Stack, logs at Error, and writes a structured error to results[idx].
• internal/execution/fanout_test.go: new test TestFanOut_PanicInFn panics from fn for one of three items, asserts the panicking item carries an Err containing the panic value AND that the other two items still succeed — i.e. one goroutine's panic doesn't cascade across the fan-out.

Test plan

• go test ./internal/execution/... ./internal/purchase/... -count=1 -short — 174/174 pass
• go build ./... — clean
• Optional post-merge: induce a deliberate panic in a test executeSinglePurchase to verify end-to-end that the execution row transitions to failed (not approved) with the panic visible in exec.Error

Cross-references

• Closes fix(execution): FanOutWithConcurrency goroutine has no recover() — purchase panics crash the whole Lambda (P1) #669
• Refs bug(purchases): approve strands execution in 'approved' (purchase never runs, no error) on interrupted sync execution #632 (status stranding when sync execute fails — recover() reduces one cause of this)
• Refs fix(purchases): EC2 RI purchase times out under transient AWS SDK retries; Lambda 60s budget blown (P0) #667 (the context-deadline timeout that prompted the concurrency audit; this PR is from that same audit, not the fix for fix(purchases): EC2 RI purchase times out under transient AWS SDK retries; Lambda 60s budget blown (P0) #667 itself)
• Original commit was pushed onto fix/purchases-add-execution-logging (PR fix(purchases): add execution-tagged logging to executeSinglePurchase (refs #667) #668) after that PR had already merged with only the logging commit — this PR is the orphaned recover() commit landed cleanly on its own branch.

Summary by CodeRabbit

• Bug Fixes
  • Enhanced concurrent processing to gracefully handle failures; individual operations that encounter issues now return errors instead of causing unexpected terminations, ensuring more robust batch processing.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a5ad9656-726f-4357-955c-16cb27cbe005

📥 Commits

Reviewing files that changed from the base of the PR and between 6fa5048 and e6fdb42.

📒 Files selected for processing (2)

• internal/execution/fanout.go
• internal/execution/fanout_test.go

---

📝 Walkthrough

Walkthrough

The pull request adds panic recovery to FanOutWithConcurrency, a helper that executes work across multiple goroutines. When the per-item function panics, the goroutine now catches it, logs the panic with a stack trace, and converts it to an error result instead of crashing the process.

Changes

Panic Recovery in Fan-Out Execution

Layer / File(s)	Summary
Panic recovery implementation internal/execution/fanout.go	Adds fmt and runtime imports, then wraps the fan-out goroutine with a defer/recover block that catches panics from fn, logs them with stack trace via runtime.Stack, and stores a corresponding Err in the appropriate results[idx] entry.
Panic recovery test internal/execution/fanout_test.go	Adds TestFanOut_PanicInFn that invokes FanOut with one item designed to panic and others that succeed, asserting that the panicking item's result contains an error with the recovered panic value and account ID, while non-panicking items return normal results.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

effort/s

Poem

🐰 A goroutine panicked with glee,
But recover() caught it with care,
Now errors stack trace so we see—
No more Lambda crashes from there!
The purchase row marks failure true,
Recovery logs light the path through. 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding panic recovery to FanOutWithConcurrency goroutines. It is concise, specific, and directly related to the changeset.
Linked Issues check	✅ Passed	The PR meets all primary coding requirements from issue #669: panic recovery is implemented in FanOutWithConcurrency, stack traces are logged at Error level, and a regression test validates panic handling.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to addressing issue #669: panic recovery implementation in fanout.go and a corresponding test in fanout_test.go. No unrelated modifications are present.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/issue-669-fanout-recover

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[cristim]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.