sec(ci): gate terraform force-unlock behind explicit input (closes #438) by cristim · Pull Request #818 · LeanerCloud/CUDly

cristim · 2026-05-28T18:36:44Z

Summary

The deploy-aws-fargate.yml workflow unconditionally deleted the S3 Terraform state lock before every terraform init, allowing concurrent dispatch runs to race and corrupt state (issue sec(ci): deploy-aws-fargate.yml unconditionally deletes Terraform state-lock file at init time #438).
Replaced the automatic deletion with a new step gated on inputs.clear_stale_lock == true (a workflow_dispatch-only boolean input, default false). The existing if: failure() || cancelled() cleanup step is unchanged.
Added runbooks/terraform-stuck-lock.md explaining when/how to use the escape hatch and why unconditional clearing is unsafe.

Test plan

Verify a normal workflow_dispatch run (without clear_stale_lock) no longer deletes the lock file at init time.
Verify a run with clear_stale_lock=true executes the new step and removes the lock object.
Verify the Release state lock on failure step still fires on job failure/cancellation.
YAML parse-check: python3 -c "import yaml; yaml.safe_load(open('.github/workflows/deploy-aws-fargate.yml'))" passes.

Summary by CodeRabbit

New Features
- Added an operator-controlled option to clear stale Terraform S3 state locks for ECS Fargate deployments via workflow dispatch input, improving recovery from interrupted runs.
Documentation
- Added a runbook for diagnosing Terraform state lock issues and performing safe recovery, including workflow-based and manual AWS CLI steps.
Operational Safety
- Removed automatic state-lock deletion on failure/cancellation to avoid interfering with concurrent deployments; lock clearing is now only operator-triggered.

cristim · 2026-05-28T18:36:52Z

@coderabbitai review

coderabbitai · 2026-05-28T18:36:53Z

Important

Review skipped

No new commits to review since the last review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: fc935300-4ac0-4c1a-9305-b66af809fa79

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

📝 Walkthrough

Walkthrough

The workflow now clears Terraform S3 state locks only when explicitly requested through clear_stale_lock=true. Automatic failure cleanup is removed, and a runbook documents lock symptoms, verification steps, workflow-dispatch recovery, and manual AWS CLI recovery.

Changes

Operator-triggered Terraform state lock recovery

Layer / File(s)	Summary
Workflow input and conditional lock-clearing step `.github/workflows/deploy-aws-fargate.yml`	Adds the `clear_stale_lock` dispatch input, conditionally deletes the S3 `.tflock` object, keeps Terraform Init focused on backend configuration, and removes automatic lock release after failure or cancellation.
Stuck lock recovery procedures `runbooks/terraform-stuck-lock.md`	Documents lock-acquisition symptoms, verification of the owning run, workflow-dispatch and AWS CLI recovery paths, and why automatic deletion is avoided.

Estimated code review effort: 2 (Simple) | ~12 minutes

Possibly related PRs

LeanerCloud/CUDly#1131: Modifies the same deployment workflow’s Terraform S3 state-lock handling and recovery flow.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures the main change: making Terraform force-unlock conditional on an explicit input.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/438-wave7

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai · 2026-05-28T18:36:58Z

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

cristim · 2026-05-30T10:22:28Z

@coderabbitai full review

coderabbitai · 2026-05-30T10:22:32Z

✅ Actions performed

Full review triggered.

cristim · 2026-06-01T23:30:59Z

@coderabbitai review

coderabbitai · 2026-06-01T23:31:03Z

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

cristim · 2026-06-04T14:04:24Z

@coderabbitai review

coderabbitai · 2026-06-04T14:04:33Z

Rate Limit Exceeded

@cristim have exceeded the limit for the number of chat messages per hour. Please wait 39 minutes and 2 seconds before sending another message.

cristim · 2026-06-08T00:31:07Z

@coderabbitai full review

Generated by Claude Code

coderabbitai · 2026-06-08T00:31:17Z

@cristim Sure, triggering a full review of PR #818 now.

🧠 Learnings used

Learnt from: cristim
Repo: LeanerCloud/CUDly PR: 0
File: :0-0
Timestamp: 2026-05-05T15:44:54.463Z
Learning: In the LeanerCloud/CUDly repository, when a CodeRabbit review flags something as "outside-diff" that was already implemented in an earlier commit of the same PR, it is likely a stale-snapshot artifact where the review diff was generated against a pre-fix view. The inline comment convention `(CR pass on PR #<number> Finding N)` is used by the maintainer (cristim) to mark lines that were changed in direct response to a prior CodeRabbit review pass.

✅ Action performed

Full review finished.

cristim · 2026-06-08T07:48:20Z

@coderabbitai full review

coderabbitai · 2026-06-08T07:48:25Z

✅ Action performed

Full review finished.

cristim · 2026-06-19T21:08:56Z

@coderabbitai review

coderabbitai · 2026-06-19T21:09:06Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/deploy-aws-fargate.yml:
- Around line 117-132: The workflow has an automatic failure-path unlock
mechanism that unconditionally removes the Terraform state lock on any failed or
cancelled run, which creates a race condition where one failing concurrent run
can delete another healthy run's active lock, causing state corruption. Either
remove the automatic failure-path unlock mechanism entirely and rely solely on
the operator-triggered clear_stale_lock step (which requires explicit
input.clear_stale_lock == true), or modify the failure-path unlock to include
ownership awareness by adding a unique identifier or token that tracks which run
created the lock before allowing removal. The safest approach is to delete the
automatic failure cleanup step and require manual operator intervention via the
clear_stale_lock input flag.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 56c13b0f-5208-4309-86a4-9983e17993f6

📥 Commits

Reviewing files that changed from the base of the PR and between 451a70f and 8f38465.

📒 Files selected for processing (2)

.github/workflows/deploy-aws-fargate.yml
runbooks/terraform-stuck-lock.md

cristim · 2026-06-19T21:41:29Z

@coderabbitai review

coderabbitai · 2026-06-19T21:41:36Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

cristim · 2026-07-09T21:20:05Z

@coderabbitai review

coderabbitai · 2026-07-09T21:20:11Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

cristim · 2026-07-10T14:50:18Z

@coderabbitai review

coderabbitai · 2026-07-10T14:50:24Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

cristim · 2026-07-17T07:53:33Z

@coderabbitai review

cristim · 2026-07-17T07:53:50Z

Merge-prep: rebased onto origin/main (27fdb06), no conflicts. Non-Go PR; go vet on internal/... clean. @coderabbitai full review

cristim · 2026-07-17T11:24:59Z

Rebased onto main (includes #1437 tflint cache + #1438 trivy skip). No Go changes; CI workflow + runbook changes only.

Unconditionally deleting the S3 state lock before every terraform init allowed concurrent dispatch runs to race and corrupt state. Remove the automatic deletion and replace it with a gated step that only runs when the operator explicitly sets clear_stale_lock=true on workflow_dispatch. The existing failure/cancellation cleanup step is unchanged. Add runbooks/terraform-stuck-lock.md with diagnosis and recovery steps.

…k.md markdownlint MD040 requires fenced code blocks to specify a language. The error-output block at line 7 had a bare ``` opener; add "text" tag.

The "Release state lock on failure" step deleted the S3 tflock on any failed/cancelled run. A run that fails because it could not acquire the lock would delete the lock a concurrent run is still actively holding, re-introducing the state-corruption race this PR aims to close. Remove the automatic failure-path unlock. Terraform already releases its own lock on a clean apply error; a lock surviving a run means the run died abnormally, which requires operator confirmation via clear_stale_lock before clearing. Update the runbook accordingly.

cristim · 2026-07-17T16:29:11Z

@coderabbitai review

coderabbitai · 2026-07-17T16:29:26Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

cristim added triaged Item has been triaged priority/p3 Polish / idea / may never ship severity/low Minor harm urgency/this-quarter Within the quarter impact/internal Team-internal only effort/xs Trivial / one-liner type/security Security finding labels May 28, 2026

cristim changed the base branch from feat/multicloud-web-frontend to main June 9, 2026 15:45

cristim force-pushed the fix/438-wave7 branch from b4ee61b to 8f38465 Compare June 19, 2026 21:08

coderabbitai Bot reviewed Jun 19, 2026

View reviewed changes

Comment thread .github/workflows/deploy-aws-fargate.yml

cristim mentioned this pull request Jun 19, 2026

sec(ci): lambda deploy has same unsafe failure-path state-lock delete as #438 #1269

Open

cristim force-pushed the fix/438-wave7 branch from 064fbc4 to 5419ead Compare July 10, 2026 14:50

cristim force-pushed the fix/438-wave7 branch from 5419ead to a855b73 Compare July 17, 2026 07:53

cristim force-pushed the fix/438-wave7 branch from a855b73 to 8dba861 Compare July 17, 2026 11:24

cristim added 3 commits July 17, 2026 19:22

fix(ci): add language tag to fenced code block in terraform-stuck-loc…

6715e24

…k.md markdownlint MD040 requires fenced code blocks to specify a language. The error-output block at line 7 had a bare ``` opener; add "text" tag.

cristim force-pushed the fix/438-wave7 branch from 8dba861 to a344ca1 Compare July 17, 2026 16:23

cristim merged commit 4aa4ead into main Jul 17, 2026
19 checks passed

cristim deleted the fix/438-wave7 branch July 27, 2026 11:09

Uh oh!

Conversation

cristim commented May 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Summary by CodeRabbit

Uh oh!

cristim commented May 28, 2026

Uh oh!

coderabbitai Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Walkthrough

Changes

Uh oh!

coderabbitai Bot commented May 28, 2026

Uh oh!

cristim commented May 30, 2026

Uh oh!

coderabbitai Bot commented May 30, 2026

Uh oh!

cristim commented Jun 1, 2026

Uh oh!

coderabbitai Bot commented Jun 1, 2026

Uh oh!

cristim commented Jun 4, 2026

Uh oh!

coderabbitai Bot commented Jun 4, 2026

Rate Limit Exceeded

Uh oh!

cristim commented Jun 8, 2026

Uh oh!

coderabbitai Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cristim commented Jun 8, 2026

Uh oh!

coderabbitai Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cristim commented Jun 19, 2026

Uh oh!

coderabbitai Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cristim commented Jun 19, 2026

Uh oh!

coderabbitai Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cristim commented Jul 9, 2026

Uh oh!

coderabbitai Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cristim commented Jul 10, 2026

Uh oh!

coderabbitai Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cristim commented Jul 17, 2026

Uh oh!

cristim commented Jul 17, 2026

Uh oh!

cristim commented Jul 17, 2026

Uh oh!

cristim commented Jul 17, 2026

Uh oh!

coderabbitai Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

cristim commented May 28, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 28, 2026 •

edited

Loading

coderabbitai Bot commented Jun 8, 2026 •

edited

Loading

coderabbitai Bot commented Jun 8, 2026 •

edited

Loading

coderabbitai Bot commented Jun 19, 2026 •

edited

Loading

coderabbitai Bot commented Jun 19, 2026 •

edited

Loading

coderabbitai Bot commented Jul 9, 2026 •

edited

Loading

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading

coderabbitai Bot commented Jul 17, 2026 •

edited

Loading