Skip to content

Reduce Redis block wait scan pressure#1139

Merged
bootjp merged 3 commits into
mainfrom
feature/redis-block-wait-fallback
Jul 22, 2026
Merged

Reduce Redis block wait scan pressure#1139
bootjp merged 3 commits into
mainfrom
feature/redis-block-wait-fallback

Conversation

@bootjp

@bootjp bootjp commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Summary

  • increase the Redis blocking-command fallback interval from 100ms to 250ms to reduce full key-type scan pressure from idle BZPOPMIN/XREAD wait loops
  • keep the fallback below common sub-second BLOCK budgets, and keep signal-driven ZADD/XADD wakeups immediate
  • make the fallback configurable so integration tests can retain a shorter cadence without forcing production to scan as often

Runtime evidence

  • Redis proxy pool pending stayed at 0 while async drops continued, so the remaining bottleneck was not connection checkout
  • post-deploy CPU profile moved from registry decode error allocation to BZPOPMIN/XREAD key-type scans
  • leader CPU dropped after the registry fast path, but proxy drops still increased with write queue depth around 1k+, requiring this follow-up

Validation

  • go test ./adapter -run 'TestRedis_BZPopMin|TestRedis_StreamXRead|TestKeyWaiterRegistry|TestCommandGetKeys_BZPOPMIN' -count=1
  • golangci-lint run ./adapter --timeout=5m

Author: bootjp

Summary by CodeRabbit

  • 新機能

    • Redisのブロッキングコマンドにおける待機フォールバック間隔を設定できるようになりました。
    • デフォルト値を維持しつつ、環境や用途に応じて待機間隔を調整できます。
  • 改善

    • ストリームやソート済みセットの待機処理が、設定された間隔とブロック残り時間に応じて動作するようになりました。
    • 接続終了時やデータ更新時の待機解除が、より適切なタイミングで行われます。

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@bootjp, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 53 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: af29cddf-fafb-4c7a-bd41-f103f5760650

📥 Commits

Reviewing files that changed from the base of the PR and between 437d1e3 and a2dd651.

📒 Files selected for processing (2)
  • adapter/redis_bzpopmin_wake_test.go
  • adapter/redis_options_test.go
📝 Walkthrough

Walkthrough

Redisブロッキングコマンドのフォールバック待機間隔をRedisServerごとに設定可能にし、XREAD・BZPOPMINの待機処理へ伝播させた。既定値と不正値の扱いを追加し、テスト設定および関連コメントを更新した。

Changes

Redisブロック待機フォールバック

Layer / File(s) Summary
フォールバック間隔の設定契約
adapter/redis.go, adapter/redis_compat_commands.go
既定値250msとRedisServerの設定フィールドを追加し、WithRedisBlockWaitFallbackで正の間隔を設定できるようにした。旧固定定数を削除した。
待機ループへの間隔伝播
adapter/redis_zset_cmds.go, adapter/redis_stream_cmds.go
BZPOPMINとXREADから設定値を待機ヘルパへ渡し、未指定時は既定値、BLOCK残り時間が短い場合はその残り時間を使用するようにした。
テスト設定と説明の更新
adapter/test_util.go, adapter/redis_key_waiters_test.go, adapter/redis_bzpopmin_wake_test.go, adapter/redis_compat_commands_stream_test.go, adapter/redis_compat_helpers.go
テスト用間隔を100msに設定し、呼び出しとフォールバック挙動に関するコメントを構成可能な間隔に合わせて更新した。

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RedisServer
  participant bzpopminWaitLoop
  participant waitForBlockedCommandUpdate
  participant BLOCKDeadline
  RedisServer->>bzpopminWaitLoop: blockWaitFallback
  bzpopminWaitLoop->>waitForBlockedCommandUpdate: fallbackInterval
  waitForBlockedCommandUpdate->>BLOCKDeadline: 残り時間で上限調整
  waitForBlockedCommandUpdate-->>bzpopminWaitLoop: 待機結果
Loading

Possibly related PRs

  • bootjp/elastickv#683: BZPOPMINのfastモードとwaitForBlockedCommandUpdateによるフォールバック処理に関連する変更。
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed Redis のブロッキング待機でのスキャン負荷を下げる変更内容を適切に要約しています。

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6edca89733

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread adapter/redis_compat_commands.go Outdated
// only a safety net for missed signals, wrong-type detection, and the
// BLOCK deadline. Keep it coarse enough that many idle blocking commands
// do not turn into a constant stream of full key-type scans.
redisBlockWaitFallback = time.Second

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve sub-second fallback wakes

For waiters that depend on the fallback rather than an in-process Signal (for example Lua XADD in the default NewRedisServer/test wiring, or any direct mutation path this safety net is meant to cover), raising this to 1s makes any sub-second BLOCK sleep until its deadline because waitForBlockedCommandUpdate caps the timer to the remaining block window. xreadBusyPoll then checks iterTimeout <= 0 before running another xreadOnce, so an entry committed during a 500ms XREAD BLOCK can still return redis.Nil instead of the new stream entry. Consider keeping the fallback below common sub-second block budgets or making every bypass path signal the waiter.

Useful? React with 👍 / 👎.

Comment thread adapter/redis_compat_commands.go Outdated
// only a safety net for missed signals, wrong-type detection, and the
// BLOCK deadline. Keep it coarse enough that many idle blocking commands
// do not turn into a constant stream of full key-type scans.
redisBlockWaitFallback = time.Second

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep BZPOPMIN wrong-type probes below short timeouts

When a multi-key BZPOPMIN is kept awake by fast zset signals on one key, bzpopminWaitLoop uses this same interval as the maximum time it may keep calling keyTypeAtExpectFast and skipping wrong-type detection. With the interval now at 1s, a sub-second block such as 500ms can time out with redis.Nil if another registered key is changed to a string during the wait, because the forced full probe is not re-armed before the deadline; keeping the wrong-type full-check cadence separate from the coarse idle fallback would preserve the existing WRONGTYPE behavior without restoring every idle scan.

Useful? React with 👍 / 👎.

@bootjp

bootjp commented Jul 22, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
adapter/redis.go (1)

343-350: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

設定オプションの契約をテーブル駆動テストで固定してください。

既定値、正値による上書き、0・負値では既定値を維持する挙動は、新しい公開オプションの重要な契約です。*_test.go に境界値を含むテーブル駆動テストを追加してください。

As per coding guidelines、Goの単体テストは *_test.go に同居させ、テーブル駆動テストを優先してください。

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@adapter/redis.go` around lines 343 - 350, Add a table-driven unit test in a
*_test.go file for WithRedisBlockWaitFallback, covering the default value, a
positive duration override, zero, and negative durations. Apply each option to a
RedisServer and assert that positive input changes blockWaitFallback while zero
and negative input preserve the default.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@adapter/redis_bzpopmin_wake_test.go`:
- Around line 177-180: Remove the unmatched closing parenthesis from the comment
in the wake-loop test, changing “previous one)” to “previous one” while leaving
the test logic unchanged.

---

Nitpick comments:
In `@adapter/redis.go`:
- Around line 343-350: Add a table-driven unit test in a *_test.go file for
WithRedisBlockWaitFallback, covering the default value, a positive duration
override, zero, and negative durations. Apply each option to a RedisServer and
assert that positive input changes blockWaitFallback while zero and negative
input preserve the default.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6923d43c-3859-4c17-b7a1-3be152fd13c1

📥 Commits

Reviewing files that changed from the base of the PR and between 8284327 and 437d1e3.

📒 Files selected for processing (9)
  • adapter/redis.go
  • adapter/redis_bzpopmin_wake_test.go
  • adapter/redis_compat_commands.go
  • adapter/redis_compat_commands_stream_test.go
  • adapter/redis_compat_helpers.go
  • adapter/redis_key_waiters_test.go
  • adapter/redis_stream_cmds.go
  • adapter/redis_zset_cmds.go
  • adapter/test_util.go

Comment thread adapter/redis_bzpopmin_wake_test.go Outdated
@bootjp

bootjp commented Jul 22, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: a2dd651e6a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@bootjp
bootjp merged commit 9c5d497 into main Jul 22, 2026
9 checks passed
@bootjp
bootjp deleted the feature/redis-block-wait-fallback branch July 22, 2026 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant