[No QA] Fix leader-election deadlock in ActiveClientManager (SequentialQueue unable to flush) by elirangoshen · Pull Request #95148 · Expensify/App

elirangoshen · 2026-07-02T09:21:31Z

Explanation of Change

The web/PWA app could intermittently become unresponsive after focusing the window: the Inbox loading bar loops forever, no new data arrives, and only a full sign-out/sign-in recovers it (Clear cache and restart does not help).

Root cause is a leader-election deadlock in ActiveClientManager. Leadership is decided purely by "last GUID in the shared ACTIVE_CLIENTS list wins". The only cleanup path is a beforeunload handler, which doesn't fire on crashes, force-quit, or mobile/PWA background kills — so a stale/ghost GUID from a dead tab can remain in the persisted list. When that ghost sits last, the only live tab believes it is not the leader, so SequentialQueue.flush() bails and OpenApp/ReconnectApp/PusherPing pile up forever with no leader to flush them. ACTIVE_CLIENTS is persisted, which is why cache-clear doesn't recover but sign-in (which resets the list) does.

This PR fixes it by never dropping the live client (ActiveClientManager): a late disk-hydration event on boot can overwrite the list and strip the live client out (the Safari setTimeout(init, 400) widens this window). The ACTIVE_CLIENTS connect callback now re-appends our own GUID if it's missing while we're still alive, so a ghost can't take over leadership from us on startup.

Fixed Issues

$ #94963
PROPOSAL: N/A — internal bug fix.

Tests

Primary validation is the automated tests. The production bug is an intermittent lifecycle/timing race, and it can't be held open by hand: the fix self-heals on the next event, and ActiveClientManager.init() re-runs on focus/visibility changes (Expensify.tsx), which re-adds this client and masks any injected stuck state. The unit tests use deterministic control (direct state injection) to exercise both the deadlock condition and the recovery.

Run:

npx jest tests/unit/ActiveClientManagerTest.ts

tests/unit/ActiveClientManagerTest.ts — leader registration; the boot-race re-add guard (a stale hydration that drops the live client is repaired).

Best-effort manual check (web) — note it may self-heal before you observe anything, which is itself the fix working:

npm run web, open the dev URL in Chrome with DevTools, sign in.
Closest-to-real trigger: open a second tab (it becomes leader), then kill that tab's renderer via Chrome → Window → Task Manager → select the tab → End process (this skips beforeunload, leaving a ghost GUID holding leadership).
From a different account, send a message to that chat. Expected: it appears in the first tab — the stuck client re-adds itself, stays leader, and the queue drains (sending from the stuck tab already worked; the regression was that it stopped receiving updates).
Requests should not permanently stack in OpenApp/ReconnectApp/PusherPing.

Verify that no errors appear in the JS console

Offline tests

No new offline behavior. The change only affects which tab flushes the shared request queue; offline queuing/replay is unchanged. Spot-check: toggle Offline in DevTools, send a message (queues optimistically), toggle back online, confirm it sends and no white screen/stale UI.

Verify that no errors appear in the JS console

QA Steps

Same as tests. Focus on the recovery behavior: an app that previously stuck with a looping loading bar after focusing the window should now recover on its own.

Verify that no errors appear in the JS console

PR Author Checklist

Screenshots/Videos

Android: Native

N/A — web/PWA leader-election path; no native code changes (native ActiveClientManager is a no-op).

Android: mWeb Chrome

iOS: Native

N/A — web/PWA leader-election path; no native code changes (native ActiveClientManager is a no-op).

iOS: mWeb Safari

MacOS: Chrome / Safari

A stale/ghost client GUID left in ACTIVE_CLIENTS (from an unclean shutdown) could hold leadership, leaving the only live tab unable to flush the SequentialQueue so all requests pile up indefinitely. - Re-append this client to ACTIVE_CLIENTS if a late hydration event drops it, so a ghost GUID can't strip the live tab out of the list on boot. - Add a self-promotion safety net: if the queue is stuck as a non-leader with queued work past a timeout, claim leadership and flush. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

melvin-bot · 2026-07-02T09:33:01Z

@thesahindia Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button]

elirangoshen · 2026-07-02T09:33:07Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 17bd5ad095

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-02T09:38:41Z

+            return;
+        }
+
+        const hasWork = getAllPersistedRequests().length > 0 || !!getPersistedOngoingRequest();


Avoid promoting over a live ongoing request

With two tabs, the follower schedules this timer when it sees queued work; if the current leader has already moved the write into PERSISTED_ONGOING_REQUESTS but the API call takes longer than 30 seconds (for example a large upload or slow backend), this getPersistedOngoingRequest() branch still marks the queue as stuck and promotes the follower. processNextPersistedRequest() returns the same ongoing request, so the follower can issue a duplicate write while the original leader is still in flight.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-02T09:38:41Z

+        if (hasInitialized && !isLeavingTab && !activeClients.includes(clientID)) {
+            activeClients.push(clientID);


Don't re-add clients pruned by the cap

When the active-client list is at the 20-client cap, the trimming loop above evicts a live client by design; this new unconditional re-add makes that evicted tab immediately append itself again. With 20+ open tabs, the next callback trims the list again and clients can keep rewriting ACTIVE_CLIENTS and rotating leadership instead of settling, so the recovery path should avoid re-adding IDs that were intentionally pruned by the max-client limit.

Useful? React with 👍 / 👎.

- Self-promote only when there are pending requests and no ongoing request, so a live leader's in-flight request is never re-issued as a duplicate. - Re-add this client to ACTIVE_CLIENTS only when a stale write dropped it, not when the max-client cap trim evicted it, avoiding leadership thrash with 20+ tabs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

mountiny

Overall, I'm not sure if I have any concerns, but I feel like the PR should be marked as no QA because I don't think the testers will be able to test it really.

elirangoshen · 2026-07-02T14:43:49Z

Overall, I'm not sure if I have any concerns, but I feel like the PR should be marked as no QA because I don't think the testers will be able to test it really.

done

mountiny · 2026-07-02T15:10:23Z

@thesahindia Can you test the PR

github-actions · 2026-07-02T15:13:12Z

🚧 mountiny has triggered a test Expensify/App build. You can view the workflow run here.

trjExpensify · 2026-07-02T16:18:16Z

PR doesn’t need product input as a refactor PR. Unassigning and unsubscribing myself.

# Conflicts: # src/libs/ActiveClientManager/index.ts

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

codecov · 2026-07-03T08:33:11Z

Codecov Report

✅ Changes either increased or maintained existing code coverage, great job!

Files with missing lines	Coverage Δ
src/libs/ActiveClientManager/index.ts	`76.47% <83.33%> (+76.47%)`	⬆️
... and 300 files with indirect coverage changes

adhorodyski · 2026-07-03T11:09:01Z

Self-promotion safety net (SequentialQueue): if the queue is stuck (non-empty, no ongoing request, not the leader) for longer than STUCK_QUEUE_LEADER_PROMOTION_TIMEOUT_MS (30s) while online and not paused, the tab logs an alert, promotes itself to leader, and flushes. The threshold is deliberately generous so a legitimately busy leader in another tab flushes well before it fires, keeping duplicate-request risk near zero.

I'd push back on this, I don't think this is anything close to reliable as we're only depending on a 30s timer, nothing deterministic.

adhorodyski · 2026-07-03T11:13:31Z

A durable follow-up (a BroadcastChannel liveness heartbeat to prune ghost GUIDs proactively) is tracked separately.

Where is it being tracked?

…nt guard The 30s self-promotion safety net relied on a non-deterministic timer and risked duplicate writes. Guard #1 (re-appending the live client in the ACTIVE_CLIENTS connect callback) deterministically closes the leader-election deadlock, so drop the timer, promoteToLeader, its constant, and its test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

elirangoshen · 2026-07-03T13:44:35Z

A durable follow-up (a BroadcastChannel liveness heartbeat to prune ghost GUIDs proactively) is tracked separately.

Where is it being tracked?

I removed it as its not relevant after I removed the stuck queue

elirangoshen · 2026-07-03T13:45:01Z

Self-promotion safety net (SequentialQueue): if the queue is stuck (non-empty, no ongoing request, not the leader) for longer than STUCK_QUEUE_LEADER_PROMOTION_TIMEOUT_MS (30s) while online and not paused, the tab logs an alert, promotes itself to leader, and flushes. The threshold is deliberately generous so a legitimately busy leader in another tab flushes well before it fires, keeping duplicate-request risk near zero.

I'd push back on this, I don't think this is anything close to reliable as we're only depending on a 30s timer, nothing deterministic.

I removed this

mountiny · 2026-07-03T15:36:49Z

@thesahindia can you give it a test?

adhorodyski · 2026-07-03T15:42:14Z

Also, for any leader election I think this needs to be reflected in the SEQUENTIAL_QUEUE.md doc.

adhorodyski · 2026-07-03T15:43:53Z

+
        activeClients = val;

        // Remove from the beginning of the list any clients that are past the limit, to avoid having thousands of them


is this comment still relevant after the removed -> changed update?

Nop, I removed it

adhorodyski · 2026-07-03T15:46:44Z

+        // A late disk-hydration event can overwrite the list and drop this live client, handing
+        // leadership to a stale/ghost GUID and stalling the SequentialQueue. Re-append ourselves when a
+        // stale write dropped us (we were never in the incoming list) — but not when the cap trim above
+        // intentionally evicted us, which would otherwise thrash leadership with 20+ open tabs.


I find this part to be very jargon-y, also, is 'late disk-hydration' really the only scenario that can update the list? I think this part would benefit from a much simpler language (or the problem is really that complex?).

Sure I changed that

thesahindia · 2026-07-03T17:44:50Z

@thesahindia can you give it a test?

On it.

thesahindia · 2026-07-04T17:31:50Z

3. Back in the first tab, send a chat message. Expected: it recovers — the client re-adds itself and stays leader, and the queue drains.

I think it should be: Send a message from a different account and verify that it appears in the first tab.

Sending message was working fine but user wasn't able to receive any message/updates. The PR tested well.

Screen.Recording.2026-07-04.at.10.58.41.PM.mov

…n self-heal - Reword the cap-trim comment so it no longer implies `changed` only means "removed" - Simplify the re-add comment: drop disk-hydration jargon, describe any stale write - Document the active-clients self-heal in SEQUENTIAL_QUEUE.md leader-election section Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

elirangoshen · 2026-07-07T11:30:54Z

Also, for any leader election I think this needs to be reflected in the SEQUENTIAL_QUEUE.md doc.

Done

mountiny · 2026-07-07T18:44:48Z

@thesahindia can you prioritize this pr now? thank you!

thesahindia · 2026-07-08T03:39:13Z

Reviewer Checklist

I have verified the author checklist is complete (all boxes are checked off).
I verified the correct issue is linked in the ### Fixed Issues section above
I verified testing steps are clear and they cover the changes made in this PR
- I verified the steps for local testing are in the Tests section
- I verified the steps for Staging and/or Production testing are in the QA steps section
- I verified the steps cover any possible failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
- I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
I checked that screenshots or videos are included for tests on all platforms
I included screenshots or videos for tests on all platforms
I verified that the composer does not automatically focus or open the keyboard on mobile unless explicitly intended. This includes checking that returning the app from the background does not unexpectedly open the keyboard.
I verified tests pass on all platforms & I tested again on:
- Android: HybridApp
- Android: mWeb Chrome
- iOS: HybridApp
- iOS: mWeb Safari
- MacOS: Chrome / Safari
If there are any errors in the console that are unrelated to this PR, I either fixed them (preferred) or linked to where I reported them in Slack
I verified proper code patterns were followed (see Reviewing the code)
- I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick).
- I verified that comments were added to code that is not self explanatory
- I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
- I verified any copy / text that was added to the app is grammatically correct in English. It adheres to proper capitalization guidelines (note: only the first word of header/labels should be capitalized), and is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
I verified that this PR follows the guidelines as stated in the Review Guidelines
I verified other components that can be impacted by these changes have been tested, and I retested again (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar have been tested & I retested again)
If a new component is created I verified that:
- A similar component doesn't exist in the codebase
- All props are defined accurately and each prop has a /** comment above it */
- The file is named correctly
- The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
- The only data being stored in the state is data necessary for rendering and nothing else
- For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
- Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
- All JSX used for rendering exists in the render method
- The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions
If any new file was added I verified that:
- The file has a description of what it does and/or why is needed at the top of the file if the code is not self explanatory
If a new CSS style is added I verified that:
- A similar style doesn't already exist
- The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(theme.componentBG)
If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
- I verified that all the inputs inside a form are aligned with each other.
- I added Design label and/or tagged @Expensify/design so the design team can review the changes.
For any bug fix or new feature in this PR, I verified that sufficient unit tests are included to prevent regressions in this flow.
If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.
I have checked off every checkbox in the PR reviewer checklist, including those that don't apply to this PR.

Screenshots/Videos

Android: HybridApp

Android: mWeb Chrome

iOS: HybridApp

iOS: mWeb Safari

MacOS: Chrome / Safari

Screen.Recording.2026-07-08.at.8.31.28.AM.mov

Screen.Recording.2026-07-04.at.10.58.41.PM.mov

thesahindia

LGTM!

mountiny

Thank you!

OSBotify · 2026-07-08T16:08:23Z

✋ This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

github-actions · 2026-07-08T17:07:31Z

🚧 luacmartins has triggered a test Expensify/App build. You can view the workflow run here.

github-actions · 2026-07-08T17:20:40Z

🧪🧪 Use the links below to test this adhoc build on Android, iOS, and Web. Happy testing! 🧪🧪
Built from App PR #95148.

Android 🤖	iOS 🍎
https://ad-hoc-expensify-cash.s3.us-east-1.amazonaws.com/rock-artifacts/ad-hoc/rock-android-Adhoc-449bb049977-706a9bc-58bace2e8157d984bf483b2c36acb8e0f02f264b/index.html	https://ad-hoc-expensify-cash.s3.us-east-1.amazonaws.com/rock-artifacts/ad-hoc/rock-ios-device-AdHoc-449bb04-706a9bc-631b8f40657b27ed3b062857c68c5b9d03bd8b5a/index.html

Web 🕸️
https://95148.pr-testing.expensify.com

👀 View the workflow run that generated this build 👀

OSBotify · 2026-07-09T11:05:56Z

🚀 Deployed to staging by https://github.com/luacmartins in version: 9.4.32-0 🚀

platform	result
🕸 web 🕸	success ✅
🤖 android 🤖	success ✅
🍎 iOS 🍎	success ✅

MelvinBot · 2026-07-09T11:07:34Z

🤖 No help site changes required — no draft PR created.

I reviewed the changes in this PR against the help site content under App/docs/articles.

Why no docs update is needed: This is a purely internal reliability fix to the client leader-election logic. It has no user-facing surface:

src/libs/ActiveClientManager/index.ts — internal logic that re-appends the live client to ACTIVE_CLIENTS so leadership can't get stuck on a dead tab.
tests/unit/ActiveClientManagerTest.ts — a new unit test.
contributingGuides/SEQUENTIAL_QUEUE.md — internal developer documentation, not a public help-site article.

There is no new feature, UI element, setting, tab, button, or user-facing workflow introduced. The fix simply lets the web/PWA app self-recover from a stuck loading state (looping Inbox loading bar) that previously required a full sign-out/sign-in. That behavior isn't documented on the help site, so nothing there needs to change.

If you believe a specific help article is affected, let me know which one and I'll take another look.

OSBotify · 2026-07-10T13:42:59Z

🚀 Deployed to production by https://github.com/grgia in version: 9.4.32-3 🚀

platform	result
🕸 web 🕸	success ✅
🤖 android 🤖	success ✅
🍎 iOS 🍎	success ✅

Bundle Size Analysis (Sentry):

Android

melvin-bot Bot assigned elirangoshen Jul 2, 2026

elirangoshen marked this pull request as ready for review July 2, 2026 09:32

elirangoshen requested review from a team as code owners July 2, 2026 09:32

melvin-bot Bot requested review from thesahindia and trjExpensify and removed request for a team July 2, 2026 09:32

melvin-bot Bot removed the request for review from a team July 2, 2026 09:33

chatgpt-codex-connector Bot reviewed Jul 2, 2026

View reviewed changes

mountiny reviewed Jul 2, 2026

View reviewed changes

elirangoshen changed the title ~~Fix leader-election deadlock in ActiveClientManager (SequentialQueue unable to flush)~~ [No QA] Fix leader-election deadlock in ActiveClientManager (SequentialQueue unable to flush) Jul 2, 2026

This comment has been minimized.

Sign in to view

trjExpensify removed their request for review July 2, 2026 16:18

elirangoshen and others added 2 commits July 3, 2026 10:23

Merge branch 'main' into eliran/94963-leader-election-deadlock

c6c8001

# Conflicts: # src/libs/ActiveClientManager/index.ts

style: apply oxfmt formatting to PR-changed files

1b651de

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

adhorodyski reviewed Jul 3, 2026

View reviewed changes

thesahindia approved these changes Jul 8, 2026

View reviewed changes

melvin-bot Bot requested review from luacmartins and mountiny July 8, 2026 03:39

mountiny approved these changes Jul 8, 2026

View reviewed changes

luacmartins approved these changes Jul 8, 2026

View reviewed changes

luacmartins merged commit 449bb04 into Expensify:main Jul 8, 2026
36 of 38 checks passed

OSBotify mentioned this pull request Jul 9, 2026

Deploy Checklist: New Expensify 2026-07-09 #95677

Closed

94 tasks

grgia mentioned this pull request Jul 9, 2026

[CP Staging] Revert "[No QA] Fix leader-election deadlock in ActiveClientManager (SequentialQueue unable to flush)" #95718

Merged

melvin-bot Bot mentioned this pull request Jul 10, 2026

[Due for payment 2026-07-17] Web/PWA app becomes unresponsive due to leader-election deadlock in ActiveClientManager (SequentialQueue unable to flush) #94963

Closed

ShridharGoel mentioned this pull request Jul 14, 2026

[$250] [$?] [Bug] Expenses fail to load in workspace expense chat (Inbox) — admin waited 45s with no content #96033

Closed

		if (hasInitialized && !isLeavingTab && !activeClients.includes(clientID)) {
		activeClients.push(clientID);


		activeClients = val;

		// Remove from the beginning of the list any clients that are past the limit, to avoid having thousands of them

Uh oh!

Conversation

elirangoshen commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Explanation of Change

Fixed Issues

Tests

Offline tests

QA Steps

PR Author Checklist

Screenshots/Videos

Uh oh!

melvin-bot Bot commented Jul 2, 2026

Uh oh!

elirangoshen commented Jul 2, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

elirangoshen Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

elirangoshen Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

mountiny left a comment

Choose a reason for hiding this comment

Uh oh!

elirangoshen commented Jul 2, 2026

Uh oh!

mountiny commented Jul 2, 2026

Uh oh!

github-actions Bot commented Jul 2, 2026

Uh oh!

This comment has been minimized.

trjExpensify commented Jul 2, 2026

Uh oh!

codecov Bot commented Jul 3, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

adhorodyski commented Jul 3, 2026

Uh oh!

adhorodyski commented Jul 3, 2026

Uh oh!

elirangoshen commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elirangoshen commented Jul 3, 2026

Uh oh!

mountiny commented Jul 3, 2026

Uh oh!

adhorodyski commented Jul 3, 2026

Uh oh!

adhorodyski Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

elirangoshen Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

adhorodyski Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

elirangoshen Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

thesahindia commented Jul 3, 2026

Uh oh!

thesahindia commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elirangoshen commented Jul 7, 2026

Uh oh!

elirangoshen commented Jul 2, 2026 •

edited

Loading

codecov Bot commented Jul 3, 2026 •

edited by github-actions Bot

Loading

elirangoshen commented Jul 3, 2026 •

edited

Loading

thesahindia commented Jul 4, 2026 •

edited

Loading