perf: parallelize investment scanning with bounded concurrency#951
Open
dangershony wants to merge 1 commit into
Open
perf: parallelize investment scanning with bounded concurrency#951dangershony wants to merge 1 commit into
dangershony wants to merge 1 commit into
Conversation
Split out of #949 for focused review. - ScanDynamicTypeInvestments: collect spent-fund work items and run them in parallel instead of O(funders x stages) sequential awaits - this dominated the 'Syncing funding data...' time - GetProjectInvestmentsTransactionsAsync: fetch tx hex + info concurrently - All scan fan-outs now go through RunBoundedAsync (SemaphoreSlim capped at 12 concurrent indexer round-trips), including the two pre-existing unbounded Task.WhenAll spots. Projects can have hundreds of investors (x stages for Fund projects); unbounded WhenAll could fire thousands of simultaneous HTTP requests - Add unit test asserting max in-flight indexer requests stays within the bound while still overlapping round-trips
dangershony
commented
Jul 25, 2026
| foreach (var stage in stageDataList) | ||
| { | ||
| var tasks = investmentsResult.Value.Select(tuple => | ||
| var taskFactories = investmentsResult.Value.Select(tuple => |
Member
Author
There was a problem hiding this comment.
the stage.StageIndex + 2 is suspicious here (needs more investigation)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #949 (per review discussion on the
ProjectInvestmentsServicechange) so the scan-perf work gets focused review.Problem
ScanDynamicTypeInvestmentsawaited each spent-fund check sequentially — O(funders × stages) indexer round-trips dominated the "Syncing funding data…" time. But naively parallelizing with unboundedTask.WhenAll(as originally proposed in #949) is risky: projects can have hundreds of investors, and Fund projects multiply that by stage count — thousands of simultaneous HTTP requests would risk socket exhaustion and indexer rate limiting.Changes
ScanDynamicTypeInvestments: collect all stage work items first, then run spent-fund checks in parallel with bounded concurrency (RunBoundedAsync, SemaphoreSlim capped at 12 concurrent indexer round-trips). Ordering is preserved (work-item index), so date-bucket grouping is byte-for-byte identical to the sequential version.GetProjectInvestmentsTransactionsAsync: fetch tx hex + tx info concurrently per investment; the per-investment fan-out (previously already unboundedTask.WhenAllover all investors) now also goes through the bound.ScanInvestTypeInvestments: the pre-existing unbounded per-stageTask.WhenAllis bounded too.Bounded parallelism captures nearly all the latency win (it comes from overlapping round-trips, not N-way parallelism) with none of the failure risk at scale.
Testing
ScanFullInvestments_ManyFundInvestors_LimitsConcurrentIndexerRequests: 30 Fund investors × 3 stages with instrumented mocks tracking max in-flight indexer calls — asserts overlap happens (>1) and never exceeds 2 × cap (hex+info pairs share a slot).FundInvestorsWithShiftedSchedules,FundMixedBucket) unchanged and green — confirms functional equivalence of the restructured loop.Suggested UAT validation
MultiFundClaimAndRecoverTest(dynamic scan end-to-end incl. spent-type discrimination) andMultiInvestClaimAndRecoverTest(Invest path + FundersView).