feat(nimbus): Include ration in the calculation for the SRM ratios alert#16011
Open
yashikakhurana wants to merge 8 commits into
Open
feat(nimbus): Include ration in the calculation for the SRM ratios alert#16011yashikakhurana wants to merge 8 commits into
yashikakhurana wants to merge 8 commits into
Conversation
added 3 commits
June 17, 2026 15:21
…itives on unequal splits
added 2 commits
June 17, 2026 15:49
…itives on unequal splits
…e positives with large populations
Contributor
Author
|
@jaredlockhart Fixed two things that were causing false SRM alerts:
|
jaredlockhart
previously requested changes
Jun 18, 2026
jaredlockhart
left a comment
Collaborator
There was a problem hiding this comment.
Oh yes good, taking the designed ratio into account good catch! And actually yeah checking telemetry shows that 5% is probably exactly what we want. One thing though I think this only updates slack, don't we also need to pass the ratios in here:
To fix it in the UI?
The UI path (monitoring_summary) called check_srm_mismatch without the configured branch ratios, so experiments with intentional unequal splits (e.g. 95/5) still showed false SRM mismatches in the UI even after the Slack path was fixed. Pass the already-in-scope branch_ratios so the chi-square test compares against the configured distribution.
Contributor
Author
ahh very good catch @jaredlockhart |
Both callers (Slack alerts and monitoring_summary) always pass branch
ratios now, so drop the branch_ratios=None defaults on compute_srm_p_value,
compute_max_ratio_deviation, and check_srm_mismatch. Tests that exercised
the equal-ratio fallback pass {} explicitly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Because
This commit
Fixes #16010