Skip to content

ci(maestro): rebalance iOS shards and isolate driver warmup from test timing#7394

Closed
diegolmello wants to merge 1 commit into
developfrom
smiling-modem
Closed

ci(maestro): rebalance iOS shards and isolate driver warmup from test timing#7394
diegolmello wants to merge 1 commit into
developfrom
smiling-modem

Conversation

@diegolmello

@diegolmello diegolmello commented Jun 12, 2026

Copy link
Copy Markdown
Member

Proposed changes

iOS Maestro shard 12 carried ~35 minutes of flow time, exceeding the step's 30-minute timeout: the Room flow was killed mid-run and Share Message never started. This PR rebalances the shard assignments using measured per-flow durations from a full develop run and cleans up the iOS Maestro workflow:

Shard rebalance (heaviest iOS shard goes from ~35 min to ~15 min):

  • E2E Encryption moves to shard 3 and gets it to itself — it is the heaviest single flow on both platforms (~13 min). Server History moves to shard 1; Valid/Invalid Workspace move to shard 2.
  • Room Info, Message Markdown Click and Share Message move from shard 12 to shard 9; Threads moves from 12 to 13. Shard 12 keeps Room + Open room with 50+ messages.
  • Delete Server moves from shard 6 to 7; Display Perf from 6 to 8.

maestro-ios.yml changes:

  • Replace nick-fields/retry with a plain timeout-minutes: 30 on the test step. run-maestro.sh already reruns flaky flows internally, so the action's only remaining effect was a second full 30-minute attempt after a timeout — doubling runner cost on oversized shards. The Android workflow already works this way.
  • Add a warmup step that installs Maestro's XCTest driver onto the booted simulator (~3 min on a cold simulator) before the timed test step, so the test step's duration reflects flow execution only.
  • Boot the simulator asynchronously while Maestro and idb install, then block on boot status, instead of waiting serially.

Flow-to-shard mapping was verified by replaying run-maestro.sh's tag grep for all 14 shards — every flow is assigned exactly once, none dropped. Room and Share Message iOS durations are estimated (they never completed on iOS recently), so shards 9 and 12 carry the most uncertainty.

A follow-up task exists to replace these static test-N tags with duration-aware auto-balancing per platform: https://rocketchat.atlassian.net/browse/NATIVE-1265

Issue(s)

https://rocketchat.atlassian.net/browse/NATIVE-1123

How to test or reproduce

Approve the E2E hold on this PR and check the Maestro iOS/Android matrix: all 14 shards should pass, no shard's "Run Maestro Tests" step should approach the 30-minute timeout, and the new "Warm Up Maestro Driver" step should absorb the ~3-minute driver install on iOS.

Screenshots

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • Improvement (non-breaking change which improves a current function)
  • New feature (non-breaking change which adds functionality)
  • Documentation update (if none of the other choices apply)

Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA
  • Lint and unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works (if applicable)
  • I have added necessary documentation (if applicable)
  • Any dependent changes have been merged and published in downstream modules

Further comments

Summary by CodeRabbit

  • Tests
    • Enhanced iOS test automation workflow by optimizing simulator startup sequence with asynchronous boot and parallel dependency installation, reducing CI/CD pipeline duration.
    • Reorganized test categorization and updated test tags across Maestro test suites for improved test management and traceability.

… timing

Shard 12 exceeded its 30-minute timeout on iOS (~35 min of flow time),
killing the Room flow mid-run and never starting Share Message. Rebalance
flows across shards using measured per-flow durations from a full develop
run, bringing the heaviest iOS shard down to ~15 min:

- E2EE gets shard 3 to itself (heaviest single flow on both platforms);
  its former shard-3 neighbors move to shards 1 and 2
- Room Info, Markdown Click and Share Message move from shard 12 to 9;
  Threads moves to 13, leaving shard 12 with Room + Open room 50+
- Delete Server and Display Perf split from shard 6 onto 7 and 8

Also in maestro-ios.yml:
- Replace nick-fields/retry with a plain 30-minute step timeout. The
  run-maestro.sh script already reruns flaky flows internally, so the
  action's only effect was a second full 30-minute attempt after a
  timeout, doubling runner cost on oversized shards.
- Add a warmup step that installs Maestro's XCTest driver (~3 min on a
  cold simulator) before the timed test step, so the test step measures
  flow execution only.
- Boot the simulator asynchronously while Maestro and idb install,
  blocking on readiness afterwards.
@diegolmello diegolmello requested a deployment to approve_e2e_testing June 12, 2026 18:27 — with GitHub Actions Waiting
@coderabbitai

coderabbitai Bot commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

Pull request was closed or merged during review

Walkthrough

This PR restructures the iOS Maestro GitHub Actions workflow to parallelize simulator boot and dependency installation, adds an XCTest driver warm-up phase, and removes retry wrapping from test invocation. It also updates test tag assignments across nine Maestro test files to reorganize test categorization.

Changes

iOS Maestro CI Optimization

Layer / File(s) Summary
Simulator boot and Maestro warm-up flow
.github/workflows/maestro-ios.yml
Splits simulator startup into asynchronous boot and synchronous wait-for-ready steps, allowing parallel Maestro/idb installation during boot. Adds a "Warm Up Maestro Driver" step that runs a temporary XCTest warm-up before invoking the main Maestro test script directly without retry wrapper.

Maestro Test Tag Updates

Layer / File(s) Summary
Test tag metadata updates
.maestro/tests/assorted/*, .maestro/tests/e2ee/*, .maestro/tests/onboarding/*, .maestro/tests/room/*
Updates tags entries across nine test files: delete-server.yaml (test-6 → test-7), display-perf.yaml (test-6 → test-8), e2e-encryption.yaml (test-9 → test-3), server-history.yaml (test-3 → test-1), invalid-workspace.yaml (test-3 → test-2), valid-workspace.yaml (test-3 → test-2), message-markdown-click.yaml (test-12 → test-9), room-info.yaml (test-12 → test-9), share-message.yaml (test-12 → test-9), threads.yaml (test-12 → test-13).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

type: chore

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main changes: CI workflow optimization through iOS shard rebalancing and driver warmup isolation from test timing.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

Warning

Review ran into problems

🔥 Problems

Errors were encountered while retrieving linked issues.

Errors (2)
  • NATIVE-1265: Request failed with status code 401
  • NATIVE-1123: Request failed with status code 401

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant