Desktop deploys are flaky

[roryabraham]

HOLDing for new release of electron-builder, which can be monitored at electron-userland/electron-builder#7592

Problem

We've seen a lot of flakiness with desktop deploys lately. Here's an example. They have all been failing in the same way:

 • electron-builder  version=24.4.0 os=21.6.0
  • loaded configuration  file=/Users/runner/work/App/App/config/electronBuilder.config.js
  • executing @electron/rebuild  arch=x64 version=22.3.7 appDir=/Users/runner/work/App/App/desktop
  • packaging       platform=darwin arch=x64 electron=22.3.7 appOutDir=desktop-build/mac
  • downloading     url=https://github.com/electron/electron/releases/download/v22.3.7/electron-v22.3.7-darwin-x64.zip size=92 MB parts=8
  • downloaded      url=https://github.com/electron/electron/releases/download/v22.3.7/electron-v22.3.7-darwin-x64.zip duration=2.91s
  • signing         file=desktop-build/mac/New Expensify.app identityName=Developer ID Application: Expensify, Inc. (368M544MTT) identityHash=0633A09738B5C8B1FF97B81A951198CBB50B87A2 provisioningProfile=none
  • notarization successful
  • building        target=macOS zip arch=x64 file=desktop-build/New Expensify-1.3.24-4-mac.zip
  • building        target=DMG arch=x64 file=desktop-build/NewExpensify.dmg
  • Above command failed, retrying 5 more times
  • building block map  blockMapFile=desktop-build/New Expensify-1.3.24-4-mac.zip.blockmap
  • publishing      publisher=s3 (bucket: staging-expensify-cash)
  • uploading       file=New Expensify-1.3.24-4-mac.zip.blockmap provider=s3
  • uploading       file=New Expensify-1.3.24-4-mac.zip provider=s3
  ⨯ Exit code: 16. Command failed: hdiutil detach -force /dev/disk3
hdiutil: couldn't eject "disk3" - Resource busy

hdiutil: couldn't eject "disk3" - Resource busy
  failedTask=build stackTrace=Error: Exit code: 16. Command failed: hdiutil detach -force /dev/disk3
hdiutil: couldn't eject "disk3" - Resource busy
                                                                                                                                                                         hdiutil: couldn't eject "disk3" - Resource busy
                                                                                                                                                                             at /Users/runner/work/App/App/node_modules/builder-util/src/util.ts:133:18
    at ChildProcess.exithandler (node:child_process:406:5)
    at ChildProcess.emit (node:events:527:28)
    at maybeClose (node:internal/child_process:1092:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

Solution

TBD. Some ideas are:

• Create a GitHub support ticket. A lot of the advice for this issue is just restart your computer
• Update electron-builder to add more retries wrapping this call to hdiutil detach. We just need to figure out which call is failing for us and verify it's not already wrapped in 5 retries.
• Update electron-builder to implement a backoff mechanism for retries rather than just waiting 1 second between retries.
• Start using macos-13 runners?
• Something else / make this external and ask for help?

[melvin-bot]

Triggered auto assignment to @jliexpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[roryabraham]

Going to start by creating a GitHub Support ticket.

[roryabraham]

GitHub Support Ticket created here

[roryabraham]

Created a PR in electron-builder, but was struggling to figure out how to build that monorepo correctly in order to use patch-package.

Going to pivot to trying the macos-13 runner

[roryabraham]

Currently have two PRs ready for review to try and fix this:

• [No QA] Use macos-13 runners for desktop builds #20291
• [No QA] Patch electron-builder submodules to implement retry in hdiutils detach #20292

[roryabraham]

macos-13 beta runners weren't available, going to revert that PR for now.