[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs by jmagman · Pull Request #7430 · flutter/packages

jmagman · 2024-08-16T17:34:33Z

Native Xcode tests will output a helpful "xcresult" package on failure containing logs, screenshots, and screen recordings. Zip and upload these results when tests fail.
Pass flutter test --debug-logs-dir flag to upload logs like Upload DerivedData logs in CI flutter#142643.
Pass flutter drive --screenshot flag to upload screenshots on timeout like Take drive screenshot on test failure before app is stopped flutter#96973.

Example of failing Xcode analyzer build has the zipped xcresult attached as a log.

The unzipped xcresult looks like this in Xcode:

A failing "native test" step build:

Fixes flutter/flutter#144795

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

stuartmorgan-g · 2024-10-08T19:40:57Z

From triage: What's the status of this? Is it still something we plan to deploy? It would definitely be useful for debugging!

jmagman · 2024-10-24T23:58:18Z

From triage: What's the status of this? Is it still something we plan to deploy? It would definitely be useful for debugging!

Sorry, I lost track of this. I need to page back in why the tests are failing.

This reverts commit d329678b5be29c3c03ac0d3e9dc18d3bc0d8d154.

jmagman · 2024-10-29T17:04:48Z

@stuartmorgan this is ready to review, when you get a chance.

stuartmorgan-g

LGTM

Is the FLUTTER_LOGS_DIR support low-level enough that it already applies to this repo, or do we need recipe changes too?

script/tool/lib/src/common/xcode.dart

stuartmorgan-g · 2024-10-31T14:22:39Z

script/tool/lib/src/drive_examples_command.dart

+      Directory? screenshotDirectory;
+      if (logsDirectory != null) {
+        screenshotDirectory = logsDirectory.childDirectory('$exampleName-drive');
+      }


Directory? screenshotDirectory = logsDirectory?.childDirectory(...);

Oops, I copied code that was written before null safety and clumsily migrated (likely by me).

stuartmorgan-g · 2024-10-31T14:26:52Z

script/tool/lib/src/drive_examples_command.dart

    for (final File target in targets) {
+      Directory? screenshotDirectory;
+      if (logsDirectory != null) {
+        screenshotDirectory = logsDirectory.childDirectory('$exampleName-drive');


exampleName is a relative posix path; IIRC childDirectory will interpret that as a path and actually make intervening directories. Is that what we want? If not we should replace / with _ or something before this.

stuartmorgan-g · 2024-10-31T14:30:24Z

script/tool/lib/src/drive_examples_command.dart

    required List<File> testFiles,
  }) async {
    final String enableExperiment = getStringArg(kEnableExperiment);
+    final Directory? logsDirectory = testFiles.isNotEmpty ? ciLogsDirectory(platform, testFiles.first.fileSystem) : null;


Why is this gated on testFiles not being empty? (I'd have to check but I wouldn't think this would ever be called with an empty test list.)

I think I was just guarding the .first, I'll remove since I think you're right it's never called with an empty list.

jmagman · 2024-11-04T19:51:12Z

Is the FLUTTER_LOGS_DIR support low-level enough that it already applies to this repo, or do we need recipe changes too?

It already applies as of https://flutter-review.googlesource.com/c/recipes/+/39703

You can see it in the env:
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8732173142577412545/+/u/Run_package_tests/xcode_analyze_deprecation/l_execution_details

…ve and integration_test runs (flutter/packages#7430)

flutter/packages@7219431...bb5a258 2024-11-06 magder@google.com [ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs (flutter/packages#7430) 2024-11-05 737941+loic-sharma@users.noreply.github.com Remove use_modular_headers! from Podfiles (flutter/packages#7796) 2024-11-05 30872003+misos1@users.noreply.github.com [camera_avfoundation] enable more than 30 fps (flutter/packages#7394) 2024-11-05 engine-flutter-autoroll@skia.org Roll Flutter from 8591d0c to 29d40f7 (25 revisions) (flutter/packages#8027) 2024-11-05 stuartmorgan@google.com [ci] Add vector_graphics and flutter_svg to autolabeler (flutter/packages#8025) 2024-11-05 alex@mariuti.com [vector_graphics_compiler] wasm compatibility (flutter/packages#8021) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

* main: (1187 commits) [various] Update example app minSdkVersions (flutter#8035) [go_router] Activate leak testing (flutter#7546) [in_app_purchase_storekit] Add restore purchases and receipts (flutter#7964) [interactive_media_ads] Adds remaining methods for internal wrapper of the Android native `BaseManager` (flutter#7943) [google_sign_in/google_identity_services] Clear-up documentation of callbacks in various APIs and uses of those APIs (flutter#8029) [flutter_svg] wasm compatibility (flutter#8014) Applied Gradle Plugins Declaratively for Multiple Plugin Example Apps (Part 2) (flutter#8019) Roll Flutter from 29d40f7 to 73546b3 (20 revisions) (flutter#8028) [ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs (flutter#7430) Remove use_modular_headers! from Podfiles (flutter#7796) [camera_avfoundation] enable more than 30 fps (flutter#7394) Roll Flutter from 8591d0c to 29d40f7 (25 revisions) (flutter#8027) [ci] Add vector_graphics and flutter_svg to autolabeler (flutter#8025) [vector_graphics_compiler] wasm compatibility (flutter#8021) [vector_graphics*] Relax dependency constraints of vector_graphics, vector_graphics_codec, vector_graphics_compiler, flutter_svg (flutter#8018) [various] Add `missing_code_block_language_in_doc_comment` lint to flutter/packages. (flutter#6473) [various] Update example apps to Kotlin 1.9.0 (flutter#7998) [go_router] add current state getter (flutter#7651) Applied Gradle Plugins Declaratively for Multiple Plugin Example Apps (flutter#7968) Roll Flutter from f86b777 to 8591d0c (16 revisions) (flutter#8015) ... # Conflicts: # packages/quick_actions/quick_actions/CHANGELOG.md # packages/quick_actions/quick_actions_ios/CHANGELOG.md # packages/quick_actions/quick_actions_platform_interface/CHANGELOG.md

jmagman marked this pull request as ready for review October 25, 2024 16:11

jmagman marked this pull request as draft October 25, 2024 16:11

jmagman force-pushed the screenshot-drive branch from 2a21d04 to d329678 Compare October 25, 2024 16:48

github-actions bot added platform-ios platform-macos p: video_player labels Oct 25, 2024

jmagman added 3 commits October 25, 2024 10:22

Add screenshot functionality to drive

f39f734

test failures

8ac52ff

Revert "test failures"

54003a8

This reverts commit d329678b5be29c3c03ac0d3e9dc18d3bc0d8d154.

jmagman force-pushed the screenshot-drive branch from d329678 to 54003a8 Compare October 25, 2024 17:23

github-actions bot removed platform-ios platform-macos p: video_player labels Oct 25, 2024

jmagman changed the title ~~Upload screenshots and Xcode test results for drive and integration_test runs~~ [ci] Upload screenshots and Xcode test results for drive and integration_test runs Oct 25, 2024

jmagman self-assigned this Oct 25, 2024

jmagman marked this pull request as ready for review October 25, 2024 17:24

jmagman changed the title ~~[ci] Upload screenshots and Xcode test results for drive and integration_test runs~~ [ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs Oct 25, 2024

jmagman requested a review from stuartmorgan-g October 25, 2024 21:10

stuartmorgan-g approved these changes Oct 31, 2024

View reviewed changes

jmagman added 5 commits November 4, 2024 21:34

Review edits

937765c

path separator

7e6e02c

deletesync

885a407

Merge branch 'main' into screenshot-drive

5719da2

New formatter

a45f634

jmagman added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 6, 2024

auto-submit bot merged commit bb5a258 into flutter:main Nov 6, 2024

jmagman deleted the screenshot-drive branch November 6, 2024 06:58

engine-flutter-autoroll mentioned this pull request Nov 6, 2024

Roll Packages from 721943194945 to bb5a25815cae (6 revisions) flutter/flutter#158267

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 6, 2024

bb5a258 [ci] Upload screenshots, logs, and Xcode test results for dri…

16cad07

…ve and integration_test runs (flutter/packages#7430)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs#7430

[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs#7430
auto-submit[bot] merged 8 commits intoflutter:mainfrom
jmagman:screenshot-drive

jmagman commented Aug 16, 2024 •

edited

Loading

Uh oh!

stuartmorgan-g commented Oct 8, 2024

Uh oh!

jmagman commented Oct 24, 2024

Uh oh!

jmagman commented Oct 29, 2024

Uh oh!

stuartmorgan-g left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stuartmorgan-g Oct 31, 2024

Uh oh!

jmagman Nov 5, 2024

Uh oh!

stuartmorgan-g Oct 31, 2024

Uh oh!

jmagman Nov 6, 2024

Uh oh!

stuartmorgan-g Oct 31, 2024

Uh oh!

jmagman Nov 5, 2024

Uh oh!

jmagman commented Nov 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jmagman commented Aug 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

stuartmorgan-g commented Oct 8, 2024

Uh oh!

jmagman commented Oct 24, 2024

Uh oh!

jmagman commented Oct 29, 2024

Uh oh!

stuartmorgan-g left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stuartmorgan-g Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

jmagman Nov 5, 2024

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-g Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

jmagman Nov 6, 2024

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-g Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

jmagman Nov 5, 2024

Choose a reason for hiding this comment

Uh oh!

jmagman commented Nov 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jmagman commented Aug 16, 2024 •

edited

Loading