Speed up the java_sdk e2e Gradle builds with concurrency and caching#69239
Merged
jason810496 merged 3 commits intoJul 10, 2026
Merged
Conversation
jason810496
marked this pull request as ready for review
July 2, 2026 09:31
jason810496
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jscheffl and
potiuk
as code owners
July 2, 2026 09:31
phanikumv
reviewed
Jul 2, 2026
phanikumv
reviewed
Jul 2, 2026
phanikumv
reviewed
Jul 2, 2026
uranusjr
reviewed
Jul 2, 2026
uranusjr
reviewed
Jul 2, 2026
uranusjr
reviewed
Jul 2, 2026
The example and scala_spark_example bundles are independent Gradle builds that both consume the SDK artifact published to the local Maven repo, so the Scala Spark dependency resolution no longer has to wait for the Java example build to finish before it starts. The bundle containers run with --network=host because Gradle's cross-process lock handover pings the lock owner over loopback, which isolated container network namespaces would break, and each build's output is captured and printed under its own header so concurrent logs do not interleave.
The java_sdk e2e job builds the SDK and its example bundles with Gradle on every canary and java-sdk-touching run, starting from an empty GRADLE_USER_HOME each time. That forces a full re-download of the example bundles' dependency tree (Spark alone is around 1GB) on every run. Reusing the resolved dependencies across runs removes that repeated download. The restore/save split saves the warm-up even when the e2e tests fail, the key hashes the Kotlin-DSL build files and version catalog that pin the SDK dependencies, and runner.arch keeps the arch-specific auto-provisioned JDKs in .gradle/jdks from crossing between amd64 and arm64 runners.
Excludes churny *.lock/journal files from the Gradle cache per GitHub's own caching guidance, makes bundle build output print stdout/stderr symmetrically through Rich with markup disabled so raw Gradle text can't be misparsed as markup, ties the build pool size to the bundle list instead of a hardcoded count, and cites the Gradle lock-handover mechanism the --network=host workaround depends on.
jason810496
force-pushed
the
ci/java-sdk/refactor-e2e-gradle-deps
branch
from
July 9, 2026 10:21
f5be378 to
f0287aa
Compare
phanikumv
approved these changes
Jul 9, 2026
Member
|
#69411 introduced a different approach to caching, we might want to think about this with both in context. This does not block the merge since the two are fundamentally different components and don’t conflict. |
Contributor
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 7a4c059 v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
Member
Author
Yes, I will follow-up to check whether building Java-SDK artifact natively in CI be better (it should but still worthwhile to double check). No need to backport, Lang-SDK lives in main. |
jason810496
added a commit
to jason810496/airflow
that referenced
this pull request
Jul 13, 2026
The java_sdk e2e mode and the lang-SDK k8s test solved the same cold-build problem two different ways (containerized Gradle with a repo-local cache vs host toolchain with salted home-dir caches), which a review follow-up on apache#69239 asked to reconcile. Converge the e2e mode on the apache#69411 native-toolchain convention: CI provisions the JDK with actions/setup-java and caches ~/.gradle, dropping the per-run eclipse-temurin image pull and the --network=host Gradle lock-handover workaround; local runs keep the containerized build so a dev host still needs no JDK.
jason810496
added a commit
to jason810496/airflow
that referenced
this pull request
Jul 13, 2026
The java_sdk e2e mode and the lang-SDK k8s test solved the same cold-build problem two different ways (containerized Gradle with a repo-local cache vs host toolchain with salted home-dir caches), which a review follow-up on apache#69239 asked to reconcile. Converge the e2e mode on the apache#69411 native-toolchain convention: CI provisions the JDK with actions/setup-java and caches ~/.gradle, dropping the per-run eclipse-temurin image pull and the --network=host Gradle lock-handover workaround; local runs keep the containerized build so a dev host still needs no JDK.
1 task
jason810496
added a commit
to jason810496/airflow
that referenced
this pull request
Jul 15, 2026
The java_sdk e2e mode and the lang-SDK k8s test solved the same cold-build problem two different ways (containerized Gradle with a repo-local cache vs host toolchain with salted home-dir caches), which a review follow-up on apache#69239 asked to reconcile. Converge the e2e mode on the apache#69411 native-toolchain convention: CI provisions the JDK with actions/setup-java and caches ~/.gradle, dropping the per-run eclipse-temurin image pull and the --network=host Gradle lock-handover workaround; local runs keep the containerized build so a dev host still needs no JDK.
jason810496
added a commit
that referenced
this pull request
Jul 15, 2026
The java_sdk e2e mode and the lang-SDK k8s test solved the same cold-build problem two different ways (containerized Gradle with a repo-local cache vs host toolchain with salted home-dir caches), which a review follow-up on #69239 asked to reconcile. Converge the e2e mode on the #69411 native-toolchain convention: CI provisions the JDK with actions/setup-java and caches ~/.gradle, dropping the per-run eclipse-temurin image pull and the --network=host Gradle lock-handover workaround; local runs keep the containerized build so a dev host still needs no JDK.
joshuabvarghese
pushed a commit
to joshuabvarghese/airflow
that referenced
this pull request
Jul 16, 2026
…pache#69239) * Build the Java SDK and Scala Spark e2e bundles concurrently The example and scala_spark_example bundles are independent Gradle builds that both consume the SDK artifact published to the local Maven repo, so the Scala Spark dependency resolution no longer has to wait for the Java example build to finish before it starts. The bundle containers run with --network=host because Gradle's cross-process lock handover pings the lock owner over loopback, which isolated container network namespaces would break, and each build's output is captured and printed under its own header so concurrent logs do not interleave. * Cache the Java SDK Gradle dependencies in the e2e job The java_sdk e2e job builds the SDK and its example bundles with Gradle on every canary and java-sdk-touching run, starting from an empty GRADLE_USER_HOME each time. That forces a full re-download of the example bundles' dependency tree (Spark alone is around 1GB) on every run. Reusing the resolved dependencies across runs removes that repeated download. The restore/save split saves the warm-up even when the e2e tests fail, the key hashes the Kotlin-DSL build files and version catalog that pin the SDK dependencies, and runner.arch keeps the arch-specific auto-provisioned JDKs in .gradle/jdks from crossing between amd64 and arm64 runners. * Address java_sdk e2e review feedback on caching and build output Excludes churny *.lock/journal files from the Gradle cache per GitHub's own caching guidance, makes bundle build output print stdout/stderr symmetrically through Rich with markup disabled so raw Gradle text can't be misparsed as markup, ties the build pool size to the bundle list instead of a hardcoded count, and cites the Gradle lock-handover mechanism the --network=host workaround depends on.
joshuabvarghese
pushed a commit
to joshuabvarghese/airflow
that referenced
this pull request
Jul 16, 2026
The java_sdk e2e mode and the lang-SDK k8s test solved the same cold-build problem two different ways (containerized Gradle with a repo-local cache vs host toolchain with salted home-dir caches), which a review follow-up on apache#69239 asked to reconcile. Converge the e2e mode on the apache#69411 native-toolchain convention: CI provisions the JDK with actions/setup-java and caches ~/.gradle, dropping the per-run eclipse-temurin image pull and the --network=host Gradle lock-handover workaround; local runs keep the containerized build so a dev host still needs no JDK.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The java_sdk e2e job builds the two example bundles sequentially and starts from an empty
GRADLE_USER_HOMEon every run, forcing a full re-download of the bundles' dependency tree (Spark alone is ~1GB) each time.What
exampleandscala_spark_examplebundles concurrently; they are independent Gradle builds that both only consume the SDK artifact published beforehand.--network=host: Gradle's cross-process lock handover pings the lock owner over loopback, which isolated container network namespaces would break (60s lock timeouts).java-sdk/.gradle(caches, wrapper, jdks) in the e2e workflow for the java_sdk mode.actions/cache/restore+actions/cache/savesplit withif: always(), so a completed Gradle warm-up is saved even when the e2e tests later fail.runner.archso arch-specific auto-provisioned JDKs do not cross between amd64 and arm64 runners.Was generative AI tooling used to co-author this PR?