fix(glue): Fix GlueJobOperator verbose logs not showing in deferrable mode#64342
Merged
o-nikolas merged 3 commits intoapache:mainfrom Apr 10, 2026
Merged
Conversation
85cb82e to
945c8a3
Compare
Contributor
|
I see that all the checks are green, but this is still marked draft. Is this ready for review @shivaam? |
945c8a3 to
763a0f0
Compare
Contributor
Author
Yes, it is ready to be reviewed. @o-nikolas |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR restores verbose CloudWatch log forwarding for GlueJobOperator(deferrable=True, verbose=True) by adding an async polling/log-forwarding path to the Glue trigger and by extracting shared log formatting/helpers into the Glue hook.
Changes:
- Override
GlueJobCompleteTrigger.run()to poll Glue job state and forward CloudWatch logs whenverbose=True, yieldingTriggerEventerrors instead of raising in the triggerer. - Extract
get_glue_log_group_names()andformat_glue_logs()intohooks/glue.pyand reuse them from both sync and async paths. - Add unit tests covering verbose deferrable behavior and
_forward_logs()pagination / no-new-events cases.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| providers/amazon/src/airflow/providers/amazon/aws/triggers/glue.py | Adds verbose-mode async polling + CloudWatch log forwarding to the Glue job completion trigger. |
| providers/amazon/src/airflow/providers/amazon/aws/hooks/glue.py | Extracts shared helpers to unify log group naming and log formatting between sync/async paths. |
| providers/amazon/tests/unit/amazon/aws/triggers/test_glue.py | Adds unit tests for verbose deferrable Glue trigger behavior and log forwarding. |
providers/amazon/src/airflow/providers/amazon/aws/triggers/glue.py
Outdated
Show resolved
Hide resolved
763a0f0 to
20e1f7d
Compare
…bOperator When using GlueJobOperator with deferrable=True and verbose=True, CloudWatch logs were silently ignored because the trigger inherited the base waiter's run() method which only polls job status. This adds a run() override and _forward_logs() helper to GlueJobCompleteTrigger that streams logs from both output and error CloudWatch log groups, matching the format used by the synchronous path. Key changes: - Extract get_glue_log_group_names() and format_glue_logs() as shared helpers in hooks/glue.py to eliminate sync/async duplication - Override run() in GlueJobCompleteTrigger for verbose log streaming - Yield TriggerEvent with error status on failure (matching base class pattern) instead of raising AirflowException in the triggerer process - Add tests for verbose success, failure, max attempts, pagination, ResourceNotFoundException, and no-new-events scenarios closes: apache#56535
…logger - Use logs_client.meta.region_name instead of self.region_name for the CloudWatch URL in ResourceNotFoundException handler, avoiding invalid URLs when region_name is None (resolved from connection) - Replace caplog with mock.patch.object(trigger.log, ...) in tests per Airflow community convention
da8c6c1 to
a2fc628
Compare
o-nikolas
approved these changes
Apr 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-land of #63086 (reverted by #64340) with corrected error handling.
Changes
When
GlueJobOperatorruns withdeferrable=Trueandverbose=True, CloudWatchlogs were silently ignored because the trigger inherited the base waiter's
run()method which only polls job status.
Improvements over the original PR:
TriggerEventwith error status instead of raisingAirflowException—matches the
AwsBaseWaiterTrigger.run()pattern soexecute_complete()handleserrors in the worker, not the triggerer. This eliminates the
AirflowExceptionimport from both the trigger and test files, which was the root cause of the revert.
get_glue_log_group_names()andformat_glue_logs()as shared helpers inhooks/glue.pyto eliminate sync/async duplication withGlueJobHook.print_job_logs.closes: #56535
related: #63086, #64340
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.6) following the guidelines