Add dedicated Glue crawler lifecycle operators

[AlejandroMorgante]

Split the AWS Glue crawler lifecycle into explicit create, update, run, and delete operators while keeping the existing GlueCrawlerOperator backward compatible during its deprecation period.

Motivation

GlueCrawlerOperator currently combines resource provisioning, configuration changes, and execution. This means a Dag intended only to run an existing crawler can also create or modify it. Operation-specific operators make each task's intent and required AWS permissions explicit.

Operators

• GlueCrawlerCreateOperator creates a crawler from its Boto3 configuration.
• GlueCrawlerUpdateOperator updates an existing crawler without starting it.
• GlueCrawlerRunOperator starts an existing crawler and supports synchronous or deferrable completion waits through the existing GlueCrawlerCompleteTrigger.
• GlueCrawlerDeleteOperator deletes an existing crawler through the Boto3 client exposed by GlueCrawlerHook.

The existing GlueCrawlerOperator remains available for compatibility and emits an AirflowProviderDeprecationWarning directing new Dags to the operation-specific operators.

System test

The Glue system test now validates the complete lifecycle:

1. Create the crawler.
2. Update its description and verify the change through Boto3.
3. Run it with deferrable=True.
4. Run it with deferrable=False.
5. Delete it during teardown with the new delete operator.

Validation

Validated end-to-end against a real AWS environment. The system test created, updated, ran, and deleted the Glue crawler, then confirmed that the crawler, Glue database, Glue job, and S3 bucket no longer existed.

The system test can be reproduced from the Airflow checkout with AWS_PROFILE, AWS_REGION, and ROLE_ARN configured in files/airflow-breeze-config/environment_variables.env:

SYSTEM_TESTS_ENV_ID=<unique-id> \
breeze testing system-tests \
  --forward-credentials \
  --test-timeout 240 \
  providers/amazon/tests/system/amazon/aws/example_glue.py \
  -q

Latest result: 1 passed in 191.86s. The focused Glue crawler operator and trigger tests also pass: 66 passed, with 100% coverage of the modified operator module. Provider mypy, regular and manual prek checks, and the Amazon provider documentation build also pass.

---

Was generative AI tooling used to co-author this PR?

• Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

[SameerMesiah97]

I am not entirely sold on this PR.

Airflow is typically quite conservative about deprecating public operators, as they're imported directly into users' DAGs and therefore represent one of the most user-facing parts of the API. In my experience reviewing provider PRs, most deprecations have been for parameters, methods or other API elements, rather than replacing an entire operator.

I agree that dedicated create/update/run/delete operators make each task's intent clearer. However, I'm not yet convinced that this alone justifies introducing a deprecation cycle. The existing GlueCrawlerOperator already provides this functionality and has been part of the provider API for some time.

Could you elaborate on why deprecating the existing operator is the preferred approach here? Is there a limitation or design issue with the current operator that cannot reasonably be addressed while preserving the existing public API? Given the migration cost for existing users and the increase in public API surface, I think the rationale for deprecating an operator should be fairly strong.

Also. have you asked @o-nikolas and/or @vincbeck before opening this PR? If you are proposing a deprecation cycle (I know it is 'just an operator' but Glue is a popular service), it would be better to ensure the maintainers are aligned so that you dont waste time on a PR that would just get closed.

[AlejandroMorgante]

I am not entirely sold on this PR.

Airflow is typically quite conservative about deprecating public operators, as they're imported directly into users' DAGs and therefore represent one of the most user-facing parts of the API. In my experience reviewing provider PRs, most deprecations have been for parameters, methods or other API elements, rather than replacing an entire operator.

I agree that dedicated create/update/run/delete operators make each task's intent clearer. However, I'm not yet convinced that this alone justifies introducing a deprecation cycle. The existing GlueCrawlerOperator already provides this functionality and has been part of the provider API for some time.

Could you elaborate on why deprecating the existing operator is the preferred approach here? Is there a limitation or design issue with the current operator that cannot reasonably be addressed while preserving the existing public API? Given the migration cost for existing users and the increase in public API surface, I think the rationale for deprecating an operator should be fairly strong.

Also. have you asked @o-nikolas and/or @vincbeck before opening this PR? If you are proposing a deprecation cycle (I know it is 'just an operator' but Glue is a popular service), it would be better to ensure the maintainers are aligned so that you dont waste time on a PR that would just get closed.

Thanks, this is a fair concern.

The limitation I am trying to address is that GlueCrawlerOperator does not merely expose several optional operations. Its execute() method always reconciles the crawler before starting it:

1. It checks whether the crawler exists.
2. It creates it when missing or updates it when present.
3. It starts the crawler.

Therefore, a Dag whose only responsibility is to run an existing crawler cannot express:

Run this crawler without modifying infrastructure.

The task also requires create/update permissions, may recreate a crawler that was intentionally removed, and may apply configuration changes during what appears to be a routine execution task.

That is the main motivation for the dedicated operators: they allow infrastructure provisioning, configuration changes, execution, and deletion to have separate ownership and narrower IAM permissions. The benefit is not only clearer naming; it is also preventing a run-only task from mutating infrastructure.

That said, I agree that this does not necessarily require deprecating GlueCrawlerOperator. There is no compatibility limitation preventing us from keeping it as a supported composite operator while adding the operation-specific alternatives. My initial intention with the deprecation was to guide new Dags toward the narrower operators, not because preserving the existing API was impossible.

Given the migration cost and Airflow's conservative approach to public operator deprecations, I would be happy to remove the deprecation and make the proposal entirely additive. Existing users could continue using GlueCrawlerOperator, while users who need isolated responsibilities and narrower permissions could use the new operators.

I had already reached out to Vincent about this, and I am currently waiting for his review.

Would keeping GlueCrawlerOperator fully supported while adding the dedicated operators address your concern?

[SameerMesiah97]

I am not entirely sold on this PR.
Airflow is typically quite conservative about deprecating public operators, as they're imported directly into users' DAGs and therefore represent one of the most user-facing parts of the API. In my experience reviewing provider PRs, most deprecations have been for parameters, methods or other API elements, rather than replacing an entire operator.
I agree that dedicated create/update/run/delete operators make each task's intent clearer. However, I'm not yet convinced that this alone justifies introducing a deprecation cycle. The existing GlueCrawlerOperator already provides this functionality and has been part of the provider API for some time.
Could you elaborate on why deprecating the existing operator is the preferred approach here? Is there a limitation or design issue with the current operator that cannot reasonably be addressed while preserving the existing public API? Given the migration cost for existing users and the increase in public API surface, I think the rationale for deprecating an operator should be fairly strong.
Also. have you asked @o-nikolas and/or @vincbeck before opening this PR? If you are proposing a deprecation cycle (I know it is 'just an operator' but Glue is a popular service), it would be better to ensure the maintainers are aligned so that you dont waste time on a PR that would just get closed.

Thanks, this is a fair concern.

The limitation I am trying to address is that GlueCrawlerOperator does not merely expose several optional operations. Its execute() method always reconciles the crawler before starting it:

1. It checks whether the crawler exists.
2. It creates it when missing or updates it when present.
3. It starts the crawler.

Therefore, a Dag whose only responsibility is to run an existing crawler cannot express:

Run this crawler without modifying infrastructure.

The task also requires create/update permissions, may recreate a crawler that was intentionally removed, and may apply configuration changes during what appears to be a routine execution task.

That is the main motivation for the dedicated operators: they allow infrastructure provisioning, configuration changes, execution, and deletion to have separate ownership and narrower IAM permissions. The benefit is not only clearer naming; it is also preventing a run-only task from mutating infrastructure.

That said, I agree that this does not necessarily require deprecating GlueCrawlerOperator. There is no compatibility limitation preventing us from keeping it as a supported composite operator while adding the operation-specific alternatives. My initial intention with the deprecation was to guide new Dags toward the narrower operators, not because preserving the existing API was impossible.

Given the migration cost and Airflow's conservative approach to public operator deprecations, I would be happy to remove the deprecation and make the proposal entirely additive. Existing users could continue using GlueCrawlerOperator, while users who need isolated responsibilities and narrower permissions could use the new operators.

I had already reached out to Vincent about this, and I am currently waiting for his review.

Would keeping GlueCrawlerOperator fully supported while adding the dedicated operators address your concern?

I completely agree with your justification for adding lifecycle-based versions of this operator. But if we were to keep the existing operator, we would effectively be maintaining duplicate functionality. Whether that is better than deprecation is something that must be decided via consensus. Honestly, I would prefer deprecation over a few release cycles versus that.

[vincbeck]

I am on the side of deprecation as well but with no real strong opinion

[o-nikolas]

I would like to see it deprecated eventually, but I think we have to tread very carefully. Glue is one of the most used AWS Services through Airflow, so the deprecation will affect MANY users. We should give them ample time (several releases) and possibly brainstorm some other ways to campaign to get users migrated. I'm not sure if you have any ideas @AlejandroMorgante for that?

Also, just throwing it out there, you can add some optional capabilities to the existing operator to allow it to behave how you'd like. Just off the top of my head adding a create_if_missing option that defaults to True, but you can change to False for your needs.

[AlejandroMorgante]

Thanks, I agree that the main concern here is providing users with a clear and predictable deprecation process, especially given how widely Glue is used.

To clarify the scope of this PR: it does not remove GlueCrawlerOperator or change its existing behavior or signature. The operator continues to create a missing crawler, update an existing crawler, and start it exactly as it does today. This PR only starts the deprecation period by emitting an AirflowProviderDeprecationWarning and directing users to the operation-specific alternatives.

Amazon provider precedents

I checked previous operator deprecations in the Amazon provider:

• RedshiftSQLOperator was deprecated in Amazon provider 6.1.0 as part of #25717 and was removed in 8.0.0 by #30755.
• AwsLambdaInvokeFunctionOperator was deprecated in Amazon provider 7.3.0 by #29749 and was also removed in 8.0.0 by #30755.

In both cases, the existing operator remained functional during the deprecation period, users received a warning pointing to its replacement, and the actual removal happened separately in a major provider release with an explicit breaking-change notice.

I propose following the same process for GlueCrawlerOperator:

1. Keep GlueCrawlerOperator fully functional and backward compatible. (Done in this PR)
2. Emit an AirflowProviderDeprecationWarning pointing users to the operation-specific operators. (Done in this PR)
3. Document the legacy operator and provide a clear migration path. (Done in this PR)
4. Keep it available for several provider releases so users have ample time to migrate. (Planned deprecation policy)
5. Consider any eventual removal only in a separate PR and a future major provider release, after maintainer consensus and with an explicit breaking-change notice. (Future safeguard)

Therefore, users would not be required to migrate immediately, and deprecation would not imply an automatic or imminent removal. Any future removal would be a separate decision, with another review and a clearly communicated migration window.

Would this provide the certainty and visibility you had in mind for the deprecation process? @vincbeck , @o-nikolas, what do you think about moving forward with this approach? If you are comfortable with it, we can proceed with the technical review to make sure everything else in the PR is in good shape.

[vincbeck]

Thanks, I agree that the main concern here is providing users with a clear and predictable deprecation process, especially given how widely Glue is used.

To clarify the scope of this PR: it does not remove GlueCrawlerOperator or change its existing behavior or signature. The operator continues to create a missing crawler, update an existing crawler, and start it exactly as it does today. This PR only starts the deprecation period by emitting an AirflowProviderDeprecationWarning and directing users to the operation-specific alternatives.

Amazon provider precedents

I checked previous operator deprecations in the Amazon provider:

• RedshiftSQLOperator was deprecated in Amazon provider 6.1.0 as part of #25717 and was removed in 8.0.0 by #30755.
• AwsLambdaInvokeFunctionOperator was deprecated in Amazon provider 7.3.0 by #29749 and was also removed in 8.0.0 by #30755.

In both cases, the existing operator remained functional during the deprecation period, users received a warning pointing to its replacement, and the actual removal happened separately in a major provider release with an explicit breaking-change notice.

I propose following the same process for GlueCrawlerOperator:

1. Keep GlueCrawlerOperator fully functional and backward compatible. (Done in this PR)
2. Emit an AirflowProviderDeprecationWarning pointing users to the operation-specific operators. (Done in this PR)
3. Document the legacy operator and provide a clear migration path. (Done in this PR)
4. Keep it available for several provider releases so users have ample time to migrate. (Planned deprecation policy)
5. Consider any eventual removal only in a separate PR and a future major provider release, after maintainer consensus and with an explicit breaking-change notice. (Future safeguard)

Therefore, users would not be required to migrate immediately, and deprecation would not imply an automatic or imminent removal. Any future removal would be a separate decision, with another review and a clearly communicated migration window.

Would this provide the certainty and visibility you had in mind for the deprecation process? @vincbeck , @o-nikolas, what do you think about moving forward with this approach? If you are comfortable with it, we can proceed with the technical review to make sure everything else in the PR is in good shape.

Guys, using AI is good and actually recommended but I would also encourage to think as the other person. Messages are getting longer and longer with AI. Please keep them short, focus. If every single person writes the equivalent of a book every-time one writes a message, it will very shortly be unscalable (if not already). Please, either instruct your AI agent to make it short, or update it yourself.

[AlejandroMorgante]

Thanks. I'll work on keeping my messages more concise and focused.

To summarize, this PR only starts the deprecation period for GlueCrawlerOperator; it does not remove the operator or change its existing behavior. It remains fully functional, now emits an AirflowProviderDeprecationWarning, and its migration path is documented.

We can keep it available for several provider releases and only consider removing it in a separate cleanup PR for a future major release, following the previous RedshiftSQLOperator and AwsLambdaInvokeFunctionOperator deprecation process (#30755).

@vincbeck, if you are ok with this safeguard?

[vincbeck]

Thanks. I'll work on keeping my messages more concise and focused.

To summarize, this PR only starts the deprecation period for GlueCrawlerOperator; it does not remove the operator or change its existing behavior. It remains fully functional, now emits an AirflowProviderDeprecationWarning, and its migration path is documented.

We can keep it available for several provider releases and only consider removing it in a separate cleanup PR for a future major release, following the previous RedshiftSQLOperator and AwsLambdaInvokeFunctionOperator deprecation process (#30755).

@vincbeck, if you are ok with this safeguard?

All good on my side

[AlejandroMorgante]

Perfect, does this approach work for you as well, @o-nikolas?

The current Amazon provider version is 9.32.0, so the plan would be to add the AirflowProviderDeprecationWarning now, keep the operator fully functional for several releases, and consider removing it no earlier than the 11.0.0 major release in a separate PR, ensuring users have at least one full major release cycle to migrate.

Once this PR is merged, I’ll open a tracking issue documenting that plan.

[vincbeck]

Instead of creating an issue for that (which I dont think it is necessary), I would leave a comment which is way easier for maintainers to follow

[o-nikolas]

Perfect, does this approach work for you as well, @o-nikolas?

The current Amazon provider version is 9.32.0, so the plan would be to add the AirflowProviderDeprecationWarning now, keep the operator fully functional for several releases, and consider removing it no earlier than the 11.0.0 major release in a separate PR, ensuring users have at least one full major release cycle to migrate.

Once this PR is merged, I’ll open a tracking issue documenting that plan.

Yeah, this is fine for me. As long as we start the deprecation now, we can decide exactly when and how to finally remove it later on.