Suspend Apache Beam Provider due to grpcio limitation#61926
Conversation
|
af4f6ef to
6e70de6
Compare
d772897 to
b348d38
Compare
jscheffl
left a comment
There was a problem hiding this comment.
Hope to resolve this soonish, such that beam can be re-enabled.
eladkal
left a comment
There was a problem hiding this comment.
LGTM but placing temporary block to avoid accidental merge.
it requires also to follow discussion and vote/lazy consensus on mailing list
Yes @eladkal , i am planning to follow that , am trying to resolve some build issues. me and @potiuk had quite a long discussion here https://apache-airflow.slack.com/archives/C015SLQF059/p1770972824043689 |
We can't rely on Slack discussions. |
Yeah correct, sorry for the confusion, am saying will send a email discussion soon, am trying to see after removing beam provider what all impacting. |
Nothing is impacted :) |
3ea2b92 to
64bc285
Compare
e959560 to
85ce6be
Compare
594cc65 to
15e5cf3
Compare
Backport failed to create: v3-1-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 917abea v3-1-testThis should apply the commit to the v3-1-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
…#61926)" This reverts commit 917abea. - Remove hacks regarding the beam provider suspension, as they are not relevant anymore. ISSUE: apache#66551
Suspending apache-airflow-providers-apache-beam package.
The current Apache Beam Python SDK pins grpcio restricts it to a range like <1.66.
This old grpcio version conflicts with several other dependencies in the project workspace, especially when using uv sync.
Newer versions of opentelemetry-exporter-otlp / opentelemetry-proto that need updated protobuf / grpcio
pydantic-ai-slim (used in common-ai provider) requiring opentelemetry-api ≥ 1.28.0
This blocks development work (including AIP-99) and prevents bumping other dependencies across the repo. the PR is here #61794 where its failing currently due to the limitation of several dependencies pinned.
The grpcio upper bound issue in Apache Beam has been known for a while, and its raised already @potiuk apache/beam#34081 long ago.
The root cause is being addressed upstream in the gRPC project grpc/grpc#37710
(they have already fixed it, with a possible release in grpcio 1.78.1 or a patch that should lift most of these pins/conflicts)
Following the official guidelines for provider suspension:
https://github.com/apache/airflow/blob/main/PROVIDERS.rst#suspending-releases-for-providers
The Apache beam provider can be removed from suspension by a PR reverting this change once grpcio limitation removed from the apache-beam.
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.