Skip to content

Document Dataproc limitations in Sovereign Cloud from Google #66706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shahar1 merged 1 commit into from
May 11, 2026
Merged

Document Dataproc limitations in Sovereign Cloud from Google #66706

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions providers/google/docs/operators/cloud/dataproc.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,52 @@ Prerequisite Tasks
.. include:: /operators/_partials/prerequisite_tasks.rst


Sovereign Cloud from Google guidance
------------------------------------

Sovereign Cloud from Google is a Google Cloud deployment with a separate service endpoint surface from
standard Google Cloud regions. Dataproc support in Sovereign Cloud from Google can therefore differ from
Dataproc support in standard Google Cloud. If a Dataproc API method or job type is not implemented in
Sovereign Cloud from Google, the corresponding Airflow operator or job configuration will not run there.
This is a Dataproc service availability limitation, not a Cloud Composer or Airflow operator limitation.

The following Dataproc job configurations were available in the tested Sovereign Cloud from Google environment:

* Hadoop jobs.
* Hive jobs.
* PySpark jobs.
* Spark jobs.
* Spark SQL jobs.
* Trino jobs.

The following Dataproc job configurations were not available in the tested environment:

* ``PigJob``.
* ``PrestoJob``.
* ``SparkRJob``.
* ``FlinkJob``.

The following operators do not work in Sovereign Cloud from Google because the corresponding Dataproc API
methods are not implemented there yet:

* Workflow template operators:
:class:`~airflow.providers.google.cloud.operators.dataproc.DataprocCreateWorkflowTemplateOperator`,
:class:`~airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateWorkflowTemplateOperator`, and
:class:`~airflow.providers.google.cloud.operators.dataproc.DataprocInstantiateInlineWorkflowTemplateOperator`.
* Batch operators:
:class:`~airflow.providers.google.cloud.operators.dataproc.DataprocCreateBatchOperator`,
:class:`~airflow.providers.google.cloud.operators.dataproc.DataprocGetBatchOperator`,
:class:`~airflow.providers.google.cloud.operators.dataproc.DataprocListBatchesOperator`,
:class:`~airflow.providers.google.cloud.operators.dataproc.DataprocDeleteBatchOperator`, and
:class:`~airflow.providers.google.cloud.operators.dataproc.DataprocCancelOperationOperator`.

The same limitation was observed with ``gcloud`` commands, so this is not a Cloud Composer or Airflow
operator limitation.

When running in Sovereign Cloud from Google, verify that the Dataproc API method and job type are available
in that environment before relying on the corresponding Airflow operator in a production DAG.


.. _howto/operator:DataprocCreateClusterOperator:
.. _howto/operator:DataprocInstantiateInlineWorkflowTemplateOperator:

Expand Down
Loading