Skip to content

BigQueryInsertJobOperator doesn't include BigQuery region/location in deferrable mode #35833

@manhld0206

Description

@manhld0206

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

I tested BigQueryInsertJobOperator with deferrable mode enabled.
I used Google Cloud Composer 2.4.0 with airflow version 2.4.3
The BigQuery's region is asia-northeast1 region. The operator ran fine before but I got Job not found error after enabling deferrable mode. At first I thought it was because I didn't specify location parameter in BigQueryInsertJobOperator. However, even after changing it, I still got the Job not found error.

After taking a quick look at the source code, I guess maybe the cause is BigQuery Trigger doesn't include location in it's parameters.
The code

job_status = await hook.get_job_status(job_id=self.job_id, project_id=self.project_id)

BigQuery get job API doc
Screenshot 2023-11-24 at 16 07 10

What you think should happen instead

No response

How to reproduce

Execute BigQueryInsertJobOperator using asia-northeast1 BigQuery region

Operating System

It's not OS. I used Google Cloud Composer 2.4.0 with airflow version 2.4.3

Versions of Apache Airflow Providers

No response

Deployment

Google Cloud Composer

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions