Status of testing Providers that were prepared on June 07, 2022

[potiuk]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider alibaba: 2.0.0rc2

• SSL Bucket, Light Logic Refactor and Docstring Update for Alibaba Provider (#23891): @koconder
• Apply per-run log templates to log handlers (#24153): @uranusjr

Provider amazon: 4.0.0rc2

• Create get_partition and create_partition methods in GlueCatalogHook (#23857): @gmcrocetti
• Add support for associating custom tags to job runs submitted via EmrContainerOperator (#23769): @skadyan
• Fix RedshiftCreateClusterOperator for single-node cluster type (#23839): @pankajastro
• fix: StepFunctionHook ignores explicit set region_name (#23976): @Taragolis
• Fix Amazon EKS example DAG raises warning during Imports (#23849): @Dark-Knight11
• Move string arg evals to execute() in EksCreateClusterOperator (#23877): @josh-fell
• Fix: AwsGlueJobOperator - patches #24215 (#24308): @gmcrocetti
• Light Refactor and Clean-up AWS Provider (#23907): @koconder
• Update sample dag and doc for RDS (#23651): @vincbeck
• Reformat the whole AWS documentation (#23810): @vincbeck
• Replace "absolute()" with "resolve()" in pathlib objects (#23675): @potiuk
• Apply per-run log templates to log handlers (#24153): @uranusjr
• Refactor GlueJobHook get_or_create_glue_job method. (#24215): @gmcrocetti
• Update the DMS Sample DAG and Docs (#23681): @ferruzzi
• Update doc and sample dag for Quicksight (#23653): @vincbeck
• Update doc and sample dag for EMR Containers (#24087): @vincbeck
• Add AWS project structure tests (re: AIP-47) (#23630): @o-nikolas
• Add doc and sample dag for GCSToS3Operator (#23730): @vincbeck
• Remove old Athena Sample DAG (#24170): @ferruzzi
• Clean up f-strings in logging calls (#23597): @josh-fell

Provider apache.beam: 4.0.0rc2

• Added missing project_id to the wait_for_job (#24020): @lwyszomi
• Support impersonation service account parameter for Dataflow runner (#23961): @lwyszomi
• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.cassandra: 3.0.0rc2

• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.druid: 3.0.0rc2

• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.hdfs: 3.0.0rc2

• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.hive: 3.0.0rc2

• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder
• AIP-47 - Migrate hive DAGs to new design #22439 (#24204): @chethanuk-plutoflume

Provider apache.kylin: 3.0.0rc2

• AIP-47 - Migrate kylin DAGs to new design #22439 (#24205): @chethanuk-plutoflume
• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.pinot: 3.0.0rc2

• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider apache.spark: 3.0.0rc2

• Add typing for airflow/configuration.py (#23716): @potiuk
• Fix backwards-compatibility introduced by fixing mypy problems (#24230): @potiuk
• AIP-47 - Migrate spark DAGs to new design #22439 (#24210): @chethanuk-plutoflume
• chore: Refactoring and Cleaning Apache Providers (#24219): @koconder

Provider cncf.kubernetes: 4.1.0rc2

• Use KubernetesHook to create api client in KubernetesPodOperator (#20578): @dstandish
• [FEATURE] KPO add optional K8S hook (#22086): @raphaelauv
• Add param docs to KubernetesHook and KubernetesPodOperator (#23955) (#24054): @mariasumedre-3pg
• Use "remote" pod when patching KPO pod as "checked" (#23676): @dstandish
• Don't use the root logger in KPO _suppress function (#23835): @dstandish
• Fix KubernetesPodOperator await_container_completion condition (#23883): @akakakakakaa
• Migrate Cncf.Kubernetes example DAGs to new design #22441 (#24132): @chethanuk-plutoflume
• Clean up f-strings in logging calls (#23597): @josh-fell

Provider databricks: 3.0.0rc2

• Add Databricks Deferrable Operators (#19736): @eskarimov
• Add git_source to DatabricksSubmitRunOperator (#23620): @akolar-db
• fix: DatabricksSubmitRunOperator and DatabricksRunNowOperator cannot define .json as template_ext (#23622) (#23641): @jonathansp
• Fix UnboundLocalError when sql is empty list in DatabricksSqlHook (#23815): @kazanzhy

Provider dbt.cloud: 2.0.0rc2

• Enable dbt Cloud provider to interact with single tenant instances (#24264): @epapineau
• Fix typo in dbt Cloud provider description (#23179): @josh-fell
• Fix new MyPy errors in main (#22884): @potiuk

Provider docker: 3.0.0rc2

• Remove xcom_push from DockerOperator (#23981): @eladkal
• Migrate Docker example DAGs to new design (#23167): @Bowrna

Provider elasticsearch: 4.0.0rc2

• Apply per-run log templates to log handlers (#24153): @uranusjr

Provider exasol: 3.0.0rc2

• Fix UnboundLocalError when sql is empty list in ExasolHook (#23812): @kazanzhy

Provider github: 2.0.0rc2

• Remove GithubOperator use in GithubSensor.__init__() (#24214): @josh-fell

Provider google: 8.0.0rc2

• Add key_secret_project_id parameter which specifies a project with KeyFile (#23930): @MaksYermak
• Added impersonation_chain for dataflow operators (#24046): @lwyszomi
• Add fields to CLOUD_SQL_EXPORT_VALIDATION. (#23724): @ishiis
• Update credentials when using ADC in Compute Engine (#23773): @MaksYermak
• set color to operators in cloud_sql.py (#24000): @ishiis
• Sql to gcs with exclude columns (#23695): @jaegwonseo
• [Issue#22846] allow option to encode or not encode UUID when uploading from Cassandra to GCS (#23766): @fuxiao224
• Workflows assets & system tests migration (AIP-47) (#24105): @wojsamjan
• Spanner assets & system tests migration (AIP-47) (#23957): @wojsamjan
• Speech To Text assets & system tests migration (AIP-47) (#23643): @wojsamjan
• Cloud SQL assets & system tests migration (AIP-47) (#23583): @wojsamjan
• Cloud Storage assets & StorageLink update (#23865): @wojsamjan
• fix BigQueryInsertJobOperator (#24165): @raphaelauv
• Fix the link to google workplace (#24080): @francois2metz
• Fix DataprocJobBaseOperator not compatible with TaskGroups (#23791): @gmcrocetti
• Remove hack from BigQuery DTS hook (#23887): @henriqueribeiro
• Fix GCSToGCSOperator cannot copy a single file/folder without copying other files/folders with that prefix (#24039): @gitstart-airflow
• Workaround job race bug on biguery to gcs transfer (#24330): @potiuk
• Fix BigQuery system tests (#24013): @bhirsz
• Ensure @contextmanager decorates generator func (#23103): @uranusjr
• Migrate Dataproc to new system tests design (#22777): @bhirsz
• AIP-47 - Migrate google leveldb DAGs to new design ##22447 (#24233): @chethanuk-plutoflume
• Apply per-run log templates to log handlers (#24153): @uranusjr

Provider hashicorp: 3.0.0rc2

• Add explanatory note for contributors about updating Changelog (#24229): @potiuk
• Prepare provider documentation 2022.05.11 (#23631): @potiuk
• pydocstyle D202 added (#24221): @Bowrna
• Prepare docs for May 2022 provider's release (#24231): @potiuk

Provider jdbc: 3.0.0rc2

• Handler parameter from JdbcOperator to JdbcHook.run (#23817): @kazanzhy

Provider jenkins: 3.0.0rc2

• Bug Fix for apache-airflow-providers-jenkins JenkinsJobTriggerOperator (#22802): @SasanAhmadi

Provider microsoft.azure: 4.0.0rc2

• Pass connection extra parameters to wasb BlobServiceClient (#24154): @tanelk
• Apply per-run log templates to log handlers (#24153): @uranusjr
• Migrate Microsoft example DAGs to new design #22452 - azure (#24141): @chethanuk-plutoflume
• Add typing to Azure Cosmos Client Hook (#23941): @potiuk

Provider oracle: 3.0.0rc2

• Add parameters to templated fields in OracleOperator (#22857): @malthe
• Make numpy effectively an optional dependency for Oracle provider (#24272): @potiuk

Provider papermill: 3.0.0rc2

• Add support to specify language name in PapermillOperator (#23916): @nhojjohn
• Fix langauge override in papermill operator (#24301): @nhojjohn

Provider presto: 3.0.0rc2

• Add PrestoToSlackOperator (#23979): @eladkal

Provider qubole: 3.0.0rc2

• Add typing for airflow/configuration.py (#23716): @potiuk
• Fix backwards-compatibility introduced by fixing mypy problems (#24230): @potiuk

Provider sftp: 3.0.0rc2

• Adding fnmatch type regex to SFTPSensor (#24084): @alexkruc

Provider snowflake: 3.0.0rc2

• Fix error when SnowflakeHook take empty list in sql param (#23767): @kazanzhy

Provider sqlite: 3.0.0rc2

• Fix SqliteHook compatibility with SQLAlchemy engine (#23790): @kaxil

Provider ssh: 3.0.0rc2

• Add disabled_algorithms as an extra parameter for SSH connections (#24090): @pauldalewilliams
• fixing SSHHook bug when using allow_host_key_change param (#24116): @alexkruc

The guidelines on how to test providers can be found in

Verify providers by contributors

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[vincbeck]

All my changes are related to doc and/or sample dag. No testing needed

[potiuk]

Cool. Bulk-updated them as "tested" :D

[raphaelauv]

#23981 work

[potiuk]

I also tested and updated all "my" changes

[raphaelauv]

#20578 I have no regression ( not using a new connection or using args of the KPO ) if still using core airlfow kubernetes settings 👍

[potiuk]

One comment. I am releasing rc2 in a moment (@josh-fell found that our RELEASE notes (part of packages) contained Airflow 2.1+ so I am re-relesing all providers with that one change included (but feel free to continue the testing - the only difference will be those descriptions.

[ishiis]

#23724 #24000 is tested.

[Dark-Knight11]

#23849 contains only refactoring of code. No testing needed

[pauldalewilliams]

I've tested #24090 and can confirm it's working as expected.

[pankajastro]

Tested #23839 working as expected.

[dstandish]

@potiuk do you know why k8s is a major?

[jonathansp]

I've tested #23622 and it works as expected.

[gilangardya]

I've tested #24165 (google, BigQueryInsertJobOperator) with composer-2.0.15-airflow-2.2.5, it works as expected.
These are the tests I did:

Job type	Statement	Details	7.0.0	2022.5.18+composer	8.0.0rc1
COPY	N/A	Copy BigQuery Table	❌	❌	✅
EXTRACT	N/A	To CSV	❌	❌	✅
LOAD	N/A	From CSV	❌	❌	✅
QUERY	N/A	SELECT query	✅	✅	✅
QUERY	N/A	WRITE_TRUNCATE with partitioned table	✅	✅	✅
QUERY	DDL	CREATE TABLE AS SELECT	✅	✅	✅
QUERY	DML	INSERT	✅	✅	✅
QUERY	DML	MERGE	✅	✅	✅
QUERY	DCL	GRANT	❌	❌	✅
QUERY	Procedural	DECLARE	❌	❌	✅
QUERY	Debugging	ASSERT	❌	❌	✅
QUERY	Other	EXPORT DATA	❌	❌	✅
QUERY	Multi-statement	DECLARE + INSERT + MERGE	❌	❌	✅