Apache Airflow version
main (development)
What happened
After refactoring DatabricksSqlHook lost it's ability to run multiple queries in one Airflow connection (basically same Spark Session). Now it doesn't care what the smart get_conn returns and simply creates new connection for each query in the queries list. It is wrong and breaks logic like:
USE your_database;
select from table_in_your_database;
In this case it will create first connection and as consequence Databricks session, runs USE your_database statement, then recreate connection/session which doesn't know anything about first query and runs select from table_in_your_database statement and obviously it fails because second session doesn't know anything about first query...
This is a breaking change and cost us some time to understand what went wrong after our upgrade from 2.2.0 to 2.5.3
What you think should happen instead
smart get_conn functions should decide when to create new connection and when not to. Otherwise why do we need so complex logic there?
How to reproduce
Try to send (using DatabricksSqlHook) to the Databricks Warehouse a string with two queries:
USE your_database;
select from table_in_your_database;
Operating System
Debian GNU/Linux 11 (bullseye)
Versions of Apache Airflow Providers
databricks 3.4.0
Deployment
Other Docker-based deployment
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct
Apache Airflow version
main (development)
What happened
After refactoring
DatabricksSqlHooklost it's ability to run multiple queries in one Airflow connection (basically same Spark Session). Now it doesn't care what the smart get_conn returns and simply creates new connection for each query in the queries list. It is wrong and breaks logic like:In this case it will create first connection and as consequence Databricks session, runs
USE your_databasestatement, then recreate connection/session which doesn't know anything about first query and runsselect from table_in_your_databasestatement and obviously it fails because second session doesn't know anything about first query...This is a breaking change and cost us some time to understand what went wrong after our upgrade from
2.2.0to2.5.3What you think should happen instead
smart get_conn functions should decide when to create new connection and when not to. Otherwise why do we need so complex logic there?
How to reproduce
Try to send (using
DatabricksSqlHook) to the Databricks Warehouse a string with two queries:Operating System
Debian GNU/Linux 11 (bullseye)
Versions of Apache Airflow Providers
databricks 3.4.0Deployment
Other Docker-based deployment
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct