Add deferred pagination mode to GenericTransfer#44809

Merged

potiuk merged 134 commits into

apache:mainfrom

dabla:feature/paginated-generic-transfer

Feb 26, 2025

Commits on Jan 2, 2025

refactor: Refactored the GenericTransfer operator to support paginated reads (in deferred mode) and introduce a SQLExecuteQueryTrigger

davidblain-infrabel
authored and
potiuk
committed
refactor: updated provider dependencies

dabla
authored and
potiuk
committed
refactor: Added TestSQLExecuteQueryTrigger and moved test code which allows you to run deferrable operators in test common test utils

davidblain-infrabel
authored and
potiuk
committed
refactor: Fixed static checks

davidblain-infrabel
authored and
potiuk
committed
refactor: Fixed static checks

davidblain-infrabel
authored and
potiuk
committed
refactor: Fixed static checks

davidblain-infrabel
authored and
potiuk
committed
refactor: Reformatted GenericTransfer

davidblain-infrabel
authored and
potiuk
committed
refactor: Moved source and destination hooks into cached properties

davidblain-infrabel
authored and
potiuk
committed
refactor: Moved imports to type checking block

davidblain-infrabel
authored and
potiuk
committed
refactor: Fixed execute method of GenericTransfer

davidblain-infrabel
authored and
potiuk
committed
refactor: Refactored get_hook method of GenericTransfer which checks if hook is instance of DbApiHook instead of checking presence of get_records and insert_rows method

davidblain-infrabel
authored and
potiuk
committed
refactor: Remove white lines from mock_context

davidblain-infrabel
authored and
potiuk
committed
refactor: Reformatted get_hook in GenericTransfer operator

davidblain-infrabel
authored and
potiuk
committed
refactor: Added sql.pyi for SQLExecuteQueryTrigger

davidblain-infrabel
authored and
potiuk
committed
refactor: Reformatted SQLExecuteQueryTrigger definition

davidblain-infrabel
authored and
potiuk
committed
refactor: Added alias in SQLExecuteQueryTrigger definition

davidblain-infrabel
authored and
potiuk
committed
Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored
Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 7, 2025

Commits on Jan 9, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 11, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 13, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 15, 2025

Commits on Jan 23, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 28, 2025

Commits on Jan 29, 2025

Commits on Jan 30, 2025

Updates docs to improve Readability (#46205)

aritra24
authored and
davidblain-infrabel
committed
Add Webserver parameters: max_form_parts, max_form_memory_size (#45749)

moiseenkov
authored and
davidblain-infrabel
committed
Fix `api_connexion` test failures in main (#46245)

vincbeck
authored and
davidblain-infrabel
committed
Add autorefresh to Task Instance and Dag Run pages (#46213)

authored and
davidblain-infrabel
committed
use populate_by_name (#46168)

rawwar
authored and
davidblain-infrabel
committed
Disable Flask-SQLAlchemy modification tracking in FAB provider (#46249)

jedcunningham
authored and
davidblain-infrabel
committed
Fix AWS auth manager compatibility test (#46246)

vincbeck
authored and
davidblain-infrabel
committed
Prevent from being added to root logger by default (#46236)

amoghrajesh
authored and
davidblain-infrabel
committed
Move more DAG parsing related config to dag_processor section (#46034)

jedcunningham
authored and
davidblain-infrabel
committed
Update test fixture docstring (#46252)

jedcunningham
authored and
davidblain-infrabel
committed
AIP-38 Invalidate DryRun query cache on submit (#46238)

pierrejeambrun
authored and
davidblain-infrabel
committed
Fix pre-commit to re-generate on dependency changes (#46259)

jscheffl
authored and
davidblain-infrabel
committed
update some icons and add dag import error to dagslist (#46251)

bbovenzi
authored and
davidblain-infrabel
committed
Limit `google-cloud-aiplatform` to fix issues in CI (#46242)

authored and
davidblain-infrabel
committed
Adding from_path.exists() check in a couple of conditions (#46255)

kunaljubce
authored and
davidblain-infrabel
committed
Remove caplog from HDFS tests (#46263)

jscheffl
authored and
davidblain-infrabel
committed
Remove caplog from Livy tests (#46272)

potiuk
authored and
davidblain-infrabel
committed
Replace caplog with patching log property in k8s tests (#46273)

potiuk
authored and
davidblain-infrabel
committed
Remove import from MySQL provider tests in generic transfer test (#46274)

potiuk
authored and
davidblain-infrabel
committed
Fix GitDagBundle to support https (include 46073/46179) (#46226)

jx2lee
authored and
davidblain-infrabel
committed
AIP-72: Port _validate_inlet_outlet_assets_activeness into Task SDK (#46020)

amoghrajesh
authored and
davidblain-infrabel
committed
Try and reduce flakiness of number of test call asserts (#46276)

amoghrajesh
authored and
davidblain-infrabel
committed
Add evaluation extra to google-cloud-aiplatform (#46270)

authored and
davidblain-infrabel
committed
Move Docker Provider to the New Structure (#46097)

bugraoz93
authored and
davidblain-infrabel
committed
Move Atlassian Jira Provider to the new structure (#46271)

bugraoz93
authored and
davidblain-infrabel
committed
moved apache.cassandra provider to new structure and fixed check-for-inclusive-language (#46171)

authored and
davidblain-infrabel
committed
Make azure test less flaky/racy (#46281)

potiuk
authored and
davidblain-infrabel
committed
move Influxdb provider to new provider structure (#46277)

authored and
davidblain-infrabel
committed
Move Apprise provider to new structure (#46161)

authored and
davidblain-infrabel
committed
Make racy test test_start_pod_startup_interval_seconds less racy (#46282)

potiuk
authored and
davidblain-infrabel
committed
Move FTP Provider to the New Structure (#46206)

aritra24
authored and
davidblain-infrabel
committed
move Trino provider to new provider structure (#46162)

authored and
davidblain-infrabel
committed
refactor(providers/slack): move slack provider to new structure (#46209)

josix
authored and
davidblain-infrabel
committed
Move Apache Livy to new provider structure (#46131)

jason810496
authored and
davidblain-infrabel
committed
refactor: Re-added test_generic_transfer under standard provider
davidblain-infrabel
committed
Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored
refactor: Moved test_generic_transfer from standard to common sql provider
davidblain-infrabel
committed
Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Jan 31, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Feb 6, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Feb 10, 2025

Commits on Feb 11, 2025

Commits on Feb 12, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Feb 13, 2025

Commits on Feb 18, 2025

Commits on Feb 20, 2025

Merge branch 'main' into feature/paginated-generic-transfer
dabla
authored

Commits on Feb 21, 2025