|
1 | | -## Versatile Data Kit Airflow provider |
| 1 | +# Versatile Data Kit Airflow provider |
2 | 2 |
|
3 | 3 | A set of Airflow operators, sensors and a connection hook intended to help schedule Versatile Data Kit jobs using Apache Airflow. |
| 4 | + |
| 5 | +# Usage |
| 6 | + |
| 7 | +To install it simply run: |
| 8 | +``` |
| 9 | +pip install airflow-provider-vdk |
| 10 | +``` |
| 11 | + |
| 12 | +Then you can create a workflow of data jobs (deployed by VDK Control Service) like this: |
| 13 | + |
| 14 | +``` |
| 15 | +from datetime import datetime |
| 16 | +
|
| 17 | +from airflow import DAG |
| 18 | +from vdk_provider.operators.vdk import VDKOperator |
| 19 | +
|
| 20 | +with DAG( |
| 21 | + "airflow_example_vdk", |
| 22 | + schedule_interval=None, |
| 23 | + start_date=datetime(2022, 1, 1), |
| 24 | + catchup=False, |
| 25 | + tags=["example", "vdk"], |
| 26 | +) as dag: |
| 27 | + trino_job1 = VDKOperator( |
| 28 | + conn_id="vdk-default", |
| 29 | + job_name="airflow-trino-job1", |
| 30 | + team_name="taurus", |
| 31 | + task_id="trino-job1", |
| 32 | + ) |
| 33 | +
|
| 34 | + trino_job2 = VDKOperator( |
| 35 | + conn_id="vdk-default", |
| 36 | + job_name="airflow-trino-job2", |
| 37 | + team_name="taurus", |
| 38 | + task_id="trino-job2", |
| 39 | + ) |
| 40 | +
|
| 41 | + transform_job = VDKOperator( |
| 42 | + conn_id="vdk-default", |
| 43 | + job_name="airflow-transform-job", |
| 44 | + team_name="taurus", |
| 45 | + task_id="transform-job", |
| 46 | + ) |
| 47 | +
|
| 48 | + [trino_job1, trino_job2] >> transform_job |
| 49 | +
|
| 50 | +``` |
| 51 | + |
| 52 | +# Example |
| 53 | + |
| 54 | +See [full example here](https://github.com/vmware/versatile-data-kit/tree/main/examples/airflow-example) |
| 55 | + |
| 56 | +# Demo |
| 57 | + |
| 58 | +You can see demo during one of the community meetings here: https://www.youtube.com/watch?v=c3j1aOALjVU&t=690s |
| 59 | + |
| 60 | +# Architecture |
| 61 | + |
| 62 | +See the [vdk enhancement proposal spec here](https://github.com/vmware/versatile-data-kit/tree/main/specs/vep-554-apache-airflow-integration) |
0 commit comments