Skip to content

Commit 59c8c6a

Browse files
authored
vdk-airflow: populate readme (#924)
1 parent 25374d0 commit 59c8c6a

File tree

1 file changed

+60
-1
lines changed
  • projects/vdk-plugins/airflow-provider-vdk

1 file changed

+60
-1
lines changed
Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,62 @@
1-
## Versatile Data Kit Airflow provider
1+
# Versatile Data Kit Airflow provider
22

33
A set of Airflow operators, sensors and a connection hook intended to help schedule Versatile Data Kit jobs using Apache Airflow.
4+
5+
# Usage
6+
7+
To install it simply run:
8+
```
9+
pip install airflow-provider-vdk
10+
```
11+
12+
Then you can create a workflow of data jobs (deployed by VDK Control Service) like this:
13+
14+
```
15+
from datetime import datetime
16+
17+
from airflow import DAG
18+
from vdk_provider.operators.vdk import VDKOperator
19+
20+
with DAG(
21+
"airflow_example_vdk",
22+
schedule_interval=None,
23+
start_date=datetime(2022, 1, 1),
24+
catchup=False,
25+
tags=["example", "vdk"],
26+
) as dag:
27+
trino_job1 = VDKOperator(
28+
conn_id="vdk-default",
29+
job_name="airflow-trino-job1",
30+
team_name="taurus",
31+
task_id="trino-job1",
32+
)
33+
34+
trino_job2 = VDKOperator(
35+
conn_id="vdk-default",
36+
job_name="airflow-trino-job2",
37+
team_name="taurus",
38+
task_id="trino-job2",
39+
)
40+
41+
transform_job = VDKOperator(
42+
conn_id="vdk-default",
43+
job_name="airflow-transform-job",
44+
team_name="taurus",
45+
task_id="transform-job",
46+
)
47+
48+
[trino_job1, trino_job2] >> transform_job
49+
50+
```
51+
52+
# Example
53+
54+
See [full example here](https://github.com/vmware/versatile-data-kit/tree/main/examples/airflow-example)
55+
56+
# Demo
57+
58+
You can see demo during one of the community meetings here: https://www.youtube.com/watch?v=c3j1aOALjVU&t=690s
59+
60+
# Architecture
61+
62+
See the [vdk enhancement proposal spec here](https://github.com/vmware/versatile-data-kit/tree/main/specs/vep-554-apache-airflow-integration)

0 commit comments

Comments
 (0)