Skip to content

Conversation

@jesrypandawa
Copy link
Contributor

@jesrypandawa jesrypandawa commented Jul 6, 2022

Currently, dagger uses flink version 1.14.3 which is already supported for Python udf. But, dagger still does not implement this feature.

This PR adds the functionality of dagger to also support adding Python udf.

  • Configuration to enable udfs.
  • code changes in dagger to register udfs from GCS.
  • Code abstraction in dagger.
  • Add CI to publish compressed python files into release assets
  • Create a dagger-py-functions directory at odpf/dagger root
  • zip the entire directory including data, requirements.txt, and publish to release assets
  • Actual python udf(in scope: One sample UDF)
  • Documentation on configuration, how to add and use python udf
  • Passed for the test

MayurGubrele and others added 6 commits April 21, 2022 13:17
* chore: publish python zip on release
* refactor: update requirements.txt
* refactor: remove unused publish python package
* docs: add python udf docs

* docs: fix docs config

* docs: update python docs

* docs: add python environment setup
@prakharmathur82 prakharmathur82 merged commit faea8c6 into main Jul 12, 2022
@prakharmathur82 prakharmathur82 deleted the feat/add-support-python-udf branch July 12, 2022 05:45
Meghajit added a commit that referenced this pull request Jul 15, 2022
* feat: dynamic header capability for http post-processor. (#134)

Co-authored-by: mayankrai09 <[email protected]>

* feat: upgrade stencil client version (#141)

* chore: bump up core dependencies version for stencil upgrade (#151)

* feat: add dependencies in build.gradle for parquet source and python udfs (#167)

* doc: rfc for python udf (#129)

* doc: rfc for python udf

* docs: update rfc

Co-authored-by: jesrypandawa <[email protected]>

* feat:add support python udf (#172)

* fix: fix docs for udf contribution  (#175)

Co-authored-by: Prakhar Mathur <[email protected]>
Co-authored-by: mayankrai09 <[email protected]>
Co-authored-by: Hari krishna <[email protected]>
Co-authored-by: gauravsinghania <[email protected]>
Co-authored-by: jesrypandawa <[email protected]>
Co-authored-by: jesrypandawa <[email protected]>
Meghajit added a commit that referenced this pull request Jul 18, 2022
* feat: dynamic header capability for http post-processor. (#134)

Co-authored-by: mayankrai09 <[email protected]>

* feat: upgrade stencil client version (#141)

* chore: bump up core dependencies version for stencil upgrade (#151)

* feat: add dependencies in build.gradle for parquet source and python udfs (#167)

* doc: rfc for python udf (#129)

* doc: rfc for python udf

* docs: update rfc

Co-authored-by: jesrypandawa <[email protected]>

* feat:add support python udf (#172)

* fix: fix docs for udf contribution  (#175)

Co-authored-by: Prakhar Mathur <[email protected]>
Co-authored-by: mayankrai09 <[email protected]>
Co-authored-by: Hari krishna <[email protected]>
Co-authored-by: gauravsinghania <[email protected]>
Co-authored-by: jesrypandawa <[email protected]>
Co-authored-by: jesrypandawa <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: enhance dagger functions to support python UDF feat: enhance dagger core to be able to add python UDF

4 participants