Skip to content

Fix serialization of Params with set data type - #19267

Merged
kaxil merged 15 commits into
apache:mainfrom
dstandish:fix-param-issue
Nov 5, 2021
Merged

Fix serialization of Params with set data type#19267
kaxil merged 15 commits into
apache:mainfrom
dstandish:fix-param-issue

Conversation

@dstandish

@dstandish dstandish commented Oct 27, 2021

Copy link
Copy Markdown
Contributor

This is a solution for #19096

Previously, the serialization of params did not run the param value through the _serialize function, resulting in non-json-serializable dictionaries. This manifested when a user, for example, tried to use params with a default value of type set.

Here we change the logic to run the param value through the serialization process. And I add a test for the set case.

Could def use extra set of eyes on schema definition. btw i added hook for formatting this -- i can do that in separate pr if that's helpful.

Comment thread airflow/serialization/serialized_objects.py Outdated
@dstandish

dstandish commented Oct 28, 2021

Copy link
Copy Markdown
Contributor Author

there's a bit of awkwardness with the Param object as relates to serialization because, while for other objects we can reuse __dict__ as kwargs, for params, default is a kwarg but not an attribute. it is tempting to add default as an attribute perhaps make value a property or just remove it, but not sure of downstream consequences.

@dstandish
dstandish marked this pull request as draft October 28, 2021 16:06
@dstandish
dstandish marked this pull request as ready for review October 28, 2021 17:08
Comment thread airflow/serialization/serialized_objects.py Outdated
@dstandish
dstandish requested a review from XD-DENG as a code owner October 28, 2021 20:26
Comment thread airflow/models/param.py
uranusjr
uranusjr previously approved these changes Oct 29, 2021
@github-actions github-actions Bot added the full tests needed We need to run full set of tests for this PR to merge label Oct 29, 2021
@github-actions

This comment has been minimized.

@uranusjr
uranusjr dismissed their stale review October 29, 2021 02:00

Hmm, I think the test failures are related to the change and need to be fixed.

Comment thread airflow/serialization/serialized_objects.py Outdated
@dstandish
dstandish marked this pull request as draft October 29, 2021 07:28
@dstandish
dstandish marked this pull request as draft October 29, 2021 07:28
@dstandish
dstandish marked this pull request as ready for review October 29, 2021 15:16
@kaxil
kaxil requested a review from msumit November 2, 2021 22:11
Comment thread airflow/serialization/serialized_objects.py Outdated
@msumit

msumit commented Nov 3, 2021

Copy link
Copy Markdown
Contributor

lgtm overall. Also nice work in handling the backward compatibility 👍

Comment thread .pre-commit-config.yaml Outdated
Comment thread airflow/serialization/serialized_objects.py Outdated
@kaxil kaxil added this to the Airflow 2.2.2 milestone Nov 3, 2021
@kaxil

kaxil commented Nov 4, 2021

Copy link
Copy Markdown
Member

@dstandish Can you take a look at the failing tests please

@dstandish

Copy link
Copy Markdown
Contributor Author

@dstandish Can you take a look at the failing tests please

yup will do

Comment thread tests/serialization/test_dag_serialization.py
@dstandish

Copy link
Copy Markdown
Contributor Author

fyi i updated Param init to check whether the default is json serializable and a deprecation warning if it is not.

Comment thread tests/serialization/test_dag_serialization.py
Comment thread tests/serialization/test_dag_serialization.py Outdated
Comment thread airflow/models/param.py Outdated
@kaxil
kaxil merged commit 8512e05 into apache:main Nov 5, 2021
jedcunningham pushed a commit that referenced this pull request Nov 5, 2021
This is a solution for #19096

Previously, the serialization of params did not run the param value through the `_serialize` function, resulting in non-json-serializable dictionaries.  This manifested when a user, for example, tried to use params with a default value of type `set`.

Here we change the logic to run the param value through the serialization process.  And I add a test for the `set` case.

closes #19096

(cherry picked from commit 8512e05)
@jedcunningham jedcunningham added the type:bug-fix Changelog: Bug Fixes label Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:serialization full tests needed We need to run full set of tests for this PR to merge type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants