Upper bound for markupsafe dependency

[tsingh2k15]

Overview

Currently Jinja2 upper bound is jinja2>=2.10.1,<4 at https://github.com/apache/airflow/blob/main/setup.cfg#L121 as part of #16595

Jinja2 seems to require MarkupSafe>=2.0 at https://github.com/pallets/jinja/blob/main/setup.py#L6 as part of pallets/jinja@e2f673e

Based on this, is it feasible to consider updating upper bound in airflow for markupsafe>=1.1.1, <2.0 to allow 2.0 at https://github.com/apache/airflow/blob/main/setup.cfg#L126 ?

Issue

I am facing a dependency conflict(using poetry) where one of the libraries I am using requires jinja2 (>=3.0,<4.0) which further requires MarkupSafe>=2.0 as mentioned above. This seems to conflict with markupsafe>=1.1.1, <2.0 required by airflow.

Please share your thoughts as I may be missing some context/rationale behind not doing this. There could be compatibility issues which I may not be aware of. I am just looking at the dependency chain and have not tried this change myself yet. I thought to ask here first in case this has already been discussed or if there are risks with doing this change.

Are you willing to submit a PR?

If there is a consensus on suggested change, then I can contribute the change. I will need to review the process and will require some guidance.

[potiuk]

You can try remove the limit.

Unfortunately markupsafe 2.0 limit is not commented and it's been moved here from setup.py and it has long history - tracing the origin of that limit might be difficult.

However I think this was just a 'safeguard' in case of future incompatibilities. Which we stopped doing pretty much. We have constraints mechanism that allows us to safeguard against future-backwards-incompatible releases, and we tend to update all the dependencies to remove all the "upper-bound" limits (unless they are needed - then we leave a comment there why).

However - I have a bad news for you - you will lilkely have different problems with poetry - because it does not support constraints, so the bad news for you is that the only supported installation method for airflow is pip install ... --constraint as described in https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html (you can find there also the description about why poetry is not supported). As a result - if you are using poetry (thus not using constraints), even if you install Airflow today it might fail to install tomorrow (because for example new Jinja release tomorrow might cause another conflict).

Just to repeat again - installing with pip and --constraint is the only "sure" way for airflow to install without causing conflicts.

And If you want you can see my talk from today where I explained why we have this: https://www.youtube.com/watch?v=_SjMdQLP30s&t=2549s , also you can see the issue that wes closed by poetry maintainer with the comment that they won't add support for constratints python-poetry/poetry#3225

What you can try to do is I think you can try to remove the upper bounds and make a PR - it will automatically try to upgrade to latest dependencies which "comply" with other limtis and will run tests for them. If those tests pass, we should be able to merge the PR and get rid of the upper bound (but it will not solve the underlying problems that trying to use poetry to install airflow consistently will not work until they support constraints). Just be aware that there might other dependencies that might hold Jinja2 from upgrding

[tsingh2k15]

@potiuk Thanks for sharing further details. I will follow the suggestion you made regarding "sure" way for airflow installation preventing conflicts. For this particular dependency, I will remove the upper bound and make a PR. If all the validations and tests pass, then it could be merged if reviewers see no risk in merging it.

[tsingh2k15]

@potiuk I have created a PR #19760 This is my first PR here, your guidance here would be helpful. Thanks!

[tsingh2k15]

Opened a new PR - #19881

[tsingh2k15]

Tracking it under new PR - #19953

[tsingh2k15]

Final PR for lifting the upper bound - #20113