Support for pip --constraint

[robsonpeixoto]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Tools like airflow has some constraints that need be respected, for example, when are adding dependencies in a docker image.

This project could add constraints that accept a list of FQDN with all constraints. For example:

contrains = [
  "https://github.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt",
  "file://{CWD}/constraints.txt"
]

[abn]

@robsonpeixoto it is not clear what you mean here. I doubt this is within the scope of Poetry. How do you imagine thise being used when generating the lock file? Feels like you are essentially removing the resolver from the equation.

If airflow needs these pinned dependencies, shouldn't this be done so within the package's core metadata? It looks like it already is.

See snipped output below.

$ curl -sL https://pypi.org/pypi/apache-airflow/json | jq '.info.requires_dist'
[
  "google-auth-httplib2 (>=0.0.1) ; extra == 'gcp'",
...
  "google-cloud-vision (<2.0.0,>=0.35.2) ; extra == 'gcp'",
  "grpcio-gcp (>=0.2.2) ; extra == 'gcp'",
  "pandas-gbq ; extra == 'gcp'",
  "PyOpenSSL ; extra == 'gcp_api'",
  "google-api-python-client (<2.0.0,>=1.6.0) ; extra == 'gcp_api'",
...
  "grpcio-gcp (>=0.2.2) ; extra == 'gcp_api'",
  "pandas-gbq ; extra == 'gcp_api'",
  "Flask-OAuthlib (>=0.9.1) ; extra == 'github_enterprise'",
  "oauthlib (!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2) ; extra == 'github_enterprise'",
  "requests-oauthlib (==1.1.0) ; extra == 'github_enterprise'",
  "Flask-OAuthlib (>=0.9.1) ; extra == 'google_auth'",
  "oauthlib (!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2) ; extra == 'google_auth'",
  "requests-oauthlib (==1.1.0) ; extra == 'google_auth'",
  "grpcio (>=1.15.0) ; extra == 'grpc'",
  "hvac (~=0.10) ; extra == 'hashicorp'",
  "snakebite (>=2.7.8) ; extra == 'hdfs'",
  "hmsclient (>=0.1.0) ; extra == 'hive'",
  "pyhive[hive] (>=0.6.0) ; extra == 'hive'",
  "JPype1 (==0.7.1) ; extra == 'jdbc'",
...
  "sphinx (==1.8.5) ; (python_version < \"3.0\") and extra == 'devel_hadoop'",
  "mock ; (python_version < \"3.3\") and extra == 'devel_hadoop'",
  "contextdecorator ; (python_version < \"3.4\") and extra == 'devel_hadoop'",
...
  "sphinx (==1.8.5) ; (python_version < \"3.0\") and extra == 'doc'",
  "sphinx (>=2.1.2) ; (python_version >= \"3.0\") and extra == 'doc'",
  "sphinx-copybutton ; (python_version >= \"3.6\") and extra == 'doc'",
  "docker (~=3.0) ; extra == 'docker'",
  "pydruid (<=0.5.8,>=0.4.1) ; extra == 'druid'",
  "elasticsearch (<6.0.0,>=5.0.0) ; extra == 'elasticsearch'",
  "elasticsearch-dsl (<6.0.0,>=5.0.0) ; extra == 'elasticsearch'",
  "boto3 (~=1.10) ; extra == 'emr'",
  "PyOpenSSL ; extra == 'gcp'",
  "google-api-python-client (<2.0.0,>=1.6.0) ; extra == 'gcp'",
  "google-auth (<2.0.0,>=1.0.0) ; extra == 'gcp'"
]

[robsonpeixoto]

For some reason Airflow are not pinning all dependencies.

❯ curl -sL https://pypi.org/pypi/apache-airflow/1.10.12/json | jq '.info.requires_dist' | grep ipython

❯ curl -sq https://github.com/apache/airflow/constraints-1.10.12/constraints-3.6.txt | grep ipython
ipython-genutils==0.2.0
ipython==7.16.1

❯ curl -sq https://github.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt | grep ipython
ipython-genutils==0.2.0
ipython==7.17.0

Makes sense add this feature? Being honest, was the first time a saw this parameter --constraints.

[abn]

That sounds like an airflow packaging issue, if ipython is really a dependency. Although, doing pip install apache-airflow==1.10.12 -c constraints.txt, will not install ipython if it is not defined in the core metadata.

This feature does not, imho, make sense for poetry. We already have a lock file, which is effectively what the cosntratint file is (minus the top level dependencies).

[stpierre]

The differences between a poetry lockfile and a constraints file, AFAICT, is that a constraints file can easily be hand-crafted, and can be shared among multiple different packages. For instance, we use a constraints file as the global version lock for our entire internal monorepo with dozens and dozens of distinct packages in it.

Is there some way to accomplish the same thing with poetry? If not, would you consider reopening this issue?

[potiuk]

Just a comment here as the original author of the Airflow constraints.idea and CI behind it. The constraints are very different from lock files of poetry as far as I understand it. They are a simple snapshot of all dependencies that are 'guaranteed' to work. But you are still free to use other versions if you want and by specifying constraint you do not fix the version, you merely set a 'starting point for your installation.

The thing is that airflow is both application and library - we want to be able to provide to our users an installation method that works and produces 'working set of non-conflicting dependencies' as well as enable the user to upgrade the library if needed after airflow is installed (as long as this upgrade is ok for airflow). Airflow has many extras and some users install just one or two extras which can easily get latest versions of dependent libraries.

But when you combine it with other extras which have different requirements it might be that latest version of the common dependency is not good. Constraints is the way to say 'with those constraints you can install any combination of extras' and not only they will cleanly install but also all 4000+ tests of Airflow pass. Yeah we run all the possible tests automatically before we (automatically) upgrade the constraints - and it is our CI that takes care of it.

[honnix]

FWIW, Openstack has this global constraints file https://github.com/openstack/requirements/blob/d0b389d840c75216ab2cc10df849cb98990b1503/tox.ini#L10 used by all its repos.

[honnix]

Took a stab at this.