Update Pipenv to 2025.0.4 and improve installation/caching#1840
Merged
Conversation
* Updates Pipenv from 2024.0.1 to 2025.0.4. Changelog: https://github.com/pypa/pipenv/blob/main/CHANGELOG.md#202504-2025-07-07 * Switches to installing Pipenv into its own virtual environment, so that it and its dependencies don't leak into the app environment. * Stops installing pip (in addition to Pipenv) when the chosen package manager is Pipenv, to match the behaviour when using Poetry or uv. * Forces the build cache to be cleared if the contents of `Pipfile.lock` has changed since the last build. This is required to work around `pipenv {install,sync}` by design not uninstalling packages if they are removed from the lockfile (?!). We also can't use `pipenv clean` since it doesn't work with `--system` / `PIPENV_SYSTEM`. Apps that have been inadvertently depending on Pipenv's own dependencies implicitly, will need to add those dependencies explicitly to their `Pipfile` and regenerate `Pipfile.lock`. Similarly, any apps that for some reason use pip later in the build in addition to Pipenv, will need to either add pip as an explicit dependency in their `Pipfile`, or else for adhoc use-cases (such as in one-off dynos), can temporarily install pip using `python -m ensurepip --default-pip`. Supersedes #1828. GUS-W-17482289. GUS-W-17482412. GUS-W-19116903.
84032db to
3aa14c1
Compare
Malax
approved these changes
Jul 23, 2025
Member
Author
edmorley
added a commit
that referenced
this pull request
Jul 23, 2025
After moving Pipenv to its own virtual environment in #1840 (to fix leaking Pipenv's dependencies into the app environment), apps that explicitly depend on a package that happen to also be a Pipenv dependency were seeing `ModuleNotFoundError` failures. For example: ``` ModuleNotFoundError: No module named 'certifi' ``` It appears that Pipenv's `--system` feature has a bug whereby it sees its own dependencies (installed in its own environment) as already installed (even though they aren't actually installed in the app environment), so incorrectly skips installing them. In general, it seems Pipenv's support for `--system` is pretty buggy: https://github.com/pypa/pipenv/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22--system%20flag%22 (for example, it doesn't work with `pipenv run` either) The classic Python buildpack doesn't currently use virtual environments (since it installs a dedicated Python install for the app, that's already separate from system Python), and switching to them would be a significant change that may cause other types of breakage (for example, to apps that hardcode the path to Python). As such for now, we need to continue to use `pipenv install --system` (as the buildpack has done for some time), but I have found the bug can be worked around if we set `VIRTUAL_ENV` to the Python install location, to trick Pipenv into thinking it's operating in a virtual environment. The Pipenv tests have been updated to confirm that an explicit `certifi` install now works. GUS-W-19118553.
edmorley
added a commit
that referenced
this pull request
Jul 23, 2025
After moving Pipenv to its own virtual environment in #1840 (to fix leaking Pipenv's dependencies into the app environment), apps that explicitly depend on a package that happen to also be a Pipenv dependency were seeing `ModuleNotFoundError` failures. For example: ``` ModuleNotFoundError: No module named 'certifi' ``` It appears that Pipenv's `--system` feature has a bug whereby it sees its own dependencies (installed in its own environment) as already installed (even though they aren't actually installed in the app environment), so incorrectly skips installing them. In general, it seems Pipenv's support for `--system` is pretty buggy: https://github.com/pypa/pipenv/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22--system%20flag%22 (for example, it doesn't work with `pipenv run` either) The classic Python buildpack doesn't currently use virtual environments (since it installs a dedicated Python install for the app, that's already separate from system Python), and switching to them would be a significant change that may cause other types of breakage (for example, to apps that hardcode the path to Python). As such for now, we need to continue to use `pipenv install --system` (as the buildpack has done for some time), but I have found the bug can be worked around if we set `VIRTUAL_ENV` to the Python install location, to trick Pipenv into thinking it's operating in a virtual environment. The Pipenv tests have been updated to confirm that an explicit `certifi` install now works. GUS-W-19118553.
edmorley
added a commit
that referenced
this pull request
Jul 23, 2025
After moving Pipenv to its own virtual environment in #1840 (to fix leaking Pipenv's dependencies into the app environment), apps that explicitly depend on a package that happen to also be a Pipenv dependency were seeing `ModuleNotFoundError` failures. For example: ``` ModuleNotFoundError: No module named 'certifi' ``` It appears that Pipenv's `--system` feature has a bug whereby it sees its own dependencies (installed in its own environment) as already installed (even though they aren't actually installed in the app environment), so incorrectly skips installing them. In general, it seems Pipenv's support for `--system` is pretty buggy: https://github.com/pypa/pipenv/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22--system%20flag%22 (for example, it doesn't work with `pipenv run` either) The classic Python buildpack doesn't currently use virtual environments (since it installs a dedicated Python install for the app, that's already separate from system Python), and switching to them would be a significant change that may cause other types of breakage (for example, to apps that hardcode the path to Python). As such for now, we need to continue to use `pipenv install --system` (as the buildpack has done for some time), but I have found the bug can be worked around if we set `VIRTUAL_ENV` to the Python install location, to trick Pipenv into thinking it's operating in a virtual environment. The Pipenv tests have been updated to confirm that an explicit `certifi` install now works. GUS-W-19118553.
Member
Author
|
This exposed an underlying bug in Pipenv's |
edmorley
added a commit
that referenced
this pull request
Jul 23, 2025
After moving Pipenv to its own virtual environment in #1840 (to fix leaking Pipenv's dependencies into the app environment), apps that explicitly depend on a package that happen to also be a Pipenv dependency were seeing `ModuleNotFoundError` failures. For example: ``` ModuleNotFoundError: No module named 'certifi' ``` Or: ``` ModuleNotFoundError: No module named 'packaging' ``` It appears that Pipenv's `--system` feature has a bug whereby it sees its own dependencies (installed in its own environment) as already installed (even though they aren't actually installed in the app environment), so incorrectly skips installing them. In general, it seems Pipenv's support for `--system` is pretty buggy (for example, it doesn't work with `pipenv run` either): https://github.com/pypa/pipenv/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22--system%20flag%22 The classic Python buildpack doesn't currently use virtual environments (since it installs a dedicated Python install for the app, that's already separate from system Python), and switching to them would be a significant change that may cause other types of breakage (for example, to apps that hardcode the path to Python). As such for now, we need to continue to use `pipenv install --system` (as the buildpack has done for some time), but I have found the bug can be worked around if we set `VIRTUAL_ENV` to the Python install location, to trick Pipenv into thinking it's operating in a virtual environment. The Pipenv tests have been updated to confirm that an explicit `certifi` install now works. GUS-W-19118553.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pipfile.lockhas changed since the last build. This is required to work aroundpipenv {install,sync}by design not uninstalling packages if they are removed from the lockfile (?!). We also can't usepipenv cleansince it doesn't work with--system/PIPENV_SYSTEM.Apps that have been inadvertently depending on Pipenv's own dependencies implicitly, will need to add those dependencies explicitly to their
Pipfileand regeneratePipfile.lock.In particular, if you see the error
ModuleNotFoundError: No module named 'pkg_resources'it means the app (or one of its dependencies) relies upon thesetuptoolspackage, but that package is missing fromPipfile/Pipfile.lockand must now be added there.Similarly, any apps that for some reason use pip later in the build in addition to Pipenv, will need to either add pip as an explicit dependency in their
Pipfile, or else for adhoc use-cases can temporarily install pip usingpython -m ensurepip --default-pip.Supersedes #1828.
GUS-W-17482289.
GUS-W-17482412.
GUS-W-19116903.