certbot/tools/pipstrap.py

#!/usr/bin/env python
"""Uses pip to upgrade Python packaging tools to pinned versions."""
import os

import pip_install

_REQUIREMENTS_PATH = os.path.join(os.path.dirname(__file__), "pipstrap_constraints.txt")


def main():
    pip_install_args = '--requirement "{0}"'.format(_REQUIREMENTS_PATH)
    pip_install.pip_install_with_print(pip_install_args)


if __name__ == '__main__':
    main()
Write a replacement for pipstrap (#8401) * Add a new, simplified version of pipstrap. * Use tools/pipstrap.py * Uncomment code * Refactor pip_install.py and provide hashes. * Fix test_sdists.sh. * Make code work on Python 2. * Call strip_hashes.py using Python 3. * Pin the oldest version of httplib2 used in distros * Strip enum34 dependency. * Remove pip pinnings from dev_constraints.txt * Correct pipstrap docstring. * Don't set working_dir twice. * Add comments 2020-11-06 05:17:41 -05:00			`#!/usr/bin/env python`
			`"""Uses pip to upgrade Python packaging tools to pinned versions."""`
			`import os`

			`import pip_install`

Enable again build isolation with proper pinning of build dependencies (#8443) Fixes #8256 First let's sum up the problem to solve. We disabled the build isolation available in pip>=19 because it could potential break certbot build without a control on our side. Basically builds are not reproductible. Indeed the build isolation triggers build of PEP-517 enabled transitive dependencies (like `cryptography`) with the build dependencies defined in their `pyproject.toml`. For `cryptography` in particular these requirements include `setuptools>=40.6.0`, and quite logically pip will install the latest version of `setuptools` for the build. And when `setuptools` broke with the version 50, our build did the same. But disabling the build isolation is not a long term solution, as more and more project will migrate on this approach and it basically provides a lot of benefit in how dependencies are built. The ideal solution would be to be able to apply version constraints on our side on the build dependencies, in order to pin `setuptools` for instance, and decide precisely when we upgrade to a newer version. However for now pip does not provide a mechanism for that (like a `--build-constraint` flag or propagation of existing `--constraint` flag). Until I saw https://github.com/pypa/pip/issues/9081 and https://github.com/pypa/pip/issues/8439. Apart the fact that https://github.com/pypa/pip/issues/9081 shows that pip maintainers are working on this issue, it explains how pip works regarding PEP-517 and infers which workaround can be used to still pin the build dependencies. It turns out that pip invokes itself in each build isolation to install the build dependencies. It means that even if some flags (like `--constraint`) are not explicitly passed to the pip sub call, the global environment remains, in particular the environment variables. Thus it is known that every pip flag can alternatively be set by environment variable using the following pattern for the variable name: `PIP_[FLAG_NAME_UPPERCASE]`. So for `--constraint`, it is `PIP_CONSTRAINT`. And so you can pass a constraint file to the pip sub call through that mechanism. I made some tests with a constraint file containing pinning for `setuptools`: indeed under isolation zone, the constraint file has been honored and the provided pinned version has been used to build the dependencies (I tested it with `cryptography`). Finally this PR takes advantage of this mechanism, by setting `PIP_CONSTRAINT` to `pip_install`, the snap building process, the Dockerfiles and the windows installer building process. I also extracted out the requirements of the new `pipstrap.py` to be reusable in these various build processes. * Use workaround to fix build requirements in build isolation, and renable build isolation * Clean imports in pipstrap * Externalize pipstrap reqs to be reusable * Inject pipstrap constraints during pip_install * Update docker build * Update snapcraft build * Prepare installer build * Fix pipstrap constraints in snap build * Add back --no-build-cache option in Docker images build * Update snap/snapcraft.yaml * Use proper flags with pip Co-authored-by: Brad Warren <bmw@users.noreply.github.com> 2020-12-16 13:49:31 -05:00			`_REQUIREMENTS_PATH = os.path.join(os.path.dirname(__file__), "pipstrap_constraints.txt")`
Write a replacement for pipstrap (#8401) * Add a new, simplified version of pipstrap. * Use tools/pipstrap.py * Uncomment code * Refactor pip_install.py and provide hashes. * Fix test_sdists.sh. * Make code work on Python 2. * Call strip_hashes.py using Python 3. * Pin the oldest version of httplib2 used in distros * Strip enum34 dependency. * Remove pip pinnings from dev_constraints.txt * Correct pipstrap docstring. * Don't set working_dir twice. * Add comments 2020-11-06 05:17:41 -05:00

			`def main():`
Enable again build isolation with proper pinning of build dependencies (#8443) Fixes #8256 First let's sum up the problem to solve. We disabled the build isolation available in pip>=19 because it could potential break certbot build without a control on our side. Basically builds are not reproductible. Indeed the build isolation triggers build of PEP-517 enabled transitive dependencies (like `cryptography`) with the build dependencies defined in their `pyproject.toml`. For `cryptography` in particular these requirements include `setuptools>=40.6.0`, and quite logically pip will install the latest version of `setuptools` for the build. And when `setuptools` broke with the version 50, our build did the same. But disabling the build isolation is not a long term solution, as more and more project will migrate on this approach and it basically provides a lot of benefit in how dependencies are built. The ideal solution would be to be able to apply version constraints on our side on the build dependencies, in order to pin `setuptools` for instance, and decide precisely when we upgrade to a newer version. However for now pip does not provide a mechanism for that (like a `--build-constraint` flag or propagation of existing `--constraint` flag). Until I saw https://github.com/pypa/pip/issues/9081 and https://github.com/pypa/pip/issues/8439. Apart the fact that https://github.com/pypa/pip/issues/9081 shows that pip maintainers are working on this issue, it explains how pip works regarding PEP-517 and infers which workaround can be used to still pin the build dependencies. It turns out that pip invokes itself in each build isolation to install the build dependencies. It means that even if some flags (like `--constraint`) are not explicitly passed to the pip sub call, the global environment remains, in particular the environment variables. Thus it is known that every pip flag can alternatively be set by environment variable using the following pattern for the variable name: `PIP_[FLAG_NAME_UPPERCASE]`. So for `--constraint`, it is `PIP_CONSTRAINT`. And so you can pass a constraint file to the pip sub call through that mechanism. I made some tests with a constraint file containing pinning for `setuptools`: indeed under isolation zone, the constraint file has been honored and the provided pinned version has been used to build the dependencies (I tested it with `cryptography`). Finally this PR takes advantage of this mechanism, by setting `PIP_CONSTRAINT` to `pip_install`, the snap building process, the Dockerfiles and the windows installer building process. I also extracted out the requirements of the new `pipstrap.py` to be reusable in these various build processes. * Use workaround to fix build requirements in build isolation, and renable build isolation * Clean imports in pipstrap * Externalize pipstrap reqs to be reusable * Inject pipstrap constraints during pip_install * Update docker build * Update snapcraft build * Prepare installer build * Fix pipstrap constraints in snap build * Add back --no-build-cache option in Docker images build * Update snap/snapcraft.yaml * Use proper flags with pip Co-authored-by: Brad Warren <bmw@users.noreply.github.com> 2020-12-16 13:49:31 -05:00			`pip_install_args = '--requirement "{0}"'.format(_REQUIREMENTS_PATH)`
			`pip_install.pip_install_with_print(pip_install_args)`
Write a replacement for pipstrap (#8401) * Add a new, simplified version of pipstrap. * Use tools/pipstrap.py * Uncomment code * Refactor pip_install.py and provide hashes. * Fix test_sdists.sh. * Make code work on Python 2. * Call strip_hashes.py using Python 3. * Pin the oldest version of httplib2 used in distros * Strip enum34 dependency. * Remove pip pinnings from dev_constraints.txt * Correct pipstrap docstring. * Don't set working_dir twice. * Add comments 2020-11-06 05:17:41 -05:00

			`if __name__ == '__main__':`
			`main()`