I think this is causing failures in some of our tests so this PR reverts the change until we can fix the problem.
The 2nd commit is to keep the change using more idiomatic wording in the changelog for another change that got included in this PR.
* Revert "Use built-in support for OCSP in cryptography >= 2.5 (#6603)"
This reverts commit 2ddaf3db04.
* keep changelog correction
In response to #6594. [Fixes #6594.]
To execute OCSP requests, certbot relies currently on a openssl binary execution. If openssl is not present in the PATH, the OCSP check will be silently ignored. Since version 2.4, cryptography has support for OCSP requests, without the need to have openssl binary available locally.
This PR takes advantage of it, and will use the built-in support of OCSP in cryptography for versions >= 2.4. Otherwise, fallback is done do a direct call to openssl binary, allowing oldest requirements to still work with legacy cryptography versions.
Update: requirement is now cryptography >= 2.5, to avoid to rely on a private method from cryptography.
* Implement logic using cryptography
* Working OSCP using pure cryptography
* Fix openssl usage in unit tests
* Reduce verbosity
* Add tests
* Improve naive skipIf
* Test resiliency
* Update ocsp.py
* Validate OCSP response. Unify OCSP URL get
* Improve resiliency checks, correct lint/mypy
* Improve hash selection
* Fix warnings when calling openssl bin
* Load OCSP tests assets as vectors.
* Update ocsp.py
* Protect against invalid ocsp response.
* Add checks to OCSP response
* Add more control on ocsp response
* Be lenient about assertion that next_update must be in the future, similarly to openssl.
* Construct a more advanced OCSP response mock to trigger more logic in ocsp module.
* Add test
* Refactor signature process to use crypto_util
* Fallback for cryptography 2.4
* Avoid a collision with a meteor.
* Correct method signature documentation
* Relax OCSP update interval
* Trigger built-in ocsp logic from cryptography with 2.5+
* Update pinned version of cryptography
* Update certbot/ocsp.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update ocsp.py
* Update ocsp_test.py
* Update CHANGELOG.md
* Update CHANGELOG.md
Fixes#6697.
This PR updates the version of setuptools pinned in pipstrap which works around the problems we have seen on recent OSes.
Why did I pick this version of setuptools? Because it's the latest and greatest, [supports all versions of Python that we do](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py#L173), [has been out for a month and a half without the need for a point release](https://setuptools.readthedocs.io/en/latest/history.html), and has no non-optional dependencies.
For the last point about dependencies, I don't think we have too much to worry about. setuptools did have a period between versions 34.0.0 and 36.0.0 where they tried to have normal dependencies on other packages, but reverted these changes. See their [changelog for 36.0.0](https://setuptools.readthedocs.io/en/latest/history.html#v36-0-0).
You can also compare their [current setup.py file](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py) to the [setup.py file for the currently pinned version](https://github.com/pypa/setuptools/blob/v29.0.1/setup.py) and you'll see [not much has changed](https://pastebin.com/nQj6d7D8).
Not only that, but I have successfully tested these changes on:
* ubuntu18.10
* ubuntu18.04LTS
* ubuntu16.04LTS
* ubuntu14.04LTS
* ubuntu14.04LTS_32bit
* debian9
* debian8.1
* amazonlinux-2015.09.1
* amazonlinux-2015.03.1
* RHEL7
* fedora23
* fedora29
* centos7
* centos6
* freebsd11
* macOS
* Update setuptools to 40.6.3.
* Build letsencrypt-auto.
* update changelog
* Don't use pipstrap in Dockerfile.centos6.
(cherry picked from commit b7211c3f39)
Fixes#6697.
This PR updates the version of setuptools pinned in pipstrap which works around the problems we have seen on recent OSes.
Why did I pick this version of setuptools? Because it's the latest and greatest, [supports all versions of Python that we do](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py#L173), [has been out for a month and a half without the need for a point release](https://setuptools.readthedocs.io/en/latest/history.html), and has no non-optional dependencies.
For the last point about dependencies, I don't think we have too much to worry about. setuptools did have a period between versions 34.0.0 and 36.0.0 where they tried to have normal dependencies on other packages, but reverted these changes. See their [changelog for 36.0.0](https://setuptools.readthedocs.io/en/latest/history.html#v36-0-0).
You can also compare their [current setup.py file](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py) to the [setup.py file for the currently pinned version](https://github.com/pypa/setuptools/blob/v29.0.1/setup.py) and you'll see [not much has changed](https://pastebin.com/nQj6d7D8).
Not only that, but I have successfully tested these changes on:
* ubuntu18.10
* ubuntu18.04LTS
* ubuntu16.04LTS
* ubuntu14.04LTS
* ubuntu14.04LTS_32bit
* debian9
* debian8.1
* amazonlinux-2015.09.1
* amazonlinux-2015.03.1
* RHEL7
* fedora23
* fedora29
* centos7
* centos6
* freebsd11
* macOS
* Update setuptools to 40.6.3.
* Build letsencrypt-auto.
* update changelog
* Don't use pipstrap in Dockerfile.centos6.
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so after virtualenv upgrades pip, downgrade to the pinned version.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Always download the pinned version of pip in pipstrap
* Run build.py
* Update changelog
* Remove unused variable
* Run build.py
(cherry picked from commit 9746c310d8)
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so after virtualenv upgrades pip, downgrade to the pinned version.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Always download the pinned version of pip in pipstrap
* Run build.py
* Update changelog
* Remove unused variable
* Run build.py
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
(cherry picked from commit 658b7b9d47)
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
These tests were running on Ubuntu Precise and Debian Wheezy which have reached their end of life and are no longer maintained by the respective distros. This updates the tests to a newer version of Debian and Ubuntu.
* Remove tests on the deprecated precise.
* Add tests for Xenial.
* update Jessie tests to use Wheezy
* update .travis.yml
GitHub notified us about a security vulnerability in our pinned version of `urllib3` earlier this week. It doesn't affect us, but we might as well upgrade anyway. I checked:
* There are no backwards incompatible features we care about listed at https://github.com/urllib3/urllib3/blob/master/CHANGES.rst.
* urllib3's dependencies don't also need to be updated according to https://github.com/urllib3/urllib3/blob/1.24.1/setup.py.
* The hashes match when obtained from different network vantage points.
Current pinned version of cffi is 1.10.0. This version does not provide pre-compiled wheels for latest Python versions on Windows. This implies on this plateform, when certbot is installed, to compile cffi from sources.
But for that, the computer will need to have the Visual C compiler available locally. This environnement is really heavy to setup, and totally outside of the scope.
This PR updates cffi to version 1.11.5, that has the required wheels, and makes certbot installable without a full .NET dev profile.
With current code, the certbot-auto self-upgrade process can make it actually to downgrade itself, because the comparison done is an equality test between local certbot-auto version and the remote one. This is a flaw for attackers, that could make certbot-auto break itself by falsely advertising it about an old version as the latest one available.
A function is added to make a more advanced comparison between version. Certbot-auto will upgrade itself only if the local version is strictly inferior to the latest one available. For instance, a version 0.28.0 will not upgrade itself if the latest one available on internet is 0.27.1. Similarly, non-official versions like 0.28.0.dev0 will never trigger a self-upgrade, to help development workflows.
This implementation relies only on the Python distribution installed by certbot-auto (supporting 2.7+) and basic shell operations, to be compatible with any UNIX-based system.
* Check version with protection again downgrade
* Create a stable version of letsencrypt-auto to use correctly self-upgrade functionality
* Update letsencrypt-auto-source/letsencrypt-auto.template
Certbot relies heavily on bash scripts to deploy a development environment and to execute tests. This is fine for Linux systems, including Travis, but problematic for Windows machines.
This PR converts all theses scripts into Python, to make them platform independant.
As a consequence, tox-win.ini is not needed anymore, and tox can be run indifferently on Windows or on Linux using a common tox.ini. AppVeyor is updated accordingly to execute tests for acme, certbot and all dns plugins. Other tests are not executed as they are for Docker, unsupported Apache/Nginx/Postfix plugins (for now) or not relevant for Windows (explicit Linux distribution tests or pylint).
Another PR will be done on certbot website to update how a dev environment can be set up.
* Replace several shell scripts by python equivalent.
* Correction on tox coverage
* Extend usage of new python scripts
* Various corrections
* Replace venv construction bash scripts by python equivalents
* Update tox.ini
* Unicode lines to compare files
* Put modifications on letsencrypt-auto-source instead of generated scripts
* Add executable permissions for Linux.
* Merge tox win tests into main tox
* Skip lock_test on Windows
* Correct appveyor config
* Update appveyor.yml
* Explicit coverage py27 or py37
* Avoid to cover non supported certbot plugins on Windows
* Update tox.ini
* Remove specific warnings during CI
* No cover on a debug code for tests only.
* Update documentation and help script on venv/venv3.py
* Customize help message for Windows
* Quote correctly executable path with potential spaces in it.
* Copy pipstrap from upstream
We released josepy 1.1.0 a while ago to work around newer versions of cryptography deprecating some of the functionality we were using. We haven't yet upgraded our pinned josepy version though and since #6169 has landed, we're now seeing these deprecation warnings in our tests. This would be shown to certbot-auto users as well.
This PR removes these warnings by upgrading our pinned version of josepy.
* update pinned josepy version
* build leauto
* update pinned dev version of josepy
