In response to #6594. [Fixes #6594.]
To execute OCSP requests, certbot relies currently on a openssl binary execution. If openssl is not present in the PATH, the OCSP check will be silently ignored. Since version 2.4, cryptography has support for OCSP requests, without the need to have openssl binary available locally.
This PR takes advantage of it, and will use the built-in support of OCSP in cryptography for versions >= 2.4. Otherwise, fallback is done do a direct call to openssl binary, allowing oldest requirements to still work with legacy cryptography versions.
Update: requirement is now cryptography >= 2.5, to avoid to rely on a private method from cryptography.
* Implement logic using cryptography
* Working OSCP using pure cryptography
* Fix openssl usage in unit tests
* Reduce verbosity
* Add tests
* Improve naive skipIf
* Test resiliency
* Update ocsp.py
* Validate OCSP response. Unify OCSP URL get
* Improve resiliency checks, correct lint/mypy
* Improve hash selection
* Fix warnings when calling openssl bin
* Load OCSP tests assets as vectors.
* Update ocsp.py
* Protect against invalid ocsp response.
* Add checks to OCSP response
* Add more control on ocsp response
* Be lenient about assertion that next_update must be in the future, similarly to openssl.
* Construct a more advanced OCSP response mock to trigger more logic in ocsp module.
* Add test
* Refactor signature process to use crypto_util
* Fallback for cryptography 2.4
* Avoid a collision with a meteor.
* Correct method signature documentation
* Relax OCSP update interval
* Trigger built-in ocsp logic from cryptography with 2.5+
* Update pinned version of cryptography
* Update certbot/ocsp.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update ocsp.py
* Update ocsp_test.py
* Update CHANGELOG.md
* Update CHANGELOG.md
Fixes#6697.
This PR updates the version of setuptools pinned in pipstrap which works around the problems we have seen on recent OSes.
Why did I pick this version of setuptools? Because it's the latest and greatest, [supports all versions of Python that we do](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py#L173), [has been out for a month and a half without the need for a point release](https://setuptools.readthedocs.io/en/latest/history.html), and has no non-optional dependencies.
For the last point about dependencies, I don't think we have too much to worry about. setuptools did have a period between versions 34.0.0 and 36.0.0 where they tried to have normal dependencies on other packages, but reverted these changes. See their [changelog for 36.0.0](https://setuptools.readthedocs.io/en/latest/history.html#v36-0-0).
You can also compare their [current setup.py file](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py) to the [setup.py file for the currently pinned version](https://github.com/pypa/setuptools/blob/v29.0.1/setup.py) and you'll see [not much has changed](https://pastebin.com/nQj6d7D8).
Not only that, but I have successfully tested these changes on:
* ubuntu18.10
* ubuntu18.04LTS
* ubuntu16.04LTS
* ubuntu14.04LTS
* ubuntu14.04LTS_32bit
* debian9
* debian8.1
* amazonlinux-2015.09.1
* amazonlinux-2015.03.1
* RHEL7
* fedora23
* fedora29
* centos7
* centos6
* freebsd11
* macOS
* Update setuptools to 40.6.3.
* Build letsencrypt-auto.
* update changelog
* Don't use pipstrap in Dockerfile.centos6.
(cherry picked from commit b7211c3f39)
Fixes#6697.
This PR updates the version of setuptools pinned in pipstrap which works around the problems we have seen on recent OSes.
Why did I pick this version of setuptools? Because it's the latest and greatest, [supports all versions of Python that we do](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py#L173), [has been out for a month and a half without the need for a point release](https://setuptools.readthedocs.io/en/latest/history.html), and has no non-optional dependencies.
For the last point about dependencies, I don't think we have too much to worry about. setuptools did have a period between versions 34.0.0 and 36.0.0 where they tried to have normal dependencies on other packages, but reverted these changes. See their [changelog for 36.0.0](https://setuptools.readthedocs.io/en/latest/history.html#v36-0-0).
You can also compare their [current setup.py file](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py) to the [setup.py file for the currently pinned version](https://github.com/pypa/setuptools/blob/v29.0.1/setup.py) and you'll see [not much has changed](https://pastebin.com/nQj6d7D8).
Not only that, but I have successfully tested these changes on:
* ubuntu18.10
* ubuntu18.04LTS
* ubuntu16.04LTS
* ubuntu14.04LTS
* ubuntu14.04LTS_32bit
* debian9
* debian8.1
* amazonlinux-2015.09.1
* amazonlinux-2015.03.1
* RHEL7
* fedora23
* fedora29
* centos7
* centos6
* freebsd11
* macOS
* Update setuptools to 40.6.3.
* Build letsencrypt-auto.
* update changelog
* Don't use pipstrap in Dockerfile.centos6.
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so after virtualenv upgrades pip, downgrade to the pinned version.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Always download the pinned version of pip in pipstrap
* Run build.py
* Update changelog
* Remove unused variable
* Run build.py
(cherry picked from commit 9746c310d8)
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so after virtualenv upgrades pip, downgrade to the pinned version.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Always download the pinned version of pip in pipstrap
* Run build.py
* Update changelog
* Remove unused variable
* Run build.py
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
(cherry picked from commit 658b7b9d47)
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
These tests were running on Ubuntu Precise and Debian Wheezy which have reached their end of life and are no longer maintained by the respective distros. This updates the tests to a newer version of Debian and Ubuntu.
* Remove tests on the deprecated precise.
* Add tests for Xenial.
* update Jessie tests to use Wheezy
* update .travis.yml
GitHub notified us about a security vulnerability in our pinned version of `urllib3` earlier this week. It doesn't affect us, but we might as well upgrade anyway. I checked:
* There are no backwards incompatible features we care about listed at https://github.com/urllib3/urllib3/blob/master/CHANGES.rst.
* urllib3's dependencies don't also need to be updated according to https://github.com/urllib3/urllib3/blob/1.24.1/setup.py.
* The hashes match when obtained from different network vantage points.
Current pinned version of cffi is 1.10.0. This version does not provide pre-compiled wheels for latest Python versions on Windows. This implies on this plateform, when certbot is installed, to compile cffi from sources.
But for that, the computer will need to have the Visual C compiler available locally. This environnement is really heavy to setup, and totally outside of the scope.
This PR updates cffi to version 1.11.5, that has the required wheels, and makes certbot installable without a full .NET dev profile.
With current code, the certbot-auto self-upgrade process can make it actually to downgrade itself, because the comparison done is an equality test between local certbot-auto version and the remote one. This is a flaw for attackers, that could make certbot-auto break itself by falsely advertising it about an old version as the latest one available.
A function is added to make a more advanced comparison between version. Certbot-auto will upgrade itself only if the local version is strictly inferior to the latest one available. For instance, a version 0.28.0 will not upgrade itself if the latest one available on internet is 0.27.1. Similarly, non-official versions like 0.28.0.dev0 will never trigger a self-upgrade, to help development workflows.
This implementation relies only on the Python distribution installed by certbot-auto (supporting 2.7+) and basic shell operations, to be compatible with any UNIX-based system.
* Check version with protection again downgrade
* Create a stable version of letsencrypt-auto to use correctly self-upgrade functionality
* Update letsencrypt-auto-source/letsencrypt-auto.template
Certbot relies heavily on bash scripts to deploy a development environment and to execute tests. This is fine for Linux systems, including Travis, but problematic for Windows machines.
This PR converts all theses scripts into Python, to make them platform independant.
As a consequence, tox-win.ini is not needed anymore, and tox can be run indifferently on Windows or on Linux using a common tox.ini. AppVeyor is updated accordingly to execute tests for acme, certbot and all dns plugins. Other tests are not executed as they are for Docker, unsupported Apache/Nginx/Postfix plugins (for now) or not relevant for Windows (explicit Linux distribution tests or pylint).
Another PR will be done on certbot website to update how a dev environment can be set up.
* Replace several shell scripts by python equivalent.
* Correction on tox coverage
* Extend usage of new python scripts
* Various corrections
* Replace venv construction bash scripts by python equivalents
* Update tox.ini
* Unicode lines to compare files
* Put modifications on letsencrypt-auto-source instead of generated scripts
* Add executable permissions for Linux.
* Merge tox win tests into main tox
* Skip lock_test on Windows
* Correct appveyor config
* Update appveyor.yml
* Explicit coverage py27 or py37
* Avoid to cover non supported certbot plugins on Windows
* Update tox.ini
* Remove specific warnings during CI
* No cover on a debug code for tests only.
* Update documentation and help script on venv/venv3.py
* Customize help message for Windows
* Quote correctly executable path with potential spaces in it.
* Copy pipstrap from upstream
We released josepy 1.1.0 a while ago to work around newer versions of cryptography deprecating some of the functionality we were using. We haven't yet upgraded our pinned josepy version though and since #6169 has landed, we're now seeing these deprecation warnings in our tests. This would be shown to certbot-auto users as well.
This PR removes these warnings by upgrading our pinned version of josepy.
* update pinned josepy version
* build leauto
* update pinned dev version of josepy
The re stdlib module requires attrs that don't exist in the backported 3.4 version.
Technically, we are changing our install behavior beyond what is necessary. Previously, enum34 was used for 3.4 and 3.5 as well, and it happened not to conflict, but I think it's better to use the latest bug-fixed stdlib versions as long as they meet the needs of `cryptography`, which is what depends on enum34. That way, at least the various stdlib modules are guaranteed not to conflict with each other.
* Release 0.22.1
(cherry picked from commit 05c75e34e2)
* Bump version to 0.23.0
(cherry picked from commit 6fd3a57791)
* Release 0.22.2
(cherry picked from commit ea445ed11e)
* Bump version to 0.23.0
(cherry picked from commit cbe87d451c66931a084f4e513d899aae085a37d3)
* Drop support for EOL Python 2.6
* Use more helpful assertIn/NotIn instead of assertTrue/False
* Drop support for EOL Python 3.3
* Remove redundant Python 3.3 code
* Restore code for RHEL 6 and virtualenv for Py2.7
* Revert pipstrap.py to upstream
* Merge py26_packages and non_py26_packages into all_packages
* Revert changes to *-auto in root
* Update by calling letsencrypt-auto-source/build.py
* Revert permissions for pipstrap.py
* Fix rebootstrapping before venv move
* add regression test
* dedupe test
* Cleanup case when two venvs exist.
* Add clarifying comment
* Add double venv test to leauto_upgrades
* Fix logic with the help of coffee
* redirect stderr
* pass VENV_PATH through sudo
* redirect stderr
* If there's no python or there's only python2.6 on red hat systems, install python3
* Always check for python2.6
* address style, documentation, nits
* factor out all initialization code
* fix up python version return value when no python installed
* add no python error and exit
* document DeterminePythonVersion parameters
* build letsencrypt-auto
* close brace
* build leauto
* fix syntax errors
* set USE_PYTHON_3 for all cases
* rip out NOCRASH
* replace NOCRASH, update LE_PYTHON set logic
* use built-in venv for py3
* switch to LE_PYTHON not affecting bootstrap selection and not overwriting LE_PYTHON
* python3ify fetch.py
* get fetch.py working with python2 and 3
* don't verify server certificates in fetch.py HttpsGetter
* Use SSLContext and an environment variable so that our tests continue to never verify server certificates.
* typo
* build
* remove commented out code
* address review comments
* add documentation for YES_FLAG and QUIET_FLAG
* Add tests to centos6 Dockerfile to make sure we install python3 if and only if appropriate to do so.
* Remove assert_called_once from dns-route53
* Remove assert_called_once from main_test.py
* Remove assert_called() usage in dns-digitalocean
* Remove assert_called() usage in dns-route53
* Downgrade mock version in certbot-auto
* Use pipstrap to install a good version of pip
* Use pytest in cb-auto tests
* Remove nose usage in auto_test.py
* remove nose dev dep
* use pytest in test_tests
* Use pytest in tox
* Update dev dependency pinnings
* remove nose multiprocess lines
* Use pytest for coverage
* Use older py and pytest for old python versions
* Add test for Error.__str__
* pin pytest in oldest test
* Fix tests for DNS-DO plugin on py26
* Work around bug for Python 3.3
* Clarify dockerfile comments
Also, switch timeout to 30 so it has every opportunity to actually work, even in bad network weather. (I posit that people are used to 30-second timeouts.)
Stop catching URLError explicitly, since it's a subclass of the already-caught IOError.
Now we always check if we have root access if --cb-auto-has-root is not given
on the command line. This allows certbot-auto to properly acquire root when
upgrading from an older version. People upgrading from 0.18.0 to 0.18.1 may
check for root access twice, however, if root's user ID is 0, this check is
essentially a noop. If root's user ID is not 0, we'll request root access a 2nd
time during this upgrade.
* Add version number to bootstrap scripts.
* Always determine Bootstrap function and version.
* Write bootstrap version into venv.
* Add PrevBootstrapVersion function.
* Add OS bootstrapping check to phase 2.
* Differentiate -n and renew when rebootstrapping.
* Quote all environment variables.
* Correct test condition
* Add loud warning about hardcoded version list.
* s/VENV_BOOTSTRAP_VERSION/BOOTSTRAP_VERSION_PATH
* Properly handle noop bootstrap functions.
* Update comment about root usage.
* run all of certbot-auto as root
* remove other $SUDO uses from template
* remove $SUDO usage from bootstrappers
* default venv path = /opt/eff.org/certbot/venv
* Create symlinks from old default venvs
* Delete old venv path when it exists.
Also, quote expansion of paths.
* fix typo
* Separate venv_dir and le_auto_path
* Deduplicate code with test_dirs()
* Ignore cleanup errors.
This is caused by subdirectories being owned by root.
* Split test into test_phase2_upgrade.
* Rename test_dirs to temp_paths for clarity.
* Check both venvs before bootstrapping again.
* Use OLD_VENV_PATH/bin
* Preserve environment with sudo.
* Remove "esp. under sudo" comment.
* Export *VENV_PATH.
* Change check for OLD_VENV installation.
This approach better handles manually set VENV_PATH values.
* Remove SUDO_ENV.
* Print message before requesting root privileges.
* Make a function for selecting root auth method.
* Address @erikrose's feedback.
* Revert "Pin python-augeas version to avoid error with 1.0.0 (#4422)"
This reverts commit 1c51ae2588.
* make dependency-requirements
* separate certbot and dependency requirements
* fix build.py
* update hashin comment
* simplify release pinning
* separate letsencrypt dependency
* pin hashes in venv
* error out when bad things happen
* use pinned dependencies in tox
* Revert "pin hashes in venv"
This reverts commit 1cd38a9e50.
* use pip_install.sh in venv_common
* quote pip install args
* bump mock version
* say -- echo which honors quiet
* error -- echo which does not honor quiet
* switch non error echos to say
* switch error echos to error
* run letsencrypt-auto-source/build.py
* Support "certbot-auto --no-bootstrap"
* Tell people about --no-bootstrap?
* Document new certbot-auto flag in its cli help
* Rebuild
* Less variables is less variability
* Alphabetize help
* Make it extra clear we only take one branch
* Add --no-bootstrap message to experimentalbootstrap exit
* add fasteners as a dependency
* add LOCK_FILE constant
* Add lock file to Certbot
* Move code to _run_subcommand
* move lock file path into CLI_CONSTANTS
* add --lock-path flag
* move locking code to separate function
* Add TestAcquireFileLock
* assert we log
* test lock contention
* add fasteners to certbot-auto
* Use a different lock file for each test in MainTest
* add get_certbot_version.sh
* Use get_certbot_version.sh in build.py
* make test_leauto_upgrades.sh more robust
* auto upgrades break tests
* OCSP experimental is not needed anymore
* Add test_sdists.sh
* Use LE_AUTO_VERSION, not repo version
* install OS deps earlier
* use readlink not realpath
* undo changes to build.py
* Factor out version code from build.py
* Use version.py in test_sdists
* Remove get_certbot_version
* workaround setuptools breakage
* Add quiet flags to package manager invocations
Add the following flags when 'certbot-auto --quiet' is invoked:
- Add '-qq' to calls to 'apt-get' in Debian
- Add '--quiet' to calls to 'yum' or 'dnf' in CentOS or Fedora
- Add '--quiet' to calls to 'urpmi' in Mageia
- Add '--quiet' to calls to 'pkg install' in FreeBSD
* Fix $QUIET flag in bootstrappers
- Set the value of $QUIET properly (i.e. s/$QUIET/QUIET when setting the
variable) in
- deb_common.sh
- mageia_common.sh
- rpm_common.sh
- Actually use $QUIET when running $tool in rpm_common.sh
* Add handling of $QUIET to Arch and Open Suse
* Add logic to set --non-interactive if --quiet
* Add missing $QUIET_FLAG to rpm_common.sh
* Run build.py
* Limit --help to 80 cols
* Update indentation within bootstrappers
* Add $QUIET_FLAG to second call to `urpmi` (redux)
* Make certbot-auto indentation consistent
Since a majority of certbot-auto uses 2 spaces per indentation level,
made indentation in letsencrypt-auto and platform-specific shell scripts
a consistent 2 spaces
Fixes#3902
* Fix last `if` statement body in rpm_common.sh
When certbot-auto cannot find the currently installed version, output the error to the end-user, instead of not showing anything, and re-installing the virtualenv.
Fixes#4034
- The others seem to either be platforms where openssl is part of the
base system, or where I can't quickly confirm that it's safe to ask
for installation of something called "openssl".
- If we miss any platforms, the OCSP checking code in "certbot
certificates" should fail gracefully.
* Added support for shells without default variable support
* Added support for BusyBox installs that do not have `command` but has `which`
* Style fixes as suggested by reviewer
* Renamed `WHERE_IS` to `EXISTS` as suggested by review
* Removed expansion of `$LE_AUTO_SUDO` to `x` as the `-n` can check empty strings.
* Added `EXISTS` to debian bootstrap as suggested in review
* certbot-auto: Print link to doc on debugging pip install error
Also, update the doc to teach the user to workaround problem on a low
memory system.
* Correct formatting
* grep the PIP_OUT and print useful info if the problem is about memory allocation
* Fix logic on string to grep
* pin requests version in py26-oldest
* Determine requests security deps dynamically
Starting with requests 2.12, pyasn1 and ndg-httpsclient are no longer
needed to inject pyopenssl into urllib3. This change allows us to
determine whether or not these dependencies are required at install
time. If an older version of requests is used, these packages are
still installed. If a new version of requests is used, they are not
reducing the number of dependencies we have.
* Bump requests version in certbot-auto
* Use pkg_resources in activate test
Due to pip's lack of dependency resolution, the change to use
requests[extras] causes errors in acme.util_test because pkg_resources
accurately detects the "missing" dependency.
There isn't a real problem here. The problem comes from a brand new
requests and ancient pyopenssl as well as a unit test for
functionality we plan to remove in our next release. I modified
the unit test to fix the problem for now.
* Use six instead of pkg_resources for test
* Require requests<=2.11.1 in py27-oldest test
If we don't do this, we get test failures for the certbot package
which is actually a good thing! pkg_resources is catching the
unlikely but possible problem I describe in #3803 and erroring out
saying it is missing the necessary dependencies to run certbot.
Good job package resources.
* Undo changes to acme.util_test
