Fixes#6697.
This PR updates the version of setuptools pinned in pipstrap which works around the problems we have seen on recent OSes.
Why did I pick this version of setuptools? Because it's the latest and greatest, [supports all versions of Python that we do](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py#L173), [has been out for a month and a half without the need for a point release](https://setuptools.readthedocs.io/en/latest/history.html), and has no non-optional dependencies.
For the last point about dependencies, I don't think we have too much to worry about. setuptools did have a period between versions 34.0.0 and 36.0.0 where they tried to have normal dependencies on other packages, but reverted these changes. See their [changelog for 36.0.0](https://setuptools.readthedocs.io/en/latest/history.html#v36-0-0).
You can also compare their [current setup.py file](https://github.com/pypa/setuptools/blob/v40.6.3/setup.py) to the [setup.py file for the currently pinned version](https://github.com/pypa/setuptools/blob/v29.0.1/setup.py) and you'll see [not much has changed](https://pastebin.com/nQj6d7D8).
Not only that, but I have successfully tested these changes on:
* ubuntu18.10
* ubuntu18.04LTS
* ubuntu16.04LTS
* ubuntu14.04LTS
* ubuntu14.04LTS_32bit
* debian9
* debian8.1
* amazonlinux-2015.09.1
* amazonlinux-2015.03.1
* RHEL7
* fedora23
* fedora29
* centos7
* centos6
* freebsd11
* macOS
* Update setuptools to 40.6.3.
* Build letsencrypt-auto.
* update changelog
* Don't use pipstrap in Dockerfile.centos6.
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so after virtualenv upgrades pip, downgrade to the pinned version.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Always download the pinned version of pip in pipstrap
* Run build.py
* Update changelog
* Remove unused variable
* Run build.py
(cherry picked from commit 9746c310d8)
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so after virtualenv upgrades pip, downgrade to the pinned version.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Always download the pinned version of pip in pipstrap
* Run build.py
* Update changelog
* Remove unused variable
* Run build.py
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
(cherry picked from commit 658b7b9d47)
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
These tests were running on Ubuntu Precise and Debian Wheezy which have reached their end of life and are no longer maintained by the respective distros. This updates the tests to a newer version of Debian and Ubuntu.
* Remove tests on the deprecated precise.
* Add tests for Xenial.
* update Jessie tests to use Wheezy
* update .travis.yml
GitHub notified us about a security vulnerability in our pinned version of `urllib3` earlier this week. It doesn't affect us, but we might as well upgrade anyway. I checked:
* There are no backwards incompatible features we care about listed at https://github.com/urllib3/urllib3/blob/master/CHANGES.rst.
* urllib3's dependencies don't also need to be updated according to https://github.com/urllib3/urllib3/blob/1.24.1/setup.py.
* The hashes match when obtained from different network vantage points.
Current pinned version of cffi is 1.10.0. This version does not provide pre-compiled wheels for latest Python versions on Windows. This implies on this plateform, when certbot is installed, to compile cffi from sources.
But for that, the computer will need to have the Visual C compiler available locally. This environnement is really heavy to setup, and totally outside of the scope.
This PR updates cffi to version 1.11.5, that has the required wheels, and makes certbot installable without a full .NET dev profile.
With current code, the certbot-auto self-upgrade process can make it actually to downgrade itself, because the comparison done is an equality test between local certbot-auto version and the remote one. This is a flaw for attackers, that could make certbot-auto break itself by falsely advertising it about an old version as the latest one available.
A function is added to make a more advanced comparison between version. Certbot-auto will upgrade itself only if the local version is strictly inferior to the latest one available. For instance, a version 0.28.0 will not upgrade itself if the latest one available on internet is 0.27.1. Similarly, non-official versions like 0.28.0.dev0 will never trigger a self-upgrade, to help development workflows.
This implementation relies only on the Python distribution installed by certbot-auto (supporting 2.7+) and basic shell operations, to be compatible with any UNIX-based system.
* Check version with protection again downgrade
* Create a stable version of letsencrypt-auto to use correctly self-upgrade functionality
* Update letsencrypt-auto-source/letsencrypt-auto.template
Certbot relies heavily on bash scripts to deploy a development environment and to execute tests. This is fine for Linux systems, including Travis, but problematic for Windows machines.
This PR converts all theses scripts into Python, to make them platform independant.
As a consequence, tox-win.ini is not needed anymore, and tox can be run indifferently on Windows or on Linux using a common tox.ini. AppVeyor is updated accordingly to execute tests for acme, certbot and all dns plugins. Other tests are not executed as they are for Docker, unsupported Apache/Nginx/Postfix plugins (for now) or not relevant for Windows (explicit Linux distribution tests or pylint).
Another PR will be done on certbot website to update how a dev environment can be set up.
* Replace several shell scripts by python equivalent.
* Correction on tox coverage
* Extend usage of new python scripts
* Various corrections
* Replace venv construction bash scripts by python equivalents
* Update tox.ini
* Unicode lines to compare files
* Put modifications on letsencrypt-auto-source instead of generated scripts
* Add executable permissions for Linux.
* Merge tox win tests into main tox
* Skip lock_test on Windows
* Correct appveyor config
* Update appveyor.yml
* Explicit coverage py27 or py37
* Avoid to cover non supported certbot plugins on Windows
* Update tox.ini
* Remove specific warnings during CI
* No cover on a debug code for tests only.
* Update documentation and help script on venv/venv3.py
* Customize help message for Windows
* Quote correctly executable path with potential spaces in it.
* Copy pipstrap from upstream
We released josepy 1.1.0 a while ago to work around newer versions of cryptography deprecating some of the functionality we were using. We haven't yet upgraded our pinned josepy version though and since #6169 has landed, we're now seeing these deprecation warnings in our tests. This would be shown to certbot-auto users as well.
This PR removes these warnings by upgrading our pinned version of josepy.
* update pinned josepy version
* build leauto
* update pinned dev version of josepy
The re stdlib module requires attrs that don't exist in the backported 3.4 version.
Technically, we are changing our install behavior beyond what is necessary. Previously, enum34 was used for 3.4 and 3.5 as well, and it happened not to conflict, but I think it's better to use the latest bug-fixed stdlib versions as long as they meet the needs of `cryptography`, which is what depends on enum34. That way, at least the various stdlib modules are guaranteed not to conflict with each other.
* Release 0.22.1
(cherry picked from commit 05c75e34e2)
* Bump version to 0.23.0
(cherry picked from commit 6fd3a57791)
* Release 0.22.2
(cherry picked from commit ea445ed11e)
* Bump version to 0.23.0
(cherry picked from commit cbe87d451c66931a084f4e513d899aae085a37d3)
* Drop support for EOL Python 2.6
* Use more helpful assertIn/NotIn instead of assertTrue/False
* Drop support for EOL Python 3.3
* Remove redundant Python 3.3 code
* Restore code for RHEL 6 and virtualenv for Py2.7
* Revert pipstrap.py to upstream
* Merge py26_packages and non_py26_packages into all_packages
* Revert changes to *-auto in root
* Update by calling letsencrypt-auto-source/build.py
* Revert permissions for pipstrap.py
* Fix rebootstrapping before venv move
* add regression test
* dedupe test
* Cleanup case when two venvs exist.
* Add clarifying comment
* Add double venv test to leauto_upgrades
* Fix logic with the help of coffee
* redirect stderr
* pass VENV_PATH through sudo
* redirect stderr
* If there's no python or there's only python2.6 on red hat systems, install python3
* Always check for python2.6
* address style, documentation, nits
* factor out all initialization code
* fix up python version return value when no python installed
* add no python error and exit
* document DeterminePythonVersion parameters
* build letsencrypt-auto
* close brace
* build leauto
* fix syntax errors
* set USE_PYTHON_3 for all cases
* rip out NOCRASH
* replace NOCRASH, update LE_PYTHON set logic
* use built-in venv for py3
* switch to LE_PYTHON not affecting bootstrap selection and not overwriting LE_PYTHON
* python3ify fetch.py
* get fetch.py working with python2 and 3
* don't verify server certificates in fetch.py HttpsGetter
* Use SSLContext and an environment variable so that our tests continue to never verify server certificates.
* typo
* build
* remove commented out code
* address review comments
* add documentation for YES_FLAG and QUIET_FLAG
* Add tests to centos6 Dockerfile to make sure we install python3 if and only if appropriate to do so.
* Remove assert_called_once from dns-route53
* Remove assert_called_once from main_test.py
* Remove assert_called() usage in dns-digitalocean
* Remove assert_called() usage in dns-route53
* Downgrade mock version in certbot-auto
* Use pipstrap to install a good version of pip
* Use pytest in cb-auto tests
* Remove nose usage in auto_test.py
* remove nose dev dep
* use pytest in test_tests
* Use pytest in tox
* Update dev dependency pinnings
* remove nose multiprocess lines
* Use pytest for coverage
* Use older py and pytest for old python versions
* Add test for Error.__str__
* pin pytest in oldest test
* Fix tests for DNS-DO plugin on py26
* Work around bug for Python 3.3
* Clarify dockerfile comments
Also, switch timeout to 30 so it has every opportunity to actually work, even in bad network weather. (I posit that people are used to 30-second timeouts.)
Stop catching URLError explicitly, since it's a subclass of the already-caught IOError.
Now we always check if we have root access if --cb-auto-has-root is not given
on the command line. This allows certbot-auto to properly acquire root when
upgrading from an older version. People upgrading from 0.18.0 to 0.18.1 may
check for root access twice, however, if root's user ID is 0, this check is
essentially a noop. If root's user ID is not 0, we'll request root access a 2nd
time during this upgrade.
* Add version number to bootstrap scripts.
* Always determine Bootstrap function and version.
* Write bootstrap version into venv.
* Add PrevBootstrapVersion function.
* Add OS bootstrapping check to phase 2.
* Differentiate -n and renew when rebootstrapping.
* Quote all environment variables.
* Correct test condition
* Add loud warning about hardcoded version list.
* s/VENV_BOOTSTRAP_VERSION/BOOTSTRAP_VERSION_PATH
* Properly handle noop bootstrap functions.
* Update comment about root usage.
* run all of certbot-auto as root
* remove other $SUDO uses from template
* remove $SUDO usage from bootstrappers
* default venv path = /opt/eff.org/certbot/venv
* Create symlinks from old default venvs
* Delete old venv path when it exists.
Also, quote expansion of paths.
* fix typo
* Separate venv_dir and le_auto_path
* Deduplicate code with test_dirs()
* Ignore cleanup errors.
This is caused by subdirectories being owned by root.
* Split test into test_phase2_upgrade.
* Rename test_dirs to temp_paths for clarity.
* Check both venvs before bootstrapping again.
* Use OLD_VENV_PATH/bin
* Preserve environment with sudo.
* Remove "esp. under sudo" comment.
* Export *VENV_PATH.
* Change check for OLD_VENV installation.
This approach better handles manually set VENV_PATH values.
* Remove SUDO_ENV.
* Print message before requesting root privileges.
* Make a function for selecting root auth method.
* Address @erikrose's feedback.
