Fixes#6955.
This updates the Fedora version used in our test farm tests to Fedora 30. The AMI ID comes from https://alt.fedoraproject.org/cloud/ where it is listed as their standard HVM AMI for the region we use us-east-1 (US East (N. Virginia)).
Unfortunately, there were a lot of small changes required for this. The big reason for this is on Fedora, there isn't a Python 2 executable installed. In fact, there's not even an executable named python. It's just python3. Rather than installing another Python in each test, I wrote a script that the test scripts can share to figure out the different paths and names that should be used in their script. (This isn't used in test_sdists.sh because the logic is a little different.)
Other changes here worth flagging are:
I changed the name of the variable RUN_PYTHON3_TESTS in test_leauto_upgrades.sh to RUN_RHEL6_TESTS. The tests that are run when this variable is set test the upgrade from Python 2 to Python 3 on RHEL 6. I think this new name is much better now that we also have Fedora running Python 3.
I made tools/simple_http_server.py work on Python 3.
You can see tests passing with these changes at https://travis-ci.com/certbot/certbot/builds/113821476. I also ran test_tests.sh and they passed.
* Update to Fedora 30 in test farm tests.
Fedora 28 is likely to reach its EOL soon.
* Add set_python_envvars.sh.
* Fix test_apache2.sh on python3 only distros.
* Fix test_leauto_upgrades.sh on python3 systems.
* Fix certonly_standalone tests with python3 only
* Fix test_sdists.sh on python3 only distros.
* Make simple_http_server.py work on Python 3.
* add comments
Fixes#7012.
Apparently, the previous test we had here doesn't catch the case when certbot-auto prints blank lines. (I don't yet understand why so if someone does, please let me know!)
Regardless, I fixed up the test and verified it fails with the version of letsencrypt-auto in master and then fixed letsencrypt-auto so the test passes.
I ran test farm tests on the changes here and they passed on all instances.
* correct test
* fixes#7012
We made this change locally yesterday while preparing the release.
I tested this change on all AMIs currently in the test farm as well as Fedora 29 and this test passed on all instances.
Recent changes are no longer compatible with the old version of boulder used in the test farm tests. This PR updates the version of boulder used, and runs it with the new way of running boulder.
A new ami was created and is used here that uses Ubuntu 18.04, so that docker-compose can be installed more properly.
Removed commented-out section about rabbitmq that was already deprecated.
Switched to using the public DNS resolver 8.8.8.8 for the tests because the way to find the correct local resolver changed.
This PR attempts to better inform people about the problem identified at https://community.letsencrypt.org/t/certbot-auto-deployment-best-practices/91979/.
I was hesitant to add the flag --no-permissions-check, however, if there's some obscure distro out there (or custom user setup) that has a strange users and groups, I didn't want us to either:
Have to put out a bug fix release
Refuse to fix the problem and let them deal with warnings on every run
* add check_permissions.py
* Update letsencrypt-auto.template.
* build letsencrypt-auto
* Add test_permissions_warnings to auto_test
* Allow uid/gid < 1000.
* Add --no-permissions-check to Certbot.
* Add --no-permissions-check to certbot-auto.
* Add test farm test that letsencrypt-auto is quiet.
As a bonus, this new test will catch problems like the one that the caused
0.33.1 point release.
* Update CHANGELOG about permissions check.
* Update permissions comment.
* Fix symlink handling.
* Use a better default in auto_test.py.
Fixes#6861.
_venv_common.py is no longer executable. The reason for this is the venv creation logic is now different between Python 2 and Python 3. We could add code that branches on the Python version running the script, but I personally think that's unnecessary.
--setuptools and --no-site-packages is no longer passed to virtualenv either. These flags were made noops in virtualenv 1.10 and 1.7 respectively, but all of CentOS 6, 7, Debian 8+, and Ubuntu 14.04+ have new enough versions of virtualenv where these flags are no longer necessary. They are not even accepted as flags to Python 3's venv module.
Use of VENV_ARGS from test_sdists.sh was also removed because that environment variable hasn't done anything in a while.
I ran test farm tests on test_apache2.sh and test_sdists.sh with these changes and they passed.
* Fixes#6861.
* _venv_common is no longer executable.
* Reenabling OCSP cryptography support
* Refactor the validation logic of OCSP response to match the OpenSSL one
* Prepare runtime for OCSP response test
* Move unrelated test to another relevant place
* Reimplement OCSP status checks in integration tests
* Clean script
* Protect OCSP check against connection errors
* Update tests/certbot-boulder-integration.sh
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Cleaning
* Add a specific script for letsencrypt-auto install+help
* Remove inconsistent assertion
* Add executable permissions
* Remove unused variable
* Move testdata
* Corrected cleanup code
* Empty commit
Fixes#6106.
AMIs were taken from https://wiki.debian.org/Cloud/AmazonEC2Image/Stretch and https://cloud-images.ubuntu.com/locator/ec2/.
I didn't update the AMI for Fedora due to #6698.
These new AMIs pass on all test farm tests we run during the release process except Ubuntu 18.04 and 18.10 fail on test_apache2.sh. This is tracked at #6706. If this PR lands before this issue is resolved, we should list these systems as expected failures in the release notes.
Adding these AMIs slows down our tests significantly. I didn't measure it, but it feels 50-100% slower at least on my setup. I think it's worth it though.
* Update test farm targets.
* use different ubuntu ami
* Fix test_leauto_upgrades.sh on newer OSes.
Fixes#6585.
I wrote up three suggestions for fixing this at https://github.com/certbot/certbot/issues/6585#issuecomment-448054502. I took the middle approach of requiring the user to provide an ACME server to use. I like this better than the other approaches which were:
> Resolve#5938 instead of this issue.
There is value in these tests as is over the compatibility tests in that they don't use Docker and run on different OSes.
> Spin up a local Python server to return the directory object.
Trying to set up a dummy ACME server seemed hacky and error prone.
Other notes about this PR are:
* I put the Pebble setup in `tox.ini` rather than `.travis.yml` as this seems much cleaner and more natural.
* I created a new `tox` environment called `apacheconftest-with-pebble` that reuses the code from `testenv:apacheconftest` so `apacheconftest` can continue to be used with servers other than Pebble like is done in our test farm tests.
* I chose the environment variable `SERVER` for consistency with our integration tests. I chose to not give this environment variable a default but to fail fast when it is not set.
* I ran test farm tests on this PR and they passed.
Certbot relies heavily on bash scripts to deploy a development environment and to execute tests. This is fine for Linux systems, including Travis, but problematic for Windows machines.
This PR converts all theses scripts into Python, to make them platform independant.
As a consequence, tox-win.ini is not needed anymore, and tox can be run indifferently on Windows or on Linux using a common tox.ini. AppVeyor is updated accordingly to execute tests for acme, certbot and all dns plugins. Other tests are not executed as they are for Docker, unsupported Apache/Nginx/Postfix plugins (for now) or not relevant for Windows (explicit Linux distribution tests or pylint).
Another PR will be done on certbot website to update how a dev environment can be set up.
* Replace several shell scripts by python equivalent.
* Correction on tox coverage
* Extend usage of new python scripts
* Various corrections
* Replace venv construction bash scripts by python equivalents
* Update tox.ini
* Unicode lines to compare files
* Put modifications on letsencrypt-auto-source instead of generated scripts
* Add executable permissions for Linux.
* Merge tox win tests into main tox
* Skip lock_test on Windows
* Correct appveyor config
* Update appveyor.yml
* Explicit coverage py27 or py37
* Avoid to cover non supported certbot plugins on Windows
* Update tox.ini
* Remove specific warnings during CI
* No cover on a debug code for tests only.
* Update documentation and help script on venv/venv3.py
* Customize help message for Windows
* Quote correctly executable path with potential spaces in it.
* Copy pipstrap from upstream
* Drop support for EOL Python 2.6
* Use more helpful assertIn/NotIn instead of assertTrue/False
* Drop support for EOL Python 3.3
* Remove redundant Python 3.3 code
* Restore code for RHEL 6 and virtualenv for Py2.7
* Revert pipstrap.py to upstream
* Merge py26_packages and non_py26_packages into all_packages
* Revert changes to *-auto in root
* Update by calling letsencrypt-auto-source/build.py
* Revert permissions for pipstrap.py
* Fix rebootstrapping before venv move
* add regression test
* dedupe test
* Cleanup case when two venvs exist.
* Add clarifying comment
* Add double venv test to leauto_upgrades
* Fix logic with the help of coffee
* redirect stderr
* pass VENV_PATH through sudo
* redirect stderr
* Make leauto_upgrades do a real upgrade
* Cleanup vars and output
* Sleep until the server is ready
* add simple_http_server.py
* Use a randomly assigned port
* s/realpath/readlink
* wait for server before getting port
* s/localhost/all interfaces
* Use pipstrap to install a good version of pip
* Use pytest in cb-auto tests
* Remove nose usage in auto_test.py
* remove nose dev dep
* use pytest in test_tests
* Use pytest in tox
* Update dev dependency pinnings
* remove nose multiprocess lines
* Use pytest for coverage
* Use older py and pytest for old python versions
* Add test for Error.__str__
* pin pytest in oldest test
* Fix tests for DNS-DO plugin on py26
* Work around bug for Python 3.3
* Clarify dockerfile comments
* prevent regressions of #5082
* Fix test_leauto_upgrades.sh
test_leauto_upgrades.sh has been incorrectly been succeeding because while peep
doesn't work with newer versions of pip and letsencrypt-auto would crash,
the output included the version number so we reported the test as passing.
This updates letsencrypt-auto to the oldest version that still works for the
purpose of the test and sets pipefail so errors are properly reported.
* Test symlink creation in test_leauto_upgrades.sh
* Pin dependencies in test_sdists.sh.
* Fix permissions errors in test_tests.sh
* add get_certbot_version.sh
* Use get_certbot_version.sh in build.py
* make test_leauto_upgrades.sh more robust
* auto upgrades break tests
* OCSP experimental is not needed anymore
* Add test_sdists.sh
* Use LE_AUTO_VERSION, not repo version
* install OS deps earlier
* use readlink not realpath
* undo changes to build.py
* Factor out version code from build.py
* Use version.py in test_sdists
* Remove get_certbot_version
* workaround setuptools breakage
Test farm tests should test the version of letsencrypt-auto that's in the git tree, not the one from the previous release.
* Test the new leauto, not the previously released one
This removes a lot of setup code we used to need in order to get Boulder to run,
and should reduce brittleness of tests based on Boulder changes.
This also unblocks Boulder from upgrading to MariaDB 10.1 in integration tests,
since changing to 10.1 syntax for user creation would break the current certbot
integration tests (which run 10.0).
