Recent changes are no longer compatible with the old version of boulder used in the test farm tests. This PR updates the version of boulder used, and runs it with the new way of running boulder.
A new ami was created and is used here that uses Ubuntu 18.04, so that docker-compose can be installed more properly.
Removed commented-out section about rabbitmq that was already deprecated.
Switched to using the public DNS resolver 8.8.8.8 for the tests because the way to find the correct local resolver changed.
This PR attempts to better inform people about the problem identified at https://community.letsencrypt.org/t/certbot-auto-deployment-best-practices/91979/.
I was hesitant to add the flag --no-permissions-check, however, if there's some obscure distro out there (or custom user setup) that has a strange users and groups, I didn't want us to either:
Have to put out a bug fix release
Refuse to fix the problem and let them deal with warnings on every run
* add check_permissions.py
* Update letsencrypt-auto.template.
* build letsencrypt-auto
* Add test_permissions_warnings to auto_test
* Allow uid/gid < 1000.
* Add --no-permissions-check to Certbot.
* Add --no-permissions-check to certbot-auto.
* Add test farm test that letsencrypt-auto is quiet.
As a bonus, this new test will catch problems like the one that the caused
0.33.1 point release.
* Update CHANGELOG about permissions check.
* Update permissions comment.
* Fix symlink handling.
* Use a better default in auto_test.py.
Fixes#6861.
_venv_common.py is no longer executable. The reason for this is the venv creation logic is now different between Python 2 and Python 3. We could add code that branches on the Python version running the script, but I personally think that's unnecessary.
--setuptools and --no-site-packages is no longer passed to virtualenv either. These flags were made noops in virtualenv 1.10 and 1.7 respectively, but all of CentOS 6, 7, Debian 8+, and Ubuntu 14.04+ have new enough versions of virtualenv where these flags are no longer necessary. They are not even accepted as flags to Python 3's venv module.
Use of VENV_ARGS from test_sdists.sh was also removed because that environment variable hasn't done anything in a while.
I ran test farm tests on test_apache2.sh and test_sdists.sh with these changes and they passed.
* Fixes#6861.
* _venv_common is no longer executable.
This PR is a part of the tls-sni-01 removal plan described in #6849.
This PR removes --tls-sni-01-port, --tls-sni-01-address and tls-sni-01/tls-sni options from --preferred-challenges. They are replace by deprecation warning, indicating that these options will be removed soon.
This deprecation, instead of complete removal, is done to avoid certbot instances to hard fail if some automated scripts still use these flags for some users.
Once this PR lands, we can remove completely theses flags in one or two release.
* Remove tls-sni related flags in cli. Add a deprecation warning instead.
* Adapt tests to cli and renewal towards tls-sni flags deprecation
* Add https_port option. Make tls_sni_01_port show a deprecation warning, but silently modify https_port if set
* Migrate last items
* Fix lint
* Update certbot/cli.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Ensure to remove all occurences of tls-sni-01
* Remove unused parameter
* Revert modifications on cli-help.txt
* Use logger.warning instead of sys.stderr
* Update the logger warning message
* Remove standalone_supported_challenges option.
* Fix order of preferred-challenges
* Remove supported_challenges property
* Fix some tests
* Fix lint
* Fix tests
* Add a changelog
* Clean code, fix test
* Update CI
* Reload
* No hard date for tls-sni removal
* Remove useless cast to list
* Update certbot/tests/renewal_test.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Add entry to the changelog
* Add entry to the changelog
* Remove tls-sni from nginx config
* Add a dedicated configuration to define what is the HTTPS port for this certbot instance.
* Correct some tests
* Reestablish default vhost creation
* Clean tls references for nginx integration tests
* Associate https_port only to tests and nginx
This PR is a part of the tls-sni-01 removal plan described in #6849.
This PR removes the tls-sni-01 challenge tests during the integration tests. The approach I used here is not to remove completely the existing test code, but simply editing it to use a http-01 challenge. Indeed:
* the current integration tests are strongly coupled, and would require more modifications that it is worth, because ...
* the certbot-ci project, that has already no tls-sni tests, will soon replace completely the current integration tests code.
Currently coverage invocation during integration tests on certbot core is misplaced, just before the OCSP statuses tests.
This PR move back the coverage invocation at the end of the script.
If you execute `tests/lock_test.py` or `tox -e integration` on a fairly recent machine, you will get the following error during tests executing against a live Nginx instance:
```
no "ssl_certificate" is defined in server listening on SSL port while SSL handshaking, client: x.x.x.x, server: y:y:y:y:z
```
Indeed, having no defined ssl certificate for a ssl port would inevitably lead to an error during the handshake SSL process between a client and this mis-configured nginx instance.
However it was not a problem one year before, because the handshake was not occurring in practice: the test just need to have a nginx started, and then immediately proceed to modify the configuration with a correct SSL setup. And nginx was able to start with a mis-configuration on SSL.
But then this fix has been done: https://trac.nginx.org/nginx/ticket/178
Basically with this, validation of the configuration is done during nginx startup, that will refuse to start with invalid configuration on SSL. Consequently, all related tests are failing with a sufficiently up-to-date nginx. For now, it is not seen on Travis because Ubuntu Trusty is used, with an old Nginx.
The PR fixes that, by generating on the fly self-signed certificates in the two impacted tests, and pushing the right parameters in the Nginx configuration.
* Fix nginx configuration with self-signed certificates generated on the fly
* Fix lint/mypy
* Fix old cryptography
* Unattended openssl
* Update lock_test.py
* Reenabling OCSP cryptography support
* Refactor the validation logic of OCSP response to match the OpenSSL one
* Prepare runtime for OCSP response test
* Move unrelated test to another relevant place
* Reimplement OCSP status checks in integration tests
* Clean script
* Protect OCSP check against connection errors
* Update tests/certbot-boulder-integration.sh
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Cleaning
* Add a specific script for letsencrypt-auto install+help
* Remove inconsistent assertion
* Add executable permissions
* Remove unused variable
* Move testdata
* Corrected cleanup code
* Empty commit
This PR updates and fixes `pebble-fetch.sh` considering latest improvements done on Pebble, to start a working instance.
* Fix the pebble fetch script
* Update pebble-fetch.sh
* Update tox.ini
This PR reactivates tls-sni-01 challenges on recent Boulder versions checkout for integration tests. This allows to continue testing this challenge until it is officially dropped from server (Boulder) and client (Certbot).
Reverts #6679.
Fixes#6106.
AMIs were taken from https://wiki.debian.org/Cloud/AmazonEC2Image/Stretch and https://cloud-images.ubuntu.com/locator/ec2/.
I didn't update the AMI for Fedora due to #6698.
These new AMIs pass on all test farm tests we run during the release process except Ubuntu 18.04 and 18.10 fail on test_apache2.sh. This is tracked at #6706. If this PR lands before this issue is resolved, we should list these systems as expected failures in the release notes.
Adding these AMIs slows down our tests significantly. I didn't measure it, but it feels 50-100% slower at least on my setup. I think it's worth it though.
* Update test farm targets.
* use different ubuntu ami
* Fix test_leauto_upgrades.sh on newer OSes.
* Remove tls-sni-01 challenges in integration tests
* Remove the tls-sni test in the less invasive way
* Correct code coverage from tls-sni logic not been tested anymore.
* Update certbot-boulder-integration.sh
Fixes#6585.
I wrote up three suggestions for fixing this at https://github.com/certbot/certbot/issues/6585#issuecomment-448054502. I took the middle approach of requiring the user to provide an ACME server to use. I like this better than the other approaches which were:
> Resolve#5938 instead of this issue.
There is value in these tests as is over the compatibility tests in that they don't use Docker and run on different OSes.
> Spin up a local Python server to return the directory object.
Trying to set up a dummy ACME server seemed hacky and error prone.
Other notes about this PR are:
* I put the Pebble setup in `tox.ini` rather than `.travis.yml` as this seems much cleaner and more natural.
* I created a new `tox` environment called `apacheconftest-with-pebble` that reuses the code from `testenv:apacheconftest` so `apacheconftest` can continue to be used with servers other than Pebble like is done in our test farm tests.
* I chose the environment variable `SERVER` for consistency with our integration tests. I chose to not give this environment variable a default but to fail fast when it is not set.
* I ran test farm tests on this PR and they passed.
#6636 broke [test-everything tests](https://travis-ci.org/certbot/certbot/builds/475173804) because `_common.sh` is a common file shared between Certbot and Nginx integration tests and `--no-random-sleep-on-renew` isn't defined for the version of Certbot used in the "oldest" integration tests.
This PR adds code to `_common.sh` to check the Certbot version and if it's new enough, add `--no-random-sleep-on-renew` to the command line. I repurposed `$store_flags` and stopped exporting it because it's not used anywhere outside of this file.
Other approaches I considered and decided against were:
1. Adding this flag in `certbot-boulder-integration.sh`. I decided against it because it's setting us up for the same problem in the future if the oldest version of Certbot is upgraded in the Nginx tests and we call `certbot renew`.
2. Just upgrading the oldest version of Certbot required by Nginx to avoid these issues. While this would work (with perhaps some unnecessary burden for our packagers), I think it's avoiding the real problem here which should now be able to addressed easily with the addition of `$other_flags` and `version_at_least`.
* Add version_at_least().
* Conditionally disable sleep.
* Consolidate store_flags and other_flags.
* update comments
Not updating the guidelines in using.rst-- I don't want to encourage people to use this. now that Certbot preserves gid, the better way to set permissions is to change the group permisisons!
* Preserve other-read bit on private keys too
* fix integration test
* fix and rename permission routines in integration tests
* Setup an integration tests env against Pebble, that enforce post-as-get
* Implement POST-as-GET requests, with fallback to GET.
* Fix unit tests
* Fix coverage.
* Fix or ignore lint errors
* Corrections after review
* Correct test
* Try a simple delegate approach
* Add a test
* Simplify test mocking
* Clean comment
Fixes#1473.
writes privkey.pem to 0600 by default for new lineages
on renewals where a new privkey is generated, preserves group mode and gid
Things this PR does not do:
we talked about forcing 0600 on privkeys when a Certbot upgrade is detected. Instead, this PR only creates new lineages with the more restrictive permission to prevent renewal breakages.
this doesn't solve many of the problems mentioned in #1473 that are not directly related to the title issue!
* safe_open on archive keyfiles
* keep group from current lineage
* clean up integration test
* safe_open can follow symlinks
* fix tests on windows, maybe
* Address Brad's comments
* Revert changes to safe_open
* Test chown is called when saving new key
* Reorder chown operation
* Changelog and documentation
* Fix documentation style
After #6485 and #6435, it appears that there is no good reason to not fail fast when test, cover or linting scripts are executed.
This PR ensures to fail fast by invoking commands throught subprocess.check_call instead of subprocess.call, and by removing the handling of non-zero exit code at the end of theses scripts.
As now coverage on Windows is executed with thresholds, I added specific thresholds for this platform. Because some portions of code that are done for Unix platform will not be executed on Windows.
Note that coverage reports from Travis and AppVeyor are accumulated on Codecov. So if a file is covered up to 50 % on Linux, and all other parts are covered on Windows, then coverage is 100 % for Codecov.
Note: that PR also fixes the ability of coverage tests to fail if thresholds are exceeded.
* Use check_call to fail fast in all scripts related to tests/lint/coverage/deploy
* Make specific coverage threshold for windows
Certbot relies heavily on bash scripts to deploy a development environment and to execute tests. This is fine for Linux systems, including Travis, but problematic for Windows machines.
This PR converts all theses scripts into Python, to make them platform independant.
As a consequence, tox-win.ini is not needed anymore, and tox can be run indifferently on Windows or on Linux using a common tox.ini. AppVeyor is updated accordingly to execute tests for acme, certbot and all dns plugins. Other tests are not executed as they are for Docker, unsupported Apache/Nginx/Postfix plugins (for now) or not relevant for Windows (explicit Linux distribution tests or pylint).
Another PR will be done on certbot website to update how a dev environment can be set up.
* Replace several shell scripts by python equivalent.
* Correction on tox coverage
* Extend usage of new python scripts
* Various corrections
* Replace venv construction bash scripts by python equivalents
* Update tox.ini
* Unicode lines to compare files
* Put modifications on letsencrypt-auto-source instead of generated scripts
* Add executable permissions for Linux.
* Merge tox win tests into main tox
* Skip lock_test on Windows
* Correct appveyor config
* Update appveyor.yml
* Explicit coverage py27 or py37
* Avoid to cover non supported certbot plugins on Windows
* Update tox.ini
* Remove specific warnings during CI
* No cover on a debug code for tests only.
* Update documentation and help script on venv/venv3.py
* Customize help message for Windows
* Quote correctly executable path with potential spaces in it.
* Copy pipstrap from upstream
* feat(nginx): add and test new parsing abstractions
* chore(nginx parser): fix mypy and address small comments
* chore(nginx parser): clean up by removing context object
* fix integration test and lint
Main piece of #5810.
* Rename Certbot integration tests
* Remove nginx from certbot tests
* allow for running individual integration tests
* fail under 65
* Add set -e
* Track Nginx coverage and omit it from report later.
* Use INTEGRATION_TEST in script
* add INTEGRATION_TEST=all
* update min certbot percentage
Debian Wheezy is no longer supported (see https://wiki.debian.org/LTS) and
Amazon shut down their Debian 7 mirrors so let's stop trying to use Debian 7
during testing.
* Initial work on new version of --reuse-key
* Test for reuse_key
* Make lint happier
* Also test a non-dry-run reuse_key renewal
* Test --reuse-key in boulder integration test
* Better reuse-key integration testing
* Log fact that key was reused
* Test that the certificates themselves are different
* Change "oldkeypath" to "old_keypath"
* Simply appearance of new-key generation logic
* Reorganize new-key logic
* Move awk logic into TotalAndDistinctLines function
* After refactor, there's now explicit None rather than missing param
* Indicate for MyPy that key can be None
* Actually import the Optional type
* magic_typing is too magical for pylint
* Remove --no-reuse-key option
* Correct pylint test disable
The value for FAKE_DNS is now always the same because Boulder's
docker-compose hardcodes it, so skip some sed.
Set a time limit on how long we'll wait for boulder to come up.
This change will allow registering/updating account with multi emails.
Detail is enclosed in #4242
* support multi emails register
* add more test cases
* update test to unregister before register
* update create path to support multi emaill
* refactor payload updating
* fix typo
* move command line doc to another place
* revert the change for updating account registration info, added unit test
* rearrange text for consistency
See https://wiki.ubuntu.com/Releases.
Ubuntu 15.* repositories have been shut down for months now causing our tests
to always fail on these systems. While the tests on Ubuntu 12.04 still work, it
has been unsupported by Canonical for almost a year and I don't think we should
hamstring ourselves trying to continue to support it ourselves.
* Remove aauthzr instance variable
* If domain begins with fail, fail the challenge.
* test --allow-subset-of-names
* Fix renewal and add extra check
* test after hook checks
* Use newer boulder config
* Use ACMEv2 endpoint if requested
* Add v2 integration tests
* Work with unset variables
* Add wildcard issuance test
* quote domains
* Drop support for EOL Python 2.6
* Use more helpful assertIn/NotIn instead of assertTrue/False
* Drop support for EOL Python 3.3
* Remove redundant Python 3.3 code
* Restore code for RHEL 6 and virtualenv for Py2.7
* Revert pipstrap.py to upstream
* Merge py26_packages and non_py26_packages into all_packages
* Revert changes to *-auto in root
* Update by calling letsencrypt-auto-source/build.py
* Revert permissions for pipstrap.py
* Fix install verb
* Fix error message, tests and remove global pylint change
* Fix boulder integration test keypath
* Also use chain_path from lineage if not defined on CLI
* Fix rebootstrapping before venv move
* add regression test
* dedupe test
* Cleanup case when two venvs exist.
* Add clarifying comment
* Add double venv test to leauto_upgrades
* Fix logic with the help of coffee
* redirect stderr
* pass VENV_PATH through sudo
* redirect stderr
* Make leauto_upgrades do a real upgrade
* Cleanup vars and output
* Sleep until the server is ready
* add simple_http_server.py
* Use a randomly assigned port
* s/realpath/readlink
* wait for server before getting port
* s/localhost/all interfaces
* Add --delete-after-revoke flags
* Use delete_after_revoke value
* Add delete_after_revoke unit tests
* Add integration tests for delete-after-revoke.
* Use pipstrap to install a good version of pip
* Use pytest in cb-auto tests
* Remove nose usage in auto_test.py
* remove nose dev dep
* use pytest in test_tests
* Use pytest in tox
* Update dev dependency pinnings
* remove nose multiprocess lines
* Use pytest for coverage
* Use older py and pytest for old python versions
* Add test for Error.__str__
* pin pytest in oldest test
* Fix tests for DNS-DO plugin on py26
* Work around bug for Python 3.3
* Clarify dockerfile comments
Occasionally a network error prevents Docker from starting boulder causing
Travis tests to fail like it did at
https://travis-ci.org/certbot/certbot/jobs/282923098. This works around the
problem by using travis_retry to try to start boulder again if it fails.
This also moves the logic of waiting for boulder to start into
tests/boulder-fetch.sh so people running integration tests locally can benefit.
* Switching from old branch (issue-4109) and addressing changes requested
in last iteration of review:
80aa857fd2
Requested changes that were addressed:
- fixed outdated docstring for `cert_path_to_lineage`
- removed `full_archive_dir_from_renewal_conf` amd replaced with `full_archive_path` (and `_full_archive_path` -> `full_archive_path`)
- matching on `cert` instead of `chain` in `cert_manager.cert_path_to_lineage`
- fixed the two coding wrongs make a right issue
Requested changes which were not addressed:
- moving `cert_path_to_lineage` from `cert_manager` to `storage`,
as it would introduce a hard to resolve circular dependency.
* Update integration tests to handle default deletion after revoke.
* Swapping test domains.
* Addressing PR feedback:
- calling storage.full_archive_path with a ConfigObj instead of None
- Removing lambda x: x.chain_path as an option to match against
* Addressing PR feedback: it's expected that len(pattern) is 0, so handle that case properly.
* Testing of conflicting values of --cert-name and --cert-path non-interactive mode.
* Silly test for when neither certname nor cert-path were specified.
* Changing archive_files to a private function, because mocking nested functions seems impossible.
* Tests for storage.cert_path_for_cert_name
* Splitting out _acceptable_matches
* Some tests for cert_manager.cert_path_to_lineage
* Offerings to the Lint God
* Cleaner way of dealing with files in archive dirs
* Handling the two different use cases of match_and_check_overlaps a bit better
* late night syntax errors
* Test for when multiple lineages share an archive dir
* Tests for certbot.cert_manager.match_and_check_overlaps
* Removing unneeded nesting
* Lint errors that Travis caught that didn't show up locally
* Adding two integration tests (matching & mismatched --cert-path, --cert-name) based on feedback.
* Asking the user if they want to delete in interactive mode.
* Add hook dir constants
* Add hook dir properties to configuration
* test hook dir properties
* reuse certbot.util.is_exe
* Add certbot.hooks.list_hooks
* test list_hooks
* Run pre-hooks in directory
* Run deploy-hooks in directory
* Run post-hooks in directory
* Refactor and update certbot/tests/hook_test.py
* Add integration tests for hook directories
* Have Certbot create hook directories.
* document renewal hook directories
* Add --no-directory-hooks
* Make minor note about locale independent sorting
Up until now, this test was written incorrectly. In addition, when it has
failed, it simply prints error messages rather than reporting that the test
failed. This fixes both of these problems.
* prevent regressions of #5082
* Fix test_leauto_upgrades.sh
test_leauto_upgrades.sh has been incorrectly been succeeding because while peep
doesn't work with newer versions of pip and letsencrypt-auto would crash,
the output included the version number so we reported the test as passing.
This updates letsencrypt-auto to the oldest version that still works for the
purpose of the test and sets pipefail so errors are properly reported.
* Test symlink creation in test_leauto_upgrades.sh
* Pin dependencies in test_sdists.sh.
* Fix permissions errors in test_tests.sh
The program readlink used to be used in integration tests so an environment
variable was used to handle differences in the executable on different systems.
This command is no longer used though so the variable can be removed.
Might help debug #4363.
Also:
make "bash" vs "sh" explicit
move the paranoia flags (-ex) from the shebang into the body
add -u (fail on unset variables)
change _common to work with -u
remove some env vars that were no longer used
remove shebang from _common.sh because it's meant to be sourced, not run
* lineage_for_certname should return None if there is no existing renewal file
* add unit test
* add regression test to integration test
* revent boulder-start to boulder-fetch
Recommend faster-running commands with clearer output first, building up to
slow, noisy, `tox` last.
Remove outdated Boulder testing advice, and outdated Boulder testing scripts
(Vagrantfile, mac-bootstrap.sh, boulder-start.sh). Update Boulder testing
section.
Fill out "directory structure" section of contributing guide a little better.
* add get_certbot_version.sh
* Use get_certbot_version.sh in build.py
* make test_leauto_upgrades.sh more robust
* auto upgrades break tests
* OCSP experimental is not needed anymore
* Add test_sdists.sh
* Use LE_AUTO_VERSION, not repo version
* install OS deps earlier
* use readlink not realpath
* undo changes to build.py
* Factor out version code from build.py
* Use version.py in test_sdists
* Remove get_certbot_version
* workaround setuptools breakage
* use challenge type strings, not objectS
* Factor out parse_preferred_challenges
* restore pref_challs
* save pref_challs
* Make CheckCertCount more flexible
* improve integration tests
* Make pref_challs more flexible
This includes two new tests in the integration test script to check that
boulder gets the correct code. The encoding is specified in RFC5280
5.3.1. The codes that boulder will accept are a subset of that,
specified in `boulder.revocation.reasons.go`.
* Start of combined manual/script plugin
* Return str from hooks.execute, not bytes
* finish manual/script rewrite
* delete old manual and script plugins
* manually specify we want chall.token
* use consistent quotes
* specify chall for uri
* s/script/hook
* fix spacing on instructions
* remove unneeded response argument
* make achall more helpful
* simplify perform
* remove old test files
* add start of manual_tests
* fix ParseTest.test_help
* stop using manual_test_mode in cli tests
* Revert "make achall more helpful"
This reverts commit 54b01cea30.
* use bad response/validation methods on achalls
* simplify perform and cleanup environment
* finish manual tests
* Add HTTP manual hook integration test
* add manual http scripts
* Add manual DNS script integration test
* remove references to the script plugin
* they're hooks, not scripts
* add --manual-public-ip-logging-ok to integration tests
* use --pref-chall for dns integration
* does dns work?
* validate hooks
* test hook validation
* Revert "does dns work?"
This reverts commit 1224cc2961.
* busy wait in manual-http-auth
* remove DNS script test for now
* Fix challenge prefix and add trailing .
* Add comment about universal_newlines
* Fix typo from 0464ba2c4
* fix nits and typos
* Generalize HookCOmmandNotFound error
* Add verify_exe_exists
* Don't duplicate code in hooks.py
* Revert changes to hooks.py
* Use consistent hook error messages
* CLI flag for forcing interactivity
* add --force-interactive
* Add force_interactive error checking and tests
* Add force_interactive parameter to FileDisplay
* add _can_interact
* Add _return_default
* Add **unused_kwargs to NoninteractiveDisplay
* improve _return_default assertion
* Change IDisplay calls and write tests
* Document force_interactive in interfaces.py
* Don't force_interactive with a new prompt
* Warn when skipping an interaction for the first time
* add specific logger.debug message
Test farm tests should test the version of letsencrypt-auto that's in the git tree, not the one from the previous release.
* Test the new leauto, not the previously released one
Print request and response bodies with newlines, rather than all on one line.
Remove "Omitted empty field" log, which gets logged meaninglessly for every JSON
serialization.
Remove duplicated logging of responses.
Log the base64 version of the nonce, rather than turning it into bytes and
logging the backslash-escaped version of those bytes.
Only pass -vv in tests.
* only verify port is available when you actually need it
* refactor code to create achalls
* Test port checks are based on achall
* test that only the port for the requested challenge is checked in standalone
This removes a lot of setup code we used to need in order to get Boulder to run,
and should reduce brittleness of tests based on Boulder changes.
This also unblocks Boulder from upgrading to MariaDB 10.1 in integration tests,
since changing to 10.1 syntax for user creation would break the current certbot
integration tests (which run 10.0).
