After #6485 and #6435, it appears that there is no good reason to not fail fast when test, cover or linting scripts are executed.
This PR ensures to fail fast by invoking commands throught subprocess.check_call instead of subprocess.call, and by removing the handling of non-zero exit code at the end of theses scripts.
As now coverage on Windows is executed with thresholds, I added specific thresholds for this platform. Because some portions of code that are done for Unix platform will not be executed on Windows.
Note that coverage reports from Travis and AppVeyor are accumulated on Codecov. So if a file is covered up to 50 % on Linux, and all other parts are covered on Windows, then coverage is 100 % for Codecov.
Note: that PR also fixes the ability of coverage tests to fail if thresholds are exceeded.
* Use check_call to fail fast in all scripts related to tests/lint/coverage/deploy
* Make specific coverage threshold for windows
Certbot relies heavily on bash scripts to deploy a development environment and to execute tests. This is fine for Linux systems, including Travis, but problematic for Windows machines.
This PR converts all theses scripts into Python, to make them platform independant.
As a consequence, tox-win.ini is not needed anymore, and tox can be run indifferently on Windows or on Linux using a common tox.ini. AppVeyor is updated accordingly to execute tests for acme, certbot and all dns plugins. Other tests are not executed as they are for Docker, unsupported Apache/Nginx/Postfix plugins (for now) or not relevant for Windows (explicit Linux distribution tests or pylint).
Another PR will be done on certbot website to update how a dev environment can be set up.
* Replace several shell scripts by python equivalent.
* Correction on tox coverage
* Extend usage of new python scripts
* Various corrections
* Replace venv construction bash scripts by python equivalents
* Update tox.ini
* Unicode lines to compare files
* Put modifications on letsencrypt-auto-source instead of generated scripts
* Add executable permissions for Linux.
* Merge tox win tests into main tox
* Skip lock_test on Windows
* Correct appveyor config
* Update appveyor.yml
* Explicit coverage py27 or py37
* Avoid to cover non supported certbot plugins on Windows
* Update tox.ini
* Remove specific warnings during CI
* No cover on a debug code for tests only.
* Update documentation and help script on venv/venv3.py
* Customize help message for Windows
* Quote correctly executable path with potential spaces in it.
* Copy pipstrap from upstream
* feat(nginx): add and test new parsing abstractions
* chore(nginx parser): fix mypy and address small comments
* chore(nginx parser): clean up by removing context object
* fix integration test and lint
Main piece of #5810.
* Rename Certbot integration tests
* Remove nginx from certbot tests
* allow for running individual integration tests
* fail under 65
* Add set -e
* Track Nginx coverage and omit it from report later.
* Use INTEGRATION_TEST in script
* add INTEGRATION_TEST=all
* update min certbot percentage
Debian Wheezy is no longer supported (see https://wiki.debian.org/LTS) and
Amazon shut down their Debian 7 mirrors so let's stop trying to use Debian 7
during testing.
* Initial work on new version of --reuse-key
* Test for reuse_key
* Make lint happier
* Also test a non-dry-run reuse_key renewal
* Test --reuse-key in boulder integration test
* Better reuse-key integration testing
* Log fact that key was reused
* Test that the certificates themselves are different
* Change "oldkeypath" to "old_keypath"
* Simply appearance of new-key generation logic
* Reorganize new-key logic
* Move awk logic into TotalAndDistinctLines function
* After refactor, there's now explicit None rather than missing param
* Indicate for MyPy that key can be None
* Actually import the Optional type
* magic_typing is too magical for pylint
* Remove --no-reuse-key option
* Correct pylint test disable
The value for FAKE_DNS is now always the same because Boulder's
docker-compose hardcodes it, so skip some sed.
Set a time limit on how long we'll wait for boulder to come up.
This change will allow registering/updating account with multi emails.
Detail is enclosed in #4242
* support multi emails register
* add more test cases
* update test to unregister before register
* update create path to support multi emaill
* refactor payload updating
* fix typo
* move command line doc to another place
* revert the change for updating account registration info, added unit test
* rearrange text for consistency
See https://wiki.ubuntu.com/Releases.
Ubuntu 15.* repositories have been shut down for months now causing our tests
to always fail on these systems. While the tests on Ubuntu 12.04 still work, it
has been unsupported by Canonical for almost a year and I don't think we should
hamstring ourselves trying to continue to support it ourselves.
* Remove aauthzr instance variable
* If domain begins with fail, fail the challenge.
* test --allow-subset-of-names
* Fix renewal and add extra check
* test after hook checks
* Use newer boulder config
* Use ACMEv2 endpoint if requested
* Add v2 integration tests
* Work with unset variables
* Add wildcard issuance test
* quote domains
* Drop support for EOL Python 2.6
* Use more helpful assertIn/NotIn instead of assertTrue/False
* Drop support for EOL Python 3.3
* Remove redundant Python 3.3 code
* Restore code for RHEL 6 and virtualenv for Py2.7
* Revert pipstrap.py to upstream
* Merge py26_packages and non_py26_packages into all_packages
* Revert changes to *-auto in root
* Update by calling letsencrypt-auto-source/build.py
* Revert permissions for pipstrap.py
* Fix install verb
* Fix error message, tests and remove global pylint change
* Fix boulder integration test keypath
* Also use chain_path from lineage if not defined on CLI
* Fix rebootstrapping before venv move
* add regression test
* dedupe test
* Cleanup case when two venvs exist.
* Add clarifying comment
* Add double venv test to leauto_upgrades
* Fix logic with the help of coffee
* redirect stderr
* pass VENV_PATH through sudo
* redirect stderr
* Make leauto_upgrades do a real upgrade
* Cleanup vars and output
* Sleep until the server is ready
* add simple_http_server.py
* Use a randomly assigned port
* s/realpath/readlink
* wait for server before getting port
* s/localhost/all interfaces
* Add --delete-after-revoke flags
* Use delete_after_revoke value
* Add delete_after_revoke unit tests
* Add integration tests for delete-after-revoke.
* Use pipstrap to install a good version of pip
* Use pytest in cb-auto tests
* Remove nose usage in auto_test.py
* remove nose dev dep
* use pytest in test_tests
* Use pytest in tox
* Update dev dependency pinnings
* remove nose multiprocess lines
* Use pytest for coverage
* Use older py and pytest for old python versions
* Add test for Error.__str__
* pin pytest in oldest test
* Fix tests for DNS-DO plugin on py26
* Work around bug for Python 3.3
* Clarify dockerfile comments
Occasionally a network error prevents Docker from starting boulder causing
Travis tests to fail like it did at
https://travis-ci.org/certbot/certbot/jobs/282923098. This works around the
problem by using travis_retry to try to start boulder again if it fails.
This also moves the logic of waiting for boulder to start into
tests/boulder-fetch.sh so people running integration tests locally can benefit.
* Switching from old branch (issue-4109) and addressing changes requested
in last iteration of review:
80aa857fd2
Requested changes that were addressed:
- fixed outdated docstring for `cert_path_to_lineage`
- removed `full_archive_dir_from_renewal_conf` amd replaced with `full_archive_path` (and `_full_archive_path` -> `full_archive_path`)
- matching on `cert` instead of `chain` in `cert_manager.cert_path_to_lineage`
- fixed the two coding wrongs make a right issue
Requested changes which were not addressed:
- moving `cert_path_to_lineage` from `cert_manager` to `storage`,
as it would introduce a hard to resolve circular dependency.
* Update integration tests to handle default deletion after revoke.
* Swapping test domains.
* Addressing PR feedback:
- calling storage.full_archive_path with a ConfigObj instead of None
- Removing lambda x: x.chain_path as an option to match against
* Addressing PR feedback: it's expected that len(pattern) is 0, so handle that case properly.
* Testing of conflicting values of --cert-name and --cert-path non-interactive mode.
* Silly test for when neither certname nor cert-path were specified.
* Changing archive_files to a private function, because mocking nested functions seems impossible.
* Tests for storage.cert_path_for_cert_name
* Splitting out _acceptable_matches
* Some tests for cert_manager.cert_path_to_lineage
* Offerings to the Lint God
* Cleaner way of dealing with files in archive dirs
* Handling the two different use cases of match_and_check_overlaps a bit better
* late night syntax errors
* Test for when multiple lineages share an archive dir
* Tests for certbot.cert_manager.match_and_check_overlaps
* Removing unneeded nesting
* Lint errors that Travis caught that didn't show up locally
* Adding two integration tests (matching & mismatched --cert-path, --cert-name) based on feedback.
* Asking the user if they want to delete in interactive mode.
* Add hook dir constants
* Add hook dir properties to configuration
* test hook dir properties
* reuse certbot.util.is_exe
* Add certbot.hooks.list_hooks
* test list_hooks
* Run pre-hooks in directory
* Run deploy-hooks in directory
* Run post-hooks in directory
* Refactor and update certbot/tests/hook_test.py
* Add integration tests for hook directories
* Have Certbot create hook directories.
* document renewal hook directories
* Add --no-directory-hooks
* Make minor note about locale independent sorting
Up until now, this test was written incorrectly. In addition, when it has
failed, it simply prints error messages rather than reporting that the test
failed. This fixes both of these problems.
* prevent regressions of #5082
* Fix test_leauto_upgrades.sh
test_leauto_upgrades.sh has been incorrectly been succeeding because while peep
doesn't work with newer versions of pip and letsencrypt-auto would crash,
the output included the version number so we reported the test as passing.
This updates letsencrypt-auto to the oldest version that still works for the
purpose of the test and sets pipefail so errors are properly reported.
* Test symlink creation in test_leauto_upgrades.sh
* Pin dependencies in test_sdists.sh.
* Fix permissions errors in test_tests.sh
The program readlink used to be used in integration tests so an environment
variable was used to handle differences in the executable on different systems.
This command is no longer used though so the variable can be removed.
Might help debug #4363.
Also:
make "bash" vs "sh" explicit
move the paranoia flags (-ex) from the shebang into the body
add -u (fail on unset variables)
change _common to work with -u
remove some env vars that were no longer used
remove shebang from _common.sh because it's meant to be sourced, not run
* lineage_for_certname should return None if there is no existing renewal file
* add unit test
* add regression test to integration test
* revent boulder-start to boulder-fetch
Recommend faster-running commands with clearer output first, building up to
slow, noisy, `tox` last.
Remove outdated Boulder testing advice, and outdated Boulder testing scripts
(Vagrantfile, mac-bootstrap.sh, boulder-start.sh). Update Boulder testing
section.
Fill out "directory structure" section of contributing guide a little better.
* add get_certbot_version.sh
* Use get_certbot_version.sh in build.py
* make test_leauto_upgrades.sh more robust
* auto upgrades break tests
* OCSP experimental is not needed anymore
* Add test_sdists.sh
* Use LE_AUTO_VERSION, not repo version
* install OS deps earlier
* use readlink not realpath
* undo changes to build.py
* Factor out version code from build.py
* Use version.py in test_sdists
* Remove get_certbot_version
* workaround setuptools breakage
* use challenge type strings, not objectS
* Factor out parse_preferred_challenges
* restore pref_challs
* save pref_challs
* Make CheckCertCount more flexible
* improve integration tests
* Make pref_challs more flexible
This includes two new tests in the integration test script to check that
boulder gets the correct code. The encoding is specified in RFC5280
5.3.1. The codes that boulder will accept are a subset of that,
specified in `boulder.revocation.reasons.go`.
* Start of combined manual/script plugin
* Return str from hooks.execute, not bytes
* finish manual/script rewrite
* delete old manual and script plugins
* manually specify we want chall.token
* use consistent quotes
* specify chall for uri
* s/script/hook
* fix spacing on instructions
* remove unneeded response argument
* make achall more helpful
* simplify perform
* remove old test files
* add start of manual_tests
* fix ParseTest.test_help
* stop using manual_test_mode in cli tests
* Revert "make achall more helpful"
This reverts commit 54b01cea30.
* use bad response/validation methods on achalls
* simplify perform and cleanup environment
* finish manual tests
* Add HTTP manual hook integration test
* add manual http scripts
* Add manual DNS script integration test
* remove references to the script plugin
* they're hooks, not scripts
* add --manual-public-ip-logging-ok to integration tests
* use --pref-chall for dns integration
* does dns work?
* validate hooks
* test hook validation
* Revert "does dns work?"
This reverts commit 1224cc2961.
* busy wait in manual-http-auth
* remove DNS script test for now
* Fix challenge prefix and add trailing .
* Add comment about universal_newlines
* Fix typo from 0464ba2c4
* fix nits and typos
* Generalize HookCOmmandNotFound error
* Add verify_exe_exists
* Don't duplicate code in hooks.py
* Revert changes to hooks.py
* Use consistent hook error messages
* CLI flag for forcing interactivity
* add --force-interactive
* Add force_interactive error checking and tests
* Add force_interactive parameter to FileDisplay
* add _can_interact
* Add _return_default
* Add **unused_kwargs to NoninteractiveDisplay
* improve _return_default assertion
* Change IDisplay calls and write tests
* Document force_interactive in interfaces.py
* Don't force_interactive with a new prompt
* Warn when skipping an interaction for the first time
* add specific logger.debug message
Test farm tests should test the version of letsencrypt-auto that's in the git tree, not the one from the previous release.
* Test the new leauto, not the previously released one
Print request and response bodies with newlines, rather than all on one line.
Remove "Omitted empty field" log, which gets logged meaninglessly for every JSON
serialization.
Remove duplicated logging of responses.
Log the base64 version of the nonce, rather than turning it into bytes and
logging the backslash-escaped version of those bytes.
Only pass -vv in tests.
* only verify port is available when you actually need it
* refactor code to create achalls
* Test port checks are based on achall
* test that only the port for the requested challenge is checked in standalone
This removes a lot of setup code we used to need in order to get Boulder to run,
and should reduce brittleness of tests based on Boulder changes.
This also unblocks Boulder from upgrading to MariaDB 10.1 in integration tests,
since changing to 10.1 syntax for user creation would break the current certbot
integration tests (which run 10.0).
Remove the old bootstrap scripts, which have been subsumed into letsencrypt-auto-source/pieces/bootstrappers. They no longer need to be dispatched among manually: everyone can just run letsencrypt-auto --os-packages-only, regardless of OS.
Make the root-level le-auto a symlink to the canonical version. It should thus still work for people running le-auto from a git checkout.
