* acme: re-populate uri in deactivate_authorization
* Use fresh authorizations in dry runs
--dry-run now deactivates 'valid' authorizations if it encounters them
when creating a new order.
Resolves#5116.
* remove unused code
* typo in local-oldest-requirements
* better error handling
* certbot-ci: AUTHREUSE to 100 + unskip dry-run test
* improve test coverage for error cases
* restore newline to local-oldest-requirements.txt
I tried to finish up #7214 by removing the code in acme but we can't really do that until #7478 is resolved which we cannot do until we release 0.40.0.
Since we have to wait, this PR adds deprecation warnings for code that uses the TLS-SNI-01 code or was only used by the long deprecated TLS-SNI-01 code.
I'd like this PR to land before our next release.
* Deprecate more code related to TLS-SNI-01.
* Assert about warning message.
Fixes#7368.
When updating the changelog, I replaced the line about running tests on Python 3.8 because I personally think that support for Python 3.8 is the most relevant information for our users/packagers about our changes in this area.
* List support for Python 3.8.
* Update changelog.
Resolves#4945. First PR in order to address #5116.
* acme: Implement authz deactivation
Resolves#4945
* update AUTHORS and CHANGELOG
* typos in mypy annotations
* formatting: missing newline
* improve test_deactivate_authorization
* improve deactivate_authorization
* test: s/STATUS_INVALID/STATUS_DEACTIVATED/
* simplify dict to keyword argument
* acme: add UpdateAuthorization
* acme: use UpdateAuthorization in deactivate_authz
and add mypy annotation
This allows deactivate_authorization to succeed for both ACME v1
and v2 servers.
Fixes#6974.
This PR removes the fallback that consists in retrying to send the keyAuthorization field during a challenge request in case of malformed request.
* Remove keyAuthorization fallback dump in challenges response
* Correct import
* Add changelog entry
This PR is a part of the tls-sni-01 removal plan described in #6849.
As `acme` is a library, we need to put some efforts to make a decent deprecation path before totally removing tls-sni in it. While initialization of `acme.challenges.TLSSNI01` was already creating deprecation warning, not all cases were covered.
For instance, and innocent call like this ...
```python
if not isinstance(challenge, acme.challenges.TLSSNI01):
print('I am not using this TLS-SNI deprecated stuff, what could possibly go wrong?')
```
... would break if we suddenly remove all objects related to this challenge.
So, I use the _Deprecator Warning Machine, Let's Pacify this Technical Debt_ (Guido ®), to make `acme.challenges` and `acme.standalone` patch themselves, and display a deprecation warning on stderr for any access to the tls-sni challenge objects.
No dev should be able to avoid the deprecation warning. I set the deprecation warning in the idea to remove the code on `0.34.0`, but the exact deprecation window is open to discussion of course.
* Modules challenges and standalone patch themselves to generated deprecation warning when tls-sni related objects are accessed.
* Correct unit tests
* Correct lint
* Update challenges_test.py
* Correct lint
* Fix an error during tests
* Update coverage
* Use multiprocessing for coverage
* Add coverage
* Update test_util.py
* Factor the logic about global deprecation warning when accessing TLS-SNI-01 attributes
* Fix coverage
* Add comment for cryptography example.
* Use warnings.
* Add a changelog
* Fix deprecation during tests
* Reload
* Update acme/acme/__init__.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update CHANGELOG.md
* Pick a random free port.
Fixes#6755.
POSTing the `keyAuthorization` in a JWS token when answering an ACME challenge, has been deprecated for some time now. Indeed, this is superfluous as the request is already authentified by the JWS signature.
Boulder still accepts to see this field in the JWS token, and ignore it. Pebble in non strict mode also. But Pebble in strict mode refuses the request, to prepare complete removal of this field in ACME v2.
Certbot still sends the `keyAuthorization` field. This PR removes it, and makes Certbot compliant with current ACME v2 protocol, and so Pebble in strict mode.
See also [letsencrypt/pebble#192](https://github.com/letsencrypt/pebble/issues/192) for implementation details server side.
* New implementation, with a fallback.
* Add deprecation on changelog
* Update acme/acme/client.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Fix an instance parameter
* Update changelog, extend coverage
* Update comment
* Add unit tests on keyAuthorization dump
* Update acme/acme/client.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Restrict the magic of setting a variable in immutable object in one place. Make a soon to be removed method private.
* Add acme library usage example
Create, edit and deactivate account.
Setup and perform http-01 challenge.
Issue, renew and revoke certificate.
* Adapt example to ACME-v2 and exclude data persistence
The code to persist/load data would length this example and distract from what is actually important.
* Fix domain names and e-mail addresses
* Remove unnecessary license header
This usage example is under the license for the acme package.
* Remove logging information
The code will be mostly read by developers, so simplify the logging info into comments.
* Revert abstraction of simple methods
All methods that are used only once in this example were expanded into the main code in order to make the process more explicit.
* Fix missing URL suffix
* Improve aesthetics and reorganize workflow
Also make words capitalization consistent and improve comments.
No complaints from pep8.
So merging the study from @bmw and me, here is what happened.
Each invocation of `certbot.logger.post_arg_parse_setup` create a file handler on `letsencrypt.log`. This function also set an atexit handler invoking `logger.shutdown()`, that have the effect to close all logger file handler not already closed at this point. This method is supposed to be called when a python process is close to exit, because it makes all logger unable to write new logs on any handler.
Before #6667 and this PR, for tests, the atexit handle would be triggered only at the end of the pytest process. It means that each test that launches `certbot.logger.post_arg_parse_setup` add a new file handler. These tests were typically connecting the file handler on a `letsencrypt.log` located in a temporary directory, and this directory and content was wipped out at each test tearDown. As a consequence, the file handles, not cleared from the logger, were accumulating in the logger, with all of them connected to a deleted file log, except the last one that was just created by the current test. Considering the number of tests concerned, there were ~300 file handler at the end of pytest execution.
One can see that, on prior #6667, by calling `print(logger.getLogger().handlers` on the `tearDown` of these tests, and see the array growing at each test execution.
Even if this represent a memory leak, this situation was not really a problem on Linux: because a file can be deleted before it is closed, it was only meaning that a given invocation of `logger.debug` for instance, during the tests, was written in 300 log files. The overhead is negligeable. On Windows however, the file handlers were failing because you cannot delete a file before it is closed.
It was one of the reason for #6667, that added a call to `logging.shutdown()` at each test tearDown, with the consequence to close all file handlers. At this point, Linux is not happy anymore. Any call to `logger.warn` will generate an error for each closed file handler. As a file handler is added for each test, the number of errors grows on each test, following an arithmetical suite divergence.
On `test_sdists.py`, that is using the bare setuptools test suite without output capturing, we can see the damages. The total output takes 216000 lines, and 23000 errors are generated. A decent machine can support this load, but a not a small AWS instance, that is crashing during the execution. Even with pytest, the captured output and the memory leak become so large that segfaults are generated.
On the current PR, the problem is solved, by resetting the file handlers array on the logging system on each test tearDown. So each fileHandler is properly closed, and removed from the stack. They do not participate anymore in the logging system, and can be garbage collected. Then we stay on always one file handler opened at any time, and tests can succeed on AWS instances.
For the record, here is all the places where the logging system is called and fail if there is still file handlers closed but not cleaned (extracted from the original huge output before correction):
```
Logged from file account.py, line 116
Logged from file account.py, line 178
Logged from file client.py, line 166
Logged from file client.py, line 295
Logged from file client.py, line 415
Logged from file client.py, line 422
Logged from file client.py, line 480
Logged from file client.py, line 503
Logged from file client.py, line 540
Logged from file client.py, line 601
Logged from file client.py, line 622
Logged from file client.py, line 750
Logged from file cli.py, line 220
Logged from file cli.py, line 226
Logged from file crypto_util.py, line 101
Logged from file crypto_util.py, line 127
Logged from file crypto_util.py, line 147
Logged from file crypto_util.py, line 261
Logged from file crypto_util.py, line 283
Logged from file crypto_util.py, line 307
Logged from file crypto_util.py, line 336
Logged from file disco.py, line 116
Logged from file disco.py, line 124
Logged from file disco.py, line 134
Logged from file disco.py, line 138
Logged from file disco.py, line 141
Logged from file dns_common_lexicon.py, line 45
Logged from file dns_common_lexicon.py, line 61
Logged from file dns_common_lexicon.py, line 67
Logged from file dns_common.py, line 316
Logged from file dns_common.py, line 64
Logged from file eff.py, line 60
Logged from file eff.py, line 73
Logged from file error_handler.py, line 105
Logged from file error_handler.py, line 110
Logged from file error_handler.py, line 87
Logged from file hooks.py, line 248
Logged from file main.py, line 1071
Logged from file main.py, line 1075
Logged from file main.py, line 1189
Logged from file ops.py, line 122
Logged from file ops.py, line 325
Logged from file ops.py, line 338
Logged from file reporter.py, line 55
Logged from file selection.py, line 110
Logged from file selection.py, line 118
Logged from file selection.py, line 123
Logged from file selection.py, line 176
Logged from file selection.py, line 231
Logged from file selection.py, line 310
Logged from file selection.py, line 66
Logged from file standalone.py, line 101
Logged from file standalone.py, line 88
Logged from file standalone.py, line 97
Logged from file standalone.py, line 98
Logged from file storage.py, line 52
Logged from file storage.py, line 59
Logged from file storage.py, line 75
Logged from file util.py, line 56
Logged from file webroot.py, line 165
Logged from file webroot.py, line 186
Logged from file webroot.py, line 187
Logged from file webroot.py, line 204
Logged from file webroot.py, line 223
Logged from file webroot.py, line 234
Logged from file webroot.py, line 235
Logged from file webroot.py, line 237
Logged from file webroot.py, line 91
```
* Reapply #6667
* Make setuptools delegates tests execution to pytest, like in acme module.
* Clean handlers at each tearDown to avoid memory leaks.
* Update changelog
The existing `acme.TLSALPN01` challenge class did not have
a `response_cls`, meaning it was not possible to use a tls-alpn-01
challenge with `client.answer_challenge`.
To support the above a simple `TLSALPN01Response` class is added that
doesn't provide the ability to solve a tls-alpn-01 challenge end to end
(e.g. generating and serving the correct response certificate) but that
does allow the challenge to be initiated. This is sufficient for users
that have set up the challenge response independent of the `acme`
module code.
Resolves#6676
This PR passes the CERTBOT_NO_PIN environment variable to the unit tests tox envs. By setting CERTBOT_NO_PIN to 1 before executing a given tox env, certbot dependencies will be installed at their latest version instead of the usual pinned version.
I also moved the unpin logic one layer below to allow it to be used potentially more widely, and avoid unnecessary merging constraints operation in this case.
As warnings are errors now, latest versions of Python will break now the tests, because collections launch a warning when some classes are imported from collections instead of collections.abc. Certbot code is patched, and warning is ignored for now, because a lot of third party libraries still depend on this behavior.
* Allow to execute a tox target without pinned dependencies
* Correct lint
* Retrigger build.
* Remove debug code
* Added test against unpinned dependencies from test-everything-unpinned-dependencies branch
* Remove duplicated assertion to pass TRAVIS and APPVEYOR in default tox environment.
I observed that the current set of oldest requirements do not correspond to any environment, except the specific Xenial image in Travis CI (and standard Xenial containers will also fail).
It is because the requirements make cryptography and requests fail against standard libraries available in the typical Linux distributions that are targeted by the oldest requirements approach (Centos 6, Centos 7, Xenial, Jessie).
This PR fixes that, by aligning the minimal version requirements of cryptography and requests to the maximal versions that are available on Centos 6. Centos 7, Jessie and Xenial stay unusable with oldest requirements for other reasons, but at least one old and supported Linux distribution is able to run the tests with oldest requirements out of the box.
A test is also corrected to match the expected error message that old versions of urllib3 will raise.
Fixes the problem at https://github.com/certbot/certbot/pull/6592#discussion_r245106383.
The tests use `eval` which neither myself or `pylint` like very much. I started to change this by splitting the path we wanted to test and repeatedly calling `getattr`, but it didn't seem worth the effort to me.
* Add missing acme.jose attribute.
* update changelog
When working on an update to our packages in Ubuntu Xenial, @NCommander noticed that importing code through acme.jose no longer works since josepy became a separate package and remembers having to fix up some code that was using acme.jose himself.
This PR should fix that problem by making all of josepy accessible through acme.jose. This is primarily beneficial to our OS package maintainers who want to avoid subtle API changes when updating packages in stable repositories. They will likely backport this change, but I figure we might as well add it ourselves to minimize divergences in our OS packages in the future and avoid problems on the off chance someone hasn't upgraded acme and was relying on this feature.
* Rough draft of External Account Binding.
* Remove parameter --eab and namespace kid and hmac. Also add parameters to "register" subcommand.
* Refactor as much as possible of the EAB functionality into ExternalAccountBinding class.
* Remove debug line.
* Added external account binding to Directory.Meta.
* Rename to account_public_key, hmac_key and make some non-optional.
Rename command line argument to --eab-hmac-key.
* Error out when the server requires External Account Binding and the user
has not supplied kid and hmac key.
* Remove whitespace.
* Refactor a bit to make it possible to set the url argument.
* Move from_data method into client.
* Revert "Move from_data method into client."
This reverts commit 8963fae
* Refactored to use json field on Registration.
* Inherit from object according to Google Python Style Guide.
* Move to two separate ifs.
* Get tests to pass after External Account Binding additions.
* messages.py back to 100% test coverage with some EAB tests.
* .encode() this JSON key.
* Set eab parameter default values to None.
* * Remove unnecessary public key mock on most of the test.
* Restructure the directory mock to be able to mock both True and False for externalAccountRequired easily.
* Add EAB client tests.
* Move external_account_required check into BackwardsCompatibleClientV2 to be able to mock it.
* Update versions.
* Try 0.29.0.
* Revert "Try 0.29.0."
This reverts commit 5779509
* Try 0.29.0 again.
* Try this.
* Fix pylint failures.
* Add tests for external_account_required method.
* Test not needed, avoid:
************* Module acme.client_test
C: 1, 0: Too many lines in module (1258/1250) (too-many-lines)
* Move real external_account_required method into ClientV2 and pass through to it in BackwardsCompatibleClientV2.
* Handle missing meta key in server ACME directory.
* Add docstring for BackwardsCompatibleClientV2.external_account_required().
* Add tests for BackwardsCompatibleClientV2.external_account_required().
* Fix coverage for ACMEv1 code in BackwardsCompatibleClientV2.
* Disable pylint too-many-lines check for client_test.py.
* Fix versions.
* Remove whitespace that accidently snuck into an earlier commit.
* Remove these two stray whitespaces also.
* And the last couple of whitespaces.
* Add External Account Binding to changelog.
* Add dev0 suffix to setup.py.
Co-Authored-By: robaman <robert@kastel.se>
* Set to "-e acme[dev]" again.
Co-Authored-By: robaman <robert@kastel.se>
* Setup an integration tests env against Pebble, that enforce post-as-get
* Implement POST-as-GET requests, with fallback to GET.
* Fix unit tests
* Fix coverage.
* Fix or ignore lint errors
* Corrections after review
* Correct test
* Try a simple delegate approach
* Add a test
* Simplify test mocking
* Clean comment
* Warn when using deprecated acme.challenges.TLSSNI01
* Update changelog
* remove specific date from warning
* add a raw assert for mypy optional type checking
Also, add checking to the newNonce HEAD request, and check responses in general before attempting to save a nonce, for a better error message.
* check response before adding nonce to the pool
* fix tests so that they test what they're supposed to test, and also allow the order of _add_nonce and _check_response to be switched
* make _get_nonce take acme_version
* Send HEAD to newNonce endpoint when using ACMEv2
* check the HEAD newNonce response
* remove unnecessary try; get returns None if the item doesn't exist
* instead of setting new_nonce_url on ClientNetwork, use the saved directory in ClientBase and pass that into ClientNetwork.post
* no need to test acme_version in _get_nonce
* pop new_nonce_url out of kwargs before passing to _send_request
So here we are: after #6361 has been merged, time is to provide an environment to execute the automated testing on Windows.
Here are the assertions used to build the CI on Windows:
every test running on Linux should ultimately be runnable on Windows, in a cross-platform compatible manner (there is one or two exception, when a test does not have any meaning for Windows),
currently some tests are not runnable on Windows: theses tests are ignored by default when the environment is Windows using a custom decorator: @broken_on_windows,
test environment should have functionalities similar to Travis, in particular an execution test matrix against various versions of Python and Windows,
so test execution is done through AppVeyor, as it supports the requirements: it add a CI step along Travis and Codecov for each PR, all of this ensuring that Certbot is entirely functional on both Linux and Windows,
code in tests can be changed, but code in Certbot should be changed as little as possible, to avoid regression risks.
So far in this PR, I focused on the tests on Certbot core and ACME library. Concerning the plugins, it will be done later, for plugins which have an interest on Windows. Test are executed against Python 3.4, 3.5, 3.6 and 3.7, for Windows Server 2012 R2 and Windows Server 2016.
I succeeded at making 258/259 of acme tests to work, and 828/868 of certbot core tests to work. Most of the errors where not because of Certbot itself, but because of how the tests are written. After redesigning some test utilitaries, and things like file path handling, or CRLF/LF, a lot of the errors vanished.
I needed also to ignore a lot of IO errors typically occurring when a tearDown test process tries to delete a file before it has been closed: this kind of behavior is acceptable for Linux, but not for Windows. As a consequence, and until the tearDown process is improved, a lot of temporary files are not cleared on Windows after a test campaign.
Remaining broken tests requires a more subtile approach to solve the errors, I will correct them progressively in future PR.
Last words about tox. I did not used the existing tox.ini for now. It is just to far from what is supported on Windows: lot of bash scripts that should be rewritten completely, and that contain test logic not ready/relevant for Windows (plugin tests, Docker compilation/test, GNU distribution versatility handling and so on). So I use an independent file tox-win.ini for now, with the goal to merge it ultimately with the existing logic.
* Define a tox configuration for windows, to execute tests against Python 3.4, 3.5, 3.6 and 3.7 + code coverage on Codecov.io
* Correct windows compatibility on certbot codebase
* Correct windows compatibility on certbot display functionalities
* Correct windows compatibility on certbot plugins
* Correct test utils to run tests on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows
* Correct tests on certbot core to run them both on windows and linux. Mark some of them as broken on windows for now.
* Lock tests are completely skipped on windows. Planned to be replace in next PR.
* Correct tests on certbot display to run them both on windows and linux. Mark some of them as broken on windows for now.
* Correct test utils for acme on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows.
* Correct acme tests to run them both on windows and linux. Allow a reduction of code coverage of 1% on acme code base.
* Create AppVeyor CI for Certbot on Windows, to run the test matrix (py34,35,36,37+coverage) on Windows Server 2012 R2 and Windows Server 2016.
* Update changelog with Windows compatibility of Certbot.
* Corrections about tox, pyreadline and CI logic
* Correct english
* Some corrections for acme
* Newlines corrections
* Remove changelog
* Use os.devnull instead of /dev/null to be used on Windows
* Uid is a always a number now.
* Correct linting
* PR https://github.com/python/typeshed/pull/2136 has been merge to third-party upstream 6 months ago, so code patch can be removed.
* And so acme coverage should be 100% again.
* More compatible tests Windows+Linux
* Use stable line separator
* Remove unused import
* Do not rely on pytest in certbot tests
* Use json.dumps to another json embedding weird characters
* Change comment
* Add import
* Test rolling builds #1
* Test rolling builds #2
* Correction on json serialization
* It seems that rolling builds are not canceling jobs on PR. Revert back to fail fast code in the pipeline.
* Raise ConflictError on attempts to create an existing account in ACME V2.
Fixes issue #6246
* Allow querying an account without calling new_account in ACMEv2
Fixed issue #6258
* find the correct url when deactivating an acmev1 account on the acmev2 endpoint
* set regr in ClientNetwork.account after deactivating on the server
* update self.net.account
* move logic into update_registration
* return methods to their original order to please git
* factor out common code
* update test_fowarding to use a method that still gets forwarded
* add acme module test coverage
* pragma no cover on correct line
* use previous regr uri
* strip unnecessary items from regr before saving
* add explanation to main.py
* add extra check to client_test.py
* use empty dict instead of empty string to indicate lack of body that we save to disk
This partially reverts commit 15f1405fff.
A basic tls-alpn-01 implementation is left so we can successfully parse the
challenge so it can be used in boulder's tests.
* Remove unneeded sys import.
Once upon a time we needed this in some of these setup.py files because we were
using sys in the file, but we aren't anymore so let's remove the import.
* use setuptools instead of distutils
The new challenge is described in https://github.com/rolandshoemaker/acme-tls-alpn.
* TLS-ALPN tests
* Implement TLS-ALPN challenge
* Skip TLS-ALPN tests on old pyopenssl
* make _selection methods private.
This change will allow registering/updating account with multi emails.
Detail is enclosed in #4242
* support multi emails register
* add more test cases
* update test to unregister before register
* update create path to support multi emaill
* refactor payload updating
* fix typo
* move command line doc to another place
* revert the change for updating account registration info, added unit test
* rearrange text for consistency
* Remove unsupported pylint disable options
* star-args removed in Pylint 1.4.3
* abstract-class-little-used removed in Pylint 1.4.3
* Fixes new lint errors
* Copy dummy-variable-rgx expression to new ignored-argument-names expression to ignore unused funtion arguments
* Notable changes
* Refactor to satisfy Pylint no-else-return warning
* Fix Pylint inconsistent-return-statements warning
* Refactor to satisfy consider-iterating-dictionary
* Remove methods with only super call to satisfy useless-super-delegation
* Refactor too-many-nested-statements where possible
* Suppress type checked errors where member is dynamically added (notably derived from josepy.JSONObjectWithFields)
* Remove None default of func parameter for ExitHandler and ErrorHandler
Resolves#5973
* check_untyped_defs in mypy with clean output for acme
* test entire acme module
* Add typing as a dependency because it's only in the stdlib for 3.5+
* Add str_utils, modified for python2.7 compatibility
* make mypy happy in acme
* typing is needed in prod
* we actually only need typing in acme so far
* add tests and more docs for str_utils
* pragma no cover
* add magic_typing
* s/from typing/from magic_typing/g
* move typing to dev_extras
* correctly set up imports
* remove str_utils
* only type: ignore for OpenSSL.SSL, not crypto
* Since we only run mypy with python3 anyway and we're fine importing it when it's not actually there, there's no actual need for typing to be present as a dependency
* comment magic_typing.py
* disable wildcard-import im magic_typing
* disable pylint errors
* add magic_typing_test
* make magic_typing tests work alongside other tests
* make sure temp_typing is set
* add typing as a dev dependency for python3.4
* run mypy with python3.4 on travis to get a little more testing with different environments
* don't stick typing into sys.modules
* reorder imports
* Allow revoke to pass in a url
* Add revocation support to ACMEv2.
* Provide regr for account based revocation.
* Add revoke wrapper to BackwardsCompat client
* Add post wrapper to automatically add acme_version
* Add uri to authzr.
* Only add kid when account is set.
* Add content_type when downloading certificate.
* Only save new_authz URL when it exists.
* Handle combinations in ACMEv1 and ACMEv2.
* Add tests for ACMEv2 "combinations".
* update order object with returned authorizations
* major structure of finalize_order shim refactor
* util methods and imports for finalize_order shim refactor
* update certbot.tests.client_test.py
* extraneous client_test imports
* remove correct import
* update renewal call
* add test for acme.dump_pyopenssl_chain
* Add test for certbot.crypto_util.cert_and_chain_from_fullchain
* add tests for acme.client and change to fetch chain failure to TimeoutError
* s/rytpe/rtype
* remove ClientV1 passthrough
* dump the wrapped cert
* remove dead code
* remove the correct dead code
* support earlier mock
* Drop support for EOL Python 2.6
* Use more helpful assertIn/NotIn instead of assertTrue/False
* Drop support for EOL Python 3.3
* Remove redundant Python 3.3 code
* Restore code for RHEL 6 and virtualenv for Py2.7
* Revert pipstrap.py to upstream
* Merge py26_packages and non_py26_packages into all_packages
* Revert changes to *-auto in root
* Update by calling letsencrypt-auto-source/build.py
* Revert permissions for pipstrap.py
* switch new_order to use crypto_util._pyopenssl_cert_or_req_san
* move certbot.crypto_util._get_names_from_loaded_cert_or_req functionality to acme.crypto_util._pyopenssl_cert_or_req_all_names
* delint
* refactor client tests
* Add test for new order and fix identifiers parsing.
* Add poll_and_finalize test
* Test and fix poll_authorizations timeout
* Add test_failed_authorizations
* Add test_poll_authorizations_success
* Test and fix finalize_order success
* add test_finalize_order_timeout
* add test_finalize_order_error
* test sleep code
This adds two new classes in messages: Order and OrderResource. It also adds methods to ClientV2 to create orders, and poll orders then request issuance.
The CSR is stored on the OrderResource so it can be carried along and submitted when it's time to finalize the order.
Detects acme version by checking for newNonce field in the directory, since it's mandatory. Also updates ClientNetwork.account on register and update_registration.
* add mechanism to detect acme version
* update ClientNetwork.account comment
* switch to MultiVersionClient object in acme
* add shim methods
* add returns
* use backwards-compatible format and implement register
* update to actual representation of tos v2
* add tos fields and pass through to v1 for partial updates
* update tests
* pass more tests
* allow instance variable pass-through and lint
* update certbot and tests to use new_account_and_tos method
* remove --agree-tos test from main_test for now because we moved the callback into acme
* add docstrings
* use hasattr
* all most review comments
* use terms_of_service for both v1 and v2
* add tests for acme/client.py
* tests for acme/messages.py
* Implement ACMEv2 signing of POST bodies.
* Add account, and make acme_version explicit.
* Remove separate NewAccount.
* Rename to add v2.
* Add terms_of_service_agreed.
* Split out wrap_in_jws_v2 test.
* Re-add too-many-public-methods.
* Split Client into ClientBase / Client / ClientV2
* Use camelCase for newAccount.
* Make acme_version optional parameter on .post().
This allows us to instantiate a ClientNetwork before knowing the version.
* Add kid unconditionally.
* Use pipstrap to install a good version of pip
* Use pytest in cb-auto tests
* Remove nose usage in auto_test.py
* remove nose dev dep
* use pytest in test_tests
* Use pytest in tox
* Update dev dependency pinnings
* remove nose multiprocess lines
* Use pytest for coverage
* Use older py and pytest for old python versions
* Add test for Error.__str__
* pin pytest in oldest test
* Fix tests for DNS-DO plugin on py26
* Work around bug for Python 3.3
* Clarify dockerfile comments
Fixes#4535
Extracts the relevant fields using a regex. We considered catching
specific exception types, and referencing their fields, but the types
raised by `requests` are not well documented and may not be long
term stable. If the regex fails to match, for instance due to a change
in the exception message, the new exception message will just be
passed through.
* Hide exceptions that occur during session.close()
This fixes#4840. Exceptions that are raised out of __del__ methods are caught
and printed to stderr. By catching any exceptions that occur, we now prevent
this from happening.
Alternative solutions to this would have been either not calling
session.close() at all or adding a close() method to acme.client.ClientNetwork,
acme.client.Client, and certbot.client.Client and using certbot.client.Client
in a context manager to ensure close() is called. The former means that users
of the ACME library never properly close their connections until their program
exits and the latter adds a lot of complexity and nesting of client code for
little benefit.
* Only catch Exceptions
* Pin oldest version of packaged python deps
* Install security extras in oldest tests
* Revert "bump requests requirement to >=2.10 (#4248)"
This reverts commit 402ad8b353.
* Use create=True when patching open on module
* add TLSSNI01DualNetworkedServers
* use DualNetworkedServers in certbot/plugins/standalone.py
also, make both servers run on the same port.
* make probe_sni connect on ipv6 and ipv4 using None
* mimic BSD-like conditions to get test coverage
* test ServerManager taking into account BSD systems
* pass tests even if python is compiled without ipv6 support
* Add an account deactivate utility script.
This is handy if you created an account with a tool other than Certbot, and want
to deactivate the account.
* Move deactivate.py to tools.
* Add test for ConflictError.
* Fix lint error.
* Document how to set server.
* Move 'jwk' and 'alg' fields to protected header.
Previously, these were in the unprotected JWS header, which Boulder currently
allows. However, the next version of the spec doesn't allow anything in the
unprotected header. Moving these fields now allows server implementers who are
implementing the Certbot/Boulder version of ACME
(https://github.com/letsencrypt/boulder/blob/master/docs/acme-divergences.md) to
use JOSE libraries that don't support unprotected headers.
Fixes#4417.
* Only protect existing headers.
* Mention python 3 support in setup.py
* Build universal (py2 and py3 compatible) wheels
* Mention Python 3.3+ support in docs
* we work on python 3.6 too
* Add url and kid to jws.
This will be required in order to implement the latest ACME spec, which uses
these protected header fields.
* Add comments and fix lint.
* Enforce mutual exclusivity of jwk and kid.
If the updated datetime collides, the comparator of heapq will move
onto the AuthorizationResource value and throws an "unorderable type"
error.
This adds an index value to the element tuple to ensure that they are
always strictly ordered.
It's relatively finicky to make a CSR appropriate to pass to poll_and_request_issuance. I think most users want to be able to give a list of domains and a private key, and get back a CSR. This branch adds that functionality to crypto_util.
Note that the two new functions take arguments, and return values, as PEM-encoded buffers. This is a departure from some existing ACME interfaces that take PyOpenSSL types. I've discussed with the Certbot team, and we agree that this is broadly the direction the ACME API should take, so that users of the module don't need to import PyOpenSSL themselves, or use its primitives.
* Add make_csr.
* accept privkey
* Tweak API.
* Remove make_csr from certbot package.
* Skip test in older Pythons.
* Move get_Extensions call under protection.
* Remove assertIn because not backwards-compatible.
* Fix encoding, and use PEM.
* Fix test
* Fix tests on py35.
* Fix error in test.
* Make import_csr_file always return PEM.
Also delete get_sans_from_csr (unused) and get_names_from_csr (newly unused).
* Fix function doc.
* Fix indent
* Fix call of obtain_certificate_from_Csr
* lint
* Handle review feedback.
* Fix test.
* Initial configuration of mypy in box, correction of base mypy errors.
* Move mypy install to toe
* Add pylint comments for typing imports.
* Remove typing module for Python 2.6 compatibility.
An early version of the spec indicated that clients should process issuance
sequentially, following Link rel=next from an account URL to an authz URL, to a
new-cert URL. However, the spec has long since moved to putting these URLs in
the directory.
Certbot nominally supports either; This change consolidates on always using the
directory, simplifying things and making the transition to the latest ACME spec
easier.
* Revert "Revert "Remove Link rel=next for authzs and new-certs." (#4277)"
This reverts commit 11ec1eb911.
* Save new_authzr_uri with account for older clients.
* Add test that new_authzr_uri exists in regr.
* Restore backwards compatibility for new_authzr_uri.
* Fix account_test.
* Add test for deprecated URI argument to request_challenges.
* Review feedback.
* Fix test
* Add omitempty to new_cert_uri.
* stop conditionally pinning mock version in acme
* stop conditionally pinning mock version in certbot
* stop conditionally pinning mock version in apache
* stop conditionally pinning mock version in nginx
* stop conditionally pinning mock version in letshelp
* stop conditionally pinning mock version in compatibility-test
The primary motivation is to avoid a branch, giving bugs one fewer place to hide. But, as a bonus, more people get a more bugfixed version of argparse. (To use the example from the argparse docs, people stuck on Python 3.2.3 can get bugfixes that made it into the stdlib only in 3.2.4.)
* Remove UnexpectedUpdate exceptions.
These exceptions trigger when the server sends the client back an object with a
field that doesn't exactly match what the client previously sent.
This causes unnecessary breakage in various cases, doesn't prevent any problems,
and isn't required by spec.
* Back out all UnexpectedUpdate removals except registration update.
An early version of the spec indicated that clients should process issuance
sequentially, following Link rel=next from an account URL to an authz URL, to a
new-cert URL. However, the spec has long since moved to putting these URLs in
the directory.
Certbot nominally supports either; This change consolidates on always using the
directory, simplifying things and making the transition to the latest ACME spec
easier.
This includes two new tests in the integration test script to check that
boulder gets the correct code. The encoding is specified in RFC5280
5.3.1. The codes that boulder will accept are a subset of that,
specified in `boulder.revocation.reasons.go`.
* Parallalelise nosetests from tox
* Parallelise even more things, break even more things
* Now unbreak all the tests that aren't ready for ||ism
* Try to pass tests!
- Remove non-working hack in reporter_test
- also be selective about ||ism in the cover environment
* Try again
* certbot-apache tests also work, given enough time
* Nginx may need more time in Travis's cloud
* Unbreak reporter_test under ||ism
* More timeout
* Working again?
* This goes way faster
* Another big win
* Split a couple more large test suites
* A last improvement
* More ||ism!
* ||ise lint too
* Allow nosetests to figure out how many cores to use
* simplify merge
* Mark the new CLI tests as ||izable
* Simplify reporter_test changes
* Rationalise ||ism flags
* Re-up coverage
* Clean up reporter tests
* Stop modifying testdata during tests
* remove unused os
* Ensure tests pass with openssl 1.1
A bunch of the acme.standalone and acme.crypto_util tests were using
weak crypto that is now prohibited :/
* lint
* lintlint
* Fix symlink
* pin requests version in py26-oldest
* Determine requests security deps dynamically
Starting with requests 2.12, pyasn1 and ndg-httpsclient are no longer
needed to inject pyopenssl into urllib3. This change allows us to
determine whether or not these dependencies are required at install
time. If an older version of requests is used, these packages are
still installed. If a new version of requests is used, they are not
reducing the number of dependencies we have.
* Bump requests version in certbot-auto
* Use pkg_resources in activate test
Due to pip's lack of dependency resolution, the change to use
requests[extras] causes errors in acme.util_test because pkg_resources
accurately detects the "missing" dependency.
There isn't a real problem here. The problem comes from a brand new
requests and ancient pyopenssl as well as a unit test for
functionality we plan to remove in our next release. I modified
the unit test to fix the problem for now.
* Use six instead of pkg_resources for test
* Require requests<=2.11.1 in py27-oldest test
If we don't do this, we get test failures for the certbot package
which is actually a good thing! pkg_resources is catching the
unlikely but possible problem I describe in #3803 and erroring out
saying it is missing the necessary dependencies to run certbot.
Good job package resources.
* Undo changes to acme.util_test
Print request and response bodies with newlines, rather than all on one line.
Remove "Omitted empty field" log, which gets logged meaninglessly for every JSON
serialization.
Remove duplicated logging of responses.
Log the base64 version of the nonce, rather than turning it into bytes and
logging the backslash-escaped version of those bytes.
Only pass -vv in tests.
* Add and test activate function to acme.
This function can be used to check if our optional dependencies are
available and they meet our version requirements.
* use activate in dns_resolver
* use activate in dns_available() in challenges_test
* Use activate in dns_resolver_test
* Use activate in certbot.plugins.util_test
* Use acme.util.activate for psutil
* Better testing and handling of missing deps
* Factored out *_available() code into a common function
* Delayed exception caused from using acme.dns_resolver without
dnspython until the function is called. This makes both
production and testing code simpler.
* Make a common subclass for already_listening tests
* Simplify mocking of USE_PSUTIL in tests
* move skipUnless to test_util
* add skip_unless to acme test_util
* Make dns_resolver_tests work with and without dnspython
* make acme.challenges_test pass when dns is unavailable
Right now the ACME client checks that the returned registration matches the
registation posted, but there's no guarantee this will always be the case, and
this only introduces unnecessary fragility.
Don't run pep8 for directories that we don't actually enforce pep8 on.
Install dependencies with -q.
Don't print reports, they make it hard to find the actual errors.
Remove deprecated fields from acme .pylintrc, they cause unnecessary messages
about deprecation.
The extra self will push along the arguments, resulting in the accurate but not very helpful error message: "AttributeError: 'JWKRSA' object has no attribute 'kty'"
When pip-installing any of these packages, pip hit our permissive, any-version "setuptools" dependency first and then ignored all subsequent, more constrained ones, like cryptography's "setuptools>=1.0". See https://github.com/pypa/pip/issues/988. It thus, on a box with setuptools 0.9.8, stuck with that version. Then, at runtime, letsencrypt crashed because pkg_resources couldn't satisfy cryptography's setuptools>=1.0 requirement.
This change lets us pip-install our packages and have it work. We'll need to make sure our direct requirements (all of them) satisfy the more constrained requirements of our dependencies. Yes, it is disgusting.
