* acme: re-populate uri in deactivate_authorization
* Use fresh authorizations in dry runs
--dry-run now deactivates 'valid' authorizations if it encounters them
when creating a new order.
Resolves#5116.
* remove unused code
* typo in local-oldest-requirements
* better error handling
* certbot-ci: AUTHREUSE to 100 + unskip dry-run test
* improve test coverage for error cases
* restore newline to local-oldest-requirements.txt
I tried to finish up #7214 by removing the code in acme but we can't really do that until #7478 is resolved which we cannot do until we release 0.40.0.
Since we have to wait, this PR adds deprecation warnings for code that uses the TLS-SNI-01 code or was only used by the long deprecated TLS-SNI-01 code.
I'd like this PR to land before our next release.
* Deprecate more code related to TLS-SNI-01.
* Assert about warning message.
Fixes#7368.
When updating the changelog, I replaced the line about running tests on Python 3.8 because I personally think that support for Python 3.8 is the most relevant information for our users/packagers about our changes in this area.
* List support for Python 3.8.
* Update changelog.
Resolves#4945. First PR in order to address #5116.
* acme: Implement authz deactivation
Resolves#4945
* update AUTHORS and CHANGELOG
* typos in mypy annotations
* formatting: missing newline
* improve test_deactivate_authorization
* improve deactivate_authorization
* test: s/STATUS_INVALID/STATUS_DEACTIVATED/
* simplify dict to keyword argument
* acme: add UpdateAuthorization
* acme: use UpdateAuthorization in deactivate_authz
and add mypy annotation
This allows deactivate_authorization to succeed for both ACME v1
and v2 servers.
Fixes#6974.
This PR removes the fallback that consists in retrying to send the keyAuthorization field during a challenge request in case of malformed request.
* Remove keyAuthorization fallback dump in challenges response
* Correct import
* Add changelog entry
This PR is a part of the tls-sni-01 removal plan described in #6849.
As `acme` is a library, we need to put some efforts to make a decent deprecation path before totally removing tls-sni in it. While initialization of `acme.challenges.TLSSNI01` was already creating deprecation warning, not all cases were covered.
For instance, and innocent call like this ...
```python
if not isinstance(challenge, acme.challenges.TLSSNI01):
print('I am not using this TLS-SNI deprecated stuff, what could possibly go wrong?')
```
... would break if we suddenly remove all objects related to this challenge.
So, I use the _Deprecator Warning Machine, Let's Pacify this Technical Debt_ (Guido ®), to make `acme.challenges` and `acme.standalone` patch themselves, and display a deprecation warning on stderr for any access to the tls-sni challenge objects.
No dev should be able to avoid the deprecation warning. I set the deprecation warning in the idea to remove the code on `0.34.0`, but the exact deprecation window is open to discussion of course.
* Modules challenges and standalone patch themselves to generated deprecation warning when tls-sni related objects are accessed.
* Correct unit tests
* Correct lint
* Update challenges_test.py
* Correct lint
* Fix an error during tests
* Update coverage
* Use multiprocessing for coverage
* Add coverage
* Update test_util.py
* Factor the logic about global deprecation warning when accessing TLS-SNI-01 attributes
* Fix coverage
* Add comment for cryptography example.
* Use warnings.
* Add a changelog
* Fix deprecation during tests
* Reload
* Update acme/acme/__init__.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Update CHANGELOG.md
* Pick a random free port.
Fixes#6755.
POSTing the `keyAuthorization` in a JWS token when answering an ACME challenge, has been deprecated for some time now. Indeed, this is superfluous as the request is already authentified by the JWS signature.
Boulder still accepts to see this field in the JWS token, and ignore it. Pebble in non strict mode also. But Pebble in strict mode refuses the request, to prepare complete removal of this field in ACME v2.
Certbot still sends the `keyAuthorization` field. This PR removes it, and makes Certbot compliant with current ACME v2 protocol, and so Pebble in strict mode.
See also [letsencrypt/pebble#192](https://github.com/letsencrypt/pebble/issues/192) for implementation details server side.
* New implementation, with a fallback.
* Add deprecation on changelog
* Update acme/acme/client.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Fix an instance parameter
* Update changelog, extend coverage
* Update comment
* Add unit tests on keyAuthorization dump
* Update acme/acme/client.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Restrict the magic of setting a variable in immutable object in one place. Make a soon to be removed method private.
* Add acme library usage example
Create, edit and deactivate account.
Setup and perform http-01 challenge.
Issue, renew and revoke certificate.
* Adapt example to ACME-v2 and exclude data persistence
The code to persist/load data would length this example and distract from what is actually important.
* Fix domain names and e-mail addresses
* Remove unnecessary license header
This usage example is under the license for the acme package.
* Remove logging information
The code will be mostly read by developers, so simplify the logging info into comments.
* Revert abstraction of simple methods
All methods that are used only once in this example were expanded into the main code in order to make the process more explicit.
* Fix missing URL suffix
* Improve aesthetics and reorganize workflow
Also make words capitalization consistent and improve comments.
No complaints from pep8.
So merging the study from @bmw and me, here is what happened.
Each invocation of `certbot.logger.post_arg_parse_setup` create a file handler on `letsencrypt.log`. This function also set an atexit handler invoking `logger.shutdown()`, that have the effect to close all logger file handler not already closed at this point. This method is supposed to be called when a python process is close to exit, because it makes all logger unable to write new logs on any handler.
Before #6667 and this PR, for tests, the atexit handle would be triggered only at the end of the pytest process. It means that each test that launches `certbot.logger.post_arg_parse_setup` add a new file handler. These tests were typically connecting the file handler on a `letsencrypt.log` located in a temporary directory, and this directory and content was wipped out at each test tearDown. As a consequence, the file handles, not cleared from the logger, were accumulating in the logger, with all of them connected to a deleted file log, except the last one that was just created by the current test. Considering the number of tests concerned, there were ~300 file handler at the end of pytest execution.
One can see that, on prior #6667, by calling `print(logger.getLogger().handlers` on the `tearDown` of these tests, and see the array growing at each test execution.
Even if this represent a memory leak, this situation was not really a problem on Linux: because a file can be deleted before it is closed, it was only meaning that a given invocation of `logger.debug` for instance, during the tests, was written in 300 log files. The overhead is negligeable. On Windows however, the file handlers were failing because you cannot delete a file before it is closed.
It was one of the reason for #6667, that added a call to `logging.shutdown()` at each test tearDown, with the consequence to close all file handlers. At this point, Linux is not happy anymore. Any call to `logger.warn` will generate an error for each closed file handler. As a file handler is added for each test, the number of errors grows on each test, following an arithmetical suite divergence.
On `test_sdists.py`, that is using the bare setuptools test suite without output capturing, we can see the damages. The total output takes 216000 lines, and 23000 errors are generated. A decent machine can support this load, but a not a small AWS instance, that is crashing during the execution. Even with pytest, the captured output and the memory leak become so large that segfaults are generated.
On the current PR, the problem is solved, by resetting the file handlers array on the logging system on each test tearDown. So each fileHandler is properly closed, and removed from the stack. They do not participate anymore in the logging system, and can be garbage collected. Then we stay on always one file handler opened at any time, and tests can succeed on AWS instances.
For the record, here is all the places where the logging system is called and fail if there is still file handlers closed but not cleaned (extracted from the original huge output before correction):
```
Logged from file account.py, line 116
Logged from file account.py, line 178
Logged from file client.py, line 166
Logged from file client.py, line 295
Logged from file client.py, line 415
Logged from file client.py, line 422
Logged from file client.py, line 480
Logged from file client.py, line 503
Logged from file client.py, line 540
Logged from file client.py, line 601
Logged from file client.py, line 622
Logged from file client.py, line 750
Logged from file cli.py, line 220
Logged from file cli.py, line 226
Logged from file crypto_util.py, line 101
Logged from file crypto_util.py, line 127
Logged from file crypto_util.py, line 147
Logged from file crypto_util.py, line 261
Logged from file crypto_util.py, line 283
Logged from file crypto_util.py, line 307
Logged from file crypto_util.py, line 336
Logged from file disco.py, line 116
Logged from file disco.py, line 124
Logged from file disco.py, line 134
Logged from file disco.py, line 138
Logged from file disco.py, line 141
Logged from file dns_common_lexicon.py, line 45
Logged from file dns_common_lexicon.py, line 61
Logged from file dns_common_lexicon.py, line 67
Logged from file dns_common.py, line 316
Logged from file dns_common.py, line 64
Logged from file eff.py, line 60
Logged from file eff.py, line 73
Logged from file error_handler.py, line 105
Logged from file error_handler.py, line 110
Logged from file error_handler.py, line 87
Logged from file hooks.py, line 248
Logged from file main.py, line 1071
Logged from file main.py, line 1075
Logged from file main.py, line 1189
Logged from file ops.py, line 122
Logged from file ops.py, line 325
Logged from file ops.py, line 338
Logged from file reporter.py, line 55
Logged from file selection.py, line 110
Logged from file selection.py, line 118
Logged from file selection.py, line 123
Logged from file selection.py, line 176
Logged from file selection.py, line 231
Logged from file selection.py, line 310
Logged from file selection.py, line 66
Logged from file standalone.py, line 101
Logged from file standalone.py, line 88
Logged from file standalone.py, line 97
Logged from file standalone.py, line 98
Logged from file storage.py, line 52
Logged from file storage.py, line 59
Logged from file storage.py, line 75
Logged from file util.py, line 56
Logged from file webroot.py, line 165
Logged from file webroot.py, line 186
Logged from file webroot.py, line 187
Logged from file webroot.py, line 204
Logged from file webroot.py, line 223
Logged from file webroot.py, line 234
Logged from file webroot.py, line 235
Logged from file webroot.py, line 237
Logged from file webroot.py, line 91
```
* Reapply #6667
* Make setuptools delegates tests execution to pytest, like in acme module.
* Clean handlers at each tearDown to avoid memory leaks.
* Update changelog
The existing `acme.TLSALPN01` challenge class did not have
a `response_cls`, meaning it was not possible to use a tls-alpn-01
challenge with `client.answer_challenge`.
To support the above a simple `TLSALPN01Response` class is added that
doesn't provide the ability to solve a tls-alpn-01 challenge end to end
(e.g. generating and serving the correct response certificate) but that
does allow the challenge to be initiated. This is sufficient for users
that have set up the challenge response independent of the `acme`
module code.
Resolves#6676
This PR passes the CERTBOT_NO_PIN environment variable to the unit tests tox envs. By setting CERTBOT_NO_PIN to 1 before executing a given tox env, certbot dependencies will be installed at their latest version instead of the usual pinned version.
I also moved the unpin logic one layer below to allow it to be used potentially more widely, and avoid unnecessary merging constraints operation in this case.
As warnings are errors now, latest versions of Python will break now the tests, because collections launch a warning when some classes are imported from collections instead of collections.abc. Certbot code is patched, and warning is ignored for now, because a lot of third party libraries still depend on this behavior.
* Allow to execute a tox target without pinned dependencies
* Correct lint
* Retrigger build.
* Remove debug code
* Added test against unpinned dependencies from test-everything-unpinned-dependencies branch
* Remove duplicated assertion to pass TRAVIS and APPVEYOR in default tox environment.
I observed that the current set of oldest requirements do not correspond to any environment, except the specific Xenial image in Travis CI (and standard Xenial containers will also fail).
It is because the requirements make cryptography and requests fail against standard libraries available in the typical Linux distributions that are targeted by the oldest requirements approach (Centos 6, Centos 7, Xenial, Jessie).
This PR fixes that, by aligning the minimal version requirements of cryptography and requests to the maximal versions that are available on Centos 6. Centos 7, Jessie and Xenial stay unusable with oldest requirements for other reasons, but at least one old and supported Linux distribution is able to run the tests with oldest requirements out of the box.
A test is also corrected to match the expected error message that old versions of urllib3 will raise.
Fixes the problem at https://github.com/certbot/certbot/pull/6592#discussion_r245106383.
The tests use `eval` which neither myself or `pylint` like very much. I started to change this by splitting the path we wanted to test and repeatedly calling `getattr`, but it didn't seem worth the effort to me.
* Add missing acme.jose attribute.
* update changelog
