The existing `acme.TLSALPN01` challenge class did not have
a `response_cls`, meaning it was not possible to use a tls-alpn-01
challenge with `client.answer_challenge`.
To support the above a simple `TLSALPN01Response` class is added that
doesn't provide the ability to solve a tls-alpn-01 challenge end to end
(e.g. generating and serving the correct response certificate) but that
does allow the challenge to be initiated. This is sufficient for users
that have set up the challenge response independent of the `acme`
module code.
Resolves#6676
This PR passes the CERTBOT_NO_PIN environment variable to the unit tests tox envs. By setting CERTBOT_NO_PIN to 1 before executing a given tox env, certbot dependencies will be installed at their latest version instead of the usual pinned version.
I also moved the unpin logic one layer below to allow it to be used potentially more widely, and avoid unnecessary merging constraints operation in this case.
As warnings are errors now, latest versions of Python will break now the tests, because collections launch a warning when some classes are imported from collections instead of collections.abc. Certbot code is patched, and warning is ignored for now, because a lot of third party libraries still depend on this behavior.
* Allow to execute a tox target without pinned dependencies
* Correct lint
* Retrigger build.
* Remove debug code
* Added test against unpinned dependencies from test-everything-unpinned-dependencies branch
* Remove duplicated assertion to pass TRAVIS and APPVEYOR in default tox environment.
I observed that the current set of oldest requirements do not correspond to any environment, except the specific Xenial image in Travis CI (and standard Xenial containers will also fail).
It is because the requirements make cryptography and requests fail against standard libraries available in the typical Linux distributions that are targeted by the oldest requirements approach (Centos 6, Centos 7, Xenial, Jessie).
This PR fixes that, by aligning the minimal version requirements of cryptography and requests to the maximal versions that are available on Centos 6. Centos 7, Jessie and Xenial stay unusable with oldest requirements for other reasons, but at least one old and supported Linux distribution is able to run the tests with oldest requirements out of the box.
A test is also corrected to match the expected error message that old versions of urllib3 will raise.
Fixes the problem at https://github.com/certbot/certbot/pull/6592#discussion_r245106383.
The tests use `eval` which neither myself or `pylint` like very much. I started to change this by splitting the path we wanted to test and repeatedly calling `getattr`, but it didn't seem worth the effort to me.
* Add missing acme.jose attribute.
* update changelog
When working on an update to our packages in Ubuntu Xenial, @NCommander noticed that importing code through acme.jose no longer works since josepy became a separate package and remembers having to fix up some code that was using acme.jose himself.
This PR should fix that problem by making all of josepy accessible through acme.jose. This is primarily beneficial to our OS package maintainers who want to avoid subtle API changes when updating packages in stable repositories. They will likely backport this change, but I figure we might as well add it ourselves to minimize divergences in our OS packages in the future and avoid problems on the off chance someone hasn't upgraded acme and was relying on this feature.
* Rough draft of External Account Binding.
* Remove parameter --eab and namespace kid and hmac. Also add parameters to "register" subcommand.
* Refactor as much as possible of the EAB functionality into ExternalAccountBinding class.
* Remove debug line.
* Added external account binding to Directory.Meta.
* Rename to account_public_key, hmac_key and make some non-optional.
Rename command line argument to --eab-hmac-key.
* Error out when the server requires External Account Binding and the user
has not supplied kid and hmac key.
* Remove whitespace.
* Refactor a bit to make it possible to set the url argument.
* Move from_data method into client.
* Revert "Move from_data method into client."
This reverts commit 8963fae
* Refactored to use json field on Registration.
* Inherit from object according to Google Python Style Guide.
* Move to two separate ifs.
* Get tests to pass after External Account Binding additions.
* messages.py back to 100% test coverage with some EAB tests.
* .encode() this JSON key.
* Set eab parameter default values to None.
* * Remove unnecessary public key mock on most of the test.
* Restructure the directory mock to be able to mock both True and False for externalAccountRequired easily.
* Add EAB client tests.
* Move external_account_required check into BackwardsCompatibleClientV2 to be able to mock it.
* Update versions.
* Try 0.29.0.
* Revert "Try 0.29.0."
This reverts commit 5779509
* Try 0.29.0 again.
* Try this.
* Fix pylint failures.
* Add tests for external_account_required method.
* Test not needed, avoid:
************* Module acme.client_test
C: 1, 0: Too many lines in module (1258/1250) (too-many-lines)
* Move real external_account_required method into ClientV2 and pass through to it in BackwardsCompatibleClientV2.
* Handle missing meta key in server ACME directory.
* Add docstring for BackwardsCompatibleClientV2.external_account_required().
* Add tests for BackwardsCompatibleClientV2.external_account_required().
* Fix coverage for ACMEv1 code in BackwardsCompatibleClientV2.
* Disable pylint too-many-lines check for client_test.py.
* Fix versions.
* Remove whitespace that accidently snuck into an earlier commit.
* Remove these two stray whitespaces also.
* And the last couple of whitespaces.
* Add External Account Binding to changelog.
* Add dev0 suffix to setup.py.
Co-Authored-By: robaman <robert@kastel.se>
* Set to "-e acme[dev]" again.
Co-Authored-By: robaman <robert@kastel.se>
* Setup an integration tests env against Pebble, that enforce post-as-get
* Implement POST-as-GET requests, with fallback to GET.
* Fix unit tests
* Fix coverage.
* Fix or ignore lint errors
* Corrections after review
* Correct test
* Try a simple delegate approach
* Add a test
* Simplify test mocking
* Clean comment
* Warn when using deprecated acme.challenges.TLSSNI01
* Update changelog
* remove specific date from warning
* add a raw assert for mypy optional type checking
Also, add checking to the newNonce HEAD request, and check responses in general before attempting to save a nonce, for a better error message.
* check response before adding nonce to the pool
* fix tests so that they test what they're supposed to test, and also allow the order of _add_nonce and _check_response to be switched
* make _get_nonce take acme_version
* Send HEAD to newNonce endpoint when using ACMEv2
* check the HEAD newNonce response
* remove unnecessary try; get returns None if the item doesn't exist
* instead of setting new_nonce_url on ClientNetwork, use the saved directory in ClientBase and pass that into ClientNetwork.post
* no need to test acme_version in _get_nonce
* pop new_nonce_url out of kwargs before passing to _send_request
So here we are: after #6361 has been merged, time is to provide an environment to execute the automated testing on Windows.
Here are the assertions used to build the CI on Windows:
every test running on Linux should ultimately be runnable on Windows, in a cross-platform compatible manner (there is one or two exception, when a test does not have any meaning for Windows),
currently some tests are not runnable on Windows: theses tests are ignored by default when the environment is Windows using a custom decorator: @broken_on_windows,
test environment should have functionalities similar to Travis, in particular an execution test matrix against various versions of Python and Windows,
so test execution is done through AppVeyor, as it supports the requirements: it add a CI step along Travis and Codecov for each PR, all of this ensuring that Certbot is entirely functional on both Linux and Windows,
code in tests can be changed, but code in Certbot should be changed as little as possible, to avoid regression risks.
So far in this PR, I focused on the tests on Certbot core and ACME library. Concerning the plugins, it will be done later, for plugins which have an interest on Windows. Test are executed against Python 3.4, 3.5, 3.6 and 3.7, for Windows Server 2012 R2 and Windows Server 2016.
I succeeded at making 258/259 of acme tests to work, and 828/868 of certbot core tests to work. Most of the errors where not because of Certbot itself, but because of how the tests are written. After redesigning some test utilitaries, and things like file path handling, or CRLF/LF, a lot of the errors vanished.
I needed also to ignore a lot of IO errors typically occurring when a tearDown test process tries to delete a file before it has been closed: this kind of behavior is acceptable for Linux, but not for Windows. As a consequence, and until the tearDown process is improved, a lot of temporary files are not cleared on Windows after a test campaign.
Remaining broken tests requires a more subtile approach to solve the errors, I will correct them progressively in future PR.
Last words about tox. I did not used the existing tox.ini for now. It is just to far from what is supported on Windows: lot of bash scripts that should be rewritten completely, and that contain test logic not ready/relevant for Windows (plugin tests, Docker compilation/test, GNU distribution versatility handling and so on). So I use an independent file tox-win.ini for now, with the goal to merge it ultimately with the existing logic.
* Define a tox configuration for windows, to execute tests against Python 3.4, 3.5, 3.6 and 3.7 + code coverage on Codecov.io
* Correct windows compatibility on certbot codebase
* Correct windows compatibility on certbot display functionalities
* Correct windows compatibility on certbot plugins
* Correct test utils to run tests on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows
* Correct tests on certbot core to run them both on windows and linux. Mark some of them as broken on windows for now.
* Lock tests are completely skipped on windows. Planned to be replace in next PR.
* Correct tests on certbot display to run them both on windows and linux. Mark some of them as broken on windows for now.
* Correct test utils for acme on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows.
* Correct acme tests to run them both on windows and linux. Allow a reduction of code coverage of 1% on acme code base.
* Create AppVeyor CI for Certbot on Windows, to run the test matrix (py34,35,36,37+coverage) on Windows Server 2012 R2 and Windows Server 2016.
* Update changelog with Windows compatibility of Certbot.
* Corrections about tox, pyreadline and CI logic
* Correct english
* Some corrections for acme
* Newlines corrections
* Remove changelog
* Use os.devnull instead of /dev/null to be used on Windows
* Uid is a always a number now.
* Correct linting
* PR https://github.com/python/typeshed/pull/2136 has been merge to third-party upstream 6 months ago, so code patch can be removed.
* And so acme coverage should be 100% again.
* More compatible tests Windows+Linux
* Use stable line separator
* Remove unused import
* Do not rely on pytest in certbot tests
* Use json.dumps to another json embedding weird characters
* Change comment
* Add import
* Test rolling builds #1
* Test rolling builds #2
* Correction on json serialization
* It seems that rolling builds are not canceling jobs on PR. Revert back to fail fast code in the pipeline.
* Raise ConflictError on attempts to create an existing account in ACME V2.
Fixes issue #6246
* Allow querying an account without calling new_account in ACMEv2
Fixed issue #6258
* find the correct url when deactivating an acmev1 account on the acmev2 endpoint
* set regr in ClientNetwork.account after deactivating on the server
* update self.net.account
* move logic into update_registration
* return methods to their original order to please git
* factor out common code
* update test_fowarding to use a method that still gets forwarded
* add acme module test coverage
* pragma no cover on correct line
* use previous regr uri
* strip unnecessary items from regr before saving
* add explanation to main.py
* add extra check to client_test.py
* use empty dict instead of empty string to indicate lack of body that we save to disk
This partially reverts commit 15f1405fff.
A basic tls-alpn-01 implementation is left so we can successfully parse the
challenge so it can be used in boulder's tests.
* Remove unneeded sys import.
Once upon a time we needed this in some of these setup.py files because we were
using sys in the file, but we aren't anymore so let's remove the import.
* use setuptools instead of distutils
The new challenge is described in https://github.com/rolandshoemaker/acme-tls-alpn.
* TLS-ALPN tests
* Implement TLS-ALPN challenge
* Skip TLS-ALPN tests on old pyopenssl
* make _selection methods private.
This change will allow registering/updating account with multi emails.
Detail is enclosed in #4242
* support multi emails register
* add more test cases
* update test to unregister before register
* update create path to support multi emaill
* refactor payload updating
* fix typo
* move command line doc to another place
* revert the change for updating account registration info, added unit test
* rearrange text for consistency
* Remove unsupported pylint disable options
* star-args removed in Pylint 1.4.3
* abstract-class-little-used removed in Pylint 1.4.3
* Fixes new lint errors
* Copy dummy-variable-rgx expression to new ignored-argument-names expression to ignore unused funtion arguments
* Notable changes
* Refactor to satisfy Pylint no-else-return warning
* Fix Pylint inconsistent-return-statements warning
* Refactor to satisfy consider-iterating-dictionary
* Remove methods with only super call to satisfy useless-super-delegation
* Refactor too-many-nested-statements where possible
* Suppress type checked errors where member is dynamically added (notably derived from josepy.JSONObjectWithFields)
* Remove None default of func parameter for ExitHandler and ErrorHandler
Resolves#5973
* check_untyped_defs in mypy with clean output for acme
* test entire acme module
* Add typing as a dependency because it's only in the stdlib for 3.5+
* Add str_utils, modified for python2.7 compatibility
* make mypy happy in acme
* typing is needed in prod
* we actually only need typing in acme so far
* add tests and more docs for str_utils
* pragma no cover
* add magic_typing
* s/from typing/from magic_typing/g
* move typing to dev_extras
* correctly set up imports
* remove str_utils
* only type: ignore for OpenSSL.SSL, not crypto
* Since we only run mypy with python3 anyway and we're fine importing it when it's not actually there, there's no actual need for typing to be present as a dependency
* comment magic_typing.py
* disable wildcard-import im magic_typing
* disable pylint errors
* add magic_typing_test
* make magic_typing tests work alongside other tests
* make sure temp_typing is set
* add typing as a dev dependency for python3.4
* run mypy with python3.4 on travis to get a little more testing with different environments
* don't stick typing into sys.modules
* reorder imports
* Allow revoke to pass in a url
* Add revocation support to ACMEv2.
* Provide regr for account based revocation.
* Add revoke wrapper to BackwardsCompat client
* Add post wrapper to automatically add acme_version
* Add uri to authzr.
* Only add kid when account is set.
* Add content_type when downloading certificate.
* Only save new_authz URL when it exists.
* Handle combinations in ACMEv1 and ACMEv2.
* Add tests for ACMEv2 "combinations".
* update order object with returned authorizations
* major structure of finalize_order shim refactor
* util methods and imports for finalize_order shim refactor
* update certbot.tests.client_test.py
* extraneous client_test imports
* remove correct import
* update renewal call
* add test for acme.dump_pyopenssl_chain
* Add test for certbot.crypto_util.cert_and_chain_from_fullchain
* add tests for acme.client and change to fetch chain failure to TimeoutError
* s/rytpe/rtype
* remove ClientV1 passthrough
* dump the wrapped cert
* remove dead code
* remove the correct dead code
* support earlier mock
* Drop support for EOL Python 2.6
* Use more helpful assertIn/NotIn instead of assertTrue/False
* Drop support for EOL Python 3.3
* Remove redundant Python 3.3 code
* Restore code for RHEL 6 and virtualenv for Py2.7
* Revert pipstrap.py to upstream
* Merge py26_packages and non_py26_packages into all_packages
* Revert changes to *-auto in root
* Update by calling letsencrypt-auto-source/build.py
* Revert permissions for pipstrap.py
* switch new_order to use crypto_util._pyopenssl_cert_or_req_san
* move certbot.crypto_util._get_names_from_loaded_cert_or_req functionality to acme.crypto_util._pyopenssl_cert_or_req_all_names
* delint
* refactor client tests
* Add test for new order and fix identifiers parsing.
* Add poll_and_finalize test
* Test and fix poll_authorizations timeout
* Add test_failed_authorizations
* Add test_poll_authorizations_success
* Test and fix finalize_order success
* add test_finalize_order_timeout
* add test_finalize_order_error
* test sleep code
This adds two new classes in messages: Order and OrderResource. It also adds methods to ClientV2 to create orders, and poll orders then request issuance.
The CSR is stored on the OrderResource so it can be carried along and submitted when it's time to finalize the order.
Detects acme version by checking for newNonce field in the directory, since it's mandatory. Also updates ClientNetwork.account on register and update_registration.
* add mechanism to detect acme version
* update ClientNetwork.account comment
* switch to MultiVersionClient object in acme
* add shim methods
* add returns
* use backwards-compatible format and implement register
* update to actual representation of tos v2
* add tos fields and pass through to v1 for partial updates
* update tests
* pass more tests
* allow instance variable pass-through and lint
* update certbot and tests to use new_account_and_tos method
* remove --agree-tos test from main_test for now because we moved the callback into acme
* add docstrings
* use hasattr
* all most review comments
* use terms_of_service for both v1 and v2
* add tests for acme/client.py
* tests for acme/messages.py
* Implement ACMEv2 signing of POST bodies.
* Add account, and make acme_version explicit.
* Remove separate NewAccount.
* Rename to add v2.
* Add terms_of_service_agreed.
* Split out wrap_in_jws_v2 test.
* Re-add too-many-public-methods.
* Split Client into ClientBase / Client / ClientV2
* Use camelCase for newAccount.
* Make acme_version optional parameter on .post().
This allows us to instantiate a ClientNetwork before knowing the version.
* Add kid unconditionally.
* Use pipstrap to install a good version of pip
* Use pytest in cb-auto tests
* Remove nose usage in auto_test.py
* remove nose dev dep
* use pytest in test_tests
* Use pytest in tox
* Update dev dependency pinnings
* remove nose multiprocess lines
* Use pytest for coverage
* Use older py and pytest for old python versions
* Add test for Error.__str__
* pin pytest in oldest test
* Fix tests for DNS-DO plugin on py26
* Work around bug for Python 3.3
* Clarify dockerfile comments
Fixes#4535
Extracts the relevant fields using a regex. We considered catching
specific exception types, and referencing their fields, but the types
raised by `requests` are not well documented and may not be long
term stable. If the regex fails to match, for instance due to a change
in the exception message, the new exception message will just be
passed through.
* Hide exceptions that occur during session.close()
This fixes#4840. Exceptions that are raised out of __del__ methods are caught
and printed to stderr. By catching any exceptions that occur, we now prevent
this from happening.
Alternative solutions to this would have been either not calling
session.close() at all or adding a close() method to acme.client.ClientNetwork,
acme.client.Client, and certbot.client.Client and using certbot.client.Client
in a context manager to ensure close() is called. The former means that users
of the ACME library never properly close their connections until their program
exits and the latter adds a lot of complexity and nesting of client code for
little benefit.
* Only catch Exceptions
* Pin oldest version of packaged python deps
* Install security extras in oldest tests
* Revert "bump requests requirement to >=2.10 (#4248)"
This reverts commit 402ad8b353.
* Use create=True when patching open on module
* add TLSSNI01DualNetworkedServers
* use DualNetworkedServers in certbot/plugins/standalone.py
also, make both servers run on the same port.
* make probe_sni connect on ipv6 and ipv4 using None
* mimic BSD-like conditions to get test coverage
* test ServerManager taking into account BSD systems
* pass tests even if python is compiled without ipv6 support
* Add an account deactivate utility script.
This is handy if you created an account with a tool other than Certbot, and want
to deactivate the account.
* Move deactivate.py to tools.
* Add test for ConflictError.
* Fix lint error.
* Document how to set server.
* Move 'jwk' and 'alg' fields to protected header.
Previously, these were in the unprotected JWS header, which Boulder currently
allows. However, the next version of the spec doesn't allow anything in the
unprotected header. Moving these fields now allows server implementers who are
implementing the Certbot/Boulder version of ACME
(https://github.com/letsencrypt/boulder/blob/master/docs/acme-divergences.md) to
use JOSE libraries that don't support unprotected headers.
Fixes#4417.
* Only protect existing headers.
