* deps: remove pyjwt dependency
* pinning: strip extras from dependencies
`poetry export` outputs in requirements.txt format, which is now
apparently producing "dep[extra]==...". We are using this output
as the constraints file for pip and pip's new resolver does not
permit extras in the constraints file.
This change filters out the extras specifiers.
* repin current dependencies
* fix new pylint complaints
* silence lint about distutils.version
We have already deprecated the function and it'll be removed in
2.0.
* docs: set sphinx language to 'en'
this is emitting a warning and failing the build
* Revert "pinning: strip extras from dependencies"
This reverts commit 11268fd231.
* pin poetry back to avoid extras issue
* repin
* fix new mypy complaints in acme/
* certbot-ci: fix challtestsrv address for boulder-v2
The port is no longer exposed on the Docker host.
* vary the challtestsrv URL by acme server
* fix mypy
* fix comment
Co-authored-by: ohemorange <ebportnoy@gmail.com>
Co-authored-by: ohemorange <ebportnoy@gmail.com>
* improve handling and ux of unexpected key type migration
* update unit tests
* update integration tests
* if --cert-name and --key-type are set, dont prompt
* Add support for revoking ecdsa keys without --cert-name.
Co-Authored-By: commonism <commonism@users.noreply.github.com>
* Move alg to acme_client.ClientNetwork instantiating in acme_from_config_key
* Fix argument for RS256/ES256
* Support also ES384 and ES512 signing algorithms.
* Work in progress
* Work in progress
* Work in progress
* Work in progress
* Fix issues around nullability of VirtualHost.path, may discuss that during review
* Work in progress
* Fix remaining types
* Various lint fixes
* Reconfigure tox and mypy to disallow untyped defs globally
* Cleanup compatibility tests
* Use cast for unused v2 logic
* Improve types
* Remove unused comment
* Fix coverage
* Better types
* Fix another type
* Update certbot-apache/certbot_apache/_internal/apacheparser.py
Co-authored-by: alexzorin <alex@zor.io>
* Update certbot-apache/certbot_apache/_internal/assertions.py
Co-authored-by: alexzorin <alex@zor.io>
* Fix type
* Various fixes
* Refactor imports
* Keep naming convention consistent on TypeVars
* Improve types
* Improve types
* Remove remaining Sequence[str] in the project
Co-authored-by: alexzorin <alex@zor.io>
* Add generic methods to save some casts, and fix lint
* Update current and oldest pinning
* Fix classes
* Remove some todos thanks to josepy 1.11.0
* Cleanup some useless pylint disable
* Finish complete typing
* Better TypeVar names
* Upgrade pinning and fix some typing errors
* Use protocol
* Fix types in apache
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Fixes https://github.com/certbot/certbot/issues/9058.
The changes to the CI config are equivalent to the ones made in https://github.com/certbot/certbot/pull/8460.
Other than ignoring some warnings raised by botocore, the main additional work that had to be done here was switching away from using `distutils.version.LooseVersion` since the entire `distutils` module was deprecated in Python 3.10. To do that, I took a few different approaches:
* If the version strings being parsed are from Python packages such as Certbot or setuptools, I switched to using [pkg_resources.parse_version](https://setuptools.pypa.io/en/latest/pkg_resources.html#parsing-utilities) from `setuptools`. This functionality has been available since [setuptools 8.0 from 2014](https://setuptools.pypa.io/en/latest/history.html#id865).
* If the version strings being parsed are not from Python packages, I added code equivalent to `distutils.version.LooseVersion` in `certbot.util.parse_loose_version`.
* The code for `CERTBOT_PIP_NO_BINARY` can be completely removed since that variable isn't used or referenced anywhere in this repo.
* add python 3.10 support
* make some version changes
* don't use looseversion in setup.py
* switch to pkg_resources
* deprecate get_strict_version
* fix route53 tests
* remove unused CERTBOT_PIP_NO_BINARY code
* stop using distutils in letstest
* add unit tests
* more changelog entries
Fixes#8899
This PR removes the pinning upper limit of mypy currently set to <0.900 and adds the required types-* stub packages to make recent versions of mypy work.
* Unpin mypy
* Improve type in TempHandler
* Add types
While bumping pinned packages in #8928, we came across a new version of pylint (2.9.3). Upgrading to this version requires some changes to Certbot's code, which is what this change is about.
* pylint: upgrade pinned verson and fix new lints
* maxsplit should be 1, not -1, for rsplit
This is part of https://github.com/certbot/certbot/issues/8782. I took it on now because the currently pinned version of `pylint` doesn't work with newer versions of `poetry` which I wanted to upgrade as part of https://github.com/certbot/certbot/issues/8787.
To say a bit more about the specific changes in this PR:
* Newer versions of `pylint` complain if `Popen` isn't used as a context manager. Instead of making this change, I switched to using `subprocess.run` which is simpler and [recommended in the Python docs](https://docs.python.org/3/library/subprocess.html#using-the-subprocess-module). I also disabled this check in a few places where no longer using `Popen` would require significant refactoring.
* The deleted code in `certbot/certbot/_internal/renewal.py` is cruft since https://github.com/certbot/certbot/pull/8685.
* The unused argument to `enable_mod` in the Apache plugin is used in some over the override classes that subclass that class.
* unpin pylint and repin dependencies
* disable raise-missing-from
* disable wrong-input-order
* remove unused code
* misc lint fixes
* remove unused import
* various lint fixes
This is one of the things that newer versions of `pylint` complains about.
* git grep -l super\( | xargs sed -i 's/super([^)]*)/super()/g'
* fix spacing
Built on top of #8748, this PR reenables mypy strict mode and adds the appropriate corrections to pass the types checks.
* Upgrade mypy
* First step for acme
* Cast for the rescue
* Fixing types for certbot
* Fix typing for certbot-nginx
* Finalize type fixes, configure no optional strict check for mypy in tox
* Align requirements
* Isort
* Pylint
* Protocol for python 3.6
* Use Python 3.9 for mypy, make code compatible with Python 3.8<
* Pylint and mypy
* Pragma no cover
* Pythonic NotImplemented constant
* More type definitions
* Add comments
* Simplify typing logic
* Use vararg tuple
* Relax constraints on mypy
* Add more type
* Do not silence error if target is not defined
* Conditionally import Protocol for type checking only
* Clean up imports
* Add comments
* Align python version linting with mypy and coverage
* Just ignore types in an unused module
* Add comments
* Fix lint
* Work in progress
* Finish type control
* Isort
* Fix pylint
* Fix imports
* Fix cli subparser
* Some fixes
* Coverage
* Remove --no-strict-optional (obviously...)
* Update certbot-apache/certbot_apache/_internal/configurator.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/certbot/_internal/display/completer.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Cleanup dns_google
* Improve lock controls and fix subparser
* Use the expected interfaces
* Fix code
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Fixes#8425
This PR upgrades mypy to the latest version available, 0.812.
Given the advanced type inference capabilities provided by this newer version, this PRs also fixes various type inconsistencies that are now detected. Here are the non obvious changes done to fix types:
* typing in mixins has been solved using `Protocol` classes, as recommended by mypy (https://mypy.readthedocs.io/en/latest/more_types.html#mixin-classes, https://mypy.readthedocs.io/en/stable/protocols.html)
* `cast` when we are playing with `Union` types
This PR also disables the strict optional checks that have been enable by default in recent versions of mypy. Once this PR is merged, I will create an issue to study how these checks can be enabled.
`typing.Protocol` is available only since Python 3.8. To keep compatibility with Python 3.6, I try to import the class `Protocol` from `typing`, and fallback to assign `object` to `Protocol` if that fails. This way the code is working with all versions of Python, but the mypy check can be run only with Python 3.8+ because it needs the protocol feature. As a consequence, tox runs mypy under Python 3.8.
Alternatives are:
* importing `typing_extensions`, that proposes backport of newest typing features to Python 3.6, but this implies to add a dependency to Certbot just to run mypy
* redesign the concerned classes to not use mixins, or use them differently, but this implies to modify the code itself even if there is nothing wrong with it and it is just a matter of instructing mypy to understand in which context the mixins can be used
* ignoring type for these classes with `# type: ignore` but we loose the benefit of mypy for them
* Upgrade mypy
* First step for acme
* Cast for the rescue
* Fixing types for certbot
* Fix typing for certbot-nginx
* Finalize type fixes, configure no optional strict check for mypy in tox
* Align requirements
* Isort
* Pylint
* Protocol for python 3.6
* Use Python 3.9 for mypy, make code compatible with Python 3.8<
* Pylint and mypy
* Pragma no cover
* Pythonic NotImplemented constant
* More type definitions
* Add comments
* Simplify typing logic
* Use vararg tuple
* Relax constraints on mypy
* Add more type
* Do not silence error if target is not defined
* Conditionally import Protocol for type checking only
* Clean up imports
* Add comments
* Align python version linting with mypy and coverage
* Just ignore types in an unused module
* Add comments
* Fix lint
Fixes#8427
This PR converts the Python 2 types hints into Python 3 types annotations. I have used the project https://github.com/ilevkivskyi/com2ann which has been designed for that specific purpose and did that very well.
The only remaining things to do were to fix broken type hints that became wrong code after migration, and to fix lines too long with the new syntax.
* Raw execution of com2ann
* Fixing broken type annotations
* Cleanup imports
There are still some left, but the `modification_check` test fails. Some are still in `tools`, and they can probably be removed as well. `with_statement` was introduced officially in Python 2.5, so there's really old stuff in the code base.
Fixes https://github.com/certbot/certbot/issues/8494.
I left the `six` dependency pinned in `tests/letstest/requirements.txt` and `tools/oldest_constraints.txt` because `six` is still a transitive dependency with our current pinnings.
The extra moving around of imports is due to me using `isort` to help me keep dependencies in sorted order after replacing imports of `six`.
* remove some six usage in acme
* remove six from acme
* remove six.add_metaclass usage
* fix six.moves.zip
* fix six.moves.builtins.open
* six.moves server fixes
* 's/six\.moves\.range/range/g'
* stop using six.moves.xrange
* fix urllib imports
* s/six\.binary_type/bytes/g
* s/six\.string_types/str/g
* 's/six\.text_type/str/g'
* fix six.iteritems usage
* fix itervalues usage
* switch from six.StringIO to io.StringIO
* remove six imports
* misc fixes
* stop using six.reload_module
* no six.PY2
* rip out six
* keep six pinned in oldest constraints
* fix log_test.py
* update changelog
* test: certbot-ci crash due to no p521 on boulder
The bugfix in #8598 added an integration test to request a certificate
for an EC P-521 key, which is unsupported when ACME_SERVER=boulder,
failing our nightly integration tests.
* add an integration test for all EC curves
* Fix EC curve name typo in crypto_util
Fix typo of secp521r1 in crypto util module.
- secp521r1 is to be supported by certbot, but a typo of "SECP521R1" in the input validation section of the make_key function results in an error being thrown
* Add myself to authors.md
Add myself to authors.md ^^
* Add test for secp521r1 key generation
Add test for secp521r1 key generation to cli-tests
[As discussed in Mattermost](https://opensource.eff.org/eff-open-source/pl/yhtp4qu4zpfczm5wxmzxhndrto), our Apache test farm tests are failing because the CA certificate in the old version of boulder we have pinned expired over the weekend. This PR fixes that by running a local Pebble instance instead of an external boulder instance.
* switch from external boulder to local pebble
* add --http-01-port to run_acme_server
* Edit certs -> certificates in user-facing text.
To reduce confusion, we should consistently use the full term.
* Edit certs->certificates in more user-facing text.
* fix failing lint (line too long)
* fix typo
Co-authored-by: Jacob Hoffman-Andrews <github@hoffman-andrews.com>
Co-authored-by: Alex Zorin <alex@zorin.id.au>
In 96a05d9, mypy testing was added to certbot-ci, but introduced an
undeclared dependency on acme.magic_typing, resulting in a crash when
run under the integration-external tox environment.
This change uses the typing module in certbot-ci in place of
acme.magic_typing. It is already provided via dev_constraints.
Fixes https://github.com/certbot/certbot/issues/8495.
To further explain the problem here, `modify_kwargs_for_default_detection` as called in `add` is simplistic and doesn't always work. See https://github.com/certbot/certbot/issues/6164 for one other example.
In this case, were bitten by the code d1e7404358/certbot/certbot/_internal/cli/helpful.py (L393-L395)
The action used for deprecated arguments isn't in `ZERO_ARG_ACTIONS` so it assumes that all deprecated flags take one parameter.
Rather than trying to fix this function (which I think can only realistically be fixed by https://github.com/certbot/certbot/issues/4493), I took the approach that was previously used in `HelpfulArgumentParser.add_deprecated_argument` of bypassing this extra logic entirely. I adapted that function to now call `HelpfulArgumentParser.add` as well for consistency and to make testing easier.
* Rename deprecated arg action class
* Skip extra parsing for deprecated arguments
* Add back test of --manual-public-ip-logging-ok
* Add changelog entry
* nginx: fix py2 unicode sandwich
The nginx parser would crash when saving configuraitons containing
Unicode, because py2's `str` type does not support Unicode.
This change fixes that crash by ensuring that a string type supporting
Unicode is used in both Python 2 and Python 3.
* nginx: add unicode to the integration test config
* update CHANGELOG
* tests: add certbot-dns-rfc2136 integration tests
* dont use 'with' form of socket.socket
fixes py2 crash
* address some feedback:
- conftest: make DNS server a global resource
- conftest: add dns_xdist parameter into node config
- conftest: add --dns-server=bind flag
- conftest: if configured, point the ACME server to the DNS server
- dnsserver: make it sort-of compatible with xdist (future-proofing)
- context: parameterize dns-rfc2136 credentials file (future proofing)
- context: reduce dns-rfc2136 propagation time to speed up tests
- tox: add a integration-dns-rfc2136 target
- rfc2136: add a test/zone for subdelegation
- rfc2136: skip tests if no DNS server is configured
* try add integration-dns-rfc2136 to CI
* mock recursive dns via RPZ
* update --dns-server args and tox.ini args
* address more feedback:
- dns_server: rename rfc2136 creds file to .tpl
- dns_server: dont vary dns server port, instead we will vary zone names (#8455)
- dns_server: log error if bind9 fails to stop cleanly
- dns_server: replace assert with raise
- context: remove redundant _worker_id
- context: remove redundant cleanup override
- context: fix seek/flush in credentials context manager
- context: rename skip_if_no_server -> ...bind_server
- context: add newline EOF
* conftest: document _setup_primary_node sideeffects
* ci: rfc2136-integration from standard->nightly
* fix _stop_bind (function was renamed to stop)
* ignore errors from shutil.rmtree during cleanup
* dns_server: check for crash while polling
* remove --dry-run from rfc2136 test
Fixes#7717
This PR adds a `--dns-server` option to the `run_acme_server` test tool, in order to provide an arbitrary DNS server to Pebble or Boulder for the integration tests.
I also take this occasion to make `run_acme_server` a real CLI tool using argparse, and set the `--server-type` (default `pebble`) option as well.
* Set --dns-server flag in run_acme_server
* Default to pebble
* Add documentation
* Configure also Boulder
Fixes#8365
This PR adds a control when `certbot certonly` or `certbot run` are called for a certificate that already exists and would eventually be replaced. As described in #8365, this control is here to ensure that the user will not modify the key type of their certificate (eg. ECDSA to RSA) without an explicit approval (set explicitly `--cert-name` and `--key-type`), since RSA is the default if not specified.
* Handle unexpected key type migration.
* Update certbot-ci/certbot_integration_tests/certbot_tests/test_main.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Thanks to @pahrohfit and @Tomoyuki-GH for previous efforts to implement
suport for this.
Co-Authored-By: Robert Dailey <rob@wargam.es>
Co-Authored-By: Tomoyuki-GH <55397638+Tomoyuki-GH@users.noreply.github.com>
