Fixes#8425
This PR upgrades mypy to the latest version available, 0.812.
Given the advanced type inference capabilities provided by this newer version, this PRs also fixes various type inconsistencies that are now detected. Here are the non obvious changes done to fix types:
* typing in mixins has been solved using `Protocol` classes, as recommended by mypy (https://mypy.readthedocs.io/en/latest/more_types.html#mixin-classes, https://mypy.readthedocs.io/en/stable/protocols.html)
* `cast` when we are playing with `Union` types
This PR also disables the strict optional checks that have been enable by default in recent versions of mypy. Once this PR is merged, I will create an issue to study how these checks can be enabled.
`typing.Protocol` is available only since Python 3.8. To keep compatibility with Python 3.6, I try to import the class `Protocol` from `typing`, and fallback to assign `object` to `Protocol` if that fails. This way the code is working with all versions of Python, but the mypy check can be run only with Python 3.8+ because it needs the protocol feature. As a consequence, tox runs mypy under Python 3.8.
Alternatives are:
* importing `typing_extensions`, that proposes backport of newest typing features to Python 3.6, but this implies to add a dependency to Certbot just to run mypy
* redesign the concerned classes to not use mixins, or use them differently, but this implies to modify the code itself even if there is nothing wrong with it and it is just a matter of instructing mypy to understand in which context the mixins can be used
* ignoring type for these classes with `# type: ignore` but we loose the benefit of mypy for them
* Upgrade mypy
* First step for acme
* Cast for the rescue
* Fixing types for certbot
* Fix typing for certbot-nginx
* Finalize type fixes, configure no optional strict check for mypy in tox
* Align requirements
* Isort
* Pylint
* Protocol for python 3.6
* Use Python 3.9 for mypy, make code compatible with Python 3.8<
* Pylint and mypy
* Pragma no cover
* Pythonic NotImplemented constant
* More type definitions
* Add comments
* Simplify typing logic
* Use vararg tuple
* Relax constraints on mypy
* Add more type
* Do not silence error if target is not defined
* Conditionally import Protocol for type checking only
* Clean up imports
* Add comments
* Align python version linting with mypy and coverage
* Just ignore types in an unused module
* Add comments
* Fix lint
I recently noticed that we only support versions of `setuptools` that support environment markers which allows us to simplify our `setup.py` files a bit.
In #8649 we added some code to trick pynsist and make it understand that `abi3` wheels for Windows are forward compatible, meaning that the cryptography wheel tagged `cp36-abi3` is in fact compatible with Python 3.6+, and not only Python 3.6.
Since pynsist 2.7 the tool now understand `abi3` wheels properly, and this trick is not needed anymore.
Please note that despite modifying the pynsist pinning in `dev_constraints.txt`, it will have no effect since pynsist currently escape the pinning system. This is handled in https://github.com/certbot/certbot/pull/8749.
* Pin pynsist
* Update dependencies
* Set windows installer a proper python project
* Optimize usage of the venvs
* Add windows-installer when venv is set up
* Fix call
* Remove env marker
Fixes#8700
Now that `snapcraft remote-build` truly uses new builds for each call, we can split the builds to have a dedicated Azure job for each target architecture. This PR does that.
* Split snap_build job on each architecture
* Also parallelize the publish_snap jobs over each architecture
Fixes#8661.
As mentioned in https://github.com/certbot/certbot/issues/8661#issuecomment-806168214, there are quite a few remaining references, but until we modify the release script, we still need those. The changes here and the list there were created by grepping for the following terms:
```
certbot-auto
cb-auto
cbauto
certbotauto
letsencrypt-auto
le-auto
leauto
letsencryptauto
LEAUTO
LE_AUTO
LETSENCRYPT_AUTO
LETSENCRYPTAUTO
CB_AUTO
CERTBOT_AUTO
CBAUTO
CERTBOTAUTO
```
* Remove references to certbot-auto from certbot code
* Remove references to LEAUTO
* Remove references to CERTBOT_AUTO
* Remove references to letsencrypt-auto
* Remove references to certbot-auto from docs and tools
* remove cli constants header files
* Remove Python virtual environment section
* Upgrade cryptography to 3.4.6
* Fix comment with instructions for how to use hashin
* run tools/rebuild_certbot_constraints.py
* add deps for building cryptography in snaps
* Update cryptography build dependencies for docker
* Update sources for test farm tests
* Remove rust if it's installed for test farm tests
* source bootstrap script and call sudo as needed
We observed recently several unexpected behavior during the execution of snap jobs in Azure. In particular it seems that `snapcraft remote-build` is tending to reattach to the latest builds on Launchpad triggered by the nightly builds on master, independently from the actual branch, status of the code, or targeted architectures.
Primarily if the builds on Launchpad are stalled for some reason, it blocks effectively any other Azure snap jobs until someone manually cancel the builds on Launchpad. Secondarily it means that the outcome of the builds may be inconsistent, because they can be the result of a build for the master source even if you are on a PR that modifieds these sources (including `snapcraft.yaml`).
After digging in `snapcraft` source code, I realized that the signature computed to understand if a build should be resumed, is not based one some hashes against the snapcraft working directory content, but is simply a hash of the working directory absolute path *itself*. It means that every builds triggered from the working directory `/my/path/certbot` for instance, are recognized as the same unique build on Launchpad side, and may be resumed if they already exist, and so independently from the source code, `snapcraft.yaml` or targeted archs.
For the record, relevant parts in `snapcraft` source code:
82024d3748/snapcraft/project/_project.py (L44)82024d3748/snapcraft/project/_project.py (L86-L89)82024d3748/snapcraft/cli/remote.py (L128-L132)
This PR makes effectively the resume build mechanism effectively a noop by moving the source code first in a temporary directory with random name before running `snapcraft remote-build`. This way the signature is never the same and builds are always recognized as brand new builds.
* Invalidate snapcraft remote-build cache by using a temporary workspace.
* Capture one more state in the build
* Precise the certificate naming convention mechanism in a note.
* Add certificate name convention in user guide, refer to it in compatibility page.
* Update certbot/docs/compatibility.rst
Co-authored-by: alexzorin <alex@zor.io>
* Update certbot/docs/using.rst
Co-authored-by: alexzorin <alex@zor.io>
* Update certbot/docs/using.rst
Co-authored-by: alexzorin <alex@zor.io>
* Improve the note about naming conventions
Co-authored-by: alexzorin <alex@zor.io>
While working on #8640, I realized that there were some hidden circular dependencies in certbot._internal.cli package. Then cerbot could break if the order of these imports changes.
This PR fixes that and apply isort on top of the result.
* Kill snapcraft build when a "Chroot problem" is encountered
* Display specific helper for "Chroot problem" status and cancel retry mechanism in this case.
* Isolate build tmp directories
* Configure XDG_CACHE_HOME
* Kill snapcraftctl with chroot problem is encountered
Fixes#8427
This PR converts the Python 2 types hints into Python 3 types annotations. I have used the project https://github.com/ilevkivskyi/com2ann which has been designed for that specific purpose and did that very well.
The only remaining things to do were to fix broken type hints that became wrong code after migration, and to fix lines too long with the new syntax.
* Raw execution of com2ann
* Fixing broken type annotations
* Cleanup imports
There are still some left, but the `modification_check` test fails. Some are still in `tools`, and they can probably be removed as well. `with_statement` was introduced officially in Python 2.5, so there's really old stuff in the code base.
Fixes https://github.com/certbot/certbot/issues/8690.
After this PR, we'll let the release script make its automated changes to certbot-auto as part of the 1.14.0 release and then never make any code changes to certbot-auto ever again!
* disable upgrades on debian
* update test_leauto_upgrades.sh
* update changelog
* revoke: try determine the server automatically
When revoking via --cert-name, use the server from the lineage (unless
overriden by the CLI).
* RenewableCert.storage might be None
* guard against an empty lineage server
* nginx: authenticate all matching vhosts for HTTP01
Previously, the nginx authenticator would set up the HTTP-01 challenge
response on a single HTTP vhost which matched the challenge domain.
The nginx authenticator will now set the challenge response on every
vhost which matches the challenge domain, including duplicates and HTTPS
vhosts.
This makes the authenticator usable behind a CDN where all origin
traffic is performed over HTTPS and also makes the authenticator work
more reliably against "invalid" nginx configurations, such as those
where there are duplicate vhosts.
* some typos
* dont authenticate the same vhost twice
One vhost may appear in both the HTTP and HTTPS vhost lists. Use a set()
to avoid trying to mod the same vhost twice.
* fix type annotations
* rewrite changelog entry
Fixing #8634. It's my first time contributing to this repository, if there's something wrong please let me know.
Before this fix
```
$ python3 extract_changelog.py 1.12.0
...
### Fixed
* Fixed the apache component on openSUSE Tumbleweed which no longer provides
an apache2ctl symlink and uses apachectl instead.
* Fixed a typo in `certbot/crypto_util.py` causing an error upon attempting `secp521r1` key generation
More details about these changes can be found on our GitHub repo.
```
After this fix
```
$ python3 extract_changelog.py 1.12.0
...
### Fixed
* Fixed the apache component on openSUSE Tumbleweed which no longer provides
an apache2ctl symlink and uses apachectl instead.
* Fixed a typo in `certbot/crypto_util.py` causing an error upon attempting `secp521r1` key generation
More details about these changes can be found on our GitHub repo.
```
