Built on top of #8748, this PR reenables mypy strict mode and adds the appropriate corrections to pass the types checks.
* Upgrade mypy
* First step for acme
* Cast for the rescue
* Fixing types for certbot
* Fix typing for certbot-nginx
* Finalize type fixes, configure no optional strict check for mypy in tox
* Align requirements
* Isort
* Pylint
* Protocol for python 3.6
* Use Python 3.9 for mypy, make code compatible with Python 3.8<
* Pylint and mypy
* Pragma no cover
* Pythonic NotImplemented constant
* More type definitions
* Add comments
* Simplify typing logic
* Use vararg tuple
* Relax constraints on mypy
* Add more type
* Do not silence error if target is not defined
* Conditionally import Protocol for type checking only
* Clean up imports
* Add comments
* Align python version linting with mypy and coverage
* Just ignore types in an unused module
* Add comments
* Fix lint
* Work in progress
* Finish type control
* Isort
* Fix pylint
* Fix imports
* Fix cli subparser
* Some fixes
* Coverage
* Remove --no-strict-optional (obviously...)
* Update certbot-apache/certbot_apache/_internal/configurator.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/certbot/_internal/display/completer.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Cleanup dns_google
* Improve lock controls and fix subparser
* Use the expected interfaces
* Fix code
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
Fixes#8427
This PR converts the Python 2 types hints into Python 3 types annotations. I have used the project https://github.com/ilevkivskyi/com2ann which has been designed for that specific purpose and did that very well.
The only remaining things to do were to fix broken type hints that became wrong code after migration, and to fix lines too long with the new syntax.
* Raw execution of com2ann
* Fixing broken type annotations
* Cleanup imports
Fixes https://github.com/certbot/certbot/issues/8494.
I left the `six` dependency pinned in `tests/letstest/requirements.txt` and `tools/oldest_constraints.txt` because `six` is still a transitive dependency with our current pinnings.
The extra moving around of imports is due to me using `isort` to help me keep dependencies in sorted order after replacing imports of `six`.
* remove some six usage in acme
* remove six from acme
* remove six.add_metaclass usage
* fix six.moves.zip
* fix six.moves.builtins.open
* six.moves server fixes
* 's/six\.moves\.range/range/g'
* stop using six.moves.xrange
* fix urllib imports
* s/six\.binary_type/bytes/g
* s/six\.string_types/str/g
* 's/six\.text_type/str/g'
* fix six.iteritems usage
* fix itervalues usage
* switch from six.StringIO to io.StringIO
* remove six imports
* misc fixes
* stop using six.reload_module
* no six.PY2
* rip out six
* keep six pinned in oldest constraints
* fix log_test.py
* update changelog
Fixes https://github.com/certbot/certbot/issues/7913.
I only added the deprecation warning to `certbot.tests.util` because that's the only place where I think someone could be using the `mock` module through our API.
* remove external mock from acme
* update Certbot's mock usage
* remove mock dependency in plugins
* remove external mock from compatibility test
* add changelog entry
Fixes#8389#8584.
This PR makes the necessary modifications to officially drop Python 2 support in the Certbot project.
I did not remove the specific Python 2 compatibility branches that has been added in various places in the codebase, to reduce the size of this PR and this will be done in a future one
* Update classifiers and python_requires in setup.py
* Remove warnings about Python 2 deprecation
* Remove Azure jobs on Python 2.7
* Remove references to python 2 in documentation
* Pin dnspython to 2.1.0
* Update changelog
* Remove warning ignore
Fixes https://github.com/certbot/certbot/issues/8134.
* Test on Python 3.9.
* Mention Python 3.9 support in changelog.
* s/\( *'Pro.*3\.\)8\(',\)/\18\2\n\19\2/
* undo changes to tox.ini
* Move more tests to Python 3.9
* Update PyYAML and packages which pinned it back
* Upgrade typed-ast
* Use <= to "pin" dnspython
* Fix lint by telling pylint it cannot be trusted
* Disable mypy on RFC plugin
* add comment about <= support
* Remove python_version from mypy.ini.
* Fix magic_typing
* Ignore msvcrt usage.
* make mypy happier
* clean up changes
* Add type for reporter queue
* More mypy fixes
* Fix pyrfc3339 str.
* Remove unused import.
* Make certbot.util mypy work in both Pythons
* Fix typo
* Fixed a few linting warnings for if not x in y.
These should have been caught by pylint, but weren't.
* Replaced "x in y.keys()" with "x in y".
It's much faster, and more Pythonic.
random25863.example.org appears in multiple port 80 virtualhosts in the
nginx testdata tarball and also is in the nginx-roundtrip-testdata.
Certbot doesn't handle these properly, which results in random test
failures.
This commit ensures that random25863.example.org only appears in a
single virtualhost and should ensure that the tests pass consistently.
Fixes https://github.com/certbot/certbot/issues/8162.
I had to update the base of the Dockerfile to get a new enough version of Python 3. I also simplified things a lot and removed a lot of the comments that were essentially just describing how Dockerfiles work.
The most complicated changes here are in `testdata`. You can find a diff of the changes to `nginx.tar.gz` at https://gist.github.com/c7727db0cecf3f15f02439f085c73848.
The first problem was that there were some complaints from the new Apache/nginx/OpenSSL version about the 1024 bit RSA key so I updated `empty_cert.pem` both inside and outside of the tarball as well as the corresponding private key in the tarball to use a 2048 bit key.
The 2nd problem is trickier to understand. If you look at the output from nginx after loading the config from `lots/` you'll see it complaining about conflicting `server_name` directives for the directives I deleted. See https://dev.azure.com/certbot/certbot/_build/results?buildId=2578&view=logs&j=250aa146-b243-5f8f-bf86-17a529c9fb7e&t=9baa2014-9673-5e78-8f4f-7a463caf2bfa&l=1516.
After switching the tests to Python 3, tests on that domain started failing. What I believe to be happening is we were just lucky these tests were passing to begin with. In both the Apache and Nginx plugin, if there are conflicting virtual hosts like this, we just arbitrarily pick one. The relevant code here for nginx is 575092d603/certbot-nginx/certbot_nginx/_internal/configurator.py (L455)
I played around with a debugger and confirmed that before I removed the conflicting server names, there were two exact matches for the domain we were searching for here.
I think all that's going on is with the switch to Python 3, the vhost we happen to choose changes and "breaks" the test. I suspect this to be due to something like getting values out of a dict somewhere where the order of items in a dict while iterating over it is different between Python 2 and 3. I didn't track where this difference happens down, but I personally don't think it's a good use of time since I think the real problem here is that the nginx config being tested was invalid with conflicting `server` blocks.
I removed all references to the `server_name` causing conflicts in that nginx configuration because both server blocks had other domains that are being tested, but I could add either back if you prefer. You can see the `nginx_compat` test passing with these changes at https://dev.azure.com/certbot/certbot/_build/results?buildId=2587&view=logs&j=250aa146-b243-5f8f-bf86-17a529c9fb7e.
* update Dockerfile
* Fix apache_compat on py3.
* Update empty_cert.pem.
The command used here was `openssl req -key
certbot/certbot/tests/testdata/rsa2048_key.pem -new -subj '/CN=example.com'
-x509 >
certbot-compatibility-test/certbot_compatibility_test/testdata/empty_cert.pem`.
* update nginx.tar.gz
* Remove conflicting server_names
According to `distutils/version.py`, StrictVersion is pretty strict in
what version numbers to accept:
> A version number consists of two or three dot-separated numeric
> components, with an optional "pre-release" tag on the end. The
> pre-release tag consists of the letter 'a' or 'b' followed by a number.
This assumption already fails for some pretty basic python libraries
itself, like setuptools, also available in `46.1.3.post20200610`, a
completely valid version number according to
https://www.python.org/dev/peps/pep-0440/#post-releases.
There doesn't seem to be a particular reason on why StrictVersion has
been used here, so let's use LooseVersion, to be compatible with these
versions.
Co-authored-by: Adrien Ferrand <adferrand@users.noreply.github.com>
* Fix dangerous default argument
* Remove unused imports
* Remove unnecessary comprehension
* Use literal syntax to create data structure
* Use literal syntax instead of function calls to create data structure
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
As pylint is evolving, it improves its accuracy, and several pylint error suppression (`# pylint: disable=ERROR) added in certbot codebase months or years ago are not needed anymore to make it happy.
There is a (disabled by default) pylint error to detect the useless suppressions (pylint-ception: `useless-suppression`). It is not working perfectly (it has also false-positives ...) but it is a good start to clean the codebase.
This PR removes several of these useless suppressions as detected by the current pylint version we use.
* Remove useless suppress
* Remove useless lines
