Fixes#8427
This PR converts the Python 2 types hints into Python 3 types annotations. I have used the project https://github.com/ilevkivskyi/com2ann which has been designed for that specific purpose and did that very well.
The only remaining things to do were to fix broken type hints that became wrong code after migration, and to fix lines too long with the new syntax.
* Raw execution of com2ann
* Fixing broken type annotations
* Cleanup imports
* nginx: authenticate all matching vhosts for HTTP01
Previously, the nginx authenticator would set up the HTTP-01 challenge
response on a single HTTP vhost which matched the challenge domain.
The nginx authenticator will now set the challenge response on every
vhost which matches the challenge domain, including duplicates and HTTPS
vhosts.
This makes the authenticator usable behind a CDN where all origin
traffic is performed over HTTPS and also makes the authenticator work
more reliably against "invalid" nginx configurations, such as those
where there are duplicate vhosts.
* some typos
* dont authenticate the same vhost twice
One vhost may appear in both the HTTP and HTTPS vhost lists. Use a set()
to avoid trying to mod the same vhost twice.
* fix type annotations
* rewrite changelog entry
Fixes https://github.com/certbot/certbot/issues/8494.
I left the `six` dependency pinned in `tests/letstest/requirements.txt` and `tools/oldest_constraints.txt` because `six` is still a transitive dependency with our current pinnings.
The extra moving around of imports is due to me using `isort` to help me keep dependencies in sorted order after replacing imports of `six`.
* remove some six usage in acme
* remove six from acme
* remove six.add_metaclass usage
* fix six.moves.zip
* fix six.moves.builtins.open
* six.moves server fixes
* 's/six\.moves\.range/range/g'
* stop using six.moves.xrange
* fix urllib imports
* s/six\.binary_type/bytes/g
* s/six\.string_types/str/g
* 's/six\.text_type/str/g'
* fix six.iteritems usage
* fix itervalues usage
* switch from six.StringIO to io.StringIO
* remove six imports
* misc fixes
* stop using six.reload_module
* no six.PY2
* rip out six
* keep six pinned in oldest constraints
* fix log_test.py
* update changelog
* nginx: fix py2 unicode sandwich
The nginx parser would crash when saving configuraitons containing
Unicode, because py2's `str` type does not support Unicode.
This change fixes that crash by ensuring that a string type supporting
Unicode is used in both Python 2 and Python 3.
* nginx: add unicode to the integration test config
* update CHANGELOG
* nginx: reduced CLI logging when reloading nginx
Hides the output of `nginx -s reload` from the CLI, moving it to
debug-level logging.
Additionally, fixes an issue where Certbot did not properly capture the
output of the nginx reload and restart commands.
Fixes#8231
* remove leftover debugging
* reorder CHANGELOG
* don't use bare asserts
This commit fixes an issue with the nginx parser where it would perform
case-sensitive matching against server_name.
This would cause the authenticator and installer to ignore existing
virtualhosts containing uppercase characters, resulting in duplicate
virtualhosts and broken configurations.
"Exact" and "wildcard" matching is now case-insensitive. Regex-based
matching will continue to respect the case mode of the pattern.
Fixes#6776.
* nginx: add --nginx-sleep-seconds
As described in #7422, reloading nginx is an asynchronous process and
Certbot does not know when it is complete. In an environment where this
reload takes a long time, the nginx plugin suffers from an issue where
it responds to and fails the ACME challenge before the nginx server is
ready to serve it.
Following the discussion in a previous PR #7740, this commit introduces
a new flag, --nginx-sleep-seconds, which may be used to increase the
duration that Certbot will wait for nginx to reload, from its previously
hard-coded value of 1s.
Fixes#7422
* update CHANGELOG
* nginx: update docstring for nginx_restart
* Fix dangerous default argument
* Remove unused imports
* Remove unnecessary comprehension
* Use literal syntax to create data structure
* Use literal syntax instead of function calls to create data structure
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
This PR builds on #7657 and cleans up additional unnecessary pylint comments and some stray comments referring to pylint: disable comments that have been deleted that I didn't notice in my review of that PR.
* Remove stray pylint link.
* Cleanup more pylint comments
* Cleanup magic_typing imports
* Remove unneeded pylint: enable comments
Part of #7550
This PR makes appropriate corrections to run pylint on Python 3.
Why not keeping the dependencies unchanged and just run pylint on Python 3?
Because the old version of pylint breaks horribly on Python 3 because of unsupported version of astroid.
Why updating pylint + astroid to the latest version ?
Because this version only fixes some internal errors occuring during the lint of Certbot code, and is also ready to run gracefully on Python 3.8.
Why upgrading mypy ?
Because the old version does not support the new version of astroid required to run pylint correctly.
Why not upgrading mypy to its latest version ?
Because this latest version includes a new typshed version, that adds a lot of new type definitions, and brings dozens of new errors on the Certbot codebase. I would like to fix that in a future PR.
That said so, the work has been to find the correct set of new dependency versions, then configure pylint for sane configuration errors in our situation, disable irrelevant lintings errors, then fixing (or ignoring for good reason) the remaining mypy errors.
I also made PyLint and MyPy checks run correctly on Windows.
* Start configuration
* Reconfigure travis
* Suspend a check specific to python 3. Start fixing code.
* Repair call_args
* Fix return + elif lints
* Reconfigure development to run mainly on python3
* Remove incompatible Python 3.4 jobs
* Suspend pylint in some assertions
* Remove pylint in dev
* Take first mypy that supports typed-ast>=1.4.0 to limit the migration path
* Various return + else lint errors
* Find a set of deps that is working with current mypy version
* Update local oldest requirements
* Remove all current pylint errors
* Rebuild letsencrypt-auto
* Update mypy to fix pylint with new astroid version, and fix mypy issues
* Explain type: ignore
* Reconfigure tox, fix none path
* Simplify pinning
* Remove useless directive
* Remove debugging code
* Remove continue
* Update requirements
* Disable unsubscriptable-object check
* Disable one check, enabling two more
* Plug certbot dev version for oldest requirements
* Remove useless disable directives
* Remove useless no-member disable
* Remove no-else-* checks. Use elif in symetric branches.
* Add back assertion
* Add new line
* Remove unused pylint disable
* Remove other pylint disable
* Connect certbot-ci to travis. Remove old bash files.
* Configure test-everything
* Protect against import error
* Remove unused ignore
* Better handling of urllib3
* Correct path
* Remove a warning
* Correct call
* Protect atexit register execution
* Update docs/contributing.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update docs/contributing.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Add again some bash scripts to avoid breaking to much retro-compatiblity on third party scripts
* Move boulder-v1 and boulder-v2 in nightly tests
* Separate oldest unit tests and oldest integration tests
* Remove try/except
* Test integration included in toxenv
* Add a wait to avoid a transient issue on OCSP status in oldest tests
* Clean travis.yml, split other tests
* Remove useless config
* Update .travis.yml
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update tox.ini
* Update tox.ini
* Remove pytest-sugar
* Remove empty pytest.ini, tests are working without it
* Remove tls-sni from nginx config
* Add a dedicated configuration to define what is the HTTPS port for this certbot instance.
* Correct some tests
* Reestablish default vhost creation
* Clean tls references for nginx integration tests
* Associate https_port only to tests and nginx
This PR is a part of the tls-sni-01 removal plan described in #6849.
This PR removes the tls-sni-01 challenge tests during the integration tests. The approach I used here is not to remove completely the existing test code, but simply editing it to use a http-01 challenge. Indeed:
* the current integration tests are strongly coupled, and would require more modifications that it is worth, because ...
* the certbot-ci project, that has already no tls-sni tests, will soon replace completely the current integration tests code.
If you execute `tests/lock_test.py` or `tox -e integration` on a fairly recent machine, you will get the following error during tests executing against a live Nginx instance:
```
no "ssl_certificate" is defined in server listening on SSL port while SSL handshaking, client: x.x.x.x, server: y:y:y:y:z
```
Indeed, having no defined ssl certificate for a ssl port would inevitably lead to an error during the handshake SSL process between a client and this mis-configured nginx instance.
However it was not a problem one year before, because the handshake was not occurring in practice: the test just need to have a nginx started, and then immediately proceed to modify the configuration with a correct SSL setup. And nginx was able to start with a mis-configuration on SSL.
But then this fix has been done: https://trac.nginx.org/nginx/ticket/178
Basically with this, validation of the configuration is done during nginx startup, that will refuse to start with invalid configuration on SSL. Consequently, all related tests are failing with a sufficiently up-to-date nginx. For now, it is not seen on Travis because Ubuntu Trusty is used, with an old Nginx.
The PR fixes that, by generating on the fly self-signed certificates in the two impacted tests, and pushing the right parameters in the Nginx configuration.
* Fix nginx configuration with self-signed certificates generated on the fly
* Fix lint/mypy
* Fix old cryptography
* Unattended openssl
* Update lock_test.py
This PR reactivates tls-sni-01 challenges on recent Boulder versions checkout for integration tests. This allows to continue testing this challenge until it is officially dropped from server (Boulder) and client (Certbot).
Reverts #6679.
* Remove tls-sni-01 challenges in integration tests
* Remove the tls-sni test in the less invasive way
* Correct code coverage from tls-sni logic not been tested anymore.
* Update certbot-boulder-integration.sh
In response to #5342.
Currently, certbot will execute the operations necessary to validate a challenge even if the challenge has already been validated before against the acme ca server. This can occur for instance if a certificate is asked and issue correctly, then deleted locally, then asked again.
It is a corner case, but it will lead to some heavy operations (like updating a DNS zone, or creating an HTTP server) that are not needed.
This PR corrects this behavior by not executing challenges already validated, and use them directly instead to issue the certificate.
Fixes#5342
* Avoid to execute a given challenge that have been already validated by acme ca server.
* Execute tls challenge on a separate dns name, to avoid reusing the existing valid http challenge.
* Align with master
* Improve log
* Simplify the implementation
* Update changelog
* Add a unit test to ensure that validated challenges are not rerun
* flip challenge preference in Nginx
* Fix Nginx tests
* Flip challenge preference in Apache
* Flip challenge preference in standalone
* update changelog
* continue to run with tls-sni in integration tests for coverage
* Add a rewrite directive for the .well-known location so we don't hit existing rewrites
* add comment
* Add (nonexistent) document root so we don't use the default value
* Add integration tests for nginx plugin
* add a sleep 5 to test on travis
* put sleep 5 in the right spot
* test return status of grep respecting -e and note that we're actually not posix compliant
* redelete newline
* get http01 challenge working
* support multiple challenge types in configurator.py
* update existing nginx tests
* lint
* refactor NginxHttp01 and NginxTlsSni01 to both now inherit from NginxChallengePerformer
* remove TODO
* challenges_test tests with both tlssni01 and http01
* Make challenges.py more abstract to make lint happier
* add pylint disables to the tests to make pylint happier about the inheritance and abstraction situation
* no need to cover raise NotImplementedError() lines
* python3 compatibility
* test that http01 perform is called
* only remove ssl from addresses during http01
* Initialize addrs_to_add
* Change Nginx http01 to modify server block so the site doesn't stop serving while getting a cert
* pass existing unit tests
* rename sni --> http01 in unit tests
* lint
* fix configurator test
* select an http block instead of https
* properly test for port number
* use domains that have matching addresses
* remove debugger
* remove access_log and error_log cruft that wasn't being executed
* continue to return None from choose_redirect_vhost when create_if_no_match is False
* add nginx integration test
* Restructure add_server_directives to take a vhost as argument. This is the first step towards fixing vhost selection in nginx.
* Save path to vhost in file while parsing in get_vhosts().
* Disable creating a new server block when no names match.
* Make parser select vhost based on information in the vhost it found previously, rather than searching again for a match.
* Make add_server_directives update the passed vhost
* Update boulder config to pass test
* Add testing code for the _do_for_subarray function
* documentation and formatting updates
