* dns-google: fix condition to don't use private dns zones
* update MD
* Fix condition
* fix condition
* update testdata
* fix identation
* update tests
* update changelog
* Update dns_google.py
* add test for split horizon dns google
* add dnsName to managed zones
* Add types in all DNS plugins
* Order imports
* Fix type
* Update certbot-dns-route53/certbot_dns_route53/_internal/dns_route53.py
Co-authored-by: alexzorin <alex@zor.io>
* Clean up imports
Co-authored-by: alexzorin <alex@zor.io>
This PR upgrades the pinned version of the dependencies. Version `1.9.0` of josepy is used so errors related to JWK serialization with EC keys (see https://github.com/certbot/josepy/issues/109) are fixed for Certbot.
Since Saturday the CI pipeline is failing due to several Sphinx errors. See https://dev.azure.com/certbot/certbot/_build/results?buildId=3928&view=logs&j=d74e04fe-9740-597d-e9fa-1d0400037dfd&t=dde413a4-f24c-59a0-9684-e33d79f9aa02
First, the build of certbot-dns-google is failing because of a particular configuration. It seems that this configuration has been written here to activate the support of the RST instruction `.. code-block:: json` in documentation. However, it does not seem to be necessary for a similar situation in certbot-dns-route53 documentation. So let's try to remove it and fix the Sphinx builds.
Second, Sphinx builds were not pinning dependencies, so Sphinx 4.x (that has been released yesterday) started to be used in the pipeline. Sadly this new version is not compatible with the plugin `repoze.sphinx.autointerface`, used to extract documentation from `zope.interface`. So I fixed the pinning and also explicitly pin Sphinx to 3.5.x for now.
Technically speaking the second action is sufficient to fix the first error, but I keep the dedicated solution because it improves the documentation in my opinion.
This situation could be fixed by not requiring `repoze.sphinx.autointerface`, but this is possible only if we remove `zope.interface` from Certbot. Luckily I started the work few days ago ;).
* Remove explicit lexer call in certbot-dns-google doc builds.
* Write a valid JSON file in the documentation
* Apply constraints to sphinx build environments
* Pin Sphinx to 3.5.4
* Update dependencies
* Pin traitlets
This is one of the things that newer versions of `pylint` complains about.
* git grep -l super\( | xargs sed -i 's/super([^)]*)/super()/g'
* fix spacing
Built on top of #8748, this PR reenables mypy strict mode and adds the appropriate corrections to pass the types checks.
* Upgrade mypy
* First step for acme
* Cast for the rescue
* Fixing types for certbot
* Fix typing for certbot-nginx
* Finalize type fixes, configure no optional strict check for mypy in tox
* Align requirements
* Isort
* Pylint
* Protocol for python 3.6
* Use Python 3.9 for mypy, make code compatible with Python 3.8<
* Pylint and mypy
* Pragma no cover
* Pythonic NotImplemented constant
* More type definitions
* Add comments
* Simplify typing logic
* Use vararg tuple
* Relax constraints on mypy
* Add more type
* Do not silence error if target is not defined
* Conditionally import Protocol for type checking only
* Clean up imports
* Add comments
* Align python version linting with mypy and coverage
* Just ignore types in an unused module
* Add comments
* Fix lint
* Work in progress
* Finish type control
* Isort
* Fix pylint
* Fix imports
* Fix cli subparser
* Some fixes
* Coverage
* Remove --no-strict-optional (obviously...)
* Update certbot-apache/certbot_apache/_internal/configurator.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/certbot/_internal/display/completer.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Cleanup dns_google
* Improve lock controls and fix subparser
* Use the expected interfaces
* Fix code
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* Fix TTL mismatch leading to HTTP 412
This PR is a follow up from #8521 where we address the
issue of potentially having a mismatch of TTL when executing
a DNS change (transaction = deletion + additions). Let's say
we have a record `foo.org 30 IN TXT foo-content` with TTL 30s,
when creating challenge or cleaning we might need to perform
a deletion operation in the transaction. Currently certbot
would ask Google API to delete the foo record like this:
`foo.org 60 in TXT foo-content` ignoring the record's original
TTL and using 60s instead. This leads to HTTP 412 as Google would
expect a perfect match of what we want to delete with what it is
on the DNS. See also #8523
* remove ttl from default data to avoid confusions
* Refactor tests and add a missing case
This commit adds a test that covers the case when we are
deleting a TXT record which contains a single rrdatas. Also,
refactoring a couple of tests.
* Make get_existing_txt_rrset documentation more precise about return value
* Add missing assertions in tests.
* fix linting issues
* Mention fix on changelog
* Explain fix around user impact
* Explain what happens when no records are returned
* Update certbot/CHANGELOG.md
* Update certbot/CHANGELOG.md
* Added note to each DNS documentation index page to mention that plugins need to be installed and are not included as standard.
* Resolved issue with white space in doc files
* Changed wording as discussed in PR.
* Changing URL to new wildcard instructions link
* Update certbot-dns-cloudflare/certbot_dns_cloudflare/__init__.py
* Fix fetch of existing records from Google DNS
There has been many complaints regarding `certbot_dns_google` plugin
failing with:
* HTTP 412 - Precondition not met
* HTTP 409 - Conflict
See #6036. This PR fixes that situation. The bug lies on how we
fetch the TXT records from google. For large amount of records
the Google API paginates the result but we ignore the subsequent
pages and assume that if the record is not in the first response then
it doesn't exist. This leads to either HTTP 409, or HTTP 412 or both.
In this PR we leverage the use of filters on the API to get exactly
the records we are looking for. Apart from fixing the problem stated
above, it has the extra benefit of making the process faster by
reducing the amount of API calls and it doesn't require us to handle
any pagination logic
* Explain changes on CHANGELOG
* Edit AUTHORS.md
* make execute static
* Update certbot/CHANGELOG.md
Being more specific for which plugin this fix bug is meant for.
Co-authored-by: alexzorin <alex@zor.io>
* Fix if expression to be more python-idiomatic
Co-authored-by: alexzorin <alex@zor.io>
* Sort AUTHORS.md
* Simplify tests
Make rrs_mock modeling simpler and refactor
* Revert "Simplify tests"
This reverts commit 9de9623ba7.
* Reimplement conditional mock
We still want to use a conditional mock by make it more
simple to understand by using MagicMock.
* Revert "Sort AUTHORS.md"
This reverts commit b3aa35bcf1.
* Add name in AUTHORS.md
Co-authored-by: alexzorin <alex@zor.io>
This adds a 'Error parsing credentials file ...' wrapper to any errors
raised inside certbot-dns-google's usage of oauth2client, to make it
obvious to the user where the problem lies.
As pylint is evolving, it improves its accuracy, and several pylint error suppression (`# pylint: disable=ERROR) added in certbot codebase months or years ago are not needed anymore to make it happy.
There is a (disabled by default) pylint error to detect the useless suppressions (pylint-ception: `useless-suppression`). It is not working perfectly (it has also false-positives ...) but it is a good start to clean the codebase.
This PR removes several of these useless suppressions as detected by the current pylint version we use.
* Remove useless suppress
* Remove useless lines
Part of #7550
This PR makes appropriate corrections to run pylint on Python 3.
Why not keeping the dependencies unchanged and just run pylint on Python 3?
Because the old version of pylint breaks horribly on Python 3 because of unsupported version of astroid.
Why updating pylint + astroid to the latest version ?
Because this version only fixes some internal errors occuring during the lint of Certbot code, and is also ready to run gracefully on Python 3.8.
Why upgrading mypy ?
Because the old version does not support the new version of astroid required to run pylint correctly.
Why not upgrading mypy to its latest version ?
Because this latest version includes a new typshed version, that adds a lot of new type definitions, and brings dozens of new errors on the Certbot codebase. I would like to fix that in a future PR.
That said so, the work has been to find the correct set of new dependency versions, then configure pylint for sane configuration errors in our situation, disable irrelevant lintings errors, then fixing (or ignoring for good reason) the remaining mypy errors.
I also made PyLint and MyPy checks run correctly on Windows.
* Start configuration
* Reconfigure travis
* Suspend a check specific to python 3. Start fixing code.
* Repair call_args
* Fix return + elif lints
* Reconfigure development to run mainly on python3
* Remove incompatible Python 3.4 jobs
* Suspend pylint in some assertions
* Remove pylint in dev
* Take first mypy that supports typed-ast>=1.4.0 to limit the migration path
* Various return + else lint errors
* Find a set of deps that is working with current mypy version
* Update local oldest requirements
* Remove all current pylint errors
* Rebuild letsencrypt-auto
* Update mypy to fix pylint with new astroid version, and fix mypy issues
* Explain type: ignore
* Reconfigure tox, fix none path
* Simplify pinning
* Remove useless directive
* Remove debugging code
* Remove continue
* Update requirements
* Disable unsubscriptable-object check
* Disable one check, enabling two more
* Plug certbot dev version for oldest requirements
* Remove useless disable directives
* Remove useless no-member disable
* Remove no-else-* checks. Use elif in symetric branches.
* Add back assertion
* Add new line
* Remove unused pylint disable
* Remove other pylint disable
* Refactor tests out of module for certbot-dns-cloudflare
* Refactor tests out of module for certbot-dns-cloudxns
* Refactor tests out of module for certbot-dns-digitalocean
* Refactor tests out of module for certbot-dns-dnsimple
* Refactor tests out of module for certbot-dns-dnsmadeeasy
* Refactor tests out of module for certbot-dns-gehirn
* Refactor tests out of module for certbot-dns-google
* Refactor tests out of module for certbot-dns-linode
* Refactor tests out of module for certbot-dns-luadns
* Refactor tests out of module for certbot-dns-nsone
* Refactor tests out of module for certbot-dns-ovh
* Refactor tests out of module for certbot-dns-rfc2136
* Refactor tests out of module for certbot-dns-sakuracloud
* Refactor tests out of module for certbot-dns-route53
* Move certbot-dns-google testdata/ under tests/
* Use pytest for dns plugins
* Exclude pycache and .py[cod]
This PR is the second part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This PR creates the module certbot.compat.os, that delegates everything to os, and that will be the safeguard against problematic methods of the standard module. On top of that, a quality check wrapper is called in the lint tox environment. This wrapper calls pylint and ensures that standard os module is no used directly in the certbot codebase.
Finally local oldest requirements are updated to ensure that tests will take the new logic when running.
* Add executable permissions
* Add the delegate certbot.compat.os module, add check coding style to enforce usage of certbot.compat.os instead of standard os
* Load certbot.compat.os instead of os
* Move existing compat test
* Update local oldest requirements
* Import sys
* Update account_test.py
* Update os.py
* Update os.py
* Update local oldest requirements
* Implement the new linter_plugin
* Fix local oldest for nginx
* Remove check coding style
* Update linter_plugin.py
* Add several comments
* Update the setup.py
* Add documentation
* Update acme dependencies
* Update certbot/compat/os.py
* Update docs/contributing.rst
* Update linter_plugin.py
* Handle os.path. Simplify checker.
* Add a comment to a reference implementation
* Update changelog
* Fix module registering
* Update docs/contributing.rst
* Update config and changelog
* Remove unsupported pylint disable options
* star-args removed in Pylint 1.4.3
* abstract-class-little-used removed in Pylint 1.4.3
* Fixes new lint errors
* Copy dummy-variable-rgx expression to new ignored-argument-names expression to ignore unused funtion arguments
* Notable changes
* Refactor to satisfy Pylint no-else-return warning
* Fix Pylint inconsistent-return-statements warning
* Refactor to satisfy consider-iterating-dictionary
* Remove methods with only super call to satisfy useless-super-delegation
* Refactor too-many-nested-statements where possible
* Suppress type checked errors where member is dynamically added (notably derived from josepy.JSONObjectWithFields)
* Remove None default of func parameter for ExitHandler and ErrorHandler
Resolves#5973
* Add rrset list fallback
* List dns.resourceRecordSets.list as required
* Handle list failures differently for add and del
* Quote record content
* disable not-callable for iter_entry_points
* List update permission
* Return str rather than bytes
Project id is returned as bytes, which causes issues when constructing the google cloud API url, converting `b'PROJECT_ID'` to `b%27PROJECT_ID%27` causing the request to fail.
* Ensure we handle both bytes and str types
* project_id should be a str or bytes, not int
- when no credentials are passed it will try to get valid credentials
using the google metadata service
- this is a feature of the google SDK, so we don't need to handle that
explicitly
- previous behaviour with a credentials file is retained
* Pin oldest version of packaged python deps
* Install security extras in oldest tests
* Revert "bump requests requirement to >=2.10 (#4248)"
This reverts commit 402ad8b353.
* Use create=True when patching open on module
Implement an Authenticator which can fulfill a dns-01 challenge using
the Google Cloud DNS API. Applicable only for domains using Google Cloud
DNS for DNS.
Testing Done:
* `tox -e py27`
* `tox -e lint`
* Manual testing:
* Used `certbot certonly --dns-google -d`, specifying a credentials
file as a command line argument. Verified that a certificate was
successfully obtained without user interaction.
* Used `certbot certonly --dns-google -d`, without specifying a
credentials file as a command line argument. Verified that the
user was prompted and that a certificate was successfully
obtained.
* Used `certbot certonly -d`. Verified that the user was prompted for
a credentials file after selecting google interactively and that
a certificate was successfully obtained.
* Used `certbot renew --force-renewal`. Verified that certificates
were renewed without user interaction.
* Negative testing:
* Path to non-existent credentials file.
* Credentials file with unsafe permissions (644).
* Domain name not registered to Google Cloud Platform account.
