Part of #7550
This PR makes appropriate corrections to run pylint on Python 3.
Why not keeping the dependencies unchanged and just run pylint on Python 3?
Because the old version of pylint breaks horribly on Python 3 because of unsupported version of astroid.
Why updating pylint + astroid to the latest version ?
Because this version only fixes some internal errors occuring during the lint of Certbot code, and is also ready to run gracefully on Python 3.8.
Why upgrading mypy ?
Because the old version does not support the new version of astroid required to run pylint correctly.
Why not upgrading mypy to its latest version ?
Because this latest version includes a new typshed version, that adds a lot of new type definitions, and brings dozens of new errors on the Certbot codebase. I would like to fix that in a future PR.
That said so, the work has been to find the correct set of new dependency versions, then configure pylint for sane configuration errors in our situation, disable irrelevant lintings errors, then fixing (or ignoring for good reason) the remaining mypy errors.
I also made PyLint and MyPy checks run correctly on Windows.
* Start configuration
* Reconfigure travis
* Suspend a check specific to python 3. Start fixing code.
* Repair call_args
* Fix return + elif lints
* Reconfigure development to run mainly on python3
* Remove incompatible Python 3.4 jobs
* Suspend pylint in some assertions
* Remove pylint in dev
* Take first mypy that supports typed-ast>=1.4.0 to limit the migration path
* Various return + else lint errors
* Find a set of deps that is working with current mypy version
* Update local oldest requirements
* Remove all current pylint errors
* Rebuild letsencrypt-auto
* Update mypy to fix pylint with new astroid version, and fix mypy issues
* Explain type: ignore
* Reconfigure tox, fix none path
* Simplify pinning
* Remove useless directive
* Remove debugging code
* Remove continue
* Update requirements
* Disable unsubscriptable-object check
* Disable one check, enabling two more
* Plug certbot dev version for oldest requirements
* Remove useless disable directives
* Remove useless no-member disable
* Remove no-else-* checks. Use elif in symetric branches.
* Add back assertion
* Add new line
* Remove unused pylint disable
* Remove other pylint disable
In the diamond-include case, where a file is included by multiple other
files in the configuration tree, nodes can have multiple ancestors.
Change interfaces and existing implementations to account for this case.
* Fix metadata & primary references in Augeas tests.
When performing actions only on one of the trees in DualNodeParser, the two
trees get out-of-sync. Similarly, we can't expect that the metadata between
the two trees will remain the same.
Did a pass over the tests to re-wire metadata and primary usage.
* Add ApacheParser skeleton.
Fix plumbing in configurator & dualparser to initialize ApacheParser
alongside AugeasParser.
* Silence coverage reports for now
* Don't call core constants from nginx plugin
* Move constants.py to _internal/
* Move ENHANCEMENTS from now-internal constants to public plugins.enhancements
* Update display.enhancements.ask from its 2015 comment
* Create _internal package for Certbot's non-public modules
* Move account.py to _internal
* Move auth_handler.py to _internal
* Move cert_manager.py to _internal
* Move client.py to _internal
* Move error_handler.py to _internal
* Move lock.py to _internal
* Move main.py to _internal
* Move notify.py to _internal
* Move ocsp.py to _internal
* Move renewal.py to _internal
* Move reporter.py to _internal
* Move storage.py to _internal
* Move updater.py to _internal
* update apache and nginx oldest requirements
* Keep the lock file as certbot.lock
* nginx oldest tests still need to rely on newer certbot
* python doesn't have good dependency resolution, so specify the transitive dependency
* update required minimum versions in nginx setup.py
This pull request addresses #7451 by removing the deprecated flags.
* Dropped deprecated flags from commands
* Updated changelog for dropped flags and deleted outdated tests
* removed init-script part of apache test
This pull request ensures that we use distro package in all the distribution version detection. It also replaces the custom systemd /etc/os-release parsing and adds a few version fingerprints to Apache override selection.
Fixes: #7405
* Revert "Try to use platform.linux_distribution() before distro equivalent (#7403)"
This reverts commit ca3077d034.
* Use distro for all os detection code
* Address review comments
* Add changelog entry
* Added tests
* Fix tests to return a consistent os name
* Do not crash on non-linux systems
* Minor fixes to distro compatibility checks
* Make the tests OS independent
* Update certbot/util.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Skip linux specific tests on other platforms
* Test fixes
* Better test state handling
* Lower the coverage target for Windows tests
This is a big part of #7214. It removes all references to TLS-SNI-01 outside of acme (and pytest.ini). Those changes will come in a subsequent PR. I thought this one was getting big enough.
* Remove references to TLS-SNI-01 in Apache plugin
* Remove references to TLS-SNI-01 from certbot-nginx
* Remove references to TLS-SNI from Certbot.
* Remove TLS-SNI reference from docs
* add certbot changelog
* Clarify test behavior
While working on #7214, I noticed that certbot.plugins.common.TLSSNI01 wasn't printing a deprecation warning and it was still being used in our Apache plugin. This PR fixes that.
* find_comments implementation and AugeasCommentNode creation
* Use dummy value for ancestor
* Add NotImplementedError when calling find_comments with exact parameter
* Remove parameter 'exact' from find_comments interface
* Fix comment
This PR fixes a regression in #7337 (0.38.0) that certbot cannot run with Apache on RHEL 6.
In RHEL 6, `distro.linux_distribution()` returns `RedHatEnterpriseServer`.
In RHEL 6:
```py
>>> import distro
>>> distro.linux_distribution()
(u'RedHatEnterpriseServer', u'6.10', u'Santiago')
>>> import platform
>>> platform.linux_distribution()
('Red Hat Enterprise Linux Server', '6.10', 'Santiago')
```
In RHEL 7:
```py
>>> import distro
>>> distro.linux_distribution()
('Red Hat Enterprise Linux Server', '7.6', 'Maipo')
>>> import platform
>>> platform.linux_distribution()
('Red Hat Enterprise Linux Server', '7.6', 'Maipo')
```
* fix to run with Apache on RHEL 6
* fix docs
* DualParserNode, DualCommentNode and DualDirectiveNode implementations
* Add DualBlockNode
* Address review comments
* Address review comments
* Call the right assertion after name change
* Simplify isPass
* Add explanation to _create_matching_list pydoc
* Break when match was found
Add metadata keyword argument to the ParserNode interface, allowing the initialization of the object from contents of the metadata - if the implementation allows it. As an example, Augeas implementation needs nothing more than the Augeas DOM path of a configuration directive to be able to populate the ParserNode instance with all data relevant to the DirectiveNode.
The checks also allow skipping the otherwise required keyword arguments if metadata is provided.
* Allow creating ParserNode instances using information from metadata dictionary
* Update certbot-apache/certbot_apache/interfaces.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update certbot-apache/certbot_apache/interfaces.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Address review comments
* Fix filepath comment
* Update certbot-apache/certbot_apache/interfaces.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
Fixes#7212
This PR forbid os.stat and os.fstat, and fix or provide alternatives to avoid its usage in certbot outside of certbot.compat.filesystem.
* Reimplement private key mode propagation
* Remove other os.stat
* Remove last call of os.stat in certbot package
* Forbid stat and fstat
* Implement mode comparison checks
* Add unit tests
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Handle case where multiple ace concerns a given SID in has_min_permissions
* Add a new test scenario
* Add a simple test for has_same_ownership
* Fix name function
* Add a comment explaining an ACE structure
* Move a test in its dedicated class
* Improve a message error
* Calculate has_min_permission result using effective permission rights to be more generic.
* Change an exception message
* Add comments, avoid to skip a test.
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
This PR contains the changes requested in initial pre-review comments of #7308
Move properties to class pydocs in interfaces.py
Prefer class ABC register() functionality instead of class inheritance for interface classes
Add apache implementation specific functions to interfaces
* Move class argument definitions to class pydoc
* Add apache specific functionality to the interface
* Bring inheritance back
* Define initialization for different ParserNode classes
* Add parsernode utils to check keyword arguments and document the defaults in pydoc
* Fix pydocs and make BlockNode a child of DirectiveNode
* Refine docs, and remove unused __init__ from BlockNode
* Split parsernode util tests to their own respective file
* Skip cover for dummy calls to super
* Add types to method documentation
* Add documentation for children
This PR adds OVERRIDE_CLASS in certbot-apache/entrypoint.py for Scientific Linux. Fixes#7248.
* add OVERRIDE_CLASS for Scientific Linux os name
* add entry for Scientific Linux using "scientific" as key
* Update changelog
See https://community.letsencrypt.org/t/ssl-error-after-cert-renew/99430.
The first commit of this PR is a simple, clean revert of #7191. Subsequent commits add back pieces of that PR we want to keep.
I also reverted #7299 which landed in a separate PR, but needs to be reverted to keep including the TLS config files in the certbot-apache package when it is built.
I tested this on Ubuntu 18.04 by installing a cert to Apache using Certbot master and then running certbot renew with this branch. I watched the Apache plugin update the configuration file to remove SSLSessionTickets off.
* Revert "Disable TLS session tickets for Apache 2.4.11+ (#7191)"
This reverts commit 9174c631d9.
* Keep hashes with TLS session tickets disabled.
* dont delete changelog entries
* add changelog entry
* Revert "Clean the useless entries in MANIFEST.in (#7299)"
This reverts commit f4d17d9a6b.
* Implement the logic
* Update tests
* Fix lint and changelog
* Update configurator.py
* Move the TLS configs in a dedicated folder. Fix the formalism of their naming and location.
* Improve existing test to check all TLS config have their hash registered in Certbot
* Corrections after review
* Improve a test
* Remove commented useless lines in TLS configs
* Add a nice warning. Because I am nice.
* Fix lint
* Add a test
Fixes#7115
This PR creates a `realpath` method in `filesystem`, whose goal is to replace any call to `os.path.realpath` in Certbot. The reason is that `os.path.realpath` is broken on some versions of Python for Windows. See https://bugs.python.org/issue9949. The function created here works consistently across Linux and Windows.
As for the other forbidden functions in `os` module, our `certbot.compat.os` will raise an exception if its `path.realpath` function is invoked, and using the `os` module from Python is forbidden from the pylint check implemented in our CI.
Every call to `os.path.realpath` is corrected in `certbot` and `certbot-apache` modules.
* Forbid os.path.realpath
* Finish implementation
* Use filesystem.realpath
* Control symlink loops also for Linux
* Add a test for forbidden method
* Import a new object from os.path module
* Use same approach of wrapping than certbot.compat.os
* Correct errors
* Fix dependencies
* Make path module internal
