Fixes#6850
This PR makes the last corrections needed to run all unit tests on Windows:
add a function to check if a hook is executable in a cross-platform compatible way
handle correctly the PATH surgery for Windows during hook execution
handle correctly an account compatibility over both ACMEv1 and ACMEv2
remove (finally!) the @broken_on_windows decorator.
* Fix account_tests
* Fix hook executable test
* Remove the temporary decorator @broken_on_windows
* Fix util_test
* No broken unit test on Windows anymore
* More elegant mock
* Fix context manager
* Adapt coverage
* Corrections
* Adapt coverage
* Forbid os.access
Fixes#7115
This PR creates a `realpath` method in `filesystem`, whose goal is to replace any call to `os.path.realpath` in Certbot. The reason is that `os.path.realpath` is broken on some versions of Python for Windows. See https://bugs.python.org/issue9949. The function created here works consistently across Linux and Windows.
As for the other forbidden functions in `os` module, our `certbot.compat.os` will raise an exception if its `path.realpath` function is invoked, and using the `os` module from Python is forbidden from the pylint check implemented in our CI.
Every call to `os.path.realpath` is corrected in `certbot` and `certbot-apache` modules.
* Forbid os.path.realpath
* Finish implementation
* Use filesystem.realpath
* Control symlink loops also for Linux
* Add a test for forbidden method
* Import a new object from os.path module
* Use same approach of wrapping than certbot.compat.os
* Correct errors
* Fix dependencies
* Make path module internal
This PR implements the filesystem.copy_ownership_and_apply_mode method from #6497.
This method is used in two places in Certbot, replacing os.chown, to copy the owner and group owner from a file to another one, and apply to the latter the given POSIX mode.
* Implement copy_ownership_and_apply_mode
* Update certbot/compat/os.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Remove default values
* Rewrite a comment.
* Relaunch CI
* Pass as keyword arguments
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Make the private key permissions transfer platform specific
* Update certbot/compat/filesystem.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Rename variable
* Fix comment0
* Add unit test for copy_ownership_and_apply_mode
* Adapt coverage
* Execute unconditionally chmod with copy_ownership_and_apply_mode. Improve doc.
This PR implements the filesystem.chmod method from #6497.
* Implement filesystem.chmod
* Conditionally add pywin32 on setuptools versions that support environment markers.
* Update apache plugin requirements
* Use a try/except import approach similar to lock
* Add comments about well-known SIDs
* Add main command
* Call filesystem.chmod in tests, remove one test
* Add test for os module
* Update environment marker
* Ensure we are not building wheels using an old version of setuptools
* Added a link to list of NTFS rights
* Simplify sid comparison
* Enable coverage
* Sometimes, double-quote is the solution
* Add entrypoint
* Add unit tests to filesystem
* Resolve recursively the link, add doc
* Move imports to the top of the file
* Remove string conversion of the ACL, fix setup
* Ensure admins have all permissions
* Simplify dacl comparison
* Conditionally raise for windows temporary workaround
* Add a test to check filesystem.chown is protected against symlink loops
This PR is the second part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This PR creates the module certbot.compat.os, that delegates everything to os, and that will be the safeguard against problematic methods of the standard module. On top of that, a quality check wrapper is called in the lint tox environment. This wrapper calls pylint and ensures that standard os module is no used directly in the certbot codebase.
Finally local oldest requirements are updated to ensure that tests will take the new logic when running.
* Add executable permissions
* Add the delegate certbot.compat.os module, add check coding style to enforce usage of certbot.compat.os instead of standard os
* Load certbot.compat.os instead of os
* Move existing compat test
* Update local oldest requirements
* Import sys
* Update account_test.py
* Update os.py
* Update os.py
* Update local oldest requirements
* Implement the new linter_plugin
* Fix local oldest for nginx
* Remove check coding style
* Update linter_plugin.py
* Add several comments
* Update the setup.py
* Add documentation
* Update acme dependencies
* Update certbot/compat/os.py
* Update docs/contributing.rst
* Update linter_plugin.py
* Handle os.path. Simplify checker.
* Add a comment to a reference implementation
* Update changelog
* Fix module registering
* Update docs/contributing.rst
* Update config and changelog
This PR is the first part of #6497 to ease the integration, following the new plan propose by @bmw here: #6497 (comment)
This step 1 refactor existing certbot.compat module into certbot.compat.misc, without any logic changed. Package certbot.compat will host the new modules that constitute the security model for Windows.
* Create the certbot.compat package. Move logic in certbot.compat.misc
* Add doc
* Fix lint
* Correct mypy
* Update client.py
This PR is a part of the tls-sni-01 removal plan described in #6849.
This PR removes --tls-sni-01-port, --tls-sni-01-address and tls-sni-01/tls-sni options from --preferred-challenges. They are replace by deprecation warning, indicating that these options will be removed soon.
This deprecation, instead of complete removal, is done to avoid certbot instances to hard fail if some automated scripts still use these flags for some users.
Once this PR lands, we can remove completely theses flags in one or two release.
* Remove tls-sni related flags in cli. Add a deprecation warning instead.
* Adapt tests to cli and renewal towards tls-sni flags deprecation
* Add https_port option. Make tls_sni_01_port show a deprecation warning, but silently modify https_port if set
* Migrate last items
* Fix lint
* Update certbot/cli.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Ensure to remove all occurences of tls-sni-01
* Remove unused parameter
* Revert modifications on cli-help.txt
* Use logger.warning instead of sys.stderr
* Update the logger warning message
* Remove standalone_supported_challenges option.
* Fix order of preferred-challenges
* Remove supported_challenges property
* Fix some tests
* Fix lint
* Fix tests
* Add a changelog
* Clean code, fix test
* Update CI
* Reload
* No hard date for tls-sni removal
* Remove useless cast to list
* Update certbot/tests/renewal_test.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Add entry to the changelog
* Add entry to the changelog
Explicit is better than implicit
When calling raise without an argument, Python will raise the last error occured from the caller except block. This makes my PyCharm very sad however. So this PR makes the function handling the error raising explicitly the error received as an argument.
This PR is a part of the effort to remove the last broken unit tests in certbot codebase for Windows, as described in #6850.
This PR fixes various unit tests on Windows, whose resolution was only to modify some logic in the tests, or minor changes in certbot codecase impacting Windows only (like handling correctly paths with DOS-style).
* Correct several tests
* Skip test definitively
* Test to be reactivated with #6497
* Mock log system to avoid errors due to multiple calls to main in main_test
* Simplify mock
* Update cli_test.py
* One test to be repaired when windows file permissions PR is merged
Fixes#5789
This PR is about allowing Certbot to respect the Retry-After HTTP header that an ACME CA server can return to a client POSTing to a challenge, to instruct him and retry the request later.
However, this feature was not easily implementable in the current code of certbot.auth_handler, because the code became really hard to read. In fact, @bmw was thinking that the code was really deceiving, and a lot of supposed functionalities declared in the comments were in fact not implemented or not functional.
So I took the time to understand what was going on, and effectively, most of the code is in fact not usable or not used. Then I did a refactoring against the bare ACME spec about what to do to prepare challenges, instruct the ACME CA server to perform them, then polling regularly the authorization resources until they are decided (valid or invalid).
And of course this implementation takes care of Retry-After ^^
I added a lot of comments in the new implementation, to explain what is going on for a future developer. The workflow I used is relying on the relationships between authorizations and challenges states as described in section 7.1.6 of the ACME spec draft: https://datatracker.ietf.org/doc/draft-ietf-acme-acme/
* Clean auth_handler a bit, and implement retry-after.
* Remove a debug logger
* Correct tests
* Fix mypy and lint. Setup max retries and default retry after accordingly.
* Ease a comparison in tests
* Update documentation
* Add tests
* Adapt windows coverage threshold to the global LOC reduction
* Update certbot/auth_handler.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Corrections under review
* Correction under review
* Update certbot/auth_handler.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Corrections under review
* Update auth_handler_test.py
* Reimplementing user readable report for failed authorizations
* Fixes two tests
* Fix another test + lint + mypy
* Update auth_handler.py
* Update auth_handler_test.py
* Fix tests
* Update certbot/auth_handler.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Raise directly the exception on polling timout
* Improve interface documentation
* Move the wait on top of the loop, to be used initially or after a new loop iteration. Do not wait for negative values.
* Always display the report about failed authorizations.
* Clarify an exception.
* Return, instead of break
* Use setdefault
* Remove useless assertion
* Adapt tests
* Improve a test about retry after value.
* Update certbot/auth_handler.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
* Add a complete test on best_effort
* Add entry to the changelog
* Gather all failed authzrs to be reported in one unique report in case of best_effort
* Build complete warn/report/raise process about failed authzrs
This PR updates Lexicon dependency to the latest version available, 3.0.6, for every lexicon-based DNS plugins. It updates also the provider construction to use the new ConfigResolverobject, and to remove the legacy configuration process.
* flip challenge preference in Nginx
* Fix Nginx tests
* Flip challenge preference in Apache
* Flip challenge preference in standalone
* update changelog
* continue to run with tls-sni in integration tests for coverage
So here we are: after #6361 has been merged, time is to provide an environment to execute the automated testing on Windows.
Here are the assertions used to build the CI on Windows:
every test running on Linux should ultimately be runnable on Windows, in a cross-platform compatible manner (there is one or two exception, when a test does not have any meaning for Windows),
currently some tests are not runnable on Windows: theses tests are ignored by default when the environment is Windows using a custom decorator: @broken_on_windows,
test environment should have functionalities similar to Travis, in particular an execution test matrix against various versions of Python and Windows,
so test execution is done through AppVeyor, as it supports the requirements: it add a CI step along Travis and Codecov for each PR, all of this ensuring that Certbot is entirely functional on both Linux and Windows,
code in tests can be changed, but code in Certbot should be changed as little as possible, to avoid regression risks.
So far in this PR, I focused on the tests on Certbot core and ACME library. Concerning the plugins, it will be done later, for plugins which have an interest on Windows. Test are executed against Python 3.4, 3.5, 3.6 and 3.7, for Windows Server 2012 R2 and Windows Server 2016.
I succeeded at making 258/259 of acme tests to work, and 828/868 of certbot core tests to work. Most of the errors where not because of Certbot itself, but because of how the tests are written. After redesigning some test utilitaries, and things like file path handling, or CRLF/LF, a lot of the errors vanished.
I needed also to ignore a lot of IO errors typically occurring when a tearDown test process tries to delete a file before it has been closed: this kind of behavior is acceptable for Linux, but not for Windows. As a consequence, and until the tearDown process is improved, a lot of temporary files are not cleared on Windows after a test campaign.
Remaining broken tests requires a more subtile approach to solve the errors, I will correct them progressively in future PR.
Last words about tox. I did not used the existing tox.ini for now. It is just to far from what is supported on Windows: lot of bash scripts that should be rewritten completely, and that contain test logic not ready/relevant for Windows (plugin tests, Docker compilation/test, GNU distribution versatility handling and so on). So I use an independent file tox-win.ini for now, with the goal to merge it ultimately with the existing logic.
* Define a tox configuration for windows, to execute tests against Python 3.4, 3.5, 3.6 and 3.7 + code coverage on Codecov.io
* Correct windows compatibility on certbot codebase
* Correct windows compatibility on certbot display functionalities
* Correct windows compatibility on certbot plugins
* Correct test utils to run tests on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows
* Correct tests on certbot core to run them both on windows and linux. Mark some of them as broken on windows for now.
* Lock tests are completely skipped on windows. Planned to be replace in next PR.
* Correct tests on certbot display to run them both on windows and linux. Mark some of them as broken on windows for now.
* Correct test utils for acme on windows. Add decorator to skip (permanently) or mark broken (temporarily) tests on windows.
* Correct acme tests to run them both on windows and linux. Allow a reduction of code coverage of 1% on acme code base.
* Create AppVeyor CI for Certbot on Windows, to run the test matrix (py34,35,36,37+coverage) on Windows Server 2012 R2 and Windows Server 2016.
* Update changelog with Windows compatibility of Certbot.
* Corrections about tox, pyreadline and CI logic
* Correct english
* Some corrections for acme
* Newlines corrections
* Remove changelog
* Use os.devnull instead of /dev/null to be used on Windows
* Uid is a always a number now.
* Correct linting
* PR https://github.com/python/typeshed/pull/2136 has been merge to third-party upstream 6 months ago, so code patch can be removed.
* And so acme coverage should be 100% again.
* More compatible tests Windows+Linux
* Use stable line separator
* Remove unused import
* Do not rely on pytest in certbot tests
* Use json.dumps to another json embedding weird characters
* Change comment
* Add import
* Test rolling builds #1
* Test rolling builds #2
* Correction on json serialization
* It seems that rolling builds are not canceling jobs on PR. Revert back to fail fast code in the pipeline.
* Warn users not to remove/undo previous challenges
* Even more specific DNS challenge message
* Fix spacing and variable names
* Create a second test DNS challenge for UI testing
* Changelog for subsequent manual challenge behavior
* Add and use a compatibility layer to allow certbot to be run on windows.
* Fix path comparison
* Corrections on compat and util for tests
* Less intrusive way to parse prefix in webroot plugin working for both linux and windows.
* Disable pylint import-error for some optional imports in compat.py
* Ensure path is normalized before prefixes are generated in webroot plugin
* Same prefixes in linux and windows, in fact root path is not needed in webroot plugin
* Check that user has administrative rights before continuing on windows (necessary for symlink creation)
* More straightforward way to test administrative rights on windows
* Try to resolve import error in travis ci
* OK. We go for full introspection to trick the ci.
* Move the administrative rights control to the certbot entrypoint
* Add comment for a really non trivial code.
* Allow some commands to be run on a shell without admin rights
* Avoid races conditions on windows for lock files
* Add sphinx doc to the compat functions.
* Remove irrelevant Windows error in the lock mechanism.
* Some corrections on compat
Implement an Authenticator which can fulfill a dns-01 challenge using the OVH DNS API. Applicable only for domains using OVH DNS.
Testing Done:
* `tox -e py27`
* `tox -e lint`
* Manual testing:
* Used `certbot certonly --dns-ovh -d`, specifying a credentials file as a command line argument. Verified that a certificate was successfully obtained without user interaction.
* Used `certbot certonly --dns-ovh -d`, without specifying a credentials file as a command line argument. Verified that the user was prompted and that a certificate was successfully obtained.
* Used `certbot certonly -d`. Verified that the user was prompted for a credentials file after selecting dnsimple interactively and that a certificate was successfully obtained.
* Used `certbot renew --force-renewal`. Verified that certificates
were renewed without user interaction.
* Negative testing:
* Path to non-existent credentials file.
* Credentials file with unsafe permissions (644).
* Path to credentials file with an invalid application key.
* Path to credentials file with an invalid application secret.
* Path to credentials file with an invalid consumer key.
* Path to credentials file with missing properties.
* Domain name not registered to OVH account.
Implement an Authenticator which can fulfill a dns-01 challenge using
the Gehirn DNS (Gehirn Infrastructure Service) API.
Applicable only for domains using Gehirn DNS for DNS.
Testing Done:
* `tox -e py27`
* `tox -e lint`
* Manual testing:
* Used `certbot certonly --dns-gehirn -d`, specifying a
credentials file as a command line argument. Verified that a
certificate was successfully obtained without user interaction.
* Negative testing:
* Path to non-existent credentials file.
* Credentials file with unsafe permissions (644).
* Domain name not registered to Gehirn DNS account.
This stops us from printing messages like:
"Could not choose appropriate plugin for updaters: Could not select or initialize the requested installer None."
when certbot renew --force-renewal is run with a lineage that doesn't have an installer.
* unquote None
* Test None values aren't saved in config file.
Implement an Authenticator which can fulfill a dns-01 challenge using
the Sakura Cloud DNS API.
Applicable only for domains using Sakura Cloud for DNS.
Testing Done:
* `tox -e py27`
* `tox -e lint`
* Manual testing:
* Used `certbot certonly --dns-sakuracloud -d`, specifying a
credentials file as a command line argument. Verified that a
certificate was successfully obtained without user interaction.
* Negative testing:
* Path to non-existent credentials file.
* Credentials file with unsafe permissions (644).
* Domain name not registered to Sakura Cloud account.
* Added DNS based authenticator plugin for Linode
* Added linode plugin to docs
* Added Dockerfile
* Added .gitignore and readthedocs.org.requirements.txt
* Updated default_propagation_seconds
* Updated according to changes requested
* Bump version to 0.26.0
* Advertise our packages work on Python 3.7.
interfaces.GenericUpdater and new enhancement interface updater functions get run on every invocation of Certbot with "renew" verb for every lineage. This causes performance problems for users with large configurations, because of plugin plumbing and preparsing happening in prepare() method of installer plugins. This PR moves the responsibility to call prepare() to the plugin (possibly) implementing a new style enhancement interface.
Fixes: #6153
* Do not call IPlugin.prepare() for updaters when running renew
* Check prepare called in tests
* Refine pydoc and make the function name more informative
* Verify the plugin type
This PR adds the functionality to enhance Apache configuration to include HTTP Strict Transport Security header with a low initial max-age value.
The max-age value will get increased on every (scheduled) run of certbot renew regardless of the certificate actually getting renewed, if the last increase took place longer than ten hours ago. The increase steps are visible in constants.AUTOHSTS_STEPS.
Upon the first actual renewal after reaching the maximum increase step, the max-age value will be made "permanent" and will get value of one year.
To achieve accurate VirtualHost discovery on subsequent runs, a comment with unique id string will be added to each enhanced VirtualHost.
* AutoHSTS code rebased on master
* Fixes to match the changes in master
* Make linter happy with metaclass registration
* Address small review comments
* Use new enhancement interfaces
* New style enhancement changes
* Do not allow --hsts and --auto-hsts simultaneuously
* MyPy annotation fixes and added test
* Change oldest requrements to point to local certbot core version
* Enable new style enhancements for run and install verbs
* Test refactor
* New test class for main.install tests
* Move a test to a correct test class
- Finishing refactor of postconf/postfix command-line utilities
- Plugin uses starttls_policy plugin to specify per-domain policies
Cleaning up TLS policy code.
Print warning when setting configuration parameter that is overridden by master.
Update client to use new policy API
Cleanup and test fixes
Documentation fix
smaller fixes
Policy is now an enhancement and reverting works
Added a README, and small documentation fixes throughout
Moving testing infra from starttls repo to certbot-postfix
fixing tests and lint
Changes against new policy API
starttls-everywhere => starttls-policy
testing(postfix): Added more varieties of certificates to test against.
Moar fixes against policy API.
Address comments on README and setup.py
Address small comments on postconf and util
Address comments in installer
Python 3 fixes and Postconf tester extends TempDir test class
Mock out postconf calls from tests and test coverage for master overrides
More various fixes. Everything minus testing done
Remove STARTTLS policy enhancement from this branch.
sphinx quickstart
99% test coverage
some cleanup and testfixing
cleanup leftover files
Remove print statement
testfix for python 3.4
Revert dockerfile change
mypy fix
fix(postfix): brad's comments
test(postfix): coverage to 100
test(postfix): mypy
import mypy types
fix(postfix docs): add .rst files and fix build
fix(postfix): tls_only and server_only params behave nicely together
some cleanup
lint
fix more comments
bump version number
* Remove unsupported pylint disable options
* star-args removed in Pylint 1.4.3
* abstract-class-little-used removed in Pylint 1.4.3
* Fixes new lint errors
* Copy dummy-variable-rgx expression to new ignored-argument-names expression to ignore unused funtion arguments
* Notable changes
* Refactor to satisfy Pylint no-else-return warning
* Fix Pylint inconsistent-return-statements warning
* Refactor to satisfy consider-iterating-dictionary
* Remove methods with only super call to satisfy useless-super-delegation
* Refactor too-many-nested-statements where possible
* Suppress type checked errors where member is dynamically added (notably derived from josepy.JSONObjectWithFields)
* Remove None default of func parameter for ExitHandler and ErrorHandler
Resolves#5973
The base class for Installer plugins `certbot.plugins.common.Installer` now provides functionality of `PluginStorage` to all installer plugins. This allows a plugin to save and retrieve variables in between of invocations.
The on disk storage is basically a JSON file at `config_dir`/`.pluginstorage.json`, usually `/etc/letsencrypt/.pluginstorage.json`. The JSON structure is automatically namespaced using the internal plugin name as a namespace key. Because the actual storage is JSON, the supported data types are: dict, list, tuple, str, unicode, int, long, float, boolean and nonetype.
To add a variable from inside the plugin class:
`self.storage.put("my_variable_name", my_var)`
To fetch a variable from inside the plugin class:
`my_var = self.storage.fetch("my_variable_key")`
The storage state isn't written on disk automatically, but needs to be called:
`self.storage.save()`
* Plugin storage implementation
* Added config_dir to existing test mocks
* PluginStorage test cases
* Saner handling of bad config_dir paths
* Storage moved to Installer and not initialized on plugin __init__
* Finetuning and renaming
* Add rrset list fallback
* List dns.resourceRecordSets.list as required
* Handle list failures differently for add and del
* Quote record content
* disable not-callable for iter_entry_points
* List update permission
