* Use josepy instead of acme.jose. (#5203)
* Parse variables without whitespace separator correctly in CentOS family of distributions (#5318)
* Pin josepy in letsencrypt-auto (#5321)
* pin josepy in le-auto
* Put pinned versions in sorted order
* Pin dependencies in oldest tests (#5316)
* Add tools/merge_requirements.py
* Revert "Fix oldest tests by pinning Google DNS deps (#5000)"
This reverts commit f68fba2be2.
* Add tools/oldest_constraints.txt
* Remove oldest constraints from tox.ini
* Rename dev constraints file
* Update tools/pip_install.sh
* Update install_and_test.sh
* Fix pip_install.sh
* Don't cat when you can cp
* Add ng-httpsclient to dev constraints for oldest tests
* Bump tested setuptools version
* Update dev_constraints comment
* Better document oldest dependencies
* test against oldest versions we say we require
* Update dev constraints
* Properly handle empty lines
* Update constraints gen in pip_install
* Remove duplicated zope.component
* Reduce pyasn1-modules dependency
* Remove blank line
* pin back google-api-python-client
* pin back uritemplate
* pin josepy for oldest tests
* Undo changes to install_and_test.sh
* Update install_and_test.sh description
* use split instead of partition
* More pip dependency resolution workarounds (#5339)
* remove pyopenssl and six deps
* remove outdated tox.ini dep requirement
* Fix auto_tests on systems with new bootstrappers (#5348)
* Fix pytest on macOS in Travis (#5360)
* Add tools/pytest.sh
* pass TRAVIS through in tox.ini
* Use tools/pytest.sh to run pytest
* Add quiet to pytest.ini
* ignore pytest cache
* print as a string (#5359)
* Use apache2ctl modules for Gentoo systems. (#5349)
* Do not call Apache binary for module reset in cleanup()
* Use apache2ctl modules for Gentoo
* Broader git ignore for pytest cache files (#5361)
Make gitignore take pytest cache directories in to account, even if
they reside in subdirectories.
If pytest is run for a certain module, ie. `pytest certbot-apache` the
cache directory is created under `certbot-apache` directory.
* Fix letsencrypt-auto name and long forms of -n (#5375)
* Deprecate Python2.6 by using Python3 on CentOS/RHEL 6 (#5329)
* If there's no python or there's only python2.6 on red hat systems, install python3
* Always check for python2.6
* address style, documentation, nits
* factor out all initialization code
* fix up python version return value when no python installed
* add no python error and exit
* document DeterminePythonVersion parameters
* build letsencrypt-auto
* close brace
* build leauto
* fix syntax errors
* set USE_PYTHON_3 for all cases
* rip out NOCRASH
* replace NOCRASH, update LE_PYTHON set logic
* use built-in venv for py3
* switch to LE_PYTHON not affecting bootstrap selection and not overwriting LE_PYTHON
* python3ify fetch.py
* get fetch.py working with python2 and 3
* don't verify server certificates in fetch.py HttpsGetter
* Use SSLContext and an environment variable so that our tests continue to never verify server certificates.
* typo
* build
* remove commented out code
* address review comments
* add documentation for YES_FLAG and QUIET_FLAG
* Add tests to centos6 Dockerfile to make sure we install python3 if and only if appropriate to do so.
* Allow non-interactive revocation without deleting certificates (#5386)
* Add --delete-after-revoke flags
* Use delete_after_revoke value
* Add delete_after_revoke unit tests
* Add integration tests for delete-after-revoke.
* Have letsencrypt-auto do a real upgrade in leauto-upgrades option 2 (#5390)
* Make leauto_upgrades do a real upgrade
* Cleanup vars and output
* Sleep until the server is ready
* add simple_http_server.py
* Use a randomly assigned port
* s/realpath/readlink
* wait for server before getting port
* s/localhost/all interfaces
* update Apache ciphersuites (#5383)
* Fix macOS builds for Python2.7 in Travis (#5378)
* Add OSX Python2 tests
* Make sure python2 is originating from homebrew on macOS
* Upgrade the already installed python2 instead of trying to reinstall
Class inheritance based approach to distro specific overrides.
How it works:
The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available.
Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file.
Generic changes:
Parser initialization has been moved to separate class method, allowing easy override where needed.
Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py
Split Debian specific code from configurator.py to debian_override.py
Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands.
Moved add_parser_mod() from configurator to parser add_mod()
Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules().
Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out).
Moved OS based constants to their respective override classes.
Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class.
tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class.
This PR includes two major generic additions that should vastly improve our parsing accuracy and quality:
Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file.
Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such).
Distribution specific changes
Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are:
CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off.
Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values.
Debian
Moved the Debian specific parts from configurator.py to Debian specific override.
CentOS
Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump.
Added CentOS default Apache configuration tree for realistic test cases.
Gentoo
Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps.
Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases.
* Distribution specific override functionality based on class inheritance
* Need to patch get_systemd_os_like to as travis has proper os-release
* Added pydoc
* Move parser initialization to a method and fix Python 3 __new__ errors
* Parser changes to parse HTTPD config
* Try to get modules and includes from httpd process for better visibility over the configuration
* Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214)
* CentOS tests and linter fixes
* Gentoo override, tests and linter fixes
* Mock the process call in all the tests that require it
* Fix CentOS test mock
* Restore reseting modules list functionality for cleanup
* Move OS fingerprinting and constant mocks to parent class
* Fixes requested in review
* New entrypoint structure and started moving OS constants to override classes
* OS constants move continued, test and linter fixes
* Removed dead code
* Apache compatibility test changest to reflect OS constant restructure
* Test fix
* Requested changes
* Moved Debian specific tests to own test file
* Removed decorator based override class registration in favor of entrypoint dict
* Fix for update_includes for some versions of Augeas
* Take fedora fix into account in tests
* Review fixes
