* Reconfigure compatibility tests to use http challenge
* Correct simple test
* Add a fake DNS resolution for HTTP simple_verify
* Debug
* More subtle approach: we monkey patch urllib3 to fake a dns resolution to the target IP, allowing every host header to be preserved.
* Private package
* Relaxed permissions on certbot temp working dir
* Move the fake DNS logic in compatibility test, to avoid degrading the acme coverage
* Fix lint
* Update certbot-compatibility-test/certbot_compatibility_test/configurators/common.py
Co-Authored-By: adferrand <adferrand@users.noreply.github.com>
This will immediately address the breakage reported in #6682 and tracked at #6685. Virtualenv downloads the latest pip, which causes issues, so tell virtualenv to not download the latest pip.
I added the flag preemptively to other files as well, they're in separate commits so it will be easy to revert any spots we don't want.
I've confirmed that this fixes the issue on a machine that fails with the version of certbot-auto currently in master: recent version of virtualenv, python 2.7.
* Update changelog
* Use an environment variable instead of a flag for compatibility with old versions
* Run build.py
Certbot relies heavily on bash scripts to deploy a development environment and to execute tests. This is fine for Linux systems, including Travis, but problematic for Windows machines.
This PR converts all theses scripts into Python, to make them platform independant.
As a consequence, tox-win.ini is not needed anymore, and tox can be run indifferently on Windows or on Linux using a common tox.ini. AppVeyor is updated accordingly to execute tests for acme, certbot and all dns plugins. Other tests are not executed as they are for Docker, unsupported Apache/Nginx/Postfix plugins (for now) or not relevant for Windows (explicit Linux distribution tests or pylint).
Another PR will be done on certbot website to update how a dev environment can be set up.
* Replace several shell scripts by python equivalent.
* Correction on tox coverage
* Extend usage of new python scripts
* Various corrections
* Replace venv construction bash scripts by python equivalents
* Update tox.ini
* Unicode lines to compare files
* Put modifications on letsencrypt-auto-source instead of generated scripts
* Add executable permissions for Linux.
* Merge tox win tests into main tox
* Skip lock_test on Windows
* Correct appveyor config
* Update appveyor.yml
* Explicit coverage py27 or py37
* Avoid to cover non supported certbot plugins on Windows
* Update tox.ini
* Remove specific warnings during CI
* No cover on a debug code for tests only.
* Update documentation and help script on venv/venv3.py
* Customize help message for Windows
* Quote correctly executable path with potential spaces in it.
* Copy pipstrap from upstream
This PR adds two new command line parameters, --apache-ctlpath and --apache-binpath both of which are used to construct commands that we shell out for.
The way that we previously fetched values either from Certbot configuration object or the dictionary of distribution based constants is now also unified, and the active options are parsed in prepare() to make it easier to override needed values for the distributions needing this behavior.
Fixes: #5338
* Added the command line options and parsing
* Refactor existing code
* Distro override updates
* Handle vhost_root from cli
* Fix compatibility tests
* Add comment about changes to command line arguments
* Check None properly
* Made help texts consistent
* Keep the old defaults
* Move to shorter CLI parameter names
* No need for specific bin path, nor apache_cmd anymore
* Make sure that we use user provided vhost-root value
* Fix alt restart commands in overrides
* Fix version_cmd defaults in overrides
* Fix comparison
* Remove cruft, and use configuration object for parser parameter
* Remove unsupported pylint disable options
* star-args removed in Pylint 1.4.3
* abstract-class-little-used removed in Pylint 1.4.3
* Fixes new lint errors
* Copy dummy-variable-rgx expression to new ignored-argument-names expression to ignore unused funtion arguments
* Notable changes
* Refactor to satisfy Pylint no-else-return warning
* Fix Pylint inconsistent-return-statements warning
* Refactor to satisfy consider-iterating-dictionary
* Remove methods with only super call to satisfy useless-super-delegation
* Refactor too-many-nested-statements where possible
* Suppress type checked errors where member is dynamically added (notably derived from josepy.JSONObjectWithFields)
* Remove None default of func parameter for ExitHandler and ErrorHandler
Resolves#5973
* Drop support for EOL Python 2.6
* Use more helpful assertIn/NotIn instead of assertTrue/False
* Drop support for EOL Python 3.3
* Remove redundant Python 3.3 code
* Restore code for RHEL 6 and virtualenv for Py2.7
* Revert pipstrap.py to upstream
* Merge py26_packages and non_py26_packages into all_packages
* Revert changes to *-auto in root
* Update by calling letsencrypt-auto-source/build.py
* Revert permissions for pipstrap.py
Class inheritance based approach to distro specific overrides.
How it works:
The certbot-apache plugin entrypoint has been changed to entrypoint.ENTRYPOINT which is a variable containing appropriate override class for system, if available.
Override classes register themselves using decorator override.register() which takes a list of distribution fingerprints (ID & LIKE variables in /etc/os-release, or platform.linux_distribution() as a fallback). These end up as keys in dict override.OVERRIDE_CLASSES and values for the keys are references to the class that called the decorator, hence allowing self-registration of override classes when they are imported. The only file importing these override classes is entrypoint.py, so adding new override classes would need only one import in addition to the actual override class file.
Generic changes:
Parser initialization has been moved to separate class method, allowing easy override where needed.
Cleaned up configurator.py a bit, and moved some helper functions to newly created apache_util.py
Split Debian specific code from configurator.py to debian_override.py
Changed define_cmd to apache_cmd because the parameters are for every distribution supporting this behavior, and we're able to use the value to build the additional configuration dump commands.
Moved add_parser_mod() from configurator to parser add_mod()
Added two new configuration dump parsing methods to update_runtime_variables() in parser: update_includes() and update_modules().
Changed init_modules() in parser to accommodate the changes above. (ie. don't throw existing self.modules out).
Moved OS based constants to their respective override classes.
Refactored configurator class discovery in tests to help easier test case creation using distribution based override configurator class.
tests.util.get_apache_configurator() now takes keyword argument os_info which is string of the desired mock OS fingerprint response that's used for picking the right override class.
This PR includes two major generic additions that should vastly improve our parsing accuracy and quality:
Includes are parsed from config dump from httpd binary. This is mandatory for some distributions (Like OpenSUSE) to get visibility over the whole configuration tree because of Include statements passed on in command line, and not via root httpd.conf file.
Modules are parsed from config dump from httpd binary. This lets us jump into correct IfModule directives if for some reason we have missed the module availability (because of one being included on command line or such).
Distribution specific changes
Because of the generic changes, there are two distributions (or distribution families) that do not provide such functionality, so it had to be overridden in their respective override files. These distributions are:
CentOS, because it deliberately limits httpd binary stdout using SELinux as a feature. We are doing opportunistic config dumps here however, in case SELinux enforcing is off.
Gentoo, because it does not provide a way to invoke httpd with command line parsed from its specific configuration file. Gentoo relies heavily on Define statements that are passed over from APACHE2_OPTS variable /etc/conf.d/apache2 file and most of the configuration in root Apache configuration are dependent on these values.
Debian
Moved the Debian specific parts from configurator.py to Debian specific override.
CentOS
Parsing of /etc/sysconfig/httpd file for additional Define statements. This could hold other parameters too, but parsing everything off it would require a full Apache lexer. For CLI parameters, I think Defines are the most common ones. This is done in addition of opportunistic parsing of httpd binary config dump.
Added CentOS default Apache configuration tree for realistic test cases.
Gentoo
Parsing Defines from /etc/conf.d/apache2 variable APACHE2_OPTS, which holds additional Define statements to enable certain functionalities, enabling parts of the configuration in the Apache2 DOM. This is done instead of trying to parse httpd binary configuration dumps.
Added default Apache configuration from Gentoo to testdata, including /etc/conf.d/apache2 file for realistic test cases.
* Distribution specific override functionality based on class inheritance
* Need to patch get_systemd_os_like to as travis has proper os-release
* Added pydoc
* Move parser initialization to a method and fix Python 3 __new__ errors
* Parser changes to parse HTTPD config
* Try to get modules and includes from httpd process for better visibility over the configuration
* Had to disable duplicate-code because of test setup (PyCQA/pylint/issues/214)
* CentOS tests and linter fixes
* Gentoo override, tests and linter fixes
* Mock the process call in all the tests that require it
* Fix CentOS test mock
* Restore reseting modules list functionality for cleanup
* Move OS fingerprinting and constant mocks to parent class
* Fixes requested in review
* New entrypoint structure and started moving OS constants to override classes
* OS constants move continued, test and linter fixes
* Removed dead code
* Apache compatibility test changest to reflect OS constant restructure
* Test fix
* Requested changes
* Moved Debian specific tests to own test file
* Removed decorator based override class registration in favor of entrypoint dict
* Fix for update_includes for some versions of Augeas
* Take fedora fix into account in tests
* Review fixes
