[Unix] Create a framework for certbot integration tests: PART 4 (#6958)
This PR is the part 4 to implement #6541. It adds the integration tests for the nginx certbot plugin, and corresponds to the certbot-ci translation of certbot-nginx/tests/boulder-integration.sh that is executed for each PR.
As with certbot core tests, tests are written in Python, and executed by pytest, against a dynamic Boulder/Pebble instance setup. Tests are parallelized, of course, and a specific IntegrationTestsContext class, extended the one from certbot core tests, is crafter for these specific tests: its main goal is to setup a specific nginx instance for the current test.
On top of that, I use the test parametrization feature of Pytest, to drastically reduce the size of the actual code: indeed, the 6 tests from the original bash script share the same logic. So using a parametrization, one unique test is written, that is then executed 6 times against 6 different sets of parameters.
Note that the module integration_tests.nginx_tests.nginx_config do the same, but in Python, than certbot-nginx/tests/boulder-integration.conf.sh. The latter will be removed in a future PR, with all other bash scripts.
* Add nginx tests
* Distribute the other_port
* Load a pre-generated key/cert for nginx config
* Correct preload, remove a test, simplify a variable
* Integrate assertion directly in the test function
* Check process is not terminated
* Add spaces in the nginx config
* Add comments
* Use indirection
* Allow external cert
* Add coverage threshold for certbot-nginx
2019-04-23 16:29:48 -04:00
|
|
|
"""General purpose nginx test configuration generator."""
|
|
|
|
|
import getpass
|
|
|
|
|
|
|
|
|
|
import pkg_resources
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def construct_nginx_config(nginx_root, nginx_webroot, http_port, https_port, other_port,
|
|
|
|
|
default_server, key_path=None, cert_path=None, wtf_prefix='le'):
|
|
|
|
|
"""
|
|
|
|
|
This method returns a full nginx configuration suitable for integration tests.
|
|
|
|
|
:param str nginx_root: nginx root configuration path
|
|
|
|
|
:param str nginx_webroot: nginx webroot path
|
|
|
|
|
:param int http_port: HTTP port to listen on
|
|
|
|
|
:param int https_port: HTTPS port to listen on
|
|
|
|
|
:param int other_port: other HTTP port to listen on
|
|
|
|
|
:param bool default_server: True to set a default server in nginx config, False otherwise
|
|
|
|
|
:param str key_path: the path to a SSL key
|
|
|
|
|
:param str cert_path: the path to a SSL certificate
|
|
|
|
|
:param str wtf_prefix: the prefix to use in all domains handled by this nginx config
|
|
|
|
|
:return: a string containing the full nginx configuration
|
|
|
|
|
:rtype: str
|
|
|
|
|
"""
|
|
|
|
|
key_path = key_path if key_path \
|
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157)
This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform.
Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore.
As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows.
Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables.
Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability.
Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble.
* Complete process
* Fix nginx cert path
* Check conditionnally docker
* Update gitignore, fix apacheconftest
* Full object
* Carriage return
* Move to official v2.1.0 of pebble
* Fix name
* Update acme_server.py
* Relaunch CI
* Update certbot-ci/certbot_integration_tests/utils/acme_server.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update certbot-ci/certbot_integration_tests/utils/acme_server.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update docstring
* Update documentation
* Configure a stdout to ACMEServer
* Map all process through defined stdout
* Remove unused variable
* Handle using signals
* Use failsafe entering context
* Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
|
|
|
else pkg_resources.resource_filename('certbot_integration_tests', 'assets/key.pem')
|
[Unix] Create a framework for certbot integration tests: PART 4 (#6958)
This PR is the part 4 to implement #6541. It adds the integration tests for the nginx certbot plugin, and corresponds to the certbot-ci translation of certbot-nginx/tests/boulder-integration.sh that is executed for each PR.
As with certbot core tests, tests are written in Python, and executed by pytest, against a dynamic Boulder/Pebble instance setup. Tests are parallelized, of course, and a specific IntegrationTestsContext class, extended the one from certbot core tests, is crafter for these specific tests: its main goal is to setup a specific nginx instance for the current test.
On top of that, I use the test parametrization feature of Pytest, to drastically reduce the size of the actual code: indeed, the 6 tests from the original bash script share the same logic. So using a parametrization, one unique test is written, that is then executed 6 times against 6 different sets of parameters.
Note that the module integration_tests.nginx_tests.nginx_config do the same, but in Python, than certbot-nginx/tests/boulder-integration.conf.sh. The latter will be removed in a future PR, with all other bash scripts.
* Add nginx tests
* Distribute the other_port
* Load a pre-generated key/cert for nginx config
* Correct preload, remove a test, simplify a variable
* Integrate assertion directly in the test function
* Check process is not terminated
* Add spaces in the nginx config
* Add comments
* Use indirection
* Allow external cert
* Add coverage threshold for certbot-nginx
2019-04-23 16:29:48 -04:00
|
|
|
cert_path = cert_path if cert_path \
|
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157)
This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform.
Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore.
As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows.
Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables.
Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability.
Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble.
* Complete process
* Fix nginx cert path
* Check conditionnally docker
* Update gitignore, fix apacheconftest
* Full object
* Carriage return
* Move to official v2.1.0 of pebble
* Fix name
* Update acme_server.py
* Relaunch CI
* Update certbot-ci/certbot_integration_tests/utils/acme_server.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update certbot-ci/certbot_integration_tests/utils/acme_server.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update docstring
* Update documentation
* Configure a stdout to ACMEServer
* Map all process through defined stdout
* Remove unused variable
* Handle using signals
* Use failsafe entering context
* Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
|
|
|
else pkg_resources.resource_filename('certbot_integration_tests', 'assets/cert.pem')
|
[Unix] Create a framework for certbot integration tests: PART 4 (#6958)
This PR is the part 4 to implement #6541. It adds the integration tests for the nginx certbot plugin, and corresponds to the certbot-ci translation of certbot-nginx/tests/boulder-integration.sh that is executed for each PR.
As with certbot core tests, tests are written in Python, and executed by pytest, against a dynamic Boulder/Pebble instance setup. Tests are parallelized, of course, and a specific IntegrationTestsContext class, extended the one from certbot core tests, is crafter for these specific tests: its main goal is to setup a specific nginx instance for the current test.
On top of that, I use the test parametrization feature of Pytest, to drastically reduce the size of the actual code: indeed, the 6 tests from the original bash script share the same logic. So using a parametrization, one unique test is written, that is then executed 6 times against 6 different sets of parameters.
Note that the module integration_tests.nginx_tests.nginx_config do the same, but in Python, than certbot-nginx/tests/boulder-integration.conf.sh. The latter will be removed in a future PR, with all other bash scripts.
* Add nginx tests
* Distribute the other_port
* Load a pre-generated key/cert for nginx config
* Correct preload, remove a test, simplify a variable
* Integrate assertion directly in the test function
* Check process is not terminated
* Add spaces in the nginx config
* Add comments
* Use indirection
* Allow external cert
* Add coverage threshold for certbot-nginx
2019-04-23 16:29:48 -04:00
|
|
|
return '''\
|
|
|
|
|
# This error log will be written regardless of server scope error_log
|
|
|
|
|
# definitions, so we have to set this here in the main scope.
|
|
|
|
|
#
|
|
|
|
|
# Even doing this, Nginx will still try to create the default error file, and
|
|
|
|
|
# log a non-fatal error when it fails. After that things will work, however.
|
|
|
|
|
error_log {nginx_root}/error.log;
|
|
|
|
|
|
|
|
|
|
# The pidfile will be written to /var/run unless this is set.
|
|
|
|
|
pid {nginx_root}/nginx.pid;
|
|
|
|
|
|
|
|
|
|
user {user};
|
|
|
|
|
worker_processes 1;
|
|
|
|
|
|
|
|
|
|
events {{
|
|
|
|
|
worker_connections 1024;
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
http {{
|
|
|
|
|
# Set an array of temp, cache and log file options that will otherwise default to
|
|
|
|
|
# restricted locations accessible only to root.
|
|
|
|
|
client_body_temp_path {nginx_root}/client_body;
|
|
|
|
|
fastcgi_temp_path {nginx_root}/fastcgi_temp;
|
|
|
|
|
proxy_temp_path {nginx_root}/proxy_temp;
|
|
|
|
|
#scgi_temp_path {nginx_root}/scgi_temp;
|
|
|
|
|
#uwsgi_temp_path {nginx_root}/uwsgi_temp;
|
|
|
|
|
access_log {nginx_root}/error.log;
|
|
|
|
|
|
|
|
|
|
# This should be turned off in a Virtualbox VM, as it can cause some
|
|
|
|
|
# interesting issues with data corruption in delivered files.
|
|
|
|
|
sendfile off;
|
|
|
|
|
|
|
|
|
|
tcp_nopush on;
|
|
|
|
|
tcp_nodelay on;
|
|
|
|
|
keepalive_timeout 65;
|
|
|
|
|
types_hash_max_size 2048;
|
|
|
|
|
|
|
|
|
|
#include /etc/nginx/mime.types;
|
|
|
|
|
index index.html index.htm index.php;
|
|
|
|
|
|
|
|
|
|
log_format main '$remote_addr - $remote_user [$time_local] $status '
|
|
|
|
|
'"$request" $body_bytes_sent "$http_referer" '
|
|
|
|
|
'"$http_user_agent" "$http_x_forwarded_for"';
|
|
|
|
|
|
|
|
|
|
default_type application/octet-stream;
|
|
|
|
|
|
|
|
|
|
server {{
|
|
|
|
|
# IPv4.
|
|
|
|
|
listen {http_port} {default_server};
|
|
|
|
|
# IPv6.
|
|
|
|
|
listen [::]:{http_port} {default_server};
|
|
|
|
|
server_name nginx.{wtf_prefix}.wtf nginx2.{wtf_prefix}.wtf;
|
|
|
|
|
|
|
|
|
|
root {nginx_webroot};
|
|
|
|
|
|
|
|
|
|
location / {{
|
|
|
|
|
# First attempt to serve request as file, then as directory, then fall
|
|
|
|
|
# back to index.html.
|
|
|
|
|
try_files $uri $uri/ /index.html;
|
|
|
|
|
}}
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
server {{
|
|
|
|
|
listen {http_port};
|
|
|
|
|
listen [::]:{http_port};
|
|
|
|
|
server_name nginx3.{wtf_prefix}.wtf;
|
|
|
|
|
|
|
|
|
|
root {nginx_webroot};
|
|
|
|
|
|
|
|
|
|
location /.well-known/ {{
|
|
|
|
|
return 404;
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
return 301 https://$host$request_uri;
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
server {{
|
|
|
|
|
listen {other_port};
|
|
|
|
|
listen [::]:{other_port};
|
|
|
|
|
server_name nginx4.{wtf_prefix}.wtf nginx5.{wtf_prefix}.wtf;
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
server {{
|
|
|
|
|
listen {http_port};
|
|
|
|
|
listen [::]:{http_port};
|
|
|
|
|
listen {https_port} ssl;
|
|
|
|
|
listen [::]:{https_port} ssl;
|
|
|
|
|
if ($scheme != "https") {{
|
|
|
|
|
return 301 https://$host$request_uri;
|
|
|
|
|
}}
|
|
|
|
|
server_name nginx6.{wtf_prefix}.wtf nginx7.{wtf_prefix}.wtf;
|
|
|
|
|
|
|
|
|
|
ssl_certificate {cert_path};
|
|
|
|
|
ssl_certificate_key {key_path};
|
|
|
|
|
}}
|
|
|
|
|
}}
|
|
|
|
|
'''.format(nginx_root=nginx_root, nginx_webroot=nginx_webroot, user=getpass.getuser(),
|
|
|
|
|
http_port=http_port, https_port=https_port, other_port=other_port,
|
|
|
|
|
default_server='default_server' if default_server else '', wtf_prefix=wtf_prefix,
|
|
|
|
|
key_path=key_path, cert_path=cert_path)
|