certbot/certbot-ci/certbot_integration_tests/utils/pebble_artifacts.py

69 lines
2.6 KiB
Python
Raw Permalink Normal View History

# pylint: disable=missing-module-docstring
import atexit
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157) This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform. Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore. As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows. Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables. Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability. Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble. * Complete process * Fix nginx cert path * Check conditionnally docker * Update gitignore, fix apacheconftest * Full object * Carriage return * Move to official v2.1.0 of pebble * Fix name * Update acme_server.py * Relaunch CI * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update docstring * Update documentation * Configure a stdout to ACMEServer * Map all process through defined stdout * Remove unused variable * Handle using signals * Use failsafe entering context * Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
import json
import os
import stat
import sys
from contextlib import ExitStack
from typing import Tuple
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157) This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform. Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore. As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows. Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables. Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability. Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble. * Complete process * Fix nginx cert path * Check conditionnally docker * Update gitignore, fix apacheconftest * Full object * Carriage return * Move to official v2.1.0 of pebble * Fix name * Update acme_server.py * Relaunch CI * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update docstring * Update documentation * Configure a stdout to ACMEServer * Map all process through defined stdout * Remove unused variable * Handle using signals * Use failsafe entering context * Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
import requests
from certbot_integration_tests.utils.constants import DEFAULT_HTTP_01_PORT
from certbot_integration_tests.utils.constants import MOCK_OCSP_SERVER_PORT
if sys.version_info >= (3, 9): # pragma: no cover
import importlib.resources as importlib_resources
else: # pragma: no cover
import importlib_resources
PEBBLE_VERSION = 'v2.3.1'
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157) This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform. Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore. As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows. Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables. Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability. Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble. * Complete process * Fix nginx cert path * Check conditionnally docker * Update gitignore, fix apacheconftest * Full object * Carriage return * Move to official v2.1.0 of pebble * Fix name * Update acme_server.py * Relaunch CI * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update docstring * Update documentation * Configure a stdout to ACMEServer * Map all process through defined stdout * Remove unused variable * Handle using signals * Use failsafe entering context * Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
def fetch(workspace: str, http_01_port: int = DEFAULT_HTTP_01_PORT) -> Tuple[str, str, str]:
# pylint: disable=missing-function-docstring
Fix integration tests on Windows (#7271) * Fix account_tests * Fix hook executable test * Remove the temporary decorator @broken_on_windows * Fix util_test * No broken unit test on Windows anymore * More elegant mock * Fix context manager * Fix lint * Fix mypy * Adapt coverage * Corrections * Fix lint * Adapt coverage * Update certbot/tests/compat/filesystem_test.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update util_test.py * Fix pylint * Forbid os.access * Update os_test.py * Update os.py * Fix lint * Update filesystem.py * Update filesystem.py * Update filesystem.py * Update os.py * Start fixing tests * Platform independent hooks * Fix probe fd close * Add broken_on_windows for integration tests * Fix a lot of tests * Use a python hook script, to prepare cross-platform * New approach to be compliant with Linux and Windows on hook scripts * New tests fixed * Test for permissions on Windows * Permissions comparison for Windows * No broken tests in certbot core anymore * Change mode * Specific config for appveyor * Use forked pebble for now * Various fixes * Assert file permissions for world on private keys * Clean code * Fix several things * Add integration target * Optimize integration env * Re-enable all AppVeyor envs * Use again official pebble * Update pebble_artifacts.py * Set PYTEST_ADDOPTS silently * Update appveyor.yml * Pin pywin32 for tests, give a minimal requirement for certbot. * Remove injection of nginx in PATH * Clean debug code * Various cleanup, ensure to remove workspace after tests * Update tox target * Improve assertions. Control the keyword echoed in hooks * Fix for virtualenv on Python 3.7.4 for Windows * Update certbot-ci/certbot_integration_tests/certbot_tests/assertions.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Add conditionally pywin in certbot-ci like in certbot
2019-08-06 18:02:16 -04:00
suffix = 'linux-amd64' if os.name != 'nt' else 'windows-amd64.exe'
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157) This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform. Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore. As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows. Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables. Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability. Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble. * Complete process * Fix nginx cert path * Check conditionnally docker * Update gitignore, fix apacheconftest * Full object * Carriage return * Move to official v2.1.0 of pebble * Fix name * Update acme_server.py * Relaunch CI * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update docstring * Update documentation * Configure a stdout to ACMEServer * Map all process through defined stdout * Remove unused variable * Handle using signals * Use failsafe entering context * Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
file_manager = ExitStack()
atexit.register(file_manager.close)
pebble_path_ref = importlib_resources.files('certbot_integration_tests') / 'assets'
assets_path = str(file_manager.enter_context(importlib_resources.as_file(pebble_path_ref)))
pebble_path = _fetch_asset('pebble', suffix, assets_path)
challtestsrv_path = _fetch_asset('pebble-challtestsrv', suffix, assets_path)
pebble_config_path = _build_pebble_config(workspace, http_01_port, assets_path)
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157) This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform. Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore. As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows. Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables. Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability. Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble. * Complete process * Fix nginx cert path * Check conditionnally docker * Update gitignore, fix apacheconftest * Full object * Carriage return * Move to official v2.1.0 of pebble * Fix name * Update acme_server.py * Relaunch CI * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update docstring * Update documentation * Configure a stdout to ACMEServer * Map all process through defined stdout * Remove unused variable * Handle using signals * Use failsafe entering context * Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
return pebble_path, challtestsrv_path, pebble_config_path
def _fetch_asset(asset: str, suffix: str, assets_path: str) -> str:
asset_path = os.path.join(assets_path, '{0}_{1}_{2}'.format(asset, PEBBLE_VERSION, suffix))
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157) This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform. Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore. As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows. Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables. Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability. Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble. * Complete process * Fix nginx cert path * Check conditionnally docker * Update gitignore, fix apacheconftest * Full object * Carriage return * Move to official v2.1.0 of pebble * Fix name * Update acme_server.py * Relaunch CI * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update docstring * Update documentation * Configure a stdout to ACMEServer * Map all process through defined stdout * Remove unused variable * Handle using signals * Use failsafe entering context * Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
if not os.path.exists(asset_path):
asset_url = ('https://github.com/letsencrypt/pebble/releases/download/{0}/{1}_{2}'
.format(PEBBLE_VERSION, asset, suffix))
response = requests.get(asset_url, timeout=30)
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157) This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform. Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore. As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows. Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables. Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability. Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble. * Complete process * Fix nginx cert path * Check conditionnally docker * Update gitignore, fix apacheconftest * Full object * Carriage return * Move to official v2.1.0 of pebble * Fix name * Update acme_server.py * Relaunch CI * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update docstring * Update documentation * Configure a stdout to ACMEServer * Map all process through defined stdout * Remove unused variable * Handle using signals * Use failsafe entering context * Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
response.raise_for_status()
with open(asset_path, 'wb') as file_h:
file_h.write(response.content)
os.chmod(asset_path, os.stat(asset_path).st_mode | stat.S_IEXEC)
return asset_path
def _build_pebble_config(workspace: str, http_01_port: int, assets_path: str) -> str:
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157) This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform. Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore. As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows. Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables. Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability. Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble. * Complete process * Fix nginx cert path * Check conditionnally docker * Update gitignore, fix apacheconftest * Full object * Carriage return * Move to official v2.1.0 of pebble * Fix name * Update acme_server.py * Relaunch CI * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update docstring * Update documentation * Configure a stdout to ACMEServer * Map all process through defined stdout * Remove unused variable * Handle using signals * Use failsafe entering context * Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
config_path = os.path.join(workspace, 'pebble-config.json')
with open(config_path, 'w') as file_h:
file_h.write(json.dumps({
'pebble': {
'listenAddress': '0.0.0.0:14000',
'managementListenAddress': '0.0.0.0:15000',
'certificate': os.path.join(assets_path, 'cert.pem'),
'privateKey': os.path.join(assets_path, 'key.pem'),
'httpPort': http_01_port,
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157) This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform. Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore. As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows. Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables. Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability. Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble. * Complete process * Fix nginx cert path * Check conditionnally docker * Update gitignore, fix apacheconftest * Full object * Carriage return * Move to official v2.1.0 of pebble * Fix name * Update acme_server.py * Relaunch CI * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update docstring * Update documentation * Configure a stdout to ACMEServer * Map all process through defined stdout * Remove unused variable * Handle using signals * Use failsafe entering context * Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
'tlsPort': 5001,
'ocspResponderURL': 'http://127.0.0.1:{0}'.format(MOCK_OCSP_SERVER_PORT),
[Windows|Linux] Launch integration tests on Pebble without Docker (#7157) This PR is a part of the actions necessary to make Certbot-CI work on Windows, in order to execute the integration tests on this platform. Following #7156, this PR changes how the integration tests are setup against Pebble to not need Docker anymore. As a reminder, one can check #7156 and letsencrypt/pebble#240 to see the rationale about why using Docker is a problem to run the integration tests on Windows. Basically, this PR executes directly Pebble using its executable, since it is build using Go, and Go produces self-contained executable that can run without any installation on Linux and on Windows. During the integration tests setup, Certbot-CI will get the Pebble (and Challtestsrv) executables for the defined target version on the GitHub releases. The binaries are persisted on the filesystem, so it is not needed to download them again on the second integration tests execution. Nonetheless, we are talking about 20MB of executables. Since the setup needs to hold a state, I also took this occasion to refactor the acme_server, in order to use on object oriented approach and improve the readability/maintainability. Once this PR and #7156 are merged, Docker will not be needed anymore for the main integration tests usecase, that is to use Pebble. * Complete process * Fix nginx cert path * Check conditionnally docker * Update gitignore, fix apacheconftest * Full object * Carriage return * Move to official v2.1.0 of pebble * Fix name * Update acme_server.py * Relaunch CI * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update certbot-ci/certbot_integration_tests/utils/acme_server.py Co-Authored-By: Brad Warren <bmw@users.noreply.github.com> * Update docstring * Update documentation * Configure a stdout to ACMEServer * Map all process through defined stdout * Remove unused variable * Handle using signals * Use failsafe entering context * Remove failsafe rmtree, that is not needed anymore
2019-07-10 17:29:57 -04:00
},
}))
return config_path