diff --git a/.azure-pipelines/INSTALL.md b/.azure-pipelines/INSTALL.md new file mode 100644 index 000000000..1a50bcb0c --- /dev/null +++ b/.azure-pipelines/INSTALL.md @@ -0,0 +1,119 @@ +# Configuring Azure Pipelines with Certbot + +Let's begin. All pipelines are defined in `.azure-pipelines`. Currently there are two: +* `.azure-pipelines/main.yml` is the main one, executed on PRs for master, and pushes to master, +* `.azure-pipelines/advanced.yml` add installer testing on top of the main pipeline, and is executed for `test-*` branches, release branches, and nightly run for master. + +Several templates are defined in `.azure-pipelines/templates`. These YAML files aggregate common jobs configuration that can be reused in several pipelines. + +Unlike Travis, where CodeCov is working without any action required, CodeCov supports Azure Pipelines +using the coverage-bash utility (not python-coverage for now) only if you provide the Codecov repo token +using the `CODECOV_TOKEN` environment variable. So `CODECOV_TOKEN` needs to be set as a secured +environment variable to allow the main pipeline to publish coverage reports to CodeCov. + +This INSTALL.md file explains how to configure Azure Pipelines with Certbot in order to execute the CI/CD logic defined in `.azure-pipelines` folder with it. +During this installation step, warnings describing user access and legal comitments will be displayed like this: +``` +!!! ACCESS REQUIRED !!! +``` + +This document suppose that the Azure DevOps organization is named _certbot_, and the Azure DevOps project is also _certbot_. + +## Useful links + +* https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema +* https://www.azuredevopslabs.com/labs/azuredevops/github-integration/ +* https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/python?view=azure-devops + +## Prerequisites + +### Having a GitHub account + +Use your GitHub user for a normal GitHub account, or a user that has administrative rights to the GitHub organization if relevant. + +### Having an Azure DevOps account +- Go to https://dev.azure.com/, click "Start free with GitHub" +- Login to GitHub + +``` +!!! ACCESS REQUIRED !!! +Personal user data (email + profile info, in read-only) +``` + +- Microsoft will create a Live account using the email referenced for the GitHub account. This account is also linked to GitHub account (meaning you can log it using GitHub authentication) +- Proceed with account registration (birth date, country), add details about name and email contact + +``` +!!! ACCESS REQUIRED !!! +Microsoft proposes to send commercial links to this mail +Azure DevOps terms of service need to be accepted +``` + +_Logged to Azure DevOps, account is ready._ + +### Installing Azure Pipelines to GitHub + +- On GitHub, go to Marketplace +- Select Azure Pipeline, and "Set up a plan" +- Select Free, then "Install it for free" +- Click "Complete order and begin installation" + +``` +!!! ACCESS !!! +Azure Pipeline needs RW on code, RO on metadata, RW on checks, commit statuses, deployments, issues, pull requests. +RW access here is required to allow update of the pipelines YAML files from Azure DevOps interface, and to +update the status of builds and PRs on GitHub side when Azure Pipelines are triggered. +Note however that no admin access is defined here: this means that Azure Pipelines cannot do anything with +protected branches, like master, and cannot modify the security context around this on GitHub. +Access can be defined for all or only selected repositories, which is nice. +``` + +- Redirected to Azure DevOps, select the account created in _Having an Azure DevOps account_ section. +- Select the organization, and click "Create a new project" (let's name it the same than the targeted github repo) +- The Visibility is public, to profit from 10 parallel jobs + +``` +!!! ACCESS !!! +Azure Pipelines needs access to the GitHub account (in term of being able to check it is valid), and the Resources shared between the GitHub account and Azure Pipelines. +``` + +_Done. We can move to pipelines configuration._ + +## Import an existing pipelines from `.azure-pipelines` folder + +- On Azure DevOps, go to your organization (eg. _certbot_) then your project (eg. _certbot_) +- Click "Pipelines" tab +- Click "New pipeline" +- Where is your code?: select "__Use the classic editor__" + +__Warning: Do not choose the GitHub option in Where is your code? section. Indeed, this option will trigger an OAuth +grant permissions from Azure Pipelines to GitHub in order to setup a GitHub OAuth Application. The permissions asked +then are way too large (admin level on almost everything), while the classic approach does not add any more +permissions, and works perfectly well.__ + +- Select GitHub in "Select your repository section", choose certbot/certbot in Repository, master in default branch. +- Click on YAML option for "Select a template" +- Choose a name for the pipeline (eg. test-pipeline), and browse to the actual pipeline YAML definition in the + "YAML file path" input (eg. `.azure-pipelines/test-pipeline.yml`) +- Click "Save & queue", choose the master branch to build the first pipeline, and click "Save and run" button. + +_Done. Pipeline is operational. Repeat to add more pipelines from existing YAML files in `.azure-pipelines`._ + +## Add a secret variable to a pipeline (like `CODECOV_TOKEN`) + +__NB: Following steps suppose that you already setup the YAML pipeline file to +consume the secret variable that these steps will create as an environment variable. +For a variable named `CODECOV_TOKEN` consuming the variable `codecov_token`, +in the YAML file this setup would take the form of the following: +``` +steps: + - script: ./do_something_that_consumes_CODECOV_TOKEN # Eg. `codecov -F windows` + env: + CODECOV_TOKEN: $(codecov_token) +``` + +To set up a variable that is shared between pipelines, follow the instructions +at +https://docs.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups. +When adding variables to a group, don't forget to tick "Keep this value secret" +if it shouldn't be shared publcily. diff --git a/.azure-pipelines/advanced.yml b/.azure-pipelines/advanced.yml new file mode 100644 index 000000000..dda7f9bfd --- /dev/null +++ b/.azure-pipelines/advanced.yml @@ -0,0 +1,23 @@ +# Advanced pipeline for isolated checks and release purpose +trigger: + # When changing these triggers, please ensure the documentation under + # "Running tests in CI" is still correct. + - azure-test-* + - test-* + - '*.x' +pr: + - test-* +# This pipeline is also nightly run on master +schedules: + - cron: "0 4 * * *" + displayName: Nightly build + branches: + include: + - master + always: true + +jobs: + # Any addition here should be reflected in the release pipeline. + # It is advised to declare all jobs here as templates to improve maintainability. + - template: templates/tests-suite.yml + - template: templates/installer-tests.yml diff --git a/.azure-pipelines/main.yml b/.azure-pipelines/main.yml new file mode 100644 index 000000000..d9609037e --- /dev/null +++ b/.azure-pipelines/main.yml @@ -0,0 +1,12 @@ +trigger: + # apache-parser-v2 is a temporary branch for doing work related to + # rewriting the parser in the Apache plugin. + - apache-parser-v2 + - master +pr: + - apache-parser-v2 + - master + - '*.x' + +jobs: + - template: templates/tests-suite.yml diff --git a/.azure-pipelines/release.yml b/.azure-pipelines/release.yml new file mode 100644 index 000000000..aeb5ee327 --- /dev/null +++ b/.azure-pipelines/release.yml @@ -0,0 +1,13 @@ +# Release pipeline to build and deploy Certbot for Windows for GitHub release tags +trigger: + tags: + include: + - v* +pr: none + +jobs: + # Any addition here should be reflected in the advanced pipeline. + # It is advised to declare all jobs here as templates to improve maintainability. + - template: templates/tests-suite.yml + - template: templates/installer-tests.yml + - template: templates/changelog.yml diff --git a/.azure-pipelines/templates/changelog.yml b/.azure-pipelines/templates/changelog.yml new file mode 100644 index 000000000..4a65e2c2b --- /dev/null +++ b/.azure-pipelines/templates/changelog.yml @@ -0,0 +1,14 @@ +jobs: + - job: changelog + pool: + vmImage: vs2017-win2016 + steps: + - bash: | + CERTBOT_VERSION="$(cd certbot && python -c "import certbot; print(certbot.__version__)" && cd ~-)" + "${BUILD_REPOSITORY_LOCALPATH}\tools\extract_changelog.py" "${CERTBOT_VERSION}" >> "${BUILD_ARTIFACTSTAGINGDIRECTORY}/release_notes.md" + displayName: Prepare changelog + - task: PublishPipelineArtifact@1 + inputs: + path: $(Build.ArtifactStagingDirectory) + artifact: changelog + displayName: Publish changelog diff --git a/.azure-pipelines/templates/installer-tests.yml b/.azure-pipelines/templates/installer-tests.yml new file mode 100644 index 000000000..6d5672339 --- /dev/null +++ b/.azure-pipelines/templates/installer-tests.yml @@ -0,0 +1,56 @@ +jobs: + - job: installer_build + pool: + vmImage: vs2017-win2016 + steps: + - task: UsePythonVersion@0 + inputs: + versionSpec: 3.7 + architecture: x86 + addToPath: true + - script: python windows-installer/construct.py + displayName: Build Certbot installer + - task: CopyFiles@2 + inputs: + sourceFolder: $(System.DefaultWorkingDirectory)/windows-installer/build/nsis + contents: '*.exe' + targetFolder: $(Build.ArtifactStagingDirectory) + - task: PublishPipelineArtifact@1 + inputs: + path: $(Build.ArtifactStagingDirectory) + artifact: windows-installer + displayName: Publish Windows installer + - job: installer_run + dependsOn: installer_build + strategy: + matrix: + win2019: + imageName: windows-2019 + win2016: + imageName: vs2017-win2016 + win2012r2: + imageName: vs2015-win2012r2 + pool: + vmImage: $(imageName) + steps: + - powershell: Invoke-WebRequest https://www.python.org/ftp/python/3.8.1/python-3.8.1-amd64-webinstall.exe -OutFile C:\py3-setup.exe + displayName: Get Python + - script: C:\py3-setup.exe /quiet PrependPath=1 InstallAllUsers=1 Include_launcher=1 InstallLauncherAllUsers=1 Include_test=0 Include_doc=0 Include_dev=1 Include_debug=0 Include_tcltk=0 TargetDir=C:\py3 + displayName: Install Python + - task: DownloadPipelineArtifact@2 + inputs: + artifact: windows-installer + path: $(Build.SourcesDirectory)/bin + displayName: Retrieve Windows installer + - script: | + py -3 -m venv venv + venv\Scripts\python tools\pip_install.py -e certbot-ci + displayName: Prepare Certbot-CI + - script: | + set PATH=%ProgramFiles(x86)%\Certbot\bin;%PATH% + venv\Scripts\python -m pytest certbot-ci\windows_installer_integration_tests --allow-persistent-changes --installer-path $(Build.SourcesDirectory)\bin\certbot-beta-installer-win32.exe + displayName: Run windows installer integration tests + - script: | + set PATH=%ProgramFiles(x86)%\Certbot\bin;%PATH% + venv\Scripts\python -m pytest certbot-ci\certbot_integration_tests\certbot_tests -n 4 + displayName: Run certbot integration tests diff --git a/.azure-pipelines/templates/tests-suite.yml b/.azure-pipelines/templates/tests-suite.yml new file mode 100644 index 000000000..119f755a6 --- /dev/null +++ b/.azure-pipelines/templates/tests-suite.yml @@ -0,0 +1,38 @@ +jobs: + - job: test + pool: + vmImage: vs2017-win2016 + strategy: + matrix: + py35: + PYTHON_VERSION: 3.5 + TOXENV: py35 + py37-cover: + PYTHON_VERSION: 3.7 + TOXENV: py37-cover + integration-certbot: + PYTHON_VERSION: 3.7 + TOXENV: integration-certbot + PYTEST_ADDOPTS: --numprocesses 4 + variables: + - group: certbot-common + steps: + - task: UsePythonVersion@0 + inputs: + versionSpec: $(PYTHON_VERSION) + addToPath: true + - script: python tools/pip_install.py -U tox coverage + displayName: Install dependencies + - script: python -m tox + displayName: Run tox + # We do not require codecov report upload to succeed. So to avoid to break the pipeline if + # something goes wrong, each command is suffixed with a command that hides any non zero exit + # codes and echoes an informative message instead. + - bash: | + curl -s https://codecov.io/bash -o codecov-bash || echo "Failed to download codecov-bash" + chmod +x codecov-bash || echo "Failed to apply execute permissions on codecov-bash" + ./codecov-bash -F windows || echo "Codecov did not collect coverage reports" + condition: in(variables['TOXENV'], 'py37-cover', 'integration-certbot') + env: + CODECOV_TOKEN: $(codecov_token) + displayName: Publish coverage diff --git a/.codecov.yml b/.codecov.yml new file mode 100644 index 000000000..0a97fffe3 --- /dev/null +++ b/.codecov.yml @@ -0,0 +1,18 @@ +coverage: + status: + project: + default: off + linux: + flags: linux + # Fixed target instead of auto set by #7173, can + # be removed when flags in Codecov are added back. + target: 97.4 + threshold: 0.1 + base: auto + windows: + flags: windows + # Fixed target instead of auto set by #7173, can + # be removed when flags in Codecov are added back. + target: 97.4 + threshold: 0.1 + base: auto diff --git a/.coveragerc b/.coveragerc index 1a87ab2da..5d2a93148 100644 --- a/.coveragerc +++ b/.coveragerc @@ -1,2 +1,5 @@ +[run] +omit = */setup.py + [report] omit = */setup.py diff --git a/.github/stale.yml b/.github/stale.yml new file mode 100644 index 000000000..2e4106314 --- /dev/null +++ b/.github/stale.yml @@ -0,0 +1,35 @@ +# Configuration for https://github.com/marketplace/stale + +# Number of days of inactivity before an Issue or Pull Request becomes stale +daysUntilStale: 365 + +# Number of days of inactivity before an Issue or Pull Request with the stale label is closed. +# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale. +# When changing this value, be sure to also update markComment below. +daysUntilClose: 30 + +# Ignore issues with an assignee (defaults to false) +exemptAssignees: true + +# Label to use when marking as stale +staleLabel: needs-update + +# Comment to post when marking as stale. Set to `false` to disable +markComment: > + We've made a lot of changes to Certbot since this issue was opened. If you + still have this issue with an up-to-date version of Certbot, can you please + add a comment letting us know? This helps us to better see what issues are + still affecting our users. If there is no activity in the next 30 days, this + issue will be automatically closed. + +# Comment to post when closing a stale Issue or Pull Request. +closeComment: > + This issue has been closed due to lack of activity, but if you think it + should be reopened, please open a new issue with a link to this one and we'll + take a look. + +# Limit the number of actions per hour, from 1-30. Default is 30 +limitPerRun: 1 + +# Don't mark pull requests as stale. +only: issues diff --git a/.gitignore b/.gitignore index 54545e883..6dd422187 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,7 @@ tags \#*# .idea .ropeproject +.vscode # auth --cert-path --chain-path /*.pem @@ -44,3 +45,8 @@ tests/letstest/venv/ # docker files .docker + +# certbot tests +.certbot_test_workspace +**/assets/pebble* +**/assets/challtestsrv* diff --git a/.isort.cfg b/.isort.cfg new file mode 100644 index 000000000..11c895f4d --- /dev/null +++ b/.isort.cfg @@ -0,0 +1,7 @@ +[settings] +skip_glob=venv* +skip=letsencrypt-auto-source +force_sort_within_sections=True +force_single_line=True +order_by_type=False +line_length=400 diff --git a/.pylintrc b/.pylintrc index 1d3f0ac4f..0e78828bd 100644 --- a/.pylintrc +++ b/.pylintrc @@ -24,6 +24,11 @@ persistent=yes # usually to register additional checkers. load-plugins=linter_plugin +# A comma-separated list of package or module names from where C extensions may +# be loaded. Extensions are loading into the active Python interpreter and may +# run arbitrary code. +extension-pkg-whitelist=pywintypes,win32api,win32file,win32security + [MESSAGES CONTROL] @@ -41,10 +46,14 @@ load-plugins=linter_plugin # --enable=similarities". If you want to run only the classes checker, but have # no Warning level messages displayed, use"--disable=all --enable=classes # --disable=W" -disable=fixme,locally-disabled,locally-enabled,abstract-class-not-used,abstract-class-little-used,bad-continuation,too-few-public-methods,no-self-use,invalid-name,too-many-instance-attributes,cyclic-import,duplicate-code -# abstract-class-not-used cannot be disabled locally (at least in -# pylint 1.4.1), same for abstract-class-little-used - +# CERTBOT COMMENT +# 1) Once certbot codebase is claimed to be compatible exclusively with Python 3, +# the useless-object-inheritance check can be enabled again, and code fixed accordingly. +# 2) Check unsubscriptable-object tends to create a lot of false positives. Let's disable it. +# See https://github.com/PyCQA/pylint/issues/1498. +# 3) Same as point 2 for no-value-for-parameter. +# See https://github.com/PyCQA/pylint/issues/2820. +disable=fixme,locally-disabled,locally-enabled,bad-continuation,no-self-use,invalid-name,cyclic-import,duplicate-code,design,import-outside-toplevel,useless-object-inheritance,unsubscriptable-object,no-value-for-parameter,no-else-return,no-else-raise,no-else-break,no-else-continue [REPORTS] @@ -251,7 +260,7 @@ ignored-modules=pkg_resources,confargparse,argparse,six.moves,six.moves.urllib # List of classes names for which member attributes should not be checked # (useful for classes with attributes dynamically set). -ignored-classes=SQLObject +ignored-classes=Field,Header,JWS,closing # When zope mode is activated, add a predefined set of Zope acquired attributes # to generated-members. @@ -297,40 +306,6 @@ valid-classmethod-first-arg=cls valid-metaclass-classmethod-first-arg=mcs -[DESIGN] - -# Maximum number of arguments for function / method -max-args=6 - -# Argument names that match this expression will be ignored. Default to name -# with leading underscore -ignored-argument-names=_.* - -# Maximum number of locals for function / method body -max-locals=15 - -# Maximum number of return / yield for function / method body -max-returns=6 - -# Maximum number of branch for function / method body -max-branches=12 - -# Maximum number of statements in function / method body -max-statements=50 - -# Maximum number of parents for a class (see R0901). -max-parents=12 - -# Maximum number of attributes for a class (see R0902). -max-attributes=7 - -# Minimum number of public methods for a class (see R0903). -min-public-methods=2 - -# Maximum number of public methods for a class (see R0904). -max-public-methods=20 - - [EXCEPTIONS] # Exceptions that will emit a warning when being caught. Defaults to diff --git a/.travis.yml b/.travis.yml index 16cb6f23f..6c5147603 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,146 +1,221 @@ language: python +dist: xenial cache: directories: - $HOME/.cache/pip before_script: - - 'if [ $TRAVIS_OS_NAME = osx ] ; then ulimit -n 1024 ; fi' + - 'if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then ulimit -n 1024 ; fi' + # On Travis, the fastest parallelization for integration tests has proved to be 4. + - 'if [[ "$TOXENV" == *"integration"* ]]; then export PYTEST_ADDOPTS="--numprocesses 4"; fi' + # Use Travis retry feature for farm tests since they are flaky + - 'if [[ "$TOXENV" == "travis-test-farm"* ]]; then export TRAVIS_RETRY=travis_retry; fi' - export TOX_TESTENV_PASSENV=TRAVIS +# Only build pushes to the master branch, PRs, and branches beginning with +# `test-`, `travis-test-`, or of the form `digit(s).digit(s).x`. This reduces +# the number of simultaneous Travis runs, which speeds turnaround time on +# review since there is a cap of on the number of simultaneous runs. +branches: + # When changing these branches, please ensure the documentation under + # "Running tests in CI" is still correct. + only: + # apache-parser-v2 is a temporary branch for doing work related to + # rewriting the parser in the Apache plugin. + - apache-parser-v2 + - master + - /^\d+\.\d+\.x$/ + - /^(travis-)?test-.*$/ + +# Jobs for the main test suite are always executed (including on PRs) except for pushes on master. +not-on-master: ¬-on-master + if: NOT (type = push AND branch = master) + +# Jobs for the extended test suite are executed for cron jobs and pushes to +# non-development branches. See the explanation for apache-parser-v2 above. +extended-test-suite: &extended-test-suite + if: type = cron OR (type = push AND branch NOT IN (apache-parser-v2, master)) + matrix: include: - # These environments are always executed + # Main test suite - python: "2.7" - env: BOULDER_INTEGRATION=v1 INTEGRATION_TEST=all TOXENV=py27_install - sudo: required - services: docker - - python: "2.7" - env: BOULDER_INTEGRATION=v2 INTEGRATION_TEST=all TOXENV=py27_install - sudo: required - services: docker + env: ACME_SERVER=pebble TOXENV=integration + <<: *not-on-master + + # This job is always executed, including on master - python: "2.7" env: TOXENV=py27-cover FYI="py27 tests + code coverage" - - sudo: required - env: TOXENV=nginx_compat - services: docker - before_install: - addons: - - python: "2.7" + + - python: "3.7" env: TOXENV=lint - - python: "3.4" - env: TOXENV=mypy + <<: *not-on-master - python: "3.5" env: TOXENV=mypy + <<: *not-on-master - python: "2.7" - env: TOXENV='py27-{acme,apache,certbot,dns,nginx,postfix}-oldest' - sudo: required - services: docker - - python: "3.4" - env: TOXENV=py34 - sudo: required - services: docker - - python: "3.7" - dist: xenial - env: TOXENV=py37 - sudo: required - services: docker + # Ubuntu Trusty or older must be used because the oldest version of + # cryptography we support cannot be compiled against the version of + # OpenSSL in Xenial or newer. + dist: trusty + env: TOXENV='py27-{acme,apache,certbot,dns,nginx}-oldest' + <<: *not-on-master + - python: "3.5" + env: TOXENV=py35 + <<: *not-on-master + - python: "3.8" + env: TOXENV=py38 + <<: *not-on-master - sudo: required env: TOXENV=apache_compat services: docker before_install: addons: - - sudo: required - env: TOXENV=le_auto_trusty - services: docker - before_install: - addons: - - python: "2.7" - env: TOXENV=apacheconftest-with-pebble - sudo: required - services: docker - - python: "2.7" - env: TOXENV=nginxroundtrip - - # These environments are executed on cron events only - - python: "3.7" - dist: xenial - env: TOXENV=py37 CERTBOT_NO_PIN=1 - if: type = cron - - python: "2.7" - env: BOULDER_INTEGRATION=v1 INTEGRATION_TEST=certbot TOXENV=py27-certbot-oldest - sudo: required - services: docker - if: type = cron - - python: "2.7" - env: BOULDER_INTEGRATION=v2 INTEGRATION_TEST=certbot TOXENV=py27-certbot-oldest - sudo: required - services: docker - if: type = cron - - python: "2.7" - env: BOULDER_INTEGRATION=v1 INTEGRATION_TEST=nginx TOXENV=py27-nginx-oldest - sudo: required - services: docker - if: type = cron - - python: "2.7" - env: BOULDER_INTEGRATION=v2 INTEGRATION_TEST=nginx TOXENV=py27-nginx-oldest - sudo: required - services: docker - if: type = cron - - python: "3.4" - env: TOXENV=py34 BOULDER_INTEGRATION=v1 - sudo: required - services: docker - if: type = cron - - python: "3.4" - env: TOXENV=py34 BOULDER_INTEGRATION=v2 - sudo: required - services: docker - if: type = cron - - python: "3.5" - env: TOXENV=py35 BOULDER_INTEGRATION=v1 - sudo: required - services: docker - if: type = cron - - python: "3.5" - env: TOXENV=py35 BOULDER_INTEGRATION=v2 - sudo: required - services: docker - if: type = cron - - python: "3.6" - env: TOXENV=py36 BOULDER_INTEGRATION=v1 - sudo: required - services: docker - if: type = cron - - python: "3.6" - env: TOXENV=py36 BOULDER_INTEGRATION=v2 - sudo: required - services: docker - if: type = cron - - python: "3.7" - dist: xenial - env: TOXENV=py37 BOULDER_INTEGRATION=v1 - sudo: required - services: docker - if: type = cron - - python: "3.7" - dist: xenial - env: TOXENV=py37 BOULDER_INTEGRATION=v2 - sudo: required - services: docker - if: type = cron + <<: *not-on-master - sudo: required env: TOXENV=le_auto_xenial services: docker - if: type = cron + <<: *not-on-master + - python: "2.7" + env: TOXENV=apacheconftest-with-pebble + <<: *not-on-master + - python: "2.7" + env: TOXENV=nginxroundtrip + <<: *not-on-master + + # Extended test suite on cron jobs and pushes to tested branches other than master + - sudo: required + env: TOXENV=nginx_compat + services: docker + before_install: + addons: + <<: *extended-test-suite + - python: "2.7" + env: + - TOXENV=travis-test-farm-apache2 + - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw=" + <<: *extended-test-suite + - python: "2.7" + env: + - TOXENV=travis-test-farm-leauto-upgrades + - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw=" + git: + depth: false # This is needed to have the history to checkout old versions of certbot-auto. + <<: *extended-test-suite + - python: "2.7" + env: + - TOXENV=travis-test-farm-certonly-standalone + - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw=" + <<: *extended-test-suite + - python: "2.7" + env: + - TOXENV=travis-test-farm-sdists + - secure: "f+j/Lj9s1lcuKo5sEFrlRd1kIAMnIJI4z0MTI7QF8jl9Fkmbx7KECGzw31TNgzrOSzxSapHbcueFYvNCLKST+kE/8ogMZBbwqXfEDuKpyF6BY3uYoJn+wPVE5pIb8Hhe08xPte8TTDSMIyHI3EyTfcAKrIreauoArePvh/cRvSw=" + <<: *extended-test-suite + - python: "3.7" + env: TOXENV=py37 CERTBOT_NO_PIN=1 + <<: *extended-test-suite + - python: "2.7" + env: ACME_SERVER=boulder-v1 TOXENV=integration + sudo: required + services: docker + <<: *extended-test-suite + - python: "2.7" + env: ACME_SERVER=boulder-v2 TOXENV=integration + sudo: required + services: docker + <<: *extended-test-suite + - python: "2.7" + env: ACME_SERVER=boulder-v1 TOXENV=integration-certbot-oldest + # Ubuntu Trusty or older must be used because the oldest version of + # cryptography we support cannot be compiled against the version of + # OpenSSL in Xenial or newer. + dist: trusty + sudo: required + services: docker + <<: *extended-test-suite + - python: "2.7" + env: ACME_SERVER=boulder-v2 TOXENV=integration-certbot-oldest + # Ubuntu Trusty or older must be used because the oldest version of + # cryptography we support cannot be compiled against the version of + # OpenSSL in Xenial or newer. + dist: trusty + sudo: required + services: docker + <<: *extended-test-suite + - python: "2.7" + env: ACME_SERVER=boulder-v1 TOXENV=integration-nginx-oldest + # Ubuntu Trusty or older must be used because the oldest version of + # cryptography we support cannot be compiled against the version of + # OpenSSL in Xenial or newer. + dist: trusty + sudo: required + services: docker + <<: *extended-test-suite + - python: "2.7" + env: ACME_SERVER=boulder-v2 TOXENV=integration-nginx-oldest + # Ubuntu Trusty or older must be used because the oldest version of + # cryptography we support cannot be compiled against the version of + # OpenSSL in Xenial or newer. + dist: trusty + sudo: required + services: docker + <<: *extended-test-suite + - python: "3.6" + env: TOXENV=py36 + <<: *extended-test-suite + - python: "3.7" + env: TOXENV=py37 + <<: *extended-test-suite + - python: "3.5" + env: ACME_SERVER=boulder-v1 TOXENV=integration + sudo: required + services: docker + <<: *extended-test-suite + - python: "3.5" + env: ACME_SERVER=boulder-v2 TOXENV=integration + sudo: required + services: docker + <<: *extended-test-suite + - python: "3.6" + env: ACME_SERVER=boulder-v1 TOXENV=integration + sudo: required + services: docker + <<: *extended-test-suite + - python: "3.6" + env: ACME_SERVER=boulder-v2 TOXENV=integration + sudo: required + services: docker + <<: *extended-test-suite + - python: "3.7" + env: ACME_SERVER=boulder-v1 TOXENV=integration + sudo: required + services: docker + <<: *extended-test-suite + - python: "3.7" + env: ACME_SERVER=boulder-v2 TOXENV=integration + sudo: required + services: docker + <<: *extended-test-suite + - python: "3.8" + env: ACME_SERVER=boulder-v1 TOXENV=integration + <<: *extended-test-suite + - python: "3.8" + env: ACME_SERVER=boulder-v2 TOXENV=integration + <<: *extended-test-suite - sudo: required env: TOXENV=le_auto_jessie services: docker - if: type = cron + <<: *extended-test-suite - sudo: required env: TOXENV=le_auto_centos6 services: docker - if: type = cron + <<: *extended-test-suite + - sudo: required + env: TOXENV=le_auto_oraclelinux6 + services: docker + <<: *extended-test-suite - sudo: required env: TOXENV=docker_dev services: docker @@ -148,7 +223,7 @@ matrix: apt: packages: # don't install nginx and apache - libaugeas0 - if: type = cron + <<: *extended-test-suite - language: generic env: TOXENV=py27 os: osx @@ -157,7 +232,7 @@ matrix: packages: - augeas - python2 - if: type = cron + <<: *extended-test-suite - language: generic env: TOXENV=py3 os: osx @@ -166,19 +241,7 @@ matrix: packages: - augeas - python3 - if: type = cron - - - -# Only build pushes to the master branch, PRs, and branches beginning with -# `test-` or of the form `digit(s).digit(s).x`. This reduces the number of -# simultaneous Travis runs, which speeds turnaround time on review since there -# is a cap of on the number of simultaneous runs. -branches: - only: - - master - - /^\d+\.\d+\.x$/ - - /^test-.*$/ + <<: *extended-test-suite # container-based infrastructure sudo: false @@ -187,7 +250,6 @@ addons: apt: packages: # Keep in sync with letsencrypt-auto-source/pieces/bootstrappers/deb_common.sh and Boulder. - python-dev - - python-virtualenv - gcc - libaugeas0 - libssl-dev @@ -197,19 +259,30 @@ addons: - nginx-light - openssl -install: "travis_retry $(command -v pip || command -v pip3) install codecov tox" -script: - - travis_retry tox - - '[ -z "${BOULDER_INTEGRATION+x}" ] || (travis_retry tests/boulder-fetch.sh && tests/tox-boulder-integration.sh)' +# tools/pip_install.py is used to pin packages to a known working version +# except in tests where the environment variable CERTBOT_NO_PIN is set. +# virtualenv is listed here explicitly to make sure it is upgraded when +# CERTBOT_NO_PIN is set to work around failures we've seen when using an older +# version of virtualenv. +install: 'tools/pip_install.py -U codecov tox virtualenv' +# Most of the time TRAVIS_RETRY is an empty string, and has no effect on the +# script command. It is set only to `travis_retry` during farm tests, in +# order to trigger the Travis retry feature, and compensate the inherent +# flakiness of these specific tests. +script: '$TRAVIS_RETRY tox' -after_success: '[ "$TOXENV" == "py27-cover" ] && codecov' +after_success: '[ "$TOXENV" == "py27-cover" ] && codecov -F linux' notifications: email: false irc: + if: NOT branch =~ ^(travis-)?test-.*$ channels: - - secure: "SGWZl3ownKx9xKVV2VnGt7DqkTmutJ89oJV9tjKhSs84kLijU6EYdPnllqISpfHMTxXflNZuxtGo0wTDYHXBuZL47w1O32W6nzuXdra5zC+i4sYQwYULUsyfOv9gJX8zWAULiK0Z3r0oho45U+FR5ZN6TPCidi8/eGU+EEPwaAw=" + # This is set to a secure variable to prevent forks from sending + # notifications. This value was created by installing + # https://github.com/travis-ci/travis.rb and running + # `travis encrypt "chat.freenode.net#certbot-devel"`. + - secure: "EWW66E2+KVPZyIPR8ViENZwfcup4Gx3/dlimmAZE0WuLwxDCshBBOd3O8Rf6pBokEoZlXM5eDT6XdyJj8n0DLslgjO62pExdunXpbcMwdY7l1ELxX2/UbnDTE6UnPYa09qVBHNG7156Z6yE0x2lH4M9Ykvp0G0cubjPQHylAwo0=" on_cancel: never on_success: never on_failure: always - use_notice: true diff --git a/AUTHORS.md b/AUTHORS.md index 6ee739bc0..80a24d3be 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -5,6 +5,7 @@ Authors * Aaron Zuehlke * Ada Lovelace * [Adam Woodbeck](https://github.com/awoodbeck) +* [Adrien Ferrand](https://github.com/adferrand) * [Aidin Gharibnavaz](https://github.com/aidin36) * [AJ ONeal](https://github.com/coolaj86) * [Alcaro](https://github.com/Alcaro) @@ -14,8 +15,10 @@ Authors * [Alex Gaynor](https://github.com/alex) * [Alex Halderman](https://github.com/jhalderm) * [Alex Jordan](https://github.com/strugee) +* [Alex Zorin](https://github.com/alexzorin) * [Amjad Mashaal](https://github.com/TheNavigat) * [Andrew Murray](https://github.com/radarhere) +* [Andrzej Górski](https://github.com/andrzej3393) * [Anselm Levskaya](https://github.com/levskaya) * [Antoine Jacoutot](https://github.com/ajacoutot) * [asaph](https://github.com/asaph) @@ -33,6 +36,7 @@ Authors * [Brad Warren](https://github.com/bmw) * [Brandon Kraft](https://github.com/kraftbj) * [Brandon Kreisel](https://github.com/kraftbj) +* [Cameron Steel](https://github.com/Tugzrida) * [Ceesjan Luiten](https://github.com/quinox) * [Chad Whitacre](https://github.com/whit537) * [Chhatoi Pritam Baral](https://github.com/pritambaral) @@ -75,6 +79,7 @@ Authors * [Fabian](https://github.com/faerbit) * [Faidon Liambotis](https://github.com/paravoid) * [Fan Jiang](https://github.com/tcz001) +* [Felix Lechner](https://github.com/lechner) * [Felix Schwarz](https://github.com/FelixSchwarz) * [Felix Yan](https://github.com/felixonmars) * [Filip Ochnik](https://github.com/filipochnik) @@ -96,6 +101,7 @@ Authors * [Harlan Lieberman-Berg](https://github.com/hlieberman) * [Henri Salo](https://github.com/fgeek) * [Henry Chen](https://github.com/henrychen95) +* [Hugo van Kemenade](https://github.com/hugovk) * [Ingolf Becker](https://github.com/watercrossing) * [Jaap Eldering](https://github.com/eldering) * [Jacob Hoffman-Andrews](https://github.com/jsha) @@ -120,10 +126,12 @@ Authors * [Jonathan Herlin](https://github.com/Jonher937) * [Jon Walsh](https://github.com/code-tree) * [Joona Hoikkala](https://github.com/joohoi) +* [Josh McCullough](https://github.com/JoshMcCullough) * [Josh Soref](https://github.com/jsoref) * [Joubin Jabbari](https://github.com/joubin) * [Juho Juopperi](https://github.com/jkjuopperi) * [Kane York](https://github.com/riking) +* [Kenichi Maehashi](https://github.com/kmaehashi) * [Kenneth Skovhede](https://github.com/kenkendk) * [Kevin Burke](https://github.com/kevinburke) * [Kevin London](https://github.com/kevinlondon) @@ -159,8 +167,10 @@ Authors * [Michael Schumacher](https://github.com/schumaml) * [Michael Strache](https://github.com/Jarodiv) * [Michael Sverdlin](https://github.com/sveder) +* [Michael Watters](https://github.com/blackknight36) * [Michal Moravec](https://github.com/https://github.com/Majkl578) * [Michal Papis](https://github.com/mpapis) +* [Mickaël Schoentgen](https://github.com/BoboTiG) * [Minn Soe](https://github.com/MinnSoe) * [Min RK](https://github.com/minrk) * [Miquel Ruiz](https://github.com/miquelruiz) @@ -224,6 +234,7 @@ Authors * [Stavros Korokithakis](https://github.com/skorokithakis) * [Stefan Weil](https://github.com/stweil) * [Steve Desmond](https://github.com/stevedesmond-ca) +* [sydneyli](https://github.com/sydneyli) * [Tan Jay Jun](https://github.com/jayjun) * [Tapple Gao](https://github.com/tapple) * [Telepenin Nikolay](https://github.com/telepenin) diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 000000000..142b31c93 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1 @@ +This project is governed by [EFF's Public Projects Code of Conduct](https://www.eff.org/pages/eppcode). \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d740b7d89..07187eb59 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -33,3 +33,5 @@ started. In particular, we recommend you read these sections - [Finding issues to work on](https://certbot.eff.org/docs/contributing.html#find-issues-to-work-on) - [Coding style](https://certbot.eff.org/docs/contributing.html#coding-style) - [Submitting a pull request](https://certbot.eff.org/docs/contributing.html#submitting-a-pull-request) + - [EFF's Public Projects Code of Conduct](https://www.eff.org/pages/eppcode) + diff --git a/Dockerfile b/Dockerfile deleted file mode 100644 index f3626dc8d..000000000 --- a/Dockerfile +++ /dev/null @@ -1,29 +0,0 @@ -FROM python:2-alpine3.7 - -ENTRYPOINT [ "certbot" ] -EXPOSE 80 443 -VOLUME /etc/letsencrypt /var/lib/letsencrypt -WORKDIR /opt/certbot - -COPY CHANGELOG.md README.rst setup.py src/ -COPY letsencrypt-auto-source/pieces/dependency-requirements.txt . -COPY acme src/acme -COPY certbot src/certbot - -RUN apk add --no-cache --virtual .certbot-deps \ - libffi \ - libssl1.0 \ - openssl \ - ca-certificates \ - binutils -RUN apk add --no-cache --virtual .build-deps \ - gcc \ - linux-headers \ - openssl-dev \ - musl-dev \ - libffi-dev \ - && pip install -r /opt/certbot/dependency-requirements.txt \ - && pip install --no-cache-dir \ - --editable /opt/certbot/src/acme \ - --editable /opt/certbot/src \ - && apk del .build-deps diff --git a/Dockerfile-dev b/Dockerfile-dev index 1ab56e081..ae197b1cb 100644 --- a/Dockerfile-dev +++ b/Dockerfile-dev @@ -1,21 +1,20 @@ # This Dockerfile builds an image for development. -FROM ubuntu:xenial +FROM debian:buster # Note: this only exposes the port to other docker containers. EXPOSE 80 443 WORKDIR /opt/certbot/src -# TODO: Install Apache/Nginx for plugin development. COPY . . RUN apt-get update && \ - apt-get install apache2 git nginx-light -y && \ - letsencrypt-auto-source/letsencrypt-auto --os-packages-only && \ + apt-get install apache2 git python3-dev python3-venv gcc libaugeas0 \ + libssl-dev libffi-dev ca-certificates openssl nginx-light -y && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* \ /tmp/* \ /var/tmp/* -RUN VENV_NAME="../venv" python tools/venv.py +RUN VENV_NAME="../venv3" python3 tools/venv3.py -ENV PATH /opt/certbot/venv/bin:$PATH +ENV PATH /opt/certbot/venv3/bin:$PATH diff --git a/Dockerfile-old b/Dockerfile-old deleted file mode 100644 index c52a9937b..000000000 --- a/Dockerfile-old +++ /dev/null @@ -1,75 +0,0 @@ -# https://github.com/letsencrypt/letsencrypt/pull/431#issuecomment-103659297 -# it is more likely developers will already have ubuntu:trusty rather -# than e.g. debian:jessie and image size differences are negligible -FROM ubuntu:trusty -MAINTAINER Jakub Warmuz -MAINTAINER William Budington - -# Note: this only exposes the port to other docker containers. You -# still have to bind to 443@host at runtime, as per the ACME spec. -EXPOSE 443 - -# TODO: make sure --config-dir and --work-dir cannot be changed -# through the CLI (certbot-docker wrapper that uses standalone -# authenticator and text mode only?) -VOLUME /etc/letsencrypt /var/lib/letsencrypt - -WORKDIR /opt/certbot - -# no need to mkdir anything: -# https://docs.docker.com/reference/builder/#copy -# If doesn't exist, it is created along with all missing -# directories in its path. - -ENV DEBIAN_FRONTEND=noninteractive - -COPY letsencrypt-auto-source/letsencrypt-auto /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto -RUN /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto --os-packages-only && \ - apt-get clean && \ - rm -rf /var/lib/apt/lists/* \ - /tmp/* \ - /var/tmp/* - -# the above is not likely to change, so by putting it further up the -# Dockerfile we make sure we cache as much as possible - - -COPY setup.py README.rst CHANGELOG.md MANIFEST.in letsencrypt-auto-source/pieces/pipstrap.py /opt/certbot/src/ - -# all above files are necessary for setup.py and venv setup, however, -# package source code directory has to be copied separately to a -# subdirectory... -# https://docs.docker.com/reference/builder/#copy: "If is a -# directory, the entire contents of the directory are copied, -# including filesystem metadata. Note: The directory itself is not -# copied, just its contents." Order again matters, three files are far -# more likely to be cached than the whole project directory - -COPY certbot /opt/certbot/src/certbot/ -COPY acme /opt/certbot/src/acme/ -COPY certbot-apache /opt/certbot/src/certbot-apache/ -COPY certbot-nginx /opt/certbot/src/certbot-nginx/ - - -RUN VIRTUALENV_NO_DOWNLOAD=1 virtualenv --no-site-packages -p python2 /opt/certbot/venv - -# PATH is set now so pipstrap upgrades the correct (v)env -ENV PATH /opt/certbot/venv/bin:$PATH -RUN /opt/certbot/venv/bin/python /opt/certbot/src/pipstrap.py && \ - /opt/certbot/venv/bin/pip install \ - -e /opt/certbot/src/acme \ - -e /opt/certbot/src \ - -e /opt/certbot/src/certbot-apache \ - -e /opt/certbot/src/certbot-nginx - -# install in editable mode (-e) to save space: it's not possible to -# "rm -rf /opt/certbot/src" (it's stays in the underlaying image); -# this might also help in debugging: you can "docker run --entrypoint -# bash" and investigate, apply patches, etc. - -# set up certbot/letsencrypt wrapper to warn people about Dockerfile changes -COPY tools/docker-warning.sh /opt/certbot/bin/certbot -RUN ln -s /opt/certbot/bin/certbot /opt/certbot/bin/letsencrypt -ENV PATH /opt/certbot/bin:$PATH - -ENTRYPOINT [ "certbot" ] diff --git a/README.rst b/README.rst deleted file mode 100644 index f55581268..000000000 --- a/README.rst +++ /dev/null @@ -1,157 +0,0 @@ -.. This file contains a series of comments that are used to include sections of this README in other files. Do not modify these comments unless you know what you are doing. tag:intro-begin - -Certbot is part of EFF’s effort to encrypt the entire Internet. Secure communication over the Web relies on HTTPS, which requires the use of a digital certificate that lets browsers verify the identity of web servers (e.g., is that really google.com?). Web servers obtain their certificates from trusted third parties called certificate authorities (CAs). Certbot is an easy-to-use client that fetches a certificate from Let’s Encrypt—an open certificate authority launched by the EFF, Mozilla, and others—and deploys it to a web server. - -Anyone who has gone through the trouble of setting up a secure website knows what a hassle getting and maintaining a certificate is. Certbot and Let’s Encrypt can automate away the pain and let you turn on and manage HTTPS with simple commands. Using Certbot and Let's Encrypt is free, so there’s no need to arrange payment. - -How you use Certbot depends on the configuration of your web server. The best way to get started is to use our `interactive guide `_. It generates instructions based on your configuration settings. In most cases, you’ll need `root or administrator access `_ to your web server to run Certbot. - -Certbot is meant to be run directly on your web server, not on your personal computer. If you’re using a hosted service and don’t have direct access to your web server, you might not be able to use Certbot. Check with your hosting provider for documentation about uploading certificates or using certificates issued by Let’s Encrypt. - -Certbot is a fully-featured, extensible client for the Let's -Encrypt CA (or any other CA that speaks the `ACME -`_ -protocol) that can automate the tasks of obtaining certificates and -configuring webservers to use them. This client runs on Unix-based operating -systems. - -To see the changes made to Certbot between versions please refer to our -`changelog `_. - -Until May 2016, Certbot was named simply ``letsencrypt`` or ``letsencrypt-auto``, -depending on install method. Instructions on the Internet, and some pieces of the -software, may still refer to this older name. - -Contributing ------------- - -If you'd like to contribute to this project please read `Developer Guide -`_. - -.. _installation: - -Installation ------------- - -The easiest way to install Certbot is by visiting `certbot.eff.org`_, where you can -find the correct installation instructions for many web server and OS combinations. -For more information, see `Get Certbot `_. - -.. _certbot.eff.org: https://certbot.eff.org/ - -How to run the client ---------------------- - -In many cases, you can just run ``certbot-auto`` or ``certbot``, and the -client will guide you through the process of obtaining and installing certs -interactively. - -For full command line help, you can type:: - - ./certbot-auto --help all - - -You can also tell it exactly what you want it to do from the command line. -For instance, if you want to obtain a cert for ``example.com``, -``www.example.com``, and ``other.example.net``, using the Apache plugin to both -obtain and install the certs, you could do this:: - - ./certbot-auto --apache -d example.com -d www.example.com -d other.example.net - -(The first time you run the command, it will make an account, and ask for an -email and agreement to the Let's Encrypt Subscriber Agreement; you can -automate those with ``--email`` and ``--agree-tos``) - -If you want to use a webserver that doesn't have full plugin support yet, you -can still use "standalone" or "webroot" plugins to obtain a certificate:: - - ./certbot-auto certonly --standalone --email admin@example.com -d example.com -d www.example.com -d other.example.net - - -Understanding the client in more depth --------------------------------------- - -To understand what the client is doing in detail, it's important to -understand the way it uses plugins. Please see the `explanation of -plugins `_ in -the User Guide. - -Links -===== - -.. Do not modify this comment unless you know what you're doing. tag:links-begin - -Documentation: https://certbot.eff.org/docs - -Software project: https://github.com/certbot/certbot - -Notes for developers: https://certbot.eff.org/docs/contributing.html - -Main Website: https://certbot.eff.org - -Let's Encrypt Website: https://letsencrypt.org - -Community: https://community.letsencrypt.org - -ACME spec: http://ietf-wg-acme.github.io/acme/ - -ACME working area in github: https://github.com/ietf-wg-acme/acme - -|build-status| |coverage| |docs| |container| - -.. |build-status| image:: https://travis-ci.com/certbot/certbot.svg?branch=master - :target: https://travis-ci.com/certbot/certbot - :alt: Travis CI status - -.. |coverage| image:: https://codecov.io/gh/certbot/certbot/branch/master/graph/badge.svg - :target: https://codecov.io/gh/certbot/certbot - :alt: Coverage status - -.. |docs| image:: https://readthedocs.org/projects/letsencrypt/badge/ - :target: https://readthedocs.org/projects/letsencrypt/ - :alt: Documentation status - -.. |container| image:: https://quay.io/repository/letsencrypt/letsencrypt/status - :target: https://quay.io/repository/letsencrypt/letsencrypt - :alt: Docker Repository on Quay.io - -.. Do not modify this comment unless you know what you're doing. tag:links-end - -System Requirements -=================== - -See https://certbot.eff.org/docs/install.html#system-requirements. - -.. Do not modify this comment unless you know what you're doing. tag:intro-end - -.. Do not modify this comment unless you know what you're doing. tag:features-begin - -Current Features -===================== - -* Supports multiple web servers: - - - apache/2.x - - nginx/0.8.48+ - - webroot (adds files to webroot directories in order to prove control of - domains and obtain certs) - - standalone (runs its own simple webserver to prove you control a domain) - - other server software via `third party plugins `_ - -* The private key is generated locally on your system. -* Can talk to the Let's Encrypt CA or optionally to other ACME - compliant services. -* Can get domain-validated (DV) certificates. -* Can revoke certificates. -* Adjustable RSA key bit-length (2048 (default), 4096, ...). -* Can optionally install a http -> https redirect, so your site effectively - runs https only (Apache only) -* Fully automated. -* Configuration changes are logged and can be reverted. -* Supports an interactive text UI, or can be driven entirely from the - command line. -* Free and Open Source Software, made with Python. - -.. Do not modify this comment unless you know what you're doing. tag:features-end - -For extensive documentation on using and contributing to Certbot, go to https://certbot.eff.org/docs. If you would like to contribute to the project or run the latest code from git, you should read our `developer guide `_. diff --git a/README.rst b/README.rst new file mode 120000 index 000000000..645fd4c78 --- /dev/null +++ b/README.rst @@ -0,0 +1 @@ +certbot/README.rst \ No newline at end of file diff --git a/acme/MANIFEST.in b/acme/MANIFEST.in index 1619bef69..de254250e 100644 --- a/acme/MANIFEST.in +++ b/acme/MANIFEST.in @@ -3,4 +3,6 @@ include README.rst include pytest.ini recursive-include docs * recursive-include examples * -recursive-include acme/testdata * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/acme/acme/__init__.py b/acme/acme/__init__.py index d91072a3b..d1679fcad 100644 --- a/acme/acme/__init__.py +++ b/acme/acme/__init__.py @@ -6,13 +6,13 @@ This module is an implementation of the `ACME protocol`_. """ import sys +import warnings # This code exists to keep backwards compatibility with people using acme.jose # before it became the standalone josepy package. # # It is based on # https://github.com/requests/requests/blob/1278ecdf71a312dc2268f3bfc0aabfab3c006dcf/requests/packages.py - import josepy as jose for mod in list(sys.modules): diff --git a/acme/acme/challenges.py b/acme/acme/challenges.py index 29b9bbb50..39c8d6269 100644 --- a/acme/acme/challenges.py +++ b/acme/acme/challenges.py @@ -3,27 +3,19 @@ import abc import functools import hashlib import logging -import socket -import warnings from cryptography.hazmat.primitives import hashes # type: ignore import josepy as jose -import OpenSSL import requests import six -from acme import errors -from acme import crypto_util from acme import fields logger = logging.getLogger(__name__) -# pylint: disable=too-few-public-methods - - class Challenge(jose.TypedJSONObjectWithFields): - # _fields_to_partial_json | pylint: disable=abstract-method + # _fields_to_partial_json """ACME challenge.""" TYPES = {} # type: dict @@ -37,7 +29,7 @@ class Challenge(jose.TypedJSONObjectWithFields): class ChallengeResponse(jose.TypedJSONObjectWithFields): - # _fields_to_partial_json | pylint: disable=abstract-method + # _fields_to_partial_json """ACME challenge response.""" TYPES = {} # type: dict resource_type = 'challenge' @@ -62,8 +54,7 @@ class UnrecognizedChallenge(Challenge): object.__setattr__(self, "jobj", jobj) def to_partial_json(self): - # pylint: disable=no-member - return self.jobj + return self.jobj # pylint: disable=no-member @classmethod def from_json(cls, jobj): @@ -96,6 +87,7 @@ class _TokenChallenge(Challenge): """ # TODO: check that path combined with uri does not go above # URI_ROOT_PATH! + # pylint: disable=unsupported-membership-test return b'..' not in self.token and b'/' not in self.token @@ -120,7 +112,7 @@ class KeyAuthorizationChallengeResponse(ChallengeResponse): :rtype: bool """ - parts = self.key_authorization.split('.') # pylint: disable=no-member + parts = self.key_authorization.split('.') if len(parts) != 2: logger.debug("Key authorization (%r) is not well formed", self.key_authorization) @@ -140,10 +132,14 @@ class KeyAuthorizationChallengeResponse(ChallengeResponse): return True + def to_partial_json(self): + jobj = super(KeyAuthorizationChallengeResponse, self).to_partial_json() + jobj.pop('keyAuthorization', None) + return jobj + @six.add_metaclass(abc.ABCMeta) class KeyAuthorizationChallenge(_TokenChallenge): - # pylint: disable=abstract-class-little-used,too-many-ancestors """Challenge based on Key Authorization. :param response_cls: Subclass of `KeyAuthorizationChallengeResponse` @@ -175,7 +171,7 @@ class KeyAuthorizationChallenge(_TokenChallenge): :rtype: KeyAuthorizationChallengeResponse """ - return self.response_cls( + return self.response_cls( # pylint: disable=not-callable key_authorization=self.key_authorization(account_key)) @abc.abstractmethod @@ -212,7 +208,7 @@ class DNS01Response(KeyAuthorizationChallengeResponse): """ACME dns-01 challenge response.""" typ = "dns-01" - def simple_verify(self, chall, domain, account_public_key): + def simple_verify(self, chall, domain, account_public_key): # pylint: disable=unused-argument """Simple verify. This method no longer checks DNS records and is a simple wrapper @@ -228,14 +224,13 @@ class DNS01Response(KeyAuthorizationChallengeResponse): :rtype: bool """ - # pylint: disable=unused-argument verified = self.verify(chall, account_public_key) if not verified: logger.debug("Verification of key authorization in response failed") return verified -@Challenge.register # pylint: disable=too-many-ancestors +@Challenge.register class DNS01(KeyAuthorizationChallenge): """ACME dns-01 challenge.""" response_cls = DNS01Response @@ -325,7 +320,7 @@ class HTTP01Response(KeyAuthorizationChallengeResponse): return True -@Challenge.register # pylint: disable=too-many-ancestors +@Challenge.register class HTTP01(KeyAuthorizationChallenge): """ACME http-01 challenge.""" response_cls = HTTP01Response @@ -365,154 +360,6 @@ class HTTP01(KeyAuthorizationChallenge): return self.key_authorization(account_key) -@ChallengeResponse.register -class TLSSNI01Response(KeyAuthorizationChallengeResponse): - """ACME tls-sni-01 challenge response.""" - typ = "tls-sni-01" - - DOMAIN_SUFFIX = b".acme.invalid" - """Domain name suffix.""" - - PORT = 443 - """Verification port as defined by the protocol. - - You can override it (e.g. for testing) by passing ``port`` to - `simple_verify`. - - """ - - @property - def z(self): # pylint: disable=invalid-name - """``z`` value used for verification. - - :rtype bytes: - - """ - return hashlib.sha256( - self.key_authorization.encode("utf-8")).hexdigest().lower().encode() - - @property - def z_domain(self): - """Domain name used for verification, generated from `z`. - - :rtype bytes: - - """ - return self.z[:32] + b'.' + self.z[32:] + self.DOMAIN_SUFFIX - - def gen_cert(self, key=None, bits=2048): - """Generate tls-sni-01 certificate. - - :param OpenSSL.crypto.PKey key: Optional private key used in - certificate generation. If not provided (``None``), then - fresh key will be generated. - :param int bits: Number of bits for newly generated key. - - :rtype: `tuple` of `OpenSSL.crypto.X509` and `OpenSSL.crypto.PKey` - - """ - if key is None: - key = OpenSSL.crypto.PKey() - key.generate_key(OpenSSL.crypto.TYPE_RSA, bits) - return crypto_util.gen_ss_cert(key, [ - # z_domain is too big to fit into CN, hence first dummy domain - 'dummy', self.z_domain.decode()], force_san=True), key - - def probe_cert(self, domain, **kwargs): - """Probe tls-sni-01 challenge certificate. - - :param unicode domain: - - """ - # TODO: domain is not necessary if host is provided - if "host" not in kwargs: - host = socket.gethostbyname(domain) - logger.debug('%s resolved to %s', domain, host) - kwargs["host"] = host - - kwargs.setdefault("port", self.PORT) - kwargs["name"] = self.z_domain - # TODO: try different methods? - # pylint: disable=protected-access - return crypto_util.probe_sni(**kwargs) - - def verify_cert(self, cert): - """Verify tls-sni-01 challenge certificate. - - :param OpensSSL.crypto.X509 cert: Challenge certificate. - - :returns: Whether the certificate was successfully verified. - :rtype: bool - - """ - # pylint: disable=protected-access - sans = crypto_util._pyopenssl_cert_or_req_san(cert) - logger.debug('Certificate %s. SANs: %s', cert.digest('sha256'), sans) - return self.z_domain.decode() in sans - - def simple_verify(self, chall, domain, account_public_key, - cert=None, **kwargs): - """Simple verify. - - Verify ``validation`` using ``account_public_key``, optionally - probe tls-sni-01 certificate and check using `verify_cert`. - - :param .challenges.TLSSNI01 chall: Corresponding challenge. - :param str domain: Domain name being validated. - :param JWK account_public_key: - :param OpenSSL.crypto.X509 cert: Optional certificate. If not - provided (``None``) certificate will be retrieved using - `probe_cert`. - :param int port: Port used to probe the certificate. - - - :returns: ``True`` iff client's control of the domain has been - verified. - :rtype: bool - - """ - if not self.verify(chall, account_public_key): - logger.debug("Verification of key authorization in response failed") - return False - - if cert is None: - try: - cert = self.probe_cert(domain=domain, **kwargs) - except errors.Error as error: - logger.debug(str(error), exc_info=True) - return False - - return self.verify_cert(cert) - - -@Challenge.register # pylint: disable=too-many-ancestors -class TLSSNI01(KeyAuthorizationChallenge): - """ACME tls-sni-01 challenge.""" - response_cls = TLSSNI01Response - typ = response_cls.typ - - # boulder#962, ietf-wg-acme#22 - #n = jose.Field("n", encoder=int, decoder=int) - - def __init__(self, *args, **kwargs): - warnings.warn("TLS-SNI-01 is deprecated, and will stop working soon.", - DeprecationWarning, stacklevel=2) - super(TLSSNI01, self).__init__(*args, **kwargs) - - def validation(self, account_key, **kwargs): - """Generate validation. - - :param JWK account_key: - :param OpenSSL.crypto.PKey cert_key: Optional private key used - in certificate generation. If not provided (``None``), then - fresh key will be generated. - - :rtype: `tuple` of `OpenSSL.crypto.X509` and `OpenSSL.crypto.PKey` - - """ - return self.response(account_key).gen_cert(key=kwargs.get('cert_key')) - - @ChallengeResponse.register class TLSALPN01Response(KeyAuthorizationChallengeResponse): """ACME TLS-ALPN-01 challenge response. @@ -524,7 +371,7 @@ class TLSALPN01Response(KeyAuthorizationChallengeResponse): typ = "tls-alpn-01" -@Challenge.register # pylint: disable=too-many-ancestors +@Challenge.register class TLSALPN01(KeyAuthorizationChallenge): """ACME tls-alpn-01 challenge. @@ -540,7 +387,7 @@ class TLSALPN01(KeyAuthorizationChallenge): raise NotImplementedError() -@Challenge.register # pylint: disable=too-many-ancestors +@Challenge.register class DNS(_TokenChallenge): """ACME "dns" challenge.""" typ = "dns" diff --git a/acme/acme/client.py b/acme/acme/client.py index 41338e17e..f48ff40b2 100644 --- a/acme/acme/client.py +++ b/acme/acme/client.py @@ -5,25 +5,26 @@ import datetime from email.utils import parsedate_tz import heapq import logging +import re +import sys import time -import six -from six.moves import http_client # pylint: disable=import-error import josepy as jose import OpenSSL -import re -from requests_toolbelt.adapters.source import SourceAddressAdapter import requests from requests.adapters import HTTPAdapter -import sys +from requests_toolbelt.adapters.source import SourceAddressAdapter +import six +from six.moves import http_client # pylint: disable=import-error from acme import crypto_util from acme import errors from acme import jws from acme import messages -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import Dict, List, Set, Text - +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Text # pylint: disable=unused-import, no-name-in-module logger = logging.getLogger(__name__) @@ -33,7 +34,6 @@ logger = logging.getLogger(__name__) # https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning if sys.version_info < (2, 7, 9): # pragma: no cover try: - # pylint: disable=no-member requests.packages.urllib3.contrib.pyopenssl.inject_into_urllib3() # type: ignore except AttributeError: import urllib3.contrib.pyopenssl # pylint: disable=import-error @@ -44,7 +44,7 @@ DEFAULT_NETWORK_TIMEOUT = 45 DER_CONTENT_TYPE = 'application/pkix-cert' -class ClientBase(object): # pylint: disable=too-many-instance-attributes +class ClientBase(object): """ACME client base object. :ivar messages.Directory directory: @@ -123,14 +123,21 @@ class ClientBase(object): # pylint: disable=too-many-instance-attributes """ return self.update_registration(regr, update={'status': 'deactivated'}) - def query_registration(self, regr): - """Query server about registration. + def deactivate_authorization(self, authzr): + # type: (messages.AuthorizationResource) -> messages.AuthorizationResource + """Deactivate authorization. - :param messages.RegistrationResource: Existing Registration - Resource. + :param messages.AuthorizationResource authzr: The Authorization resource + to be deactivated. + + :returns: The Authorization resource that was deactivated. + :rtype: `.AuthorizationResource` """ - return self._send_recv_regr(regr, messages.UpdateRegistration()) + body = messages.UpdateAuthorization(status='deactivated') + response = self._post(authzr.uri, body) + return self._authzr_from_response(response, + authzr.body.identifier, authzr.uri) def _authzr_from_response(self, response, identifier=None, uri=None): authzr = messages.AuthorizationResource( @@ -247,7 +254,6 @@ class Client(ClientBase): URI from which the resource will be downloaded. """ - # pylint: disable=too-many-arguments self.key = key if net is None: net = ClientNetwork(key, alg=alg, verify_ssl=verify_ssl) @@ -273,9 +279,17 @@ class Client(ClientBase): assert response.status_code == http_client.CREATED # "Instance of 'Field' has no key/contact member" bug: - # pylint: disable=no-member return self._regr_from_response(response) + def query_registration(self, regr): + """Query server about registration. + + :param messages.RegistrationResource: Existing Registration + Resource. + + """ + return self._send_recv_regr(regr, messages.UpdateRegistration()) + def agree_to_tos(self, regr): """Agree to the terms-of-service. @@ -419,7 +433,6 @@ class Client(ClientBase): was marked by the CA as invalid """ - # pylint: disable=too-many-locals assert max_attempts > 0 attempts = collections.defaultdict(int) # type: Dict[messages.AuthorizationResource, int] exhausted = set() @@ -450,7 +463,6 @@ class Client(ClientBase): updated[authzr] = updated_authzr attempts[authzr] += 1 - # pylint: disable=no-member if updated_authzr.body.status not in ( messages.STATUS_VALID, messages.STATUS_INVALID): if attempts[authzr] < max_attempts: @@ -591,7 +603,6 @@ class ClientV2(ClientBase): if response.status_code == 200 and 'Location' in response.headers: raise errors.ConflictError(response.headers.get('Location')) # "Instance of 'Field' has no key/contact member" bug: - # pylint: disable=no-member regr = self._regr_from_response(response) self.net.account = regr return regr @@ -603,10 +614,13 @@ class ClientV2(ClientBase): Resource. """ - self.net.account = regr - updated_regr = super(ClientV2, self).query_registration(regr) - self.net.account = updated_regr - return updated_regr + self.net.account = regr # See certbot/certbot#6258 + # ACME v2 requires to use a POST-as-GET request (POST an empty JWS) here. + # This is done by passing None instead of an empty UpdateRegistration to _post(). + response = self._post(regr.uri, None) + self.net.account = self._regr_from_response(response, uri=regr.uri, + terms_of_service=regr.terms_of_service) + return self.net.account def update_registration(self, regr, update=None): """Update registration. @@ -652,7 +666,7 @@ class ClientV2(ClientBase): response = self._post(self.directory['newOrder'], order) body = messages.Order.from_json(response.json()) authorizations = [] - for url in body.authorizations: + for url in body.authorizations: # pylint: disable=not-an-iterable authorizations.append(self._authzr_from_response(self._post_as_get(url), uri=url)) return messages.OrderResource( body=body, @@ -712,9 +726,9 @@ class ClientV2(ClientBase): for authzr in responses: if authzr.body.status != messages.STATUS_VALID: for chall in authzr.body.challenges: - if chall.error != None: + if chall.error is not None: failed.append(authzr) - if len(failed) > 0: + if failed: raise errors.ValidationError(failed) return orderr.update(authorizations=responses) @@ -739,8 +753,7 @@ class ClientV2(ClientBase): if body.error is not None: raise errors.IssuanceError(body.error) if body.certificate is not None: - certificate_response = self._post_as_get(body.certificate, - content_type=DER_CONTENT_TYPE).text + certificate_response = self._post_as_get(body.certificate).text return orderr.update(body=body, fullchain_pem=certificate_response) raise errors.TimeoutError() @@ -759,36 +772,17 @@ class ClientV2(ClientBase): def external_account_required(self): """Checks if ACME server requires External Account Binding authentication.""" - if hasattr(self.directory, 'meta') and self.directory.meta.external_account_required: - return True - else: - return False + return hasattr(self.directory, 'meta') and self.directory.meta.external_account_required def _post_as_get(self, *args, **kwargs): """ - Send GET request using the POST-as-GET protocol if needed. - The request will be first issued using POST-as-GET for ACME v2. If the ACME CA servers do - not support this yet and return an error, request will be retried using GET. - For ACME v1, only GET request will be tried, as POST-as-GET is not supported. + Send GET request using the POST-as-GET protocol. :param args: :param kwargs: :return: """ - if self.acme_version >= 2: - # We add an empty payload for POST-as-GET requests - new_args = args[:1] + (None,) + args[1:] - try: - return self._post(*new_args, **kwargs) # pylint: disable=star-args - except messages.Error as error: - if error.code == 'malformed': - logger.debug('Error during a POST-as-GET request, ' - 'your ACME CA may not support it:\n%s', error) - logger.debug('Retrying request with GET.') - else: # pragma: no cover - raise - - # If POST-as-GET is not supported yet, we use a GET instead. - return self.net.get(*args, **kwargs) + new_args = args[:1] + (None,) + args[1:] + return self._post(*new_args, **kwargs) class BackwardsCompatibleClientV2(object): @@ -866,8 +860,7 @@ class BackwardsCompatibleClientV2(object): for domain in dnsNames: authorizations.append(self.client.request_domain_challenges(domain)) return messages.OrderResource(authorizations=authorizations, csr_pem=csr_pem) - else: - return self.client.new_order(csr_pem) + return self.client.new_order(csr_pem) def finalize_order(self, orderr, deadline): """Finalize an order and obtain a certificate. @@ -904,8 +897,7 @@ class BackwardsCompatibleClientV2(object): chain = crypto_util.dump_pyopenssl_chain(chain).decode() return orderr.update(fullchain_pem=(cert + chain)) - else: - return self.client.finalize_order(orderr, deadline) + return self.client.finalize_order(orderr, deadline) def revoke(self, cert, rsn): """Revoke certificate. @@ -923,8 +915,7 @@ class BackwardsCompatibleClientV2(object): def _acme_version_from_directory(self, directory): if hasattr(directory, 'newNonce'): return 2 - else: - return 1 + return 1 def external_account_required(self): """Checks if the server requires an external account for ACMEv2 servers. @@ -932,11 +923,10 @@ class BackwardsCompatibleClientV2(object): Always return False for ACMEv1 servers, as it doesn't use External Account Binding.""" if self.acme_version == 1: return False - else: - return self.client.external_account_required() + return self.client.external_account_required() -class ClientNetwork(object): # pylint: disable=too-many-instance-attributes +class ClientNetwork(object): """Wrapper around requests that signs POSTs for authentication. Also adds user agent, and handles Content-Type. @@ -952,7 +942,7 @@ class ClientNetwork(object): # pylint: disable=too-many-instance-attributes :param messages.RegistrationResource account: Account object. Required if you are planning to use .post() with acme_version=2 for anything other than creating a new account; may be set later after registering. - :param josepy.JWASignature alg: Algoritm to use in signing JWS. + :param josepy.JWASignature alg: Algorithm to use in signing JWS. :param bool verify_ssl: Whether to verify certificates on SSL connections. :param str user_agent: String to send as User-Agent header. :param float timeout: Timeout for requests. @@ -962,7 +952,6 @@ class ClientNetwork(object): # pylint: disable=too-many-instance-attributes def __init__(self, key, account=None, alg=jose.RS256, verify_ssl=True, user_agent='acme-python', timeout=DEFAULT_NETWORK_TIMEOUT, source_address=None): - # pylint: disable=too-many-arguments self.key = key self.account = account self.alg = alg @@ -1011,7 +1000,6 @@ class ClientNetwork(object): # pylint: disable=too-many-instance-attributes if self.account is not None: kwargs["kid"] = self.account["uri"] kwargs["key"] = self.key - # pylint: disable=star-args return jws.JWS.sign(jobj, **kwargs).json_dumps(indent=2) @classmethod @@ -1071,7 +1059,6 @@ class ClientNetwork(object): # pylint: disable=too-many-instance-attributes return response def _send_request(self, method, url, *args, **kwargs): - # pylint: disable=too-many-locals """Send HTTP request. Makes sure that `verify_ssl` is respected. Logs request and @@ -1118,10 +1105,9 @@ class ClientNetwork(object): # pylint: disable=too-many-instance-attributes err_regex = r".*host='(\S*)'.*Max retries exceeded with url\: (\/\w*).*(\[Errno \d+\])([A-Za-z ]*)" m = re.match(err_regex, str(e)) if m is None: - raise # pragma: no cover - else: - host, path, _err_no, err_msg = m.groups() - raise ValueError("Requesting {0}{1}:{2}".format(host, path, err_msg)) + raise # pragma: no cover + host, path, _err_no, err_msg = m.groups() + raise ValueError("Requesting {0}{1}:{2}".format(host, path, err_msg)) # If content is DER, log the base64 of it instead of raw bytes, to keep # binary data out of the logs. @@ -1187,15 +1173,11 @@ class ClientNetwork(object): # pylint: disable=too-many-instance-attributes if error.code == 'badNonce': logger.debug('Retrying request after error:\n%s', error) return self._post_once(*args, **kwargs) - else: - raise + raise def _post_once(self, url, obj, content_type=JOSE_CONTENT_TYPE, acme_version=1, **kwargs): - try: - new_nonce_url = kwargs.pop('new_nonce_url') - except KeyError: - new_nonce_url = None + new_nonce_url = kwargs.pop('new_nonce_url', None) data = self._wrap_in_jws(obj, self._get_nonce(url, new_nonce_url), url, acme_version) kwargs.setdefault('headers', {'Content-Type': content_type}) response = self._send_request('POST', url, data=data, **kwargs) diff --git a/acme/acme/crypto_util.py b/acme/acme/crypto_util.py index c88cab943..66dfc738c 100644 --- a/acme/acme/crypto_util.py +++ b/acme/acme/crypto_util.py @@ -6,32 +6,29 @@ import os import re import socket -from OpenSSL import crypto -from OpenSSL import SSL # type: ignore # https://github.com/python/typeshed/issues/2052 import josepy as jose +from OpenSSL import crypto +from OpenSSL import SSL # type: ignore # https://github.com/python/typeshed/issues/2052 from acme import errors -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import Callable, Union, Tuple, Optional -# pylint: enable=unused-import, no-name-in-module - +from acme.magic_typing import Callable # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Optional # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Tuple # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Union # pylint: disable=unused-import, no-name-in-module logger = logging.getLogger(__name__) -# TLSSNI01 certificate serving and probing is not affected by SSL -# vulnerabilities: prober needs to check certificate for expected -# contents anyway. Working SNI is the only thing that's necessary for -# the challenge and thus scoping down SSL/TLS method (version) would -# cause interoperability issues: TLSv1_METHOD is only compatible with +# Default SSL method selected here is the most compatible, while secure +# SSL method: TLSv1_METHOD is only compatible with # TLSv1_METHOD, while SSLv23_METHOD is compatible with all other # methods, including TLSv2_METHOD (read more at # https://www.openssl.org/docs/ssl/SSLv23_method.html). _serve_sni # should be changed to use "set_options" to disable SSLv2 and SSLv3, # in case it's used for things other than probing/serving! -_DEFAULT_TLSSNI01_SSL_METHOD = SSL.SSLv23_METHOD # type: ignore +_DEFAULT_SSL_METHOD = SSL.SSLv23_METHOD # type: ignore -class SSLSocket(object): # pylint: disable=too-few-public-methods +class SSLSocket(object): """SSL wrapper for sockets. :ivar socket sock: Original wrapped socket. @@ -40,7 +37,7 @@ class SSLSocket(object): # pylint: disable=too-few-public-methods :ivar method: See `OpenSSL.SSL.Context` for allowed values. """ - def __init__(self, sock, certs, method=_DEFAULT_TLSSNI01_SSL_METHOD): + def __init__(self, sock, certs, method=_DEFAULT_SSL_METHOD): self.sock = sock self.certs = certs self.method = method @@ -77,7 +74,7 @@ class SSLSocket(object): # pylint: disable=too-few-public-methods class FakeConnection(object): """Fake OpenSSL.SSL.Connection.""" - # pylint: disable=too-few-public-methods,missing-docstring + # pylint: disable=missing-docstring def __init__(self, connection): self._wrapped = connection @@ -112,7 +109,7 @@ class SSLSocket(object): # pylint: disable=too-few-public-methods def probe_sni(name, host, port=443, timeout=300, - method=_DEFAULT_TLSSNI01_SSL_METHOD, source_address=('', 0)): + method=_DEFAULT_SSL_METHOD, source_address=('', 0)): """Probe SNI server for SSL certificate. :param bytes name: Byte string to send as the server name in the @@ -137,7 +134,6 @@ def probe_sni(name, host, port=443, timeout=300, socket_kwargs = {'source_address': source_address} try: - # pylint: disable=star-args logger.debug( "Attempting to connect to %s:%d%s.", host, port, " from {0}:{1}".format( @@ -198,8 +194,7 @@ def _pyopenssl_cert_or_req_all_names(loaded_cert_or_req): if common_name is None: return sans - else: - return [common_name] + [d for d in sans if d != common_name] + return [common_name] + [d for d in sans if d != common_name] def _pyopenssl_cert_or_req_san(cert_or_req): """Get Subject Alternative Names from certificate or CSR using pyOpenSSL. diff --git a/acme/acme/errors.py b/acme/acme/errors.py index 3a0f8c596..806657940 100644 --- a/acme/acme/errors.py +++ b/acme/acme/errors.py @@ -29,7 +29,12 @@ class NonceError(ClientError): class BadNonce(NonceError): """Bad nonce error.""" def __init__(self, nonce, error, *args, **kwargs): - super(BadNonce, self).__init__(*args, **kwargs) + # MyPy complains here that there is too many arguments for BaseException constructor. + # This is an error fixed in typeshed, see https://github.com/python/mypy/issues/4183 + # The fix is included in MyPy>=0.740, but upgrading it would bring dozen of errors due to + # new types definitions. So we ignore the error until the code base is fixed to match + # with MyPy>=0.740 referential. + super(BadNonce, self).__init__(*args, **kwargs) # type: ignore self.nonce = nonce self.error = error @@ -48,7 +53,8 @@ class MissingNonce(NonceError): """ def __init__(self, response, *args, **kwargs): - super(MissingNonce, self).__init__(*args, **kwargs) + # See comment in BadNonce constructor above for an explanation of type: ignore here. + super(MissingNonce, self).__init__(*args, **kwargs) # type: ignore self.response = response def __str__(self): @@ -83,6 +89,7 @@ class PollError(ClientError): return '{0}(exhausted={1!r}, updated={2!r})'.format( self.__class__.__name__, self.exhausted, self.updated) + class ValidationError(Error): """Error for authorization failures. Contains a list of authorization resources, each of which is invalid and should have an error field. @@ -91,9 +98,11 @@ class ValidationError(Error): self.failed_authzrs = failed_authzrs super(ValidationError, self).__init__() -class TimeoutError(Error): + +class TimeoutError(Error): # pylint: disable=redefined-builtin """Error for when polling an authorization or an order times out.""" + class IssuanceError(Error): """Error sent by the server after requesting issuance of a certificate.""" @@ -105,6 +114,7 @@ class IssuanceError(Error): self.error = error super(IssuanceError, self).__init__() + class ConflictError(ClientError): """Error for when the server returns a 409 (Conflict) HTTP status. diff --git a/acme/acme/fields.py b/acme/acme/fields.py index d7ec78403..3b5672283 100644 --- a/acme/acme/fields.py +++ b/acme/acme/fields.py @@ -4,7 +4,6 @@ import logging import josepy as jose import pyrfc3339 - logger = logging.getLogger(__name__) diff --git a/acme/acme/jws.py b/acme/acme/jws.py index c92d226d4..894e69f3d 100644 --- a/acme/acme/jws.py +++ b/acme/acme/jws.py @@ -40,10 +40,10 @@ class Signature(jose.Signature): class JWS(jose.JWS): """ACME-specific JWS. Includes none, url, and kid in protected header.""" signature_cls = Signature - __slots__ = jose.JWS._orig_slots # pylint: disable=no-member + __slots__ = jose.JWS._orig_slots @classmethod - # pylint: disable=arguments-differ,too-many-arguments + # pylint: disable=arguments-differ def sign(cls, payload, key, alg, nonce, url=None, kid=None): # Per ACME spec, jwk and kid are mutually exclusive, so only include a # jwk field if kid is not provided. diff --git a/acme/acme/magic_typing.py b/acme/acme/magic_typing.py index 471b8dfa9..5a6358c69 100644 --- a/acme/acme/magic_typing.py +++ b/acme/acme/magic_typing.py @@ -1,6 +1,7 @@ """Shim class to not have to depend on typing module in prod.""" import sys + class TypingClass(object): """Ignore import errors by getting anything""" def __getattr__(self, name): diff --git a/acme/acme/messages.py b/acme/acme/messages.py index 7c82c8507..e82d12890 100644 --- a/acme/acme/messages.py +++ b/acme/acme/messages.py @@ -1,37 +1,55 @@ """ACME protocol messages.""" -import six import json -try: - from collections.abc import Hashable # pylint: disable=no-name-in-module -except ImportError: - from collections import Hashable import josepy as jose +import six from acme import challenges from acme import errors from acme import fields -from acme import util from acme import jws +from acme import util + +try: + from collections.abc import Hashable # pylint: disable=no-name-in-module +except ImportError: # pragma: no cover + from collections import Hashable + + OLD_ERROR_PREFIX = "urn:acme:error:" ERROR_PREFIX = "urn:ietf:params:acme:error:" ERROR_CODES = { + 'accountDoesNotExist': 'The request specified an account that does not exist', + 'alreadyRevoked': 'The request specified a certificate to be revoked that has' \ + ' already been revoked', 'badCSR': 'The CSR is unacceptable (e.g., due to a short key)', 'badNonce': 'The client sent an unacceptable anti-replay nonce', + 'badPublicKey': 'The JWS was signed by a public key the server does not support', + 'badRevocationReason': 'The revocation reason provided is not allowed by the server', + 'badSignatureAlgorithm': 'The JWS was signed with an algorithm the server does not support', + 'caa': 'Certification Authority Authorization (CAA) records forbid the CA from issuing' \ + ' a certificate', + 'compound': 'Specific error conditions are indicated in the "subproblems" array', 'connection': ('The server could not connect to the client to verify the' ' domain'), + 'dns': 'There was a problem with a DNS query during identifier validation', 'dnssec': 'The server could not validate a DNSSEC signed domain', + 'incorrectResponse': 'Response received didn\'t match the challenge\'s requirements', # deprecate invalidEmail 'invalidEmail': 'The provided email for a registration was invalid', 'invalidContact': 'The provided contact URI was invalid', 'malformed': 'The request message was malformed', + 'rejectedIdentifier': 'The server will not issue certificates for the identifier', + 'orderNotReady': 'The request attempted to finalize an order that is not ready to be finalized', 'rateLimited': 'There were too many requests of a given type', 'serverInternal': 'The server experienced an internal error', 'tls': 'The server experienced a TLS error during domain verification', 'unauthorized': 'The client lacks sufficient authorization', + 'unsupportedContact': 'A contact URL for an account used an unsupported protocol scheme', 'unknownHost': 'The server could not resolve a domain name', + 'unsupportedIdentifier': 'An identifier is of an unsupported type', 'externalAccountRequired': 'The server requires external account binding', } @@ -46,8 +64,7 @@ def is_acme_error(err): """Check if argument is an ACME error.""" if isinstance(err, Error) and (err.typ is not None): return (ERROR_PREFIX in err.typ) or (OLD_ERROR_PREFIX in err.typ) - else: - return False + return False @six.python_2_unicode_compatible @@ -102,6 +119,7 @@ class Error(jose.JSONObjectWithFields, errors.Error): code = str(self.typ).split(':')[-1] if code in ERROR_CODES: return code + return None def __str__(self): return b' :: '.join( @@ -116,18 +134,19 @@ class _Constant(jose.JSONDeSerializable, Hashable): # type: ignore POSSIBLE_NAMES = NotImplemented def __init__(self, name): - self.POSSIBLE_NAMES[name] = self + super(_Constant, self).__init__() + self.POSSIBLE_NAMES[name] = self # pylint: disable=unsupported-assignment-operation self.name = name def to_partial_json(self): return self.name @classmethod - def from_json(cls, value): - if value not in cls.POSSIBLE_NAMES: + def from_json(cls, jobj): + if jobj not in cls.POSSIBLE_NAMES: # pylint: disable=unsupported-membership-test raise jose.DeserializationError( '{0} not recognized'.format(cls.__name__)) - return cls.POSSIBLE_NAMES[value] + return cls.POSSIBLE_NAMES[jobj] def __repr__(self): return '{0}({1})'.format(self.__class__.__name__, self.name) @@ -152,6 +171,7 @@ STATUS_VALID = Status('valid') STATUS_INVALID = Status('invalid') STATUS_REVOKED = Status('revoked') STATUS_READY = Status('ready') +STATUS_DEACTIVATED = Status('deactivated') class IdentifierType(_Constant): @@ -186,7 +206,6 @@ class Directory(jose.JSONDeSerializable): def __init__(self, **kwargs): kwargs = dict((self._internal_name(k), v) for k, v in kwargs.items()) - # pylint: disable=star-args super(Directory.Meta, self).__init__(**kwargs) @property @@ -226,13 +245,13 @@ class Directory(jose.JSONDeSerializable): try: return self[name.replace('_', '-')] except KeyError as error: - raise AttributeError(str(error) + ': ' + name) + raise AttributeError(str(error)) def __getitem__(self, name): try: return self._jobj[self._canon_key(name)] except KeyError: - raise KeyError('Directory field not found') + raise KeyError('Directory field "' + self._canon_key(name) + '" not found') def to_partial_json(self): return self._jobj @@ -322,7 +341,7 @@ class Registration(ResourceBody): def _filter_contact(self, prefix): return tuple( - detail[len(prefix):] for detail in self.contact + detail[len(prefix):] for detail in self.contact # pylint: disable=not-an-iterable if detail.startswith(prefix)) @property @@ -394,7 +413,6 @@ class ChallengeBody(ResourceBody): def __init__(self, **kwargs): kwargs = dict((self._internal_name(k), v) for k, v in kwargs.items()) - # pylint: disable=star-args super(ChallengeBody, self).__init__(**kwargs) def encode(self, name): @@ -457,7 +475,7 @@ class Authorization(ResourceBody): :ivar datetime.datetime expires: """ - identifier = jose.Field('identifier', decoder=Identifier.from_json) + identifier = jose.Field('identifier', decoder=Identifier.from_json, omitempty=True) challenges = jose.Field('challenges', omitempty=True) combinations = jose.Field('combinations', omitempty=True) @@ -477,7 +495,7 @@ class Authorization(ResourceBody): def resolved_combinations(self): """Combinations with challenges instead of indices.""" return tuple(tuple(self.challenges[idx] for idx in combo) - for combo in self.combinations) + for combo in self.combinations) # pylint: disable=not-an-iterable @Directory.register @@ -487,6 +505,12 @@ class NewAuthorization(Authorization): resource = fields.Resource(resource_type) +class UpdateAuthorization(Authorization): + """Update authorization.""" + resource_type = 'authz' + resource = fields.Resource(resource_type) + + class AuthorizationResource(ResourceWithURI): """Authorization Resource. diff --git a/acme/acme/standalone.py b/acme/acme/standalone.py index ff9159933..cf0da4e86 100644 --- a/acme/acme/standalone.py +++ b/acme/acme/standalone.py @@ -1,28 +1,22 @@ """Support for standalone client challenge solvers. """ -import argparse import collections import functools import logging -import os import socket -import sys import threading from six.moves import BaseHTTPServer # type: ignore # pylint: disable=import-error from six.moves import http_client # pylint: disable=import-error from six.moves import socketserver # type: ignore # pylint: disable=import-error -import OpenSSL - from acme import challenges from acme import crypto_util -from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module - +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module logger = logging.getLogger(__name__) # six.moves.* | pylint: disable=no-member,attribute-defined-outside-init -# pylint: disable=too-few-public-methods,no-init +# pylint: disable=no-init class TLSServer(socketserver.TCPServer): @@ -37,7 +31,7 @@ class TLSServer(socketserver.TCPServer): self.certs = kwargs.pop("certs", {}) self.method = kwargs.pop( # pylint: disable=protected-access - "method", crypto_util._DEFAULT_TLSSNI01_SSL_METHOD) + "method", crypto_util._DEFAULT_SSL_METHOD) self.allow_reuse_address = kwargs.pop("allow_reuse_address", True) socketserver.TCPServer.__init__(self, *args, **kwargs) @@ -50,7 +44,7 @@ class TLSServer(socketserver.TCPServer): return socketserver.TCPServer.server_bind(self) -class ACMEServerMixin: # pylint: disable=old-style-class +class ACMEServerMixin: """ACME server common settings mixin.""" # TODO: c.f. #858 server_version = "ACME client standalone challenge solver" @@ -82,7 +76,7 @@ class BaseDualNetworkedServers(object): kwargs["ipv6"] = ip_version new_address = (server_address[0],) + (port,) + server_address[2:] new_args = (new_address,) + remaining_args - server = ServerClass(*new_args, **kwargs) # pylint: disable=star-args + server = ServerClass(*new_args, **kwargs) logger.debug( "Successfully bound to %s:%s using %s", new_address[0], new_address[1], "IPv6" if ip_version else "IPv4") @@ -90,8 +84,8 @@ class BaseDualNetworkedServers(object): if self.servers: # Already bound using IPv6. logger.debug( - "Certbot wasn't able to bind to %s:%s using %s, this " + - "is often expected due to the dual stack nature of " + + "Certbot wasn't able to bind to %s:%s using %s, this " + "is often expected due to the dual stack nature of " "IPv6 socket implementations.", new_address[0], new_address[1], "IPv6" if ip_version else "IPv4") @@ -104,14 +98,13 @@ class BaseDualNetworkedServers(object): # If two servers are set up and port 0 was passed in, ensure we always # bind to the same port for both servers. port = server.socket.getsockname()[1] - if len(self.servers) == 0: + if not self.servers: raise socket.error("Could not bind to IPv4 or IPv6.") def serve_forever(self): """Wraps socketserver.TCPServer.serve_forever""" for server in self.servers: thread = threading.Thread( - # pylint: disable=no-member target=server.serve_forever) thread.start() self.threads.append(thread) @@ -131,35 +124,6 @@ class BaseDualNetworkedServers(object): self.threads = [] -class TLSSNI01Server(TLSServer, ACMEServerMixin): - """TLSSNI01 Server.""" - - def __init__(self, server_address, certs, ipv6=False): - TLSServer.__init__( - self, server_address, BaseRequestHandlerWithLogging, certs=certs, ipv6=ipv6) - - -class TLSSNI01DualNetworkedServers(BaseDualNetworkedServers): - """TLSSNI01Server Wrapper. Tries everything for both. Failures for one don't - affect the other.""" - - def __init__(self, *args, **kwargs): - BaseDualNetworkedServers.__init__(self, TLSSNI01Server, *args, **kwargs) - - -class BaseRequestHandlerWithLogging(socketserver.BaseRequestHandler): - """BaseRequestHandler with logging.""" - - def log_message(self, format, *args): # pylint: disable=redefined-builtin - """Log arbitrary message.""" - logger.debug("%s - - %s", self.client_address[0], format % args) - - def handle(self): - """Handle request.""" - self.log_message("Incoming request") - socketserver.BaseRequestHandler.handle(self) - - class HTTPServer(BaseHTTPServer.HTTPServer): """Generic HTTP Server.""" @@ -262,39 +226,3 @@ class HTTP01RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler): """ return functools.partial( cls, simple_http_resources=simple_http_resources) - - -def simple_tls_sni_01_server(cli_args, forever=True): - """Run simple standalone TLSSNI01 server.""" - logging.basicConfig(level=logging.DEBUG) - - parser = argparse.ArgumentParser() - parser.add_argument( - "-p", "--port", default=0, help="Port to serve at. By default " - "picks random free port.") - args = parser.parse_args(cli_args[1:]) - - certs = {} - - _, hosts, _ = next(os.walk('.')) # type: ignore # https://github.com/python/mypy/issues/465 - for host in hosts: - with open(os.path.join(host, "cert.pem")) as cert_file: - cert_contents = cert_file.read() - with open(os.path.join(host, "key.pem")) as key_file: - key_contents = key_file.read() - certs[host.encode()] = ( - OpenSSL.crypto.load_privatekey( - OpenSSL.crypto.FILETYPE_PEM, key_contents), - OpenSSL.crypto.load_certificate( - OpenSSL.crypto.FILETYPE_PEM, cert_contents)) - - server = TLSSNI01Server(('', int(args.port)), certs=certs) - logger.info("Serving at https://%s:%s...", *server.socket.getsockname()[:2]) - if forever: # pragma: no cover - server.serve_forever() - else: - server.handle_request() - - -if __name__ == "__main__": - sys.exit(simple_tls_sni_01_server(sys.argv)) # pragma: no cover diff --git a/acme/docs/conf.py b/acme/docs/conf.py index e70651648..8c1689128 100644 --- a/acme/docs/conf.py +++ b/acme/docs/conf.py @@ -12,10 +12,9 @@ # All configuration values have a default; values that are commented out # serve to show the default. -import sys import os import shlex - +import sys here = os.path.abspath(os.path.dirname(__file__)) @@ -42,7 +41,7 @@ extensions = [ ] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/acme/examples/http01_example.py b/acme/examples/http01_example.py new file mode 100644 index 000000000..2dc197d09 --- /dev/null +++ b/acme/examples/http01_example.py @@ -0,0 +1,241 @@ +"""Example ACME-V2 API for HTTP-01 challenge. + +Brief: + +This a complete usage example of the python-acme API. + +Limitations of this example: + - Works for only one Domain name + - Performs only HTTP-01 challenge + - Uses ACME-v2 + +Workflow: + (Account creation) + - Create account key + - Register account and accept TOS + (Certificate actions) + - Select HTTP-01 within offered challenges by the CA server + - Set up http challenge resource + - Set up standalone web server + - Create domain private key and CSR + - Issue certificate + - Renew certificate + - Revoke certificate + (Account update actions) + - Change contact information + - Deactivate Account +""" +from contextlib import contextmanager + +from cryptography.hazmat.backends import default_backend +from cryptography.hazmat.primitives.asymmetric import rsa +import josepy as jose +import OpenSSL + +from acme import challenges +from acme import client +from acme import crypto_util +from acme import errors +from acme import messages +from acme import standalone + +# Constants: + +# This is the staging point for ACME-V2 within Let's Encrypt. +DIRECTORY_URL = 'https://acme-staging-v02.api.letsencrypt.org/directory' + +USER_AGENT = 'python-acme-example' + +# Account key size +ACC_KEY_BITS = 2048 + +# Certificate private key size +CERT_PKEY_BITS = 2048 + +# Domain name for the certificate. +DOMAIN = 'client.example.com' + +# If you are running Boulder locally, it is possible to configure any port +# number to execute the challenge, but real CA servers will always use port +# 80, as described in the ACME specification. +PORT = 80 + + +# Useful methods and classes: + + +def new_csr_comp(domain_name, pkey_pem=None): + """Create certificate signing request.""" + if pkey_pem is None: + # Create private key. + pkey = OpenSSL.crypto.PKey() + pkey.generate_key(OpenSSL.crypto.TYPE_RSA, CERT_PKEY_BITS) + pkey_pem = OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, + pkey) + csr_pem = crypto_util.make_csr(pkey_pem, [domain_name]) + return pkey_pem, csr_pem + + +def select_http01_chall(orderr): + """Extract authorization resource from within order resource.""" + # Authorization Resource: authz. + # This object holds the offered challenges by the server and their status. + authz_list = orderr.authorizations + + for authz in authz_list: + # Choosing challenge. + # authz.body.challenges is a set of ChallengeBody objects. + for i in authz.body.challenges: + # Find the supported challenge. + if isinstance(i.chall, challenges.HTTP01): + return i + + raise Exception('HTTP-01 challenge was not offered by the CA server.') + + +@contextmanager +def challenge_server(http_01_resources): + """Manage standalone server set up and shutdown.""" + + # Setting up a fake server that binds at PORT and any address. + address = ('', PORT) + try: + servers = standalone.HTTP01DualNetworkedServers(address, + http_01_resources) + # Start client standalone web server. + servers.serve_forever() + yield servers + finally: + # Shutdown client web server and unbind from PORT + servers.shutdown_and_server_close() + + +def perform_http01(client_acme, challb, orderr): + """Set up standalone webserver and perform HTTP-01 challenge.""" + + response, validation = challb.response_and_validation(client_acme.net.key) + + resource = standalone.HTTP01RequestHandler.HTTP01Resource( + chall=challb.chall, response=response, validation=validation) + + with challenge_server({resource}): + # Let the CA server know that we are ready for the challenge. + client_acme.answer_challenge(challb, response) + + # Wait for challenge status and then issue a certificate. + # It is possible to set a deadline time. + finalized_orderr = client_acme.poll_and_finalize(orderr) + + return finalized_orderr.fullchain_pem + + +# Main examples: + + +def example_http(): + """This example executes the whole process of fulfilling a HTTP-01 + challenge for one specific domain. + + The workflow consists of: + (Account creation) + - Create account key + - Register account and accept TOS + (Certificate actions) + - Select HTTP-01 within offered challenges by the CA server + - Set up http challenge resource + - Set up standalone web server + - Create domain private key and CSR + - Issue certificate + - Renew certificate + - Revoke certificate + (Account update actions) + - Change contact information + - Deactivate Account + + """ + # Create account key + + acc_key = jose.JWKRSA( + key=rsa.generate_private_key(public_exponent=65537, + key_size=ACC_KEY_BITS, + backend=default_backend())) + + # Register account and accept TOS + + net = client.ClientNetwork(acc_key, user_agent=USER_AGENT) + directory = messages.Directory.from_json(net.get(DIRECTORY_URL).json()) + client_acme = client.ClientV2(directory, net=net) + + # Terms of Service URL is in client_acme.directory.meta.terms_of_service + # Registration Resource: regr + # Creates account with contact information. + email = ('fake@example.com') + regr = client_acme.new_account( + messages.NewRegistration.from_data( + email=email, terms_of_service_agreed=True)) + + # Create domain private key and CSR + pkey_pem, csr_pem = new_csr_comp(DOMAIN) + + # Issue certificate + + orderr = client_acme.new_order(csr_pem) + + # Select HTTP-01 within offered challenges by the CA server + challb = select_http01_chall(orderr) + + # The certificate is ready to be used in the variable "fullchain_pem". + fullchain_pem = perform_http01(client_acme, challb, orderr) + + # Renew certificate + + _, csr_pem = new_csr_comp(DOMAIN, pkey_pem) + + orderr = client_acme.new_order(csr_pem) + + challb = select_http01_chall(orderr) + + # Performing challenge + fullchain_pem = perform_http01(client_acme, challb, orderr) + + # Revoke certificate + + fullchain_com = jose.ComparableX509( + OpenSSL.crypto.load_certificate( + OpenSSL.crypto.FILETYPE_PEM, fullchain_pem)) + + try: + client_acme.revoke(fullchain_com, 0) # revocation reason = 0 + except errors.ConflictError: + # Certificate already revoked. + pass + + # Query registration status. + client_acme.net.account = regr + try: + regr = client_acme.query_registration(regr) + except errors.Error as err: + if err.typ == messages.OLD_ERROR_PREFIX + 'unauthorized' \ + or err.typ == messages.ERROR_PREFIX + 'unauthorized': + # Status is deactivated. + pass + raise + + # Change contact information + + email = 'newfake@example.com' + regr = client_acme.update_registration( + regr.update( + body=regr.body.update( + contact=('mailto:' + email,) + ) + ) + ) + + # Deactivate account/registration + + regr = client_acme.deactivate_registration(regr) + + +if __name__ == "__main__": + example_http() diff --git a/acme/readthedocs.org.requirements.txt b/acme/readthedocs.org.requirements.txt index 65e6c7cf3..168af8013 100644 --- a/acme/readthedocs.org.requirements.txt +++ b/acme/readthedocs.org.requirements.txt @@ -1,10 +1,10 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e acme[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install acme[docs]" does not work as +# expected and "pip install -e acme[docs]" must be used instead -e acme[docs] diff --git a/acme/setup.py b/acme/setup.py index eac3974fa..0e11779ba 100644 --- a/acme/setup.py +++ b/acme/setup.py @@ -1,9 +1,10 @@ -from setuptools import setup -from setuptools import find_packages -from setuptools.command.test import test as TestCommand import sys -version = '0.31.0.dev0' +from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand + +version = '1.3.0.dev0' # Please update tox.ini when modifying dependency version requirements install_requires = [ @@ -11,9 +12,11 @@ install_requires = [ # rsa_recover_prime_factors (>=0.8) 'cryptography>=1.2.3', # formerly known as acme.jose: - 'josepy>=1.0.0', - # Connection.set_tlsext_host_name (>=0.13) + # 1.1.0+ is required to avoid the warnings described at + # https://github.com/certbot/josepy/issues/13. + 'josepy>=1.1.0', 'mock', + # Connection.set_tlsext_host_name (>=0.13) 'PyOpenSSL>=0.13.1', 'pyrfc3339', 'pytz', @@ -34,6 +37,7 @@ docs_extras = [ 'sphinx_rtd_theme', ] + class PyTest(TestCommand): user_options = [] @@ -48,6 +52,7 @@ class PyTest(TestCommand): errno = pytest.main(shlex.split(self.pytest_args)) sys.exit(errno) + setup( name='acme', version=version, @@ -56,7 +61,7 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', @@ -65,10 +70,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', ], @@ -80,7 +85,7 @@ setup( 'dev': dev_extras, 'docs': docs_extras, }, - tests_require=["pytest"], test_suite='acme', + tests_require=["pytest"], cmdclass={"test": PyTest}, ) diff --git a/acme/acme/challenges_test.py b/acme/tests/challenges_test.py similarity index 69% rename from acme/acme/challenges_test.py rename to acme/tests/challenges_test.py index be15e5b1a..adebaffc5 100644 --- a/acme/acme/challenges_test.py +++ b/acme/tests/challenges_test.py @@ -1,16 +1,12 @@ """Tests for acme.challenges.""" import unittest -import warnings import josepy as jose import mock -import OpenSSL import requests +from six.moves.urllib import parse as urllib_parse -from six.moves.urllib import parse as urllib_parse # pylint: disable=import-error - -from acme import errors -from acme import test_util +import test_util CERT = test_util.load_comparable_cert('cert.pem') KEY = jose.JWKRSA(key=test_util.load_rsa_private_key('rsa512_key.pem')) @@ -22,7 +18,6 @@ class ChallengeTest(unittest.TestCase): from acme.challenges import Challenge from acme.challenges import UnrecognizedChallenge chall = UnrecognizedChallenge({"type": "foo"}) - # pylint: disable=no-member self.assertEqual(chall, Challenge.from_json(chall.jobj)) @@ -78,7 +73,6 @@ class KeyAuthorizationChallengeResponseTest(unittest.TestCase): class DNS01ResponseTest(unittest.TestCase): - # pylint: disable=too-many-instance-attributes def setUp(self): from acme.challenges import DNS01Response @@ -94,7 +88,8 @@ class DNS01ResponseTest(unittest.TestCase): self.response = self.chall.response(KEY) def test_to_partial_json(self): - self.assertEqual(self.jmsg, self.msg.to_partial_json()) + self.assertEqual({k: v for k, v in self.jmsg.items() if k != 'keyAuthorization'}, + self.msg.to_partial_json()) def test_from_json(self): from acme.challenges import DNS01Response @@ -149,7 +144,6 @@ class DNS01Test(unittest.TestCase): class HTTP01ResponseTest(unittest.TestCase): - # pylint: disable=too-many-instance-attributes def setUp(self): from acme.challenges import HTTP01Response @@ -165,7 +159,8 @@ class HTTP01ResponseTest(unittest.TestCase): self.response = self.chall.response(KEY) def test_to_partial_json(self): - self.assertEqual(self.jmsg, self.msg.to_partial_json()) + self.assertEqual({k: v for k, v in self.jmsg.items() if k != 'keyAuthorization'}, + self.msg.to_partial_json()) def test_from_json(self): from acme.challenges import HTTP01Response @@ -258,152 +253,7 @@ class HTTP01Test(unittest.TestCase): self.msg.update(token=b'..').good_token) -class TLSSNI01ResponseTest(unittest.TestCase): - # pylint: disable=too-many-instance-attributes - - def setUp(self): - from acme.challenges import TLSSNI01 - self.chall = TLSSNI01( - token=jose.b64decode(b'a82d5ff8ef740d12881f6d3c2277ab2e')) - - self.response = self.chall.response(KEY) - self.jmsg = { - 'resource': 'challenge', - 'type': 'tls-sni-01', - 'keyAuthorization': self.response.key_authorization, - } - - # pylint: disable=invalid-name - label1 = b'dc38d9c3fa1a4fdcc3a5501f2d38583f' - label2 = b'b7793728f084394f2a1afd459556bb5c' - self.z = label1 + label2 - self.z_domain = label1 + b'.' + label2 + b'.acme.invalid' - self.domain = 'foo.com' - - def test_z_and_domain(self): - self.assertEqual(self.z, self.response.z) - self.assertEqual(self.z_domain, self.response.z_domain) - - def test_to_partial_json(self): - self.assertEqual(self.jmsg, self.response.to_partial_json()) - - def test_from_json(self): - from acme.challenges import TLSSNI01Response - self.assertEqual(self.response, TLSSNI01Response.from_json(self.jmsg)) - - def test_from_json_hashable(self): - from acme.challenges import TLSSNI01Response - hash(TLSSNI01Response.from_json(self.jmsg)) - - @mock.patch('acme.challenges.socket.gethostbyname') - @mock.patch('acme.challenges.crypto_util.probe_sni') - def test_probe_cert(self, mock_probe_sni, mock_gethostbyname): - mock_gethostbyname.return_value = '127.0.0.1' - self.response.probe_cert('foo.com') - mock_gethostbyname.assert_called_once_with('foo.com') - mock_probe_sni.assert_called_once_with( - host='127.0.0.1', port=self.response.PORT, - name=self.z_domain) - - self.response.probe_cert('foo.com', host='8.8.8.8') - mock_probe_sni.assert_called_with( - host='8.8.8.8', port=mock.ANY, name=mock.ANY) - - self.response.probe_cert('foo.com', port=1234) - mock_probe_sni.assert_called_with( - host=mock.ANY, port=1234, name=mock.ANY) - - self.response.probe_cert('foo.com', bar='baz') - mock_probe_sni.assert_called_with( - host=mock.ANY, port=mock.ANY, name=mock.ANY, bar='baz') - - self.response.probe_cert('foo.com', name=b'xxx') - mock_probe_sni.assert_called_with( - host=mock.ANY, port=mock.ANY, - name=self.z_domain) - - def test_gen_verify_cert(self): - key1 = test_util.load_pyopenssl_private_key('rsa512_key.pem') - cert, key2 = self.response.gen_cert(key1) - self.assertEqual(key1, key2) - self.assertTrue(self.response.verify_cert(cert)) - - def test_gen_verify_cert_gen_key(self): - cert, key = self.response.gen_cert() - self.assertTrue(isinstance(key, OpenSSL.crypto.PKey)) - self.assertTrue(self.response.verify_cert(cert)) - - def test_verify_bad_cert(self): - self.assertFalse(self.response.verify_cert( - test_util.load_cert('cert.pem'))) - - def test_simple_verify_bad_key_authorization(self): - key2 = jose.JWKRSA.load(test_util.load_vector('rsa256_key.pem')) - self.response.simple_verify(self.chall, "local", key2.public_key()) - - @mock.patch('acme.challenges.TLSSNI01Response.verify_cert', autospec=True) - def test_simple_verify(self, mock_verify_cert): - mock_verify_cert.return_value = mock.sentinel.verification - self.assertEqual( - mock.sentinel.verification, self.response.simple_verify( - self.chall, self.domain, KEY.public_key(), - cert=mock.sentinel.cert)) - mock_verify_cert.assert_called_once_with( - self.response, mock.sentinel.cert) - - @mock.patch('acme.challenges.TLSSNI01Response.probe_cert') - def test_simple_verify_false_on_probe_error(self, mock_probe_cert): - mock_probe_cert.side_effect = errors.Error - self.assertFalse(self.response.simple_verify( - self.chall, self.domain, KEY.public_key())) - - -class TLSSNI01Test(unittest.TestCase): - - def setUp(self): - self.jmsg = { - 'type': 'tls-sni-01', - 'token': 'a82d5ff8ef740d12881f6d3c2277ab2e', - } - - def _msg(self): - from acme.challenges import TLSSNI01 - with warnings.catch_warnings(record=True) as warn: - warnings.simplefilter("always") - msg = TLSSNI01( - token=jose.b64decode('a82d5ff8ef740d12881f6d3c2277ab2e')) - assert warn is not None # using a raw assert for mypy - self.assertTrue(len(warn) == 1) - self.assertTrue(issubclass(warn[-1].category, DeprecationWarning)) - self.assertTrue('deprecated' in str(warn[-1].message)) - return msg - - def test_to_partial_json(self): - self.assertEqual(self.jmsg, self._msg().to_partial_json()) - - def test_from_json(self): - from acme.challenges import TLSSNI01 - self.assertEqual(self._msg(), TLSSNI01.from_json(self.jmsg)) - - def test_from_json_hashable(self): - from acme.challenges import TLSSNI01 - hash(TLSSNI01.from_json(self.jmsg)) - - def test_from_json_invalid_token_length(self): - from acme.challenges import TLSSNI01 - self.jmsg['token'] = jose.encode_b64jose(b'abcd') - self.assertRaises( - jose.DeserializationError, TLSSNI01.from_json, self.jmsg) - - @mock.patch('acme.challenges.TLSSNI01Response.gen_cert') - def test_validation(self, mock_gen_cert): - mock_gen_cert.return_value = ('cert', 'key') - self.assertEqual(('cert', 'key'), self._msg().validation( - KEY, cert_key=mock.sentinel.cert_key)) - mock_gen_cert.assert_called_once_with(key=mock.sentinel.cert_key) - class TLSALPN01ResponseTest(unittest.TestCase): - # pylint: disable=too-many-instance-attributes def setUp(self): from acme.challenges import TLSALPN01Response @@ -419,7 +269,8 @@ class TLSALPN01ResponseTest(unittest.TestCase): self.response = self.chall.response(KEY) def test_to_partial_json(self): - self.assertEqual(self.jmsg, self.msg.to_partial_json()) + self.assertEqual({k: v for k, v in self.jmsg.items() if k != 'keyAuthorization'}, + self.msg.to_partial_json()) def test_from_json(self): from acme.challenges import TLSALPN01Response diff --git a/acme/acme/client_test.py b/acme/tests/client_test.py similarity index 97% rename from acme/acme/client_test.py rename to acme/tests/client_test.py index b3d0f1921..a38fedbd6 100644 --- a/acme/acme/client_test.py +++ b/acme/tests/client_test.py @@ -5,21 +5,19 @@ import datetime import json import unittest -from six.moves import http_client # pylint: disable=import-error - import josepy as jose import mock import OpenSSL import requests +from six.moves import http_client # pylint: disable=import-error from acme import challenges from acme import errors from acme import jws as acme_jws from acme import messages -from acme import messages_test -from acme import test_util -from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module - +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +import messages_test +import test_util CERT_DER = test_util.load_vector('cert.der') CERT_SAN_PEM = test_util.load_vector('cert-san.pem') @@ -63,8 +61,8 @@ class ClientTestBase(unittest.TestCase): self.contact = ('mailto:cert-admin@example.com', 'tel:+12025551212') reg = messages.Registration( contact=self.contact, key=KEY.public_key()) - the_arg = dict(reg) # type: Dict - self.new_reg = messages.NewRegistration(**the_arg) # pylint: disable=star-args + the_arg = dict(reg) # type: Dict + self.new_reg = messages.NewRegistration(**the_arg) self.regr = messages.RegistrationResource( body=reg, uri='https://www.letsencrypt-demo.org/acme/reg/1') @@ -318,7 +316,6 @@ class BackwardsCompatibleClientV2Test(ClientTestBase): class ClientTest(ClientTestBase): """Tests for acme.client.Client.""" - # pylint: disable=too-many-instance-attributes,too-many-public-methods def setUp(self): super(ClientTest, self).setUp() @@ -358,7 +355,6 @@ class ClientTest(ClientTestBase): def test_register(self): # "Instance of 'Field' has no to_json/update member" bug: - # pylint: disable=no-member self.response.status_code = http_client.CREATED self.response.json.return_value = self.regr.body.to_json() self.response.headers['Location'] = self.regr.uri @@ -371,7 +367,6 @@ class ClientTest(ClientTestBase): def test_update_registration(self): # "Instance of 'Field' has no to_json/update member" bug: - # pylint: disable=no-member self.response.headers['Location'] = self.regr.uri self.response.json.return_value = self.regr.body.to_json() self.assertEqual(self.regr, self.client.update_registration(self.regr)) @@ -639,6 +634,14 @@ class ClientTest(ClientTestBase): errors.PollError, self.client.poll_and_request_issuance, csr, authzrs, mintime=mintime, max_attempts=2) + def test_deactivate_authorization(self): + authzb = self.authzr.body.update(status=messages.STATUS_DEACTIVATED) + self.response.json.return_value = authzb.to_json() + authzr = self.client.deactivate_authorization(self.authzr) + self.assertEqual(authzb, authzr.body) + self.assertEqual(self.client.net.post.call_count, 1) + self.assertTrue(self.authzr.uri in self.net.post.call_args_list[0][0]) + def test_check_cert(self): self.response.headers['Location'] = self.certr.uri self.response.content = CERT_DER @@ -844,7 +847,6 @@ class ClientV2Test(ClientTestBase): def test_update_registration(self): # "Instance of 'Field' has no to_json/update member" bug: - # pylint: disable=no-member self.response.headers['Location'] = self.regr.uri self.response.json.return_value = self.regr.body.to_json() self.assertEqual(self.regr, self.client.update_registration(self.regr)) @@ -883,19 +885,6 @@ class ClientV2Test(ClientTestBase): new_nonce_url='https://www.letsencrypt-demo.org/acme/new-nonce') self.client.net.get.assert_not_called() - class FakeError(messages.Error): # pylint: disable=too-many-ancestors - """Fake error to reproduce a malformed request ACME error""" - def __init__(self): # pylint: disable=super-init-not-called - pass - @property - def code(self): - return 'malformed' - self.client.net.post.side_effect = FakeError() - - self.client.poll(self.authzr2) # pylint: disable=protected-access - - self.client.net.get.assert_called_once_with(self.authzr2.uri) - class MockJSONDeSerializable(jose.JSONDeSerializable): # pylint: disable=missing-docstring @@ -906,13 +895,12 @@ class MockJSONDeSerializable(jose.JSONDeSerializable): return {'foo': self.value} @classmethod - def from_json(cls, value): + def from_json(cls, jobj): pass # pragma: no cover class ClientNetworkTest(unittest.TestCase): """Tests for acme.client.ClientNetwork.""" - # pylint: disable=too-many-public-methods def setUp(self): self.verify_ssl = mock.MagicMock() @@ -962,8 +950,8 @@ class ClientNetworkTest(unittest.TestCase): def test_check_response_not_ok_jobj_error(self): self.response.ok = False - self.response.json.return_value = messages.Error( - detail='foo', typ='serverInternal', title='some title').to_json() + self.response.json.return_value = messages.Error.with_code( + 'serverInternal', detail='foo', title='some title').to_json() # pylint: disable=protected-access self.assertRaises( messages.Error, self.net._check_response, self.response) @@ -988,7 +976,7 @@ class ClientNetworkTest(unittest.TestCase): self.response.json.side_effect = ValueError for response_ct in [self.net.JSON_CONTENT_TYPE, 'foo']: self.response.headers['Content-Type'] = response_ct - # pylint: disable=protected-access,no-value-for-parameter + # pylint: disable=protected-access self.assertEqual( self.response, self.net._check_response(self.response)) @@ -1002,7 +990,7 @@ class ClientNetworkTest(unittest.TestCase): self.response.json.return_value = {} for response_ct in [self.net.JSON_CONTENT_TYPE, 'foo']: self.response.headers['Content-Type'] = response_ct - # pylint: disable=protected-access,no-value-for-parameter + # pylint: disable=protected-access self.assertEqual( self.response, self.net._check_response(self.response)) @@ -1118,7 +1106,6 @@ class ClientNetworkTest(unittest.TestCase): class ClientNetworkWithMockedResponseTest(unittest.TestCase): """Tests for acme.client.ClientNetwork which mock out response.""" - # pylint: disable=too-many-instance-attributes def setUp(self): from acme.client import ClientNetwork @@ -1128,8 +1115,8 @@ class ClientNetworkWithMockedResponseTest(unittest.TestCase): self.response.headers = {} self.response.links = {} self.response.checked = False - self.acmev1_nonce_response = mock.MagicMock(ok=False, - status_code=http_client.METHOD_NOT_ALLOWED) + self.acmev1_nonce_response = mock.MagicMock( + ok=False, status_code=http_client.METHOD_NOT_ALLOWED) self.acmev1_nonce_response.headers = {} self.obj = mock.MagicMock() self.wrapped_obj = mock.MagicMock() diff --git a/acme/acme/crypto_util_test.py b/acme/tests/crypto_util_test.py similarity index 96% rename from acme/acme/crypto_util_test.py rename to acme/tests/crypto_util_test.py index 44b245bbe..41640ed60 100644 --- a/acme/acme/crypto_util_test.py +++ b/acme/tests/crypto_util_test.py @@ -5,15 +5,14 @@ import threading import time import unittest -import six -from six.moves import socketserver #type: ignore # pylint: disable=import-error - import josepy as jose import OpenSSL +import six +from six.moves import socketserver # type: ignore # pylint: disable=import-error from acme import errors -from acme import test_util -from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +import test_util class SSLSocketAndProbeSNITest(unittest.TestCase): @@ -30,7 +29,6 @@ class SSLSocketAndProbeSNITest(unittest.TestCase): class _TestServer(socketserver.TCPServer): - # pylint: disable=too-few-public-methods # six.moves.* | pylint: disable=attribute-defined-outside-init,no-init def server_bind(self): # pylint: disable=missing-docstring @@ -40,7 +38,6 @@ class SSLSocketAndProbeSNITest(unittest.TestCase): self.server = _TestServer(('', 0), socketserver.BaseRequestHandler) self.port = self.server.socket.getsockname()[1] self.server_thread = threading.Thread( - # pylint: disable=no-member target=self.server.handle_request) def tearDown(self): @@ -67,7 +64,7 @@ class SSLSocketAndProbeSNITest(unittest.TestCase): def test_probe_connection_error(self): # pylint has a hard time with six - self.server.server_close() # pylint: disable=no-member + self.server.server_close() original_timeout = socket.getdefaulttimeout() try: socket.setdefaulttimeout(1) diff --git a/acme/acme/errors_test.py b/acme/tests/errors_test.py similarity index 100% rename from acme/acme/errors_test.py rename to acme/tests/errors_test.py diff --git a/acme/acme/fields_test.py b/acme/tests/fields_test.py similarity index 100% rename from acme/acme/fields_test.py rename to acme/tests/fields_test.py diff --git a/acme/acme/jose_test.py b/acme/tests/jose_test.py similarity index 87% rename from acme/acme/jose_test.py rename to acme/tests/jose_test.py index 340624a4f..e008cb6fc 100644 --- a/acme/acme/jose_test.py +++ b/acme/tests/jose_test.py @@ -2,6 +2,7 @@ import importlib import unittest + class JoseTest(unittest.TestCase): """Tests for acme.jose shim.""" @@ -20,11 +21,10 @@ class JoseTest(unittest.TestCase): # We use the imports below with eval, but pylint doesn't # understand that. - # pylint: disable=eval-used,unused-variable - import acme - import josepy - acme_jose_mod = eval(acme_jose_path) - josepy_mod = eval(josepy_path) + import acme # pylint: disable=unused-import + import josepy # pylint: disable=unused-import + acme_jose_mod = eval(acme_jose_path) # pylint: disable=eval-used + josepy_mod = eval(josepy_path) # pylint: disable=eval-used self.assertIs(acme_jose_mod, josepy_mod) self.assertIs(getattr(acme_jose_mod, attribute), getattr(josepy_mod, attribute)) diff --git a/acme/acme/jws_test.py b/acme/tests/jws_test.py similarity index 98% rename from acme/acme/jws_test.py rename to acme/tests/jws_test.py index aa3ccb700..2e6ad72dd 100644 --- a/acme/acme/jws_test.py +++ b/acme/tests/jws_test.py @@ -3,8 +3,7 @@ import unittest import josepy as jose -from acme import test_util - +import test_util KEY = jose.JWKRSA.load(test_util.load_vector('rsa512_key.pem')) diff --git a/acme/acme/magic_typing_test.py b/acme/tests/magic_typing_test.py similarity index 100% rename from acme/acme/magic_typing_test.py rename to acme/tests/magic_typing_test.py diff --git a/acme/acme/messages_test.py b/acme/tests/messages_test.py similarity index 95% rename from acme/acme/messages_test.py rename to acme/tests/messages_test.py index 7efaaa1a3..b9b70266b 100644 --- a/acme/acme/messages_test.py +++ b/acme/tests/messages_test.py @@ -5,9 +5,8 @@ import josepy as jose import mock from acme import challenges -from acme import test_util -from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module - +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +import test_util CERT = test_util.load_comparable_cert('cert.der') CSR = test_util.load_comparable_csr('csr.der') @@ -19,8 +18,7 @@ class ErrorTest(unittest.TestCase): def setUp(self): from acme.messages import Error, ERROR_PREFIX - self.error = Error( - detail='foo', typ=ERROR_PREFIX + 'malformed', title='title') + self.error = Error.with_code('malformed', detail='foo', title='title') self.jobj = { 'detail': 'foo', 'title': 'some title', @@ -28,7 +26,6 @@ class ErrorTest(unittest.TestCase): } self.error_custom = Error(typ='custom', detail='bar') self.empty_error = Error() - self.jobj_custom = {'type': 'custom', 'detail': 'bar'} def test_default_typ(self): from acme.messages import Error @@ -43,8 +40,7 @@ class ErrorTest(unittest.TestCase): hash(Error.from_json(self.error.to_json())) def test_description(self): - self.assertEqual( - 'The request message was malformed', self.error.description) + self.assertEqual('The request message was malformed', self.error.description) self.assertTrue(self.error_custom.description is None) def test_code(self): @@ -54,17 +50,17 @@ class ErrorTest(unittest.TestCase): self.assertEqual(None, Error().code) def test_is_acme_error(self): - from acme.messages import is_acme_error + from acme.messages import is_acme_error, Error self.assertTrue(is_acme_error(self.error)) self.assertFalse(is_acme_error(self.error_custom)) + self.assertFalse(is_acme_error(Error())) self.assertFalse(is_acme_error(self.empty_error)) self.assertFalse(is_acme_error("must pet all the {dogs|rabbits}")) def test_unicode_error(self): - from acme.messages import Error, ERROR_PREFIX, is_acme_error - arabic_error = Error( - detail=u'\u0639\u062f\u0627\u0644\u0629', typ=ERROR_PREFIX + 'malformed', - title='title') + from acme.messages import Error, is_acme_error + arabic_error = Error.with_code( + 'malformed', detail=u'\u0639\u062f\u0627\u0644\u0629', title='title') self.assertTrue(is_acme_error(arabic_error)) def test_with_code(self): @@ -305,8 +301,7 @@ class ChallengeBodyTest(unittest.TestCase): from acme.messages import Error from acme.messages import STATUS_INVALID self.status = STATUS_INVALID - error = Error(typ='urn:ietf:params:acme:error:serverInternal', - detail='Unable to communicate with DNS server') + error = Error.with_code('serverInternal', detail='Unable to communicate with DNS server') self.challb = ChallengeBody( uri='http://challb', chall=self.chall, status=self.status, error=error) diff --git a/acme/acme/standalone_test.py b/acme/tests/standalone_test.py similarity index 60% rename from acme/acme/standalone_test.py rename to acme/tests/standalone_test.py index ee527782a..83ced12b0 100644 --- a/acme/acme/standalone_test.py +++ b/acme/tests/standalone_test.py @@ -1,23 +1,17 @@ """Tests for acme.standalone.""" -import os -import shutil import socket import threading -import tempfile import unittest -from six.moves import http_client # pylint: disable=import-error -from six.moves import queue # pylint: disable=import-error -from six.moves import socketserver # type: ignore # pylint: disable=import-error - import josepy as jose import mock import requests +from six.moves import http_client # pylint: disable=import-error +from six.moves import socketserver # type: ignore # pylint: disable=import-error from acme import challenges -from acme import crypto_util -from acme import test_util -from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module +import test_util class TLSServerTest(unittest.TestCase): @@ -28,41 +22,14 @@ class TLSServerTest(unittest.TestCase): from acme.standalone import TLSServer server = TLSServer( ('', 0), socketserver.BaseRequestHandler, bind_and_activate=True) - server.server_close() # pylint: disable=no-member + server.server_close() def test_ipv6(self): if socket.has_ipv6: from acme.standalone import TLSServer server = TLSServer( ('', 0), socketserver.BaseRequestHandler, bind_and_activate=True, ipv6=True) - server.server_close() # pylint: disable=no-member - - -class TLSSNI01ServerTest(unittest.TestCase): - """Test for acme.standalone.TLSSNI01Server.""" - - - def setUp(self): - self.certs = {b'localhost': ( - test_util.load_pyopenssl_private_key('rsa2048_key.pem'), - test_util.load_cert('rsa2048_cert.pem'), - )} - from acme.standalone import TLSSNI01Server - self.server = TLSSNI01Server(('localhost', 0), certs=self.certs) - # pylint: disable=no-member - self.thread = threading.Thread(target=self.server.serve_forever) - self.thread.start() - - def tearDown(self): - self.server.shutdown() # pylint: disable=no-member - self.thread.join() - - def test_it(self): - host, port = self.server.socket.getsockname()[:2] - cert = crypto_util.probe_sni( - b'localhost', host=host, port=port, timeout=1) - self.assertEqual(jose.ComparableX509(cert), - jose.ComparableX509(self.certs[b'localhost'][1])) + server.server_close() class HTTP01ServerTest(unittest.TestCase): @@ -77,13 +44,12 @@ class HTTP01ServerTest(unittest.TestCase): from acme.standalone import HTTP01Server self.server = HTTP01Server(('', 0), resources=self.resources) - # pylint: disable=no-member self.port = self.server.socket.getsockname()[1] self.thread = threading.Thread(target=self.server.serve_forever) self.thread.start() def tearDown(self): - self.server.shutdown() # pylint: disable=no-member + self.server.shutdown() self.thread.join() def test_index(self): @@ -136,7 +102,6 @@ class BaseDualNetworkedServersTest(unittest.TestCase): # NB: On Windows, socket.IPPROTO_IPV6 constant may be missing. # We use the corresponding value (41) instead. level = getattr(socket, "IPPROTO_IPV6", 41) - # pylint: disable=no-member self.socket.setsockopt(level, socket.IPV6_V6ONLY, 1) try: self.server_bind() @@ -170,33 +135,6 @@ class BaseDualNetworkedServersTest(unittest.TestCase): prev_port = port -class TLSSNI01DualNetworkedServersTest(unittest.TestCase): - """Test for acme.standalone.TLSSNI01DualNetworkedServers.""" - - - def setUp(self): - self.certs = {b'localhost': ( - test_util.load_pyopenssl_private_key('rsa2048_key.pem'), - test_util.load_cert('rsa2048_cert.pem'), - )} - from acme.standalone import TLSSNI01DualNetworkedServers - self.servers = TLSSNI01DualNetworkedServers(('localhost', 0), certs=self.certs) - self.servers.serve_forever() - - def tearDown(self): - self.servers.shutdown_and_server_close() - - def test_connect(self): - socknames = self.servers.getsocknames() - # connect to all addresses - for sockname in socknames: - host, port = sockname[:2] - cert = crypto_util.probe_sni( - b'localhost', host=host, port=port, timeout=1) - self.assertEqual(jose.ComparableX509(cert), - jose.ComparableX509(self.certs[b'localhost'][1])) - - class HTTP01DualNetworkedServersTest(unittest.TestCase): """Tests for acme.standalone.HTTP01DualNetworkedServers.""" @@ -209,7 +147,6 @@ class HTTP01DualNetworkedServersTest(unittest.TestCase): from acme.standalone import HTTP01DualNetworkedServers self.servers = HTTP01DualNetworkedServers(('', 0), resources=self.resources) - # pylint: disable=no-member self.port = self.servers.getsocknames()[0][1] self.servers.serve_forever() @@ -248,51 +185,5 @@ class HTTP01DualNetworkedServersTest(unittest.TestCase): self.assertFalse(self._test_http01(add=False)) -@test_util.broken_on_windows -class TestSimpleTLSSNI01Server(unittest.TestCase): - """Tests for acme.standalone.simple_tls_sni_01_server.""" - - - def setUp(self): - # mirror ../examples/standalone - self.test_cwd = tempfile.mkdtemp() - localhost_dir = os.path.join(self.test_cwd, 'localhost') - os.makedirs(localhost_dir) - shutil.copy(test_util.vector_path('rsa2048_cert.pem'), - os.path.join(localhost_dir, 'cert.pem')) - shutil.copy(test_util.vector_path('rsa2048_key.pem'), - os.path.join(localhost_dir, 'key.pem')) - - from acme.standalone import simple_tls_sni_01_server - self.thread = threading.Thread( - target=simple_tls_sni_01_server, kwargs={ - 'cli_args': ('filename',), - 'forever': False, - }, - ) - self.old_cwd = os.getcwd() - os.chdir(self.test_cwd) - - def tearDown(self): - os.chdir(self.old_cwd) - self.thread.join() - shutil.rmtree(self.test_cwd) - - @mock.patch('acme.standalone.logger') - def test_it(self, mock_logger): - # Use a Queue because mock objects aren't thread safe. - q = queue.Queue() # type: queue.Queue[int] - # Add port number to the queue. - mock_logger.info.side_effect = lambda *args: q.put(args[-1]) - self.thread.start() - - # After the timeout, an exception is raised if the queue is empty. - port = q.get(timeout=5) - cert = crypto_util.probe_sni(b'localhost', b'0.0.0.0', port) - self.assertEqual(jose.ComparableX509(cert), - test_util.load_comparable_cert( - 'rsa2048_cert.pem')) - - if __name__ == "__main__": unittest.main() # pragma: no cover diff --git a/acme/acme/test_util.py b/acme/tests/test_util.py similarity index 60% rename from acme/acme/test_util.py rename to acme/tests/test_util.py index f97614700..d4a45272d 100644 --- a/acme/acme/test_util.py +++ b/acme/tests/test_util.py @@ -4,20 +4,12 @@ """ import os -import sys -import pkg_resources -import unittest from cryptography.hazmat.backends import default_backend from cryptography.hazmat.primitives import serialization import josepy as jose from OpenSSL import crypto - - -def vector_path(*names): - """Path to a test vector.""" - return pkg_resources.resource_filename( - __name__, os.path.join('testdata', *names)) +import pkg_resources def load_vector(*names): @@ -33,8 +25,7 @@ def _guess_loader(filename, loader_pem, loader_der): return loader_pem elif ext.lower() == '.der': return loader_der - else: # pragma: no cover - raise ValueError("Loader could not be recognized based on extension") + raise ValueError("Loader could not be recognized based on extension") # pragma: no cover def load_cert(*names): @@ -74,32 +65,3 @@ def load_pyopenssl_private_key(*names): loader = _guess_loader( names[-1], crypto.FILETYPE_PEM, crypto.FILETYPE_ASN1) return crypto.load_privatekey(loader, load_vector(*names)) - - -def skip_unless(condition, reason): # pragma: no cover - """Skip tests unless a condition holds. - - This implements the basic functionality of unittest.skipUnless - which is only available on Python 2.7+. - - :param bool condition: If ``False``, the test will be skipped - :param str reason: the reason for skipping the test - - :rtype: callable - :returns: decorator that hides tests unless condition is ``True`` - - """ - if hasattr(unittest, "skipUnless"): - return unittest.skipUnless(condition, reason) - elif condition: - return lambda cls: cls - else: - return lambda cls: None - -def broken_on_windows(function): - """Decorator to skip temporarily a broken test on Windows.""" - reason = 'Test is broken and ignored on windows but should be fixed.' - return unittest.skipIf( - sys.platform == 'win32' - and os.environ.get('SKIP_BROKEN_TESTS_ON_WINDOWS', 'true') == 'true', - reason)(function) diff --git a/acme/acme/testdata/README b/acme/tests/testdata/README similarity index 100% rename from acme/acme/testdata/README rename to acme/tests/testdata/README diff --git a/acme/acme/testdata/cert-100sans.pem b/acme/tests/testdata/cert-100sans.pem similarity index 100% rename from acme/acme/testdata/cert-100sans.pem rename to acme/tests/testdata/cert-100sans.pem diff --git a/acme/acme/testdata/cert-idnsans.pem b/acme/tests/testdata/cert-idnsans.pem similarity index 100% rename from acme/acme/testdata/cert-idnsans.pem rename to acme/tests/testdata/cert-idnsans.pem diff --git a/acme/acme/testdata/cert-nocn.der b/acme/tests/testdata/cert-nocn.der similarity index 100% rename from acme/acme/testdata/cert-nocn.der rename to acme/tests/testdata/cert-nocn.der diff --git a/acme/acme/testdata/cert-san.pem b/acme/tests/testdata/cert-san.pem similarity index 100% rename from acme/acme/testdata/cert-san.pem rename to acme/tests/testdata/cert-san.pem diff --git a/acme/acme/testdata/cert.der b/acme/tests/testdata/cert.der similarity index 100% rename from acme/acme/testdata/cert.der rename to acme/tests/testdata/cert.der diff --git a/acme/acme/testdata/cert.pem b/acme/tests/testdata/cert.pem similarity index 100% rename from acme/acme/testdata/cert.pem rename to acme/tests/testdata/cert.pem diff --git a/acme/acme/testdata/critical-san.pem b/acme/tests/testdata/critical-san.pem similarity index 100% rename from acme/acme/testdata/critical-san.pem rename to acme/tests/testdata/critical-san.pem diff --git a/acme/acme/testdata/csr-100sans.pem b/acme/tests/testdata/csr-100sans.pem similarity index 100% rename from acme/acme/testdata/csr-100sans.pem rename to acme/tests/testdata/csr-100sans.pem diff --git a/acme/acme/testdata/csr-6sans.pem b/acme/tests/testdata/csr-6sans.pem similarity index 100% rename from acme/acme/testdata/csr-6sans.pem rename to acme/tests/testdata/csr-6sans.pem diff --git a/acme/acme/testdata/csr-idnsans.pem b/acme/tests/testdata/csr-idnsans.pem similarity index 100% rename from acme/acme/testdata/csr-idnsans.pem rename to acme/tests/testdata/csr-idnsans.pem diff --git a/acme/acme/testdata/csr-nosans.pem b/acme/tests/testdata/csr-nosans.pem similarity index 100% rename from acme/acme/testdata/csr-nosans.pem rename to acme/tests/testdata/csr-nosans.pem diff --git a/acme/acme/testdata/csr-san.pem b/acme/tests/testdata/csr-san.pem similarity index 100% rename from acme/acme/testdata/csr-san.pem rename to acme/tests/testdata/csr-san.pem diff --git a/acme/acme/testdata/csr.der b/acme/tests/testdata/csr.der similarity index 100% rename from acme/acme/testdata/csr.der rename to acme/tests/testdata/csr.der diff --git a/acme/acme/testdata/csr.pem b/acme/tests/testdata/csr.pem similarity index 100% rename from acme/acme/testdata/csr.pem rename to acme/tests/testdata/csr.pem diff --git a/acme/acme/testdata/dsa512_key.pem b/acme/tests/testdata/dsa512_key.pem similarity index 100% rename from acme/acme/testdata/dsa512_key.pem rename to acme/tests/testdata/dsa512_key.pem diff --git a/acme/acme/testdata/rsa1024_key.pem b/acme/tests/testdata/rsa1024_key.pem similarity index 100% rename from acme/acme/testdata/rsa1024_key.pem rename to acme/tests/testdata/rsa1024_key.pem diff --git a/acme/acme/testdata/rsa2048_cert.pem b/acme/tests/testdata/rsa2048_cert.pem similarity index 100% rename from acme/acme/testdata/rsa2048_cert.pem rename to acme/tests/testdata/rsa2048_cert.pem diff --git a/acme/acme/testdata/rsa2048_key.pem b/acme/tests/testdata/rsa2048_key.pem similarity index 100% rename from acme/acme/testdata/rsa2048_key.pem rename to acme/tests/testdata/rsa2048_key.pem diff --git a/acme/acme/testdata/rsa256_key.pem b/acme/tests/testdata/rsa256_key.pem similarity index 100% rename from acme/acme/testdata/rsa256_key.pem rename to acme/tests/testdata/rsa256_key.pem diff --git a/acme/acme/testdata/rsa512_key.pem b/acme/tests/testdata/rsa512_key.pem similarity index 100% rename from acme/acme/testdata/rsa512_key.pem rename to acme/tests/testdata/rsa512_key.pem diff --git a/acme/acme/util_test.py b/acme/tests/util_test.py similarity index 100% rename from acme/acme/util_test.py rename to acme/tests/util_test.py diff --git a/appveyor.yml b/appveyor.yml deleted file mode 100644 index 2b6b82747..000000000 --- a/appveyor.yml +++ /dev/null @@ -1,32 +0,0 @@ -image: Visual Studio 2015 - -environment: - matrix: - - TOXENV: py35 - - TOXENV: py37-cover - -branches: - only: - - master - - /^\d+\.\d+\.x$/ # Version branches like X.X.X - - /^test-.*$/ - -install: - # Use Python 3.7 by default - - "SET PATH=C:\\Python37;C:\\Python37\\Scripts;%PATH%" - # Check env - - "python --version" - # Upgrade pip to avoid warnings - - "python -m pip install --upgrade pip" - # Ready to install tox and coverage - - "pip install tox codecov" - -build: off - -test_script: - - set TOX_TESTENV_PASSENV=APPVEYOR - # Test env is set by TOXENV env variable - - tox - -on_success: - - if exist .coverage codecov diff --git a/certbot-apache/MANIFEST.in b/certbot-apache/MANIFEST.in index 3e594a953..2316983bb 100644 --- a/certbot-apache/MANIFEST.in +++ b/certbot-apache/MANIFEST.in @@ -1,7 +1,7 @@ include LICENSE.txt include README.rst -recursive-include docs * -recursive-include certbot_apache/tests/testdata * -include certbot_apache/centos-options-ssl-apache.conf -include certbot_apache/options-ssl-apache.conf -recursive-include certbot_apache/augeas_lens *.aug +recursive-include tests * +include certbot_apache/_internal/options-ssl-apache.conf +recursive-include certbot_apache/_internal/augeas_lens *.aug +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-apache/certbot_apache/_internal/__init__.py b/certbot-apache/certbot_apache/_internal/__init__.py new file mode 100644 index 000000000..9c195ccc7 --- /dev/null +++ b/certbot-apache/certbot_apache/_internal/__init__.py @@ -0,0 +1 @@ +"""Certbot Apache plugin.""" diff --git a/certbot-apache/certbot_apache/apache_util.py b/certbot-apache/certbot_apache/_internal/apache_util.py similarity index 99% rename from certbot-apache/certbot_apache/apache_util.py rename to certbot-apache/certbot_apache/_internal/apache_util.py index 62342004f..7a2ecf49b 100644 --- a/certbot-apache/certbot_apache/apache_util.py +++ b/certbot-apache/certbot_apache/_internal/apache_util.py @@ -1,8 +1,9 @@ """ Utility functions for certbot-apache plugin """ import binascii -import os from certbot import util +from certbot.compat import os + def get_mod_deps(mod_name): """Get known module dependencies. diff --git a/certbot-apache/certbot_apache/augeas_lens/README b/certbot-apache/certbot_apache/_internal/augeas_lens/README similarity index 100% rename from certbot-apache/certbot_apache/augeas_lens/README rename to certbot-apache/certbot_apache/_internal/augeas_lens/README diff --git a/certbot-apache/certbot_apache/augeas_lens/httpd.aug b/certbot-apache/certbot_apache/_internal/augeas_lens/httpd.aug similarity index 100% rename from certbot-apache/certbot_apache/augeas_lens/httpd.aug rename to certbot-apache/certbot_apache/_internal/augeas_lens/httpd.aug diff --git a/certbot-apache/certbot_apache/configurator.py b/certbot-apache/certbot_apache/_internal/configurator.py similarity index 89% rename from certbot-apache/certbot_apache/configurator.py rename to certbot-apache/certbot_apache/_internal/configurator.py index 16de3a3d8..84b59d2c7 100644 --- a/certbot-apache/certbot_apache/configurator.py +++ b/certbot-apache/certbot_apache/_internal/configurator.py @@ -1,40 +1,39 @@ -"""Apache Configuration based off of Augeas Configurator.""" +"""Apache Configurator.""" # pylint: disable=too-many-lines +from collections import defaultdict import copy import fnmatch import logging -import os -import pkg_resources import re -import six import socket import time +import pkg_resources +import six import zope.component import zope.interface from acme import challenges -from acme.magic_typing import Any, DefaultDict, Dict, List, Set, Union # pylint: disable=unused-import, no-name-in-module - +from acme.magic_typing import DefaultDict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Union # pylint: disable=unused-import, no-name-in-module from certbot import errors from certbot import interfaces from certbot import util - from certbot.achallenges import KeyAuthorizationAnnotatedChallenge # pylint: disable=unused-import +from certbot.compat import filesystem +from certbot.compat import os from certbot.plugins import common -from certbot.plugins.util import path_surgery from certbot.plugins.enhancements import AutoHSTSEnhancement - -from certbot_apache import apache_util -from certbot_apache import augeas_configurator -from certbot_apache import constants -from certbot_apache import display_ops -from certbot_apache import http_01 -from certbot_apache import obj -from certbot_apache import parser -from certbot_apache import tls_sni_01 - -from collections import defaultdict +from certbot.plugins.util import path_surgery +from certbot_apache._internal import apache_util +from certbot_apache._internal import constants +from certbot_apache._internal import display_ops +from certbot_apache._internal import http_01 +from certbot_apache._internal import obj +from certbot_apache._internal import parser logger = logging.getLogger(__name__) @@ -70,28 +69,29 @@ logger = logging.getLogger(__name__) @zope.interface.implementer(interfaces.IAuthenticator, interfaces.IInstaller) @zope.interface.provider(interfaces.IPluginFactory) -class ApacheConfigurator(augeas_configurator.AugeasConfigurator): - # pylint: disable=too-many-instance-attributes,too-many-public-methods +class ApacheConfigurator(common.Installer): """Apache configurator. - State of Configurator: This code has been been tested and built for Ubuntu - 14.04 Apache 2.4 and it works for Ubuntu 12.04 Apache 2.2 - :ivar config: Configuration. :type config: :class:`~certbot.interfaces.IConfig` :ivar parser: Handles low level parsing - :type parser: :class:`~certbot_apache.parser` + :type parser: :class:`~certbot_apache._internal.parser` :ivar tup version: version of Apache :ivar list vhosts: All vhosts found in the configuration - (:class:`list` of :class:`~certbot_apache.obj.VirtualHost`) + (:class:`list` of :class:`~certbot_apache._internal.obj.VirtualHost`) :ivar dict assoc: Mapping between domains and vhosts """ description = "Apache Web Server plugin" + if os.environ.get("CERTBOT_DOCS") == "1": + description += ( # pragma: no cover + " (Please note that the default values of the Apache plugin options" + " change depending on the operating system Certbot is run on.)" + ) OS_DEFAULTS = dict( server_root="/etc/apache2", @@ -109,7 +109,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): handle_sites=False, challenge_location="/etc/apache2", MOD_SSL_CONF_SRC=pkg_resources.resource_filename( - "certbot_apache", "options-ssl-apache.conf") + "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf")) ) def option(self, key): @@ -141,31 +141,37 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): # When adding, modifying or deleting command line arguments, be sure to # include the changes in the list used in method _prepare_options() to # ensure consistent behavior. - add("enmod", default=cls.OS_DEFAULTS["enmod"], + + # Respect CERTBOT_DOCS environment variable and use default values from + # base class regardless of the underlying distribution (overrides). + if os.environ.get("CERTBOT_DOCS") == "1": + DEFAULTS = ApacheConfigurator.OS_DEFAULTS + else: + # cls.OS_DEFAULTS can be distribution specific, see override classes + DEFAULTS = cls.OS_DEFAULTS + add("enmod", default=DEFAULTS["enmod"], help="Path to the Apache 'a2enmod' binary") - add("dismod", default=cls.OS_DEFAULTS["dismod"], + add("dismod", default=DEFAULTS["dismod"], help="Path to the Apache 'a2dismod' binary") - add("le-vhost-ext", default=cls.OS_DEFAULTS["le_vhost_ext"], + add("le-vhost-ext", default=DEFAULTS["le_vhost_ext"], help="SSL vhost configuration extension") - add("server-root", default=cls.OS_DEFAULTS["server_root"], + add("server-root", default=DEFAULTS["server_root"], help="Apache server root directory") add("vhost-root", default=None, help="Apache server VirtualHost configuration root") - add("logs-root", default=cls.OS_DEFAULTS["logs_root"], + add("logs-root", default=DEFAULTS["logs_root"], help="Apache server logs directory") add("challenge-location", - default=cls.OS_DEFAULTS["challenge_location"], + default=DEFAULTS["challenge_location"], help="Directory path for challenge configuration") - add("handle-modules", default=cls.OS_DEFAULTS["handle_modules"], + add("handle-modules", default=DEFAULTS["handle_modules"], help="Let installer handle enabling required modules for you " + "(Only Ubuntu/Debian currently)") - add("handle-sites", default=cls.OS_DEFAULTS["handle_sites"], + add("handle-sites", default=DEFAULTS["handle_sites"], help="Let installer handle enabling sites for you " + "(Only Ubuntu/Debian currently)") - add("ctl", default=cls.OS_DEFAULTS["ctl"], + add("ctl", default=DEFAULTS["ctl"], help="Full path to Apache control script") - util.add_deprecated_argument( - add, argument_name="init-script", nargs=1) def __init__(self, *args, **kwargs): """Initialize an Apache Configurator. @@ -188,6 +194,8 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): self._enhanced_vhosts = defaultdict(set) # type: DefaultDict[str, Set[obj.VirtualHost]] # Temporary state for AutoHSTS enhancement self._autohsts = {} # type: Dict[str, Dict[str, Union[int, float]]] + # Reverter save notes + self.save_notes = "" # These will be set in the prepare function self._prepared = False @@ -202,15 +210,13 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): @property def mod_ssl_conf(self): """Full absolute path to SSL configuration file.""" - return os.path.join(self.config.config_dir, - constants.MOD_SSL_CONF_DEST) + return os.path.join(self.config.config_dir, constants.MOD_SSL_CONF_DEST) @property def updated_mod_ssl_conf_digest(self): """Full absolute path to digest of updated SSL configuration file.""" return os.path.join(self.config.config_dir, constants.UPDATED_MOD_SSL_CONF_DIGEST) - def prepare(self): """Prepare the authenticator/installer. @@ -220,12 +226,6 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): :raises .errors.PluginError: If there is any other error """ - # Perform the actual Augeas initialization to be able to react - try: - self.init_augeas() - except ImportError: - raise errors.NoInstallationError("Problem in Augeas installation") - self._prepare_options() # Verify Apache is installed @@ -241,18 +241,16 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): '.'.join(str(i) for i in self.version)) if self.version < (2, 2): raise errors.NotSupportedError( - "Apache Version %s not supported.", str(self.version)) - - if not self._check_aug_version(): - raise errors.NotSupportedError( - "Apache plugin support requires libaugeas0 and augeas-lenses " - "version 1.2.0 or higher, please make sure you have you have " - "those installed.") + "Apache Version {0} not supported.".format(str(self.version))) + # Recover from previous crash before Augeas initialization to have the + # correct parse tree from the get go. + self.recovery_routine() + # Perform the actual Augeas initialization to be able to react self.parser = self.get_parser() # Check for errors in parsing files with Augeas - self.check_parsing_errors("httpd.aug") + self.parser.check_parsing_errors("httpd.aug") # Get all of the available vhosts self.vhosts = self.get_virtual_hosts() @@ -266,9 +264,72 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): except (OSError, errors.LockError): logger.debug("Encountered error:", exc_info=True) raise errors.PluginError( - "Unable to lock %s", self.option("server_root")) + "Unable to create a lock file in {0}. Are you running" + " Certbot with sufficient privileges to modify your" + " Apache configuration?".format(self.option("server_root"))) self._prepared = True + def save(self, title=None, temporary=False): + """Saves all changes to the configuration files. + + This function first checks for save errors, if none are found, + all configuration changes made will be saved. According to the + function parameters. If an exception is raised, a new checkpoint + was not created. + + :param str title: The title of the save. If a title is given, the + configuration will be saved as a new checkpoint and put in a + timestamped directory. + + :param bool temporary: Indicates whether the changes made will + be quickly reversed in the future (ie. challenges) + + """ + save_files = self.parser.unsaved_files() + if save_files: + self.add_to_checkpoint(save_files, + self.save_notes, temporary=temporary) + # Handle the parser specific tasks + self.parser.save(save_files) + if title and not temporary: + self.finalize_checkpoint(title) + + def recovery_routine(self): + """Revert all previously modified files. + + Reverts all modified files that have not been saved as a checkpoint + + :raises .errors.PluginError: If unable to recover the configuration + + """ + super(ApacheConfigurator, self).recovery_routine() + # Reload configuration after these changes take effect if needed + # ie. ApacheParser has been initialized. + if self.parser: + # TODO: wrap into non-implementation specific parser interface + self.parser.aug.load() + + def revert_challenge_config(self): + """Used to cleanup challenge configurations. + + :raises .errors.PluginError: If unable to revert the challenge config. + + """ + self.revert_temporary_config() + self.parser.aug.load() + + def rollback_checkpoints(self, rollback=1): + """Rollback saved checkpoints. + + :param int rollback: Number of checkpoints to revert + + :raises .errors.PluginError: If there is a problem with the input or + the function is unable to correctly revert the configuration + + """ + super(ApacheConfigurator, self).rollback_checkpoints(rollback) + self.parser.aug.load() + def _verify_exe_availability(self, exe): """Checks availability of Apache executable""" if not util.exe_exists(exe): @@ -276,26 +337,11 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): raise errors.NoInstallationError( 'Cannot find Apache executable {0}'.format(exe)) - def _check_aug_version(self): - """ Checks that we have recent enough version of libaugeas. - If augeas version is recent enough, it will support case insensitive - regexp matching""" - - self.aug.set("/test/path/testing/arg", "aRgUMeNT") - try: - matches = self.aug.match( - "/test//*[self::arg=~regexp('argument', 'i')]") - except RuntimeError: - self.aug.remove("/test/path") - return False - self.aug.remove("/test/path") - return matches - def get_parser(self): """Initializes the ApacheParser""" # If user provided vhost_root value in command line, use it return parser.ApacheParser( - self.aug, self.option("server_root"), self.conf("vhost-root"), + self.option("server_root"), self.conf("vhost-root"), self.version, configurator=self) def _wildcard_domain(self, domain): @@ -344,7 +390,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): counterpart, should one get created :returns: List of VirtualHosts or None - :rtype: `list` of :class:`~certbot_apache.obj.VirtualHost` + :rtype: `list` of :class:`~certbot_apache._internal.obj.VirtualHost` """ if self._wildcard_domain(domain): @@ -382,7 +428,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """ if len(name.split(".")) == len(domain.split(".")): return fnmatch.fnmatch(name, domain) - + return None def _choose_vhosts_wildcard(self, domain, create_ssl=True): """Prompts user to choose vhosts to install a wildcard certificate for""" @@ -403,7 +449,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): filtered_vhosts[name] = vhost # Only unique VHost objects - dialog_input = set([vhost for vhost in filtered_vhosts.values()]) + dialog_input = set(filtered_vhosts.values()) # Ask the user which of names to enable, expect list of names back dialog_output = display_ops.select_vhost_multiple(list(dialog_input)) @@ -428,7 +474,6 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): self._wildcard_vhosts[domain] = return_vhosts return return_vhosts - def _deploy_cert(self, vhost, cert_path, key_path, chain_path, fullchain_path): """ Helper function for deploy_cert() that handles the actual deployment @@ -436,8 +481,6 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): domain originally passed for deploy_cert(). This is especially true with wildcard certificates """ - - # This is done first so that ssl module is enabled and cert_path, # cert_key... can all be parsed appropriately self.prepare_server_https("443") @@ -477,8 +520,8 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): # install SSLCertificateFile, SSLCertificateKeyFile, # and SSLCertificateChainFile directives set_cert_path = cert_path - self.aug.set(path["cert_path"][-1], cert_path) - self.aug.set(path["cert_key"][-1], key_path) + self.parser.aug.set(path["cert_path"][-1], cert_path) + self.parser.aug.set(path["cert_key"][-1], key_path) if chain_path is not None: self.parser.add_dir(vhost.path, "SSLCertificateChainFile", chain_path) @@ -490,8 +533,8 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): raise errors.PluginError("Please provide the --fullchain-path " "option pointing to your full chain file") set_cert_path = fullchain_path - self.aug.set(path["cert_path"][-1], fullchain_path) - self.aug.set(path["cert_key"][-1], key_path) + self.parser.aug.set(path["cert_path"][-1], fullchain_path) + self.parser.aug.set(path["cert_key"][-1], key_path) # Enable the new vhost if needed if not vhost.enabled: @@ -522,7 +565,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): counterpart, should one get created :returns: vhost associated with name - :rtype: :class:`~certbot_apache.obj.VirtualHost` + :rtype: :class:`~certbot_apache._internal.obj.VirtualHost` :raises .errors.PluginError: If no vhost is available or chosen @@ -557,9 +600,9 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): "in the Apache config.", target_name) raise errors.PluginError("No vhost selected") - elif temp: + if temp: return vhost - elif not vhost.ssl: + if not vhost.ssl: addrs = self._get_proposed_addrs(vhost, "443") # TODO: Conflicts is too conservative if not any(vhost.enabled and vhost.conflicts(addrs) for @@ -577,8 +620,9 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): self.assoc[target_name] = vhost return vhost - def included_in_wildcard(self, names, target_name): - """Is target_name covered by a wildcard? + def domain_in_names(self, names, target_name): + """Checks if target domain is covered by one or more of the provided + names. The target name is matched by wildcard as well as exact match. :param names: server aliases :type names: `collections.Iterable` of `str` @@ -624,7 +668,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): :param str target_name: domain handled by the desired vhost :param vhosts: vhosts to consider - :type vhosts: `collections.Iterable` of :class:`~certbot_apache.obj.VirtualHost` + :type vhosts: `collections.Iterable` of :class:`~certbot_apache._internal.obj.VirtualHost` :param bool filter_defaults: whether a vhost with a _default_ addr is acceptable @@ -649,7 +693,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): names = vhost.get_names() if target_name in names: points = 3 - elif self.included_in_wildcard(names, target_name): + elif self.domain_in_names(names, target_name): points = 2 elif any(addr.get_addr() == target_name for addr in vhost.addrs): points = 1 @@ -708,7 +752,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): if name: all_names.add(name) - if len(vhost_macro) > 0: + if vhost_macro: zope.component.getUtility(interfaces.IDisplay).notification( "Apache mod_macro seems to be in use in file(s):\n{0}" "\n\nUnfortunately mod_macro is not yet supported".format( @@ -766,7 +810,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """Helper function for get_virtual_hosts(). :param host: In progress vhost whose names will be added - :type host: :class:`~certbot_apache.obj.VirtualHost` + :type host: :class:`~certbot_apache._internal.obj.VirtualHost` """ @@ -785,12 +829,12 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): :param str path: Augeas path to virtual host :returns: newly created vhost - :rtype: :class:`~certbot_apache.obj.VirtualHost` + :rtype: :class:`~certbot_apache._internal.obj.VirtualHost` """ addrs = set() try: - args = self.aug.match(path + "/arg") + args = self.parser.aug.match(path + "/arg") except RuntimeError: logger.warning("Encountered a problem while parsing file: %s, skipping", path) return None @@ -808,7 +852,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): is_ssl = True filename = apache_util.get_file_path( - self.aug.get("/augeas/files%s/path" % apache_util.get_file_path(path))) + self.parser.aug.get("/augeas/files%s/path" % apache_util.get_file_path(path))) if filename is None: return None @@ -826,7 +870,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): def get_virtual_hosts(self): """Returns list of virtual hosts found in the Apache configuration. - :returns: List of :class:`~certbot_apache.obj.VirtualHost` + :returns: List of :class:`~certbot_apache._internal.obj.VirtualHost` objects found in configuration :rtype: list @@ -838,7 +882,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): # Make a list of parser paths because the parser_paths # dictionary may be modified during the loop. for vhost_path in list(self.parser.parser_paths): - paths = self.aug.match( + paths = self.parser.aug.match( ("/files%s//*[label()=~regexp('%s')]" % (vhost_path, parser.case_i("VirtualHost")))) paths = [path for path in paths if @@ -848,7 +892,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): if not new_vhost: continue internal_path = apache_util.get_internal_aug_path(new_vhost.path) - realpath = os.path.realpath(new_vhost.filep) + realpath = filesystem.realpath(new_vhost.filep) if realpath not in file_paths: file_paths[realpath] = new_vhost.filep internal_paths[realpath].add(internal_path) @@ -883,7 +927,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): now NameVirtualHosts. If version is earlier than 2.4, check if addr has a NameVirtualHost directive in the Apache config - :param certbot_apache.obj.Addr target_addr: vhost address + :param certbot_apache._internal.obj.Addr target_addr: vhost address :returns: Success :rtype: bool @@ -901,19 +945,18 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """Adds NameVirtualHost directive for given address. :param addr: Address that will be added as NameVirtualHost directive - :type addr: :class:`~certbot_apache.obj.Addr` + :type addr: :class:`~certbot_apache._internal.obj.Addr` """ loc = parser.get_aug_path(self.parser.loc["name"]) if addr.get_port() == "443": - path = self.parser.add_dir_to_ifmodssl( + self.parser.add_dir_to_ifmodssl( loc, "NameVirtualHost", [str(addr)]) else: - path = self.parser.add_dir(loc, "NameVirtualHost", [str(addr)]) + self.parser.add_dir(loc, "NameVirtualHost", [str(addr)]) - msg = ("Setting %s to be NameBasedVirtualHost\n" - "\tDirective added to %s\n" % (addr, path)) + msg = "Setting {0} to be NameBasedVirtualHost\n".format(addr) logger.debug(msg) self.save_notes += msg @@ -1054,6 +1097,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): # Ugly but takes care of protocol def, eg: 1.1.1.1:443 https if listen.split(":")[-1].split(" ")[0] == port: return True + return None def prepare_https_modules(self, temp): """Helper method for prepare_server_https, taking care of enabling @@ -1069,24 +1113,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): if "ssl_module" not in self.parser.modules: self.enable_mod("ssl", temp=temp) - def make_addrs_sni_ready(self, addrs): - """Checks to see if the server is ready for SNI challenges. - - :param addrs: Addresses to check SNI compatibility - :type addrs: :class:`~certbot_apache.obj.Addr` - - """ - # Version 2.4 and later are automatically SNI ready. - if self.version >= (2, 4): - return - - for addr in addrs: - if not self.is_name_vhost(addr): - logger.debug("Setting VirtualHost at %s to be a name " - "based virtual host", addr) - self.add_name_vhost(addr) - - def make_vhost_ssl(self, nonssl_vhost): # pylint: disable=too-many-locals + def make_vhost_ssl(self, nonssl_vhost): """Makes an ssl_vhost version of a nonssl_vhost. Duplicates vhost and adds default ssl options @@ -1096,10 +1123,10 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): .. note:: This function saves the configuration :param nonssl_vhost: Valid VH that doesn't have SSLEngine on - :type nonssl_vhost: :class:`~certbot_apache.obj.VirtualHost` + :type nonssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :returns: SSL vhost - :rtype: :class:`~certbot_apache.obj.VirtualHost` + :rtype: :class:`~certbot_apache._internal.obj.VirtualHost` :raises .errors.PluginError: If more than one virtual host is in the file or if plugin is unable to write/read vhost files. @@ -1108,16 +1135,16 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): avail_fp = nonssl_vhost.filep ssl_fp = self._get_ssl_vhost_path(avail_fp) - orig_matches = self.aug.match("/files%s//* [label()=~regexp('%s')]" % + orig_matches = self.parser.aug.match("/files%s//* [label()=~regexp('%s')]" % (self._escape(ssl_fp), parser.case_i("VirtualHost"))) self._copy_create_ssl_vhost_skeleton(nonssl_vhost, ssl_fp) # Reload augeas to take into account the new vhost - self.aug.load() + self.parser.aug.load() # Get Vhost augeas path for new vhost - new_matches = self.aug.match("/files%s//* [label()=~regexp('%s')]" % + new_matches = self.parser.aug.match("/files%s//* [label()=~regexp('%s')]" % (self._escape(ssl_fp), parser.case_i("VirtualHost"))) @@ -1128,7 +1155,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): # Make Augeas aware of the new vhost self.parser.parse_file(ssl_fp) # Try to search again - new_matches = self.aug.match( + new_matches = self.parser.aug.match( "/files%s//* [label()=~regexp('%s')]" % (self._escape(ssl_fp), parser.case_i("VirtualHost"))) @@ -1190,16 +1217,15 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """ if self.conf("vhost-root") and os.path.exists(self.conf("vhost-root")): - fp = os.path.join(os.path.realpath(self.option("vhost_root")), + fp = os.path.join(filesystem.realpath(self.option("vhost_root")), os.path.basename(non_ssl_vh_fp)) else: # Use non-ssl filepath - fp = os.path.realpath(non_ssl_vh_fp) + fp = filesystem.realpath(non_ssl_vh_fp) if fp.endswith(".conf"): return fp[:-(len(".conf"))] + self.option("le_vhost_ext") - else: - return fp + self.option("le_vhost_ext") + return fp + self.option("le_vhost_ext") def _sift_rewrite_rule(self, line): """Decides whether a line should be copied to a SSL vhost. @@ -1279,8 +1305,8 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): "vhost for your HTTPS site located at {1} because they have " "the potential to create redirection loops.".format( vhost.filep, ssl_fp), reporter.MEDIUM_PRIORITY) - self.aug.set("/augeas/files%s/mtime" % (self._escape(ssl_fp)), "0") - self.aug.set("/augeas/files%s/mtime" % (self._escape(vhost.filep)), "0") + self.parser.aug.set("/augeas/files%s/mtime" % (self._escape(ssl_fp)), "0") + self.parser.aug.set("/augeas/files%s/mtime" % (self._escape(vhost.filep)), "0") def _sift_rewrite_rules(self, contents): """ Helper function for _copy_create_ssl_vhost_skeleton to prepare the @@ -1338,12 +1364,9 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): result.append(comment) sift = True - result.append('\n'.join( - ['# ' + l for l in chunk])) - continue + result.append('\n'.join(['# ' + l for l in chunk])) else: result.append('\n'.join(chunk)) - continue return result, sift def _get_vhost_block(self, vhost): @@ -1355,7 +1378,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """ try: - span_val = self.aug.span(vhost.path) + span_val = self.parser.aug.span(vhost.path) except ValueError: logger.critical("Error while reading the VirtualHost %s from " "file %s", vhost.name, vhost.filep, exc_info=True) @@ -1390,13 +1413,13 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): def _update_ssl_vhosts_addrs(self, vh_path): ssl_addrs = set() - ssl_addr_p = self.aug.match(vh_path + "/arg") + ssl_addr_p = self.parser.aug.match(vh_path + "/arg") for addr in ssl_addr_p: old_addr = obj.Addr.fromstring( str(self.parser.get_arg(addr))) ssl_addr = old_addr.get_addr_obj("443") - self.aug.set(addr, str(ssl_addr)) + self.parser.aug.set(addr, str(ssl_addr)) ssl_addrs.add(ssl_addr) return ssl_addrs @@ -1415,15 +1438,14 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): vh_path, False)) > 1: directive_path = self.parser.find_dir(directive, None, vh_path, False) - self.aug.remove(re.sub(r"/\w*$", "", directive_path[0])) + self.parser.aug.remove(re.sub(r"/\w*$", "", directive_path[0])) def _remove_directives(self, vh_path, directives): for directive in directives: - while len(self.parser.find_dir(directive, None, - vh_path, False)) > 0: + while self.parser.find_dir(directive, None, vh_path, False): directive_path = self.parser.find_dir(directive, None, vh_path, False) - self.aug.remove(re.sub(r"/\w*$", "", directive_path[0])) + self.parser.aug.remove(re.sub(r"/\w*$", "", directive_path[0])) def _add_dummy_ssl_directives(self, vh_path): self.parser.add_dir(vh_path, "SSLCertificateFile", @@ -1462,8 +1484,8 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """ matches = self.parser.find_dir( "ServerAlias", start=vh_path, exclude=False) - aliases = (self.aug.get(match) for match in matches) - return self.included_in_wildcard(aliases, target_name) + aliases = (self.parser.aug.get(match) for match in matches) + return self.domain_in_names(aliases, target_name) def _add_name_vhost_if_necessary(self, vhost): """Add NameVirtualHost Directives if necessary for new vhost. @@ -1472,7 +1494,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): https://httpd.apache.org/docs/2.2/mod/core.html#namevirtualhost :param vhost: New virtual host that was recently created. - :type vhost: :class:`~certbot_apache.obj.VirtualHost` + :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost` """ need_to_save = False @@ -1507,7 +1529,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): :param str id_str: Id string for matching :returns: The matched VirtualHost or None - :rtype: :class:`~certbot_apache.obj.VirtualHost` or None + :rtype: :class:`~certbot_apache._internal.obj.VirtualHost` or None :raises .errors.PluginError: If no VirtualHost is found """ @@ -1524,7 +1546,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): used for keeping track of VirtualHost directive over time. :param vhost: Virtual host to add the id - :type vhost: :class:`~certbot_apache.obj.VirtualHost` + :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :returns: The unique ID or None :rtype: str or None @@ -1546,7 +1568,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): If ID already exists, returns that instead. :param vhost: Virtual host to add or find the id - :type vhost: :class:`~certbot_apache.obj.VirtualHost` + :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :returns: The unique ID for vhost :rtype: str or None @@ -1584,9 +1606,9 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): :param str domain: domain to enhance :param str enhancement: enhancement type defined in - :const:`~certbot.constants.ENHANCEMENTS` + :const:`~certbot.plugins.enhancements.ENHANCEMENTS` :param options: options for the enhancement - See :const:`~certbot.constants.ENHANCEMENTS` + See :const:`~certbot.plugins.enhancements.ENHANCEMENTS` documentation for appropriate parameter. :raises .errors.PluginError: If Enhancement is not supported, or if @@ -1624,7 +1646,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """Increase the AutoHSTS max-age value :param vhost: Virtual host object to modify - :type vhost: :class:`~certbot_apache.obj.VirtualHost` + :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :param str id_str: The unique ID string of VirtualHost @@ -1645,7 +1667,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): if header_path: pat = '(?:[ "]|^)(strict-transport-security)(?:[ "]|$)' for match in header_path: - if re.search(pat, self.aug.get(match).lower()): + if re.search(pat, self.parser.aug.get(match).lower()): hsts_dirpath = match if not hsts_dirpath: err_msg = ("Certbot was unable to find the existing HSTS header " @@ -1659,7 +1681,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): # Our match statement was for string strict-transport-security, but # we need to update the value instead. The next index is for the value hsts_dirpath = hsts_dirpath.replace("arg[3]", "arg[4]") - self.aug.set(hsts_dirpath, hsts_maxage) + self.parser.aug.set(hsts_dirpath, hsts_maxage) note_msg = ("Increasing HSTS max-age value to {0} for VirtualHost " "in {1}\n".format(nextstep_value, vhost.filep)) logger.debug(note_msg) @@ -1708,13 +1730,13 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): .. note:: This function saves the configuration :param ssl_vhost: Destination of traffic, an ssl enabled vhost - :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost` + :type ssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :param unused_options: Not currently used :type unused_options: Not Available :returns: Success, general_vhost (HTTP vhost) - :rtype: (bool, :class:`~certbot_apache.obj.VirtualHost`) + :rtype: (bool, :class:`~certbot_apache._internal.obj.VirtualHost`) """ min_apache_ver = (2, 3, 3) @@ -1741,7 +1763,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): # We'll simply delete the directive, so that we'll have a # consistent OCSP cache path. if stapling_cache_aug_path: - self.aug.remove( + self.parser.aug.remove( re.sub(r"/\w*$", "", stapling_cache_aug_path[0])) self.parser.add_dir_to_ifmodssl(ssl_vhost_aug_path, @@ -1764,14 +1786,14 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): .. note:: This function saves the configuration :param ssl_vhost: Destination of traffic, an ssl enabled vhost - :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost` + :type ssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :param header_substring: string that uniquely identifies a header. e.g: Strict-Transport-Security, Upgrade-Insecure-Requests. :type str :returns: Success, general_vhost (HTTP vhost) - :rtype: (bool, :class:`~certbot_apache.obj.VirtualHost`) + :rtype: (bool, :class:`~certbot_apache._internal.obj.VirtualHost`) :raises .errors.PluginError: If no viable HTTP host can be created or set with header header_substring. @@ -1795,11 +1817,11 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): ssl_vhost.filep) def _verify_no_matching_http_header(self, ssl_vhost, header_substring): - """Checks to see if an there is an existing Header directive that + """Checks to see if there is an existing Header directive that contains the string header_substring. :param ssl_vhost: vhost to check - :type vhost: :class:`~certbot_apache.obj.VirtualHost` + :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :param header_substring: string that uniquely identifies a header. e.g: Strict-Transport-Security, Upgrade-Insecure-Requests. @@ -1818,7 +1840,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): # "Existing Header directive for virtualhost" pat = '(?:[ "]|^)(%s)(?:[ "]|$)' % (header_substring.lower()) for match in header_path: - if re.search(pat, self.aug.get(match).lower()): + if re.search(pat, self.parser.aug.get(match).lower()): raise errors.PluginEnhancementAlreadyPresent( "Existing %s header" % (header_substring)) @@ -1836,7 +1858,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): .. note:: This function saves the configuration :param ssl_vhost: Destination of traffic, an ssl enabled vhost - :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost` + :type ssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :param unused_options: Not currently used :type unused_options: Not Available @@ -1911,7 +1933,6 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): self.parser.add_dir(vhost.path, "RewriteRule", constants.REWRITE_HTTPS_ARGS) - def _verify_no_certbot_redirect(self, vhost): """Checks to see if a redirect was already installed by certbot. @@ -1922,7 +1943,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): delete certbot's old rewrite rules and set the new one instead. :param vhost: vhost to check - :type vhost: :class:`~certbot_apache.obj.VirtualHost` + :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :raises errors.PluginEnhancementAlreadyPresent: When the exact certbot redirection WriteRule exists in virtual host. @@ -1946,11 +1967,11 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): constants.REWRITE_HTTPS_ARGS_WITH_END] for dir_path, args_paths in rewrite_args_dict.items(): - arg_vals = [self.aug.get(x) for x in args_paths] + arg_vals = [self.parser.aug.get(x) for x in args_paths] # Search for past redirection rule, delete it, set the new one if arg_vals in constants.OLD_REWRITE_HTTPS_ARGS: - self.aug.remove(dir_path) + self.parser.aug.remove(dir_path) self._set_https_redirection_rewrite_rule(vhost) self.save() raise errors.PluginEnhancementAlreadyPresent( @@ -1964,7 +1985,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """Checks if there exists a RewriteRule directive in vhost :param vhost: vhost to check - :type vhost: :class:`~certbot_apache.obj.VirtualHost` + :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :returns: True if a RewriteRule directive exists. :rtype: bool @@ -1978,7 +1999,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """Checks if a RewriteEngine directive is on :param vhost: vhost to check - :type vhost: :class:`~certbot_apache.obj.VirtualHost` + :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost` """ rewrite_engine_path_list = self.parser.find_dir("RewriteEngine", "on", @@ -1995,10 +2016,10 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """Creates an http_vhost specifically to redirect for the ssl_vhost. :param ssl_vhost: ssl vhost - :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost` + :type ssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :returns: tuple of the form - (`success`, :class:`~certbot_apache.obj.VirtualHost`) + (`success`, :class:`~certbot_apache._internal.obj.VirtualHost`) :rtype: tuple """ @@ -2006,7 +2027,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): redirect_filepath = self._write_out_redirect(ssl_vhost, text) - self.aug.load() + self.parser.aug.load() # Make a new vhost data structure and add it to the lists new_vhost = self._create_vhost(parser.get_aug_path(self._escape(redirect_filepath))) self.vhosts.append(new_vhost) @@ -2124,7 +2145,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): of this method where available. :param vhost: vhost to enable - :type vhost: :class:`~certbot_apache.obj.VirtualHost` + :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :raises .errors.NotSupportedError: If filesystem layout is not supported. @@ -2142,7 +2163,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): vhost.enabled = True return - def enable_mod(self, mod_name, temp=False): # pylint: disable=unused-argument + def enable_mod(self, mod_name, temp=False): # pylint: disable=unused-argument """Enables module in Apache. Both enables and reloads Apache so module is active. @@ -2179,7 +2200,6 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): :raises .errors.MisconfigurationError: If reload fails """ - error = "" try: util.run_script(self.option("restart_cmd")) except errors.SubprocessError as err: @@ -2253,7 +2273,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): ########################################################################### def get_chall_pref(self, unused_domain): # pylint: disable=no-self-use """Return list of challenge preferences.""" - return [challenges.HTTP01, challenges.TLSSNI01] + return [challenges.HTTP01] def perform(self, achalls): """Perform the configuration related challenge. @@ -2266,20 +2286,15 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): self._chall_out.update(achalls) responses = [None] * len(achalls) http_doer = http_01.ApacheHttp01(self) - sni_doer = tls_sni_01.ApacheTlsSni01(self) for i, achall in enumerate(achalls): # Currently also have chall_doer hold associated index of the # challenge. This helps to put all of the responses back together # when they are all complete. - if isinstance(achall.chall, challenges.HTTP01): - http_doer.add_chall(achall, i) - else: # tls-sni-01 - sni_doer.add_chall(achall, i) + http_doer.add_chall(achall, i) http_response = http_doer.perform() - sni_response = sni_doer.perform() - if http_response or sni_response: + if http_response: # Must reload in order to activate the challenges. # Handled here because we may be able to load up other challenge # types @@ -2290,7 +2305,6 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): time.sleep(3) self._update_responses(responses, http_response, http_doer) - self._update_responses(responses, sni_response, sni_doer) return responses @@ -2325,7 +2339,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): Enable the AutoHSTS enhancement for defined domains :param _unused_lineage: Certificate lineage object, unused - :type _unused_lineage: certbot.storage.RenewableCert + :type _unused_lineage: certbot._internal.storage.RenewableCert :param domains: List of domains in certificate to enhance :type domains: str @@ -2368,7 +2382,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): """Do the initial AutoHSTS deployment to a vhost :param ssl_vhost: The VirtualHost object to deploy the AutoHSTS - :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost` or None + :type ssl_vhost: :class:`~certbot_apache._internal.obj.VirtualHost` or None :raises errors.PluginEnhancementAlreadyPresent: When already enhanced @@ -2450,7 +2464,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): and changes the HSTS max-age to a high value. :param lineage: Certificate lineage object - :type lineage: certbot.storage.RenewableCert + :type lineage: certbot._internal.storage.RenewableCert """ self._autohsts_fetch_state() if not self._autohsts: @@ -2495,4 +2509,4 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator): self._autohsts_save_state() -AutoHSTSEnhancement.register(ApacheConfigurator) # pylint: disable=no-member +AutoHSTSEnhancement.register(ApacheConfigurator) diff --git a/certbot-apache/certbot_apache/constants.py b/certbot-apache/certbot_apache/_internal/constants.py similarity index 80% rename from certbot-apache/certbot_apache/constants.py rename to certbot-apache/certbot_apache/_internal/constants.py index 23a7b7afd..a37bebac5 100644 --- a/certbot-apache/certbot_apache/constants.py +++ b/certbot-apache/certbot_apache/_internal/constants.py @@ -1,6 +1,7 @@ """Apache plugin constants.""" import pkg_resources +from certbot.compat import os MOD_SSL_CONF_DEST = "options-ssl-apache.conf" """Name of the mod_ssl config file as saved in `IConfig.config_dir`.""" @@ -9,6 +10,7 @@ MOD_SSL_CONF_DEST = "options-ssl-apache.conf" UPDATED_MOD_SSL_CONF_DIGEST = ".updated-options-ssl-apache-conf-digest.txt" """Name of the hash of the updated or informed mod_ssl_conf as saved in `IConfig.config_dir`.""" +# NEVER REMOVE A SINGLE HASH FROM THIS LIST UNLESS YOU KNOW EXACTLY WHAT YOU ARE DOING! ALL_SSL_OPTIONS_HASHES = [ '2086bca02db48daf93468332543c60ac6acdb6f0b58c7bfdf578a5d47092f82a', '4844d36c9a0f587172d9fa10f4f1c9518e3bcfa1947379f155e16a70a728c21a', @@ -18,11 +20,17 @@ ALL_SSL_OPTIONS_HASHES = [ 'cfdd7c18d2025836ea3307399f509cfb1ebf2612c87dd600a65da2a8e2f2797b', '80720bd171ccdc2e6b917ded340defae66919e4624962396b992b7218a561791', 'c0c022ea6b8a51ecc8f1003d0a04af6c3f2bc1c3ce506b3c2dfc1f11ef931082', + '717b0a89f5e4c39b09a42813ac6e747cfbdeb93439499e73f4f70a1fe1473f20', + '0fcdc81280cd179a07ec4d29d3595068b9326b455c488de4b09f585d5dafc137', + '86cc09ad5415cd6d5f09a947fe2501a9344328b1e8a8b458107ea903e80baa6c', + '06675349e457eae856120cdebb564efe546f0b87399f2264baeb41e442c724c7', + '5cc003edd93fb9cd03d40c7686495f8f058f485f75b5e764b789245a386e6daf', + '007cd497a56a3bb8b6a2c1aeb4997789e7e38992f74e44cc5d13a625a738ac73', ] """SHA256 hashes of the contents of previous versions of all versions of MOD_SSL_CONF_SRC""" AUGEAS_LENS_DIR = pkg_resources.resource_filename( - "certbot_apache", "augeas_lens") + "certbot_apache", os.path.join("_internal", "augeas_lens")) """Path to the Augeas lens directory""" REWRITE_HTTPS_ARGS = [ diff --git a/certbot-apache/certbot_apache/display_ops.py b/certbot-apache/certbot_apache/_internal/display_ops.py similarity index 95% rename from certbot-apache/certbot_apache/display_ops.py rename to certbot-apache/certbot_apache/_internal/display_ops.py index db1c3cca4..1ae32bb47 100644 --- a/certbot-apache/certbot_apache/display_ops.py +++ b/certbot-apache/certbot_apache/_internal/display_ops.py @@ -1,15 +1,13 @@ """Contains UI methods for Apache operations.""" import logging -import os import zope.component from certbot import errors from certbot import interfaces - +from certbot.compat import os import certbot.display.util as display_util - logger = logging.getLogger(__name__) @@ -26,7 +24,7 @@ def select_vhost_multiple(vhosts): return list() tags_list = [vhost.display_repr()+"\n" for vhost in vhosts] # Remove the extra newline from the last entry - if len(tags_list): + if tags_list: tags_list[-1] = tags_list[-1][:-1] code, names = zope.component.getUtility(interfaces.IDisplay).checklist( "Which VirtualHosts would you like to install the wildcard certificate for?", @@ -62,8 +60,7 @@ def select_vhost(domain, vhosts): code, tag = _vhost_menu(domain, vhosts) if code == display_util.OK: return vhosts[tag] - else: - return None + return None def _vhost_menu(domain, vhosts): """Select an appropriate Apache Vhost. @@ -80,7 +77,7 @@ def _vhost_menu(domain, vhosts): if free_chars < 2: logger.debug("Display size is too small for " - "certbot_apache.display_ops._vhost_menu()") + "certbot_apache._internal.display_ops._vhost_menu()") # This runs the edge off the screen, but it doesn't cause an "error" filename_size = 1 disp_name_size = 1 @@ -93,7 +90,7 @@ def _vhost_menu(domain, vhosts): for vhost in vhosts: if len(vhost.get_names()) == 1: disp_name = next(iter(vhost.get_names())) - elif len(vhost.get_names()) == 0: + elif not vhost.get_names(): disp_name = "" else: disp_name = "Multiple Names" diff --git a/certbot-apache/certbot_apache/entrypoint.py b/certbot-apache/certbot_apache/_internal/entrypoint.py similarity index 50% rename from certbot-apache/certbot_apache/entrypoint.py rename to certbot-apache/certbot_apache/_internal/entrypoint.py index 6f1443507..d43094976 100644 --- a/certbot-apache/certbot_apache/entrypoint.py +++ b/certbot-apache/certbot_apache/_internal/entrypoint.py @@ -1,23 +1,32 @@ """ Entry point for Apache Plugin """ -from certbot import util +# Pylint does not like disutils.version when running inside a venv. +# See: https://github.com/PyCQA/pylint/issues/73 +from distutils.version import LooseVersion # pylint: disable=no-name-in-module,import-error -from certbot_apache import configurator -from certbot_apache import override_arch -from certbot_apache import override_darwin -from certbot_apache import override_debian -from certbot_apache import override_centos -from certbot_apache import override_gentoo -from certbot_apache import override_suse +from certbot import util +from certbot_apache._internal import configurator +from certbot_apache._internal import override_arch +from certbot_apache._internal import override_centos +from certbot_apache._internal import override_darwin +from certbot_apache._internal import override_debian +from certbot_apache._internal import override_fedora +from certbot_apache._internal import override_gentoo +from certbot_apache._internal import override_suse OVERRIDE_CLASSES = { "arch": override_arch.ArchConfigurator, + "cloudlinux": override_centos.CentOSConfigurator, "darwin": override_darwin.DarwinConfigurator, "debian": override_debian.DebianConfigurator, "ubuntu": override_debian.DebianConfigurator, "centos": override_centos.CentOSConfigurator, "centos linux": override_centos.CentOSConfigurator, - "fedora": override_centos.CentOSConfigurator, + "fedora_old": override_centos.CentOSConfigurator, + "fedora": override_fedora.FedoraConfigurator, + "linuxmint": override_debian.DebianConfigurator, "ol": override_centos.CentOSConfigurator, + "oracle": override_centos.CentOSConfigurator, + "redhatenterpriseserver": override_centos.CentOSConfigurator, "red hat enterprise linux server": override_centos.CentOSConfigurator, "rhel": override_centos.CentOSConfigurator, "amazon": override_centos.CentOSConfigurator, @@ -25,14 +34,24 @@ OVERRIDE_CLASSES = { "gentoo base system": override_gentoo.GentooConfigurator, "opensuse": override_suse.OpenSUSEConfigurator, "suse": override_suse.OpenSUSEConfigurator, + "sles": override_suse.OpenSUSEConfigurator, + "scientific": override_centos.CentOSConfigurator, + "scientific linux": override_centos.CentOSConfigurator, } + def get_configurator(): """ Get correct configurator class based on the OS fingerprint """ - os_info = util.get_os_info() + os_name, os_version = util.get_os_info() + os_name = os_name.lower() override_class = None + + # Special case for older Fedora versions + if os_name == 'fedora' and LooseVersion(os_version) < LooseVersion('29'): + os_name = 'fedora_old' + try: - override_class = OVERRIDE_CLASSES[os_info[0].lower()] + override_class = OVERRIDE_CLASSES[os_name] except KeyError: # OS not found in the list os_like = util.get_systemd_os_like() @@ -45,4 +64,5 @@ def get_configurator(): override_class = configurator.ApacheConfigurator return override_class + ENTRYPOINT = get_configurator() diff --git a/certbot-apache/certbot_apache/http_01.py b/certbot-apache/certbot_apache/_internal/http_01.py similarity index 75% rename from certbot-apache/certbot_apache/http_01.py rename to certbot-apache/certbot_apache/_internal/http_01.py index 22598baca..c34abc2b4 100644 --- a/certbot-apache/certbot_apache/http_01.py +++ b/certbot-apache/certbot_apache/_internal/http_01.py @@ -1,16 +1,19 @@ """A class that performs HTTP-01 challenges for Apache""" import logging -import os +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module from certbot import errors +from certbot.compat import filesystem +from certbot.compat import os from certbot.plugins import common -from certbot_apache.obj import VirtualHost # pylint: disable=unused-import -from certbot_apache.parser import get_aug_path +from certbot_apache._internal.obj import VirtualHost # pylint: disable=unused-import +from certbot_apache._internal.parser import get_aug_path logger = logging.getLogger(__name__) -class ApacheHttp01(common.TLSSNI01): + +class ApacheHttp01(common.ChallengePerformer): """Class that performs HTTP-01 challenges within the Apache configurator.""" CONFIG_TEMPLATE22_PRE = """\ @@ -89,15 +92,27 @@ class ApacheHttp01(common.TLSSNI01): self.configurator.enable_mod(mod, temp=True) def _mod_config(self): + selected_vhosts = [] # type: List[VirtualHost] + http_port = str(self.configurator.config.http01_port) for chall in self.achalls: - vh = self.configurator.find_best_http_vhost( - chall.domain, filter_defaults=False, - port=str(self.configurator.config.http01_port)) - if vh: - self._set_up_include_directives(vh) - else: - for vh in self._relevant_vhosts(): - self._set_up_include_directives(vh) + # Search for matching VirtualHosts + for vh in self._matching_vhosts(chall.domain): + selected_vhosts.append(vh) + + # Ensure that we have one or more VirtualHosts that we can continue + # with. (one that listens to port configured with --http-01-port) + found = False + for vhost in selected_vhosts: + if any(a.is_wildcard() or a.get_port() == http_port for a in vhost.addrs): + found = True + + if not found: + for vh in self._relevant_vhosts(): + selected_vhosts.append(vh) + + # Add the challenge configuration + for vh in selected_vhosts: + self._set_up_include_directives(vh) self.configurator.reverter.register_file_creation( True, self.challenge_conf_pre) @@ -121,6 +136,20 @@ class ApacheHttp01(common.TLSSNI01): with open(self.challenge_conf_post, "w") as new_conf: new_conf.write(config_text_post) + def _matching_vhosts(self, domain): + """Return all VirtualHost objects that have the requested domain name or + a wildcard name that would match the domain in ServerName or ServerAlias + directive. + """ + matching_vhosts = [] + for vhost in self.configurator.vhosts: + if self.configurator.domain_in_names(vhost.get_names(), domain): + # domain_in_names also matches the exact names, so no need + # to check "domain in vhost.get_names()" explicitly here + matching_vhosts.append(vhost) + + return matching_vhosts + def _relevant_vhosts(self): http01_port = str(self.configurator.config.http01_port) relevant_vhosts = [] @@ -139,8 +168,7 @@ class ApacheHttp01(common.TLSSNI01): def _set_up_challenges(self): if not os.path.isdir(self.challenge_dir): - os.makedirs(self.challenge_dir) - os.chmod(self.challenge_dir, 0o755) + filesystem.makedirs(self.challenge_dir, 0o755) responses = [] for achall in self.achalls: @@ -156,7 +184,7 @@ class ApacheHttp01(common.TLSSNI01): self.configurator.reverter.register_file_creation(True, name) with open(name, 'wb') as f: f.write(validation.encode()) - os.chmod(name, 0o644) + filesystem.chmod(name, 0o644) return response @@ -166,8 +194,8 @@ class ApacheHttp01(common.TLSSNI01): if vhost not in self.moded_vhosts: logger.debug( - "Adding a temporary challenge validation Include for name: %s " + - "in: %s", vhost.name, vhost.filep) + "Adding a temporary challenge validation Include for name: %s in: %s", + vhost.name, vhost.filep) self.configurator.parser.add_dir_beginning( vhost.path, "Include", self.challenge_conf_pre) self.configurator.parser.add_dir( diff --git a/certbot-apache/certbot_apache/obj.py b/certbot-apache/certbot_apache/_internal/obj.py similarity index 96% rename from certbot-apache/certbot_apache/obj.py rename to certbot-apache/certbot_apache/_internal/obj.py index 290979f27..8b3aeb376 100644 --- a/certbot-apache/certbot_apache/obj.py +++ b/certbot-apache/certbot_apache/_internal/obj.py @@ -1,7 +1,7 @@ """Module contains classes used by the Apache Configurator.""" import re -from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module from certbot.plugins import common @@ -24,9 +24,9 @@ class Addr(common.Addr): return not self.__eq__(other) def __repr__(self): - return "certbot_apache.obj.Addr(" + repr(self.tup) + ")" + return "certbot_apache._internal.obj.Addr(" + repr(self.tup) + ")" - def __hash__(self): + def __hash__(self): # pylint: disable=useless-super-delegation # Python 3 requires explicit overridden for __hash__ if __eq__ or # __cmp__ is overridden. See https://bugs.python.org/issue2235 return super(Addr, self).__hash__() @@ -47,8 +47,7 @@ class Addr(common.Addr): return 0 elif self.get_addr() == "*": return 1 - else: - return 2 + return 2 def conflicts(self, addr): r"""Returns if address could conflict with correct function of self. @@ -99,7 +98,7 @@ class Addr(common.Addr): return self.get_addr_obj(port) -class VirtualHost(object): # pylint: disable=too-few-public-methods +class VirtualHost(object): """Represents an Apache Virtualhost. :ivar str filep: file path of VH @@ -127,7 +126,6 @@ class VirtualHost(object): # pylint: disable=too-few-public-methods def __init__(self, filep, path, addrs, ssl, enabled, name=None, aliases=None, modmacro=False, ancestor=None): - # pylint: disable=too-many-arguments """Initialize a VH.""" self.filep = filep self.path = path diff --git a/certbot-apache/certbot_apache/_internal/options-ssl-apache.conf b/certbot-apache/certbot_apache/_internal/options-ssl-apache.conf new file mode 100644 index 000000000..1a3799628 --- /dev/null +++ b/certbot-apache/certbot_apache/_internal/options-ssl-apache.conf @@ -0,0 +1,18 @@ +# This file contains important security parameters. If you modify this file +# manually, Certbot will be unable to automatically provide future security +# updates. Instead, Certbot will print and log an error message with a path to +# the up-to-date file that you will need to refer to when manually updating +# this file. + +SSLEngine on + +# Intermediate configuration, tweak to your needs +SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1 +SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 +SSLHonorCipherOrder off + +SSLOptions +StrictRequire + +# Add vhost name to log entries: +LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined +LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common diff --git a/certbot-apache/certbot_apache/override_arch.py b/certbot-apache/certbot_apache/_internal/override_arch.py similarity index 84% rename from certbot-apache/certbot_apache/override_arch.py rename to certbot-apache/certbot_apache/_internal/override_arch.py index c5620e9f9..2765bd238 100644 --- a/certbot-apache/certbot_apache/override_arch.py +++ b/certbot-apache/certbot_apache/_internal/override_arch.py @@ -1,11 +1,11 @@ """ Distribution specific override class for Arch Linux """ import pkg_resources - import zope.interface from certbot import interfaces +from certbot.compat import os +from certbot_apache._internal import configurator -from certbot_apache import configurator @zope.interface.provider(interfaces.IPluginFactory) class ArchConfigurator(configurator.ApacheConfigurator): @@ -27,5 +27,5 @@ class ArchConfigurator(configurator.ApacheConfigurator): handle_sites=False, challenge_location="/etc/httpd/conf", MOD_SSL_CONF_SRC=pkg_resources.resource_filename( - "certbot_apache", "options-ssl-apache.conf") + "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf")) ) diff --git a/certbot-apache/certbot_apache/_internal/override_centos.py b/certbot-apache/certbot_apache/_internal/override_centos.py new file mode 100644 index 000000000..a3ef2d760 --- /dev/null +++ b/certbot-apache/certbot_apache/_internal/override_centos.py @@ -0,0 +1,215 @@ +""" Distribution specific override class for CentOS family (RHEL, Fedora) """ +import logging + +import pkg_resources +import zope.interface + +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from certbot import errors +from certbot import interfaces +from certbot import util +from certbot.compat import os +from certbot.errors import MisconfigurationError +from certbot_apache._internal import apache_util +from certbot_apache._internal import configurator +from certbot_apache._internal import parser + +logger = logging.getLogger(__name__) + + +@zope.interface.provider(interfaces.IPluginFactory) +class CentOSConfigurator(configurator.ApacheConfigurator): + """CentOS specific ApacheConfigurator override class""" + + OS_DEFAULTS = dict( + server_root="/etc/httpd", + vhost_root="/etc/httpd/conf.d", + vhost_files="*.conf", + logs_root="/var/log/httpd", + ctl="apachectl", + version_cmd=['apachectl', '-v'], + restart_cmd=['apachectl', 'graceful'], + restart_cmd_alt=['apachectl', 'restart'], + conftest_cmd=['apachectl', 'configtest'], + enmod=None, + dismod=None, + le_vhost_ext="-le-ssl.conf", + handle_modules=False, + handle_sites=False, + challenge_location="/etc/httpd/conf.d", + MOD_SSL_CONF_SRC=pkg_resources.resource_filename( + "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf")) + ) + + def config_test(self): + """ + Override config_test to mitigate configtest error in vanilla installation + of mod_ssl in Fedora. The error is caused by non-existent self-signed + certificates referenced by the configuration, that would be autogenerated + during the first (re)start of httpd. + """ + + os_info = util.get_os_info() + fedora = os_info[0].lower() == "fedora" + + try: + super(CentOSConfigurator, self).config_test() + except errors.MisconfigurationError: + if fedora: + self._try_restart_fedora() + else: + raise + + def _try_restart_fedora(self): + """ + Tries to restart httpd using systemctl to generate the self signed keypair. + """ + + try: + util.run_script(['systemctl', 'restart', 'httpd']) + except errors.SubprocessError as err: + raise errors.MisconfigurationError(str(err)) + + # Finish with actual config check to see if systemctl restart helped + super(CentOSConfigurator, self).config_test() + + def _prepare_options(self): + """ + Override the options dictionary initialization in order to support + alternative restart cmd used in CentOS. + """ + super(CentOSConfigurator, self)._prepare_options() + self.options["restart_cmd_alt"][0] = self.option("ctl") + + def get_parser(self): + """Initializes the ApacheParser""" + return CentOSParser( + self.option("server_root"), self.option("vhost_root"), + self.version, configurator=self) + + def _deploy_cert(self, *args, **kwargs): # pylint: disable=arguments-differ + """ + Override _deploy_cert in order to ensure that the Apache configuration + has "LoadModule ssl_module..." before parsing the VirtualHost configuration + that was created by Certbot + """ + super(CentOSConfigurator, self)._deploy_cert(*args, **kwargs) + if self.version < (2, 4, 0): + self._deploy_loadmodule_ssl_if_needed() + + def _deploy_loadmodule_ssl_if_needed(self): + """ + Add "LoadModule ssl_module " to main httpd.conf if + it doesn't exist there already. + """ + + loadmods = self.parser.find_dir("LoadModule", "ssl_module", exclude=False) + + correct_ifmods = [] # type: List[str] + loadmod_args = [] # type: List[str] + loadmod_paths = [] # type: List[str] + for m in loadmods: + noarg_path = m.rpartition("/")[0] + path_args = self.parser.get_all_args(noarg_path) + if loadmod_args: + if loadmod_args != path_args: + msg = ("Certbot encountered multiple LoadModule directives " + "for LoadModule ssl_module with differing library paths. " + "Please remove or comment out the one(s) that are not in " + "use, and run Certbot again.") + raise MisconfigurationError(msg) + else: + loadmod_args = path_args + + if self.parser.not_modssl_ifmodule(noarg_path): # pylint: disable=no-member + if self.parser.loc["default"] in noarg_path: + # LoadModule already in the main configuration file + if ("ifmodule/" in noarg_path.lower() or + "ifmodule[1]" in noarg_path.lower()): + # It's the first or only IfModule in the file + return + # Populate the list of known !mod_ssl.c IfModules + nodir_path = noarg_path.rpartition("/directive")[0] + correct_ifmods.append(nodir_path) + else: + loadmod_paths.append(noarg_path) + + if not loadmod_args: + # Do not try to enable mod_ssl + return + + # Force creation as the directive wasn't found from the beginning of + # httpd.conf + rootconf_ifmod = self.parser.create_ifmod( + parser.get_aug_path(self.parser.loc["default"]), + "!mod_ssl.c", beginning=True) + # parser.get_ifmod returns a path postfixed with "/", remove that + self.parser.add_dir(rootconf_ifmod[:-1], "LoadModule", loadmod_args) + correct_ifmods.append(rootconf_ifmod[:-1]) + self.save_notes += "Added LoadModule ssl_module to main configuration.\n" + + # Wrap LoadModule mod_ssl inside of if it's not + # configured like this already. + for loadmod_path in loadmod_paths: + nodir_path = loadmod_path.split("/directive")[0] + # Remove the old LoadModule directive + self.parser.aug.remove(loadmod_path) + + # Create a new IfModule !mod_ssl.c if not already found on path + ssl_ifmod = self.parser.get_ifmod(nodir_path, "!mod_ssl.c", + beginning=True)[:-1] + if ssl_ifmod not in correct_ifmods: + self.parser.add_dir(ssl_ifmod, "LoadModule", loadmod_args) + correct_ifmods.append(ssl_ifmod) + self.save_notes += ("Wrapped pre-existing LoadModule ssl_module " + "inside of block.\n") + + +class CentOSParser(parser.ApacheParser): + """CentOS specific ApacheParser override class""" + def __init__(self, *args, **kwargs): + # CentOS specific configuration file for Apache + self.sysconfig_filep = "/etc/sysconfig/httpd" + super(CentOSParser, self).__init__(*args, **kwargs) + + def update_runtime_variables(self): + """ Override for update_runtime_variables for custom parsing """ + # Opportunistic, works if SELinux not enforced + super(CentOSParser, self).update_runtime_variables() + self.parse_sysconfig_var() + + def parse_sysconfig_var(self): + """ Parses Apache CLI options from CentOS configuration file """ + defines = apache_util.parse_define_file(self.sysconfig_filep, "OPTIONS") + for k in defines: + self.variables[k] = defines[k] + + def not_modssl_ifmodule(self, path): + """Checks if the provided Augeas path has argument !mod_ssl""" + + if "ifmodule" not in path.lower(): + return False + + # Trim the path to the last ifmodule + workpath = path.lower() + while workpath: + # Get path to the last IfModule (ignore the tail) + parts = workpath.rpartition("ifmodule") + + if not parts[0]: + # IfModule not found + break + ifmod_path = parts[0] + parts[1] + # Check if ifmodule had an index + if parts[2].startswith("["): + # Append the index from tail + ifmod_path += parts[2].partition("/")[0] + # Get the original path trimmed to correct length + # This is required to preserve cases + ifmod_real_path = path[0:len(ifmod_path)] + if "!mod_ssl.c" in self.get_all_args(ifmod_real_path): + return True + # Set the workpath to the heading part + workpath = parts[0] + + return False diff --git a/certbot-apache/certbot_apache/override_darwin.py b/certbot-apache/certbot_apache/_internal/override_darwin.py similarity index 84% rename from certbot-apache/certbot_apache/override_darwin.py rename to certbot-apache/certbot_apache/_internal/override_darwin.py index 4e2a6acac..00faff623 100644 --- a/certbot-apache/certbot_apache/override_darwin.py +++ b/certbot-apache/certbot_apache/_internal/override_darwin.py @@ -1,11 +1,11 @@ """ Distribution specific override class for macOS """ import pkg_resources - import zope.interface from certbot import interfaces +from certbot.compat import os +from certbot_apache._internal import configurator -from certbot_apache import configurator @zope.interface.provider(interfaces.IPluginFactory) class DarwinConfigurator(configurator.ApacheConfigurator): @@ -27,5 +27,5 @@ class DarwinConfigurator(configurator.ApacheConfigurator): handle_sites=False, challenge_location="/etc/apache2/other", MOD_SSL_CONF_SRC=pkg_resources.resource_filename( - "certbot_apache", "options-ssl-apache.conf") + "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf")) ) diff --git a/certbot-apache/certbot_apache/override_debian.py b/certbot-apache/certbot_apache/_internal/override_debian.py similarity index 83% rename from certbot-apache/certbot_apache/override_debian.py rename to certbot-apache/certbot_apache/_internal/override_debian.py index 0caa619d2..77ced6a3f 100644 --- a/certbot-apache/certbot_apache/override_debian.py +++ b/certbot-apache/certbot_apache/_internal/override_debian.py @@ -1,19 +1,20 @@ """ Distribution specific override class for Debian family (Ubuntu/Debian) """ import logging -import os -import pkg_resources +import pkg_resources import zope.interface from certbot import errors from certbot import interfaces from certbot import util - -from certbot_apache import apache_util -from certbot_apache import configurator +from certbot.compat import filesystem +from certbot.compat import os +from certbot_apache._internal import apache_util +from certbot_apache._internal import configurator logger = logging.getLogger(__name__) + @zope.interface.provider(interfaces.IPluginFactory) class DebianConfigurator(configurator.ApacheConfigurator): """Debian specific ApacheConfigurator override class""" @@ -34,7 +35,7 @@ class DebianConfigurator(configurator.ApacheConfigurator): handle_sites=True, challenge_location="/etc/apache2", MOD_SSL_CONF_SRC=pkg_resources.resource_filename( - "certbot_apache", "options-ssl-apache.conf") + "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf")) ) def enable_site(self, vhost): @@ -44,14 +45,14 @@ class DebianConfigurator(configurator.ApacheConfigurator): modules are enabled appropriately. :param vhost: vhost to enable - :type vhost: :class:`~certbot_apache.obj.VirtualHost` + :type vhost: :class:`~certbot_apache._internal.obj.VirtualHost` :raises .errors.NotSupportedError: If filesystem layout is not supported. """ if vhost.enabled: - return + return None enabled_path = ("%s/sites-enabled/%s" % (self.parser.root, @@ -64,26 +65,25 @@ class DebianConfigurator(configurator.ApacheConfigurator): try: os.symlink(vhost.filep, enabled_path) except OSError as err: - if os.path.islink(enabled_path) and os.path.realpath( + if os.path.islink(enabled_path) and filesystem.realpath( enabled_path) == vhost.filep: # Already in shape vhost.enabled = True - return - else: - logger.warning( - "Could not symlink %s to %s, got error: %s", enabled_path, - vhost.filep, err.strerror) - errstring = ("Encountered error while trying to enable a " + - "newly created VirtualHost located at {0} by " + - "linking to it from {1}") - raise errors.NotSupportedError(errstring.format(vhost.filep, - enabled_path)) + return None + logger.warning( + "Could not symlink %s to %s, got error: %s", enabled_path, + vhost.filep, err.strerror) + errstring = ("Encountered error while trying to enable a " + + "newly created VirtualHost located at {0} by " + + "linking to it from {1}") + raise errors.NotSupportedError(errstring.format(vhost.filep, + enabled_path)) vhost.enabled = True logger.info("Enabling available site: %s", vhost.filep) self.save_notes += "Enabled site %s\n" % vhost.filep + return None def enable_mod(self, mod_name, temp=False): - # pylint: disable=unused-argument """Enables module in Apache. Both enables and reloads Apache so module is active. diff --git a/certbot-apache/certbot_apache/_internal/override_fedora.py b/certbot-apache/certbot_apache/_internal/override_fedora.py new file mode 100644 index 000000000..8197b0dcd --- /dev/null +++ b/certbot-apache/certbot_apache/_internal/override_fedora.py @@ -0,0 +1,98 @@ +""" Distribution specific override class for Fedora 29+ """ +import pkg_resources +import zope.interface + +from certbot import errors +from certbot import interfaces +from certbot import util +from certbot.compat import os +from certbot_apache._internal import apache_util +from certbot_apache._internal import configurator +from certbot_apache._internal import parser + + +@zope.interface.provider(interfaces.IPluginFactory) +class FedoraConfigurator(configurator.ApacheConfigurator): + """Fedora 29+ specific ApacheConfigurator override class""" + + OS_DEFAULTS = dict( + server_root="/etc/httpd", + vhost_root="/etc/httpd/conf.d", + vhost_files="*.conf", + logs_root="/var/log/httpd", + ctl="httpd", + version_cmd=['httpd', '-v'], + restart_cmd=['apachectl', 'graceful'], + restart_cmd_alt=['apachectl', 'restart'], + conftest_cmd=['apachectl', 'configtest'], + enmod=None, + dismod=None, + le_vhost_ext="-le-ssl.conf", + handle_modules=False, + handle_sites=False, + challenge_location="/etc/httpd/conf.d", + MOD_SSL_CONF_SRC=pkg_resources.resource_filename( + # TODO: eventually newest version of Fedora will need their own config + "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf")) + ) + + def config_test(self): + """ + Override config_test to mitigate configtest error in vanilla installation + of mod_ssl in Fedora. The error is caused by non-existent self-signed + certificates referenced by the configuration, that would be autogenerated + during the first (re)start of httpd. + """ + try: + super(FedoraConfigurator, self).config_test() + except errors.MisconfigurationError: + self._try_restart_fedora() + + def get_parser(self): + """Initializes the ApacheParser""" + return FedoraParser( + self.option("server_root"), self.option("vhost_root"), + self.version, configurator=self) + + def _try_restart_fedora(self): + """ + Tries to restart httpd using systemctl to generate the self signed keypair. + """ + try: + util.run_script(['systemctl', 'restart', 'httpd']) + except errors.SubprocessError as err: + raise errors.MisconfigurationError(str(err)) + + # Finish with actual config check to see if systemctl restart helped + super(FedoraConfigurator, self).config_test() + + def _prepare_options(self): + """ + Override the options dictionary initialization to keep using apachectl + instead of httpd and so take advantages of this new bash script in newer versions + of Fedora to restart httpd. + """ + super(FedoraConfigurator, self)._prepare_options() + self.options["restart_cmd"][0] = 'apachectl' + self.options["restart_cmd_alt"][0] = 'apachectl' + self.options["conftest_cmd"][0] = 'apachectl' + + +class FedoraParser(parser.ApacheParser): + """Fedora 29+ specific ApacheParser override class""" + def __init__(self, *args, **kwargs): + # Fedora 29+ specific configuration file for Apache + self.sysconfig_filep = "/etc/sysconfig/httpd" + super(FedoraParser, self).__init__(*args, **kwargs) + + def update_runtime_variables(self): + """ Override for update_runtime_variables for custom parsing """ + # Opportunistic, works if SELinux not enforced + super(FedoraParser, self).update_runtime_variables() + self._parse_sysconfig_var() + + def _parse_sysconfig_var(self): + """ Parses Apache CLI options from Fedora configuration file """ + defines = apache_util.parse_define_file(self.sysconfig_filep, "OPTIONS") + for k in defines: + self.variables[k] = defines[k] diff --git a/certbot-apache/certbot_apache/override_gentoo.py b/certbot-apache/certbot_apache/_internal/override_gentoo.py similarity index 87% rename from certbot-apache/certbot_apache/override_gentoo.py rename to certbot-apache/certbot_apache/_internal/override_gentoo.py index 556e3225e..38f8aebe9 100644 --- a/certbot-apache/certbot_apache/override_gentoo.py +++ b/certbot-apache/certbot_apache/_internal/override_gentoo.py @@ -1,13 +1,13 @@ """ Distribution specific override class for Gentoo Linux """ import pkg_resources - import zope.interface from certbot import interfaces +from certbot.compat import os +from certbot_apache._internal import apache_util +from certbot_apache._internal import configurator +from certbot_apache._internal import parser -from certbot_apache import apache_util -from certbot_apache import configurator -from certbot_apache import parser @zope.interface.provider(interfaces.IPluginFactory) class GentooConfigurator(configurator.ApacheConfigurator): @@ -30,7 +30,7 @@ class GentooConfigurator(configurator.ApacheConfigurator): handle_sites=False, challenge_location="/etc/apache2/vhosts.d", MOD_SSL_CONF_SRC=pkg_resources.resource_filename( - "certbot_apache", "options-ssl-apache.conf") + "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf")) ) def _prepare_options(self): @@ -44,7 +44,7 @@ class GentooConfigurator(configurator.ApacheConfigurator): def get_parser(self): """Initializes the ApacheParser""" return GentooParser( - self.aug, self.option("server_root"), self.option("vhost_root"), + self.option("server_root"), self.option("vhost_root"), self.version, configurator=self) @@ -64,7 +64,7 @@ class GentooParser(parser.ApacheParser): """ Parses Apache CLI options from Gentoo configuration file """ defines = apache_util.parse_define_file(self.apacheconfig_filep, "APACHE2_OPTS") - for k in defines.keys(): + for k in defines: self.variables[k] = defines[k] def update_modules(self): diff --git a/certbot-apache/certbot_apache/override_suse.py b/certbot-apache/certbot_apache/_internal/override_suse.py similarity index 84% rename from certbot-apache/certbot_apache/override_suse.py rename to certbot-apache/certbot_apache/_internal/override_suse.py index 3d0043afe..0c9219e6d 100644 --- a/certbot-apache/certbot_apache/override_suse.py +++ b/certbot-apache/certbot_apache/_internal/override_suse.py @@ -1,11 +1,11 @@ """ Distribution specific override class for OpenSUSE """ import pkg_resources - import zope.interface from certbot import interfaces +from certbot.compat import os +from certbot_apache._internal import configurator -from certbot_apache import configurator @zope.interface.provider(interfaces.IPluginFactory) class OpenSUSEConfigurator(configurator.ApacheConfigurator): @@ -27,5 +27,5 @@ class OpenSUSEConfigurator(configurator.ApacheConfigurator): handle_sites=False, challenge_location="/etc/apache2/vhosts.d", MOD_SSL_CONF_SRC=pkg_resources.resource_filename( - "certbot_apache", "options-ssl-apache.conf") + "certbot_apache", os.path.join("_internal", "options-ssl-apache.conf")) ) diff --git a/certbot-apache/certbot_apache/parser.py b/certbot-apache/certbot_apache/_internal/parser.py similarity index 76% rename from certbot-apache/certbot_apache/parser.py rename to certbot-apache/certbot_apache/_internal/parser.py index 148f052d0..0703b8fb5 100644 --- a/certbot-apache/certbot_apache/parser.py +++ b/certbot-apache/certbot_apache/_internal/parser.py @@ -2,21 +2,23 @@ import copy import fnmatch import logging -import os import re import subprocess import sys import six -from acme.magic_typing import Dict, List, Set # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module from certbot import errors +from certbot.compat import os +from certbot_apache._internal import constants logger = logging.getLogger(__name__) class ApacheParser(object): - # pylint: disable=too-many-public-methods """Class handles the fine details of parsing the Apache Configuration. .. todo:: Make parsing general... remove sites-available etc... @@ -31,7 +33,7 @@ class ApacheParser(object): arg_var_interpreter = re.compile(r"\$\{[^ \}]*}") fnmatch_chars = set(["*", "?", "\\", "[", "]"]) - def __init__(self, aug, root, vhostroot=None, version=(2, 4), + def __init__(self, root, vhostroot=None, version=(2, 4), configurator=None): # Note: Order is important here. @@ -40,11 +42,20 @@ class ApacheParser(object): # issues with aug.load() after adding new files / defines to parse tree self.configurator = configurator + # Initialize augeas + self.aug = None + self.init_augeas() + + if not self.check_aug_version(): + raise errors.NotSupportedError( + "Apache plugin support requires libaugeas0 and augeas-lenses " + "version 1.2.0 or higher, please make sure you have you have " + "those installed.") + self.modules = set() # type: Set[str] self.parser_paths = {} # type: Dict[str, List[str]] self.variables = {} # type: Dict[str, str] - self.aug = aug # Find configuration root and make sure augeas can parse it. self.root = os.path.abspath(root) self.loc = {"root": self._find_config_root()} @@ -76,6 +87,146 @@ class ApacheParser(object): if self.find_dir("Define", exclude=False): raise errors.PluginError("Error parsing runtime variables") + def init_augeas(self): + """ Initialize the actual Augeas instance """ + + try: + import augeas + except ImportError: # pragma: no cover + raise errors.NoInstallationError("Problem in Augeas installation") + + self.aug = augeas.Augeas( + # specify a directory to load our preferred lens from + loadpath=constants.AUGEAS_LENS_DIR, + # Do not save backup (we do it ourselves), do not load + # anything by default + flags=(augeas.Augeas.NONE | + augeas.Augeas.NO_MODL_AUTOLOAD | + augeas.Augeas.ENABLE_SPAN)) + + def check_parsing_errors(self, lens): + """Verify Augeas can parse all of the lens files. + + :param str lens: lens to check for errors + + :raises .errors.PluginError: If there has been an error in parsing with + the specified lens. + + """ + error_files = self.aug.match("/augeas//error") + + for path in error_files: + # Check to see if it was an error resulting from the use of + # the httpd lens + lens_path = self.aug.get(path + "/lens") + # As aug.get may return null + if lens_path and lens in lens_path: + msg = ( + "There has been an error in parsing the file {0} on line {1}: " + "{2}".format( + # Strip off /augeas/files and /error + path[13:len(path) - 6], + self.aug.get(path + "/line"), + self.aug.get(path + "/message"))) + raise errors.PluginError(msg) + + def check_aug_version(self): + """ Checks that we have recent enough version of libaugeas. + If augeas version is recent enough, it will support case insensitive + regexp matching""" + + self.aug.set("/test/path/testing/arg", "aRgUMeNT") + try: + matches = self.aug.match( + "/test//*[self::arg=~regexp('argument', 'i')]") + except RuntimeError: + self.aug.remove("/test/path") + return False + self.aug.remove("/test/path") + return matches + + def unsaved_files(self): + """Lists files that have modified Augeas DOM but the changes have not + been written to the filesystem yet, used by `self.save()` and + ApacheConfigurator to check the file state. + + :raises .errors.PluginError: If there was an error in Augeas, in + an attempt to save the configuration, or an error creating a + checkpoint + + :returns: `set` of unsaved files + """ + save_state = self.aug.get("/augeas/save") + self.aug.set("/augeas/save", "noop") + # Existing Errors + ex_errs = self.aug.match("/augeas//error") + try: + # This is a noop save + self.aug.save() + except (RuntimeError, IOError): + self._log_save_errors(ex_errs) + # Erase Save Notes + self.configurator.save_notes = "" + raise errors.PluginError( + "Error saving files, check logs for more info.") + + # Return the original save method + self.aug.set("/augeas/save", save_state) + + # Retrieve list of modified files + # Note: Noop saves can cause the file to be listed twice, I used a + # set to remove this possibility. This is a known augeas 0.10 error. + save_paths = self.aug.match("/augeas/events/saved") + + save_files = set() + if save_paths: + for path in save_paths: + save_files.add(self.aug.get(path)[6:]) + return save_files + + def ensure_augeas_state(self): + """Makes sure that all Augeas dom changes are written to files to avoid + loss of configuration directives when doing additional augeas parsing, + causing a possible augeas.load() resulting dom reset + """ + + if self.unsaved_files(): + self.configurator.save_notes += "(autosave)" + self.configurator.save() + + def save(self, save_files): + """Saves all changes to the configuration files. + + save() is called from ApacheConfigurator to handle the parser specific + tasks of saving. + + :param list save_files: list of strings of file paths that we need to save. + + """ + self.configurator.save_notes = "" + self.aug.save() + + # Force reload if files were modified + # This is needed to recalculate augeas directive span + if save_files: + for sf in save_files: + self.aug.remove("/files/"+sf) + self.aug.load() + + def _log_save_errors(self, ex_errs): + """Log errors due to bad Augeas save. + + :param list ex_errs: Existing errors before save + + """ + # Check for the root of save problems + new_errs = self.aug.match("/augeas//error") + # logger.error("During Save - %s", mod_conf) + logger.error("Unable to save files: %s. Attempted Save Notes: %s", + ", ".join(err[13:len(err) - 6] for err in new_errs + # Only new errors caused by recent save + if err not in ex_errs), self.configurator.save_notes) + def add_include(self, main_config, inc_path): """Add Include for a new configuration file if one does not exist @@ -83,7 +234,7 @@ class ApacheParser(object): :param str inc_path: path of file to include """ - if len(self.find_dir(case_i("Include"), inc_path)) == 0: + if not self.find_dir(case_i("Include"), inc_path): logger.debug("Adding Include %s to %s", inc_path, get_aug_path(main_config)) self.add_dir( @@ -93,12 +244,7 @@ class ApacheParser(object): # Add new path to parser paths new_dir = os.path.dirname(inc_path) new_file = os.path.basename(inc_path) - if new_dir in self.existing_paths.keys(): - # Add to existing path - self.existing_paths[new_dir].append(new_file) - else: - # Create a new path - self.existing_paths[new_dir] = [new_file] + self.existing_paths.setdefault(new_dir, []).append(new_file) def add_mod(self, mod_name): """Shortcut for updating parser modules.""" @@ -138,8 +284,8 @@ class ApacheParser(object): mods.add(mod_name) mods.add(os.path.basename(mod_filename)[:-2] + "c") else: - logger.debug("Could not read LoadModule directive from " + - "Augeas path: {0}".format(match_name[6:])) + logger.debug("Could not read LoadModule directive from Augeas path: %s", + match_name[6:]) self.modules.update(mods) def update_runtime_variables(self): @@ -229,8 +375,8 @@ class ApacheParser(object): "Error running command %s for runtime parameters!%s", command, os.linesep) raise errors.MisconfigurationError( - "Error accessing loaded Apache parameters: %s", - command) + "Error accessing loaded Apache parameters: {0}".format( + command)) # Small errors that do not impede if proc.returncode != 0: logger.warning("Error in checking parameter list: %s", stderr) @@ -256,12 +402,12 @@ class ApacheParser(object): """ filtered = [] if args == 1: - for i in range(len(matches)): - if matches[i].endswith("/arg"): + for i, match in enumerate(matches): + if match.endswith("/arg"): filtered.append(matches[i][:-4]) else: - for i in range(len(matches)): - if matches[i].endswith("/arg[%d]" % args): + for i, match in enumerate(matches): + if match.endswith("/arg[%d]" % args): # Make sure we don't cause an IndexError (end of list) # Check to make sure arg + 1 doesn't exist if (i == (len(matches) - 1) or @@ -286,7 +432,7 @@ class ApacheParser(object): """ # TODO: Add error checking code... does the path given even exist? # Does it throw exceptions? - if_mod_path = self._get_ifmod(aug_conf_path, "mod_ssl.c") + if_mod_path = self.get_ifmod(aug_conf_path, "mod_ssl.c") # IfModule can have only one valid argument, so append after self.aug.insert(if_mod_path + "arg", "directive", False) nvh_path = if_mod_path + "directive[1]" @@ -297,22 +443,54 @@ class ApacheParser(object): for i, arg in enumerate(args): self.aug.set("%s/arg[%d]" % (nvh_path, i + 1), arg) - def _get_ifmod(self, aug_conf_path, mod): + def get_ifmod(self, aug_conf_path, mod, beginning=False): """Returns the path to and creates one if it doesn't exist. :param str aug_conf_path: Augeas configuration path :param str mod: module ie. mod_ssl.c + :param bool beginning: If the IfModule should be created to the beginning + of augeas path DOM tree. + + :returns: Augeas path of the requested IfModule directive that pre-existed + or was created during the process. The path may be dynamic, + i.e. .../IfModule[last()] + :rtype: str """ if_mods = self.aug.match(("%s/IfModule/*[self::arg='%s']" % (aug_conf_path, mod))) - if len(if_mods) == 0: - self.aug.set("%s/IfModule[last() + 1]" % aug_conf_path, "") - self.aug.set("%s/IfModule[last()]/arg" % aug_conf_path, mod) - if_mods = self.aug.match(("%s/IfModule/*[self::arg='%s']" % - (aug_conf_path, mod))) + if not if_mods: + return self.create_ifmod(aug_conf_path, mod, beginning) + # Strip off "arg" at end of first ifmod path - return if_mods[0][:len(if_mods[0]) - 3] + return if_mods[0].rpartition("arg")[0] + + def create_ifmod(self, aug_conf_path, mod, beginning=False): + """Creates a new and returns its path. + + :param str aug_conf_path: Augeas configuration path + :param str mod: module ie. mod_ssl.c + :param bool beginning: If the IfModule should be created to the beginning + of augeas path DOM tree. + + :returns: Augeas path of the newly created IfModule directive. + The path may be dynamic, i.e. .../IfModule[last()] + :rtype: str + + """ + if beginning: + c_path_arg = "{}/IfModule[1]/arg".format(aug_conf_path) + # Insert IfModule before the first directive + self.aug.insert("{}/directive[1]".format(aug_conf_path), + "IfModule", True) + retpath = "{}/IfModule[1]/".format(aug_conf_path) + else: + c_path = "{}/IfModule[last() + 1]".format(aug_conf_path) + c_path_arg = "{}/IfModule[last()]/arg".format(aug_conf_path) + self.aug.set(c_path, "") + retpath = "{}/IfModule[last()]/".format(aug_conf_path) + self.aug.set(c_path_arg, mod) + return retpath def add_dir(self, aug_conf_path, directive, args): """Appends directive to the end fo the file given by aug_conf_path. @@ -447,7 +625,7 @@ class ApacheParser(object): # https://httpd.apache.org/docs/2.4/mod/core.html#include for match in matches: dir_ = self.aug.get(match).lower() - if dir_ == "include" or dir_ == "includeoptional": + if dir_ in ("include", "includeoptional"): ordered_matches.extend(self.find_dir( directive, arg, self._get_include_path(self.get_arg(match + "/arg")), @@ -458,6 +636,20 @@ class ApacheParser(object): return ordered_matches + def get_all_args(self, match): + """ + Tries to fetch all arguments for a directive. See get_arg. + + Note that if match is an ancestor node, it returns all names of + child directives as well as the list of arguments. + + """ + + if match[-1] != "/": + match = match+"/" + allargs = self.aug.match(match + '*') + return [self.get_arg(arg) for arg in allargs] + def get_arg(self, match): """Uses augeas.get to get argument value and interprets result. @@ -473,8 +665,7 @@ class ApacheParser(object): # e.g. strip now, not later if not value: return None - else: - value = value.strip("'\"") + value = value.strip("'\"") variables = ApacheParser.arg_var_interpreter.findall(value) @@ -573,7 +764,7 @@ class ApacheParser(object): split_arg = arg.split("/") for idx, split in enumerate(split_arg): if any(char in ApacheParser.fnmatch_chars for char in split): - # Turn it into a augeas regex + # Turn it into an augeas regex # TODO: Can this instead be an augeas glob instead of regex split_arg[idx] = ("* [label()=~regexp('%s')]" % self.fnmatch_to_re(split)) @@ -601,9 +792,8 @@ class ApacheParser(object): if sys.version_info < (3, 6): # This strips off final /Z(?ms) return fnmatch.translate(clean_fn_match)[:-7] - else: # pragma: no cover - # Since Python 3.6, it returns a different pattern like (?s:.*\.load)\Z - return fnmatch.translate(clean_fn_match)[4:-3] + # Since Python 3.6, it returns a different pattern like (?s:.*\.load)\Z + return fnmatch.translate(clean_fn_match)[4:-3] # pragma: no cover def parse_file(self, filepath): """Parse file with Augeas @@ -617,8 +807,7 @@ class ApacheParser(object): use_new, remove_old = self._check_path_actions(filepath) # Ensure that we have the latest Augeas DOM state on disk before # calling aug.load() which reloads the state from disk - if self.configurator: - self.configurator.ensure_augeas_state() + self.ensure_augeas_state() # Test if augeas included file for Httpd.lens # Note: This works for augeas globs, ie. *.conf if use_new: @@ -685,10 +874,7 @@ class ApacheParser(object): use_new = False else: use_new = True - if new_file_match == "*": - remove_old = True - else: - remove_old = False + remove_old = new_file_match == "*" except KeyError: use_new = True remove_old = False diff --git a/certbot-apache/certbot_apache/augeas_configurator.py b/certbot-apache/certbot_apache/augeas_configurator.py deleted file mode 100644 index a32c65c41..000000000 --- a/certbot-apache/certbot_apache/augeas_configurator.py +++ /dev/null @@ -1,207 +0,0 @@ -"""Class of Augeas Configurators.""" -import logging - - -from certbot import errors -from certbot.plugins import common - -from certbot_apache import constants - -logger = logging.getLogger(__name__) - - -class AugeasConfigurator(common.Installer): - """Base Augeas Configurator class. - - :ivar config: Configuration. - :type config: :class:`~certbot.interfaces.IConfig` - - :ivar aug: Augeas object - :type aug: :class:`augeas.Augeas` - - :ivar str save_notes: Human-readable configuration change notes - :ivar reverter: saves and reverts checkpoints - :type reverter: :class:`certbot.reverter.Reverter` - - """ - def __init__(self, *args, **kwargs): - super(AugeasConfigurator, self).__init__(*args, **kwargs) - - # Placeholder for augeas - self.aug = None - - self.save_notes = "" - - - def init_augeas(self): - """ Initialize the actual Augeas instance """ - import augeas - self.aug = augeas.Augeas( - # specify a directory to load our preferred lens from - loadpath=constants.AUGEAS_LENS_DIR, - # Do not save backup (we do it ourselves), do not load - # anything by default - flags=(augeas.Augeas.NONE | - augeas.Augeas.NO_MODL_AUTOLOAD | - augeas.Augeas.ENABLE_SPAN)) - # See if any temporary changes need to be recovered - # This needs to occur before VirtualHost objects are setup... - # because this will change the underlying configuration and potential - # vhosts - self.recovery_routine() - - def check_parsing_errors(self, lens): - """Verify Augeas can parse all of the lens files. - - :param str lens: lens to check for errors - - :raises .errors.PluginError: If there has been an error in parsing with - the specified lens. - - """ - error_files = self.aug.match("/augeas//error") - - for path in error_files: - # Check to see if it was an error resulting from the use of - # the httpd lens - lens_path = self.aug.get(path + "/lens") - # As aug.get may return null - if lens_path and lens in lens_path: - msg = ( - "There has been an error in parsing the file {0} on line {1}: " - "{2}".format( - # Strip off /augeas/files and /error - path[13:len(path) - 6], - self.aug.get(path + "/line"), - self.aug.get(path + "/message"))) - raise errors.PluginError(msg) - - def ensure_augeas_state(self): - """Makes sure that all Augeas dom changes are written to files to avoid - loss of configuration directives when doing additional augeas parsing, - causing a possible augeas.load() resulting dom reset - """ - - if self.unsaved_files(): - self.save_notes += "(autosave)" - self.save() - - def unsaved_files(self): - """Lists files that have modified Augeas DOM but the changes have not - been written to the filesystem yet, used by `self.save()` and - ApacheConfigurator to check the file state. - - :raises .errors.PluginError: If there was an error in Augeas, in - an attempt to save the configuration, or an error creating a - checkpoint - - :returns: `set` of unsaved files - """ - save_state = self.aug.get("/augeas/save") - self.aug.set("/augeas/save", "noop") - # Existing Errors - ex_errs = self.aug.match("/augeas//error") - try: - # This is a noop save - self.aug.save() - except (RuntimeError, IOError): - self._log_save_errors(ex_errs) - # Erase Save Notes - self.save_notes = "" - raise errors.PluginError( - "Error saving files, check logs for more info.") - - # Return the original save method - self.aug.set("/augeas/save", save_state) - - # Retrieve list of modified files - # Note: Noop saves can cause the file to be listed twice, I used a - # set to remove this possibility. This is a known augeas 0.10 error. - save_paths = self.aug.match("/augeas/events/saved") - - save_files = set() - if save_paths: - for path in save_paths: - save_files.add(self.aug.get(path)[6:]) - return save_files - - def save(self, title=None, temporary=False): - """Saves all changes to the configuration files. - - This function first checks for save errors, if none are found, - all configuration changes made will be saved. According to the - function parameters. If an exception is raised, a new checkpoint - was not created. - - :param str title: The title of the save. If a title is given, the - configuration will be saved as a new checkpoint and put in a - timestamped directory. - - :param bool temporary: Indicates whether the changes made will - be quickly reversed in the future (ie. challenges) - - """ - save_files = self.unsaved_files() - if save_files: - self.add_to_checkpoint(save_files, - self.save_notes, temporary=temporary) - - self.save_notes = "" - self.aug.save() - - # Force reload if files were modified - # This is needed to recalculate augeas directive span - if save_files: - for sf in save_files: - self.aug.remove("/files/"+sf) - self.aug.load() - if title and not temporary: - self.finalize_checkpoint(title) - - def _log_save_errors(self, ex_errs): - """Log errors due to bad Augeas save. - - :param list ex_errs: Existing errors before save - - """ - # Check for the root of save problems - new_errs = self.aug.match("/augeas//error") - # logger.error("During Save - %s", mod_conf) - logger.error("Unable to save files: %s. Attempted Save Notes: %s", - ", ".join(err[13:len(err) - 6] for err in new_errs - # Only new errors caused by recent save - if err not in ex_errs), self.save_notes) - - # Wrapper functions for Reverter class - def recovery_routine(self): - """Revert all previously modified files. - - Reverts all modified files that have not been saved as a checkpoint - - :raises .errors.PluginError: If unable to recover the configuration - - """ - super(AugeasConfigurator, self).recovery_routine() - # Need to reload configuration after these changes take effect - self.aug.load() - - def revert_challenge_config(self): - """Used to cleanup challenge configurations. - - :raises .errors.PluginError: If unable to revert the challenge config. - - """ - self.revert_temporary_config() - self.aug.load() - - def rollback_checkpoints(self, rollback=1): - """Rollback saved checkpoints. - - :param int rollback: Number of checkpoints to revert - - :raises .errors.PluginError: If there is a problem with the input or - the function is unable to correctly revert the configuration - - """ - super(AugeasConfigurator, self).rollback_checkpoints(rollback) - self.aug.load() diff --git a/certbot-apache/certbot_apache/centos-options-ssl-apache.conf b/certbot-apache/certbot_apache/centos-options-ssl-apache.conf deleted file mode 100644 index 56c946a4e..000000000 --- a/certbot-apache/certbot_apache/centos-options-ssl-apache.conf +++ /dev/null @@ -1,25 +0,0 @@ -# This file contains important security parameters. If you modify this file -# manually, Certbot will be unable to automatically provide future security -# updates. Instead, Certbot will print and log an error message with a path to -# the up-to-date file that you will need to refer to when manually updating -# this file. - -SSLEngine on - -# Intermediate configuration, tweak to your needs -SSLProtocol all -SSLv2 -SSLv3 -SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS -SSLHonorCipherOrder on - -SSLOptions +StrictRequire - -# Add vhost name to log entries: -LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined -LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common - -#CustomLog /var/log/apache2/access.log vhost_combined -#LogLevel warn -#ErrorLog /var/log/apache2/error.log - -# Always ensure Cookies have "Secure" set (JAH 2012/1) -#Header edit Set-Cookie (?i)^(.*)(;\s*secure)??((\s*;)?(.*)) "$1; Secure$3$4" diff --git a/certbot-apache/certbot_apache/options-ssl-apache.conf b/certbot-apache/certbot_apache/options-ssl-apache.conf deleted file mode 100644 index 8113ee81e..000000000 --- a/certbot-apache/certbot_apache/options-ssl-apache.conf +++ /dev/null @@ -1,26 +0,0 @@ -# This file contains important security parameters. If you modify this file -# manually, Certbot will be unable to automatically provide future security -# updates. Instead, Certbot will print and log an error message with a path to -# the up-to-date file that you will need to refer to when manually updating -# this file. - -SSLEngine on - -# Intermediate configuration, tweak to your needs -SSLProtocol all -SSLv2 -SSLv3 -SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS -SSLHonorCipherOrder on -SSLCompression off - -SSLOptions +StrictRequire - -# Add vhost name to log entries: -LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined -LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common - -#CustomLog /var/log/apache2/access.log vhost_combined -#LogLevel warn -#ErrorLog /var/log/apache2/error.log - -# Always ensure Cookies have "Secure" set (JAH 2012/1) -#Header edit Set-Cookie (?i)^(.*)(;\s*secure)??((\s*;)?(.*)) "$1; Secure$3$4" diff --git a/certbot-apache/certbot_apache/override_centos.py b/certbot-apache/certbot_apache/override_centos.py deleted file mode 100644 index a4f1b84ec..000000000 --- a/certbot-apache/certbot_apache/override_centos.py +++ /dev/null @@ -1,68 +0,0 @@ -""" Distribution specific override class for CentOS family (RHEL, Fedora) """ -import pkg_resources - -import zope.interface - -from certbot import interfaces - -from certbot_apache import apache_util -from certbot_apache import configurator -from certbot_apache import parser - -@zope.interface.provider(interfaces.IPluginFactory) -class CentOSConfigurator(configurator.ApacheConfigurator): - """CentOS specific ApacheConfigurator override class""" - - OS_DEFAULTS = dict( - server_root="/etc/httpd", - vhost_root="/etc/httpd/conf.d", - vhost_files="*.conf", - logs_root="/var/log/httpd", - ctl="apachectl", - version_cmd=['apachectl', '-v'], - restart_cmd=['apachectl', 'graceful'], - restart_cmd_alt=['apachectl', 'restart'], - conftest_cmd=['apachectl', 'configtest'], - enmod=None, - dismod=None, - le_vhost_ext="-le-ssl.conf", - handle_modules=False, - handle_sites=False, - challenge_location="/etc/httpd/conf.d", - MOD_SSL_CONF_SRC=pkg_resources.resource_filename( - "certbot_apache", "centos-options-ssl-apache.conf") - ) - - def _prepare_options(self): - """ - Override the options dictionary initialization in order to support - alternative restart cmd used in CentOS. - """ - super(CentOSConfigurator, self)._prepare_options() - self.options["restart_cmd_alt"][0] = self.option("ctl") - - def get_parser(self): - """Initializes the ApacheParser""" - return CentOSParser( - self.aug, self.option("server_root"), self.option("vhost_root"), - self.version, configurator=self) - - -class CentOSParser(parser.ApacheParser): - """CentOS specific ApacheParser override class""" - def __init__(self, *args, **kwargs): - # CentOS specific configuration file for Apache - self.sysconfig_filep = "/etc/sysconfig/httpd" - super(CentOSParser, self).__init__(*args, **kwargs) - - def update_runtime_variables(self): - """ Override for update_runtime_variables for custom parsing """ - # Opportunistic, works if SELinux not enforced - super(CentOSParser, self).update_runtime_variables() - self.parse_sysconfig_var() - - def parse_sysconfig_var(self): - """ Parses Apache CLI options from CentOS configuration file """ - defines = apache_util.parse_define_file(self.sysconfig_filep, "OPTIONS") - for k in defines.keys(): - self.variables[k] = defines[k] diff --git a/certbot-apache/certbot_apache/tests/__init__.py b/certbot-apache/certbot_apache/tests/__init__.py deleted file mode 100644 index 7e7d39fa4..000000000 --- a/certbot-apache/certbot_apache/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Certbot Apache Tests""" diff --git a/certbot-apache/certbot_apache/tests/tls_sni_01_test.py b/certbot-apache/certbot_apache/tests/tls_sni_01_test.py deleted file mode 100644 index 8cea97f04..000000000 --- a/certbot-apache/certbot_apache/tests/tls_sni_01_test.py +++ /dev/null @@ -1,151 +0,0 @@ -"""Test for certbot_apache.tls_sni_01.""" -import shutil -import unittest - -import mock - -from certbot import errors -from certbot.plugins import common_test - -from certbot_apache import obj -from certbot_apache.tests import util - -from six.moves import xrange # pylint: disable=redefined-builtin, import-error - - -class TlsSniPerformTest(util.ApacheTest): - """Test the ApacheTlsSni01 challenge.""" - - auth_key = common_test.AUTH_KEY - achalls = common_test.ACHALLS - - def setUp(self): # pylint: disable=arguments-differ - super(TlsSniPerformTest, self).setUp() - - config = util.get_apache_configurator( - self.config_path, self.vhost_path, self.config_dir, - self.work_dir) - config.config.tls_sni_01_port = 443 - - from certbot_apache import tls_sni_01 - self.sni = tls_sni_01.ApacheTlsSni01(config) - - def tearDown(self): - shutil.rmtree(self.temp_dir) - shutil.rmtree(self.config_dir) - shutil.rmtree(self.work_dir) - - def test_perform0(self): - resp = self.sni.perform() - self.assertEqual(len(resp), 0) - - @mock.patch("certbot.util.exe_exists") - @mock.patch("certbot.util.run_script") - def test_perform1(self, _, mock_exists): - self.sni.configurator.parser.modules.add("socache_shmcb_module") - self.sni.configurator.parser.modules.add("ssl_module") - - mock_exists.return_value = True - self.sni.configurator.parser.update_runtime_variables = mock.Mock() - - achall = self.achalls[0] - self.sni.add_chall(achall) - response = self.achalls[0].response(self.auth_key) - mock_setup_cert = mock.MagicMock(return_value=response) - # pylint: disable=protected-access - self.sni._setup_challenge_cert = mock_setup_cert - - responses = self.sni.perform() - mock_setup_cert.assert_called_once_with(achall) - - # Check to make sure challenge config path is included in apache config - self.assertEqual( - len(self.sni.configurator.parser.find_dir( - "Include", self.sni.challenge_conf)), 1) - self.assertEqual(len(responses), 1) - self.assertEqual(responses[0], response) - - def test_perform2(self): - # Avoid load module - self.sni.configurator.parser.modules.add("ssl_module") - self.sni.configurator.parser.modules.add("socache_shmcb_module") - acme_responses = [] - for achall in self.achalls: - self.sni.add_chall(achall) - acme_responses.append(achall.response(self.auth_key)) - - mock_setup_cert = mock.MagicMock(side_effect=acme_responses) - # pylint: disable=protected-access - self.sni._setup_challenge_cert = mock_setup_cert - - with mock.patch( - "certbot_apache.override_debian.DebianConfigurator.enable_mod"): - sni_responses = self.sni.perform() - - self.assertEqual(mock_setup_cert.call_count, 2) - - # Make sure calls made to mocked function were correct - self.assertEqual( - mock_setup_cert.call_args_list[0], mock.call(self.achalls[0])) - self.assertEqual( - mock_setup_cert.call_args_list[1], mock.call(self.achalls[1])) - - self.assertEqual( - len(self.sni.configurator.parser.find_dir( - "Include", self.sni.challenge_conf)), - 1) - self.assertEqual(len(sni_responses), 2) - for i in xrange(2): - self.assertEqual(sni_responses[i], acme_responses[i]) - - def test_mod_config(self): - z_domains = [] - for achall in self.achalls: - self.sni.add_chall(achall) - z_domain = achall.response(self.auth_key).z_domain - z_domains.append(set([z_domain.decode('ascii')])) - - self.sni._mod_config() # pylint: disable=protected-access - self.sni.configurator.save() - - self.sni.configurator.parser.find_dir( - "Include", self.sni.challenge_conf) - vh_match = self.sni.configurator.aug.match( - "/files" + self.sni.challenge_conf + "//VirtualHost") - - vhs = [] - for match in vh_match: - # pylint: disable=protected-access - vhs.append(self.sni.configurator._create_vhost(match)) - self.assertEqual(len(vhs), 2) - for vhost in vhs: - self.assertEqual(vhost.addrs, set([obj.Addr.fromstring("*:443")])) - names = vhost.get_names() - self.assertTrue(names in z_domains) - - def test_get_addrs_default(self): - self.sni.configurator.choose_vhost = mock.Mock( - return_value=obj.VirtualHost( - "path", "aug_path", - set([obj.Addr.fromstring("_default_:443")]), - False, False) - ) - - # pylint: disable=protected-access - self.assertEqual( - set([obj.Addr.fromstring("*:443")]), - self.sni._get_addrs(self.achalls[0])) - - def test_get_addrs_no_vhost_found(self): - self.sni.configurator.choose_vhost = mock.Mock( - side_effect=errors.MissingCommandlineFlag( - "Failed to run Apache plugin non-interactively")) - - # pylint: disable=protected-access - self.assertEqual( - set([obj.Addr.fromstring("*:443")]), - self.sni._get_addrs(self.achalls[0])) - - -if __name__ == "__main__": - unittest.main() # pragma: no cover diff --git a/certbot-apache/certbot_apache/tls_sni_01.py b/certbot-apache/certbot_apache/tls_sni_01.py deleted file mode 100644 index 65230cdcb..000000000 --- a/certbot-apache/certbot_apache/tls_sni_01.py +++ /dev/null @@ -1,174 +0,0 @@ -"""A class that performs TLS-SNI-01 challenges for Apache""" - -import os -import logging - -from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module -from certbot.plugins import common -from certbot.errors import PluginError, MissingCommandlineFlag - -from certbot_apache import obj - -logger = logging.getLogger(__name__) - - -class ApacheTlsSni01(common.TLSSNI01): - """Class that performs TLS-SNI-01 challenges within the Apache configurator - - :ivar configurator: ApacheConfigurator object - :type configurator: :class:`~apache.configurator.ApacheConfigurator` - - :ivar list achalls: Annotated TLS-SNI-01 - (`.KeyAuthorizationAnnotatedChallenge`) challenges. - - :param list indices: Meant to hold indices of challenges in a - larger array. ApacheTlsSni01 is capable of solving many challenges - at once which causes an indexing issue within ApacheConfigurator - who must return all responses in order. Imagine ApacheConfigurator - maintaining state about where all of the http-01 Challenges, - TLS-SNI-01 Challenges belong in the response array. This is an - optional utility. - - :param str challenge_conf: location of the challenge config file - - """ - - VHOST_TEMPLATE = """\ - - ServerName {server_name} - UseCanonicalName on - SSLStrictSNIVHostCheck on - - LimitRequestBody 1048576 - - Include {ssl_options_conf_path} - SSLCertificateFile {cert_path} - SSLCertificateKeyFile {key_path} - - DocumentRoot {document_root} - - -""" - - def __init__(self, *args, **kwargs): - super(ApacheTlsSni01, self).__init__(*args, **kwargs) - - self.challenge_conf = os.path.join( - self.configurator.conf("challenge-location"), - "le_tls_sni_01_cert_challenge.conf") - - def perform(self): - """Perform a TLS-SNI-01 challenge.""" - if not self.achalls: - return [] - # Save any changes to the configuration as a precaution - # About to make temporary changes to the config - self.configurator.save("Changes before challenge setup", True) - - # Prepare the server for HTTPS - self.configurator.prepare_server_https( - str(self.configurator.config.tls_sni_01_port), True) - - responses = [] - - # Create all of the challenge certs - for achall in self.achalls: - responses.append(self._setup_challenge_cert(achall)) - - # Setup the configuration - addrs = self._mod_config() - self.configurator.save("Don't lose mod_config changes", True) - self.configurator.make_addrs_sni_ready(addrs) - - # Save reversible changes - self.configurator.save("SNI Challenge", True) - - return responses - - def _mod_config(self): - """Modifies Apache config files to include challenge vhosts. - - Result: Apache config includes virtual servers for issued challs - - :returns: All TLS-SNI-01 addresses used - :rtype: set - - """ - addrs = set() # type: Set[obj.Addr] - config_text = "\n" - - for achall in self.achalls: - achall_addrs = self._get_addrs(achall) - addrs.update(achall_addrs) - - config_text += self._get_config_text(achall, achall_addrs) - - config_text += "\n" - - self.configurator.parser.add_include( - self.configurator.parser.loc["default"], self.challenge_conf) - self.configurator.reverter.register_file_creation( - True, self.challenge_conf) - - logger.debug("writing a config file with text:\n %s", config_text) - with open(self.challenge_conf, "w") as new_conf: - new_conf.write(config_text) - - return addrs - - def _get_addrs(self, achall): - """Return the Apache addresses needed for TLS-SNI-01.""" - # TODO: Checkout _default_ rules. - addrs = set() - default_addr = obj.Addr(("*", str( - self.configurator.config.tls_sni_01_port))) - - try: - vhost = self.configurator.choose_vhost(achall.domain, - create_if_no_ssl=False) - except (PluginError, MissingCommandlineFlag): - # We couldn't find the virtualhost for this domain, possibly - # because it's a new vhost that's not configured yet - # (GH #677). See also GH #2600. - logger.warning("Falling back to default vhost %s...", default_addr) - addrs.add(default_addr) - return addrs - - for addr in vhost.addrs: - if "_default_" == addr.get_addr(): - addrs.add(default_addr) - else: - addrs.add( - addr.get_sni_addr( - self.configurator.config.tls_sni_01_port)) - - return addrs - - def _get_config_text(self, achall, ip_addrs): - """Chocolate virtual server configuration text - - :param .KeyAuthorizationAnnotatedChallenge achall: Annotated - TLS-SNI-01 challenge. - - :param list ip_addrs: addresses of challenged domain - :class:`list` of type `~.obj.Addr` - - :returns: virtual host configuration text - :rtype: str - - """ - ips = " ".join(str(i) for i in ip_addrs) - document_root = os.path.join( - self.configurator.config.work_dir, "tls_sni_01_page/") - # TODO: Python docs is not clear how multiline string literal - # newlines are parsed on different platforms. At least on - # Linux (Debian sid), when source file uses CRLF, Python still - # parses it as "\n"... c.f.: - # https://docs.python.org/2.7/reference/lexical_analysis.html - return self.VHOST_TEMPLATE.format( - vhost=ips, - server_name=achall.response(achall.account_key).z_domain.decode('ascii'), - ssl_options_conf_path=self.configurator.mod_ssl_conf, - cert_path=self.get_cert_path(achall), - key_path=self.get_key_path(achall), - document_root=document_root).replace("\n", os.linesep) diff --git a/certbot-apache/docs/Makefile b/certbot-apache/docs/Makefile deleted file mode 100644 index 0e611ecec..000000000 --- a/certbot-apache/docs/Makefile +++ /dev/null @@ -1,192 +0,0 @@ -# Makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -PAPER = -BUILDDIR = _build - -# User-friendly check for sphinx-build -ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) -$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) -endif - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . -# the i18n builder cannot share the environment and doctrees with the others -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext - -help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " singlehtml to make a single large HTML file" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " applehelp to make an Apple Help Book" - @echo " devhelp to make HTML files and a Devhelp project" - @echo " epub to make an epub" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " latexpdf to make LaTeX files and run them through pdflatex" - @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" - @echo " text to make text files" - @echo " man to make manual pages" - @echo " texinfo to make Texinfo files" - @echo " info to make Texinfo files and run them through makeinfo" - @echo " gettext to make PO message catalogs" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " xml to make Docutils-native XML files" - @echo " pseudoxml to make pseudoxml-XML files for display purposes" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - @echo " coverage to run coverage check of the documentation (if enabled)" - -clean: - rm -rf $(BUILDDIR)/* - -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." - -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." - -singlehtml: - $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml - @echo - @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." - -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/certbot-apache.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/certbot-apache.qhc" - -applehelp: - $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp - @echo - @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." - @echo "N.B. You won't be able to view it unless you put it in" \ - "~/Library/Documentation/Help or install it in your application" \ - "bundle." - -devhelp: - $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp - @echo - @echo "Build finished." - @echo "To view the help file:" - @echo "# mkdir -p $$HOME/.local/share/devhelp/certbot-apache" - @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/certbot-apache" - @echo "# devhelp" - -epub: - $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub - @echo - @echo "Build finished. The epub file is in $(BUILDDIR)/epub." - -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make' in that directory to run these through (pdf)latex" \ - "(use \`make latexpdf' here to do that automatically)." - -latexpdf: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through pdflatex..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -latexpdfja: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through platex and dvipdfmx..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -text: - $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text - @echo - @echo "Build finished. The text files are in $(BUILDDIR)/text." - -man: - $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man - @echo - @echo "Build finished. The manual pages are in $(BUILDDIR)/man." - -texinfo: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo - @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." - @echo "Run \`make' in that directory to run these through makeinfo" \ - "(use \`make info' here to do that automatically)." - -info: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo "Running Texinfo files through makeinfo..." - make -C $(BUILDDIR)/texinfo info - @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." - -gettext: - $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale - @echo - @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." - -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." - -coverage: - $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage - @echo "Testing of coverage in the sources finished, look at the " \ - "results in $(BUILDDIR)/coverage/python.txt." - -xml: - $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml - @echo - @echo "Build finished. The XML files are in $(BUILDDIR)/xml." - -pseudoxml: - $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml - @echo - @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." diff --git a/certbot-apache/docs/api/augeas_configurator.rst b/certbot-apache/docs/api/augeas_configurator.rst deleted file mode 100644 index b47ffbc6b..000000000 --- a/certbot-apache/docs/api/augeas_configurator.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_apache.augeas_configurator` ---------------------------------------------- - -.. automodule:: certbot_apache.augeas_configurator - :members: diff --git a/certbot-apache/docs/api/configurator.rst b/certbot-apache/docs/api/configurator.rst deleted file mode 100644 index 8ec266d1a..000000000 --- a/certbot-apache/docs/api/configurator.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_apache.configurator` --------------------------------------- - -.. automodule:: certbot_apache.configurator - :members: diff --git a/certbot-apache/docs/api/display_ops.rst b/certbot-apache/docs/api/display_ops.rst deleted file mode 100644 index 26d3ed3dc..000000000 --- a/certbot-apache/docs/api/display_ops.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_apache.display_ops` -------------------------------------- - -.. automodule:: certbot_apache.display_ops - :members: diff --git a/certbot-apache/docs/api/obj.rst b/certbot-apache/docs/api/obj.rst deleted file mode 100644 index 82e58df3f..000000000 --- a/certbot-apache/docs/api/obj.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_apache.obj` ------------------------------ - -.. automodule:: certbot_apache.obj - :members: diff --git a/certbot-apache/docs/api/parser.rst b/certbot-apache/docs/api/parser.rst deleted file mode 100644 index 3427735be..000000000 --- a/certbot-apache/docs/api/parser.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_apache.parser` --------------------------------- - -.. automodule:: certbot_apache.parser - :members: diff --git a/certbot-apache/docs/api/tls_sni_01.rst b/certbot-apache/docs/api/tls_sni_01.rst deleted file mode 100644 index 3ecd0a365..000000000 --- a/certbot-apache/docs/api/tls_sni_01.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_apache.tls_sni_01` ------------------------------------- - -.. automodule:: certbot_apache.tls_sni_01 - :members: diff --git a/certbot-apache/docs/conf.py b/certbot-apache/docs/conf.py deleted file mode 100644 index d2fe15581..000000000 --- a/certbot-apache/docs/conf.py +++ /dev/null @@ -1,318 +0,0 @@ -# -*- coding: utf-8 -*- -# -# certbot-apache documentation build configuration file, created by -# sphinx-quickstart on Sun Oct 18 13:39:26 2015. -# -# This file is execfile()d with the current directory set to its -# containing dir. -# -# Note that not all possible configuration values are present in this -# autogenerated file. -# -# All configuration values have a default; values that are commented out -# serve to show the default. - -import sys -import os -import shlex - -import mock - - -# http://docs.readthedocs.org/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules -# c.f. #262 -sys.modules.update( - (mod_name, mock.MagicMock()) for mod_name in ['augeas']) - -here = os.path.abspath(os.path.dirname(__file__)) - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -sys.path.insert(0, os.path.abspath(os.path.join(here, '..'))) - -# -- General configuration ------------------------------------------------ - -# If your documentation needs a minimal Sphinx version, state it here. -needs_sphinx = '1.0' - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom -# ones. -extensions = [ - 'sphinx.ext.autodoc', - 'sphinx.ext.intersphinx', - 'sphinx.ext.todo', - 'sphinx.ext.coverage', - 'sphinx.ext.viewcode', -] - -autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# The suffix(es) of source filenames. -# You can specify multiple suffix as a list of string: -# source_suffix = ['.rst', '.md'] -source_suffix = '.rst' - -# The encoding of source files. -#source_encoding = 'utf-8-sig' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'certbot-apache' -copyright = u'2014-2015, Let\'s Encrypt Project' -author = u'Certbot Project' - -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. -# -# The short X.Y version. -version = '0' -# The full version, including alpha/beta/rc tags. -release = '0' - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -# -# This is also used if you do content translation via gettext catalogs. -# Usually you set "language" from the command line for these cases. -language = 'en' - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -exclude_patterns = ['_build'] - -# The reST default role (used for this markup: `text`) to use for all -# documents. -default_role = 'py:obj' - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] - -# If true, keep warnings as "system message" paragraphs in the built documents. -#keep_warnings = False - -# If true, `todo` and `todoList` produce output, else they produce nothing. -todo_include_todos = True - - -# -- Options for HTML output ---------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. - -# http://docs.readthedocs.org/en/latest/theme.html#how-do-i-use-this-locally-and-on-read-the-docs -# on_rtd is whether we are on readthedocs.org -on_rtd = os.environ.get('READTHEDOCS', None) == 'True' -if not on_rtd: # only import and set the theme if we're building docs locally - import sphinx_rtd_theme - html_theme = 'sphinx_rtd_theme' - html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] -# otherwise, readthedocs.org uses their theme by default, so no need to specify it - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -#html_theme_options = {} - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -#html_title = None - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -#html_logo = None - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -#html_favicon = None - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - -# Add any extra paths that contain custom files (such as robots.txt or -# .htaccess) here, relative to this directory. These files are copied -# directly to the root of the documentation. -#html_extra_path = [] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_domain_indices = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True - -# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True - -# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None - -# Language to be used for generating the HTML full-text search index. -# Sphinx supports the following languages: -# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' -# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' -#html_search_language = 'en' - -# A dictionary with options for the search language support, empty by default. -# Now only 'ja' uses this config value -#html_search_options = {'type': 'default'} - -# The name of a javascript file (relative to the configuration directory) that -# implements a search results scorer. If empty, the default will be used. -#html_search_scorer = 'scorer.js' - -# Output file base name for HTML help builder. -htmlhelp_basename = 'certbot-apachedoc' - -# -- Options for LaTeX output --------------------------------------------- - -latex_elements = { - # The paper size ('letterpaper' or 'a4paper'). - #'papersize': 'letterpaper', - - # The font size ('10pt', '11pt' or '12pt'). - #'pointsize': '10pt', - - # Additional stuff for the LaTeX preamble. - #'preamble': '', - - # Latex figure (float) alignment - #'figure_align': 'htbp', -} - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, -# author, documentclass [howto, manual, or own class]). -latex_documents = [ - (master_doc, 'certbot-apache.tex', u'certbot-apache Documentation', - u'Certbot Project', 'manual'), -] - -# The name of an image file (relative to this directory) to place at the top of -# the title page. -#latex_logo = None - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# If true, show page references after internal links. -#latex_show_pagerefs = False - -# If true, show URL addresses after external links. -#latex_show_urls = False - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_domain_indices = True - - -# -- Options for manual page output --------------------------------------- - -# One entry per manual page. List of tuples -# (source start file, name, description, authors, manual section). -man_pages = [ - (master_doc, 'certbot-apache', u'certbot-apache Documentation', - [author], 1) -] - -# If true, show URL addresses after external links. -#man_show_urls = False - - -# -- Options for Texinfo output ------------------------------------------- - -# Grouping the document tree into Texinfo files. List of tuples -# (source start file, target name, title, author, -# dir menu entry, description, category) -texinfo_documents = [ - (master_doc, 'certbot-apache', u'certbot-apache Documentation', - author, 'certbot-apache', 'One line description of project.', - 'Miscellaneous'), -] - -# Documents to append as an appendix to all manuals. -#texinfo_appendices = [] - -# If false, no module index is generated. -#texinfo_domain_indices = True - -# How to display URL addresses: 'footnote', 'no', or 'inline'. -#texinfo_show_urls = 'footnote' - -# If true, do not generate a @detailmenu in the "Top" node's menu. -#texinfo_no_detailmenu = False - - -intersphinx_mapping = { - 'python': ('https://docs.python.org/', None), - 'acme': ('https://acme-python.readthedocs.org/en/latest/', None), - 'certbot': ('https://certbot.eff.org/docs/', None), -} diff --git a/certbot-apache/docs/index.rst b/certbot-apache/docs/index.rst deleted file mode 100644 index bfe4d245c..000000000 --- a/certbot-apache/docs/index.rst +++ /dev/null @@ -1,31 +0,0 @@ -.. certbot-apache documentation master file, created by - sphinx-quickstart on Sun Oct 18 13:39:26 2015. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to certbot-apache's documentation! -============================================== - -Contents: - -.. toctree:: - :maxdepth: 2 - - -.. toctree:: - :maxdepth: 1 - - api - - -.. automodule:: certbot_apache - :members: - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` - diff --git a/certbot-apache/docs/make.bat b/certbot-apache/docs/make.bat deleted file mode 100644 index 3a7818940..000000000 --- a/certbot-apache/docs/make.bat +++ /dev/null @@ -1,263 +0,0 @@ -@ECHO OFF - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set BUILDDIR=_build -set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . -set I18NSPHINXOPTS=%SPHINXOPTS% . -if NOT "%PAPER%" == "" ( - set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% - set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% -) - -if "%1" == "" goto help - -if "%1" == "help" ( - :help - echo.Please use `make ^` where ^ is one of - echo. html to make standalone HTML files - echo. dirhtml to make HTML files named index.html in directories - echo. singlehtml to make a single large HTML file - echo. pickle to make pickle files - echo. json to make JSON files - echo. htmlhelp to make HTML files and a HTML help project - echo. qthelp to make HTML files and a qthelp project - echo. devhelp to make HTML files and a Devhelp project - echo. epub to make an epub - echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter - echo. text to make text files - echo. man to make manual pages - echo. texinfo to make Texinfo files - echo. gettext to make PO message catalogs - echo. changes to make an overview over all changed/added/deprecated items - echo. xml to make Docutils-native XML files - echo. pseudoxml to make pseudoxml-XML files for display purposes - echo. linkcheck to check all external links for integrity - echo. doctest to run all doctests embedded in the documentation if enabled - echo. coverage to run coverage check of the documentation if enabled - goto end -) - -if "%1" == "clean" ( - for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i - del /q /s %BUILDDIR%\* - goto end -) - - -REM Check if sphinx-build is available and fallback to Python version if any -%SPHINXBUILD% 2> nul -if errorlevel 9009 goto sphinx_python -goto sphinx_ok - -:sphinx_python - -set SPHINXBUILD=python -m sphinx.__init__ -%SPHINXBUILD% 2> nul -if errorlevel 9009 ( - echo. - echo.The 'sphinx-build' command was not found. Make sure you have Sphinx - echo.installed, then set the SPHINXBUILD environment variable to point - echo.to the full path of the 'sphinx-build' executable. Alternatively you - echo.may add the Sphinx directory to PATH. - echo. - echo.If you don't have Sphinx installed, grab it from - echo.http://sphinx-doc.org/ - exit /b 1 -) - -:sphinx_ok - - -if "%1" == "html" ( - %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/html. - goto end -) - -if "%1" == "dirhtml" ( - %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. - goto end -) - -if "%1" == "singlehtml" ( - %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. - goto end -) - -if "%1" == "pickle" ( - %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the pickle files. - goto end -) - -if "%1" == "json" ( - %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the JSON files. - goto end -) - -if "%1" == "htmlhelp" ( - %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run HTML Help Workshop with the ^ -.hhp project file in %BUILDDIR%/htmlhelp. - goto end -) - -if "%1" == "qthelp" ( - %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run "qcollectiongenerator" with the ^ -.qhcp project file in %BUILDDIR%/qthelp, like this: - echo.^> qcollectiongenerator %BUILDDIR%\qthelp\certbot-apache.qhcp - echo.To view the help file: - echo.^> assistant -collectionFile %BUILDDIR%\qthelp\certbot-apache.ghc - goto end -) - -if "%1" == "devhelp" ( - %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. - goto end -) - -if "%1" == "epub" ( - %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The epub file is in %BUILDDIR%/epub. - goto end -) - -if "%1" == "latex" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "latexpdf" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - cd %BUILDDIR%/latex - make all-pdf - cd %~dp0 - echo. - echo.Build finished; the PDF files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "latexpdfja" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - cd %BUILDDIR%/latex - make all-pdf-ja - cd %~dp0 - echo. - echo.Build finished; the PDF files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "text" ( - %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The text files are in %BUILDDIR%/text. - goto end -) - -if "%1" == "man" ( - %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The manual pages are in %BUILDDIR%/man. - goto end -) - -if "%1" == "texinfo" ( - %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. - goto end -) - -if "%1" == "gettext" ( - %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The message catalogs are in %BUILDDIR%/locale. - goto end -) - -if "%1" == "changes" ( - %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes - if errorlevel 1 exit /b 1 - echo. - echo.The overview file is in %BUILDDIR%/changes. - goto end -) - -if "%1" == "linkcheck" ( - %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck - if errorlevel 1 exit /b 1 - echo. - echo.Link check complete; look for any errors in the above output ^ -or in %BUILDDIR%/linkcheck/output.txt. - goto end -) - -if "%1" == "doctest" ( - %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest - if errorlevel 1 exit /b 1 - echo. - echo.Testing of doctests in the sources finished, look at the ^ -results in %BUILDDIR%/doctest/output.txt. - goto end -) - -if "%1" == "coverage" ( - %SPHINXBUILD% -b coverage %ALLSPHINXOPTS% %BUILDDIR%/coverage - if errorlevel 1 exit /b 1 - echo. - echo.Testing of coverage in the sources finished, look at the ^ -results in %BUILDDIR%/coverage/python.txt. - goto end -) - -if "%1" == "xml" ( - %SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The XML files are in %BUILDDIR%/xml. - goto end -) - -if "%1" == "pseudoxml" ( - %SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml. - goto end -) - -:end diff --git a/certbot-apache/local-oldest-requirements.txt b/certbot-apache/local-oldest-requirements.txt index fd8869f7c..cf61c15a5 100644 --- a/certbot-apache/local-oldest-requirements.txt +++ b/certbot-apache/local-oldest-requirements.txt @@ -1,2 +1,3 @@ -acme[dev]==0.25.0 -certbot[dev]==0.26.0 +# Remember to update setup.py to match the package versions below. +acme[dev]==0.29.0 +certbot[dev]==1.1.0 diff --git a/certbot-apache/readthedocs.org.requirements.txt b/certbot-apache/readthedocs.org.requirements.txt deleted file mode 100644 index fe30ab1dc..000000000 --- a/certbot-apache/readthedocs.org.requirements.txt +++ /dev/null @@ -1,12 +0,0 @@ -# readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation -# dependencies), but it allows to specify a requirements.txt file at -# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) - -# Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead - --e acme --e . --e certbot-apache[docs] diff --git a/certbot-apache/setup.py b/certbot-apache/setup.py index 14d6cacb6..f9b85008b 100644 --- a/certbot-apache/setup.py +++ b/certbot-apache/setup.py @@ -1,14 +1,16 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.25.0', - 'certbot>=0.26.0', + 'acme>=0.29.0', + 'certbot>=1.1.0', 'mock', 'python-augeas', 'setuptools', @@ -16,10 +18,21 @@ install_requires = [ 'zope.interface', ] -docs_extras = [ - 'Sphinx>=1.0', # autodoc_member_order = 'bysource', autodoc_default_flags - 'sphinx_rtd_theme', -] + +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-apache', @@ -29,7 +42,7 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', @@ -40,10 +53,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -55,13 +68,12 @@ setup( packages=find_packages(), include_package_data=True, install_requires=install_requires, - extras_require={ - 'docs': docs_extras, - }, entry_points={ 'certbot.plugins': [ - 'apache = certbot_apache.entrypoint:ENTRYPOINT', + 'apache = certbot_apache._internal.entrypoint:ENTRYPOINT', ], }, test_suite='certbot_apache', + tests_require=["pytest"], + cmdclass={"test": PyTest}, ) diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/NEEDED.txt b/certbot-apache/tests/apache-conf-files/NEEDED.txt similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/NEEDED.txt rename to certbot-apache/tests/apache-conf-files/NEEDED.txt diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/apache-conf-test b/certbot-apache/tests/apache-conf-files/apache-conf-test similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/apache-conf-test rename to certbot-apache/tests/apache-conf-files/apache-conf-test diff --git a/certbot-apache/tests/apache-conf-files/apache-conf-test-pebble.py b/certbot-apache/tests/apache-conf-files/apache-conf-test-pebble.py new file mode 100755 index 000000000..68bd6287d --- /dev/null +++ b/certbot-apache/tests/apache-conf-files/apache-conf-test-pebble.py @@ -0,0 +1,27 @@ +#!/usr/bin/env python +""" +This executable script wraps the apache-conf-test bash script, in order to setup a pebble instance +before its execution. Directory URL is passed through the SERVER environment variable. +""" +import os +import subprocess +import sys + +from certbot_integration_tests.utils import acme_server + +SCRIPT_DIRNAME = os.path.dirname(__file__) + + +def main(args=None): + if not args: + args = sys.argv[1:] + with acme_server.ACMEServer('pebble', [], False) as acme_xdist: + environ = os.environ.copy() + environ['SERVER'] = acme_xdist['directory_url'] + command = [os.path.join(SCRIPT_DIRNAME, 'apache-conf-test')] + command.extend(args) + return subprocess.call(command, env=environ) + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/failing/missing-double-quote-1724.conf b/certbot-apache/tests/apache-conf-files/failing/missing-double-quote-1724.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/failing/missing-double-quote-1724.conf rename to certbot-apache/tests/apache-conf-files/failing/missing-double-quote-1724.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/failing/multivhost-1093.conf b/certbot-apache/tests/apache-conf-files/failing/multivhost-1093.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/failing/multivhost-1093.conf rename to certbot-apache/tests/apache-conf-files/failing/multivhost-1093.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/failing/multivhost-1093b.conf b/certbot-apache/tests/apache-conf-files/failing/multivhost-1093b.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/failing/multivhost-1093b.conf rename to certbot-apache/tests/apache-conf-files/failing/multivhost-1093b.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/1626-1531.conf b/certbot-apache/tests/apache-conf-files/passing/1626-1531.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/1626-1531.conf rename to certbot-apache/tests/apache-conf-files/passing/1626-1531.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/README.modules b/certbot-apache/tests/apache-conf-files/passing/README.modules similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/README.modules rename to certbot-apache/tests/apache-conf-files/passing/README.modules diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/anarcat-1531.conf b/certbot-apache/tests/apache-conf-files/passing/anarcat-1531.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/anarcat-1531.conf rename to certbot-apache/tests/apache-conf-files/passing/anarcat-1531.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/comment-continuations-2050.conf b/certbot-apache/tests/apache-conf-files/passing/comment-continuations-2050.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/comment-continuations-2050.conf rename to certbot-apache/tests/apache-conf-files/passing/comment-continuations-2050.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/drupal-errordocument-arg-1724.conf b/certbot-apache/tests/apache-conf-files/passing/drupal-errordocument-arg-1724.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/drupal-errordocument-arg-1724.conf rename to certbot-apache/tests/apache-conf-files/passing/drupal-errordocument-arg-1724.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/drupal-htaccess-1531.conf b/certbot-apache/tests/apache-conf-files/passing/drupal-htaccess-1531.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/drupal-htaccess-1531.conf rename to certbot-apache/tests/apache-conf-files/passing/drupal-htaccess-1531.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/escaped-space-arguments-2735.conf b/certbot-apache/tests/apache-conf-files/passing/escaped-space-arguments-2735.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/escaped-space-arguments-2735.conf rename to certbot-apache/tests/apache-conf-files/passing/escaped-space-arguments-2735.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/example-1755.conf b/certbot-apache/tests/apache-conf-files/passing/example-1755.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/example-1755.conf rename to certbot-apache/tests/apache-conf-files/passing/example-1755.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/example-ssl.conf b/certbot-apache/tests/apache-conf-files/passing/example-ssl.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/example-ssl.conf rename to certbot-apache/tests/apache-conf-files/passing/example-ssl.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/example.conf b/certbot-apache/tests/apache-conf-files/passing/example.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/example.conf rename to certbot-apache/tests/apache-conf-files/passing/example.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/finalize-1243.apache2.conf.txt b/certbot-apache/tests/apache-conf-files/passing/finalize-1243.apache2.conf.txt similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/finalize-1243.apache2.conf.txt rename to certbot-apache/tests/apache-conf-files/passing/finalize-1243.apache2.conf.txt diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/finalize-1243.conf b/certbot-apache/tests/apache-conf-files/passing/finalize-1243.conf similarity index 98% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/finalize-1243.conf rename to certbot-apache/tests/apache-conf-files/passing/finalize-1243.conf index 0918e5669..dbfae3765 100644 --- a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/finalize-1243.conf +++ b/certbot-apache/tests/apache-conf-files/passing/finalize-1243.conf @@ -1,7 +1,7 @@ #LoadModule ssl_module modules/mod_ssl.so -Listen 443 - +Listen 4443 + # The ServerName directive sets the request scheme, hostname and port that # the server uses to identify itself. This is used when creating # redirection URLs. In the context of virtual hosts, the ServerName diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/graphite-quote-1934.conf b/certbot-apache/tests/apache-conf-files/passing/graphite-quote-1934.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/graphite-quote-1934.conf rename to certbot-apache/tests/apache-conf-files/passing/graphite-quote-1934.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/ipv6-1143.conf b/certbot-apache/tests/apache-conf-files/passing/ipv6-1143.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/ipv6-1143.conf rename to certbot-apache/tests/apache-conf-files/passing/ipv6-1143.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/ipv6-1143b.conf b/certbot-apache/tests/apache-conf-files/passing/ipv6-1143b.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/ipv6-1143b.conf rename to certbot-apache/tests/apache-conf-files/passing/ipv6-1143b.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/ipv6-1143c.conf b/certbot-apache/tests/apache-conf-files/passing/ipv6-1143c.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/ipv6-1143c.conf rename to certbot-apache/tests/apache-conf-files/passing/ipv6-1143c.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/ipv6-1143d.conf b/certbot-apache/tests/apache-conf-files/passing/ipv6-1143d.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/ipv6-1143d.conf rename to certbot-apache/tests/apache-conf-files/passing/ipv6-1143d.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/missing-quote-1724.conf b/certbot-apache/tests/apache-conf-files/passing/missing-quote-1724.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/missing-quote-1724.conf rename to certbot-apache/tests/apache-conf-files/passing/missing-quote-1724.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/modmacro-1385.conf b/certbot-apache/tests/apache-conf-files/passing/modmacro-1385.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/modmacro-1385.conf rename to certbot-apache/tests/apache-conf-files/passing/modmacro-1385.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/owncloud-1264.conf b/certbot-apache/tests/apache-conf-files/passing/owncloud-1264.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/owncloud-1264.conf rename to certbot-apache/tests/apache-conf-files/passing/owncloud-1264.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/rewrite-quote-1960.conf b/certbot-apache/tests/apache-conf-files/passing/rewrite-quote-1960.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/rewrite-quote-1960.conf rename to certbot-apache/tests/apache-conf-files/passing/rewrite-quote-1960.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/roundcube-1222.conf b/certbot-apache/tests/apache-conf-files/passing/roundcube-1222.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/roundcube-1222.conf rename to certbot-apache/tests/apache-conf-files/passing/roundcube-1222.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/section-continuations-2525.conf b/certbot-apache/tests/apache-conf-files/passing/section-continuations-2525.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/section-continuations-2525.conf rename to certbot-apache/tests/apache-conf-files/passing/section-continuations-2525.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/section-empty-continuations-2731.conf b/certbot-apache/tests/apache-conf-files/passing/section-empty-continuations-2731.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/section-empty-continuations-2731.conf rename to certbot-apache/tests/apache-conf-files/passing/section-empty-continuations-2731.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/semacode-1598.conf b/certbot-apache/tests/apache-conf-files/passing/semacode-1598.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/semacode-1598.conf rename to certbot-apache/tests/apache-conf-files/passing/semacode-1598.conf diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/sslrequire-wordlist-1827.htaccess b/certbot-apache/tests/apache-conf-files/passing/sslrequire-wordlist-1827.htaccess similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/sslrequire-wordlist-1827.htaccess rename to certbot-apache/tests/apache-conf-files/passing/sslrequire-wordlist-1827.htaccess diff --git a/certbot-apache/certbot_apache/tests/apache-conf-files/passing/two-blocks-one-line-1693.conf b/certbot-apache/tests/apache-conf-files/passing/two-blocks-one-line-1693.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/apache-conf-files/passing/two-blocks-one-line-1693.conf rename to certbot-apache/tests/apache-conf-files/passing/two-blocks-one-line-1693.conf diff --git a/certbot-apache/certbot_apache/tests/autohsts_test.py b/certbot-apache/tests/autohsts_test.py similarity index 80% rename from certbot-apache/certbot_apache/tests/autohsts_test.py rename to certbot-apache/tests/autohsts_test.py index bf92a13ff..c9901ecdb 100644 --- a/certbot-apache/certbot_apache/tests/autohsts_test.py +++ b/certbot-apache/tests/autohsts_test.py @@ -1,14 +1,14 @@ -# pylint: disable=too-many-public-methods,too-many-lines -"""Test for certbot_apache.configurator AutoHSTS functionality""" +# pylint: disable=too-many-lines +"""Test for certbot_apache._internal.configurator AutoHSTS functionality""" import re import unittest + import mock -# six is used in mock.patch() -import six # pylint: disable=unused-import +import six # pylint: disable=unused-import # six is used in mock.patch() from certbot import errors -from certbot_apache import constants -from certbot_apache.tests import util +from certbot_apache._internal import constants +import util class AutoHSTSTest(util.ApacheTest): @@ -35,27 +35,28 @@ class AutoHSTSTest(util.ApacheTest): pat = '(?:[ "]|^)(strict-transport-security)(?:[ "]|$)' for head in header_path: if re.search(pat, self.config.parser.aug.get(head).lower()): - return self.config.parser.aug.get(head.replace("arg[3]", - "arg[4]")) + return self.config.parser.aug.get( + head.replace("arg[3]", "arg[4]")) + return None # pragma: no cover - @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart") - @mock.patch("certbot_apache.configurator.ApacheConfigurator.enable_mod") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.restart") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.enable_mod") def test_autohsts_enable_headers_mod(self, mock_enable, _restart): self.config.parser.modules.discard("headers_module") self.config.parser.modules.discard("mod_header.c") self.config.enable_autohsts(mock.MagicMock(), ["ocspvhost.com"]) self.assertTrue(mock_enable.called) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.restart") def test_autohsts_deploy_already_exists(self, _restart): self.config.enable_autohsts(mock.MagicMock(), ["ocspvhost.com"]) self.assertRaises(errors.PluginEnhancementAlreadyPresent, self.config.enable_autohsts, mock.MagicMock(), ["ocspvhost.com"]) - @mock.patch("certbot_apache.constants.AUTOHSTS_FREQ", 0) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart") - @mock.patch("certbot_apache.configurator.ApacheConfigurator.prepare") + @mock.patch("certbot_apache._internal.constants.AUTOHSTS_FREQ", 0) + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.restart") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.prepare") def test_autohsts_increase(self, mock_prepare, _mock_restart): self.config._prepared = False maxage = "\"max-age={0}\"" @@ -73,8 +74,8 @@ class AutoHSTSTest(util.ApacheTest): inc_val) self.assertTrue(mock_prepare.called) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart") - @mock.patch("certbot_apache.configurator.ApacheConfigurator._autohsts_increase") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.restart") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator._autohsts_increase") def test_autohsts_increase_noop(self, mock_increase, _restart): maxage = "\"max-age={0}\"" initial_val = maxage.format(constants.AUTOHSTS_STEPS[0]) @@ -88,8 +89,8 @@ class AutoHSTSTest(util.ApacheTest): self.assertFalse(mock_increase.called) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart") - @mock.patch("certbot_apache.constants.AUTOHSTS_FREQ", 0) + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.restart") + @mock.patch("certbot_apache._internal.constants.AUTOHSTS_FREQ", 0) def test_autohsts_increase_no_header(self, _restart): self.config.enable_autohsts(mock.MagicMock(), ["ocspvhost.com"]) # Remove the header @@ -101,8 +102,8 @@ class AutoHSTSTest(util.ApacheTest): self.config.update_autohsts, mock.MagicMock()) - @mock.patch("certbot_apache.constants.AUTOHSTS_FREQ", 0) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart") + @mock.patch("certbot_apache._internal.constants.AUTOHSTS_FREQ", 0) + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.restart") def test_autohsts_increase_and_make_permanent(self, _mock_restart): maxage = "\"max-age={0}\"" max_val = maxage.format(constants.AUTOHSTS_PERMANENT) @@ -140,18 +141,18 @@ class AutoHSTSTest(util.ApacheTest): # Make sure that the execution does not continue when no entries in store self.assertFalse(self.config.storage.put.called) - @mock.patch("certbot_apache.display_ops.select_vhost") + @mock.patch("certbot_apache._internal.display_ops.select_vhost") def test_autohsts_no_ssl_vhost(self, mock_select): mock_select.return_value = self.vh_truth[0] - with mock.patch("certbot_apache.configurator.logger.warning") as mock_log: + with mock.patch("certbot_apache._internal.configurator.logger.warning") as mock_log: self.assertRaises(errors.PluginError, self.config.enable_autohsts, mock.MagicMock(), "invalid.example.com") self.assertTrue( "Certbot was not able to find SSL" in mock_log.call_args[0][0]) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart") - @mock.patch("certbot_apache.configurator.ApacheConfigurator.add_vhost_id") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.restart") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.add_vhost_id") def test_autohsts_dont_enhance_twice(self, mock_id, _restart): mock_id.return_value = "1234567" self.config.enable_autohsts(mock.MagicMock(), @@ -176,7 +177,7 @@ class AutoHSTSTest(util.ApacheTest): self.config._autohsts_fetch_state() self.config._autohsts["orphan_id"] = {"laststep": 999, "timestamp": 0} self.config._autohsts_save_state() - with mock.patch("certbot_apache.configurator.logger.warning") as mock_log: + with mock.patch("certbot_apache._internal.configurator.logger.warning") as mock_log: self.config.deploy_autohsts(mock.MagicMock()) self.assertTrue(mock_log.called) self.assertTrue( diff --git a/certbot-apache/tests/centos6_test.py b/certbot-apache/tests/centos6_test.py new file mode 100644 index 000000000..15d086600 --- /dev/null +++ b/certbot-apache/tests/centos6_test.py @@ -0,0 +1,221 @@ +"""Test for certbot_apache._internal.configurator for CentOS 6 overrides""" +import unittest + +from certbot.compat import os +from certbot.errors import MisconfigurationError +from certbot_apache._internal import obj +from certbot_apache._internal import override_centos +from certbot_apache._internal import parser +import util + + +def get_vh_truth(temp_dir, config_name): + """Return the ground truth for the specified directory.""" + prefix = os.path.join( + temp_dir, config_name, "httpd/conf.d") + + aug_pre = "/files" + prefix + vh_truth = [ + obj.VirtualHost( + os.path.join(prefix, "test.example.com.conf"), + os.path.join(aug_pre, "test.example.com.conf/VirtualHost"), + set([obj.Addr.fromstring("*:80")]), + False, True, "test.example.com"), + obj.VirtualHost( + os.path.join(prefix, "ssl.conf"), + os.path.join(aug_pre, "ssl.conf/VirtualHost"), + set([obj.Addr.fromstring("_default_:443")]), + True, True, None) + ] + return vh_truth + +class CentOS6Tests(util.ApacheTest): + """Tests for CentOS 6""" + + def setUp(self): # pylint: disable=arguments-differ + test_dir = "centos6_apache/apache" + config_root = "centos6_apache/apache/httpd" + vhost_root = "centos6_apache/apache/httpd/conf.d" + super(CentOS6Tests, self).setUp(test_dir=test_dir, + config_root=config_root, + vhost_root=vhost_root) + + self.config = util.get_apache_configurator( + self.config_path, self.vhost_path, self.config_dir, self.work_dir, + version=(2, 2, 15), os_info="centos") + self.vh_truth = get_vh_truth( + self.temp_dir, "centos6_apache/apache") + + def test_get_parser(self): + self.assertTrue(isinstance(self.config.parser, + override_centos.CentOSParser)) + + def test_get_virtual_hosts(self): + """Make sure all vhosts are being properly found.""" + vhs = self.config.get_virtual_hosts() + self.assertEqual(len(vhs), 2) + found = 0 + + for vhost in vhs: + for centos_truth in self.vh_truth: + if vhost == centos_truth: + found += 1 + break + else: + raise Exception("Missed: %s" % vhost) # pragma: no cover + self.assertEqual(found, 2) + + def test_loadmod_default(self): + ssl_loadmods = self.config.parser.find_dir( + "LoadModule", "ssl_module", exclude=False) + self.assertEqual(len(ssl_loadmods), 1) + # Make sure the LoadModule ssl_module is in ssl.conf (default) + self.assertTrue("ssl.conf" in ssl_loadmods[0]) + # ...and that it's not inside of + self.assertFalse("IfModule" in ssl_loadmods[0]) + + # Get the example vhost + self.config.assoc["test.example.com"] = self.vh_truth[0] + self.config.deploy_cert( + "random.demo", "example/cert.pem", "example/key.pem", + "example/cert_chain.pem", "example/fullchain.pem") + self.config.save() + + post_loadmods = self.config.parser.find_dir( + "LoadModule", "ssl_module", exclude=False) + + # We should now have LoadModule ssl_module in root conf and ssl.conf + self.assertEqual(len(post_loadmods), 2) + for lm in post_loadmods: + # lm[:-7] removes "/arg[#]" from the path + arguments = self.config.parser.get_all_args(lm[:-7]) + self.assertEqual(arguments, ["ssl_module", "modules/mod_ssl.so"]) + # ...and both of them should be wrapped in + # lm[:-17] strips off /directive/arg[1] from the path. + ifmod_args = self.config.parser.get_all_args(lm[:-17]) + self.assertTrue("!mod_ssl.c" in ifmod_args) + + def test_loadmod_multiple(self): + sslmod_args = ["ssl_module", "modules/mod_ssl.so"] + # Adds another LoadModule to main httpd.conf in addtition to ssl.conf + self.config.parser.add_dir(self.config.parser.loc["default"], "LoadModule", + sslmod_args) + self.config.save() + pre_loadmods = self.config.parser.find_dir( + "LoadModule", "ssl_module", exclude=False) + # LoadModules are not within IfModule blocks + self.assertFalse(any(["ifmodule" in m.lower() for m in pre_loadmods])) + self.config.assoc["test.example.com"] = self.vh_truth[0] + self.config.deploy_cert( + "random.demo", "example/cert.pem", "example/key.pem", + "example/cert_chain.pem", "example/fullchain.pem") + post_loadmods = self.config.parser.find_dir( + "LoadModule", "ssl_module", exclude=False) + + for mod in post_loadmods: + self.assertTrue(self.config.parser.not_modssl_ifmodule(mod)) #pylint: disable=no-member + + def test_loadmod_rootconf_exists(self): + sslmod_args = ["ssl_module", "modules/mod_ssl.so"] + rootconf_ifmod = self.config.parser.get_ifmod( + parser.get_aug_path(self.config.parser.loc["default"]), + "!mod_ssl.c", beginning=True) + self.config.parser.add_dir(rootconf_ifmod[:-1], "LoadModule", sslmod_args) + self.config.save() + # Get the example vhost + self.config.assoc["test.example.com"] = self.vh_truth[0] + self.config.deploy_cert( + "random.demo", "example/cert.pem", "example/key.pem", + "example/cert_chain.pem", "example/fullchain.pem") + self.config.save() + + root_loadmods = self.config.parser.find_dir( + "LoadModule", "ssl_module", + start=parser.get_aug_path(self.config.parser.loc["default"]), + exclude=False) + + mods = [lm for lm in root_loadmods if self.config.parser.loc["default"] in lm] + + self.assertEqual(len(mods), 1) + # [:-7] removes "/arg[#]" from the path + self.assertEqual( + self.config.parser.get_all_args(mods[0][:-7]), + sslmod_args) + + def test_neg_loadmod_already_on_path(self): + loadmod_args = ["ssl_module", "modules/mod_ssl.so"] + ifmod = self.config.parser.get_ifmod( + self.vh_truth[1].path, "!mod_ssl.c", beginning=True) + self.config.parser.add_dir(ifmod[:-1], "LoadModule", loadmod_args) + self.config.parser.add_dir(self.vh_truth[1].path, "LoadModule", loadmod_args) + self.config.save() + pre_loadmods = self.config.parser.find_dir( + "LoadModule", "ssl_module", start=self.vh_truth[1].path, exclude=False) + self.assertEqual(len(pre_loadmods), 2) + # The ssl.conf now has two LoadModule directives, one inside of + # !mod_ssl.c IfModule + self.config.assoc["test.example.com"] = self.vh_truth[0] + self.config.deploy_cert( + "random.demo", "example/cert.pem", "example/key.pem", + "example/cert_chain.pem", "example/fullchain.pem") + self.config.save() + # Ensure that the additional LoadModule wasn't written into the IfModule + post_loadmods = self.config.parser.find_dir( + "LoadModule", "ssl_module", start=self.vh_truth[1].path, exclude=False) + self.assertEqual(len(post_loadmods), 1) + + def test_loadmod_non_duplicate(self): + # the modules/mod_ssl.so exists in ssl.conf + sslmod_args = ["ssl_module", "modules/mod_somethingelse.so"] + rootconf_ifmod = self.config.parser.get_ifmod( + parser.get_aug_path(self.config.parser.loc["default"]), + "!mod_ssl.c", beginning=True) + self.config.parser.add_dir(rootconf_ifmod[:-1], "LoadModule", sslmod_args) + self.config.save() + self.config.assoc["test.example.com"] = self.vh_truth[0] + pre_matches = self.config.parser.find_dir("LoadModule", + "ssl_module", exclude=False) + + self.assertRaises(MisconfigurationError, self.config.deploy_cert, + "random.demo", "example/cert.pem", "example/key.pem", + "example/cert_chain.pem", "example/fullchain.pem") + + post_matches = self.config.parser.find_dir("LoadModule", + "ssl_module", exclude=False) + # Make sure that none was changed + self.assertEqual(pre_matches, post_matches) + + def test_loadmod_not_found(self): + # Remove all existing LoadModule ssl_module... directives + orig_loadmods = self.config.parser.find_dir("LoadModule", + "ssl_module", + exclude=False) + for mod in orig_loadmods: + noarg_path = mod.rpartition("/")[0] + self.config.parser.aug.remove(noarg_path) + self.config.save() + self.config.deploy_cert( + "random.demo", "example/cert.pem", "example/key.pem", + "example/cert_chain.pem", "example/fullchain.pem") + + post_loadmods = self.config.parser.find_dir("LoadModule", + "ssl_module", + exclude=False) + self.assertFalse(post_loadmods) + + def test_no_ifmod_search_false(self): + #pylint: disable=no-member + + self.assertFalse(self.config.parser.not_modssl_ifmodule( + "/path/does/not/include/ifmod" + )) + self.assertFalse(self.config.parser.not_modssl_ifmodule( + "" + )) + self.assertFalse(self.config.parser.not_modssl_ifmodule( + "/path/includes/IfModule/but/no/arguments" + )) + + +if __name__ == "__main__": + unittest.main() # pragma: no cover diff --git a/certbot-apache/certbot_apache/tests/centos_test.py b/certbot-apache/tests/centos_test.py similarity index 60% rename from certbot-apache/certbot_apache/tests/centos_test.py rename to certbot-apache/tests/centos_test.py index a27916c32..8959d73b8 100644 --- a/certbot-apache/certbot_apache/tests/centos_test.py +++ b/certbot-apache/tests/centos_test.py @@ -1,14 +1,15 @@ -"""Test for certbot_apache.configurator for Centos overrides""" -import os +"""Test for certbot_apache._internal.configurator for Centos overrides""" import unittest import mock from certbot import errors +from certbot.compat import filesystem +from certbot.compat import os +from certbot_apache._internal import obj +from certbot_apache._internal import override_centos +import util -from certbot_apache import obj -from certbot_apache import override_centos -from certbot_apache.tests import util def get_vh_truth(temp_dir, config_name): """Return the ground truth for the specified directory.""" @@ -30,6 +31,59 @@ def get_vh_truth(temp_dir, config_name): ] return vh_truth +class FedoraRestartTest(util.ApacheTest): + """Tests for Fedora specific self-signed certificate override""" + + def setUp(self): # pylint: disable=arguments-differ + test_dir = "centos7_apache/apache" + config_root = "centos7_apache/apache/httpd" + vhost_root = "centos7_apache/apache/httpd/conf.d" + super(FedoraRestartTest, self).setUp(test_dir=test_dir, + config_root=config_root, + vhost_root=vhost_root) + self.config = util.get_apache_configurator( + self.config_path, self.vhost_path, self.config_dir, self.work_dir, + os_info="fedora_old") + self.vh_truth = get_vh_truth( + self.temp_dir, "centos7_apache/apache") + + def _run_fedora_test(self): + self.assertIsInstance(self.config, override_centos.CentOSConfigurator) + with mock.patch("certbot.util.get_os_info") as mock_info: + mock_info.return_value = ["fedora", "28"] + self.config.config_test() + + def test_non_fedora_error(self): + c_test = "certbot_apache._internal.configurator.ApacheConfigurator.config_test" + with mock.patch(c_test) as mock_test: + mock_test.side_effect = errors.MisconfigurationError + with mock.patch("certbot.util.get_os_info") as mock_info: + mock_info.return_value = ["not_fedora"] + self.assertRaises(errors.MisconfigurationError, + self.config.config_test) + + def test_fedora_restart_error(self): + c_test = "certbot_apache._internal.configurator.ApacheConfigurator.config_test" + with mock.patch(c_test) as mock_test: + # First call raises error, second doesn't + mock_test.side_effect = [errors.MisconfigurationError, ''] + with mock.patch("certbot.util.run_script") as mock_run: + mock_run.side_effect = errors.SubprocessError + self.assertRaises(errors.MisconfigurationError, + self._run_fedora_test) + + def test_fedora_restart(self): + c_test = "certbot_apache._internal.configurator.ApacheConfigurator.config_test" + with mock.patch(c_test) as mock_test: + with mock.patch("certbot.util.run_script") as mock_run: + # First call raises error, second doesn't + mock_test.side_effect = [errors.MisconfigurationError, ''] + self._run_fedora_test() + self.assertEqual(mock_test.call_count, 2) + self.assertEqual(mock_run.call_args[0][0], + ['systemctl', 'restart', 'httpd']) + + class MultipleVhostsTestCentOS(util.ApacheTest): """Multiple vhost tests for CentOS / RHEL family of distros""" @@ -50,10 +104,9 @@ class MultipleVhostsTestCentOS(util.ApacheTest): self.temp_dir, "centos7_apache/apache") def test_get_parser(self): - self.assertTrue(isinstance(self.config.parser, - override_centos.CentOSParser)) + self.assertIsInstance(self.config.parser, override_centos.CentOSParser) - @mock.patch("certbot_apache.parser.ApacheParser._get_runtime_cfg") + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") def test_opportunistic_httpd_runtime_parsing(self, mock_get): define_val = ( 'Define: TEST1\n' @@ -102,12 +155,12 @@ class MultipleVhostsTestCentOS(util.ApacheTest): raise Exception("Missed: %s" % vhost) # pragma: no cover self.assertEqual(found, 2) - @mock.patch("certbot_apache.parser.ApacheParser._get_runtime_cfg") + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") def test_get_sysconfig_vars(self, mock_cfg): """Make sure we read the sysconfig OPTIONS variable correctly""" # Return nothing for the process calls mock_cfg.return_value = "" - self.config.parser.sysconfig_filep = os.path.realpath( + self.config.parser.sysconfig_filep = filesystem.realpath( os.path.join(self.config.parser.root, "../sysconfig/httpd")) self.config.parser.variables = {} @@ -123,13 +176,13 @@ class MultipleVhostsTestCentOS(util.ApacheTest): self.assertTrue("MOCK_NOSEP" in self.config.parser.variables.keys()) self.assertEqual("NOSEP_VAL", self.config.parser.variables["NOSEP_TWO"]) - @mock.patch("certbot_apache.configurator.util.run_script") + @mock.patch("certbot_apache._internal.configurator.util.run_script") def test_alt_restart_works(self, mock_run_script): mock_run_script.side_effect = [None, errors.SubprocessError, None] self.config.restart() self.assertEqual(mock_run_script.call_count, 3) - @mock.patch("certbot_apache.configurator.util.run_script") + @mock.patch("certbot_apache._internal.configurator.util.run_script") def test_alt_restart_errors(self, mock_run_script): mock_run_script.side_effect = [None, errors.SubprocessError, diff --git a/certbot-apache/certbot_apache/tests/complex_parsing_test.py b/certbot-apache/tests/complex_parsing_test.py similarity index 96% rename from certbot-apache/certbot_apache/tests/complex_parsing_test.py rename to certbot-apache/tests/complex_parsing_test.py index a296fb0eb..8b795b0b6 100644 --- a/certbot-apache/certbot_apache/tests/complex_parsing_test.py +++ b/certbot-apache/tests/complex_parsing_test.py @@ -1,11 +1,10 @@ -"""Tests for certbot_apache.parser.""" -import os +"""Tests for certbot_apache._internal.parser.""" import shutil import unittest from certbot import errors - -from certbot_apache.tests import util +from certbot.compat import os +import util class ComplexParserTest(util.ParserTest): @@ -88,7 +87,7 @@ class ComplexParserTest(util.ParserTest): def verify_fnmatch(self, arg, hit=True): """Test if Include was correctly parsed.""" - from certbot_apache import parser + from certbot_apache._internal import parser self.parser.add_dir(parser.get_aug_path(self.parser.loc["default"]), "Include", [arg]) if hit: diff --git a/certbot-apache/certbot_apache/tests/augeas_configurator_test.py b/certbot-apache/tests/configurator_reverter_test.py similarity index 61% rename from certbot-apache/certbot_apache/tests/augeas_configurator_test.py rename to certbot-apache/tests/configurator_reverter_test.py index c121ecdf3..ad8e73347 100644 --- a/certbot-apache/certbot_apache/tests/augeas_configurator_test.py +++ b/certbot-apache/tests/configurator_reverter_test.py @@ -1,21 +1,19 @@ -"""Test for certbot_apache.augeas_configurator.""" -import os +"""Test for certbot_apache._internal.configurator implementations of reverter""" import shutil import unittest import mock from certbot import errors - -from certbot_apache.tests import util +import util -class AugeasConfiguratorTest(util.ApacheTest): - """Test for Augeas Configurator base class.""" +class ConfiguratorReverterTest(util.ApacheTest): + """Test for ApacheConfigurator reverter methods""" def setUp(self): # pylint: disable=arguments-differ - super(AugeasConfiguratorTest, self).setUp() + super(ConfiguratorReverterTest, self).setUp() self.config = util.get_apache_configurator( self.config_path, self.vhost_path, self.config_dir, self.work_dir) @@ -28,20 +26,6 @@ class AugeasConfiguratorTest(util.ApacheTest): shutil.rmtree(self.work_dir) shutil.rmtree(self.temp_dir) - def test_bad_parse(self): - # pylint: disable=protected-access - self.config.parser.parse_file(os.path.join( - self.config.parser.root, "conf-available", "bad_conf_file.conf")) - self.assertRaises( - errors.PluginError, self.config.check_parsing_errors, "httpd.aug") - - def test_bad_save(self): - mock_save = mock.Mock() - mock_save.side_effect = IOError - self.config.aug.save = mock_save - - self.assertRaises(errors.PluginError, self.config.save) - def test_bad_save_checkpoint(self): self.config.reverter.add_to_checkpoint = mock.Mock( side_effect=errors.ReverterError) @@ -63,23 +47,9 @@ class AugeasConfiguratorTest(util.ApacheTest): self.assertTrue(mock_finalize.is_called) - def test_recovery_routine(self): - mock_load = mock.Mock() - self.config.aug.load = mock_load - - self.config.recovery_routine() - self.assertEqual(mock_load.call_count, 1) - - def test_recovery_routine_error(self): - self.config.reverter.recovery_routine = mock.Mock( - side_effect=errors.ReverterError) - - self.assertRaises( - errors.PluginError, self.config.recovery_routine) - def test_revert_challenge_config(self): mock_load = mock.Mock() - self.config.aug.load = mock_load + self.config.parser.aug.load = mock_load self.config.revert_challenge_config() self.assertEqual(mock_load.call_count, 1) @@ -93,7 +63,7 @@ class AugeasConfiguratorTest(util.ApacheTest): def test_rollback_checkpoints(self): mock_load = mock.Mock() - self.config.aug.load = mock_load + self.config.parser.aug.load = mock_load self.config.rollback_checkpoints() self.assertEqual(mock_load.call_count, 1) @@ -103,13 +73,11 @@ class AugeasConfiguratorTest(util.ApacheTest): side_effect=errors.ReverterError) self.assertRaises(errors.PluginError, self.config.rollback_checkpoints) - def test_view_config_changes(self): - self.config.view_config_changes() - - def test_view_config_changes_error(self): - self.config.reverter.view_config_changes = mock.Mock( - side_effect=errors.ReverterError) - self.assertRaises(errors.PluginError, self.config.view_config_changes) + def test_recovery_routine_reload(self): + mock_load = mock.Mock() + self.config.parser.aug.load = mock_load + self.config.recovery_routine() + self.assertEqual(mock_load.call_count, 1) if __name__ == "__main__": diff --git a/certbot-apache/certbot_apache/tests/configurator_test.py b/certbot-apache/tests/configurator_test.py similarity index 89% rename from certbot-apache/certbot_apache/tests/configurator_test.py rename to certbot-apache/tests/configurator_test.py index 4aaa23ea4..9fab5ea5d 100644 --- a/certbot-apache/certbot_apache/tests/configurator_test.py +++ b/certbot-apache/tests/configurator_test.py @@ -1,36 +1,32 @@ -# pylint: disable=too-many-public-methods,too-many-lines -"""Test for certbot_apache.configurator.""" -import os +# pylint: disable=too-many-lines +"""Test for certbot_apache._internal.configurator.""" +import copy import shutil import socket import tempfile import unittest import mock -# six is used in mock.patch() -import six # pylint: disable=unused-import +import six # pylint: disable=unused-import # six is used in mock.patch() from acme import challenges - from certbot import achallenges from certbot import crypto_util from certbot import errors - +from certbot.compat import filesystem +from certbot.compat import os from certbot.tests import acme_util from certbot.tests import util as certbot_util - -from certbot_apache import apache_util -from certbot_apache import constants -from certbot_apache import parser -from certbot_apache import obj - -from certbot_apache.tests import util +from certbot_apache._internal import apache_util +from certbot_apache._internal import constants +from certbot_apache._internal import obj +from certbot_apache._internal import parser +import util class MultipleVhostsTest(util.ApacheTest): """Test two standard well-configured HTTP vhosts.""" - def setUp(self): # pylint: disable=arguments-differ super(MultipleVhostsTest, self).setUp() @@ -46,33 +42,22 @@ class MultipleVhostsTest(util.ApacheTest): def mocked_deploy_cert(*args, **kwargs): """a helper to mock a deployed cert""" - g_mod = "certbot_apache.configurator.ApacheConfigurator.enable_mod" + g_mod = "certbot_apache._internal.configurator.ApacheConfigurator.enable_mod" with mock.patch(g_mod): config.real_deploy_cert(*args, **kwargs) self.config.deploy_cert = mocked_deploy_cert return self.config - @mock.patch("certbot_apache.configurator.ApacheConfigurator.init_augeas") - @mock.patch("certbot_apache.configurator.path_surgery") - def test_prepare_no_install(self, mock_surgery, _init_augeas): + @mock.patch("certbot_apache._internal.configurator.path_surgery") + def test_prepare_no_install(self, mock_surgery): silly_path = {"PATH": "/tmp/nothingness2342"} mock_surgery.return_value = False with mock.patch.dict('os.environ', silly_path): self.assertRaises(errors.NoInstallationError, self.config.prepare) self.assertEqual(mock_surgery.call_count, 1) - @mock.patch("certbot_apache.augeas_configurator.AugeasConfigurator.init_augeas") - def test_prepare_no_augeas(self, mock_init_augeas): - """ Test augeas initialization ImportError """ - def side_effect_error(): - """ Side effect error for the test """ - raise ImportError - mock_init_augeas.side_effect = side_effect_error - self.assertRaises( - errors.NoInstallationError, self.config.prepare) - - @mock.patch("certbot_apache.parser.ApacheParser") - @mock.patch("certbot_apache.configurator.util.exe_exists") + @mock.patch("certbot_apache._internal.parser.ApacheParser") + @mock.patch("certbot_apache._internal.configurator.util.exe_exists") def test_prepare_version(self, mock_exe_exists, _): mock_exe_exists.return_value = True self.config.version = None @@ -82,24 +67,14 @@ class MultipleVhostsTest(util.ApacheTest): self.assertRaises( errors.NotSupportedError, self.config.prepare) - @mock.patch("certbot_apache.parser.ApacheParser") - @mock.patch("certbot_apache.configurator.util.exe_exists") - def test_prepare_old_aug(self, mock_exe_exists, _): - mock_exe_exists.return_value = True - self.config.config_test = mock.Mock() - # pylint: disable=protected-access - self.config._check_aug_version = mock.Mock(return_value=False) - self.assertRaises( - errors.NotSupportedError, self.config.prepare) - def test_prepare_locked(self): server_root = self.config.conf("server-root") self.config.config_test = mock.Mock() os.remove(os.path.join(server_root, ".certbot.lock")) certbot_util.lock_and_call(self._test_prepare_locked, server_root) - @mock.patch("certbot_apache.parser.ApacheParser") - @mock.patch("certbot_apache.configurator.util.exe_exists") + @mock.patch("certbot_apache._internal.parser.ApacheParser") + @mock.patch("certbot_apache._internal.configurator.util.exe_exists") def _test_prepare_locked(self, unused_parser, unused_exe_exists): try: self.config.prepare() @@ -111,18 +86,45 @@ class MultipleVhostsTest(util.ApacheTest): self.fail("Exception wasn't raised!") def test_add_parser_arguments(self): # pylint: disable=no-self-use - from certbot_apache.configurator import ApacheConfigurator + from certbot_apache._internal.configurator import ApacheConfigurator # Weak test.. ApacheConfigurator.add_parser_arguments(mock.MagicMock()) + def test_docs_parser_arguments(self): + os.environ["CERTBOT_DOCS"] = "1" + from certbot_apache._internal.configurator import ApacheConfigurator + mock_add = mock.MagicMock() + ApacheConfigurator.add_parser_arguments(mock_add) + parserargs = ["server_root", "enmod", "dismod", "le_vhost_ext", + "vhost_root", "logs_root", "challenge_location", + "handle_modules", "handle_sites", "ctl"] + exp = dict() + + for k in ApacheConfigurator.OS_DEFAULTS: + if k in parserargs: + exp[k.replace("_", "-")] = ApacheConfigurator.OS_DEFAULTS[k] + # Special cases + exp["vhost-root"] = None + + found = set() + for call in mock_add.call_args_list: + found.add(call[0][0]) + + # Make sure that all (and only) the expected values exist + self.assertEqual(len(mock_add.call_args_list), len(found)) + for e in exp: + self.assertTrue(e in found) + + del os.environ["CERTBOT_DOCS"] + def test_add_parser_arguments_all_configurators(self): # pylint: disable=no-self-use - from certbot_apache.entrypoint import OVERRIDE_CLASSES + from certbot_apache._internal.entrypoint import OVERRIDE_CLASSES for cls in OVERRIDE_CLASSES.values(): cls.add_parser_arguments(mock.MagicMock()) def test_all_configurators_defaults_defined(self): - from certbot_apache.entrypoint import OVERRIDE_CLASSES - from certbot_apache.configurator import ApacheConfigurator + from certbot_apache._internal.entrypoint import OVERRIDE_CLASSES + from certbot_apache._internal.configurator import ApacheConfigurator parameters = set(ApacheConfigurator.OS_DEFAULTS.keys()) for cls in OVERRIDE_CLASSES.values(): self.assertTrue(parameters.issubset(set(cls.OS_DEFAULTS.keys()))) @@ -139,11 +141,12 @@ class MultipleVhostsTest(util.ApacheTest): names = self.config.get_all_names() self.assertEqual(names, set( ["certbot.demo", "ocspvhost.com", "encryption-example.demo", - "nonsym.link", "vhost.in.rootconf", "www.certbot.demo"] + "nonsym.link", "vhost.in.rootconf", "www.certbot.demo", + "duplicate.example.com"] )) @certbot_util.patch_get_utility() - @mock.patch("certbot_apache.configurator.socket.gethostbyaddr") + @mock.patch("certbot_apache._internal.configurator.socket.gethostbyaddr") def test_get_all_names_addrs(self, mock_gethost, mock_getutility): mock_gethost.side_effect = [("google.com", "", ""), socket.error] mock_utility = mock_getutility() @@ -158,8 +161,7 @@ class MultipleVhostsTest(util.ApacheTest): self.config.vhosts.append(vhost) names = self.config.get_all_names() - # Names get filtered, only 5 are returned - self.assertEqual(len(names), 8) + self.assertEqual(len(names), 9) self.assertTrue("zombo.com" in names) self.assertTrue("google.com" in names) self.assertTrue("certbot.demo" in names) @@ -170,7 +172,7 @@ class MultipleVhostsTest(util.ApacheTest): self.assertEqual(self.config._create_vhost("nonexistent"), None) # pylint: disable=protected-access def test_get_aug_internal_path(self): - from certbot_apache.apache_util import get_internal_aug_path + from certbot_apache._internal.apache_util import get_internal_aug_path internal_paths = [ "Virtualhost", "IfModule/VirtualHost", "VirtualHost", "VirtualHost", "Macro/VirtualHost", "IfModule/VirtualHost", "VirtualHost", @@ -200,7 +202,7 @@ class MultipleVhostsTest(util.ApacheTest): def test_get_virtual_hosts(self): """Make sure all vhosts are being properly found.""" vhs = self.config.get_virtual_hosts() - self.assertEqual(len(vhs), 10) + self.assertEqual(len(vhs), 12) found = 0 for vhost in vhs: @@ -211,30 +213,30 @@ class MultipleVhostsTest(util.ApacheTest): else: raise Exception("Missed: %s" % vhost) # pragma: no cover - self.assertEqual(found, 10) + self.assertEqual(found, 12) # Handle case of non-debian layout get_virtual_hosts with mock.patch( - "certbot_apache.configurator.ApacheConfigurator.conf" + "certbot_apache._internal.configurator.ApacheConfigurator.conf" ) as mock_conf: mock_conf.return_value = False vhs = self.config.get_virtual_hosts() - self.assertEqual(len(vhs), 10) + self.assertEqual(len(vhs), 12) - @mock.patch("certbot_apache.display_ops.select_vhost") + @mock.patch("certbot_apache._internal.display_ops.select_vhost") def test_choose_vhost_none_avail(self, mock_select): mock_select.return_value = None self.assertRaises( errors.PluginError, self.config.choose_vhost, "none.com") - @mock.patch("certbot_apache.display_ops.select_vhost") + @mock.patch("certbot_apache._internal.display_ops.select_vhost") def test_choose_vhost_select_vhost_ssl(self, mock_select): mock_select.return_value = self.vh_truth[1] self.assertEqual( self.vh_truth[1], self.config.choose_vhost("none.com")) - @mock.patch("certbot_apache.display_ops.select_vhost") - @mock.patch("certbot_apache.obj.VirtualHost.conflicts") + @mock.patch("certbot_apache._internal.display_ops.select_vhost") + @mock.patch("certbot_apache._internal.obj.VirtualHost.conflicts") def test_choose_vhost_select_vhost_non_ssl(self, mock_conf, mock_select): mock_select.return_value = self.vh_truth[0] mock_conf.return_value = False @@ -247,8 +249,8 @@ class MultipleVhostsTest(util.ApacheTest): self.assertFalse(self.vh_truth[0].ssl) self.assertTrue(chosen_vhost.ssl) - @mock.patch("certbot_apache.configurator.ApacheConfigurator._find_best_vhost") - @mock.patch("certbot_apache.parser.ApacheParser.add_dir") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator._find_best_vhost") + @mock.patch("certbot_apache._internal.parser.ApacheParser.add_dir") def test_choose_vhost_and_servername_addition(self, mock_add, mock_find): ret_vh = self.vh_truth[8] ret_vh.enabled = False @@ -256,13 +258,13 @@ class MultipleVhostsTest(util.ApacheTest): self.config.choose_vhost("whatever.com") self.assertTrue(mock_add.called) - @mock.patch("certbot_apache.display_ops.select_vhost") + @mock.patch("certbot_apache._internal.display_ops.select_vhost") def test_choose_vhost_select_vhost_with_temp(self, mock_select): mock_select.return_value = self.vh_truth[0] chosen_vhost = self.config.choose_vhost("none.com", create_if_no_ssl=False) self.assertEqual(self.vh_truth[0], chosen_vhost) - @mock.patch("certbot_apache.display_ops.select_vhost") + @mock.patch("certbot_apache._internal.display_ops.select_vhost") def test_choose_vhost_select_vhost_conflicting_non_ssl(self, mock_select): mock_select.return_value = self.vh_truth[3] conflicting_vhost = obj.VirtualHost( @@ -322,7 +324,7 @@ class MultipleVhostsTest(util.ApacheTest): self.config.vhosts = [ vh for vh in self.config.vhosts if vh.name not in ["certbot.demo", "nonsym.link", - "encryption-example.demo", + "encryption-example.demo", "duplicate.example.com", "ocspvhost.com", "vhost.in.rootconf"] and "*.blue.purple.com" not in vh.aliases ] @@ -333,7 +335,7 @@ class MultipleVhostsTest(util.ApacheTest): def test_non_default_vhosts(self): # pylint: disable=protected-access vhosts = self.config._non_default_vhosts(self.config.vhosts) - self.assertEqual(len(vhosts), 8) + self.assertEqual(len(vhosts), 10) def test_deploy_cert_enable_new_vhost(self): # Create @@ -353,6 +355,7 @@ class MultipleVhostsTest(util.ApacheTest): """Mock method for parser.find_dir""" if directive == "Include" and argument.endswith("options-ssl-apache.conf"): return ["/path/to/whatever"] + return None # pragma: no cover mock_add = mock.MagicMock() self.config.parser.add_dir = mock_add @@ -464,8 +467,7 @@ class MultipleVhostsTest(util.ApacheTest): but an SSLCertificateKeyFile directive is missing.""" if "SSLCertificateFile" in args: return ["example/cert.pem"] - else: - return [] + return [] mock_find_dir = mock.MagicMock(return_value=[]) mock_find_dir.side_effect = side_effect @@ -645,7 +647,7 @@ class MultipleVhostsTest(util.ApacheTest): # span excludes the closing tag in older versions # of Augeas return_value = [self.vh_truth[0].filep, 1, 12, 0, 0, 0, 1142] - with mock.patch.object(self.config.aug, 'span') as mock_span: + with mock.patch.object(self.config.parser.aug, 'span') as mock_span: mock_span.return_value = return_value self.test_make_vhost_ssl() @@ -653,7 +655,7 @@ class MultipleVhostsTest(util.ApacheTest): # span includes the closing tag in newer versions # of Augeas return_value = [self.vh_truth[0].filep, 1, 12, 0, 0, 0, 1157] - with mock.patch.object(self.config.aug, 'span') as mock_span: + with mock.patch.object(self.config.parser.aug, 'span') as mock_span: mock_span.return_value = return_value self.test_make_vhost_ssl() @@ -666,8 +668,7 @@ class MultipleVhostsTest(util.ApacheTest): def test_make_vhost_ssl_nonexistent_vhost_path(self): ssl_vhost = self.config.make_vhost_ssl(self.vh_truth[1]) self.assertEqual(os.path.dirname(ssl_vhost.filep), - os.path.dirname(os.path.realpath( - self.vh_truth[1].filep))) + os.path.dirname(filesystem.realpath(self.vh_truth[1].filep))) def test_make_vhost_ssl(self): ssl_vhost = self.config.make_vhost_ssl(self.vh_truth[0]) @@ -688,7 +689,7 @@ class MultipleVhostsTest(util.ApacheTest): self.assertEqual(self.config.is_name_vhost(self.vh_truth[0]), self.config.is_name_vhost(ssl_vhost)) - self.assertEqual(len(self.config.vhosts), 11) + self.assertEqual(len(self.config.vhosts), 13) def test_clean_vhost_ssl(self): # pylint: disable=protected-access @@ -780,38 +781,25 @@ class MultipleVhostsTest(util.ApacheTest): self.config._add_name_vhost_if_necessary(self.vh_truth[0]) self.assertEqual(self.config.add_name_vhost.call_count, 2) - @mock.patch("certbot_apache.configurator.http_01.ApacheHttp01.perform") - @mock.patch("certbot_apache.configurator.tls_sni_01.ApacheTlsSni01.perform") - @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart") - def test_perform(self, mock_restart, mock_tls_perform, mock_http_perform): + @mock.patch("certbot_apache._internal.configurator.http_01.ApacheHttp01.perform") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.restart") + def test_perform(self, mock_restart, mock_http_perform): # Only tests functionality specific to configurator.perform # Note: As more challenges are offered this will have to be expanded account_key, achalls = self.get_key_and_achalls() - all_expected = [] - http_expected = [] - tls_expected = [] - for achall in achalls: - response = achall.response(account_key) - if isinstance(achall.chall, challenges.HTTP01): - http_expected.append(response) - else: - tls_expected.append(response) - all_expected.append(response) - - mock_http_perform.return_value = http_expected - mock_tls_perform.return_value = tls_expected + expected = [achall.response(account_key) for achall in achalls] + mock_http_perform.return_value = expected responses = self.config.perform(achalls) self.assertEqual(mock_http_perform.call_count, 1) - self.assertEqual(mock_tls_perform.call_count, 1) - self.assertEqual(responses, all_expected) + self.assertEqual(responses, expected) self.assertEqual(mock_restart.call_count, 1) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart") - @mock.patch("certbot_apache.parser.ApacheParser._get_runtime_cfg") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.restart") + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") def test_cleanup(self, mock_cfg, mock_restart): mock_cfg.return_value = "" _, achalls = self.get_key_and_achalls() @@ -826,8 +814,8 @@ class MultipleVhostsTest(util.ApacheTest): else: self.assertFalse(mock_restart.called) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart") - @mock.patch("certbot_apache.parser.ApacheParser._get_runtime_cfg") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.restart") + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") def test_cleanup_no_errors(self, mock_cfg, mock_restart): mock_cfg.return_value = "" _, achalls = self.get_key_and_achalls() @@ -864,11 +852,11 @@ class MultipleVhostsTest(util.ApacheTest): mock_script.side_effect = errors.SubprocessError("Can't find program") self.assertRaises(errors.PluginError, self.config.get_version) - @mock.patch("certbot_apache.configurator.util.run_script") + @mock.patch("certbot_apache._internal.configurator.util.run_script") def test_restart(self, _): self.config.restart() - @mock.patch("certbot_apache.configurator.util.run_script") + @mock.patch("certbot_apache._internal.configurator.util.run_script") def test_restart_bad_process(self, mock_run_script): mock_run_script.side_effect = [None, errors.SubprocessError] @@ -911,8 +899,8 @@ class MultipleVhostsTest(util.ApacheTest): self.assertEqual(self.vh_truth[0].name, res.name) self.assertEqual(self.vh_truth[0].aliases, res.aliases) - @mock.patch("certbot_apache.configurator.ApacheConfigurator._get_http_vhost") - @mock.patch("certbot_apache.display_ops.select_vhost") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator._get_http_vhost") + @mock.patch("certbot_apache._internal.display_ops.select_vhost") @mock.patch("certbot.util.exe_exists") def test_enhance_unknown_vhost(self, mock_exe, mock_sel_vhost, mock_get): self.config.parser.modules.add("rewrite_module") @@ -935,7 +923,7 @@ class MultipleVhostsTest(util.ApacheTest): self.config.enhance, "certbot.demo", "unknown_enhancement") def test_enhance_no_ssl_vhost(self): - with mock.patch("certbot_apache.configurator.logger.warning") as mock_log: + with mock.patch("certbot_apache._internal.configurator.logger.warning") as mock_log: self.assertRaises(errors.PluginError, self.config.enhance, "certbot.demo", "redirect") # Check that correct logger.warning was printed @@ -1026,7 +1014,7 @@ class MultipleVhostsTest(util.ApacheTest): # pylint: disable=protected-access http_vh = self.config._get_http_vhost(ssl_vh) - self.assertTrue(http_vh.ssl == False) + self.assertFalse(http_vh.ssl) @mock.patch("certbot.util.run_script") @mock.patch("certbot.util.exe_exists") @@ -1215,7 +1203,7 @@ class MultipleVhostsTest(util.ApacheTest): except errors.PluginEnhancementAlreadyPresent: args_paths = self.config.parser.find_dir( "RewriteRule", None, http_vhost.path, False) - arg_vals = [self.config.aug.get(x) for x in args_paths] + arg_vals = [self.config.parser.aug.get(x) for x in args_paths] self.assertEqual(arg_vals, constants.REWRITE_HTTPS_ARGS) @@ -1240,7 +1228,7 @@ class MultipleVhostsTest(util.ApacheTest): self.config.choose_vhost("red.blue.purple.com") self.config.enhance("red.blue.purple.com", "redirect") - verify_no_redirect = ("certbot_apache.configurator." + verify_no_redirect = ("certbot_apache._internal.configurator." "ApacheConfigurator._verify_no_certbot_redirect") with mock.patch(verify_no_redirect) as mock_verify: self.config.enhance("green.blue.purple.com", "redirect") @@ -1269,7 +1257,7 @@ class MultipleVhostsTest(util.ApacheTest): # pylint: disable=protected-access self.config._enable_redirect(self.vh_truth[1], "") - self.assertEqual(len(self.config.vhosts), 11) + self.assertEqual(len(self.config.vhosts), 13) def test_create_own_redirect_for_old_apache_version(self): self.config.parser.modules.add("rewrite_module") @@ -1280,7 +1268,7 @@ class MultipleVhostsTest(util.ApacheTest): # pylint: disable=protected-access self.config._enable_redirect(self.vh_truth[1], "") - self.assertEqual(len(self.config.vhosts), 11) + self.assertEqual(len(self.config.vhosts), 13) def test_sift_rewrite_rule(self): # pylint: disable=protected-access @@ -1301,13 +1289,13 @@ class MultipleVhostsTest(util.ApacheTest): account_key = self.rsa512jwk achall1 = achallenges.KeyAuthorizationAnnotatedChallenge( challb=acme_util.chall_to_challb( - challenges.TLSSNI01( + challenges.HTTP01( token=b"jIq_Xy1mXGN37tb4L6Xj_es58fW571ZNyXekdZzhh7Q"), "pending"), domain="encryption-example.demo", account_key=account_key) achall2 = achallenges.KeyAuthorizationAnnotatedChallenge( challb=acme_util.chall_to_challb( - challenges.TLSSNI01( + challenges.HTTP01( token=b"uqnaPzxtrndteOqtrXb0Asl5gOJfWAnnx6QJyvcmlDU"), "pending"), domain="certbot.demo", account_key=account_key) @@ -1318,24 +1306,6 @@ class MultipleVhostsTest(util.ApacheTest): return account_key, (achall1, achall2, achall3) - def test_make_addrs_sni_ready(self): - self.config.version = (2, 2) - self.config.make_addrs_sni_ready( - set([obj.Addr.fromstring("*:443"), obj.Addr.fromstring("*:80")])) - self.assertTrue(self.config.parser.find_dir( - "NameVirtualHost", "*:80", exclude=False)) - self.assertTrue(self.config.parser.find_dir( - "NameVirtualHost", "*:443", exclude=False)) - - def test_aug_version(self): - mock_match = mock.Mock(return_value=["something"]) - self.config.aug.match = mock_match - # pylint: disable=protected-access - self.assertEqual(self.config._check_aug_version(), - ["something"]) - self.config.aug.match.side_effect = RuntimeError - self.assertFalse(self.config._check_aug_version()) - def test_enable_site_nondebian(self): inc_path = "/path/to/wherever" vhost = self.vh_truth[0] @@ -1358,10 +1328,10 @@ class MultipleVhostsTest(util.ApacheTest): self.config.parser.modules.add("ssl_module") self.config.parser.modules.add("mod_ssl.c") self.config.parser.modules.add("socache_shmcb_module") - tmp_path = os.path.realpath(tempfile.mkdtemp("vhostroot")) - os.chmod(tmp_path, 0o755) - mock_p = "certbot_apache.configurator.ApacheConfigurator._get_ssl_vhost_path" - mock_a = "certbot_apache.parser.ApacheParser.add_include" + tmp_path = filesystem.realpath(tempfile.mkdtemp("vhostroot")) + filesystem.chmod(tmp_path, 0o755) + mock_p = "certbot_apache._internal.configurator.ApacheConfigurator._get_ssl_vhost_path" + mock_a = "certbot_apache._internal.parser.ApacheParser.add_include" with mock.patch(mock_p) as mock_path: mock_path.return_value = os.path.join(tmp_path, "whatever.conf") @@ -1374,7 +1344,7 @@ class MultipleVhostsTest(util.ApacheTest): self.assertTrue(mock_add.called) shutil.rmtree(tmp_path) - @mock.patch("certbot_apache.parser.ApacheParser.parsed_in_original") + @mock.patch("certbot_apache._internal.parser.ApacheParser.parsed_in_original") def test_choose_vhost_and_servername_addition_parsed(self, mock_parsed): ret_vh = self.vh_truth[8] ret_vh.enabled = True @@ -1391,12 +1361,12 @@ class MultipleVhostsTest(util.ApacheTest): # pylint: disable=protected-access cases = {u"*.example.org": True, b"*.x.example.org": True, u"a.example.org": False, b"a.x.example.org": False} - for key in cases.keys(): + for key in cases: self.assertEqual(self.config._wildcard_domain(key), cases[key]) def test_choose_vhosts_wildcard(self): # pylint: disable=protected-access - mock_path = "certbot_apache.display_ops.select_vhost_multiple" + mock_path = "certbot_apache._internal.display_ops.select_vhost_multiple" with mock.patch(mock_path) as mock_select_vhs: mock_select_vhs.return_value = [self.vh_truth[3]] vhs = self.config._choose_vhosts_wildcard("*.certbot.demo", @@ -1412,10 +1382,10 @@ class MultipleVhostsTest(util.ApacheTest): self.assertFalse(vhs[0] == self.vh_truth[3]) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.make_vhost_ssl") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.make_vhost_ssl") def test_choose_vhosts_wildcard_no_ssl(self, mock_makessl): # pylint: disable=protected-access - mock_path = "certbot_apache.display_ops.select_vhost_multiple" + mock_path = "certbot_apache._internal.display_ops.select_vhost_multiple" with mock.patch(mock_path) as mock_select_vhs: mock_select_vhs.return_value = [self.vh_truth[1]] vhs = self.config._choose_vhosts_wildcard("*.certbot.demo", @@ -1423,13 +1393,13 @@ class MultipleVhostsTest(util.ApacheTest): self.assertFalse(mock_makessl.called) self.assertEqual(vhs[0], self.vh_truth[1]) - @mock.patch("certbot_apache.configurator.ApacheConfigurator._vhosts_for_wildcard") - @mock.patch("certbot_apache.configurator.ApacheConfigurator.make_vhost_ssl") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator._vhosts_for_wildcard") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.make_vhost_ssl") def test_choose_vhosts_wildcard_already_ssl(self, mock_makessl, mock_vh_for_w): # pylint: disable=protected-access # Already SSL vhost mock_vh_for_w.return_value = [self.vh_truth[7]] - mock_path = "certbot_apache.display_ops.select_vhost_multiple" + mock_path = "certbot_apache._internal.display_ops.select_vhost_multiple" with mock.patch(mock_path) as mock_select_vhs: mock_select_vhs.return_value = [self.vh_truth[7]] vhs = self.config._choose_vhosts_wildcard("whatever", @@ -1450,7 +1420,7 @@ class MultipleVhostsTest(util.ApacheTest): mock_choose_vhosts = mock.MagicMock() mock_choose_vhosts.return_value = [self.vh_truth[7]] self.config._choose_vhosts_wildcard = mock_choose_vhosts - mock_d = "certbot_apache.configurator.ApacheConfigurator._deploy_cert" + mock_d = "certbot_apache._internal.configurator.ApacheConfigurator._deploy_cert" with mock.patch(mock_d) as mock_dep: self.config.deploy_cert("*.wildcard.example.org", "/tmp/path", "/tmp/path", "/tmp/path", "/tmp/path") @@ -1458,7 +1428,7 @@ class MultipleVhostsTest(util.ApacheTest): self.assertEqual(len(mock_dep.call_args_list), 1) self.assertEqual(self.vh_truth[7], mock_dep.call_args_list[0][0][0]) - @mock.patch("certbot_apache.display_ops.select_vhost_multiple") + @mock.patch("certbot_apache._internal.display_ops.select_vhost_multiple") def test_deploy_cert_wildcard_no_vhosts(self, mock_dialog): # pylint: disable=protected-access mock_dialog.return_value = [] @@ -1467,7 +1437,7 @@ class MultipleVhostsTest(util.ApacheTest): "*.wild.cat", "/tmp/path", "/tmp/path", "/tmp/path", "/tmp/path") - @mock.patch("certbot_apache.configurator.ApacheConfigurator._choose_vhosts_wildcard") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator._choose_vhosts_wildcard") def test_enhance_wildcard_after_install(self, mock_choose): # pylint: disable=protected-access self.config.parser.modules.add("mod_ssl.c") @@ -1478,7 +1448,7 @@ class MultipleVhostsTest(util.ApacheTest): "Upgrade-Insecure-Requests") self.assertFalse(mock_choose.called) - @mock.patch("certbot_apache.configurator.ApacheConfigurator._choose_vhosts_wildcard") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator._choose_vhosts_wildcard") def test_enhance_wildcard_no_install(self, mock_choose): self.vh_truth[3].ssl = True mock_choose.return_value = [self.vh_truth[3]] @@ -1503,6 +1473,29 @@ class MultipleVhostsTest(util.ApacheTest): second_id = self.config.add_vhost_id(self.vh_truth[0]) self.assertEqual(first_id, second_id) + def test_realpath_replaces_symlink(self): + orig_match = self.config.parser.aug.match + mock_vhost = copy.deepcopy(self.vh_truth[0]) + mock_vhost.filep = mock_vhost.filep.replace('sites-enabled', u'sites-available') + mock_vhost.path = mock_vhost.path.replace('sites-enabled', 'sites-available') + mock_vhost.enabled = False + self.config.parser.parse_file(mock_vhost.filep) + + def mock_match(aug_expr): + """Return a mocked match list of VirtualHosts""" + if "/mocked/path" in aug_expr: + return [self.vh_truth[1].path, self.vh_truth[0].path, mock_vhost.path] + return orig_match(aug_expr) + + self.config.parser.parser_paths = ["/mocked/path"] + self.config.parser.aug.match = mock_match + vhs = self.config.get_virtual_hosts() + self.assertEqual(len(vhs), 2) + self.assertTrue(vhs[0] == self.vh_truth[1]) + # mock_vhost should have replaced the vh_truth[0], because its filepath + # isn't a symlink + self.assertTrue(vhs[1] == mock_vhost) + class AugeasVhostsTest(util.ApacheTest): """Test vhosts with illegal names dependent on augeas version.""" @@ -1521,8 +1514,8 @@ class AugeasVhostsTest(util.ApacheTest): self.work_dir) def test_choosevhost_with_illegal_name(self): - self.config.aug = mock.MagicMock() - self.config.aug.match.side_effect = RuntimeError + self.config.parser.aug = mock.MagicMock() + self.config.parser.aug.match.side_effect = RuntimeError path = "debian_apache_2_4/augeas_vhosts/apache2/sites-available/old-and-default.conf" chosen_vhost = self.config._create_vhost(path) self.assertEqual(None, chosen_vhost) @@ -1530,9 +1523,9 @@ class AugeasVhostsTest(util.ApacheTest): def test_choosevhost_works(self): path = "debian_apache_2_4/augeas_vhosts/apache2/sites-available/old-and-default.conf" chosen_vhost = self.config._create_vhost(path) - self.assertTrue(chosen_vhost == None or chosen_vhost.path == path) + self.assertTrue(chosen_vhost is None or chosen_vhost.path == path) - @mock.patch("certbot_apache.configurator.ApacheConfigurator._create_vhost") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator._create_vhost") def test_get_vhost_continue(self, mock_vhost): mock_vhost.return_value = None vhs = self.config.get_virtual_hosts() @@ -1544,18 +1537,18 @@ class AugeasVhostsTest(util.ApacheTest): for name in names: self.assertFalse(name in self.config.choose_vhost(name).aliases) - @mock.patch("certbot_apache.obj.VirtualHost.conflicts") + @mock.patch("certbot_apache._internal.obj.VirtualHost.conflicts") def test_choose_vhost_without_matching_wildcard(self, mock_conflicts): mock_conflicts.return_value = False - mock_path = "certbot_apache.display_ops.select_vhost" + mock_path = "certbot_apache._internal.display_ops.select_vhost" with mock.patch(mock_path, lambda _, vhosts: vhosts[0]): for name in ("a.example.net", "other.example.net"): self.assertTrue(name in self.config.choose_vhost(name).aliases) - @mock.patch("certbot_apache.obj.VirtualHost.conflicts") + @mock.patch("certbot_apache._internal.obj.VirtualHost.conflicts") def test_choose_vhost_wildcard_not_found(self, mock_conflicts): mock_conflicts.return_value = False - mock_path = "certbot_apache.display_ops.select_vhost" + mock_path = "certbot_apache._internal.display_ops.select_vhost" names = ( "abc.example.net", "not.there.tld", "aa.wildcard.tld" ) @@ -1567,7 +1560,7 @@ class AugeasVhostsTest(util.ApacheTest): self.assertEqual(mock_select.call_count - orig_cc, 1) def test_choose_vhost_wildcard_found(self): - mock_path = "certbot_apache.display_ops.select_vhost" + mock_path = "certbot_apache._internal.display_ops.select_vhost" names = ( "ab.example.net", "a.wildcard.tld", "yetanother.example.net" ) @@ -1621,7 +1614,7 @@ class MultiVhostsTest(util.ApacheTest): self.assertEqual(self.config.is_name_vhost(self.vh_truth[1]), self.config.is_name_vhost(ssl_vhost)) - mock_path = "certbot_apache.configurator.ApacheConfigurator._get_new_vh_path" + mock_path = "certbot_apache._internal.configurator.ApacheConfigurator._get_new_vh_path" with mock.patch(mock_path) as mock_getpath: mock_getpath.return_value = None self.assertRaises(errors.PluginError, self.config.make_vhost_ssl, @@ -1727,7 +1720,7 @@ class InstallSslOptionsConfTest(util.ApacheTest): self._assert_current_file() def test_prev_file_updates_to_current(self): - from certbot_apache.constants import ALL_SSL_OPTIONS_HASHES + from certbot_apache._internal.constants import ALL_SSL_OPTIONS_HASHES ALL_SSL_OPTIONS_HASHES.insert(0, "test_hash_does_not_match") with mock.patch('certbot.crypto_util.sha256sum') as mock_sha256: mock_sha256.return_value = ALL_SSL_OPTIONS_HASHES[0] @@ -1766,7 +1759,7 @@ class InstallSslOptionsConfTest(util.ApacheTest): self.assertFalse(mock_logger.warning.called) def test_current_file_hash_in_all_hashes(self): - from certbot_apache.constants import ALL_SSL_OPTIONS_HASHES + from certbot_apache._internal.constants import ALL_SSL_OPTIONS_HASHES self.assertTrue(self._current_ssl_options_hash() in ALL_SSL_OPTIONS_HASHES, "Constants.ALL_SSL_OPTIONS_HASHES must be appended" " with the sha256 hash of self.config.mod_ssl_conf when it is updated.") diff --git a/certbot-apache/certbot_apache/tests/debian_test.py b/certbot-apache/tests/debian_test.py similarity index 92% rename from certbot-apache/certbot_apache/tests/debian_test.py rename to certbot-apache/tests/debian_test.py index bb1d64278..6e63a9bd3 100644 --- a/certbot-apache/certbot_apache/tests/debian_test.py +++ b/certbot-apache/tests/debian_test.py @@ -1,15 +1,14 @@ -"""Test for certbot_apache.configurator for Debian overrides""" -import os +"""Test for certbot_apache._internal.configurator for Debian overrides""" import shutil import unittest import mock from certbot import errors - -from certbot_apache import apache_util -from certbot_apache import obj -from certbot_apache.tests import util +from certbot.compat import os +from certbot_apache._internal import apache_util +from certbot_apache._internal import obj +import util class MultipleVhostsTestDebian(util.ApacheTest): @@ -32,8 +31,8 @@ class MultipleVhostsTestDebian(util.ApacheTest): def mocked_deploy_cert(*args, **kwargs): """a helper to mock a deployed cert""" - g_mod = "certbot_apache.configurator.ApacheConfigurator.enable_mod" - d_mod = "certbot_apache.override_debian.DebianConfigurator.enable_mod" + g_mod = "certbot_apache._internal.configurator.ApacheConfigurator.enable_mod" + d_mod = "certbot_apache._internal.override_debian.DebianConfigurator.enable_mod" with mock.patch(g_mod): with mock.patch(d_mod): config.real_deploy_cert(*args, **kwargs) @@ -47,7 +46,7 @@ class MultipleVhostsTestDebian(util.ApacheTest): @mock.patch("certbot.util.run_script") @mock.patch("certbot.util.exe_exists") - @mock.patch("certbot_apache.parser.subprocess.Popen") + @mock.patch("certbot_apache._internal.parser.subprocess.Popen") def test_enable_mod(self, mock_popen, mock_exe_exists, mock_run_script): mock_popen().communicate.return_value = ("Define: DUMP_RUN_CFG", "") mock_popen().returncode = 0 @@ -79,9 +78,9 @@ class MultipleVhostsTestDebian(util.ApacheTest): def test_enable_site_failure(self): self.config.parser.root = "/tmp/nonexistent" - with mock.patch("os.path.isdir") as mock_dir: + with mock.patch("certbot.compat.os.path.isdir") as mock_dir: mock_dir.return_value = True - with mock.patch("os.path.islink") as mock_link: + with mock.patch("certbot.compat.os.path.islink") as mock_link: mock_link.return_value = False self.assertRaises( errors.NotSupportedError, @@ -196,7 +195,7 @@ class MultipleVhostsTestDebian(util.ApacheTest): def test_enable_site_call_parent(self): with mock.patch( - "certbot_apache.configurator.ApacheConfigurator.enable_site") as e_s: + "certbot_apache._internal.configurator.ApacheConfigurator.enable_site") as e_s: self.config.parser.root = "/tmp/nonexistent" vh = self.vh_truth[0] vh.enabled = False diff --git a/certbot-apache/certbot_apache/tests/display_ops_test.py b/certbot-apache/tests/display_ops_test.py similarity index 86% rename from certbot-apache/certbot_apache/tests/display_ops_test.py rename to certbot-apache/tests/display_ops_test.py index df5cdbac0..50bdc03cf 100644 --- a/certbot-apache/certbot_apache/tests/display_ops_test.py +++ b/certbot-apache/tests/display_ops_test.py @@ -1,22 +1,18 @@ -"""Test certbot_apache.display_ops.""" +"""Test certbot_apache._internal.display_ops.""" import unittest import mock from certbot import errors - from certbot.display import util as display_util - from certbot.tests import util as certbot_util - -from certbot_apache import obj - -from certbot_apache.display_ops import select_vhost_multiple -from certbot_apache.tests import util +from certbot_apache._internal import obj +from certbot_apache._internal.display_ops import select_vhost_multiple +import util class SelectVhostMultiTest(unittest.TestCase): - """Tests for certbot_apache.display_ops.select_vhost_multiple.""" + """Tests for certbot_apache._internal.display_ops.select_vhost_multiple.""" def setUp(self): self.base_dir = "/example_path" @@ -45,7 +41,7 @@ class SelectVhostMultiTest(unittest.TestCase): self.assertFalse(vhs) class SelectVhostTest(unittest.TestCase): - """Tests for certbot_apache.display_ops.select_vhost.""" + """Tests for certbot_apache._internal.display_ops.select_vhost.""" def setUp(self): self.base_dir = "/example_path" @@ -54,7 +50,7 @@ class SelectVhostTest(unittest.TestCase): @classmethod def _call(cls, vhosts): - from certbot_apache.display_ops import select_vhost + from certbot_apache._internal.display_ops import select_vhost return select_vhost("example.com", vhosts) @certbot_util.patch_get_utility() @@ -81,9 +77,9 @@ class SelectVhostTest(unittest.TestCase): def test_no_vhosts(self): self.assertEqual(self._call([]), None) - @mock.patch("certbot_apache.display_ops.display_util") + @mock.patch("certbot_apache._internal.display_ops.display_util") @certbot_util.patch_get_utility() - @mock.patch("certbot_apache.display_ops.logger") + @mock.patch("certbot_apache._internal.display_ops.logger") def test_small_display(self, mock_logger, mock_util, mock_display_util): mock_display_util.WIDTH = 20 mock_util().menu.return_value = (display_util.OK, 0) diff --git a/certbot-apache/certbot_apache/tests/entrypoint_test.py b/certbot-apache/tests/entrypoint_test.py similarity index 69% rename from certbot-apache/certbot_apache/tests/entrypoint_test.py rename to certbot-apache/tests/entrypoint_test.py index c04611465..04c393bdf 100644 --- a/certbot-apache/certbot_apache/tests/entrypoint_test.py +++ b/certbot-apache/tests/entrypoint_test.py @@ -1,10 +1,11 @@ -"""Test for certbot_apache.entrypoint for override class resolution""" +"""Test for certbot_apache._internal.entrypoint for override class resolution""" import unittest import mock -from certbot_apache import configurator -from certbot_apache import entrypoint +from certbot_apache._internal import configurator +from certbot_apache._internal import entrypoint + class EntryPointTest(unittest.TestCase): """Entrypoint tests""" @@ -14,8 +15,13 @@ class EntryPointTest(unittest.TestCase): def test_get_configurator(self): with mock.patch("certbot.util.get_os_info") as mock_info: - for distro in entrypoint.OVERRIDE_CLASSES.keys(): - mock_info.return_value = (distro, "whatever") + for distro in entrypoint.OVERRIDE_CLASSES: + return_value = (distro, "whatever") + if distro == 'fedora_old': + return_value = ('fedora', '28') + elif distro == 'fedora': + return_value = ('fedora', '29') + mock_info.return_value = return_value self.assertEqual(entrypoint.get_configurator(), entrypoint.OVERRIDE_CLASSES[distro]) @@ -23,7 +29,7 @@ class EntryPointTest(unittest.TestCase): with mock.patch("certbot.util.get_os_info") as mock_info: mock_info.return_value = ("nonexistent", "irrelevant") with mock.patch("certbot.util.get_systemd_os_like") as mock_like: - for like in entrypoint.OVERRIDE_CLASSES.keys(): + for like in entrypoint.OVERRIDE_CLASSES: mock_like.return_value = [like] self.assertEqual(entrypoint.get_configurator(), entrypoint.OVERRIDE_CLASSES[like]) diff --git a/certbot-apache/tests/fedora_test.py b/certbot-apache/tests/fedora_test.py new file mode 100644 index 000000000..2bfd6babb --- /dev/null +++ b/certbot-apache/tests/fedora_test.py @@ -0,0 +1,194 @@ +"""Test for certbot_apache._internal.configurator for Fedora 29+ overrides""" +import unittest + +import mock + +from certbot import errors +from certbot.compat import filesystem +from certbot.compat import os +from certbot_apache._internal import obj +from certbot_apache._internal import override_fedora +import util + + +def get_vh_truth(temp_dir, config_name): + """Return the ground truth for the specified directory.""" + prefix = os.path.join( + temp_dir, config_name, "httpd/conf.d") + + aug_pre = "/files" + prefix + # TODO: eventually, these tests should have a dedicated configuration instead + # of reusing the ones from centos_test + vh_truth = [ + obj.VirtualHost( + os.path.join(prefix, "centos.example.com.conf"), + os.path.join(aug_pre, "centos.example.com.conf/VirtualHost"), + {obj.Addr.fromstring("*:80")}, + False, True, "centos.example.com"), + obj.VirtualHost( + os.path.join(prefix, "ssl.conf"), + os.path.join(aug_pre, "ssl.conf/VirtualHost"), + {obj.Addr.fromstring("_default_:443")}, + True, True, None) + ] + return vh_truth + + +class FedoraRestartTest(util.ApacheTest): + """Tests for Fedora specific self-signed certificate override""" + + # TODO: eventually, these tests should have a dedicated configuration instead + # of reusing the ones from centos_test + def setUp(self): # pylint: disable=arguments-differ + test_dir = "centos7_apache/apache" + config_root = "centos7_apache/apache/httpd" + vhost_root = "centos7_apache/apache/httpd/conf.d" + super(FedoraRestartTest, self).setUp(test_dir=test_dir, + config_root=config_root, + vhost_root=vhost_root) + self.config = util.get_apache_configurator( + self.config_path, self.vhost_path, self.config_dir, self.work_dir, + os_info="fedora") + self.vh_truth = get_vh_truth( + self.temp_dir, "centos7_apache/apache") + + def _run_fedora_test(self): + self.assertIsInstance(self.config, override_fedora.FedoraConfigurator) + self.config.config_test() + + def test_fedora_restart_error(self): + c_test = "certbot_apache._internal.configurator.ApacheConfigurator.config_test" + with mock.patch(c_test) as mock_test: + # First call raises error, second doesn't + mock_test.side_effect = [errors.MisconfigurationError, ''] + with mock.patch("certbot.util.run_script") as mock_run: + mock_run.side_effect = errors.SubprocessError + self.assertRaises(errors.MisconfigurationError, + self._run_fedora_test) + + def test_fedora_restart(self): + c_test = "certbot_apache._internal.configurator.ApacheConfigurator.config_test" + with mock.patch(c_test) as mock_test: + with mock.patch("certbot.util.run_script") as mock_run: + # First call raises error, second doesn't + mock_test.side_effect = [errors.MisconfigurationError, ''] + self._run_fedora_test() + self.assertEqual(mock_test.call_count, 2) + self.assertEqual(mock_run.call_args[0][0], + ['systemctl', 'restart', 'httpd']) + + +class MultipleVhostsTestFedora(util.ApacheTest): + """Multiple vhost tests for CentOS / RHEL family of distros""" + + _multiprocess_can_split_ = True + + def setUp(self): # pylint: disable=arguments-differ + test_dir = "centos7_apache/apache" + config_root = "centos7_apache/apache/httpd" + vhost_root = "centos7_apache/apache/httpd/conf.d" + super(MultipleVhostsTestFedora, self).setUp(test_dir=test_dir, + config_root=config_root, + vhost_root=vhost_root) + + self.config = util.get_apache_configurator( + self.config_path, self.vhost_path, self.config_dir, self.work_dir, + os_info="fedora") + self.vh_truth = get_vh_truth( + self.temp_dir, "centos7_apache/apache") + + def test_get_parser(self): + self.assertIsInstance(self.config.parser, override_fedora.FedoraParser) + + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") + def test_opportunistic_httpd_runtime_parsing(self, mock_get): + define_val = ( + 'Define: TEST1\n' + 'Define: TEST2\n' + 'Define: DUMP_RUN_CFG\n' + ) + mod_val = ( + 'Loaded Modules:\n' + ' mock_module (static)\n' + ' another_module (static)\n' + ) + def mock_get_cfg(command): + """Mock httpd process stdout""" + if command == ['httpd', '-t', '-D', 'DUMP_RUN_CFG']: + return define_val + elif command == ['httpd', '-t', '-D', 'DUMP_MODULES']: + return mod_val + return "" + mock_get.side_effect = mock_get_cfg + self.config.parser.modules = set() + self.config.parser.variables = {} + + with mock.patch("certbot.util.get_os_info") as mock_osi: + # Make sure we have the have the CentOS httpd constants + mock_osi.return_value = ("fedora", "29") + self.config.parser.update_runtime_variables() + + self.assertEqual(mock_get.call_count, 3) + self.assertEqual(len(self.config.parser.modules), 4) + self.assertEqual(len(self.config.parser.variables), 2) + self.assertTrue("TEST2" in self.config.parser.variables.keys()) + self.assertTrue("mod_another.c" in self.config.parser.modules) + + @mock.patch("certbot_apache._internal.configurator.util.run_script") + def test_get_version(self, mock_run_script): + mock_run_script.return_value = ('', None) + self.assertRaises(errors.PluginError, self.config.get_version) + self.assertEqual(mock_run_script.call_args[0][0][0], 'httpd') + + def test_get_virtual_hosts(self): + """Make sure all vhosts are being properly found.""" + vhs = self.config.get_virtual_hosts() + self.assertEqual(len(vhs), 2) + found = 0 + + for vhost in vhs: + for centos_truth in self.vh_truth: + if vhost == centos_truth: + found += 1 + break + else: + raise Exception("Missed: %s" % vhost) # pragma: no cover + self.assertEqual(found, 2) + + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") + def test_get_sysconfig_vars(self, mock_cfg): + """Make sure we read the sysconfig OPTIONS variable correctly""" + # Return nothing for the process calls + mock_cfg.return_value = "" + self.config.parser.sysconfig_filep = filesystem.realpath( + os.path.join(self.config.parser.root, "../sysconfig/httpd")) + self.config.parser.variables = {} + + with mock.patch("certbot.util.get_os_info") as mock_osi: + # Make sure we have the have the CentOS httpd constants + mock_osi.return_value = ("fedora", "29") + self.config.parser.update_runtime_variables() + + self.assertTrue("mock_define" in self.config.parser.variables.keys()) + self.assertTrue("mock_define_too" in self.config.parser.variables.keys()) + self.assertTrue("mock_value" in self.config.parser.variables.keys()) + self.assertEqual("TRUE", self.config.parser.variables["mock_value"]) + self.assertTrue("MOCK_NOSEP" in self.config.parser.variables.keys()) + self.assertEqual("NOSEP_VAL", self.config.parser.variables["NOSEP_TWO"]) + + @mock.patch("certbot_apache._internal.configurator.util.run_script") + def test_alt_restart_works(self, mock_run_script): + mock_run_script.side_effect = [None, errors.SubprocessError, None] + self.config.restart() + self.assertEqual(mock_run_script.call_count, 3) + + @mock.patch("certbot_apache._internal.configurator.util.run_script") + def test_alt_restart_errors(self, mock_run_script): + mock_run_script.side_effect = [None, + errors.SubprocessError, + errors.SubprocessError] + self.assertRaises(errors.MisconfigurationError, self.config.restart) + + +if __name__ == "__main__": + unittest.main() # pragma: no cover diff --git a/certbot-apache/certbot_apache/tests/gentoo_test.py b/certbot-apache/tests/gentoo_test.py similarity index 83% rename from certbot-apache/certbot_apache/tests/gentoo_test.py rename to certbot-apache/tests/gentoo_test.py index f09d742a4..90a163fd3 100644 --- a/certbot-apache/certbot_apache/tests/gentoo_test.py +++ b/certbot-apache/tests/gentoo_test.py @@ -1,14 +1,15 @@ -"""Test for certbot_apache.configurator for Gentoo overrides""" -import os +"""Test for certbot_apache._internal.configurator for Gentoo overrides""" import unittest import mock from certbot import errors +from certbot.compat import filesystem +from certbot.compat import os +from certbot_apache._internal import obj +from certbot_apache._internal import override_gentoo +import util -from certbot_apache import override_gentoo -from certbot_apache import obj -from certbot_apache.tests import util def get_vh_truth(temp_dir, config_name): """Return the ground truth for the specified directory.""" @@ -50,7 +51,8 @@ class MultipleVhostsTestGentoo(util.ApacheTest): config_root=config_root, vhost_root=vhost_root) - with mock.patch("certbot_apache.override_gentoo.GentooParser.update_runtime_variables"): + # pylint: disable=line-too-long + with mock.patch("certbot_apache._internal.override_gentoo.GentooParser.update_runtime_variables"): self.config = util.get_apache_configurator( self.config_path, self.vhost_path, self.config_dir, self.work_dir, os_info="gentoo") @@ -80,20 +82,20 @@ class MultipleVhostsTestGentoo(util.ApacheTest): """Make sure we read the Gentoo APACHE2_OPTS variable correctly""" defines = ['DEFAULT_VHOST', 'INFO', 'SSL', 'SSL_DEFAULT_VHOST', 'LANGUAGE'] - self.config.parser.apacheconfig_filep = os.path.realpath( + self.config.parser.apacheconfig_filep = filesystem.realpath( os.path.join(self.config.parser.root, "../conf.d/apache2")) self.config.parser.variables = {} - with mock.patch("certbot_apache.override_gentoo.GentooParser.update_modules"): + with mock.patch("certbot_apache._internal.override_gentoo.GentooParser.update_modules"): self.config.parser.update_runtime_variables() for define in defines: self.assertTrue(define in self.config.parser.variables.keys()) - @mock.patch("certbot_apache.parser.ApacheParser.parse_from_subprocess") + @mock.patch("certbot_apache._internal.parser.ApacheParser.parse_from_subprocess") def test_no_binary_configdump(self, mock_subprocess): """Make sure we don't call binary dumps other than modules from Apache as this is not supported in Gentoo currently""" - with mock.patch("certbot_apache.override_gentoo.GentooParser.update_modules"): + with mock.patch("certbot_apache._internal.override_gentoo.GentooParser.update_modules"): self.config.parser.update_runtime_variables() self.config.parser.reset_modules() self.assertFalse(mock_subprocess.called) @@ -102,7 +104,7 @@ class MultipleVhostsTestGentoo(util.ApacheTest): self.config.parser.reset_modules() self.assertTrue(mock_subprocess.called) - @mock.patch("certbot_apache.parser.ApacheParser._get_runtime_cfg") + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") def test_opportunistic_httpd_runtime_parsing(self, mock_get): mod_val = ( 'Loaded Modules:\n' @@ -113,6 +115,7 @@ class MultipleVhostsTestGentoo(util.ApacheTest): """Mock httpd process stdout""" if command == ['apache2ctl', 'modules']: return mod_val + return None # pragma: no cover mock_get.side_effect = mock_get_cfg self.config.parser.modules = set() @@ -125,7 +128,7 @@ class MultipleVhostsTestGentoo(util.ApacheTest): self.assertEqual(len(self.config.parser.modules), 4) self.assertTrue("mod_another.c" in self.config.parser.modules) - @mock.patch("certbot_apache.configurator.util.run_script") + @mock.patch("certbot_apache._internal.configurator.util.run_script") def test_alt_restart_works(self, mock_run_script): mock_run_script.side_effect = [None, errors.SubprocessError, None] self.config.restart() diff --git a/certbot-apache/certbot_apache/tests/http_01_test.py b/certbot-apache/tests/http_01_test.py similarity index 79% rename from certbot-apache/certbot_apache/tests/http_01_test.py rename to certbot-apache/tests/http_01_test.py index 9c729b08c..643a6bdd5 100644 --- a/certbot-apache/certbot_apache/tests/http_01_test.py +++ b/certbot-apache/tests/http_01_test.py @@ -1,34 +1,33 @@ -"""Test for certbot_apache.http_01.""" -import mock -import os +"""Test for certbot_apache._internal.http_01.""" import unittest +import mock + from acme import challenges from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module - from certbot import achallenges from certbot import errors - +from certbot.compat import filesystem +from certbot.compat import os from certbot.tests import acme_util -from certbot_apache.parser import get_aug_path -from certbot_apache.tests import util - +from certbot_apache._internal.parser import get_aug_path +import util NUM_ACHALLS = 3 class ApacheHttp01Test(util.ApacheTest): - """Test for certbot_apache.http_01.ApacheHttp01.""" + """Test for certbot_apache._internal.http_01.ApacheHttp01.""" - def setUp(self, *args, **kwargs): + def setUp(self, *args, **kwargs): # pylint: disable=arguments-differ super(ApacheHttp01Test, self).setUp(*args, **kwargs) self.account_key = self.rsa512jwk self.achalls = [] # type: List[achallenges.KeyAuthorizationAnnotatedChallenge] vh_truth = util.get_vh_truth( self.temp_dir, "debian_apache_2_4/multiple_vhosts") - # Takes the vhosts for encryption-example.demo, certbot.demo, and - # vhost.in.rootconf + # Takes the vhosts for encryption-example.demo, certbot.demo + # and vhost.in.rootconf self.vhosts = [vh_truth[0], vh_truth[3], vh_truth[10]] for i in range(NUM_ACHALLS): @@ -39,18 +38,18 @@ class ApacheHttp01Test(util.ApacheTest): "pending"), domain=self.vhosts[i].name, account_key=self.account_key)) - modules = ["rewrite", "authz_core", "authz_host"] + modules = ["ssl", "rewrite", "authz_core", "authz_host"] for mod in modules: self.config.parser.modules.add("mod_{0}.c".format(mod)) self.config.parser.modules.add(mod + "_module") - from certbot_apache.http_01 import ApacheHttp01 + from certbot_apache._internal.http_01 import ApacheHttp01 self.http = ApacheHttp01(self.config) def test_empty_perform(self): self.assertFalse(self.http.perform()) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.enable_mod") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.enable_mod") def test_enable_modules_apache_2_2(self, mock_enmod): self.config.version = (2, 2) self.config.parser.modules.remove("authz_host_module") @@ -59,7 +58,7 @@ class ApacheHttp01Test(util.ApacheTest): enmod_calls = self.common_enable_modules_test(mock_enmod) self.assertEqual(enmod_calls[0][0][0], "authz_host") - @mock.patch("certbot_apache.configurator.ApacheConfigurator.enable_mod") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.enable_mod") def test_enable_modules_apache_2_4(self, mock_enmod): self.config.parser.modules.remove("authz_core_module") self.config.parser.modules.remove("mod_authz_core.c") @@ -78,7 +77,7 @@ class ApacheHttp01Test(util.ApacheTest): calls = mock_enmod.call_args_list other_calls = [] for call in calls: - if "rewrite" != call[0][0]: + if call[0][0] != "rewrite": other_calls.append(call) # If these lists are equal, we never enabled mod_rewrite @@ -111,6 +110,17 @@ class ApacheHttp01Test(util.ApacheTest): domain="something.nonexistent", account_key=self.account_key)] self.common_perform_test(achalls, vhosts) + def test_configure_multiple_vhosts(self): + vhosts = [v for v in self.config.vhosts if "duplicate.example.com" in v.get_names()] + self.assertEqual(len(vhosts), 2) + achalls = [ + achallenges.KeyAuthorizationAnnotatedChallenge( + challb=acme_util.chall_to_challb( + challenges.HTTP01(token=((b'a' * 16))), + "pending"), + domain="duplicate.example.com", account_key=self.account_key)] + self.common_perform_test(achalls, vhosts) + def test_no_vhost(self): for achall in self.achalls: self.http.add_chall(achall) @@ -169,22 +179,21 @@ class ApacheHttp01Test(util.ApacheTest): self.assertEqual(self.http.perform(), expected_response) self.assertTrue(os.path.isdir(self.http.challenge_dir)) - self._has_min_permissions(self.http.challenge_dir, 0o755) + self.assertTrue(filesystem.has_min_permissions(self.http.challenge_dir, 0o755)) self._test_challenge_conf() for achall in achalls: self._test_challenge_file(achall) for vhost in vhosts: - if not vhost.ssl: - matches = self.config.parser.find_dir("Include", - self.http.challenge_conf_pre, - vhost.path) - self.assertEqual(len(matches), 1) - matches = self.config.parser.find_dir("Include", - self.http.challenge_conf_post, - vhost.path) - self.assertEqual(len(matches), 1) + matches = self.config.parser.find_dir("Include", + self.http.challenge_conf_pre, + vhost.path) + self.assertEqual(len(matches), 1) + matches = self.config.parser.find_dir("Include", + self.http.challenge_conf_post, + vhost.path) + self.assertEqual(len(matches), 1) self.assertTrue(os.path.exists(challenge_dir)) @@ -208,15 +217,10 @@ class ApacheHttp01Test(util.ApacheTest): name = os.path.join(self.http.challenge_dir, achall.chall.encode("token")) validation = achall.validation(self.account_key) - self._has_min_permissions(name, 0o644) + self.assertTrue(filesystem.has_min_permissions(name, 0o644)) with open(name, 'rb') as f: self.assertEqual(f.read(), validation.encode()) - def _has_min_permissions(self, path, min_mode): - """Tests the given file has at least the permissions in mode.""" - st_mode = os.stat(path).st_mode - self.assertEqual(st_mode, st_mode | min_mode) - if __name__ == "__main__": unittest.main() # pragma: no cover diff --git a/certbot-apache/certbot_apache/tests/obj_test.py b/certbot-apache/tests/obj_test.py similarity index 89% rename from certbot-apache/certbot_apache/tests/obj_test.py rename to certbot-apache/tests/obj_test.py index 10dba18bc..1761b9c94 100644 --- a/certbot-apache/certbot_apache/tests/obj_test.py +++ b/certbot-apache/tests/obj_test.py @@ -1,4 +1,4 @@ -"""Tests for certbot_apache.obj.""" +"""Tests for certbot_apache._internal.obj.""" import unittest @@ -6,8 +6,8 @@ class VirtualHostTest(unittest.TestCase): """Test the VirtualHost class.""" def setUp(self): - from certbot_apache.obj import Addr - from certbot_apache.obj import VirtualHost + from certbot_apache._internal.obj import Addr + from certbot_apache._internal.obj import VirtualHost self.addr1 = Addr.fromstring("127.0.0.1") self.addr2 = Addr.fromstring("127.0.0.1:443") @@ -23,7 +23,8 @@ class VirtualHostTest(unittest.TestCase): "fp", "vhp", set([self.addr2]), False, False, "localhost") def test_repr(self): - self.assertEqual(repr(self.addr2), "certbot_apache.obj.Addr(('127.0.0.1', '443'))") + self.assertEqual(repr(self.addr2), + "certbot_apache._internal.obj.Addr(('127.0.0.1', '443'))") def test_eq(self): self.assertTrue(self.vhost1b == self.vhost1) @@ -36,8 +37,8 @@ class VirtualHostTest(unittest.TestCase): self.assertFalse(self.vhost1 != self.vhost1b) def test_conflicts(self): - from certbot_apache.obj import Addr - from certbot_apache.obj import VirtualHost + from certbot_apache._internal.obj import Addr + from certbot_apache._internal.obj import VirtualHost complex_vh = VirtualHost( "fp", "vhp", @@ -54,7 +55,7 @@ class VirtualHostTest(unittest.TestCase): self.addr_default])) def test_same_server(self): - from certbot_apache.obj import VirtualHost + from certbot_apache._internal.obj import VirtualHost no_name1 = VirtualHost( "fp", "vhp", set([self.addr1]), False, False, None) no_name2 = VirtualHost( @@ -77,7 +78,7 @@ class VirtualHostTest(unittest.TestCase): class AddrTest(unittest.TestCase): """Test obj.Addr.""" def setUp(self): - from certbot_apache.obj import Addr + from certbot_apache._internal.obj import Addr self.addr = Addr.fromstring("*:443") self.addr1 = Addr.fromstring("127.0.0.1") @@ -92,7 +93,7 @@ class AddrTest(unittest.TestCase): self.assertTrue(self.addr2.is_wildcard()) def test_get_sni_addr(self): - from certbot_apache.obj import Addr + from certbot_apache._internal.obj import Addr self.assertEqual( self.addr.get_sni_addr("443"), Addr.fromstring("*:443")) self.assertEqual( diff --git a/certbot-apache/certbot_apache/tests/parser_test.py b/certbot-apache/tests/parser_test.py similarity index 75% rename from certbot-apache/certbot_apache/tests/parser_test.py rename to certbot-apache/tests/parser_test.py index a089ec471..b334ce52e 100644 --- a/certbot-apache/certbot_apache/tests/parser_test.py +++ b/certbot-apache/tests/parser_test.py @@ -1,14 +1,12 @@ -"""Tests for certbot_apache.parser.""" -import os +"""Tests for certbot_apache._internal.parser.""" import shutil import unittest -import augeas import mock from certbot import errors - -from certbot_apache.tests import util +from certbot.compat import os +import util class BasicParserTest(util.ParserTest): @@ -22,6 +20,27 @@ class BasicParserTest(util.ParserTest): shutil.rmtree(self.config_dir) shutil.rmtree(self.work_dir) + def test_bad_parse(self): + self.parser.parse_file(os.path.join(self.parser.root, + "conf-available", "bad_conf_file.conf")) + self.assertRaises( + errors.PluginError, self.parser.check_parsing_errors, "httpd.aug") + + def test_bad_save(self): + mock_save = mock.Mock() + mock_save.side_effect = IOError + self.parser.aug.save = mock_save + self.assertRaises(errors.PluginError, self.parser.unsaved_files) + + def test_aug_version(self): + mock_match = mock.Mock(return_value=["something"]) + self.parser.aug.match = mock_match + # pylint: disable=protected-access + self.assertEqual(self.parser.check_aug_version(), + ["something"]) + self.parser.aug.match.side_effect = RuntimeError + self.assertFalse(self.parser.check_aug_version()) + def test_find_config_root_no_root(self): # pylint: disable=protected-access os.remove(self.parser.loc["root"]) @@ -52,7 +71,7 @@ class BasicParserTest(util.ParserTest): test2 = self.parser.find_dir("documentroot") self.assertEqual(len(test), 1) - self.assertEqual(len(test2), 7) + self.assertEqual(len(test2), 8) def test_add_dir(self): aug_default = "/files" + self.parser.loc["default"] @@ -93,7 +112,7 @@ class BasicParserTest(util.ParserTest): Path must be valid before attempting to add to augeas """ - from certbot_apache.parser import get_aug_path + from certbot_apache._internal.parser import get_aug_path # This makes sure that find_dir will work self.parser.modules.add("mod_ssl.c") @@ -107,7 +126,7 @@ class BasicParserTest(util.ParserTest): self.assertTrue("IfModule" in matches[0]) def test_add_dir_to_ifmodssl_multiple(self): - from certbot_apache.parser import get_aug_path + from certbot_apache._internal.parser import get_aug_path # This makes sure that find_dir will work self.parser.modules.add("mod_ssl.c") @@ -121,11 +140,11 @@ class BasicParserTest(util.ParserTest): self.assertTrue("IfModule" in matches[0]) def test_get_aug_path(self): - from certbot_apache.parser import get_aug_path + from certbot_apache._internal.parser import get_aug_path self.assertEqual("/files/etc/apache", get_aug_path("/etc/apache")) def test_set_locations(self): - with mock.patch("certbot_apache.parser.os.path") as mock_path: + with mock.patch("certbot_apache._internal.parser.os.path") as mock_path: mock_path.isfile.side_effect = [False, False] @@ -135,18 +154,18 @@ class BasicParserTest(util.ParserTest): self.assertEqual(results["default"], results["listen"]) self.assertEqual(results["default"], results["name"]) - @mock.patch("certbot_apache.parser.ApacheParser.find_dir") - @mock.patch("certbot_apache.parser.ApacheParser.get_arg") + @mock.patch("certbot_apache._internal.parser.ApacheParser.find_dir") + @mock.patch("certbot_apache._internal.parser.ApacheParser.get_arg") def test_parse_modules_bad_syntax(self, mock_arg, mock_find): mock_find.return_value = ["1", "2", "3", "4", "5", "6", "7", "8"] mock_arg.return_value = None - with mock.patch("certbot_apache.parser.logger") as mock_logger: + with mock.patch("certbot_apache._internal.parser.logger") as mock_logger: self.parser.parse_modules() # Make sure that we got None return value and logged the file self.assertTrue(mock_logger.debug.called) - @mock.patch("certbot_apache.parser.ApacheParser.find_dir") - @mock.patch("certbot_apache.parser.ApacheParser._get_runtime_cfg") + @mock.patch("certbot_apache._internal.parser.ApacheParser.find_dir") + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") def test_update_runtime_variables(self, mock_cfg, _): define_val = ( 'ServerRoot: "/etc/apache2"\n' @@ -234,6 +253,7 @@ class BasicParserTest(util.ParserTest): return inc_val elif cmd[-1] == "DUMP_MODULES": return mod_val + return None # pragma: no cover mock_cfg.side_effect = mock_get_vars @@ -242,7 +262,7 @@ class BasicParserTest(util.ParserTest): self.parser.modules = set() with mock.patch( - "certbot_apache.parser.ApacheParser.parse_file") as mock_parse: + "certbot_apache._internal.parser.ApacheParser.parse_file") as mock_parse: self.parser.update_runtime_variables() self.assertEqual(self.parser.variables, expected_vars) self.assertEqual(len(self.parser.modules), 58) @@ -250,8 +270,8 @@ class BasicParserTest(util.ParserTest): # Make sure we tried to include them all. self.assertEqual(mock_parse.call_count, 25) - @mock.patch("certbot_apache.parser.ApacheParser.find_dir") - @mock.patch("certbot_apache.parser.ApacheParser._get_runtime_cfg") + @mock.patch("certbot_apache._internal.parser.ApacheParser.find_dir") + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") def test_update_runtime_variables_alt_values(self, mock_cfg, _): inc_val = ( 'Included configuration files:\n' @@ -265,7 +285,7 @@ class BasicParserTest(util.ParserTest): self.parser.modules = set() with mock.patch( - "certbot_apache.parser.ApacheParser.parse_file") as mock_parse: + "certbot_apache._internal.parser.ApacheParser.parse_file") as mock_parse: self.parser.update_runtime_variables() # No matching modules should have been found self.assertEqual(len(self.parser.modules), 0) @@ -273,7 +293,7 @@ class BasicParserTest(util.ParserTest): # path derived from root configuration Include statements self.assertEqual(mock_parse.call_count, 1) - @mock.patch("certbot_apache.parser.ApacheParser._get_runtime_cfg") + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") def test_update_runtime_vars_bad_output(self, mock_cfg): mock_cfg.return_value = "Define: TLS=443=24" self.parser.update_runtime_variables() @@ -282,8 +302,8 @@ class BasicParserTest(util.ParserTest): self.assertRaises( errors.PluginError, self.parser.update_runtime_variables) - @mock.patch("certbot_apache.configurator.ApacheConfigurator.option") - @mock.patch("certbot_apache.parser.subprocess.Popen") + @mock.patch("certbot_apache._internal.configurator.ApacheConfigurator.option") + @mock.patch("certbot_apache._internal.parser.subprocess.Popen") def test_update_runtime_vars_bad_ctl(self, mock_popen, mock_opt): mock_popen.side_effect = OSError mock_opt.return_value = "nonexistent" @@ -291,7 +311,7 @@ class BasicParserTest(util.ParserTest): errors.MisconfigurationError, self.parser.update_runtime_variables) - @mock.patch("certbot_apache.parser.subprocess.Popen") + @mock.patch("certbot_apache._internal.parser.subprocess.Popen") def test_update_runtime_vars_bad_exit(self, mock_popen): mock_popen().communicate.return_value = ("", "") mock_popen.returncode = -1 @@ -300,7 +320,7 @@ class BasicParserTest(util.ParserTest): self.parser.update_runtime_variables) def test_add_comment(self): - from certbot_apache.parser import get_aug_path + from certbot_apache._internal.parser import get_aug_path self.parser.add_comment(get_aug_path(self.parser.loc["name"]), "123456") comm = self.parser.find_comments("123456") self.assertEqual(len(comm), 1) @@ -310,53 +330,69 @@ class BasicParserTest(util.ParserTest): class ParserInitTest(util.ApacheTest): def setUp(self): # pylint: disable=arguments-differ super(ParserInitTest, self).setUp() - self.aug = augeas.Augeas( - flags=augeas.Augeas.NONE | augeas.Augeas.NO_MODL_AUTOLOAD) def tearDown(self): shutil.rmtree(self.temp_dir) shutil.rmtree(self.config_dir) shutil.rmtree(self.work_dir) - @mock.patch("certbot_apache.parser.ApacheParser._get_runtime_cfg") + @mock.patch("certbot_apache._internal.parser.ApacheParser.init_augeas") + def test_prepare_no_augeas(self, mock_init_augeas): + from certbot_apache._internal.parser import ApacheParser + mock_init_augeas.side_effect = errors.NoInstallationError + self.config.config_test = mock.Mock() + self.assertRaises( + errors.NoInstallationError, ApacheParser, + os.path.relpath(self.config_path), "/dummy/vhostpath", + version=(2, 4, 22), configurator=self.config) + + def test_init_old_aug(self): + from certbot_apache._internal.parser import ApacheParser + with mock.patch("certbot_apache._internal.parser.ApacheParser.check_aug_version") as mock_c: + mock_c.return_value = False + self.assertRaises( + errors.NotSupportedError, + ApacheParser, os.path.relpath(self.config_path), + "/dummy/vhostpath", version=(2, 4, 22), configurator=self.config) + + @mock.patch("certbot_apache._internal.parser.ApacheParser._get_runtime_cfg") def test_unparseable(self, mock_cfg): - from certbot_apache.parser import ApacheParser + from certbot_apache._internal.parser import ApacheParser mock_cfg.return_value = ('Define: TEST') self.assertRaises( errors.PluginError, - ApacheParser, self.aug, os.path.relpath(self.config_path), + ApacheParser, os.path.relpath(self.config_path), "/dummy/vhostpath", version=(2, 2, 22), configurator=self.config) def test_root_normalized(self): - from certbot_apache.parser import ApacheParser + from certbot_apache._internal.parser import ApacheParser - with mock.patch("certbot_apache.parser.ApacheParser." + with mock.patch("certbot_apache._internal.parser.ApacheParser." "update_runtime_variables"): path = os.path.join( self.temp_dir, "debian_apache_2_4/////multiple_vhosts/../multiple_vhosts/apache2") - parser = ApacheParser(self.aug, path, - "/dummy/vhostpath", configurator=self.config) + parser = ApacheParser(path, "/dummy/vhostpath", configurator=self.config) self.assertEqual(parser.root, self.config_path) def test_root_absolute(self): - from certbot_apache.parser import ApacheParser - with mock.patch("certbot_apache.parser.ApacheParser." + from certbot_apache._internal.parser import ApacheParser + with mock.patch("certbot_apache._internal.parser.ApacheParser." "update_runtime_variables"): parser = ApacheParser( - self.aug, os.path.relpath(self.config_path), + os.path.relpath(self.config_path), "/dummy/vhostpath", configurator=self.config) self.assertEqual(parser.root, self.config_path) def test_root_no_trailing_slash(self): - from certbot_apache.parser import ApacheParser - with mock.patch("certbot_apache.parser.ApacheParser." + from certbot_apache._internal.parser import ApacheParser + with mock.patch("certbot_apache._internal.parser.ApacheParser." "update_runtime_variables"): parser = ApacheParser( - self.aug, self.config_path + os.path.sep, + self.config_path + os.path.sep, "/dummy/vhostpath", configurator=self.config) self.assertEqual(parser.root, self.config_path) diff --git a/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/README b/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/README new file mode 100644 index 000000000..c12e149f2 --- /dev/null +++ b/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/README @@ -0,0 +1,9 @@ + +This directory holds Apache 2.0 module-specific configuration files; +any files in this directory which have the ".conf" extension will be +processed as Apache configuration files. + +Files are processed in alphabetical order, so if using configuration +directives which depend on, say, mod_perl being loaded, ensure that +these are placed in a filename later in the sort order than "perl.conf". + diff --git a/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/ssl.conf b/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/ssl.conf new file mode 100644 index 000000000..abe07dd0c --- /dev/null +++ b/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/ssl.conf @@ -0,0 +1,222 @@ +# +# This is the Apache server configuration file providing SSL support. +# It contains the configuration directives to instruct the server how to +# serve pages over an https connection. For detailing information about these +# directives see +# +# Do NOT simply read the instructions in here without understanding +# what they do. They're here only as hints or reminders. If you are unsure +# consult the online docs. You have been warned. +# + +LoadModule ssl_module modules/mod_ssl.so + +# +# When we also provide SSL we have to listen to the +# the HTTPS port in addition. +# +Listen 443 + +## +## SSL Global Context +## +## All SSL configuration in this context applies both to +## the main server and all SSL-enabled virtual hosts. +## + +# Pass Phrase Dialog: +# Configure the pass phrase gathering process. +# The filtering dialog program (`builtin' is an internal +# terminal dialog) has to provide the pass phrase on stdout. +SSLPassPhraseDialog builtin + +# Inter-Process Session Cache: +# Configure the SSL Session Cache: First the mechanism +# to use and second the expiring timeout (in seconds). +SSLSessionCache shmcb:/var/cache/mod_ssl/scache(512000) +SSLSessionCacheTimeout 300 + +# Semaphore: +# Configure the path to the mutual exclusion semaphore the +# SSL engine uses internally for inter-process synchronization. +SSLMutex default + +# Pseudo Random Number Generator (PRNG): +# Configure one or more sources to seed the PRNG of the +# SSL library. The seed data should be of good random quality. +# WARNING! On some platforms /dev/random blocks if not enough entropy +# is available. This means you then cannot use the /dev/random device +# because it would lead to very long connection times (as long as +# it requires to make more entropy available). But usually those +# platforms additionally provide a /dev/urandom device which doesn't +# block. So, if available, use this one instead. Read the mod_ssl User +# Manual for more details. +SSLRandomSeed startup file:/dev/urandom 256 +SSLRandomSeed connect builtin +#SSLRandomSeed startup file:/dev/random 512 +#SSLRandomSeed connect file:/dev/random 512 +#SSLRandomSeed connect file:/dev/urandom 512 + +# +# Use "SSLCryptoDevice" to enable any supported hardware +# accelerators. Use "openssl engine -v" to list supported +# engine names. NOTE: If you enable an accelerator and the +# server does not start, consult the error logs and ensure +# your accelerator is functioning properly. +# +SSLCryptoDevice builtin +#SSLCryptoDevice ubsec + +## +## SSL Virtual Host Context +## + + + +# General setup for the virtual host, inherited from global configuration +#DocumentRoot "/var/www/html" +#ServerName www.example.com:443 + +# Use separate log files for the SSL virtual host; note that LogLevel +# is not inherited from httpd.conf. +ErrorLog logs/ssl_error_log +TransferLog logs/ssl_access_log +LogLevel warn + +# SSL Engine Switch: +# Enable/Disable SSL for this virtual host. +SSLEngine on + +# SSL Protocol support: +# List the enable protocol levels with which clients will be able to +# connect. Disable SSLv2 access by default: +SSLProtocol all -SSLv2 + +# SSL Cipher Suite: +# List the ciphers that the client is permitted to negotiate. +# See the mod_ssl documentation for a complete list. +SSLCipherSuite DEFAULT:!EXP:!SSLv2:!DES:!IDEA:!SEED:+3DES + +# Server Certificate: +# Point SSLCertificateFile at a PEM encoded certificate. If +# the certificate is encrypted, then you will be prompted for a +# pass phrase. Note that a kill -HUP will prompt again. A new +# certificate can be generated using the genkey(1) command. +SSLCertificateFile /etc/pki/tls/certs/localhost.crt + +# Server Private Key: +# If the key is not combined with the certificate, use this +# directive to point at the key file. Keep in mind that if +# you've both a RSA and a DSA private key you can configure +# both in parallel (to also allow the use of DSA ciphers, etc.) +SSLCertificateKeyFile /etc/pki/tls/private/localhost.key + +# Server Certificate Chain: +# Point SSLCertificateChainFile at a file containing the +# concatenation of PEM encoded CA certificates which form the +# certificate chain for the server certificate. Alternatively +# the referenced file can be the same as SSLCertificateFile +# when the CA certificates are directly appended to the server +# certificate for convinience. +#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt + +# Certificate Authority (CA): +# Set the CA certificate verification path where to find CA +# certificates for client authentication or alternatively one +# huge file containing all of them (file must be PEM encoded) +#SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt + +# Client Authentication (Type): +# Client certificate verification type and depth. Types are +# none, optional, require and optional_no_ca. Depth is a +# number which specifies how deeply to verify the certificate +# issuer chain before deciding the certificate is not valid. +#SSLVerifyClient require +#SSLVerifyDepth 10 + +# Access Control: +# With SSLRequire you can do per-directory access control based +# on arbitrary complex boolean expressions containing server +# variable checks and other lookup directives. The syntax is a +# mixture between C and Perl. See the mod_ssl documentation +# for more details. +# +#SSLRequire ( %{SSL_CIPHER} !~ m/^(EXP|NULL)/ \ +# and %{SSL_CLIENT_S_DN_O} eq "Snake Oil, Ltd." \ +# and %{SSL_CLIENT_S_DN_OU} in {"Staff", "CA", "Dev"} \ +# and %{TIME_WDAY} >= 1 and %{TIME_WDAY} <= 5 \ +# and %{TIME_HOUR} >= 8 and %{TIME_HOUR} <= 20 ) \ +# or %{REMOTE_ADDR} =~ m/^192\.76\.162\.[0-9]+$/ +# + +# SSL Engine Options: +# Set various options for the SSL engine. +# o FakeBasicAuth: +# Translate the client X.509 into a Basic Authorisation. This means that +# the standard Auth/DBMAuth methods can be used for access control. The +# user name is the `one line' version of the client's X.509 certificate. +# Note that no password is obtained from the user. Every entry in the user +# file needs this password: `xxj31ZMTZzkVA'. +# o ExportCertData: +# This exports two additional environment variables: SSL_CLIENT_CERT and +# SSL_SERVER_CERT. These contain the PEM-encoded certificates of the +# server (always existing) and the client (only existing when client +# authentication is used). This can be used to import the certificates +# into CGI scripts. +# o StdEnvVars: +# This exports the standard SSL/TLS related `SSL_*' environment variables. +# Per default this exportation is switched off for performance reasons, +# because the extraction step is an expensive operation and is usually +# useless for serving static content. So one usually enables the +# exportation for CGI and SSI requests only. +# o StrictRequire: +# This denies access when "SSLRequireSSL" or "SSLRequire" applied even +# under a "Satisfy any" situation, i.e. when it applies access is denied +# and no other module can change it. +# o OptRenegotiate: +# This enables optimized SSL connection renegotiation handling when SSL +# directives are used in per-directory context. +#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire + + SSLOptions +StdEnvVars + + + SSLOptions +StdEnvVars + + +# SSL Protocol Adjustments: +# The safe and default but still SSL/TLS standard compliant shutdown +# approach is that mod_ssl sends the close notify alert but doesn't wait for +# the close notify alert from client. When you need a different shutdown +# approach you can use one of the following variables: +# o ssl-unclean-shutdown: +# This forces an unclean shutdown when the connection is closed, i.e. no +# SSL close notify alert is send or allowed to received. This violates +# the SSL/TLS standard but is needed for some brain-dead browsers. Use +# this when you receive I/O errors because of the standard approach where +# mod_ssl sends the close notify alert. +# o ssl-accurate-shutdown: +# This forces an accurate shutdown when the connection is closed, i.e. a +# SSL close notify alert is send and mod_ssl waits for the close notify +# alert of the client. This is 100% SSL/TLS standard compliant, but in +# practice often causes hanging connections with brain-dead browsers. Use +# this only for browsers where you know that their SSL implementation +# works correctly. +# Notice: Most problems of broken clients are also related to the HTTP +# keep-alive facility, so you usually additionally want to disable +# keep-alive for those clients, too. Use variable "nokeepalive" for this. +# Similarly, one has to force some clients to use HTTP/1.0 to workaround +# their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and +# "force-response-1.0" for this. +SetEnvIf User-Agent ".*MSIE.*" \ + nokeepalive ssl-unclean-shutdown \ + downgrade-1.0 force-response-1.0 + +# Per-Server Logging: +# The home of a custom SSL log file. Use this when you want a +# compact non-error SSL logfile on a virtual host basis. +CustomLog logs/ssl_request_log \ + "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" + + + diff --git a/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/test.example.com.conf b/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/test.example.com.conf new file mode 100644 index 000000000..3dd7b18f1 --- /dev/null +++ b/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/test.example.com.conf @@ -0,0 +1,7 @@ + + ServerName test.example.com + ServerAdmin webmaster@dummy-host.example.com + DocumentRoot /var/www/htdocs + ErrorLog logs/dummy-host.example.com-error_log + CustomLog logs/dummy-host.example.com-access_log common + diff --git a/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/welcome.conf b/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/welcome.conf new file mode 100644 index 000000000..c1d23c512 --- /dev/null +++ b/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf.d/welcome.conf @@ -0,0 +1,11 @@ +# +# This configuration file enables the default "Welcome" +# page if there is no default index page present for +# the root URL. To disable the Welcome page, comment +# out all the lines below. +# + + Options -Indexes + ErrorDocument 403 /error/noindex.html + + diff --git a/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf/httpd.conf b/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf/httpd.conf new file mode 100644 index 000000000..eac6143da --- /dev/null +++ b/certbot-apache/tests/testdata/centos6_apache/apache/httpd/conf/httpd.conf @@ -0,0 +1,1009 @@ +# +# This is the main Apache server configuration file. It contains the +# configuration directives that give the server its instructions. +# See for detailed information. +# In particular, see +# +# for a discussion of each configuration directive. +# +# +# Do NOT simply read the instructions in here without understanding +# what they do. They're here only as hints or reminders. If you are unsure +# consult the online docs. You have been warned. +# +# The configuration directives are grouped into three basic sections: +# 1. Directives that control the operation of the Apache server process as a +# whole (the 'global environment'). +# 2. Directives that define the parameters of the 'main' or 'default' server, +# which responds to requests that aren't handled by a virtual host. +# These directives also provide default values for the settings +# of all virtual hosts. +# 3. Settings for virtual hosts, which allow Web requests to be sent to +# different IP addresses or hostnames and have them handled by the +# same Apache server process. +# +# Configuration and logfile names: If the filenames you specify for many +# of the server's control files begin with "/" (or "drive:/" for Win32), the +# server will use that explicit path. If the filenames do *not* begin +# with "/", the value of ServerRoot is prepended -- so "logs/foo.log" +# with ServerRoot set to "/etc/httpd" will be interpreted by the +# server as "/etc/httpd/logs/foo.log". +# + +### Section 1: Global Environment +# +# The directives in this section affect the overall operation of Apache, +# such as the number of concurrent requests it can handle or where it +# can find its configuration files. +# + +# +# Don't give away too much information about all the subcomponents +# we are running. Comment out this line if you don't mind remote sites +# finding out what major optional modules you are running +ServerTokens OS + +# +# ServerRoot: The top of the directory tree under which the server's +# configuration, error, and log files are kept. +# +# NOTE! If you intend to place this on an NFS (or otherwise network) +# mounted filesystem then please read the LockFile documentation +# (available at ); +# you will save yourself a lot of trouble. +# +# Do NOT add a slash at the end of the directory path. +# +ServerRoot "/etc/httpd" + +# +# PidFile: The file in which the server should record its process +# identification number when it starts. Note the PIDFILE variable in +# /etc/sysconfig/httpd must be set appropriately if this location is +# changed. +# +PidFile run/httpd.pid + +# +# Timeout: The number of seconds before receives and sends time out. +# +Timeout 60 + +# +# KeepAlive: Whether or not to allow persistent connections (more than +# one request per connection). Set to "Off" to deactivate. +# +KeepAlive Off + +# +# MaxKeepAliveRequests: The maximum number of requests to allow +# during a persistent connection. Set to 0 to allow an unlimited amount. +# We recommend you leave this number high, for maximum performance. +# +MaxKeepAliveRequests 100 + +# +# KeepAliveTimeout: Number of seconds to wait for the next request from the +# same client on the same connection. +# +KeepAliveTimeout 15 + +## +## Server-Pool Size Regulation (MPM specific) +## + +# prefork MPM +# StartServers: number of server processes to start +# MinSpareServers: minimum number of server processes which are kept spare +# MaxSpareServers: maximum number of server processes which are kept spare +# ServerLimit: maximum value for MaxClients for the lifetime of the server +# MaxClients: maximum number of server processes allowed to start +# MaxRequestsPerChild: maximum number of requests a server process serves + +StartServers 8 +MinSpareServers 5 +MaxSpareServers 20 +ServerLimit 256 +MaxClients 256 +MaxRequestsPerChild 4000 + + +# worker MPM +# StartServers: initial number of server processes to start +# MaxClients: maximum number of simultaneous client connections +# MinSpareThreads: minimum number of worker threads which are kept spare +# MaxSpareThreads: maximum number of worker threads which are kept spare +# ThreadsPerChild: constant number of worker threads in each server process +# MaxRequestsPerChild: maximum number of requests a server process serves + +StartServers 4 +MaxClients 300 +MinSpareThreads 25 +MaxSpareThreads 75 +ThreadsPerChild 25 +MaxRequestsPerChild 0 + + +# +# Listen: Allows you to bind Apache to specific IP addresses and/or +# ports, in addition to the default. See also the +# directive. +# +# Change this to Listen on specific IP addresses as shown below to +# prevent Apache from glomming onto all bound IP addresses (0.0.0.0) +# +#Listen 12.34.56.78:80 +Listen 80 + +# +# Dynamic Shared Object (DSO) Support +# +# To be able to use the functionality of a module which was built as a DSO you +# have to place corresponding `LoadModule' lines at this location so the +# directives contained in it are actually available _before_ they are used. +# Statically compiled modules (those listed by `httpd -l') do not need +# to be loaded here. +# +# Example: +# LoadModule foo_module modules/mod_foo.so +# +LoadModule auth_basic_module modules/mod_auth_basic.so +LoadModule auth_digest_module modules/mod_auth_digest.so +LoadModule authn_file_module modules/mod_authn_file.so +LoadModule authn_alias_module modules/mod_authn_alias.so +LoadModule authn_anon_module modules/mod_authn_anon.so +LoadModule authn_dbm_module modules/mod_authn_dbm.so +LoadModule authn_default_module modules/mod_authn_default.so +LoadModule authz_host_module modules/mod_authz_host.so +LoadModule authz_user_module modules/mod_authz_user.so +LoadModule authz_owner_module modules/mod_authz_owner.so +LoadModule authz_groupfile_module modules/mod_authz_groupfile.so +LoadModule authz_dbm_module modules/mod_authz_dbm.so +LoadModule authz_default_module modules/mod_authz_default.so +LoadModule ldap_module modules/mod_ldap.so +LoadModule authnz_ldap_module modules/mod_authnz_ldap.so +LoadModule include_module modules/mod_include.so +LoadModule log_config_module modules/mod_log_config.so +LoadModule logio_module modules/mod_logio.so +LoadModule env_module modules/mod_env.so +LoadModule ext_filter_module modules/mod_ext_filter.so +LoadModule mime_magic_module modules/mod_mime_magic.so +LoadModule expires_module modules/mod_expires.so +LoadModule deflate_module modules/mod_deflate.so +LoadModule headers_module modules/mod_headers.so +LoadModule usertrack_module modules/mod_usertrack.so +LoadModule setenvif_module modules/mod_setenvif.so +LoadModule mime_module modules/mod_mime.so +LoadModule dav_module modules/mod_dav.so +LoadModule status_module modules/mod_status.so +LoadModule autoindex_module modules/mod_autoindex.so +LoadModule info_module modules/mod_info.so +LoadModule dav_fs_module modules/mod_dav_fs.so +LoadModule vhost_alias_module modules/mod_vhost_alias.so +LoadModule negotiation_module modules/mod_negotiation.so +LoadModule dir_module modules/mod_dir.so +LoadModule actions_module modules/mod_actions.so +LoadModule speling_module modules/mod_speling.so +LoadModule userdir_module modules/mod_userdir.so +LoadModule alias_module modules/mod_alias.so +LoadModule substitute_module modules/mod_substitute.so +LoadModule rewrite_module modules/mod_rewrite.so +LoadModule proxy_module modules/mod_proxy.so +LoadModule proxy_balancer_module modules/mod_proxy_balancer.so +LoadModule proxy_ftp_module modules/mod_proxy_ftp.so +LoadModule proxy_http_module modules/mod_proxy_http.so +LoadModule proxy_ajp_module modules/mod_proxy_ajp.so +LoadModule proxy_connect_module modules/mod_proxy_connect.so +LoadModule cache_module modules/mod_cache.so +LoadModule suexec_module modules/mod_suexec.so +LoadModule disk_cache_module modules/mod_disk_cache.so +LoadModule cgi_module modules/mod_cgi.so +LoadModule version_module modules/mod_version.so + +# +# The following modules are not loaded by default: +# +#LoadModule asis_module modules/mod_asis.so +#LoadModule authn_dbd_module modules/mod_authn_dbd.so +#LoadModule cern_meta_module modules/mod_cern_meta.so +#LoadModule cgid_module modules/mod_cgid.so +#LoadModule dbd_module modules/mod_dbd.so +#LoadModule dumpio_module modules/mod_dumpio.so +#LoadModule filter_module modules/mod_filter.so +#LoadModule ident_module modules/mod_ident.so +#LoadModule log_forensic_module modules/mod_log_forensic.so +#LoadModule unique_id_module modules/mod_unique_id.so +# + +# +# Load config files from the config directory "/etc/httpd/conf.d". +# +Include conf.d/*.conf + +# +# ExtendedStatus controls whether Apache will generate "full" status +# information (ExtendedStatus On) or just basic information (ExtendedStatus +# Off) when the "server-status" handler is called. The default is Off. +# +#ExtendedStatus On + +# +# If you wish httpd to run as a different user or group, you must run +# httpd as root initially and it will switch. +# +# User/Group: The name (or #number) of the user/group to run httpd as. +# . On SCO (ODT 3) use "User nouser" and "Group nogroup". +# . On HPUX you may not be able to use shared memory as nobody, and the +# suggested workaround is to create a user www and use that user. +# NOTE that some kernels refuse to setgid(Group) or semctl(IPC_SET) +# when the value of (unsigned)Group is above 60000; +# don't use Group #-1 on these systems! +# +User apache +Group apache + +### Section 2: 'Main' server configuration +# +# The directives in this section set up the values used by the 'main' +# server, which responds to any requests that aren't handled by a +# definition. These values also provide defaults for +# any containers you may define later in the file. +# +# All of these directives may appear inside containers, +# in which case these default settings will be overridden for the +# virtual host being defined. +# + +# +# ServerAdmin: Your address, where problems with the server should be +# e-mailed. This address appears on some server-generated pages, such +# as error documents. e.g. admin@your-domain.com +# +ServerAdmin root@localhost + +# +# ServerName gives the name and port that the server uses to identify itself. +# This can often be determined automatically, but we recommend you specify +# it explicitly to prevent problems during startup. +# +# If this is not set to valid DNS name for your host, server-generated +# redirections will not work. See also the UseCanonicalName directive. +# +# If your host doesn't have a registered DNS name, enter its IP address here. +# You will have to access it by its address anyway, and this will make +# redirections work in a sensible way. +# +#ServerName www.example.com:80 + +# +# UseCanonicalName: Determines how Apache constructs self-referencing +# URLs and the SERVER_NAME and SERVER_PORT variables. +# When set "Off", Apache will use the Hostname and Port supplied +# by the client. When set "On", Apache will use the value of the +# ServerName directive. +# +UseCanonicalName Off + +# +# DocumentRoot: The directory out of which you will serve your +# documents. By default, all requests are taken from this directory, but +# symbolic links and aliases may be used to point to other locations. +# +DocumentRoot "/var/www/html" + +# +# Each directory to which Apache has access can be configured with respect +# to which services and features are allowed and/or disabled in that +# directory (and its subdirectories). +# +# First, we configure the "default" to be a very restrictive set of +# features. +# + + Options FollowSymLinks + AllowOverride None + + +# +# Note that from this point forward you must specifically allow +# particular features to be enabled - so if something's not working as +# you might expect, make sure that you have specifically enabled it +# below. +# + +# +# This should be changed to whatever you set DocumentRoot to. +# + + +# +# Possible values for the Options directive are "None", "All", +# or any combination of: +# Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews +# +# Note that "MultiViews" must be named *explicitly* --- "Options All" +# doesn't give it to you. +# +# The Options directive is both complicated and important. Please see +# http://httpd.apache.org/docs/2.2/mod/core.html#options +# for more information. +# + Options Indexes FollowSymLinks + +# +# AllowOverride controls what directives may be placed in .htaccess files. +# It can be "All", "None", or any combination of the keywords: +# Options FileInfo AuthConfig Limit +# + AllowOverride None + +# +# Controls who can get stuff from this server. +# + Order allow,deny + Allow from all + + + +# +# UserDir: The name of the directory that is appended onto a user's home +# directory if a ~user request is received. +# +# The path to the end user account 'public_html' directory must be +# accessible to the webserver userid. This usually means that ~userid +# must have permissions of 711, ~userid/public_html must have permissions +# of 755, and documents contained therein must be world-readable. +# Otherwise, the client will only receive a "403 Forbidden" message. +# +# See also: http://httpd.apache.org/docs/misc/FAQ.html#forbidden +# + + # + # UserDir is disabled by default since it can confirm the presence + # of a username on the system (depending on home directory + # permissions). + # + UserDir disabled + + # + # To enable requests to /~user/ to serve the user's public_html + # directory, remove the "UserDir disabled" line above, and uncomment + # the following line instead: + # + #UserDir public_html + + + +# +# Control access to UserDir directories. The following is an example +# for a site where these directories are restricted to read-only. +# +# +# AllowOverride FileInfo AuthConfig Limit +# Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec +# +# Order allow,deny +# Allow from all +# +# +# Order deny,allow +# Deny from all +# +# + +# +# DirectoryIndex: sets the file that Apache will serve if a directory +# is requested. +# +# The index.html.var file (a type-map) is used to deliver content- +# negotiated documents. The MultiViews Option can be used for the +# same purpose, but it is much slower. +# +DirectoryIndex index.html index.html.var + +# +# AccessFileName: The name of the file to look for in each directory +# for additional configuration directives. See also the AllowOverride +# directive. +# +AccessFileName .htaccess + +# +# The following lines prevent .htaccess and .htpasswd files from being +# viewed by Web clients. +# + + Order allow,deny + Deny from all + Satisfy All + + +# +# TypesConfig describes where the mime.types file (or equivalent) is +# to be found. +# +TypesConfig /etc/mime.types + +# +# DefaultType is the default MIME type the server will use for a document +# if it cannot otherwise determine one, such as from filename extensions. +# If your server contains mostly text or HTML documents, "text/plain" is +# a good value. If most of your content is binary, such as applications +# or images, you may want to use "application/octet-stream" instead to +# keep browsers from trying to display binary files as though they are +# text. +# +DefaultType text/plain + +# +# The mod_mime_magic module allows the server to use various hints from the +# contents of the file itself to determine its type. The MIMEMagicFile +# directive tells the module where the hint definitions are located. +# + +# MIMEMagicFile /usr/share/magic.mime + MIMEMagicFile conf/magic + + +# +# HostnameLookups: Log the names of clients or just their IP addresses +# e.g., www.apache.org (on) or 204.62.129.132 (off). +# The default is off because it'd be overall better for the net if people +# had to knowingly turn this feature on, since enabling it means that +# each client request will result in AT LEAST one lookup request to the +# nameserver. +# +HostnameLookups Off + +# +# EnableMMAP: Control whether memory-mapping is used to deliver +# files (assuming that the underlying OS supports it). +# The default is on; turn this off if you serve from NFS-mounted +# filesystems. On some systems, turning it off (regardless of +# filesystem) can improve performance; for details, please see +# http://httpd.apache.org/docs/2.2/mod/core.html#enablemmap +# +#EnableMMAP off + +# +# EnableSendfile: Control whether the sendfile kernel support is +# used to deliver files (assuming that the OS supports it). +# The default is on; turn this off if you serve from NFS-mounted +# filesystems. Please see +# http://httpd.apache.org/docs/2.2/mod/core.html#enablesendfile +# +#EnableSendfile off + +# +# ErrorLog: The location of the error log file. +# If you do not specify an ErrorLog directive within a +# container, error messages relating to that virtual host will be +# logged here. If you *do* define an error logfile for a +# container, that host's errors will be logged there and not here. +# +ErrorLog logs/error_log + +# +# LogLevel: Control the number of messages logged to the error_log. +# Possible values include: debug, info, notice, warn, error, crit, +# alert, emerg. +# +LogLevel warn + +# +# The following directives define some format nicknames for use with +# a CustomLog directive (see below). +# +LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined +LogFormat "%h %l %u %t \"%r\" %>s %b" common +LogFormat "%{Referer}i -> %U" referer +LogFormat "%{User-agent}i" agent + +# "combinedio" includes actual counts of actual bytes received (%I) and sent (%O); this +# requires the mod_logio module to be loaded. +#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio + +# +# The location and format of the access logfile (Common Logfile Format). +# If you do not define any access logfiles within a +# container, they will be logged here. Contrariwise, if you *do* +# define per- access logfiles, transactions will be +# logged therein and *not* in this file. +# +#CustomLog logs/access_log common + +# +# If you would like to have separate agent and referer logfiles, uncomment +# the following directives. +# +#CustomLog logs/referer_log referer +#CustomLog logs/agent_log agent + +# +# For a single logfile with access, agent, and referer information +# (Combined Logfile Format), use the following directive: +# +CustomLog logs/access_log combined + +# +# Optionally add a line containing the server version and virtual host +# name to server-generated pages (internal error documents, FTP directory +# listings, mod_status and mod_info output etc., but not CGI generated +# documents or custom error documents). +# Set to "EMail" to also include a mailto: link to the ServerAdmin. +# Set to one of: On | Off | EMail +# +ServerSignature On + +# +# Aliases: Add here as many aliases as you need (with no limit). The format is +# Alias fakename realname +# +# Note that if you include a trailing / on fakename then the server will +# require it to be present in the URL. So "/icons" isn't aliased in this +# example, only "/icons/". If the fakename is slash-terminated, then the +# realname must also be slash terminated, and if the fakename omits the +# trailing slash, the realname must also omit it. +# +# We include the /icons/ alias for FancyIndexed directory listings. If you +# do not use FancyIndexing, you may comment this out. +# +Alias /icons/ "/var/www/icons/" + + + Options Indexes MultiViews FollowSymLinks + AllowOverride None + Order allow,deny + Allow from all + + +# +# WebDAV module configuration section. +# + + # Location of the WebDAV lock database. + DAVLockDB /var/lib/dav/lockdb + + +# +# ScriptAlias: This controls which directories contain server scripts. +# ScriptAliases are essentially the same as Aliases, except that +# documents in the realname directory are treated as applications and +# run by the server when requested rather than as documents sent to the client. +# The same rules about trailing "/" apply to ScriptAlias directives as to +# Alias. +# +ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" + +# +# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased +# CGI directory exists, if you have that configured. +# + + AllowOverride None + Options None + Order allow,deny + Allow from all + + +# +# Redirect allows you to tell clients about documents which used to exist in +# your server's namespace, but do not anymore. This allows you to tell the +# clients where to look for the relocated document. +# Example: +# Redirect permanent /foo http://www.example.com/bar + +# +# Directives controlling the display of server-generated directory listings. +# + +# +# IndexOptions: Controls the appearance of server-generated directory +# listings. +# +IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable Charset=UTF-8 + +# +# AddIcon* directives tell the server which icon to show for different +# files or filename extensions. These are only displayed for +# FancyIndexed directories. +# +AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip + +AddIconByType (TXT,/icons/text.gif) text/* +AddIconByType (IMG,/icons/image2.gif) image/* +AddIconByType (SND,/icons/sound2.gif) audio/* +AddIconByType (VID,/icons/movie.gif) video/* + +AddIcon /icons/binary.gif .bin .exe +AddIcon /icons/binhex.gif .hqx +AddIcon /icons/tar.gif .tar +AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv +AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip +AddIcon /icons/a.gif .ps .ai .eps +AddIcon /icons/layout.gif .html .shtml .htm .pdf +AddIcon /icons/text.gif .txt +AddIcon /icons/c.gif .c +AddIcon /icons/p.gif .pl .py +AddIcon /icons/f.gif .for +AddIcon /icons/dvi.gif .dvi +AddIcon /icons/uuencoded.gif .uu +AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl +AddIcon /icons/tex.gif .tex +AddIcon /icons/bomb.gif /core + +AddIcon /icons/back.gif .. +AddIcon /icons/hand.right.gif README +AddIcon /icons/folder.gif ^^DIRECTORY^^ +AddIcon /icons/blank.gif ^^BLANKICON^^ + +# +# DefaultIcon is which icon to show for files which do not have an icon +# explicitly set. +# +DefaultIcon /icons/unknown.gif + +# +# AddDescription allows you to place a short description after a file in +# server-generated indexes. These are only displayed for FancyIndexed +# directories. +# Format: AddDescription "description" filename +# +#AddDescription "GZIP compressed document" .gz +#AddDescription "tar archive" .tar +#AddDescription "GZIP compressed tar archive" .tgz + +# +# ReadmeName is the name of the README file the server will look for by +# default, and append to directory listings. +# +# HeaderName is the name of a file which should be prepended to +# directory indexes. +ReadmeName README.html +HeaderName HEADER.html + +# +# IndexIgnore is a set of filenames which directory indexing should ignore +# and not include in the listing. Shell-style wildcarding is permitted. +# +IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t + +# +# DefaultLanguage and AddLanguage allows you to specify the language of +# a document. You can then use content negotiation to give a browser a +# file in a language the user can understand. +# +# Specify a default language. This means that all data +# going out without a specific language tag (see below) will +# be marked with this one. You probably do NOT want to set +# this unless you are sure it is correct for all cases. +# +# * It is generally better to not mark a page as +# * being a certain language than marking it with the wrong +# * language! +# +# DefaultLanguage nl +# +# Note 1: The suffix does not have to be the same as the language +# keyword --- those with documents in Polish (whose net-standard +# language code is pl) may wish to use "AddLanguage pl .po" to +# avoid the ambiguity with the common suffix for perl scripts. +# +# Note 2: The example entries below illustrate that in some cases +# the two character 'Language' abbreviation is not identical to +# the two character 'Country' code for its country, +# E.g. 'Danmark/dk' versus 'Danish/da'. +# +# Note 3: In the case of 'ltz' we violate the RFC by using a three char +# specifier. There is 'work in progress' to fix this and get +# the reference data for rfc1766 cleaned up. +# +# Catalan (ca) - Croatian (hr) - Czech (cs) - Danish (da) - Dutch (nl) +# English (en) - Esperanto (eo) - Estonian (et) - French (fr) - German (de) +# Greek-Modern (el) - Hebrew (he) - Italian (it) - Japanese (ja) +# Korean (ko) - Luxembourgeois* (ltz) - Norwegian Nynorsk (nn) +# Norwegian (no) - Polish (pl) - Portuguese (pt) +# Brazilian Portuguese (pt-BR) - Russian (ru) - Swedish (sv) +# Simplified Chinese (zh-CN) - Spanish (es) - Traditional Chinese (zh-TW) +# +AddLanguage ca .ca +AddLanguage cs .cz .cs +AddLanguage da .dk +AddLanguage de .de +AddLanguage el .el +AddLanguage en .en +AddLanguage eo .eo +AddLanguage es .es +AddLanguage et .et +AddLanguage fr .fr +AddLanguage he .he +AddLanguage hr .hr +AddLanguage it .it +AddLanguage ja .ja +AddLanguage ko .ko +AddLanguage ltz .ltz +AddLanguage nl .nl +AddLanguage nn .nn +AddLanguage no .no +AddLanguage pl .po +AddLanguage pt .pt +AddLanguage pt-BR .pt-br +AddLanguage ru .ru +AddLanguage sv .sv +AddLanguage zh-CN .zh-cn +AddLanguage zh-TW .zh-tw + +# +# LanguagePriority allows you to give precedence to some languages +# in case of a tie during content negotiation. +# +# Just list the languages in decreasing order of preference. We have +# more or less alphabetized them here. You probably want to change this. +# +LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv zh-CN zh-TW + +# +# ForceLanguagePriority allows you to serve a result page rather than +# MULTIPLE CHOICES (Prefer) [in case of a tie] or NOT ACCEPTABLE (Fallback) +# [in case no accepted languages matched the available variants] +# +ForceLanguagePriority Prefer Fallback + +# +# Specify a default charset for all content served; this enables +# interpretation of all content as UTF-8 by default. To use the +# default browser choice (ISO-8859-1), or to allow the META tags +# in HTML content to override this choice, comment out this +# directive: +# +AddDefaultCharset UTF-8 + +# +# AddType allows you to add to or override the MIME configuration +# file mime.types for specific file types. +# +#AddType application/x-tar .tgz + +# +# AddEncoding allows you to have certain browsers uncompress +# information on the fly. Note: Not all browsers support this. +# Despite the name similarity, the following Add* directives have nothing +# to do with the FancyIndexing customization directives above. +# +#AddEncoding x-compress .Z +#AddEncoding x-gzip .gz .tgz + +# If the AddEncoding directives above are commented-out, then you +# probably should define those extensions to indicate media types: +# +AddType application/x-compress .Z +AddType application/x-gzip .gz .tgz + +# +# MIME-types for downloading Certificates and CRLs +# +AddType application/x-x509-ca-cert .crt +AddType application/x-pkcs7-crl .crl + +# +# AddHandler allows you to map certain file extensions to "handlers": +# actions unrelated to filetype. These can be either built into the server +# or added with the Action directive (see below) +# +# To use CGI scripts outside of ScriptAliased directories: +# (You will also need to add "ExecCGI" to the "Options" directive.) +# +#AddHandler cgi-script .cgi + +# +# For files that include their own HTTP headers: +# +#AddHandler send-as-is asis + +# +# For type maps (negotiated resources): +# (This is enabled by default to allow the Apache "It Worked" page +# to be distributed in multiple languages.) +# +AddHandler type-map var + +# +# Filters allow you to process content before it is sent to the client. +# +# To parse .shtml files for server-side includes (SSI): +# (You will also need to add "Includes" to the "Options" directive.) +# +AddType text/html .shtml +AddOutputFilter INCLUDES .shtml + +# +# Action lets you define media types that will execute a script whenever +# a matching file is called. This eliminates the need for repeated URL +# pathnames for oft-used CGI file processors. +# Format: Action media/type /cgi-script/location +# Format: Action handler-name /cgi-script/location +# + +# +# Customizable error responses come in three flavors: +# 1) plain text 2) local redirects 3) external redirects +# +# Some examples: +#ErrorDocument 500 "The server made a boo boo." +#ErrorDocument 404 /missing.html +#ErrorDocument 404 "/cgi-bin/missing_handler.pl" +#ErrorDocument 402 http://www.example.com/subscription_info.html +# + +# +# Putting this all together, we can internationalize error responses. +# +# We use Alias to redirect any /error/HTTP_.html.var response to +# our collection of by-error message multi-language collections. We use +# includes to substitute the appropriate text. +# +# You can modify the messages' appearance without changing any of the +# default HTTP_.html.var files by adding the line: +# +# Alias /error/include/ "/your/include/path/" +# +# which allows you to create your own set of files by starting with the +# /var/www/error/include/ files and +# copying them to /your/include/path/, even on a per-VirtualHost basis. +# + +Alias /error/ "/var/www/error/" + + + + + AllowOverride None + Options IncludesNoExec + AddOutputFilter Includes html + AddHandler type-map var + Order allow,deny + Allow from all + LanguagePriority en es de fr + ForceLanguagePriority Prefer Fallback + + +# ErrorDocument 400 /error/HTTP_BAD_REQUEST.html.var +# ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var +# ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var +# ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var +# ErrorDocument 405 /error/HTTP_METHOD_NOT_ALLOWED.html.var +# ErrorDocument 408 /error/HTTP_REQUEST_TIME_OUT.html.var +# ErrorDocument 410 /error/HTTP_GONE.html.var +# ErrorDocument 411 /error/HTTP_LENGTH_REQUIRED.html.var +# ErrorDocument 412 /error/HTTP_PRECONDITION_FAILED.html.var +# ErrorDocument 413 /error/HTTP_REQUEST_ENTITY_TOO_LARGE.html.var +# ErrorDocument 414 /error/HTTP_REQUEST_URI_TOO_LARGE.html.var +# ErrorDocument 415 /error/HTTP_UNSUPPORTED_MEDIA_TYPE.html.var +# ErrorDocument 500 /error/HTTP_INTERNAL_SERVER_ERROR.html.var +# ErrorDocument 501 /error/HTTP_NOT_IMPLEMENTED.html.var +# ErrorDocument 502 /error/HTTP_BAD_GATEWAY.html.var +# ErrorDocument 503 /error/HTTP_SERVICE_UNAVAILABLE.html.var +# ErrorDocument 506 /error/HTTP_VARIANT_ALSO_VARIES.html.var + + + + +# +# The following directives modify normal HTTP response behavior to +# handle known problems with browser implementations. +# +BrowserMatch "Mozilla/2" nokeepalive +BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0 +BrowserMatch "RealPlayer 4\.0" force-response-1.0 +BrowserMatch "Java/1\.0" force-response-1.0 +BrowserMatch "JDK/1\.0" force-response-1.0 + +# +# The following directive disables redirects on non-GET requests for +# a directory that does not include the trailing slash. This fixes a +# problem with Microsoft WebFolders which does not appropriately handle +# redirects for folders with DAV methods. +# Same deal with Apple's DAV filesystem and Gnome VFS support for DAV. +# +BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully +BrowserMatch "MS FrontPage" redirect-carefully +BrowserMatch "^WebDrive" redirect-carefully +BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully +BrowserMatch "^gnome-vfs/1.0" redirect-carefully +BrowserMatch "^XML Spy" redirect-carefully +BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully + +# +# Allow server status reports generated by mod_status, +# with the URL of http://servername/server-status +# Change the ".example.com" to match your domain to enable. +# +# +# SetHandler server-status +# Order deny,allow +# Deny from all +# Allow from .example.com +# + +# +# Allow remote server configuration reports, with the URL of +# http://servername/server-info (requires that mod_info.c be loaded). +# Change the ".example.com" to match your domain to enable. +# +# +# SetHandler server-info +# Order deny,allow +# Deny from all +# Allow from .example.com +# + +# +# Proxy Server directives. Uncomment the following lines to +# enable the proxy server: +# +# +#ProxyRequests On +# +# +# Order deny,allow +# Deny from all +# Allow from .example.com +# + +# +# Enable/disable the handling of HTTP/1.1 "Via:" headers. +# ("Full" adds the server version; "Block" removes all outgoing Via: headers) +# Set to one of: Off | On | Full | Block +# +#ProxyVia On + +# +# To enable a cache of proxied content, uncomment the following lines. +# See http://httpd.apache.org/docs/2.2/mod/mod_cache.html for more details. +# +# +# CacheEnable disk / +# CacheRoot "/var/cache/mod_proxy" +# +# + +# +# End of proxy directives. + +### Section 3: Virtual Hosts +# +# VirtualHost: If you want to maintain multiple domains/hostnames on your +# machine you can setup VirtualHost containers for them. Most configurations +# use only name-based virtual hosts so the server doesn't need to worry about +# IP addresses. This is indicated by the asterisks in the directives below. +# +# Please see the documentation at +# +# for further details before you try to setup virtual hosts. +# +# You may use the command line option '-S' to verify your virtual host +# configuration. + +# +# Use name-based virtual hosting. +# +#NameVirtualHost *:80 +# +# NOTE: NameVirtualHost cannot be used without a port specifier +# (e.g. :80) if mod_ssl is being used, due to the nature of the +# SSL protocol. +# + +# +# VirtualHost example: +# Almost any Apache directive may go into a VirtualHost container. +# The first VirtualHost section is used for requests without a known +# server name. +# +# +# ServerAdmin webmaster@dummy-host.example.com +# DocumentRoot /www/docs/dummy-host.example.com +# ServerName dummy-host.example.com +# ErrorLog logs/dummy-host.example.com-error_log +# CustomLog logs/dummy-host.example.com-access_log common +# diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/README b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/README similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/README rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/README diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/autoindex.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/autoindex.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/autoindex.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/autoindex.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/centos.example.com.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/centos.example.com.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/centos.example.com.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/centos.example.com.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/ssl.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/ssl.conf similarity index 99% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/ssl.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/ssl.conf index 6e2502e9a..c90fc780f 100644 --- a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/ssl.conf +++ b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/ssl.conf @@ -13,7 +13,7 @@ Listen 443 https # Pass Phrase Dialog: # Configure the pass phrase gathering process. -# The filtering dialog program (`builtin' is a internal +# The filtering dialog program (`builtin' is an internal # terminal dialog) has to provide the pass phrase on stdout. SSLPassPhraseDialog exec:/usr/libexec/httpd-ssl-pass-dialog diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/userdir.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/userdir.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/userdir.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/userdir.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/welcome.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/welcome.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.d/welcome.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.d/welcome.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-base.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-base.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-base.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-base.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-dav.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-dav.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-dav.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-dav.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-lua.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-lua.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-lua.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-lua.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-mpm.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-mpm.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-mpm.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-mpm.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-proxy.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-proxy.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-proxy.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-proxy.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-ssl.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-ssl.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-ssl.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-ssl.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-systemd.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-systemd.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-systemd.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/00-systemd.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/01-cgi.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/01-cgi.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/01-cgi.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf.modules.d/01-cgi.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf/httpd.conf b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf/httpd.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf/httpd.conf rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf/httpd.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf/magic b/certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf/magic similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/httpd/conf/magic rename to certbot-apache/tests/testdata/centos7_apache/apache/httpd/conf/magic diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/sites b/certbot-apache/tests/testdata/centos7_apache/apache/sites similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/sites rename to certbot-apache/tests/testdata/centos7_apache/apache/sites diff --git a/certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/sysconfig/httpd b/certbot-apache/tests/testdata/centos7_apache/apache/sysconfig/httpd similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/centos7_apache/apache/sysconfig/httpd rename to certbot-apache/tests/testdata/centos7_apache/apache/sysconfig/httpd diff --git a/certbot-apache/certbot_apache/tests/testdata/complex_parsing/apache2.conf b/certbot-apache/tests/testdata/complex_parsing/apache2.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/complex_parsing/apache2.conf rename to certbot-apache/tests/testdata/complex_parsing/apache2.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/complex_parsing/conf-enabled/dummy.conf b/certbot-apache/tests/testdata/complex_parsing/conf-enabled/dummy.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/complex_parsing/conf-enabled/dummy.conf rename to certbot-apache/tests/testdata/complex_parsing/conf-enabled/dummy.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/complex_parsing/test_fnmatch.conf b/certbot-apache/tests/testdata/complex_parsing/test_fnmatch.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/complex_parsing/test_fnmatch.conf rename to certbot-apache/tests/testdata/complex_parsing/test_fnmatch.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/complex_parsing/test_variables.conf b/certbot-apache/tests/testdata/complex_parsing/test_variables.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/complex_parsing/test_variables.conf rename to certbot-apache/tests/testdata/complex_parsing/test_variables.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/apache2.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/apache2.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/apache2.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/apache2.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/bad_conf_file.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/bad_conf_file.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/bad_conf_file.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/bad_conf_file.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/other-vhosts-access-log.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/other-vhosts-access-log.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/other-vhosts-access-log.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/other-vhosts-access-log.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/security.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/security.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/security.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/security.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/serve-cgi-bin.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/serve-cgi-bin.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/serve-cgi-bin.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-available/serve-cgi-bin.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/other-vhosts-access-log.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/other-vhosts-access-log.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/other-vhosts-access-log.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/other-vhosts-access-log.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/security.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/security.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/security.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/security.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/serve-cgi-bin.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/serve-cgi-bin.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/serve-cgi-bin.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/conf-enabled/serve-cgi-bin.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/envvars b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/envvars similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/envvars rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/envvars diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/authz_svn.load b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/authz_svn.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/authz_svn.load rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/authz_svn.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav.load b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav.load rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav_svn.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav_svn.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav_svn.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav_svn.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav_svn.load b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav_svn.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav_svn.load rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/dav_svn.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/rewrite.load b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/rewrite.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/rewrite.load rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/rewrite.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/mods-available/ssl.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/ssl.conf similarity index 98% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/mods-available/ssl.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/ssl.conf index e9fcf4f9b..65baec874 100644 --- a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/mods-available/ssl.conf +++ b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/ssl.conf @@ -31,7 +31,7 @@ # Pass Phrase Dialog: # Configure the pass phrase gathering process. - # The filtering dialog program (`builtin' is a internal + # The filtering dialog program (`builtin' is an internal # terminal dialog) has to provide the pass phrase on stdout. SSLPassPhraseDialog exec:/usr/share/apache2/ask-for-passphrase diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/ssl.load b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/ssl.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/ssl.load rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/ssl.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/authz_svn.load b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/authz_svn.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/authz_svn.load rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/authz_svn.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav.load b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav.load rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav_svn.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav_svn.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav_svn.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav_svn.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav_svn.load b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav_svn.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav_svn.load rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-enabled/dav_svn.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/ports.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/ports.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/ports.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/ports.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/another_wildcard.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/another_wildcard.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/another_wildcard.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/another_wildcard.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/old-and-default.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/old-and-default.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/old-and-default.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/old-and-default.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/wildcard.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/wildcard.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/wildcard.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-available/wildcard.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/another_wildcard.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/another_wildcard.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/another_wildcard.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/another_wildcard.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/old-and-default.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/old-and-default.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/old-and-default.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/old-and-default.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/wildcard.conf b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/wildcard.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/wildcard.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/wildcard.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/sites b/certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/sites similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/sites rename to certbot-apache/tests/testdata/debian_apache_2_4/augeas_vhosts/sites diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/apache2.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/apache2.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/apache2.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/apache2.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/other-vhosts-access-log.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/other-vhosts-access-log.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/other-vhosts-access-log.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/other-vhosts-access-log.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/security.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/security.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/security.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/security.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/serve-cgi-bin.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/serve-cgi-bin.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/serve-cgi-bin.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-available/serve-cgi-bin.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/other-vhosts-access-log.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/other-vhosts-access-log.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/other-vhosts-access-log.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/other-vhosts-access-log.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/security.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/security.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/security.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/security.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/serve-cgi-bin.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/serve-cgi-bin.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/serve-cgi-bin.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/conf-enabled/serve-cgi-bin.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/envvars b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/envvars similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/envvars rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/envvars diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/ssl.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/mods-available/ssl.conf similarity index 98% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/ssl.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/mods-available/ssl.conf index e9fcf4f9b..65baec874 100644 --- a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/ssl.conf +++ b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/mods-available/ssl.conf @@ -31,7 +31,7 @@ # Pass Phrase Dialog: # Configure the pass phrase gathering process. - # The filtering dialog program (`builtin' is a internal + # The filtering dialog program (`builtin' is an internal # terminal dialog) has to provide the pass phrase on stdout. SSLPassPhraseDialog exec:/usr/share/apache2/ask-for-passphrase diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/mods-available/ssl.load b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/mods-available/ssl.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/mods-available/ssl.load rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/mods-available/ssl.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/ports.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/ports.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/ports.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/ports.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-available/000-default.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-available/000-default.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-available/000-default.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-available/000-default.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-available/default-ssl.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-available/default-ssl.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-available/default-ssl.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-available/default-ssl.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-enabled/000-default.conf b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-enabled/000-default.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-enabled/000-default.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/apache2/sites-enabled/000-default.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/sites b/certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/sites similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/default_vhost/sites rename to certbot-apache/tests/testdata/debian_apache_2_4/default_vhost/sites diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/apache2.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/apache2.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/apache2.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/apache2.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/envvars b/certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/envvars similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/envvars rename to certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/envvars diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/ports.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/ports.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/ports.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/ports.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-available/default.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-available/default.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-available/default.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-available/default.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-available/multi-vhost.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-available/multi-vhost.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-available/multi-vhost.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-available/multi-vhost.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-enabled/default.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-enabled/default.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-enabled/default.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-enabled/default.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-enabled/multi-vhost.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-enabled/multi-vhost.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-enabled/multi-vhost.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multi_vhosts/apache2/sites-enabled/multi-vhost.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/apache2.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/apache2.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/apache2.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/apache2.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/bad_conf_file.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/bad_conf_file.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/bad_conf_file.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/bad_conf_file.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/other-vhosts-access-log.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/other-vhosts-access-log.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/other-vhosts-access-log.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/other-vhosts-access-log.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/security.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/security.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/security.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/security.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/serve-cgi-bin.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/serve-cgi-bin.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/serve-cgi-bin.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-available/serve-cgi-bin.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/other-vhosts-access-log.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/other-vhosts-access-log.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/other-vhosts-access-log.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/other-vhosts-access-log.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/security.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/security.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/security.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/security.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/serve-cgi-bin.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/serve-cgi-bin.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/serve-cgi-bin.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/conf-enabled/serve-cgi-bin.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/envvars b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/envvars similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/envvars rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/envvars diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/authz_svn.load b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/authz_svn.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/authz_svn.load rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/authz_svn.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav.load b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav.load rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav_svn.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav_svn.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav_svn.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav_svn.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav_svn.load b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav_svn.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav_svn.load rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/dav_svn.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/rewrite.load b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/rewrite.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/rewrite.load rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/rewrite.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/ssl.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/ssl.conf similarity index 98% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/ssl.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/ssl.conf index e9fcf4f9b..65baec874 100644 --- a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/mods-available/ssl.conf +++ b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/ssl.conf @@ -31,7 +31,7 @@ # Pass Phrase Dialog: # Configure the pass phrase gathering process. - # The filtering dialog program (`builtin' is a internal + # The filtering dialog program (`builtin' is an internal # terminal dialog) has to provide the pass phrase on stdout. SSLPassPhraseDialog exec:/usr/share/apache2/ask-for-passphrase diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/ssl.load b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/ssl.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/ssl.load rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-available/ssl.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/authz_svn.load b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/authz_svn.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/authz_svn.load rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/authz_svn.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav.load b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav.load rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav_svn.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav_svn.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav_svn.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav_svn.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav_svn.load b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav_svn.load similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav_svn.load rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/mods-enabled/dav_svn.load diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/ports.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/ports.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/ports.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/ports.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/000-default.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/000-default.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/000-default.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/000-default.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/certbot.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/certbot.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/certbot.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/certbot.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/default-ssl-port-only.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/default-ssl-port-only.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/default-ssl-port-only.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/default-ssl-port-only.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/default-ssl.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/default-ssl.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/default-ssl.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/default-ssl.conf diff --git a/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/duplicatehttp.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/duplicatehttp.conf new file mode 100644 index 000000000..5684651fb --- /dev/null +++ b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/duplicatehttp.conf @@ -0,0 +1,9 @@ + + ServerName duplicate.example.com + + ServerAdmin webmaster@certbot.demo + DocumentRoot /var/www/html + + ErrorLog ${APACHE_LOG_DIR}/error.log + CustomLog ${APACHE_LOG_DIR}/access.log combined + diff --git a/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/duplicatehttps.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/duplicatehttps.conf new file mode 100644 index 000000000..e3ac21fac --- /dev/null +++ b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/duplicatehttps.conf @@ -0,0 +1,14 @@ + + + ServerName duplicate.example.com + + ServerAdmin webmaster@certbot.demo + DocumentRoot /var/www/html + + ErrorLog ${APACHE_LOG_DIR}/error.log + CustomLog ${APACHE_LOG_DIR}/access.log combined + +SSLCertificateFile /etc/apache2/certs/certbot-cert_5.pem +SSLCertificateKeyFile /etc/apache2/ssl/key-certbot_15.pem + + diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/encryption-example.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/encryption-example.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/encryption-example.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/encryption-example.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/mod_macro-example.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/mod_macro-example.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/mod_macro-example.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/mod_macro-example.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/ocsp-ssl.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/ocsp-ssl.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/ocsp-ssl.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/ocsp-ssl.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/wildcard.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/wildcard.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/wildcard.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/wildcard.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/000-default.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/000-default.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/000-default.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/000-default.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/certbot.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/certbot.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/certbot.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/certbot.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/default-ssl-port-only.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/default-ssl-port-only.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/default-ssl-port-only.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/default-ssl-port-only.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/default-ssl.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/default-ssl.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/default-ssl.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/default-ssl.conf diff --git a/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/duplicatehttp.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/duplicatehttp.conf new file mode 120000 index 000000000..a69ee3c1d --- /dev/null +++ b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/duplicatehttp.conf @@ -0,0 +1 @@ +../sites-available/duplicatehttp.conf \ No newline at end of file diff --git a/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/duplicatehttps.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/duplicatehttps.conf new file mode 120000 index 000000000..a52ee1ccb --- /dev/null +++ b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/duplicatehttps.conf @@ -0,0 +1 @@ +../sites-available/duplicatehttps.conf \ No newline at end of file diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/encryption-example.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/encryption-example.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/encryption-example.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/encryption-example.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/mod_macro-example.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/mod_macro-example.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/mod_macro-example.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/mod_macro-example.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/non-symlink.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/non-symlink.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/non-symlink.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/non-symlink.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/ocsp-ssl.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/ocsp-ssl.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/ocsp-ssl.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/ocsp-ssl.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/wildcard.conf b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/wildcard.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/wildcard.conf rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/wildcard.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/sites b/certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/sites similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/sites rename to certbot-apache/tests/testdata/debian_apache_2_4/multiple_vhosts/sites diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/httpd.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/httpd.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/httpd.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/httpd.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/magic b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/magic similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/magic rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/magic diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_default_settings.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_default_settings.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_default_settings.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_default_settings.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_error_documents.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_error_documents.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_error_documents.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_error_documents.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_languages.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_languages.conf similarity index 99% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_languages.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_languages.conf index c429bf94c..10cf3fb54 100644 --- a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_languages.conf +++ b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_languages.conf @@ -33,7 +33,7 @@ # English (en) - Esperanto (eo) - Estonian (et) - French (fr) - German (de) # Greek-Modern (el) - Hebrew (he) - Italian (it) - Japanese (ja) # Korean (ko) - Luxembourgeois* (ltz) - Norwegian Nynorsk (nn) -# Norwegian (no) - Polish (pl) - Portugese (pt) +# Norwegian (no) - Polish (pl) - Portuguese (pt) # Brazilian Portuguese (pt-BR) - Russian (ru) - Swedish (sv) # Simplified Chinese (zh-CN) - Spanish (es) - Traditional Chinese (zh-TW) AddLanguage ca .ca diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_autoindex.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_autoindex.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_autoindex.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_autoindex.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_info.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_info.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_info.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_info.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_log_config.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_log_config.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_log_config.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_log_config.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_mime.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_mime.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_mime.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_mime.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_status.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_status.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_status.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_status.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_userdir.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_userdir.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_userdir.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mod_userdir.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mpm.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mpm.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mpm.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/00_mpm.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/10_mod_mem_cache.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/10_mod_mem_cache.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/10_mod_mem_cache.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/10_mod_mem_cache.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/40_mod_ssl.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/40_mod_ssl.conf similarity index 97% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/40_mod_ssl.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/40_mod_ssl.conf index f51de4641..7f3cef423 100644 --- a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/40_mod_ssl.conf +++ b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/40_mod_ssl.conf @@ -43,7 +43,7 @@ SSLRandomSeed connect builtin ## Pass Phrase Dialog: # Configure the pass phrase gathering process. The filtering dialog program -# (`builtin' is a internal terminal dialog) has to provide the pass phrase on +# (`builtin' is an internal terminal dialog) has to provide the pass phrase on # stdout. SSLPassPhraseDialog builtin diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/41_mod_http2.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/41_mod_http2.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/41_mod_http2.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/41_mod_http2.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/45_mod_dav.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/45_mod_dav.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/45_mod_dav.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/45_mod_dav.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/46_mod_ldap.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/46_mod_ldap.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/46_mod_ldap.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/modules.d/46_mod_ldap.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/00_default_ssl_vhost.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/00_default_ssl_vhost.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/00_default_ssl_vhost.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/00_default_ssl_vhost.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/00_default_vhost.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/00_default_vhost.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/00_default_vhost.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/00_default_vhost.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/default_vhost.include b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/default_vhost.include similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/default_vhost.include rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/default_vhost.include diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/gentoo.example.com.conf b/certbot-apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/gentoo.example.com.conf similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/gentoo.example.com.conf rename to certbot-apache/tests/testdata/gentoo_apache/apache/apache2/vhosts.d/gentoo.example.com.conf diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/conf.d/apache2 b/certbot-apache/tests/testdata/gentoo_apache/apache/conf.d/apache2 similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/conf.d/apache2 rename to certbot-apache/tests/testdata/gentoo_apache/apache/conf.d/apache2 diff --git a/certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/sites b/certbot-apache/tests/testdata/gentoo_apache/apache/sites similarity index 100% rename from certbot-apache/certbot_apache/tests/testdata/gentoo_apache/apache/sites rename to certbot-apache/tests/testdata/gentoo_apache/apache/sites diff --git a/certbot-apache/certbot_apache/tests/util.py b/certbot-apache/tests/util.py similarity index 87% rename from certbot-apache/certbot_apache/tests/util.py rename to certbot-apache/tests/util.py index 9329ccb20..57b20dc9d 100644 --- a/certbot-apache/certbot_apache/tests/util.py +++ b/certbot-apache/tests/util.py @@ -1,5 +1,4 @@ """Common utilities for certbot_apache.""" -import os import shutil import sys import unittest @@ -9,18 +8,16 @@ import josepy as jose import mock import zope.component +from certbot.compat import os from certbot.display import util as display_util - from certbot.plugins import common - from certbot.tests import util as test_util - -from certbot_apache import configurator -from certbot_apache import entrypoint -from certbot_apache import obj +from certbot_apache._internal import configurator +from certbot_apache._internal import entrypoint +from certbot_apache._internal import obj -class ApacheTest(unittest.TestCase): # pylint: disable=too-few-public-methods +class ApacheTest(unittest.TestCase): def setUp(self, test_dir="debian_apache_2_4/multiple_vhosts", config_root="debian_apache_2_4/multiple_vhosts/apache2", @@ -30,7 +27,7 @@ class ApacheTest(unittest.TestCase): # pylint: disable=too-few-public-methods self.temp_dir, self.config_dir, self.work_dir = common.dir_setup( test_dir=test_dir, - pkg="certbot_apache.tests") + pkg=__name__) self.config_path = os.path.join(self.temp_dir, config_root) self.vhost_path = os.path.join(self.temp_dir, vhost_root) @@ -74,17 +71,16 @@ class ParserTest(ApacheTest): zope.component.provideUtility(display_util.FileDisplay(sys.stdout, False)) - from certbot_apache.parser import ApacheParser + from certbot_apache._internal.parser import ApacheParser self.aug = augeas.Augeas( flags=augeas.Augeas.NONE | augeas.Augeas.NO_MODL_AUTOLOAD) - with mock.patch("certbot_apache.parser.ApacheParser." + with mock.patch("certbot_apache._internal.parser.ApacheParser." "update_runtime_variables"): self.parser = ApacheParser( - self.aug, self.config_path, self.vhost_path, - configurator=self.config) + self.config_path, self.vhost_path, configurator=self.config) -def get_apache_configurator( # pylint: disable=too-many-arguments, too-many-locals +def get_apache_configurator( config_path, vhost_path, config_dir, work_dir, version=(2, 4, 7), os_info="generic", @@ -108,11 +104,11 @@ def get_apache_configurator( # pylint: disable=too-many-arguments, too-many-loc in_progress_dir=os.path.join(backups, "IN_PROGRESS"), work_dir=work_dir) - with mock.patch("certbot_apache.configurator.util.run_script"): - with mock.patch("certbot_apache.configurator.util." + with mock.patch("certbot_apache._internal.configurator.util.run_script"): + with mock.patch("certbot_apache._internal.configurator.util." "exe_exists") as mock_exe_exists: mock_exe_exists.return_value = True - with mock.patch("certbot_apache.parser.ApacheParser." + with mock.patch("certbot_apache._internal.parser.ApacheParser." "update_runtime_variables"): try: config_class = entrypoint.OVERRIDE_CLASSES[os_info] @@ -196,7 +192,17 @@ def get_vh_truth(temp_dir, config_name): "/files" + os.path.join(temp_dir, config_name, "apache2/apache2.conf/VirtualHost"), set([obj.Addr.fromstring("*:80")]), False, True, - "vhost.in.rootconf")] + "vhost.in.rootconf"), + obj.VirtualHost( + os.path.join(prefix, "duplicatehttp.conf"), + os.path.join(aug_pre, "duplicatehttp.conf/VirtualHost"), + set([obj.Addr.fromstring("10.2.3.4:80")]), False, True, + "duplicate.example.com"), + obj.VirtualHost( + os.path.join(prefix, "duplicatehttps.conf"), + os.path.join(aug_pre, "duplicatehttps.conf/IfModule/VirtualHost"), + set([obj.Addr.fromstring("10.2.3.4:443")]), True, True, + "duplicate.example.com")] return vh_truth if config_name == "debian_apache_2_4/multi_vhosts": prefix = os.path.join( diff --git a/certbot-auto b/certbot-auto index a79a9c5ae..cea58e2cb 100755 --- a/certbot-auto +++ b/certbot-auto @@ -31,7 +31,7 @@ if [ -z "$VENV_PATH" ]; then fi VENV_BIN="$VENV_PATH/bin" BOOTSTRAP_VERSION_PATH="$VENV_PATH/certbot-auto-bootstrap-version.txt" -LE_AUTO_VERSION="0.30.2" +LE_AUTO_VERSION="1.2.0" BASENAME=$(basename $0) USAGE="Usage: $BASENAME [OPTIONS] A self-updating wrapper script for the Certbot ACME client. When run, updates @@ -45,6 +45,7 @@ Help for certbot itself cannot be provided until it is installed. -h, --help print this help -n, --non-interactive, --noninteractive run without asking for user input --no-bootstrap do not install OS dependencies + --no-permissions-check do not warn about file system permissions --no-self-upgrade do not download updates --os-packages-only install OS dependencies and exit --install-only install certbot, upgrade if needed, and exit @@ -67,6 +68,8 @@ for arg in "$@" ; do # Do not upgrade this script (also prevents client upgrades, because each # copy of the script pins a hash of the python client) NO_SELF_UPGRADE=1;; + --no-permissions-check) + NO_PERMISSIONS_CHECK=1;; --no-bootstrap) NO_BOOTSTRAP=1;; --help) @@ -172,7 +175,11 @@ SetRootAuthMechanism() { sudo) SUDO="sudo -E" ;; - '') ;; # Nothing to do for plain root method. + '') + # If we're not running with root, don't check that this script can only + # be modified by system users and groups. + NO_PERMISSIONS_CHECK=1 + ;; *) error "Error: unknown root authorization mechanism '$LE_AUTO_SUDO'." exit 1 @@ -249,20 +256,28 @@ DeprecationBootstrap() { fi } -MIN_PYTHON_VERSION="2.7" -MIN_PYVER=$(echo "$MIN_PYTHON_VERSION" | sed 's/\.//') +MIN_PYTHON_2_VERSION="2.7" +MIN_PYVER2=$(echo "$MIN_PYTHON_2_VERSION" | sed 's/\.//') +MIN_PYTHON_3_VERSION="3.5" +MIN_PYVER3=$(echo "$MIN_PYTHON_3_VERSION" | sed 's/\.//') # Sets LE_PYTHON to Python version string and PYVER to the first two -# digits of the python version +# digits of the python version. +# MIN_PYVER and MIN_PYTHON_VERSION are also set by this function, and their +# values depend on if we try to use Python 3 or Python 2. DeterminePythonVersion() { # Arguments: "NOCRASH" if we shouldn't crash if we don't find a good python # # If no Python is found, PYVER is set to 0. if [ "$USE_PYTHON_3" = 1 ]; then + MIN_PYVER=$MIN_PYVER3 + MIN_PYTHON_VERSION=$MIN_PYTHON_3_VERSION for LE_PYTHON in "$LE_PYTHON" python3; do # Break (while keeping the LE_PYTHON value) if found. $EXISTS "$LE_PYTHON" > /dev/null && break done else + MIN_PYVER=$MIN_PYVER2 + MIN_PYTHON_VERSION=$MIN_PYTHON_2_VERSION for LE_PYTHON in "$LE_PYTHON" python2.7 python27 python2 python; do # Break (while keeping the LE_PYTHON value) if found. $EXISTS "$LE_PYTHON" > /dev/null && break @@ -278,7 +293,7 @@ DeterminePythonVersion() { fi fi - PYVER=`"$LE_PYTHON" -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//'` + PYVER=$("$LE_PYTHON" -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') if [ "$PYVER" -lt "$MIN_PYVER" ]; then if [ "$1" != "NOCRASH" ]; then error "You have an ancient version of Python entombed in your operating system..." @@ -333,63 +348,11 @@ BootstrapDebCommon() { fi augeas_pkg="libaugeas0 augeas-lenses" - AUGVERSION=`LC_ALL=C apt-cache show --no-all-versions libaugeas0 | grep ^Version: | cut -d" " -f2` if [ "$ASSUME_YES" = 1 ]; then YES_FLAG="-y" fi - AddBackportRepo() { - # ARGS: - BACKPORT_NAME="$1" - BACKPORT_SOURCELINE="$2" - say "To use the Apache Certbot plugin, augeas needs to be installed from $BACKPORT_NAME." - if ! grep -v -e ' *#' /etc/apt/sources.list | grep -q "$BACKPORT_NAME" ; then - # This can theoretically error if sources.list.d is empty, but in that case we don't care. - if ! grep -v -e ' *#' /etc/apt/sources.list.d/* 2>/dev/null | grep -q "$BACKPORT_NAME"; then - if [ "$ASSUME_YES" = 1 ]; then - /bin/echo -n "Installing augeas from $BACKPORT_NAME in 3 seconds..." - sleep 1s - /bin/echo -ne "\e[0K\rInstalling augeas from $BACKPORT_NAME in 2 seconds..." - sleep 1s - /bin/echo -e "\e[0K\rInstalling augeas from $BACKPORT_NAME in 1 second ..." - sleep 1s - add_backports=1 - else - read -p "Would you like to enable the $BACKPORT_NAME repository [Y/n]? " response - case $response in - [yY][eE][sS]|[yY]|"") - add_backports=1;; - *) - add_backports=0;; - esac - fi - if [ "$add_backports" = 1 ]; then - sh -c "echo $BACKPORT_SOURCELINE >> /etc/apt/sources.list.d/$BACKPORT_NAME.list" - apt-get $QUIET_FLAG update - fi - fi - fi - if [ "$add_backports" != 0 ]; then - apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends -t "$BACKPORT_NAME" $augeas_pkg - augeas_pkg= - fi - } - - - if dpkg --compare-versions 1.0 gt "$AUGVERSION" ; then - if lsb_release -a | grep -q wheezy ; then - AddBackportRepo wheezy-backports "deb http://http.debian.net/debian wheezy-backports main" - elif lsb_release -a | grep -q precise ; then - # XXX add ARM case - AddBackportRepo precise-backports "deb http://archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse" - else - echo "No libaugeas0 version is available that's new enough to run the" - echo "Certbot apache plugin..." - fi - # XXX add a case for ubuntu PPAs - fi - apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends \ python \ python-dev \ @@ -413,7 +376,9 @@ BootstrapDebCommon() { # Sets TOOL to the name of the package manager # Sets appropriate values for YES_FLAG and QUIET_FLAG based on $ASSUME_YES and $QUIET_FLAG. -# Enables EPEL if applicable and possible. +# Note: this function is called both while selecting the bootstrap scripts and +# during the actual bootstrap. Some things like prompting to user can be done in the latter +# case, but not in the former one. InitializeRPMCommonBase() { if type dnf 2>/dev/null then @@ -433,26 +398,6 @@ InitializeRPMCommonBase() { if [ "$QUIET" = 1 ]; then QUIET_FLAG='--quiet' fi - - if ! $TOOL list *virtualenv >/dev/null 2>&1; then - echo "To use Certbot, packages from the EPEL repository need to be installed." - if ! $TOOL list epel-release >/dev/null 2>&1; then - error "Enable the EPEL repository and try running Certbot again." - exit 1 - fi - if [ "$ASSUME_YES" = 1 ]; then - /bin/echo -n "Enabling the EPEL repository in 3 seconds..." - sleep 1s - /bin/echo -ne "\e[0K\rEnabling the EPEL repository in 2 seconds..." - sleep 1s - /bin/echo -e "\e[0K\rEnabling the EPEL repository in 1 second..." - sleep 1s - fi - if ! $TOOL install $YES_FLAG $QUIET_FLAG epel-release; then - error "Could not enable EPEL. Aborting bootstrap!" - exit 1 - fi - fi } BootstrapRpmCommonBase() { @@ -535,19 +480,98 @@ BootstrapRpmCommon() { # If new packages are installed by BootstrapRpmPython3 below, this version # number must be increased. -BOOTSTRAP_RPM_PYTHON3_VERSION=1 +BOOTSTRAP_RPM_PYTHON3_LEGACY_VERSION=1 -BootstrapRpmPython3() { +# Checks if rh-python36 can be installed. +Python36SclIsAvailable() { + InitializeRPMCommonBase >/dev/null 2>&1; + + if "${TOOL}" list rh-python36 >/dev/null 2>&1; then + return 0 + fi + if "${TOOL}" list centos-release-scl >/dev/null 2>&1; then + return 0 + fi + return 1 +} + +# Try to enable rh-python36 from SCL if it is necessary and possible. +EnablePython36SCL() { + if "$EXISTS" python3.6 > /dev/null 2> /dev/null; then + return 0 + fi + if [ ! -f /opt/rh/rh-python36/enable ]; then + return 0 + fi + set +e + if ! . /opt/rh/rh-python36/enable; then + error 'Unable to enable rh-python36!' + exit 1 + fi + set -e +} + +# This bootstrap concerns old RedHat-based distributions that do not ship by default +# with Python 2.7, but only Python 2.6. We bootstrap them by enabling SCL and installing +# Python 3.6. Some of these distributions are: CentOS/RHEL/OL/SL 6. +BootstrapRpmPython3Legacy() { # Tested with: # - CentOS 6 InitializeRPMCommonBase - # EPEL uses python34 - if $TOOL list python34 >/dev/null 2>&1; then - python_pkgs="python34 - python34-devel - python34-tools + if ! "${TOOL}" list rh-python36 >/dev/null 2>&1; then + echo "To use Certbot on this operating system, packages from the SCL repository need to be installed." + if ! "${TOOL}" list centos-release-scl >/dev/null 2>&1; then + error "Enable the SCL repository and try running Certbot again." + exit 1 + fi + if [ "${ASSUME_YES}" = 1 ]; then + /bin/echo -n "Enabling the SCL repository in 3 seconds... (Press Ctrl-C to cancel)" + sleep 1s + /bin/echo -ne "\e[0K\rEnabling the SCL repository in 2 seconds... (Press Ctrl-C to cancel)" + sleep 1s + /bin/echo -e "\e[0K\rEnabling the SCL repository in 1 second... (Press Ctrl-C to cancel)" + sleep 1s + fi + if ! "${TOOL}" install "${YES_FLAG}" "${QUIET_FLAG}" centos-release-scl; then + error "Could not enable SCL. Aborting bootstrap!" + exit 1 + fi + fi + + # CentOS 6 must use rh-python36 from SCL + if "${TOOL}" list rh-python36 >/dev/null 2>&1; then + python_pkgs="rh-python36-python + rh-python36-python-virtualenv + rh-python36-python-devel + " + else + error "No supported Python package available to install. Aborting bootstrap!" + exit 1 + fi + + BootstrapRpmCommonBase "${python_pkgs}" + + # Enable SCL rh-python36 after bootstrapping. + EnablePython36SCL +} + +# If new packages are installed by BootstrapRpmPython3 below, this version +# number must be increased. +BOOTSTRAP_RPM_PYTHON3_VERSION=1 + +BootstrapRpmPython3() { + # Tested with: + # - Fedora 29 + + InitializeRPMCommonBase + + # Fedora 29 must use python3-virtualenv + if $TOOL list python3-virtualenv >/dev/null 2>&1; then + python_pkgs="python3 + python3-virtualenv + python3-devel " else error "No supported Python package available to install. Aborting bootstrap!" @@ -573,10 +597,20 @@ BootstrapSuseCommon() { QUIET_FLAG='-qq' fi + if zypper search -x python-virtualenv >/dev/null 2>&1; then + OPENSUSE_VIRTUALENV_PACKAGES="python-virtualenv" + else + # Since Leap 15.0 (and associated Tumbleweed version), python-virtualenv + # is a source package, and python2-virtualenv must be used instead. + # Also currently python2-setuptools is not a dependency of python2-virtualenv, + # while it should be. Installing it explicitly until upstream fix. + OPENSUSE_VIRTUALENV_PACKAGES="python2-virtualenv python2-setuptools" + fi + zypper $QUIET_FLAG $zypper_flags in $install_flags \ python \ python-devel \ - python-virtualenv \ + $OPENSUSE_VIRTUALENV_PACKAGES \ gcc \ augeas-lenses \ libopenssl-devel \ @@ -783,20 +817,71 @@ elif [ -f /etc/redhat-release ]; then prev_le_python="$LE_PYTHON" unset LE_PYTHON DeterminePythonVersion "NOCRASH" + + RPM_DIST_NAME=`(. /etc/os-release 2> /dev/null && echo $ID) || echo "unknown"` + + if [ "$PYVER" -eq 26 -a $(uname -m) != 'x86_64' ]; then + # 32 bits CentOS 6 and affiliates are not supported anymore by certbot-auto. + DEPRECATED_OS=1 + fi + + # Set RPM_DIST_VERSION to VERSION_ID from /etc/os-release after splitting on + # '.' characters (e.g. "8.0" becomes "8"). If the command exits with an + # error, RPM_DIST_VERSION is set to "unknown". + RPM_DIST_VERSION=$( (. /etc/os-release 2> /dev/null && echo "$VERSION_ID") | cut -d '.' -f1 || echo "unknown") + + # If RPM_DIST_VERSION is an empty string or it contains any nonnumeric + # characters, the value is unexpected so we set RPM_DIST_VERSION to 0. + if [ -z "$RPM_DIST_VERSION" ] || [ -n "$(echo "$RPM_DIST_VERSION" | tr -d '[0-9]')" ]; then + RPM_DIST_VERSION=0 + fi + + # Handle legacy RPM distributions if [ "$PYVER" -eq 26 ]; then + # Check if an automated bootstrap can be achieved on this system. + if ! Python36SclIsAvailable; then + INTERACTIVE_BOOTSTRAP=1 + fi + Bootstrap() { - BootstrapMessage "RedHat-based OSes that will use Python3" - BootstrapRpmPython3 + BootstrapMessage "Legacy RedHat-based OSes that will use Python3" + BootstrapRpmPython3Legacy } USE_PYTHON_3=1 - BOOTSTRAP_VERSION="BootstrapRpmPython3 $BOOTSTRAP_RPM_PYTHON3_VERSION" + BOOTSTRAP_VERSION="BootstrapRpmPython3Legacy $BOOTSTRAP_RPM_PYTHON3_LEGACY_VERSION" + + # Try now to enable SCL rh-python36 for systems already bootstrapped + # NB: EnablePython36SCL has been defined along with BootstrapRpmPython3Legacy in certbot-auto + EnablePython36SCL else - Bootstrap() { - BootstrapMessage "RedHat-based OSes" - BootstrapRpmCommon - } - BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION" + # Starting to Fedora 29, python2 is on a deprecation path. Let's move to python3 then. + # RHEL 8 also uses python3 by default. + if [ "$RPM_DIST_NAME" = "fedora" -a "$RPM_DIST_VERSION" -ge 29 ]; then + RPM_USE_PYTHON_3=1 + elif [ "$RPM_DIST_NAME" = "rhel" -a "$RPM_DIST_VERSION" -ge 8 ]; then + RPM_USE_PYTHON_3=1 + elif [ "$RPM_DIST_NAME" = "centos" -a "$RPM_DIST_VERSION" -ge 8 ]; then + RPM_USE_PYTHON_3=1 + else + RPM_USE_PYTHON_3=0 + fi + + if [ "$RPM_USE_PYTHON_3" = 1 ]; then + Bootstrap() { + BootstrapMessage "RedHat-based OSes that will use Python3" + BootstrapRpmPython3 + } + USE_PYTHON_3=1 + BOOTSTRAP_VERSION="BootstrapRpmPython3 $BOOTSTRAP_RPM_PYTHON3_VERSION" + else + Bootstrap() { + BootstrapMessage "RedHat-based OSes" + BootstrapRpmCommon + } + BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION" + fi fi + LE_PYTHON="$prev_le_python" elif [ -f /etc/os-release ] && `grep -q openSUSE /etc/os-release` ; then Bootstrap() { @@ -871,6 +956,13 @@ if [ "$NO_BOOTSTRAP" = 1 ]; then unset BOOTSTRAP_VERSION fi +if [ "$DEPRECATED_OS" = 1 ]; then + Bootstrap() { + error "Skipping bootstrap because certbot-auto is deprecated on this system." + } + unset BOOTSTRAP_VERSION +fi + # Sets PREV_BOOTSTRAP_VERSION to the identifier for the bootstrap script used # to install OS dependencies on this system. PREV_BOOTSTRAP_VERSION isn't set # if it is unknown how OS dependencies were installed on this system. @@ -940,10 +1032,156 @@ else: UNLIKELY_EOF } +# Create a new virtual environment for Certbot. It will overwrite any existing one. +# Parameters: LE_PYTHON, VENV_PATH, PYVER, VERBOSE +CreateVenv() { + "$1" - "$2" "$3" "$4" << "UNLIKELY_EOF" +#!/usr/bin/env python +import os +import shutil +import subprocess +import sys + + +def create_venv(venv_path, pyver, verbose): + if os.path.exists(venv_path): + shutil.rmtree(venv_path) + + stdout = sys.stdout if verbose == '1' else open(os.devnull, 'w') + + if int(pyver) <= 27: + # Use virtualenv binary + environ = os.environ.copy() + environ['VIRTUALENV_NO_DOWNLOAD'] = '1' + command = ['virtualenv', '--no-site-packages', '--python', sys.executable, venv_path] + subprocess.check_call(command, stdout=stdout, env=environ) + else: + # Use embedded venv module in Python 3 + command = [sys.executable, '-m', 'venv', venv_path] + subprocess.check_call(command, stdout=stdout) + + +if __name__ == '__main__': + create_venv(*sys.argv[1:]) + +UNLIKELY_EOF +} + +# Check that the given PATH_TO_CHECK has secured permissions. +# Parameters: LE_PYTHON, PATH_TO_CHECK +CheckPathPermissions() { + "$1" - "$2" << "UNLIKELY_EOF" +"""Verifies certbot-auto cannot be modified by unprivileged users. + +This script takes the path to certbot-auto as its only command line +argument. It then checks that the file can only be modified by uid/gid +< 1000 and if other users can modify the file, it prints a warning with +a suggestion on how to solve the problem. + +Permissions on symlinks in the absolute path of certbot-auto are ignored +and only the canonical path to certbot-auto is checked. There could be +permissions problems due to the symlinks that are unreported by this +script, however, issues like this were not caused by our documentation +and are ignored for the sake of simplicity. + +All warnings are printed to stdout rather than stderr so all stderr +output from this script can be suppressed to avoid printing messages if +this script fails for some reason. + +""" +from __future__ import print_function + +import os +import stat +import sys + + +FORUM_POST_URL = 'https://community.letsencrypt.org/t/certbot-auto-deployment-best-practices/91979/' + + +def has_safe_permissions(path): + """Returns True if the given path has secure permissions. + + The permissions are considered safe if the file is only writable by + uid/gid < 1000. + + The reason we allow more IDs than 0 is because on some systems such + as Debian, system users/groups other than uid/gid 0 are used for the + path we recommend in our instructions which is /usr/local/bin. 1000 + was chosen because on Debian 0-999 is reserved for system IDs[1] and + on RHEL either 0-499 or 0-999 is reserved depending on the + version[2][3]. Due to these differences across different OSes, this + detection isn't perfect so we only determine permissions are + insecure when we can be reasonably confident there is a problem + regardless of the underlying OS. + + [1] https://www.debian.org/doc/debian-policy/ch-opersys.html#uid-and-gid-classes + [2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/ch-managing_users_and_groups + [3] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-managing_users_and_groups + + :param str path: filesystem path to check + :returns: True if the path has secure permissions, otherwise, False + :rtype: bool + + """ + # os.stat follows symlinks before obtaining information about a file. + stat_result = os.stat(path) + if stat_result.st_mode & stat.S_IWOTH: + return False + if stat_result.st_mode & stat.S_IWGRP and stat_result.st_gid >= 1000: + return False + if stat_result.st_mode & stat.S_IWUSR and stat_result.st_uid >= 1000: + return False + return True + + +def main(certbot_auto_path): + current_path = os.path.realpath(certbot_auto_path) + last_path = None + permissions_ok = True + # This loop makes use of the fact that os.path.dirname('/') == '/'. + while current_path != last_path and permissions_ok: + permissions_ok = has_safe_permissions(current_path) + last_path = current_path + current_path = os.path.dirname(current_path) + + if not permissions_ok: + print('{0} has insecure permissions!'.format(certbot_auto_path)) + print('To learn how to fix them, visit {0}'.format(FORUM_POST_URL)) + + +if __name__ == '__main__': + main(sys.argv[1]) + +UNLIKELY_EOF +} + if [ "$1" = "--le-auto-phase2" ]; then # Phase 2: Create venv, install LE, and run. shift 1 # the --le-auto-phase2 arg + + if [ "$DEPRECATED_OS" = 1 ]; then + # Phase 2 damage control mode for deprecated OSes. + # In this situation, we bypass any bootstrap or certbot venv setup. + error "Your system is not supported by certbot-auto anymore." + + if [ ! -d "$VENV_PATH" ] && OldVenvExists; then + VENV_BIN="$OLD_VENV_PATH/bin" + fi + + if [ -f "$VENV_BIN/letsencrypt" -a "$INSTALL_ONLY" != 1 ]; then + error "Certbot will no longer receive updates." + error "Please visit https://certbot.eff.org/ to check for other alternatives." + "$VENV_BIN/letsencrypt" "$@" + exit 0 + else + error "Certbot cannot be installed." + error "Please visit https://certbot.eff.org/ to check for other alternatives." + exit 1 + fi + fi + SetPrevBootstrapVersion if [ -z "$PHASE_1_VERSION" -a "$USE_PYTHON_3" = 1 ]; then @@ -955,8 +1193,15 @@ if [ "$1" = "--le-auto-phase2" ]; then # If the selected Bootstrap function isn't a noop and it differs from the # previously used version if [ -n "$BOOTSTRAP_VERSION" -a "$BOOTSTRAP_VERSION" != "$PREV_BOOTSTRAP_VERSION" ]; then - # if non-interactive mode or stdin and stdout are connected to a terminal - if [ \( "$NONINTERACTIVE" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then + # Check if we can rebootstrap without manual user intervention: this requires that + # certbot-auto is in non-interactive mode AND selected bootstrap does not claim to + # require a manual user intervention. + if [ "$NONINTERACTIVE" = 1 -a "$INTERACTIVE_BOOTSTRAP" != 1 ]; then + CAN_REBOOTSTRAP=1 + fi + # Check if rebootstrap can be done non-interactively and current shell is non-interactive + # (true if stdin and stdout are not attached to a terminal). + if [ \( "$CAN_REBOOTSTRAP" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then if [ -d "$VENV_PATH" ]; then rm -rf "$VENV_PATH" fi @@ -967,12 +1212,21 @@ if [ "$1" = "--le-auto-phase2" ]; then ln -s "$VENV_PATH" "$OLD_VENV_PATH" fi RerunWithArgs "$@" + # Otherwise bootstrap needs to be done manually by the user. else - error "Skipping upgrade because new OS dependencies may need to be installed." - error - error "To upgrade to a newer version, please run this script again manually so you can" - error "approve changes or with --non-interactive on the command line to automatically" - error "install any required packages." + # If it is because bootstrapping is interactive, --non-interactive will be of no use. + if [ "$INTERACTIVE_BOOTSTRAP" = 1 ]; then + error "Skipping upgrade because new OS dependencies may need to be installed." + error "This requires manual user intervention: please run this script again manually." + # If this is because of the environment (eg. non interactive shell without + # --non-interactive flag set), help the user in that direction. + else + error "Skipping upgrade because new OS dependencies may need to be installed." + error + error "To upgrade to a newer version, please run this script again manually so you can" + error "approve changes or with --non-interactive on the command line to automatically" + error "install any required packages." + fi # Set INSTALLED_VERSION to be the same so we don't update the venv INSTALLED_VERSION="$LE_AUTO_VERSION" # Continue to use OLD_VENV_PATH if the new venv doesn't exist @@ -995,22 +1249,7 @@ if [ "$1" = "--le-auto-phase2" ]; then if [ "$LE_AUTO_VERSION" != "$INSTALLED_VERSION" ]; then say "Creating virtual environment..." DeterminePythonVersion - rm -rf "$VENV_PATH" - if [ "$PYVER" -le 27 ]; then - # Use an environment variable instead of a flag for compatibility with old versions - if [ "$VERBOSE" = 1 ]; then - VIRTUALENV_NO_DOWNLOAD=1 virtualenv --no-site-packages --python "$LE_PYTHON" "$VENV_PATH" - else - VIRTUALENV_NO_DOWNLOAD=1 virtualenv --no-site-packages --python "$LE_PYTHON" "$VENV_PATH" \ - > /dev/null - fi - else - if [ "$VERBOSE" = 1 ]; then - "$LE_PYTHON" -m venv "$VENV_PATH" - else - "$LE_PYTHON" -m venv "$VENV_PATH" > /dev/null - fi - fi + CreateVenv "$LE_PYTHON" "$VENV_PATH" "$PYVER" "$VERBOSE" if [ -n "$BOOTSTRAP_VERSION" ]; then echo "$BOOTSTRAP_VERSION" > "$BOOTSTRAP_VERSION_PATH" @@ -1024,202 +1263,271 @@ if [ "$1" = "--le-auto-phase2" ]; then # There is no $ interpolation due to quotes on starting heredoc delimiter. # ------------------------------------------------------------------------- cat << "UNLIKELY_EOF" > "$TEMP_DIR/letsencrypt-auto-requirements.txt" -# This is the flattened list of packages certbot-auto installs. To generate -# this, do -# `pip install --no-cache-dir -e acme -e . -e certbot-apache -e certbot-nginx`, -# and then use `hashin` or a more secure method to gather the hashes. - -# Hashin example: +# This is the flattened list of packages certbot-auto installs. +# To generate this, do (with docker and package hashin installed): +# ``` +# letsencrypt-auto-source/rebuild_dependencies.py \ +# letsencrypt-auto-source/pieces/dependency-requirements.txt +# ``` +# If you want to update a single dependency, run commands similar to these: +# ``` # pip install hashin # hashin -r dependency-requirements.txt cryptography==1.5.2 -# sets the new certbot-auto pinned version of cryptography to 1.5.2 - -argparse==1.4.0 \ - --hash=sha256:c31647edb69fd3d465a847ea3157d37bed1f95f19760b11a47aa91c04b666314 \ - --hash=sha256:62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4 - -# This comes before cffi because cffi will otherwise install an unchecked -# version via setup_requires. -pycparser==2.14 \ - --hash=sha256:7959b4a74abdc27b312fed1c21e6caf9309ce0b29ea86b591fd2e99ecdf27f73 \ - --no-binary pycparser - -asn1crypto==0.22.0 \ - --hash=sha256:d232509fefcfcdb9a331f37e9c9dc20441019ad927c7d2176cf18ed5da0ba097 \ - --hash=sha256:cbbadd640d3165ab24b06ef25d1dca09a3441611ac15f6a6b452474fdf0aed1a -cffi==1.11.5 \ - --hash=sha256:1b0493c091a1898f1136e3f4f991a784437fac3673780ff9de3bcf46c80b6b50 \ - --hash=sha256:87f37fe5130574ff76c17cab61e7d2538a16f843bb7bca8ebbc4b12de3078596 \ - --hash=sha256:1553d1e99f035ace1c0544050622b7bc963374a00c467edafac50ad7bd276aef \ - --hash=sha256:151b7eefd035c56b2b2e1eb9963c90c6302dc15fbd8c1c0a83a163ff2c7d7743 \ - --hash=sha256:edabd457cd23a02965166026fd9bfd196f4324fe6032e866d0f3bd0301cd486f \ - --hash=sha256:ba5e697569f84b13640c9e193170e89c13c6244c24400fc57e88724ef610cd31 \ - --hash=sha256:79f9b6f7c46ae1f8ded75f68cf8ad50e5729ed4d590c74840471fc2823457d04 \ - --hash=sha256:b0f7d4a3df8f06cf49f9f121bead236e328074de6449866515cea4907bbc63d6 \ - --hash=sha256:4c91af6e967c2015729d3e69c2e51d92f9898c330d6a851bf8f121236f3defd3 \ - --hash=sha256:7a33145e04d44ce95bcd71e522b478d282ad0eafaf34fe1ec5bbd73e662f22b6 \ - --hash=sha256:95d5251e4b5ca00061f9d9f3d6fe537247e145a8524ae9fd30a2f8fbce993b5b \ - --hash=sha256:b75110fb114fa366b29a027d0c9be3709579602ae111ff61674d28c93606acca \ - --hash=sha256:ae5e35a2c189d397b91034642cb0eab0e346f776ec2eb44a49a459e6615d6e2e \ - --hash=sha256:fdf1c1dc5bafc32bc5d08b054f94d659422b05aba244d6be4ddc1c72d9aa70fb \ - --hash=sha256:9d1d3e63a4afdc29bd76ce6aa9d58c771cd1599fbba8cf5057e7860b203710dd \ - --hash=sha256:be2a9b390f77fd7676d80bc3cdc4f8edb940d8c198ed2d8c0be1319018c778e1 \ - --hash=sha256:ed01918d545a38998bfa5902c7c00e0fee90e957ce036a4000a88e3fe2264917 \ - --hash=sha256:857959354ae3a6fa3da6651b966d13b0a8bed6bbc87a0de7b38a549db1d2a359 \ - --hash=sha256:2ba8a45822b7aee805ab49abfe7eec16b90587f7f26df20c71dd89e45a97076f \ - --hash=sha256:a36c5c154f9d42ec176e6e620cb0dd275744aa1d804786a71ac37dc3661a5e95 \ - --hash=sha256:e55e22ac0a30023426564b1059b035973ec82186ddddbac867078435801c7801 \ - --hash=sha256:3eb6434197633b7748cea30bf0ba9f66727cdce45117a712b29a443943733257 \ - --hash=sha256:ecbb7b01409e9b782df5ded849c178a0aa7c906cf8c5a67368047daab282b184 \ - --hash=sha256:770f3782b31f50b68627e22f91cb182c48c47c02eb405fd689472aa7b7aa16dc \ - --hash=sha256:d5d8555d9bfc3f02385c1c37e9f998e2011f0db4f90e250e5bc0c0a85a813085 \ - --hash=sha256:3c85641778460581c42924384f5e68076d724ceac0f267d66c757f7535069c93 \ - --hash=sha256:ca1bd81f40adc59011f58159e4aa6445fc585a32bb8ac9badf7a2c1aa23822f2 \ - --hash=sha256:3bb6bd7266598f318063e584378b8e27c67de998a43362e8fce664c54ee52d30 \ - --hash=sha256:a6a5cb8809091ec9ac03edde9304b3ad82ad4466333432b16d78ef40e0cce0d5 \ - --hash=sha256:57b2533356cb2d8fac1555815929f7f5f14d68ac77b085d2326b571310f34f6e \ - --hash=sha256:495c5c2d43bf6cebe0178eb3e88f9c4aa48d8934aa6e3cddb865c058da76756b \ - --hash=sha256:e90f17980e6ab0f3c2f3730e56d1fe9bcba1891eeea58966e89d352492cc74f4 -ConfigArgParse==0.12.0 \ - --hash=sha256:28cd7d67669651f2a4518367838c49539457504584a139709b2b8f6c208ef339 \ - --no-binary ConfigArgParse +# ``` +ConfigArgParse==1.0 \ + --hash=sha256:bf378245bc9cdc403a527e5b7406b991680c2a530e7e81af747880b54eb57133 +certifi==2019.11.28 \ + --hash=sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3 \ + --hash=sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f +cffi==1.13.2 \ + --hash=sha256:0b49274afc941c626b605fb59b59c3485c17dc776dc3cc7cc14aca74cc19cc42 \ + --hash=sha256:0e3ea92942cb1168e38c05c1d56b0527ce31f1a370f6117f1d490b8dcd6b3a04 \ + --hash=sha256:135f69aecbf4517d5b3d6429207b2dff49c876be724ac0c8bf8e1ea99df3d7e5 \ + --hash=sha256:19db0cdd6e516f13329cba4903368bff9bb5a9331d3410b1b448daaadc495e54 \ + --hash=sha256:2781e9ad0e9d47173c0093321bb5435a9dfae0ed6a762aabafa13108f5f7b2ba \ + --hash=sha256:291f7c42e21d72144bb1c1b2e825ec60f46d0a7468f5346841860454c7aa8f57 \ + --hash=sha256:2c5e309ec482556397cb21ede0350c5e82f0eb2621de04b2633588d118da4396 \ + --hash=sha256:2e9c80a8c3344a92cb04661115898a9129c074f7ab82011ef4b612f645939f12 \ + --hash=sha256:32a262e2b90ffcfdd97c7a5e24a6012a43c61f1f5a57789ad80af1d26c6acd97 \ + --hash=sha256:3c9fff570f13480b201e9ab69453108f6d98244a7f495e91b6c654a47486ba43 \ + --hash=sha256:415bdc7ca8c1c634a6d7163d43fb0ea885a07e9618a64bda407e04b04333b7db \ + --hash=sha256:42194f54c11abc8583417a7cf4eaff544ce0de8187abaf5d29029c91b1725ad3 \ + --hash=sha256:4424e42199e86b21fc4db83bd76909a6fc2a2aefb352cb5414833c030f6ed71b \ + --hash=sha256:4a43c91840bda5f55249413037b7a9b79c90b1184ed504883b72c4df70778579 \ + --hash=sha256:599a1e8ff057ac530c9ad1778293c665cb81a791421f46922d80a86473c13346 \ + --hash=sha256:5c4fae4e9cdd18c82ba3a134be256e98dc0596af1e7285a3d2602c97dcfa5159 \ + --hash=sha256:5ecfa867dea6fabe2a58f03ac9186ea64da1386af2159196da51c4904e11d652 \ + --hash=sha256:62f2578358d3a92e4ab2d830cd1c2049c9c0d0e6d3c58322993cc341bdeac22e \ + --hash=sha256:6471a82d5abea994e38d2c2abc77164b4f7fbaaf80261cb98394d5793f11b12a \ + --hash=sha256:6d4f18483d040e18546108eb13b1dfa1000a089bcf8529e30346116ea6240506 \ + --hash=sha256:71a608532ab3bd26223c8d841dde43f3516aa5d2bf37b50ac410bb5e99053e8f \ + --hash=sha256:74a1d8c85fb6ff0b30fbfa8ad0ac23cd601a138f7509dc617ebc65ef305bb98d \ + --hash=sha256:7b93a885bb13073afb0aa73ad82059a4c41f4b7d8eb8368980448b52d4c7dc2c \ + --hash=sha256:7d4751da932caaec419d514eaa4215eaf14b612cff66398dd51129ac22680b20 \ + --hash=sha256:7f627141a26b551bdebbc4855c1157feeef18241b4b8366ed22a5c7d672ef858 \ + --hash=sha256:8169cf44dd8f9071b2b9248c35fc35e8677451c52f795daa2bb4643f32a540bc \ + --hash=sha256:aa00d66c0fab27373ae44ae26a66a9e43ff2a678bf63a9c7c1a9a4d61172827a \ + --hash=sha256:ccb032fda0873254380aa2bfad2582aedc2959186cce61e3a17abc1a55ff89c3 \ + --hash=sha256:d754f39e0d1603b5b24a7f8484b22d2904fa551fe865fd0d4c3332f078d20d4e \ + --hash=sha256:d75c461e20e29afc0aee7172a0950157c704ff0dd51613506bd7d82b718e7410 \ + --hash=sha256:dcd65317dd15bc0451f3e01c80da2216a31916bdcffd6221ca1202d96584aa25 \ + --hash=sha256:e570d3ab32e2c2861c4ebe6ffcad6a8abf9347432a37608fe1fbd157b3f0036b \ + --hash=sha256:fd43a88e045cf992ed09fa724b5315b790525f2676883a6ea64e3263bae6549d +chardet==3.0.4 \ + --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \ + --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 configobj==5.0.6 \ - --hash=sha256:a2f5650770e1c87fb335af19a9b7eb73fc05ccf22144eb68db7d00cd2bcb0902 \ - --no-binary configobj -cryptography==2.2.2 \ - --hash=sha256:3f3b65d5a16e6b52fba63dc860b62ca9832f51f1a2ae5083c78b6840275f12dd \ - --hash=sha256:5251e7de0de66810833606439ca65c9b9e45da62196b0c88bfadf27740aac09f \ - --hash=sha256:551a3abfe0c8c6833df4192a63371aa2ff43afd8f570ed345d31f251d78e7e04 \ - --hash=sha256:5cb990056b7cadcca26813311187ad751ea644712022a3976443691168781b6f \ - --hash=sha256:60bda7f12ecb828358be53095fc9c6edda7de8f1ef571f96c00b2363643fa3cd \ - --hash=sha256:64b5c67acc9a7c83fbb4b69166f3105a0ab722d27934fac2cb26456718eec2ba \ - --hash=sha256:6fef51ec447fe9f8351894024e94736862900d3a9aa2961528e602eb65c92bdb \ - --hash=sha256:77d0ad229d47a6e0272d00f6bf8ac06ce14715a9fd02c9a97f5a2869aab3ccb2 \ - --hash=sha256:808fe471b1a6b777f026f7dc7bd9a4959da4bfab64972f2bbe91e22527c1c037 \ - --hash=sha256:9b62fb4d18529c84b961efd9187fecbb48e89aa1a0f9f4161c61b7fc42a101bd \ - --hash=sha256:9e5bed45ec6b4f828866ac6a6bedf08388ffcfa68abe9e94b34bb40977aba531 \ - --hash=sha256:9fc295bf69130a342e7a19a39d7bbeb15c0bcaabc7382ec33ef3b2b7d18d2f63 \ - --hash=sha256:abd070b5849ed64e6d349199bef955ee0ad99aefbad792f0c587f8effa681a5e \ - --hash=sha256:ba6a774749b6e510cffc2fb98535f717e0e5fd91c7c99a61d223293df79ab351 \ - --hash=sha256:c332118647f084c983c6a3e1dba0f3bcb051f69d12baccac68db8d62d177eb8a \ - --hash=sha256:d6f46e862ee36df81e6342c2177ba84e70f722d9dc9c6c394f9f1f434c4a5563 \ - --hash=sha256:db6013746f73bf8edd9c3d1d3f94db635b9422f503db3fc5ef105233d4c011ab \ - --hash=sha256:f57008eaff597c69cf692c3518f6d4800f0309253bb138b526a37fe9ef0c7471 \ - --hash=sha256:f6c821ac253c19f2ad4c8691633ae1d1a17f120d5b01ea1d256d7b602bc59887 -enum34==1.1.2 ; python_version < '3.4' \ - --hash=sha256:2475d7fcddf5951e92ff546972758802de5260bf409319a9f1934e6bbc8b1dc7 \ - --hash=sha256:35907defb0f992b75ab7788f65fedc1cf20ffa22688e0e6f6f12afc06b3ea501 + --hash=sha256:a2f5650770e1c87fb335af19a9b7eb73fc05ccf22144eb68db7d00cd2bcb0902 +cryptography==2.8 \ + --hash=sha256:02079a6addc7b5140ba0825f542c0869ff4df9a69c360e339ecead5baefa843c \ + --hash=sha256:1df22371fbf2004c6f64e927668734070a8953362cd8370ddd336774d6743595 \ + --hash=sha256:369d2346db5934345787451504853ad9d342d7f721ae82d098083e1f49a582ad \ + --hash=sha256:3cda1f0ed8747339bbdf71b9f38ca74c7b592f24f65cdb3ab3765e4b02871651 \ + --hash=sha256:44ff04138935882fef7c686878e1c8fd80a723161ad6a98da31e14b7553170c2 \ + --hash=sha256:4b1030728872c59687badcca1e225a9103440e467c17d6d1730ab3d2d64bfeff \ + --hash=sha256:58363dbd966afb4f89b3b11dfb8ff200058fbc3b947507675c19ceb46104b48d \ + --hash=sha256:6ec280fb24d27e3d97aa731e16207d58bd8ae94ef6eab97249a2afe4ba643d42 \ + --hash=sha256:7270a6c29199adc1297776937a05b59720e8a782531f1f122f2eb8467f9aab4d \ + --hash=sha256:73fd30c57fa2d0a1d7a49c561c40c2f79c7d6c374cc7750e9ac7c99176f6428e \ + --hash=sha256:7f09806ed4fbea8f51585231ba742b58cbcfbfe823ea197d8c89a5e433c7e912 \ + --hash=sha256:90df0cc93e1f8d2fba8365fb59a858f51a11a394d64dbf3ef844f783844cc793 \ + --hash=sha256:971221ed40f058f5662a604bd1ae6e4521d84e6cad0b7b170564cc34169c8f13 \ + --hash=sha256:a518c153a2b5ed6b8cc03f7ae79d5ffad7315ad4569b2d5333a13c38d64bd8d7 \ + --hash=sha256:b0de590a8b0979649ebeef8bb9f54394d3a41f66c5584fff4220901739b6b2f0 \ + --hash=sha256:b43f53f29816ba1db8525f006fa6f49292e9b029554b3eb56a189a70f2a40879 \ + --hash=sha256:d31402aad60ed889c7e57934a03477b572a03af7794fa8fb1780f21ea8f6551f \ + --hash=sha256:de96157ec73458a7f14e3d26f17f8128c959084931e8997b9e655a39c8fde9f9 \ + --hash=sha256:df6b4dca2e11865e6cfbfb708e800efb18370f5a46fd601d3755bc7f85b3a8a2 \ + --hash=sha256:ecadccc7ba52193963c0475ac9f6fa28ac01e01349a2ca48509667ef41ffd2cf \ + --hash=sha256:fb81c17e0ebe3358486cd8cc3ad78adbae58af12fc2bf2bc0bb84e8090fa5ce8 +distro==1.4.0 \ + --hash=sha256:362dde65d846d23baee4b5c058c8586f219b5a54be1cf5fc6ff55c4578392f57 \ + --hash=sha256:eedf82a470ebe7d010f1872c17237c79ab04097948800029994fa458e52fb4b4 +enum34==1.1.6 \ + --hash=sha256:2d81cbbe0e73112bdfe6ef8576f2238f2ba27dd0d55752a776c41d38b7da2850 \ + --hash=sha256:644837f692e5f550741432dd3f223bbb9852018674981b1664e5dc339387588a \ + --hash=sha256:6bd0f6ad48ec2aa117d3d141940d484deccda84d4fcd884f5c3d93c23ecd8c79 \ + --hash=sha256:8ad8c4783bf61ded74527bffb48ed9b54166685e4230386a9ed9b1279e2df5b1 funcsigs==1.0.2 \ --hash=sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca \ --hash=sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50 -idna==2.5 \ - --hash=sha256:cc19709fd6d0cbfed39ea875d29ba6d4e22c0cebc510a76d6302a28385e8bb70 \ - --hash=sha256:3cb5ce08046c4e3a560fc02f138d0ac63e00f8ce5901a56b32ec8b7994082aab -ipaddress==1.0.16 \ - --hash=sha256:935712800ce4760701d89ad677666cd52691fd2f6f0b340c8b4239a3c17988a5 \ - --hash=sha256:5a3182b322a706525c46282ca6f064d27a02cffbd449f9f47416f1dc96aa71b0 -josepy==1.1.0 \ - --hash=sha256:1309a25aac3caeff5239729c58ff9b583f7d022ffdb1553406ddfc8e5b52b76e \ - --hash=sha256:fb5c62c77d26e04df29cb5ecd01b9ce69b6fcc9e521eb1ca193b7faa2afa7086 -linecache2==1.0.0 \ - --hash=sha256:e78be9c0a0dfcbac712fe04fbf92b96cddae80b1b842f24248214c8496f006ef \ - --hash=sha256:4b26ff4e7110db76eeb6f5a7b64a82623839d595c2038eeda662f2a2db78e97c -# Using an older version of mock here prevents regressions of #5276. +idna==2.8 \ + --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 \ + --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c +ipaddress==1.0.23 \ + --hash=sha256:6e0f4a39e66cb5bb9a137b00276a2eff74f93b71dcbdad6f10ff7df9d3557fcc \ + --hash=sha256:b7f8e0369580bb4a24d5ba1d7cc29660a4a6987763faf1d8a8046830e020e7e2 +josepy==1.2.0 \ + --hash=sha256:8ea15573203f28653c00f4ac0142520777b1c59d9eddd8da3f256c6ba3cac916 \ + --hash=sha256:9cec9a839fe9520f0420e4f38e7219525daccce4813296627436fe444cd002d3 mock==1.3.0 \ - --hash=sha256:3f573a18be94de886d1191f27c168427ef693e8dcfcecf95b170577b2eb69cbb \ - --hash=sha256:1e247dbecc6ce057299eb7ee019ad68314bb93152e81d9a6110d35f4d5eca0f6 -ordereddict==1.1 \ - --hash=sha256:1c35b4ac206cef2d24816c89f89cf289dd3d38cf7c449bb3fab7bf6d43f01b1f \ - --no-binary ordereddict -packaging==16.8 \ - --hash=sha256:99276dc6e3a7851f32027a68f1095cd3f77c148091b092ea867a351811cfe388 \ - --hash=sha256:5d50835fdf0a7edf0b55e311b7c887786504efea1177abd7e69329a8e5ea619e -parsedatetime==2.1 \ - --hash=sha256:ce9d422165cf6e963905cd5f74f274ebf7cc98c941916169178ef93f0e557838 \ - --hash=sha256:17c578775520c99131634e09cfca5a05ea9e1bd2a05cd06967ebece10df7af2d -pbr==1.8.1 \ - --hash=sha256:46c8db75ae75a056bd1cc07fa21734fe2e603d11a07833ecc1eeb74c35c72e0c \ - --hash=sha256:e2127626a91e6c885db89668976db31020f0af2da728924b56480fc7ccf09649 -pyOpenSSL==16.2.0 \ - --hash=sha256:26ca380ddf272f7556e48064bbcd5bd71f83dfc144f3583501c7ddbd9434ee17 \ - --hash=sha256:7779a3bbb74e79db234af6a08775568c6769b5821faecf6e2f4143edb227516e -pyparsing==2.1.8 \ - --hash=sha256:2f0f5ceb14eccd5aef809d6382e87df22ca1da583c79f6db01675ce7d7f49c18 \ - --hash=sha256:03a4869b9f3493807ee1f1cb405e6d576a1a2ca4d81a982677c0c1ad6177c56b \ - --hash=sha256:ab09aee814c0241ff0c503cff30018219fe1fc14501d89f406f4664a0ec9fbcd \ - --hash=sha256:6e9a7f052f8e26bcf749e4033e3115b6dc7e3c85aafcb794b9a88c9d9ef13c97 \ - --hash=sha256:9f463a6bcc4eeb6c08f1ed84439b17818e2085937c0dee0d7674ac127c67c12b \ - --hash=sha256:3626b4d81cfb300dad57f52f2f791caaf7b06c09b368c0aa7b868e53a5775424 \ - --hash=sha256:367b90cc877b46af56d4580cd0ae278062903f02b8204ab631f5a2c0f50adfd0 \ - --hash=sha256:9f1ea360086cd68681e7f4ca8f1f38df47bf81942a0d76a9673c2d23eff35b13 -pyRFC3339==1.0 \ - --hash=sha256:eea31835c56e2096af4363a5745a784878a61d043e247d3a6d6a0a32a9741f56 \ - --hash=sha256:8dfbc6c458b8daba1c0f3620a8c78008b323a268b27b7359e92a4ae41325f535 + --hash=sha256:1e247dbecc6ce057299eb7ee019ad68314bb93152e81d9a6110d35f4d5eca0f6 \ + --hash=sha256:3f573a18be94de886d1191f27c168427ef693e8dcfcecf95b170577b2eb69cbb +parsedatetime==2.5 \ + --hash=sha256:3b835fc54e472c17ef447be37458b400e3fefdf14bb1ffdedb5d2c853acf4ba1 \ + --hash=sha256:d2e9ddb1e463de871d32088a3f3cea3dc8282b1b2800e081bd0ef86900451667 +pbr==5.4.4 \ + --hash=sha256:139d2625547dbfa5fb0b81daebb39601c478c21956dc57e2e07b74450a8c506b \ + --hash=sha256:61aa52a0f18b71c5cc58232d2cf8f8d09cd67fcad60b742a60124cb8d6951488 +pyOpenSSL==19.1.0 \ + --hash=sha256:621880965a720b8ece2f1b2f54ea2071966ab00e2970ad2ce11d596102063504 \ + --hash=sha256:9a24494b2602aaf402be5c9e30a0b82d4a5c67528fe8fb475e3f3bc00dd69507 +pyRFC3339==1.1 \ + --hash=sha256:67196cb83b470709c580bb4738b83165e67c6cc60e1f2e4f286cfcb402a926f4 \ + --hash=sha256:81b8cbe1519cdb79bed04910dd6fa4e181faf8c88dff1e1b987b5f7ab23a5b1a +pycparser==2.19 \ + --hash=sha256:a988718abfad80b6b157acce7bf130a30876d27603738ac39f140993246b25b3 +pyparsing==2.4.6 \ + --hash=sha256:4c830582a84fb022400b85429791bc551f1f4871c33f23e44f353119e92f969f \ + --hash=sha256:c342dccb5250c08d45fd6f8b4a559613ca603b57498511740e65cd11a2e7dcec python-augeas==0.5.0 \ - --hash=sha256:67d59d66cdba8d624e0389b87b2a83a176f21f16a87553b50f5703b23f29bac2 \ - --no-binary python-augeas -pytz==2015.7 \ - --hash=sha256:3abe6a6d3fc2fbbe4c60144211f45da2edbe3182a6f6511af6bbba0598b1f992 \ - --hash=sha256:939ef9c1e1224d980405689a97ffcf7828c56d1517b31d73464356c1f2b7769e \ - --hash=sha256:ead4aefa7007249e05e51b01095719d5a8dd95760089f5730aac5698b1932918 \ - --hash=sha256:3cca0df08bd0ed98432390494ce3ded003f5e661aa460be7a734bffe35983605 \ - --hash=sha256:3ede470d3d17ba3c07638dfa0d10452bc1b6e5ad326127a65ba77e6aaeb11bec \ - --hash=sha256:68c47964f7186eec306b13629627722b9079cd4447ed9e5ecaecd4eac84ca734 \ - --hash=sha256:dd5d3991950aae40a6c81de1578942e73d629808cefc51d12cd157980e6cfc18 \ - --hash=sha256:a77c52062c07eb7c7b30545dbc73e32995b7e117eea750317b5cb5c7a4618f14 \ - --hash=sha256:81af9aec4bc960a9a0127c488f18772dae4634689233f06f65443e7b11ebeb51 \ - --hash=sha256:e079b1dadc5c06246cc1bb6fe1b23a50b1d1173f2edd5104efd40bb73a28f406 \ - --hash=sha256:fbd26746772c24cb93c8b97cbdad5cb9e46c86bbdb1b9d8a743ee00e2fb1fc5d \ - --hash=sha256:99266ef30a37e43932deec2b7ca73e83c8dbc3b9ff703ec73eca6b1dae6befea \ - --hash=sha256:8b6ce1c993909783bc96e0b4f34ea223bff7a4df2c90bdb9c4e0f1ac928689e3 -requests==2.20.0 \ - --hash=sha256:99dcfdaaeb17caf6e526f32b6a7b780461512ab3f1d992187801694cba42770c \ - --hash=sha256:a84b8c9ab6239b578f22d1c21d51b696dcfe004032bb80ea832398d6909d7279 -six==1.10.0 \ - --hash=sha256:0ff78c403d9bccf5a425a6d31a12aa6b47f1c21ca4dc2573a7e2f32a97335eb1 \ - --hash=sha256:105f8d68616f8248e24bf0e9372ef04d3cc10104f1980f54d57b2ce73a5ad56a -traceback2==1.4.0 \ - --hash=sha256:8253cebec4b19094d67cc5ed5af99bf1dba1285292226e98a31929f87a5d6b23 \ - --hash=sha256:05acc67a09980c2ecfedd3423f7ae0104839eccb55fc645773e1caa0951c3030 -unittest2==1.1.0 \ - --hash=sha256:13f77d0875db6d9b435e1d4f41e74ad4cc2eb6e1d5c824996092b3430f088bb8 \ - --hash=sha256:22882a0e418c284e1f718a822b3b022944d53d2d908e1690b319a9d3eb2c0579 -zope.component==4.2.2 \ - --hash=sha256:282c112b55dd8e3c869a3571f86767c150ab1284a9ace2bdec226c592acaf81a \ - --no-binary zope.component -zope.event==4.1.0 \ - --hash=sha256:dc7a59a2fd91730d3793131a5d261b29e93ec4e2a97f1bc487ce8defee2fe786 \ - --no-binary zope.event -zope.interface==4.1.3 \ - --hash=sha256:f07b631f7a601cd8cbd3332d54f43142c7088a83299f859356f08d1d4d4259b3 \ - --hash=sha256:de5cca083b9439d8002fb76bbe6b4998c5a5a721fab25b84298967f002df4c94 \ - --hash=sha256:6788416f7ea7f5b8a97be94825377aa25e8bdc73463e07baaf9858b29e737077 \ - --hash=sha256:6f3230f7254518201e5a3708cbb2de98c848304f06e3ded8bfb39e5825cba2e1 \ - --hash=sha256:5fa575a5240f04200c3088427d0d4b7b737f6e9018818a51d8d0f927a6a2517a \ - --hash=sha256:522194ad6a545735edd75c8a83f48d65d1af064e432a7d320d64f56bafc12e99 \ - --hash=sha256:e8c7b2d40943f71c99148c97f66caa7f5134147f57423f8db5b4825099ce9a09 \ - --hash=sha256:279024f0208601c3caa907c53876e37ad88625f7eaf1cb3842dbe360b2287017 \ - --hash=sha256:2e221a9eec7ccc58889a278ea13dcfed5ef939d80b07819a9a8b3cb1c681484f \ - --hash=sha256:69118965410ec86d44dc6b9017ee3ddbd582e0c0abeef62b3a19dbf6c8ad132b \ - --hash=sha256:d04df8686ec864d0cade8cf199f7f83aecd416109a20834d568f8310ded12dea \ - --hash=sha256:e75a947e15ee97e7e71e02ea302feb2fc62d3a2bb4668bf9dfbed43a506ac7e7 \ - --hash=sha256:4e45d22fb883222a5ab9f282a116fec5ee2e8d1a568ccff6a2d75bbd0eb6bcfc \ - --hash=sha256:bce9339bb3c7a55e0803b63d21c5839e8e479bc85c4adf42ae415b72f94facb2 \ - --hash=sha256:928138365245a0e8869a5999fbcc2a45475a0a6ed52a494d60dbdc540335fedd \ - --hash=sha256:0d841ba1bb840eea0e6489dc5ecafa6125554971f53b5acb87764441e61bceba \ - --hash=sha256:b09c8c1d47b3531c400e0195697f1414a63221de6ef478598a4f1460f7d9a392 -requests-toolbelt==0.8.0 \ - --hash=sha256:42c9c170abc2cacb78b8ab23ac957945c7716249206f90874651971a4acff237 \ - --hash=sha256:f6a531936c6fa4c6cfce1b9c10d5c4f498d16528d2a54a22ca00011205a187b5 -chardet==3.0.2 \ - --hash=sha256:4f7832e7c583348a9eddd927ee8514b3bf717c061f57b21dbe7697211454d9bb \ - --hash=sha256:6ebf56457934fdce01fb5ada5582762a84eed94cad43ed877964aebbdd8174c0 -urllib3==1.24.1 \ - --hash=sha256:61bf29cada3fc2fbefad4fdf059ea4bd1b4a86d2b6d15e1c7c0b582b9752fe39 \ - --hash=sha256:de9529817c93f27c8ccbfead6985011db27bd0ddfcdb2d86f3f663385c6a9c22 -certifi==2017.4.17 \ - --hash=sha256:f4318671072f030a33c7ca6acaef720ddd50ff124d1388e50c1bda4cbd6d7010 \ - --hash=sha256:f7527ebf7461582ce95f7a9e03dd141ce810d40590834f4ec20cddd54234c10a + --hash=sha256:67d59d66cdba8d624e0389b87b2a83a176f21f16a87553b50f5703b23f29bac2 +pytz==2019.3 \ + --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \ + --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be +requests==2.22.0 \ + --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 \ + --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31 +requests-toolbelt==0.9.1 \ + --hash=sha256:380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f \ + --hash=sha256:968089d4584ad4ad7c171454f0a5c6dac23971e9472521ea3b6d49d610aa6fc0 +six==1.14.0 \ + --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \ + --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c +urllib3==1.25.8 \ + --hash=sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc \ + --hash=sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc +zope.component==4.6 \ + --hash=sha256:ec2afc5bbe611dcace98bb39822c122d44743d635dafc7315b9aef25097db9e6 +zope.deferredimport==4.3.1 \ + --hash=sha256:57b2345e7b5eef47efcd4f634ff16c93e4265de3dcf325afc7315ade48d909e1 \ + --hash=sha256:9a0c211df44aa95f1c4e6d2626f90b400f56989180d3ef96032d708da3d23e0a +zope.deprecation==4.4.0 \ + --hash=sha256:0d453338f04bacf91bbfba545d8bcdf529aa829e67b705eac8c1a7fdce66e2df \ + --hash=sha256:f1480b74995958b24ce37b0ef04d3663d2683e5d6debc96726eff18acf4ea113 +zope.event==4.4 \ + --hash=sha256:69c27debad9bdacd9ce9b735dad382142281ac770c4a432b533d6d65c4614bcf \ + --hash=sha256:d8e97d165fd5a0997b45f5303ae11ea3338becfe68c401dd88ffd2113fe5cae7 +zope.hookable==5.0.0 \ + --hash=sha256:0992a0dd692003c09fb958e1480cebd1a28f2ef32faa4857d864f3ca8e9d6952 \ + --hash=sha256:0f325838dbac827a1e2ed5d482c1f2656b6844dc96aa098f7727e76395fcd694 \ + --hash=sha256:22a317ba00f61bac99eac1a5e330be7cb8c316275a21269ec58aa396b602af0c \ + --hash=sha256:25531cb5e7b35e8a6d1d6eddef624b9a22ce5dcf8f4448ef0f165acfa8c3fc21 \ + --hash=sha256:30890892652766fc80d11f078aca9a5b8150bef6b88aba23799581a53515c404 \ + --hash=sha256:342d682d93937e5b8c232baffb32a87d5eee605d44f74566657c64a239b7f342 \ + --hash=sha256:46b2fddf1f5aeb526e02b91f7e62afbb9fff4ffd7aafc97cdb00a0d717641567 \ + --hash=sha256:523318ff96df9b8d378d997c00c5d4cbfbff68dc48ff5ee5addabdb697d27528 \ + --hash=sha256:53aa02eb8921d4e667c69d76adeed8fe426e43870c101cb08dcd2f3468aff742 \ + --hash=sha256:62e79e8fdde087cb20822d7874758f5acbedbffaf3c0fbe06309eb8a41ee4e06 \ + --hash=sha256:74bf2f757f7385b56dc3548adae508d8b3ef952d600b4b12b88f7d1706b05dcc \ + --hash=sha256:751ee9d89eb96e00c1d7048da9725ce392a708ed43406416dc5ed61e4d199764 \ + --hash=sha256:7b83bc341e682771fe810b360cd5d9c886a948976aea4b979ff214e10b8b523b \ + --hash=sha256:81eeeb27dbb0ddaed8070daee529f0d1bfe4f74c7351cce2aaca3ea287c4cc32 \ + --hash=sha256:856509191e16930335af4d773c0fc31a17bae8991eb6f167a09d5eddf25b56cc \ + --hash=sha256:8853e81fd07b18fa9193b19e070dc0557848d9945b1d2dac3b7782543458c87d \ + --hash=sha256:94506a732da2832029aecdfe6ea07eb1b70ee06d802fff34e1b3618fe7cdf026 \ + --hash=sha256:95ad874a8cc94e786969215d660143817f745225579bfe318c4676e218d3147c \ + --hash=sha256:9758ec9174966ffe5c499b6c3d149f80aa0a9238020006a2b87c6af5963fcf48 \ + --hash=sha256:a169823e331da939aa7178fc152e65699aeb78957e46c6f80ccb50ee4c3616c2 \ + --hash=sha256:a67878a798f6ca292729a28c2226592b3d000dc6ee7825d31887b553686c7ac7 \ + --hash=sha256:a9a6d9eb2319a09905670810e2de971d6c49013843700b4975e2fc0afe96c8db \ + --hash=sha256:b3e118b58a3d2301960e6f5f25736d92f6b9f861728d3b8c26d69f54d8a157d2 \ + --hash=sha256:ca6705c2a1fb5059a4efbe9f5426be4cdf71b3c9564816916fc7aa7902f19ede \ + --hash=sha256:cf711527c9d4ae72085f137caffb4be74fc007ffb17cd103628c7d5ba17e205f \ + --hash=sha256:d087602a6845ebe9d5a1c5a949fedde2c45f372d77fbce4f7fe44b68b28a1d03 \ + --hash=sha256:d1080e1074ddf75ad6662a9b34626650759c19a9093e1a32a503d37e48da135b \ + --hash=sha256:db9c60368aff2b7e6c47115f3ad9bd6e96aa298b12ed5f8cb13f5673b30be565 \ + --hash=sha256:dbeb127a04473f5a989169eb400b67beb921c749599b77650941c21fe39cb8d9 \ + --hash=sha256:dca336ca3682d869d291d7cd18284f6ff6876e4244eb1821430323056b000e2c \ + --hash=sha256:dd69a9be95346d10c853b6233fcafe3c0315b89424b378f2ad45170d8e161568 \ + --hash=sha256:dd79f8fae5894f1ee0a0042214685f2d039341250c994b825c10a4cd075d80f6 \ + --hash=sha256:e647d850aa1286d98910133cee12bd87c354f7b7bb3f3cd816a62ba7fa2f7007 \ + --hash=sha256:f37a210b5c04b2d4e4bac494ab15b70196f219a1e1649ddca78560757d4278fb \ + --hash=sha256:f67820b6d33a705dc3c1c457156e51686f7b350ff57f2112e1a9a4dad38ec268 \ + --hash=sha256:f68969978ccf0e6123902f7365aae5b7a9e99169d4b9105c47cf28e788116894 \ + --hash=sha256:f717a0b34460ae1ac0064e91b267c0588ac2c098ffd695992e72cd5462d97a67 \ + --hash=sha256:f9d58ccec8684ca276d5a4e7b0dfacca028336300a8f715d616d9f0ce9ae8096 \ + --hash=sha256:fcc3513a54e656067cbf7b98bab0d6b9534b9eabc666d1f78aad6acdf0962736 +zope.interface==4.7.1 \ + --hash=sha256:048b16ac882a05bc7ef534e8b9f15c9d7a6c190e24e8938a19b7617af4ed854a \ + --hash=sha256:05816cf8e7407cf62f2ec95c0a5d69ec4fa5741d9ccd10db9f21691916a9a098 \ + --hash=sha256:065d6a1ac89d35445168813bed45048ed4e67a4cdfc5a68fdb626a770378869f \ + --hash=sha256:14157421f4121a57625002cc4f48ac7521ea238d697c4a4459a884b62132b977 \ + --hash=sha256:18dc895945694f397a0be86be760ff664b790f95d8e7752d5bab80284ff9105d \ + --hash=sha256:1962c9f838bd6ae4075d0014f72697510daefc7e1c7e48b2607df0b6e157989c \ + --hash=sha256:1a67408cacd198c7e6274a19920bb4568d56459e659e23c4915528686ac1763a \ + --hash=sha256:21bf781076dd616bd07cf0223f79d61ab4f45176076f90bc2890e18c48195da4 \ + --hash=sha256:21c0a5d98650aebb84efa16ce2c8df1a46bdc4fe8a9e33237d0ca0b23f416ead \ + --hash=sha256:23cfeea25d1e42ff3bf4f9a0c31e9d5950aa9e7c4b12f0c4bd086f378f7b7a71 \ + --hash=sha256:24b6fce1fb71abf9f4093e3259084efcc0ef479f89356757780685bd2b06ef37 \ + --hash=sha256:24f84ce24eb6b5fcdcb38ad9761524f1ae96f7126abb5e597f8a3973d9921409 \ + --hash=sha256:25e0ef4a824017809d6d8b0ce4ab3288594ba283e4d4f94d8cfb81d73ed65114 \ + --hash=sha256:2e8fdd625e9aba31228e7ddbc36bad5c38dc3ee99a86aa420f89a290bd987ce9 \ + --hash=sha256:2f3bc2f49b67b1bea82b942d25bc958d4f4ea6709b411cb2b6b9718adf7914ce \ + --hash=sha256:35d24be9d04d50da3a6f4d61de028c1dd087045385a0ff374d93ef85af61b584 \ + --hash=sha256:35dbe4e8c73003dff40dfaeb15902910a4360699375e7b47d3c909a83ff27cd0 \ + --hash=sha256:3dfce831b824ab5cf446ed0c350b793ac6fa5fe33b984305cb4c966a86a8fb79 \ + --hash=sha256:3f7866365df5a36a7b8de8056cd1c605648f56f9a226d918ed84c85d25e8d55f \ + --hash=sha256:455cc8c01de3bac6f9c223967cea41f4449f58b4c2e724ec8177382ddd183ab4 \ + --hash=sha256:4bb937e998be9d5e345f486693e477ba79e4344674484001a0b646be1d530487 \ + --hash=sha256:52303a20902ca0888dfb83230ca3ee6fbe63c0ad1dd60aa0bba7958ccff454d8 \ + --hash=sha256:6e0a897d4e09859cc80c6a16a29697406ead752292ace17f1805126a4f63c838 \ + --hash=sha256:6e1816e7c10966330d77af45f77501f9a68818c065dec0ad11d22b50a0e212e7 \ + --hash=sha256:73b5921c5c6ce3358c836461b5470bf675601c96d5e5d8f2a446951470614f67 \ + --hash=sha256:8093cd45cdb5f6c8591cfd1af03d32b32965b0f79b94684cd0c9afdf841982bb \ + --hash=sha256:864b4a94b60db301899cf373579fd9ef92edddbf0fb2cd5ae99f53ef423ccc56 \ + --hash=sha256:8a27b4d3ea9c6d086ce8e7cdb3e8d319b6752e2a03238a388ccc83ccbe165f50 \ + --hash=sha256:91b847969d4784abd855165a2d163f72ac1e58e6dce09a5e46c20e58f19cc96d \ + --hash=sha256:b47b1028be4758c3167e474884ccc079b94835f058984b15c145966c4df64d27 \ + --hash=sha256:b68814a322835d8ad671b7acc23a3b2acecba527bb14f4b53fc925f8a27e44d8 \ + --hash=sha256:bcb50a032c3b6ec7fb281b3a83d2b31ab5246c5b119588725b1350d3a1d9f6a3 \ + --hash=sha256:c56db7d10b25ce8918b6aec6b08ac401842b47e6c136773bfb3b590753f7fb67 \ + --hash=sha256:c94b77a13d4f47883e4f97f9fa00f5feadd38af3e6b3c7be45cfdb0a14c7149b \ + --hash=sha256:db381f6fdaef483ad435f778086ccc4890120aff8df2ba5cfeeac24d280b3145 \ + --hash=sha256:e6487d01c8b7ed86af30ea141fcc4f93f8a7dde26f94177c1ad637c353bd5c07 \ + --hash=sha256:e86923fa728dfba39c5bb6046a450bd4eec8ad949ac404eca728cfce320d1732 \ + --hash=sha256:f6ca36dc1e9eeb46d779869c60001b3065fb670b5775c51421c099ea2a77c3c9 \ + --hash=sha256:fb62f2cbe790a50d95593fb40e8cca261c31a2f5637455ea39440d6457c2ba25 +zope.proxy==4.3.3 \ + --hash=sha256:04646ac04ffa9c8e32fb2b5c3cd42995b2548ea14251f3c21ca704afae88e42c \ + --hash=sha256:07b6bceea232559d24358832f1cd2ed344bbf05ca83855a5b9698b5f23c5ed60 \ + --hash=sha256:1ef452cc02e0e2f8e3c917b1a5b936ef3280f2c2ca854ee70ac2164d1655f7e6 \ + --hash=sha256:22bf61857c5977f34d4e391476d40f9a3b8c6ab24fb0cac448d42d8f8b9bf7b2 \ + --hash=sha256:299870e3428cbff1cd9f9b34144e76ecdc1d9e3192a8cf5f1b0258f47a239f58 \ + --hash=sha256:2bfc36bfccbe047671170ea5677efd3d5ab730a55d7e45611d76d495e5b96766 \ + --hash=sha256:32e82d5a640febc688c0789e15ea875bf696a10cf358f049e1ed841f01710a9b \ + --hash=sha256:3b2051bdc4bc3f02fa52483f6381cf40d4d48167645241993f9d7ebbd142ed9b \ + --hash=sha256:3f734bd8a08f5185a64fb6abb8f14dc97ec27a689ca808fb7a83cdd38d745e4f \ + --hash=sha256:3f78dd8de3112df8bbd970f0916ac876dc3fbe63810bd1cf7cc5eec4cbac4f04 \ + --hash=sha256:4eabeb48508953ba1f3590ad0773b8daea9e104eec66d661917e9bbcd7125a67 \ + --hash=sha256:4f05ecc33808187f430f249cb1ccab35c38f570b181f2d380fbe253da94b18d8 \ + --hash=sha256:4f4f4cbf23d3afc1526294a31e7b3eaa0f682cc28ac5366065dc1d6bb18bd7be \ + --hash=sha256:5483d5e70aacd06f0aa3effec9fed597c0b50f45060956eeeb1203c44d4338c3 \ + --hash=sha256:56a5f9b46892b115a75d0a1f2292431ad5988461175826600acc69a24cb3edee \ + --hash=sha256:64bb63af8a06f736927d260efdd4dfc5253d42244f281a8063e4b9eea2ddcbc5 \ + --hash=sha256:653f8cbefcf7c6ac4cece2cdef367c4faa2b7c19795d52bd7cbec11a8739a7c1 \ + --hash=sha256:664211d63306e4bd4eec35bf2b4bd9db61c394037911cf2d1804c43b511a49f1 \ + --hash=sha256:6651e6caed66a8fff0fef1a3e81c0ed2253bf361c0fdc834500488732c5d16e9 \ + --hash=sha256:6c1fba6cdfdf105739d3069cf7b07664f2944d82a8098218ab2300a82d8f40fc \ + --hash=sha256:6e64246e6e9044a4534a69dca1283c6ddab6e757be5e6874f69024329b3aa61f \ + --hash=sha256:838390245c7ec137af4993c0c8052f49d5ec79e422b4451bfa37fee9b9ccaa01 \ + --hash=sha256:856b410a14793069d8ba35f33fff667213ea66f2df25a0024cc72a7493c56d4c \ + --hash=sha256:8b932c364c1d1605a91907a41128ed0ee8a2d326fc0fafb2c55cd46f545f4599 \ + --hash=sha256:9086cf6d20f08dae7f296a78f6c77d1f8d24079d448f023ee0eb329078dd35e1 \ + --hash=sha256:9698533c14afa0548188de4968a7932d1f3f965f3f5ba1474de673596bb875af \ + --hash=sha256:9b12b05dd7c28f5068387c1afee8cb94f9d02501e7ef495a7c5c7e27139b96ad \ + --hash=sha256:a884c7426a5bc6fb7fc71a55ad14e66818e13f05b78b20a6f37175f324b7acb8 \ + --hash=sha256:abe9e7f1a3e76286c5f5baf2bf5162d41dc0310da493b34a2c36555f38d928f7 \ + --hash=sha256:bd6fde63b015a27262be06bd6bbdd895273cc2bdf2d4c7e1c83711d26a8fbace \ + --hash=sha256:bda7c62c954f47b87ed9a89f525eee1b318ec7c2162dfdba76c2ccfa334e0caa \ + --hash=sha256:be8a4908dd3f6e965993c0068b006bdbd0474fbcbd1da4893b49356e73fc1557 \ + --hash=sha256:ced65fc3c7d7205267506d854bb1815bb445899cca9d21d1d4b949070a635546 \ + --hash=sha256:dac4279aa05055d3897ab5e5ee5a7b39db121f91df65a530f8b1ac7f9bd93119 \ + --hash=sha256:e4f1863056e3e4f399c285b67fa816f411a7bfa1c81ef50e186126164e396e59 \ + --hash=sha256:ecd85f68b8cd9ab78a0141e87ea9a53b2f31fd9b1350a1c44da1f7481b5363ef \ + --hash=sha256:ed269b83750413e8fc5c96276372f49ee3fcb7ed61c49fe8e5a67f54459a5a4a \ + --hash=sha256:f19b0b80cba73b204dee68501870b11067711d21d243fb6774256d3ca2e5391f \ + --hash=sha256:ffdafb98db7574f9da84c489a10a5d582079a888cb43c64e9e6b0e3fe1034685 # Contains the requirements for the letsencrypt package. # @@ -1232,18 +1540,18 @@ letsencrypt==0.7.0 \ --hash=sha256:105a5fb107e45bcd0722eb89696986dcf5f08a86a321d6aef25a0c7c63375ade \ --hash=sha256:c36e532c486a7e92155ee09da54b436a3c420813ec1c590b98f635d924720de9 -certbot==0.30.2 \ - --hash=sha256:e411b72fa86eec1018e6de28e649e8c9c71191a7431dcc77f207b57ca9484c11 \ - --hash=sha256:534487cb552ced8e47948ba3d2e7ca12c3a439133fc609485012b1a02fc7776e -acme==0.30.2 \ - --hash=sha256:68982576492dfa99c7e2be0fce4371adc9344740b05420ce0ab53238d2bb9b3b \ - --hash=sha256:295a5b7fce9f908e6e5cff8c40be1a3daf3e1ebabd2e139a4c87274e68eeb8f2 -certbot-apache==0.30.2 \ - --hash=sha256:3b7fa4e59772da7c9975ef2a49ceff157c9d7cb31eb9475928b5986d89701a3a \ - --hash=sha256:32fa915a8a51810fdfe828ac1361da4425c231d7384891e49e6338e4741464b2 -certbot-nginx==0.30.2 \ - --hash=sha256:7dc785f6f0c0c57b19cea8d98f9ea8feef53945613967b52c9348c81327010e2 \ - --hash=sha256:6ba4dd772d0c7cdfb3383ca325b35639e01ac9e142e4baa6445cd85c7fb59552 +certbot==1.2.0 \ + --hash=sha256:e25c17125c00b3398c8e9b9d54ef473c0e8f5aff53389f313a51b06cf472d335 \ + --hash=sha256:95dcbae085f8e4eb18442fe7b12994b08964a9a6e8e352e556cdb4a8a625373c +acme==1.2.0 \ + --hash=sha256:284d22fde75687a8ea72d737cac6bcbdc91f3c796221aa25378b8732ba6f6875 \ + --hash=sha256:0630c740d49bda945e97bd35fc8d6f02d082c8cb9e18f8fec0dbb3d395ac26ab +certbot-apache==1.2.0 \ + --hash=sha256:3f7493918353d3bd6067d446a2cf263e03831c4c10ec685b83d644b47767090d \ + --hash=sha256:b46e9def272103a68108e48bf7e410ea46801529b1ea6954f6506b14dd9df9b3 +certbot-nginx==1.2.0 \ + --hash=sha256:efd32a2b32f2439279da446b6bf67684f591f289323c5f494ebfd86a566a28fd \ + --hash=sha256:6fd7cf4f2545ad66e57000343227df9ccccaf04420e835e05cb3250fac1fa6db UNLIKELY_EOF # ------------------------------------------------------------------------- @@ -1273,7 +1581,6 @@ from distutils.version import StrictVersion from hashlib import sha256 from os import environ from os.path import join -from pipes import quote from shutil import rmtree try: from subprocess import check_output @@ -1293,7 +1600,7 @@ except ImportError: cmd = popenargs[0] raise CalledProcessError(retcode, cmd) return output -from sys import exit, version_info +import sys from tempfile import mkdtemp try: from urllib2 import build_opener, HTTPHandler, HTTPSHandler @@ -1315,7 +1622,7 @@ maybe_argparse = ( [('18/dd/e617cfc3f6210ae183374cd9f6a26b20514bbb5a792af97949c5aacddf0f/' 'argparse-1.4.0.tar.gz', '62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4')] - if version_info < (2, 7, 0) else []) + if sys.version_info < (2, 7, 0) else []) PACKAGES = maybe_argparse + [ @@ -1396,7 +1703,8 @@ def get_index_base(): def main(): - pip_version = StrictVersion(check_output(['pip', '--version']) + python = sys.executable or 'python' + pip_version = StrictVersion(check_output([python, '-m', 'pip', '--version']) .decode('utf-8').split()[1]) has_pip_cache = pip_version >= StrictVersion('6.0') index_base = get_index_base() @@ -1406,12 +1714,12 @@ def main(): temp, digest) for path, digest in PACKAGES] - check_output('pip install --no-index --no-deps -U ' + - # Disable cache since we're not using it and it otherwise - # sometimes throws permission warnings: - ('--no-cache-dir ' if has_pip_cache else '') + - ' '.join(quote(d) for d in downloads), - shell=True) + # Calling pip as a module is the preferred way to avoid problems about pip self-upgrade. + command = [python, '-m', 'pip', 'install', '--no-index', '--no-deps', '-U'] + # Disable cache since it is not used and it otherwise sometimes throws permission warnings: + command.extend(['--no-cache-dir'] if has_pip_cache else []) + command.extend(downloads) + check_output(command) except HashError as exc: print(exc) except Exception: @@ -1424,7 +1732,7 @@ def main(): if __name__ == '__main__': - exit(main()) + sys.exit(main()) UNLIKELY_EOF # ------------------------------------------------------------------------- @@ -1477,6 +1785,9 @@ UNLIKELY_EOF say "Installation succeeded." fi + # If you're modifying any of the code after this point in this current `if` block, you + # may need to update the "$DEPRECATED_OS" = 1 case at the beginning of phase 2 as well. + if [ "$INSTALL_ONLY" = 1 ]; then say "Certbot is installed." exit 0 @@ -1508,6 +1819,24 @@ else exit 0 fi + DeterminePythonVersion "NOCRASH" + # Don't warn about file permissions if the user disabled the check or we + # can't find an up-to-date Python. + if [ "$PYVER" -ge "$MIN_PYVER" -a "$NO_PERMISSIONS_CHECK" != 1 ]; then + # If the script fails for some reason, don't break certbot-auto. + set +e + # Suppress unexpected error output. + CHECK_PERM_OUT=$(CheckPathPermissions "$LE_PYTHON" "$0" 2>/dev/null) + CHECK_PERM_STATUS="$?" + set -e + # Only print output if the script ran successfully and it actually produced + # output. The latter check resolves + # https://github.com/certbot/certbot/issues/7012. + if [ "$CHECK_PERM_STATUS" = 0 -a -n "$CHECK_PERM_OUT" ]; then + error "$CHECK_PERM_OUT" + fi + fi + if [ "$NO_SELF_UPGRADE" != 1 ]; then TEMP_DIR=$(TempDir) trap 'rm -rf "$TEMP_DIR"' EXIT @@ -1664,37 +1993,41 @@ if __name__ == '__main__': UNLIKELY_EOF # --------------------------------------------------------------------------- - DeterminePythonVersion "NOCRASH" if [ "$PYVER" -lt "$MIN_PYVER" ]; then error "WARNING: couldn't find Python $MIN_PYTHON_VERSION+ to check for updates." elif ! REMOTE_VERSION=`"$LE_PYTHON" "$TEMP_DIR/fetch.py" --latest-version` ; then error "WARNING: unable to check for updates." fi - LE_VERSION_STATE=`CompareVersions "$LE_PYTHON" "$LE_AUTO_VERSION" "$REMOTE_VERSION"` - if [ "$LE_VERSION_STATE" = "UNOFFICIAL" ]; then - say "Unofficial certbot-auto version detected, self-upgrade is disabled: $LE_AUTO_VERSION" - elif [ "$LE_VERSION_STATE" = "OUTDATED" ]; then - say "Upgrading certbot-auto $LE_AUTO_VERSION to $REMOTE_VERSION..." + # If for any reason REMOTE_VERSION is not set, let's assume certbot-auto is up-to-date, + # and do not go into the self-upgrading process. + if [ -n "$REMOTE_VERSION" ]; then + LE_VERSION_STATE=`CompareVersions "$LE_PYTHON" "$LE_AUTO_VERSION" "$REMOTE_VERSION"` - # Now we drop into Python so we don't have to install even more - # dependencies (curl, etc.), for better flow control, and for the option of - # future Windows compatibility. - "$LE_PYTHON" "$TEMP_DIR/fetch.py" --le-auto-script "v$REMOTE_VERSION" + if [ "$LE_VERSION_STATE" = "UNOFFICIAL" ]; then + say "Unofficial certbot-auto version detected, self-upgrade is disabled: $LE_AUTO_VERSION" + elif [ "$LE_VERSION_STATE" = "OUTDATED" ]; then + say "Upgrading certbot-auto $LE_AUTO_VERSION to $REMOTE_VERSION..." - # Install new copy of certbot-auto. - # TODO: Deal with quotes in pathnames. - say "Replacing certbot-auto..." - # Clone permissions with cp. chmod and chown don't have a --reference - # option on macOS or BSD, and stat -c on Linux is stat -f on macOS and BSD: - cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone" - cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone" - # Using mv rather than cp leaves the old file descriptor pointing to the - # original copy so the shell can continue to read it unmolested. mv across - # filesystems is non-atomic, doing `rm dest, cp src dest, rm src`, but the - # cp is unlikely to fail if the rm doesn't. - mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0" - fi # A newer version is available. + # Now we drop into Python so we don't have to install even more + # dependencies (curl, etc.), for better flow control, and for the option of + # future Windows compatibility. + "$LE_PYTHON" "$TEMP_DIR/fetch.py" --le-auto-script "v$REMOTE_VERSION" + + # Install new copy of certbot-auto. + # TODO: Deal with quotes in pathnames. + say "Replacing certbot-auto..." + # Clone permissions with cp. chmod and chown don't have a --reference + # option on macOS or BSD, and stat -c on Linux is stat -f on macOS and BSD: + cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone" + cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone" + # Using mv rather than cp leaves the old file descriptor pointing to the + # original copy so the shell can continue to read it unmolested. mv across + # filesystems is non-atomic, doing `rm dest, cp src dest, rm src`, but the + # cp is unlikely to fail if the rm doesn't. + mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0" + fi # A newer version is available. + fi fi # Self-upgrading is allowed. RerunWithArgs --le-auto-phase2 "$@" diff --git a/certbot-ci/MANIFEST.in b/certbot-ci/MANIFEST.in new file mode 100644 index 000000000..7a18def43 --- /dev/null +++ b/certbot-ci/MANIFEST.in @@ -0,0 +1 @@ +recursive-include certbot_integration_tests/assets * diff --git a/certbot-ci/certbot_integration_tests/.coveragerc b/certbot-ci/certbot_integration_tests/.coveragerc new file mode 100644 index 000000000..72f7c6adf --- /dev/null +++ b/certbot-ci/certbot_integration_tests/.coveragerc @@ -0,0 +1,9 @@ +[run] +# Avoid false warnings because certbot packages are not installed in the thread that executes +# the coverage: indeed, certbot is launched as a CLI from a subprocess. +disable_warnings = module-not-imported,no-data-collected +omit = **/*_test.py,**/tests/*,**/dns_common*,**/certbot_nginx/_internal/parser_obj.py + +[report] +# Exclude unit tests in coverage during integration tests. +omit = **/*_test.py,**/tests/*,**/dns_common*,**/certbot_nginx/_internal/parser_obj.py diff --git a/certbot-ci/certbot_integration_tests/__init__.py b/certbot-ci/certbot_integration_tests/__init__.py new file mode 100644 index 000000000..434a85a23 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/__init__.py @@ -0,0 +1 @@ +"""Package certbot_integration_test is for tests that require a live acme ca server instance""" diff --git a/certbot-ci/certbot_integration_tests/assets/cert.pem b/certbot-ci/certbot_integration_tests/assets/cert.pem new file mode 100644 index 000000000..5aae58f25 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/assets/cert.pem @@ -0,0 +1,32 @@ +-----BEGIN CERTIFICATE----- +MIIFlTCCA32gAwIBAgIUR3wbM8qFE68f8NxfciHhUjR1GeUwDQYJKoZIhvcNAQEL +BQAwWTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM +GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDESMBAGA1UEAwwJbmdpbngud3RmMCAX +DTE5MDQxODIwMDUwM1oYDzIyOTMwMTMwMjAwNTAzWjBZMQswCQYDVQQGEwJBVTET +MBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQ +dHkgTHRkMRIwEAYDVQQDDAluZ2lueC53dGYwggIiMA0GCSqGSIb3DQEBAQUAA4IC +DwAwggIKAoICAQC/W+yxYE0PWJOS4df71Yx596fDjW03I9JZuu9kfP7mneMgy+OC +HyRm0TEhl6FPUp9tD9YeEHloUZNjHEOg/qrnbEOspv3Ha3RFinzrzkMwbzEPR3Xf +0go+aVsWelDhapFl8fccw4tWwijVZQquhBsWOUnPenS3Txe96kEv2NNJlJ0qFUa+ +rOTruzRzOzlbgKv5WRb4+BxxWonHLkAQ5IT87GBlsCerVIyPD+BnZveZGl6e9oMH +ZlZvUT6aWRnzFWjAnQGiJpVIw7l9r4EW0jq1z7wqb37FrqrFbtWrOfUZVE7AlqXH +aKIR82/xwkcZfFk3sCAM0IcZc8B2SDLi4zNZtDivW6qQgTC/3z5yf1hnJ+j00dtE +X5qYlgXRaM2raOn31lxcerk5pjgagQ7Zj+v3YZS0QnenrgyXJcdnXLDj+cIARzx4 +QHtoO0nyP0RJqxvwX/H98513JTkeqFBc/Bx11UWYsUv20Qoo9IAuz0VDARu6rquu +k9anv56yvxo77qZ8r80l3z8eMyDA+UjuSD2p1Za09RAHfva7o8rMUqULHNQ4pfFH +JIUozHoinAg/9lBC/W80fcbILks+Sdi6E9WQ0n8PLl7oFLx9prEDCycKuC0z76J/ +Shb6R6sWr1YtzUFUc5EH2g9pMriaqT8uGO4CMOeRemXahrdT/H+Xg5m4TQIDAQAB +o1MwUTAdBgNVHQ4EFgQU46gJeu9ZOfTQ6c4vfbWbSLUpEMowHwYDVR0jBBgwFoAU +46gJeu9ZOfTQ6c4vfbWbSLUpEMowDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0B +AQsFAAOCAgEAcnfkXDUTsEGs0MleegkGbRCVy72a3U7tv1KVTLB8qLPc3tpPJJoT +D4PbOuw9+yIE+HetZTZooOpaZoorLQdiwAEjlQ44RVuXSHSARQ8KW9ZZeiWN/Qvl +Ip4xJ/cHxcKTFKSc/99o8M+kmPKEXF9SUMfKPc5jXarNxCsnA3VriYqJ+CnYEox2 +duNUEe3A9Y2d8ZxjmscBqlcXpk1kFwsCRT5UYVoUYwyjYznLkO5A+GJ0ZnMyRMQp +obUiB34hUrNgyOaBvizk+pNh9EV4rEBPRQwhy4vDMco4AjQcwLWQAQ9G4GSt/E+Q +62XdVDa6CAuOvBCudDPki7kEqNLbj1tMY1K/gsbgb6TYA/xTOVulAnqm4OEZ2svJ +0Jqw3BzMfRTaxsNU6jxm8WehVL15GjoJUzfs7Te+l7Vm/QNc1Dv2pmEhVfBibwMa +YxUZ8ClQtQ1lsOpne97Og0p/Cm93kKELNBLTjzXtpXGGPPYisAyNwe0Hadq8SiOd +pXeNwXa5vHOXHv8xBENzBvFJ3TRN2GmMlHBp/eOfVUx/huNSpcnh2gO3fn5EbMj7 +43IaR133JW5yWbneYAMJOEAMdEB5EthRmEDtLVA7kLqLc/ywFTQ4VbS2b+PsOr5O +501nzt0OTMMEz+UafvGXj5OPJBhe26RtnYXzVwwLfto/F5udM5zglWo= +-----END CERTIFICATE----- diff --git a/certbot-ci/certbot_integration_tests/assets/hook.py b/certbot-ci/certbot_integration_tests/assets/hook.py new file mode 100755 index 000000000..39aa72ac5 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/assets/hook.py @@ -0,0 +1,11 @@ +#!/usr/bin/env python +import os +import sys + +hook_script_type = os.path.basename(os.path.dirname(sys.argv[1])) +if hook_script_type == 'deploy' and ('RENEWED_DOMAINS' not in os.environ or 'RENEWED_LINEAGE' not in os.environ): + sys.stderr.write('Environment variables not properly set!\n') + sys.exit(1) + +with open(sys.argv[2], 'a') as file_h: + file_h.write(hook_script_type + '\n') diff --git a/certbot-ci/certbot_integration_tests/assets/key.pem b/certbot-ci/certbot_integration_tests/assets/key.pem new file mode 100644 index 000000000..1f768f079 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/assets/key.pem @@ -0,0 +1,52 @@ +-----BEGIN PRIVATE KEY----- +MIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQC/W+yxYE0PWJOS +4df71Yx596fDjW03I9JZuu9kfP7mneMgy+OCHyRm0TEhl6FPUp9tD9YeEHloUZNj +HEOg/qrnbEOspv3Ha3RFinzrzkMwbzEPR3Xf0go+aVsWelDhapFl8fccw4tWwijV +ZQquhBsWOUnPenS3Txe96kEv2NNJlJ0qFUa+rOTruzRzOzlbgKv5WRb4+BxxWonH +LkAQ5IT87GBlsCerVIyPD+BnZveZGl6e9oMHZlZvUT6aWRnzFWjAnQGiJpVIw7l9 +r4EW0jq1z7wqb37FrqrFbtWrOfUZVE7AlqXHaKIR82/xwkcZfFk3sCAM0IcZc8B2 +SDLi4zNZtDivW6qQgTC/3z5yf1hnJ+j00dtEX5qYlgXRaM2raOn31lxcerk5pjga +gQ7Zj+v3YZS0QnenrgyXJcdnXLDj+cIARzx4QHtoO0nyP0RJqxvwX/H98513JTke +qFBc/Bx11UWYsUv20Qoo9IAuz0VDARu6rquuk9anv56yvxo77qZ8r80l3z8eMyDA ++UjuSD2p1Za09RAHfva7o8rMUqULHNQ4pfFHJIUozHoinAg/9lBC/W80fcbILks+ +Sdi6E9WQ0n8PLl7oFLx9prEDCycKuC0z76J/Shb6R6sWr1YtzUFUc5EH2g9pMria +qT8uGO4CMOeRemXahrdT/H+Xg5m4TQIDAQABAoICAAGGL+pxw+tdXz+KQPgmiUnn +aRSrqbUIugIw9Pst67HWjBqUxSkiKl4PSH7mAEjrdY2e1KvEodLs42mkrf04ShAx +0pArfFX8Sx7KrZgLOonGOPPQM+YmfCJnIGybaM2C1cmkFb3K6O81+LFKbr1ZHAYf +SrE2XnufS6cdmItTBMvPPTk6lieqpOAjy5UnYZuS+Muxo/czsrZMbFCD08rOpyiE +kXf94TMCJ2R0UetA7LPxe9N0TzLd485bLU55azV+dCkklwC9oe7EcFPJ9BNEdWdB +UlRcMvxMGdwct+L3QTaEb2QlTwi5kqDl+XxJeduAHA3Pf1Haz1iqjVvj01PvT1di +Cs0+ZeFBsa+BfiGDe9ONwuSQljda1CuI+vDv5bGUExulOSG1dHJ7RK9PBaXFaR/b +/9tRBwAw1Erm7s1JIkjda5Oc46jFb3HzDaZYB1n5hUmEIrYM8HhUOGITyVT3hxDO +AWlaV3aveQ0MmMXLptVXDgbjPGbWDGMLD9d5vUE9R7IyOLeXOmjthYlCH2rj378M +r2PkgX2tD0A/yoEZ8XCFdtBWSVajLdL0/gkm7sKosMABBy3yrSCxbHeq5TFuTAXA +hOdypX4NOZkA6WJU+hn3GkQyIScLqSrvGRA9kzHGoEWVZDKkB9DXg+dmTARZDWXD +mCnHkJo6+FcbhUpXniuZAoIBAQDmE94vvdstB+HEtXxN1uNDY7H8gPc/BUonU6a9 +G5YOIbjByCfEDcXF8AUWekc6lc8DNG3ydx0dnb2ZAkxmdlsaD8GLqHGILzlSsOwR +sez8nR4+4n9vYMfx9Qal8Ren5xEP9Z9sJcNqbKVGta1WFtQzrgYbpVXXf/Luv0xS +YoVK8KaEACciD6XX4wmajrAXPPQgThvqQtXuTn/AxWsUDg1DK0tw1VRUuOJuJwpw +f6qocM9AyqUNvdeVyjFx8Slag34ZI7fmxPtHX/e6opTg3zVXab1Ow8AMICOHMRL6 +m5/+wnWa9xMoKI4kYfk/QFqeTccnLDlwi6kQM8WRfbwr9AyPAoIBAQDU60wrX6Lm +0vIfngv1/4j/w+AGAwjvxiuJ7Q7LwQ2fGsZGOIfMK/lpBxCn543kGbQT+KQKNOjO ++EywObftnJ6Y2+om2NoLkCnCiptsfr5WlN8pxtIPQu2iu5xXA67WpQv4Nc4769PM +wJGVW3pmPKi6H0QjjqYAZd1NAXdN9Au14zZVh3KBWoz82kTHWKSL6Ld1UClG728W +k/moyCFFMMGTXX/LVliQzDVLM6L5jbAOaG317qAuxZIqFJ9NLwHFW9uH/i1S6Qfp ++lOmOfVYKu1O/qh1DUBQfuJkR1XIn2ifZEjxOsxeTmWu1LXpyoZy526JRu49pk8Z +DdEu+w7hsdNjAoIBAD1YWsub8Y6GJXpPcX9HpnzXXiOXN1VEUcs+kJyneFD4SMzS +U1gA3BS0tIaTv94tB28xUYdunwLAhkb/x+Mh95RxUwert+m5va0Ao1DsgeWw9tmJ +hrTptyYaUNV5/Pa1s2Tv9rvdLcd4hHDgDAGCQL4uzk4cvVCiOuHRe8YTorqig6N6 +bvSz+2IelPbyyJzJkcXzTZoei+/oWkPJ340PWhXou0qwdrXIPgdkvXHVeGlE+t2p +qmyJi6vSp3Bb/sy1dq+5SFVtfBpBykmnA88ZdJ2EAge4RcJ150MqoIbVa8l/i9/v +tNnmRlAJF233+LFwx4L4VbBebIt3YlwyjDOj9J0CggEAIknKOGnsV/O8ni7bikAe +leG7X/x5IfPt6wZMDbAHO4oaSBCufcjPH4TNv9xgU014XIb8E9C1dS8zWmXRIujH ++aHgsWTWqGoM75FWukAm8taCob2s8lw63KwN301uiI6HwO8ZSTkPILgaOc1DhtdZ +7K9AT+GXBhVhcBc+WUVl5WKzy05GuGIWtlmIHfo+dXGCqdfA7fV9FEu8NtwTz4qs +gcja3aoIFTltk7C7HCkfIxLaMnK9RQr4IOK1TL63MEs8rUfXkLSKW7m+YtSOmCZB +lSkZg9AgfVYRq0h5nhddx91kicSISN+jLGaA7Sd6Q2LVwDG2CCOSNVyuRTyVBu+W +NQKCAQAWN6vB6oToNIoBLdOThm0HD07cNHcrnBjtaKsYsQDgqbr2m8LRCRzNRML4 +jG0IAOWpuCiEGsgUPxywiI1Ufvyq7ZSNT1QQNzCR47NM3Ve6S2abrQkMIk9VJ+za +CB9c1BH92GokoRxqswb/BiMttG2EIP8L8/pSRYEcVnaaxAkf9QOhEwj4LJPGX0mS +t7kWIUVHPdFJ67F25dYr3mUHgyV+QJupQICkkkgY3nBOU1fS42vAugaxqH0wAP3T +53FlpY3NuE7+kYC3FjfcBer99F1pOac3X9jxhk26w9dr2/QNA33xhDXHKYvoLUCG +RPQylahJByU7IrtQzSCf/RE7q4v0 +-----END PRIVATE KEY----- diff --git a/tests/letstest/testdata/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/meta.json b/certbot-ci/certbot_integration_tests/assets/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/meta.json similarity index 100% rename from tests/letstest/testdata/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/meta.json rename to certbot-ci/certbot_integration_tests/assets/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/meta.json diff --git a/tests/letstest/testdata/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/private_key.json b/certbot-ci/certbot_integration_tests/assets/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/private_key.json similarity index 100% rename from tests/letstest/testdata/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/private_key.json rename to certbot-ci/certbot_integration_tests/assets/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/private_key.json diff --git a/tests/letstest/testdata/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/regr.json b/certbot-ci/certbot_integration_tests/assets/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/regr.json similarity index 100% rename from tests/letstest/testdata/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/regr.json rename to certbot-ci/certbot_integration_tests/assets/sample-config/accounts/acme-staging.api.letsencrypt.org/directory/48d6b9e8d767eccf7e4d877d6ffa81e3/regr.json diff --git a/tests/letstest/testdata/sample-config/archive/a.encryption-example.com/cert1.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/archive/a.encryption-example.com/cert1.pem similarity index 100% rename from tests/letstest/testdata/sample-config/archive/a.encryption-example.com/cert1.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/archive/a.encryption-example.com/cert1.pem diff --git a/tests/letstest/testdata/sample-config/archive/a.encryption-example.com/chain1.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/archive/a.encryption-example.com/chain1.pem similarity index 100% rename from tests/letstest/testdata/sample-config/archive/a.encryption-example.com/chain1.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/archive/a.encryption-example.com/chain1.pem diff --git a/tests/letstest/testdata/sample-config/archive/a.encryption-example.com/fullchain1.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/archive/a.encryption-example.com/fullchain1.pem similarity index 100% rename from tests/letstest/testdata/sample-config/archive/a.encryption-example.com/fullchain1.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/archive/a.encryption-example.com/fullchain1.pem diff --git a/tests/letstest/testdata/sample-config/archive/a.encryption-example.com/privkey1.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/archive/a.encryption-example.com/privkey1.pem similarity index 100% rename from tests/letstest/testdata/sample-config/archive/a.encryption-example.com/privkey1.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/archive/a.encryption-example.com/privkey1.pem diff --git a/tests/letstest/testdata/sample-config/archive/b.encryption-example.com/cert1.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/archive/b.encryption-example.com/cert1.pem similarity index 100% rename from tests/letstest/testdata/sample-config/archive/b.encryption-example.com/cert1.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/archive/b.encryption-example.com/cert1.pem diff --git a/tests/letstest/testdata/sample-config/archive/b.encryption-example.com/chain1.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/archive/b.encryption-example.com/chain1.pem similarity index 100% rename from tests/letstest/testdata/sample-config/archive/b.encryption-example.com/chain1.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/archive/b.encryption-example.com/chain1.pem diff --git a/tests/letstest/testdata/sample-config/archive/b.encryption-example.com/fullchain1.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/archive/b.encryption-example.com/fullchain1.pem similarity index 100% rename from tests/letstest/testdata/sample-config/archive/b.encryption-example.com/fullchain1.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/archive/b.encryption-example.com/fullchain1.pem diff --git a/tests/letstest/testdata/sample-config/archive/b.encryption-example.com/privkey1.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/archive/b.encryption-example.com/privkey1.pem similarity index 100% rename from tests/letstest/testdata/sample-config/archive/b.encryption-example.com/privkey1.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/archive/b.encryption-example.com/privkey1.pem diff --git a/tests/letstest/testdata/sample-config/csr/0000_csr-certbot.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/csr/0000_csr-certbot.pem similarity index 100% rename from tests/letstest/testdata/sample-config/csr/0000_csr-certbot.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/csr/0000_csr-certbot.pem diff --git a/tests/letstest/testdata/sample-config/csr/0001_csr-certbot.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/csr/0001_csr-certbot.pem similarity index 100% rename from tests/letstest/testdata/sample-config/csr/0001_csr-certbot.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/csr/0001_csr-certbot.pem diff --git a/tests/letstest/testdata/sample-config/csr/0002_csr-certbot.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/csr/0002_csr-certbot.pem similarity index 100% rename from tests/letstest/testdata/sample-config/csr/0002_csr-certbot.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/csr/0002_csr-certbot.pem diff --git a/tests/letstest/testdata/sample-config/csr/0003_csr-certbot.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/csr/0003_csr-certbot.pem similarity index 100% rename from tests/letstest/testdata/sample-config/csr/0003_csr-certbot.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/csr/0003_csr-certbot.pem diff --git a/tests/letstest/testdata/sample-config/keys/0000_key-certbot.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/keys/0000_key-certbot.pem similarity index 100% rename from tests/letstest/testdata/sample-config/keys/0000_key-certbot.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/keys/0000_key-certbot.pem diff --git a/tests/letstest/testdata/sample-config/keys/0001_key-certbot.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/keys/0001_key-certbot.pem similarity index 100% rename from tests/letstest/testdata/sample-config/keys/0001_key-certbot.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/keys/0001_key-certbot.pem diff --git a/tests/letstest/testdata/sample-config/keys/0002_key-certbot.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/keys/0002_key-certbot.pem similarity index 100% rename from tests/letstest/testdata/sample-config/keys/0002_key-certbot.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/keys/0002_key-certbot.pem diff --git a/tests/letstest/testdata/sample-config/keys/0003_key-certbot.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/keys/0003_key-certbot.pem similarity index 100% rename from tests/letstest/testdata/sample-config/keys/0003_key-certbot.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/keys/0003_key-certbot.pem diff --git a/tests/letstest/testdata/sample-config/live/a.encryption-example.com/README b/certbot-ci/certbot_integration_tests/assets/sample-config/live/a.encryption-example.com/README similarity index 100% rename from tests/letstest/testdata/sample-config/live/a.encryption-example.com/README rename to certbot-ci/certbot_integration_tests/assets/sample-config/live/a.encryption-example.com/README diff --git a/tests/letstest/testdata/sample-config/live/a.encryption-example.com/cert.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/live/a.encryption-example.com/cert.pem similarity index 100% rename from tests/letstest/testdata/sample-config/live/a.encryption-example.com/cert.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/live/a.encryption-example.com/cert.pem diff --git a/tests/letstest/testdata/sample-config/live/a.encryption-example.com/chain.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/live/a.encryption-example.com/chain.pem similarity index 100% rename from tests/letstest/testdata/sample-config/live/a.encryption-example.com/chain.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/live/a.encryption-example.com/chain.pem diff --git a/tests/letstest/testdata/sample-config/live/a.encryption-example.com/fullchain.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/live/a.encryption-example.com/fullchain.pem similarity index 100% rename from tests/letstest/testdata/sample-config/live/a.encryption-example.com/fullchain.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/live/a.encryption-example.com/fullchain.pem diff --git a/tests/letstest/testdata/sample-config/live/a.encryption-example.com/privkey.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/live/a.encryption-example.com/privkey.pem similarity index 100% rename from tests/letstest/testdata/sample-config/live/a.encryption-example.com/privkey.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/live/a.encryption-example.com/privkey.pem diff --git a/tests/letstest/testdata/sample-config/live/b.encryption-example.com/README b/certbot-ci/certbot_integration_tests/assets/sample-config/live/b.encryption-example.com/README similarity index 100% rename from tests/letstest/testdata/sample-config/live/b.encryption-example.com/README rename to certbot-ci/certbot_integration_tests/assets/sample-config/live/b.encryption-example.com/README diff --git a/tests/letstest/testdata/sample-config/live/b.encryption-example.com/cert.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/live/b.encryption-example.com/cert.pem similarity index 100% rename from tests/letstest/testdata/sample-config/live/b.encryption-example.com/cert.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/live/b.encryption-example.com/cert.pem diff --git a/tests/letstest/testdata/sample-config/live/b.encryption-example.com/chain.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/live/b.encryption-example.com/chain.pem similarity index 100% rename from tests/letstest/testdata/sample-config/live/b.encryption-example.com/chain.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/live/b.encryption-example.com/chain.pem diff --git a/tests/letstest/testdata/sample-config/live/b.encryption-example.com/fullchain.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/live/b.encryption-example.com/fullchain.pem similarity index 100% rename from tests/letstest/testdata/sample-config/live/b.encryption-example.com/fullchain.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/live/b.encryption-example.com/fullchain.pem diff --git a/tests/letstest/testdata/sample-config/live/b.encryption-example.com/privkey.pem b/certbot-ci/certbot_integration_tests/assets/sample-config/live/b.encryption-example.com/privkey.pem similarity index 100% rename from tests/letstest/testdata/sample-config/live/b.encryption-example.com/privkey.pem rename to certbot-ci/certbot_integration_tests/assets/sample-config/live/b.encryption-example.com/privkey.pem diff --git a/tests/letstest/testdata/sample-config/options-ssl-apache.conf b/certbot-ci/certbot_integration_tests/assets/sample-config/options-ssl-apache.conf similarity index 100% rename from tests/letstest/testdata/sample-config/options-ssl-apache.conf rename to certbot-ci/certbot_integration_tests/assets/sample-config/options-ssl-apache.conf diff --git a/tests/letstest/testdata/sample-config/renewal/a.encryption-example.com.conf b/certbot-ci/certbot_integration_tests/assets/sample-config/renewal/a.encryption-example.com.conf similarity index 100% rename from tests/letstest/testdata/sample-config/renewal/a.encryption-example.com.conf rename to certbot-ci/certbot_integration_tests/assets/sample-config/renewal/a.encryption-example.com.conf diff --git a/tests/letstest/testdata/sample-config/renewal/b.encryption-example.com.conf b/certbot-ci/certbot_integration_tests/assets/sample-config/renewal/b.encryption-example.com.conf similarity index 100% rename from tests/letstest/testdata/sample-config/renewal/b.encryption-example.com.conf rename to certbot-ci/certbot_integration_tests/assets/sample-config/renewal/b.encryption-example.com.conf diff --git a/certbot-ci/certbot_integration_tests/certbot_tests/__init__.py b/certbot-ci/certbot_integration_tests/certbot_tests/__init__.py new file mode 100644 index 000000000..60c2fcdd8 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/certbot_tests/__init__.py @@ -0,0 +1,5 @@ +import pytest + +# Custom assertions defined in the following package need to be registered to be properly +# displayed in a pytest report when they are failing. +pytest.register_assert_rewrite('certbot_integration_tests.certbot_tests.assertions') diff --git a/certbot-ci/certbot_integration_tests/certbot_tests/assertions.py b/certbot-ci/certbot_integration_tests/certbot_tests/assertions.py new file mode 100644 index 000000000..1b5914d1a --- /dev/null +++ b/certbot-ci/certbot_integration_tests/certbot_tests/assertions.py @@ -0,0 +1,161 @@ +"""This module contains advanced assertions for the certbot integration tests.""" +import os + +try: + import grp + POSIX_MODE = True +except ImportError: + import win32api + import win32security + import ntsecuritycon + POSIX_MODE = False + +EVERYBODY_SID = 'S-1-1-0' +SYSTEM_SID = 'S-1-5-18' +ADMINS_SID = 'S-1-5-32-544' + + +def assert_hook_execution(probe_path, probe_content): + """ + Assert that a certbot hook has been executed + :param probe_path: path to the file that received the hook output + :param probe_content: content expected when the hook is executed + """ + with open(probe_path, 'r') as file: + data = file.read() + + lines = [line.strip() for line in data.splitlines()] + assert probe_content in lines + + +def assert_saved_renew_hook(config_dir, lineage): + """ + Assert that the renew hook configuration of a lineage has been saved. + :param config_dir: location of the certbot configuration + :param lineage: lineage domain name + """ + with open(os.path.join(config_dir, 'renewal', '{0}.conf'.format(lineage))) as file_h: + assert 'renew_hook' in file_h.read() + + +def assert_cert_count_for_lineage(config_dir, lineage, count): + """ + Assert the number of certificates generated for a lineage. + :param config_dir: location of the certbot configuration + :param lineage: lineage domain name + :param count: number of expected certificates + """ + archive_dir = os.path.join(config_dir, 'archive') + lineage_dir = os.path.join(archive_dir, lineage) + certs = [file for file in os.listdir(lineage_dir) if file.startswith('cert')] + assert len(certs) == count + + +def assert_equals_group_permissions(file1, file2): + """ + Assert that two files have the same permissions for group owner. + :param file1: first file path to compare + :param file2: second file path to compare + """ + # On Windows there is no group, so this assertion does nothing on this platform + if POSIX_MODE: + mode_file1 = os.stat(file1).st_mode & 0o070 + mode_file2 = os.stat(file2).st_mode & 0o070 + + assert mode_file1 == mode_file2 + + +def assert_equals_world_read_permissions(file1, file2): + """ + Assert that two files have the same read permissions for everyone. + :param file1: first file path to compare + :param file2: second file path to compare + """ + if POSIX_MODE: + mode_file1 = os.stat(file1).st_mode & 0o004 + mode_file2 = os.stat(file2).st_mode & 0o004 + else: + everybody = win32security.ConvertStringSidToSid(EVERYBODY_SID) + + security1 = win32security.GetFileSecurity(file1, win32security.DACL_SECURITY_INFORMATION) + dacl1 = security1.GetSecurityDescriptorDacl() + + mode_file1 = dacl1.GetEffectiveRightsFromAcl({ + 'TrusteeForm': win32security.TRUSTEE_IS_SID, + 'TrusteeType': win32security.TRUSTEE_IS_USER, + 'Identifier': everybody, + }) + mode_file1 = mode_file1 & ntsecuritycon.FILE_GENERIC_READ + + security2 = win32security.GetFileSecurity(file2, win32security.DACL_SECURITY_INFORMATION) + dacl2 = security2.GetSecurityDescriptorDacl() + + mode_file2 = dacl2.GetEffectiveRightsFromAcl({ + 'TrusteeForm': win32security.TRUSTEE_IS_SID, + 'TrusteeType': win32security.TRUSTEE_IS_USER, + 'Identifier': everybody, + }) + mode_file2 = mode_file2 & ntsecuritycon.FILE_GENERIC_READ + + assert mode_file1 == mode_file2 + + +def assert_equals_group_owner(file1, file2): + """ + Assert that two files have the same group owner. + :param file1: first file path to compare + :param file2: second file path to compare + """ + # On Windows there is no group, so this assertion does nothing on this platform + if POSIX_MODE: + group_owner_file1 = grp.getgrgid(os.stat(file1).st_gid)[0] + group_owner_file2 = grp.getgrgid(os.stat(file2).st_gid)[0] + + assert group_owner_file1 == group_owner_file2 + + +def assert_world_no_permissions(file): + """ + Assert that the given file is not world-readable. + :param file: path of the file to check + """ + if POSIX_MODE: + mode_file_all = os.stat(file).st_mode & 0o007 + assert mode_file_all == 0 + else: + security = win32security.GetFileSecurity(file, win32security.DACL_SECURITY_INFORMATION) + dacl = security.GetSecurityDescriptorDacl() + mode = dacl.GetEffectiveRightsFromAcl({ + 'TrusteeForm': win32security.TRUSTEE_IS_SID, + 'TrusteeType': win32security.TRUSTEE_IS_USER, + 'Identifier': win32security.ConvertStringSidToSid(EVERYBODY_SID), + }) + + assert not mode + + +def assert_world_read_permissions(file): + """ + Assert that the given file is world-readable, but not world-writable or world-executable. + :param file: path of the file to check + """ + if POSIX_MODE: + mode_file_all = os.stat(file).st_mode & 0o007 + assert mode_file_all == 4 + else: + security = win32security.GetFileSecurity(file, win32security.DACL_SECURITY_INFORMATION) + dacl = security.GetSecurityDescriptorDacl() + mode = dacl.GetEffectiveRightsFromAcl({ + 'TrusteeForm': win32security.TRUSTEE_IS_SID, + 'TrusteeType': win32security.TRUSTEE_IS_USER, + 'Identifier': win32security.ConvertStringSidToSid(EVERYBODY_SID), + }) + + assert not mode & ntsecuritycon.FILE_GENERIC_WRITE + assert not mode & ntsecuritycon.FILE_GENERIC_EXECUTE + assert mode & ntsecuritycon.FILE_GENERIC_READ == ntsecuritycon.FILE_GENERIC_READ + + +def _get_current_user(): + account_name = win32api.GetUserNameEx(win32api.NameSamCompatible) + return win32security.LookupAccountName(None, account_name)[0] diff --git a/certbot-ci/certbot_integration_tests/certbot_tests/context.py b/certbot-ci/certbot_integration_tests/certbot_tests/context.py new file mode 100644 index 000000000..6f8670000 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/certbot_tests/context.py @@ -0,0 +1,83 @@ +"""Module to handle the context of integration tests.""" +import logging +import os +import shutil +import sys +import tempfile + +from certbot_integration_tests.utils import certbot_call + + +class IntegrationTestsContext(object): + """General fixture describing a certbot integration tests context""" + def __init__(self, request): + self.request = request + + if hasattr(request.config, 'slaveinput'): # Worker node + self.worker_id = request.config.slaveinput['slaveid'] + acme_xdist = request.config.slaveinput['acme_xdist'] + else: # Primary node + self.worker_id = 'primary' + acme_xdist = request.config.acme_xdist + + self.acme_server = acme_xdist['acme_server'] + self.directory_url = acme_xdist['directory_url'] + self.tls_alpn_01_port = acme_xdist['https_port'][self.worker_id] + self.http_01_port = acme_xdist['http_port'][self.worker_id] + self.other_port = acme_xdist['other_port'][self.worker_id] + # Challtestsrv REST API, that exposes entrypoints to register new DNS entries, + # is listening on challtestsrv_port. + self.challtestsrv_port = acme_xdist['challtestsrv_port'] + + self.workspace = tempfile.mkdtemp() + self.config_dir = os.path.join(self.workspace, 'conf') + + probe = tempfile.mkstemp(dir=self.workspace) + os.close(probe[0]) + self.hook_probe = probe[1] + + self.manual_dns_auth_hook = ( + '{0} -c "import os; import requests; import json; ' + "assert not os.environ.get('CERTBOT_DOMAIN').startswith('fail'); " + "data = {{'host':'_acme-challenge.{{0}}.'.format(os.environ.get('CERTBOT_DOMAIN'))," + "'value':os.environ.get('CERTBOT_VALIDATION')}}; " + "request = requests.post('http://localhost:{1}/set-txt', data=json.dumps(data)); " + "request.raise_for_status(); " + '"' + ).format(sys.executable, self.challtestsrv_port) + self.manual_dns_cleanup_hook = ( + '{0} -c "import os; import requests; import json; ' + "data = {{'host':'_acme-challenge.{{0}}.'.format(os.environ.get('CERTBOT_DOMAIN'))}}; " + "request = requests.post('http://localhost:{1}/clear-txt', data=json.dumps(data)); " + "request.raise_for_status(); " + '"' + ).format(sys.executable, self.challtestsrv_port) + + def cleanup(self): + """Cleanup the integration test context.""" + shutil.rmtree(self.workspace) + + def certbot(self, args, force_renew=True): + """ + Execute certbot with given args, not renewing certificates by default. + :param args: args to pass to certbot + :param force_renew: set to False to not renew by default + :return: output of certbot execution + """ + command = ['--authenticator', 'standalone', '--installer', 'null'] + command.extend(args) + return certbot_call.certbot_test( + command, self.directory_url, self.http_01_port, self.tls_alpn_01_port, + self.config_dir, self.workspace, force_renew=force_renew) + + def get_domain(self, subdomain='le'): + """ + Generate a certificate domain name suitable for distributed certbot integration tests. + This is a requirement to let the distribution know how to redirect the challenge check + from the ACME server to the relevant pytest-xdist worker. This resolution is done by + appending the pytest worker id to the subdomain, using this pattern: + {subdomain}.{worker_id}.wtf + :param subdomain: the subdomain to use in the generated domain (default 'le') + :return: the well-formed domain suitable for redirection on + """ + return '{0}.{1}.wtf'.format(subdomain, self.worker_id) diff --git a/certbot-ci/certbot_integration_tests/certbot_tests/test_main.py b/certbot-ci/certbot_integration_tests/certbot_tests/test_main.py new file mode 100644 index 000000000..94e76cf79 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/certbot_tests/test_main.py @@ -0,0 +1,613 @@ +"""Module executing integration tests against certbot core.""" +from __future__ import print_function + +import os +from os.path import exists +from os.path import join +import re +import shutil +import subprocess +import time + +import pytest + +from certbot_integration_tests.certbot_tests import context as certbot_context +from certbot_integration_tests.certbot_tests.assertions import assert_cert_count_for_lineage +from certbot_integration_tests.certbot_tests.assertions import assert_equals_group_owner +from certbot_integration_tests.certbot_tests.assertions import assert_equals_group_permissions +from certbot_integration_tests.certbot_tests.assertions import assert_equals_world_read_permissions +from certbot_integration_tests.certbot_tests.assertions import assert_hook_execution +from certbot_integration_tests.certbot_tests.assertions import assert_saved_renew_hook +from certbot_integration_tests.certbot_tests.assertions import assert_world_no_permissions +from certbot_integration_tests.certbot_tests.assertions import assert_world_read_permissions +from certbot_integration_tests.certbot_tests.assertions import EVERYBODY_SID +from certbot_integration_tests.utils import misc + + +@pytest.fixture() +def context(request): + # Fixture request is a built-in pytest fixture describing current test request. + integration_test_context = certbot_context.IntegrationTestsContext(request) + try: + yield integration_test_context + finally: + integration_test_context.cleanup() + + +def test_basic_commands(context): + """Test simple commands on Certbot CLI.""" + # TMPDIR env variable is set to workspace for the certbot subprocess. + # So tempdir module will create any temporary files/dirs in workspace, + # and its content can be tested to check correct certbot cleanup. + initial_count_tmpfiles = len(os.listdir(context.workspace)) + + context.certbot(['--help']) + context.certbot(['--help', 'all']) + context.certbot(['--version']) + + with pytest.raises(subprocess.CalledProcessError): + context.certbot(['--csr']) + + new_count_tmpfiles = len(os.listdir(context.workspace)) + assert initial_count_tmpfiles == new_count_tmpfiles + + +def test_hook_dirs_creation(context): + """Test all hooks directory are created during Certbot startup.""" + context.certbot(['register']) + + for hook_dir in misc.list_renewal_hooks_dirs(context.config_dir): + assert os.path.isdir(hook_dir) + + +def test_registration_override(context): + """Test correct register/unregister, and registration override.""" + context.certbot(['register']) + context.certbot(['unregister']) + context.certbot(['register', '--email', 'ex1@domain.org,ex2@domain.org']) + + context.certbot(['update_account', '--email', 'example@domain.org']) + context.certbot(['update_account', '--email', 'ex1@domain.org,ex2@domain.org']) + + +def test_prepare_plugins(context): + """Test that plugins are correctly instantiated and displayed.""" + output = context.certbot(['plugins', '--init', '--prepare']) + + assert 'webroot' in output + + +def test_http_01(context): + """Test the HTTP-01 challenge using standalone plugin.""" + # We start a server listening on the port for the + # TLS-SNI challenge to prevent regressions in #3601. + with misc.create_http_server(context.tls_alpn_01_port): + certname = context.get_domain('le2') + context.certbot([ + '--domains', certname, '--preferred-challenges', 'http-01', 'run', + '--cert-name', certname, + '--pre-hook', misc.echo('wtf_pre', context.hook_probe), + '--post-hook', misc.echo('wtf_post', context.hook_probe), + '--deploy-hook', misc.echo('deploy', context.hook_probe), + ]) + + assert_hook_execution(context.hook_probe, 'deploy') + assert_saved_renew_hook(context.config_dir, certname) + + +def test_manual_http_auth(context): + """Test the HTTP-01 challenge using manual plugin.""" + with misc.create_http_server(context.http_01_port) as webroot,\ + misc.manual_http_hooks(webroot, context.http_01_port) as scripts: + + certname = context.get_domain() + context.certbot([ + 'certonly', '-a', 'manual', '-d', certname, + '--cert-name', certname, + '--manual-auth-hook', scripts[0], + '--manual-cleanup-hook', scripts[1], + '--pre-hook', misc.echo('wtf_pre', context.hook_probe), + '--post-hook', misc.echo('wtf_post', context.hook_probe), + '--renew-hook', misc.echo('renew', context.hook_probe), + ]) + + with pytest.raises(AssertionError): + assert_hook_execution(context.hook_probe, 'renew') + assert_saved_renew_hook(context.config_dir, certname) + + +def test_manual_dns_auth(context): + """Test the DNS-01 challenge using manual plugin.""" + certname = context.get_domain('dns') + context.certbot([ + '-a', 'manual', '-d', certname, '--preferred-challenges', 'dns', + 'run', '--cert-name', certname, + '--manual-auth-hook', context.manual_dns_auth_hook, + '--manual-cleanup-hook', context.manual_dns_cleanup_hook, + '--pre-hook', misc.echo('wtf_pre', context.hook_probe), + '--post-hook', misc.echo('wtf_post', context.hook_probe), + '--renew-hook', misc.echo('renew', context.hook_probe), + ]) + + with pytest.raises(AssertionError): + assert_hook_execution(context.hook_probe, 'renew') + assert_saved_renew_hook(context.config_dir, certname) + + context.certbot(['renew', '--cert-name', certname, '--authenticator', 'manual']) + + assert_cert_count_for_lineage(context.config_dir, certname, 2) + + +def test_certonly(context): + """Test the certonly verb on certbot.""" + context.certbot(['certonly', '--cert-name', 'newname', '-d', context.get_domain('newname')]) + + +def test_auth_and_install_with_csr(context): + """Test certificate issuance and install using an existing CSR.""" + certname = context.get_domain('le3') + key_path = join(context.workspace, 'key.pem') + csr_path = join(context.workspace, 'csr.der') + + misc.generate_csr([certname], key_path, csr_path) + + cert_path = join(context.workspace, 'csr', 'cert.pem') + chain_path = join(context.workspace, 'csr', 'chain.pem') + + context.certbot([ + 'auth', '--csr', csr_path, + '--cert-path', cert_path, + '--chain-path', chain_path + ]) + + print(misc.read_certificate(cert_path)) + print(misc.read_certificate(chain_path)) + + context.certbot([ + '--domains', certname, 'install', + '--cert-path', cert_path, + '--key-path', key_path + ]) + + +def test_renew_files_permissions(context): + """Test proper certificate file permissions upon renewal""" + certname = context.get_domain('renew') + context.certbot(['-d', certname]) + + privkey1 = join(context.config_dir, 'archive', certname, 'privkey1.pem') + privkey2 = join(context.config_dir, 'archive', certname, 'privkey2.pem') + + assert_cert_count_for_lineage(context.config_dir, certname, 1) + assert_world_no_permissions(privkey1) + + context.certbot(['renew']) + + assert_cert_count_for_lineage(context.config_dir, certname, 2) + assert_world_no_permissions(privkey2) + assert_equals_group_owner(privkey1, privkey2) + assert_equals_world_read_permissions(privkey1, privkey2) + assert_equals_group_permissions(privkey1, privkey2) + + +def test_renew_with_hook_scripts(context): + """Test certificate renewal with script hooks.""" + certname = context.get_domain('renew') + context.certbot(['-d', certname]) + + assert_cert_count_for_lineage(context.config_dir, certname, 1) + + misc.generate_test_file_hooks(context.config_dir, context.hook_probe) + context.certbot(['renew']) + + assert_cert_count_for_lineage(context.config_dir, certname, 2) + assert_hook_execution(context.hook_probe, 'deploy') + + +def test_renew_files_propagate_permissions(context): + """Test proper certificate renewal with custom permissions propagated on private key.""" + certname = context.get_domain('renew') + context.certbot(['-d', certname]) + + assert_cert_count_for_lineage(context.config_dir, certname, 1) + + privkey1 = join(context.config_dir, 'archive', certname, 'privkey1.pem') + privkey2 = join(context.config_dir, 'archive', certname, 'privkey2.pem') + + if os.name != 'nt': + os.chmod(privkey1, 0o444) + else: + import win32security + import ntsecuritycon + # Get the current DACL of the private key + security = win32security.GetFileSecurity(privkey1, win32security.DACL_SECURITY_INFORMATION) + dacl = security.GetSecurityDescriptorDacl() + # Create a read permission for Everybody group + everybody = win32security.ConvertStringSidToSid(EVERYBODY_SID) + dacl.AddAccessAllowedAce(win32security.ACL_REVISION, ntsecuritycon.FILE_GENERIC_READ, everybody) + # Apply the updated DACL to the private key + security.SetSecurityDescriptorDacl(1, dacl, 0) + win32security.SetFileSecurity(privkey1, win32security.DACL_SECURITY_INFORMATION, security) + + context.certbot(['renew']) + + assert_cert_count_for_lineage(context.config_dir, certname, 2) + if os.name != 'nt': + # On Linux, read world permissions + all group permissions will be copied from the previous private key + assert_world_read_permissions(privkey2) + assert_equals_world_read_permissions(privkey1, privkey2) + assert_equals_group_permissions(privkey1, privkey2) + else: + # On Windows, world will never have any permissions, and group permission is irrelevant for this platform + assert_world_no_permissions(privkey2) + + +def test_graceful_renew_it_is_not_time(context): + """Test graceful renew is not done when it is not due time.""" + certname = context.get_domain('renew') + context.certbot(['-d', certname]) + + assert_cert_count_for_lineage(context.config_dir, certname, 1) + + context.certbot(['renew', '--deploy-hook', misc.echo('deploy', context.hook_probe)], + force_renew=False) + + assert_cert_count_for_lineage(context.config_dir, certname, 1) + with pytest.raises(AssertionError): + assert_hook_execution(context.hook_probe, 'deploy') + + +def test_graceful_renew_it_is_time(context): + """Test graceful renew is done when it is due time.""" + certname = context.get_domain('renew') + context.certbot(['-d', certname]) + + assert_cert_count_for_lineage(context.config_dir, certname, 1) + + with open(join(context.config_dir, 'renewal', '{0}.conf'.format(certname)), 'r') as file: + lines = file.readlines() + lines.insert(4, 'renew_before_expiry = 100 years{0}'.format(os.linesep)) + with open(join(context.config_dir, 'renewal', '{0}.conf'.format(certname)), 'w') as file: + file.writelines(lines) + + context.certbot(['renew', '--deploy-hook', misc.echo('deploy', context.hook_probe)], + force_renew=False) + + assert_cert_count_for_lineage(context.config_dir, certname, 2) + assert_hook_execution(context.hook_probe, 'deploy') + + +def test_renew_with_changed_private_key_complexity(context): + """Test proper renew with updated private key complexity.""" + certname = context.get_domain('renew') + context.certbot(['-d', certname, '--rsa-key-size', '4096']) + + key1 = join(context.config_dir, 'archive', certname, 'privkey1.pem') + assert os.stat(key1).st_size > 3000 # 4096 bits keys takes more than 3000 bytes + assert_cert_count_for_lineage(context.config_dir, certname, 1) + + context.certbot(['renew']) + + assert_cert_count_for_lineage(context.config_dir, certname, 2) + key2 = join(context.config_dir, 'archive', certname, 'privkey2.pem') + assert os.stat(key2).st_size > 3000 + + context.certbot(['renew', '--rsa-key-size', '2048']) + + assert_cert_count_for_lineage(context.config_dir, certname, 3) + key3 = join(context.config_dir, 'archive', certname, 'privkey3.pem') + assert os.stat(key3).st_size < 1800 # 2048 bits keys takes less than 1800 bytes + + +def test_renew_ignoring_directory_hooks(context): + """Test hooks are ignored during renewal with relevant CLI flag.""" + certname = context.get_domain('renew') + context.certbot(['-d', certname]) + + assert_cert_count_for_lineage(context.config_dir, certname, 1) + + misc.generate_test_file_hooks(context.config_dir, context.hook_probe) + context.certbot(['renew', '--no-directory-hooks']) + + assert_cert_count_for_lineage(context.config_dir, certname, 2) + with pytest.raises(AssertionError): + assert_hook_execution(context.hook_probe, 'deploy') + + +def test_renew_empty_hook_scripts(context): + """Test proper renew with empty hook scripts.""" + certname = context.get_domain('renew') + context.certbot(['-d', certname]) + + assert_cert_count_for_lineage(context.config_dir, certname, 1) + + misc.generate_test_file_hooks(context.config_dir, context.hook_probe) + for hook_dir in misc.list_renewal_hooks_dirs(context.config_dir): + shutil.rmtree(hook_dir) + os.makedirs(join(hook_dir, 'dir')) + open(join(hook_dir, 'file'), 'w').close() + context.certbot(['renew']) + + assert_cert_count_for_lineage(context.config_dir, certname, 2) + + +def test_renew_hook_override(context): + """Test correct hook override on renew.""" + certname = context.get_domain('override') + context.certbot([ + 'certonly', '-d', certname, + '--preferred-challenges', 'http-01', + '--pre-hook', misc.echo('pre', context.hook_probe), + '--post-hook', misc.echo('post', context.hook_probe), + '--deploy-hook', misc.echo('deploy', context.hook_probe), + ]) + + assert_hook_execution(context.hook_probe, 'pre') + assert_hook_execution(context.hook_probe, 'post') + assert_hook_execution(context.hook_probe, 'deploy') + + # Now we override all previous hooks during next renew. + open(context.hook_probe, 'w').close() + context.certbot([ + 'renew', '--cert-name', certname, + '--pre-hook', misc.echo('pre_override', context.hook_probe), + '--post-hook', misc.echo('post_override', context.hook_probe), + '--deploy-hook', misc.echo('deploy_override', context.hook_probe), + ]) + + assert_hook_execution(context.hook_probe, 'pre_override') + assert_hook_execution(context.hook_probe, 'post_override') + assert_hook_execution(context.hook_probe, 'deploy_override') + with pytest.raises(AssertionError): + assert_hook_execution(context.hook_probe, 'pre') + with pytest.raises(AssertionError): + assert_hook_execution(context.hook_probe, 'post') + with pytest.raises(AssertionError): + assert_hook_execution(context.hook_probe, 'deploy') + + # Expect that this renew will reuse new hooks registered in the previous renew. + open(context.hook_probe, 'w').close() + context.certbot(['renew', '--cert-name', certname]) + + assert_hook_execution(context.hook_probe, 'pre_override') + assert_hook_execution(context.hook_probe, 'post_override') + assert_hook_execution(context.hook_probe, 'deploy_override') + + +def test_invalid_domain_with_dns_challenge(context): + """Test certificate issuance failure with DNS-01 challenge.""" + # Manual dns auth hooks from misc are designed to fail if the domain contains 'fail-*'. + domains = ','.join([context.get_domain('dns1'), context.get_domain('fail-dns1')]) + context.certbot([ + '-a', 'manual', '-d', domains, + '--allow-subset-of-names', + '--preferred-challenges', 'dns', + '--manual-auth-hook', context.manual_dns_auth_hook, + '--manual-cleanup-hook', context.manual_dns_cleanup_hook + ]) + + output = context.certbot(['certificates']) + + assert context.get_domain('fail-dns1') not in output + + +def test_reuse_key(context): + """Test various scenarios where a key is reused.""" + certname = context.get_domain('reusekey') + context.certbot(['--domains', certname, '--reuse-key']) + context.certbot(['renew', '--cert-name', certname]) + + with open(join(context.config_dir, 'archive/{0}/privkey1.pem').format(certname), 'r') as file: + privkey1 = file.read() + with open(join(context.config_dir, 'archive/{0}/privkey2.pem').format(certname), 'r') as file: + privkey2 = file.read() + assert privkey1 == privkey2 + + context.certbot(['--cert-name', certname, '--domains', certname, '--force-renewal']) + + with open(join(context.config_dir, 'archive/{0}/privkey3.pem').format(certname), 'r') as file: + privkey3 = file.read() + assert privkey2 != privkey3 + + with open(join(context.config_dir, 'archive/{0}/cert1.pem').format(certname), 'r') as file: + cert1 = file.read() + with open(join(context.config_dir, 'archive/{0}/cert2.pem').format(certname), 'r') as file: + cert2 = file.read() + with open(join(context.config_dir, 'archive/{0}/cert3.pem').format(certname), 'r') as file: + cert3 = file.read() + + assert len({cert1, cert2, cert3}) == 3 + + +def test_ecdsa(context): + """Test certificate issuance with ECDSA key.""" + key_path = join(context.workspace, 'privkey-p384.pem') + csr_path = join(context.workspace, 'csr-p384.der') + cert_path = join(context.workspace, 'cert-p384.pem') + chain_path = join(context.workspace, 'chain-p384.pem') + + misc.generate_csr([context.get_domain('ecdsa')], key_path, csr_path, key_type=misc.ECDSA_KEY_TYPE) + context.certbot(['auth', '--csr', csr_path, '--cert-path', cert_path, '--chain-path', chain_path]) + + certificate = misc.read_certificate(cert_path) + assert 'ASN1 OID: secp384r1' in certificate + + +def test_ocsp_must_staple(context): + """Test that OCSP Must-Staple is correctly set in the generated certificate.""" + if context.acme_server == 'pebble': + pytest.skip('Pebble does not support OCSP Must-Staple.') + + certname = context.get_domain('must-staple') + context.certbot(['auth', '--must-staple', '--domains', certname]) + + certificate = misc.read_certificate(join(context.config_dir, + 'live/{0}/cert.pem').format(certname)) + assert 'status_request' in certificate or '1.3.6.1.5.5.7.1.24' in certificate + + +def test_revoke_simple(context): + """Test various scenarios that revokes a certificate.""" + # Default action after revoke is to delete the certificate. + certname = context.get_domain() + cert_path = join(context.config_dir, 'live', certname, 'cert.pem') + context.certbot(['-d', certname]) + context.certbot(['revoke', '--cert-path', cert_path, '--delete-after-revoke']) + + assert not exists(cert_path) + + # Check default deletion is overridden. + certname = context.get_domain('le1') + cert_path = join(context.config_dir, 'live', certname, 'cert.pem') + context.certbot(['-d', certname]) + context.certbot(['revoke', '--cert-path', cert_path, '--no-delete-after-revoke']) + + assert exists(cert_path) + + context.certbot(['delete', '--cert-name', certname]) + + assert not exists(join(context.config_dir, 'archive', certname)) + assert not exists(join(context.config_dir, 'live', certname)) + assert not exists(join(context.config_dir, 'renewal', '{0}.conf'.format(certname))) + + certname = context.get_domain('le2') + key_path = join(context.config_dir, 'live', certname, 'privkey.pem') + cert_path = join(context.config_dir, 'live', certname, 'cert.pem') + context.certbot(['-d', certname]) + context.certbot(['revoke', '--cert-path', cert_path, '--key-path', key_path]) + + +def test_revoke_and_unregister(context): + """Test revoke with a reason then unregister.""" + cert1 = context.get_domain('le1') + cert2 = context.get_domain('le2') + cert3 = context.get_domain('le3') + + cert_path1 = join(context.config_dir, 'live', cert1, 'cert.pem') + key_path2 = join(context.config_dir, 'live', cert2, 'privkey.pem') + cert_path2 = join(context.config_dir, 'live', cert2, 'cert.pem') + + context.certbot(['-d', cert1]) + context.certbot(['-d', cert2]) + context.certbot(['-d', cert3]) + + context.certbot(['revoke', '--cert-path', cert_path1, + '--reason', 'cessationOfOperation']) + context.certbot(['revoke', '--cert-path', cert_path2, '--key-path', key_path2, + '--reason', 'keyCompromise']) + + context.certbot(['unregister']) + + output = context.certbot(['certificates']) + + assert cert1 not in output + assert cert2 not in output + assert cert3 in output + + +def test_revoke_mutual_exclusive_flags(context): + """Test --cert-path and --cert-name cannot be used during revoke.""" + cert = context.get_domain('le1') + context.certbot(['-d', cert]) + with pytest.raises(subprocess.CalledProcessError) as error: + context.certbot([ + 'revoke', '--cert-name', cert, + '--cert-path', join(context.config_dir, 'live', cert, 'fullchain.pem') + ]) + assert 'Exactly one of --cert-path or --cert-name must be specified' in error.out + + +def test_revoke_multiple_lineages(context): + """Test revoke does not delete certs if multiple lineages share the same dir.""" + cert1 = context.get_domain('le1') + context.certbot(['-d', cert1]) + + assert os.path.isfile(join(context.config_dir, 'renewal', '{0}.conf'.format(cert1))) + + cert2 = context.get_domain('le2') + context.certbot(['-d', cert2]) + + # Copy over renewal configuration of cert1 into renewal configuration of cert2. + with open(join(context.config_dir, 'renewal', '{0}.conf'.format(cert2)), 'r') as file: + data = file.read() + + data = re.sub('archive_dir = .*\n', + 'archive_dir = {0}\n'.format(join(context.config_dir, 'archive', cert1).replace('\\', '\\\\')), + data) + + with open(join(context.config_dir, 'renewal', '{0}.conf'.format(cert2)), 'w') as file: + file.write(data) + + output = context.certbot([ + 'revoke', '--cert-path', join(context.config_dir, 'live', cert1, 'cert.pem') + ]) + + assert 'Not deleting revoked certs due to overlapping archive dirs' in output + + +def test_wildcard_certificates(context): + """Test wildcard certificate issuance.""" + if context.acme_server == 'boulder-v1': + pytest.skip('Wildcard certificates are not supported on ACME v1') + + certname = context.get_domain('wild') + + context.certbot([ + '-a', 'manual', '-d', '*.{0},{0}'.format(certname), + '--preferred-challenge', 'dns', + '--manual-auth-hook', context.manual_dns_auth_hook, + '--manual-cleanup-hook', context.manual_dns_cleanup_hook + ]) + + assert exists(join(context.config_dir, 'live', certname, 'fullchain.pem')) + + +def test_ocsp_status_stale(context): + """Test retrieval of OCSP statuses for staled config""" + sample_data_path = misc.load_sample_data_path(context.workspace) + output = context.certbot(['certificates', '--config-dir', sample_data_path]) + + assert output.count('TEST_CERT') == 2, ('Did not find two test certs as expected ({0})' + .format(output.count('TEST_CERT'))) + assert output.count('EXPIRED') == 2, ('Did not find two expired certs as expected ({0})' + .format(output.count('EXPIRED'))) + + +def test_ocsp_status_live(context): + """Test retrieval of OCSP statuses for live config""" + cert = context.get_domain('ocsp-check') + + # OSCP 1: Check live certificate OCSP status (VALID) + context.certbot(['--domains', cert]) + output = context.certbot(['certificates']) + + assert output.count('VALID') == 1, 'Expected {0} to be VALID'.format(cert) + assert output.count('EXPIRED') == 0, 'Did not expect {0} to be EXPIRED'.format(cert) + + # OSCP 2: Check live certificate OCSP status (REVOKED) + context.certbot(['revoke', '--cert-name', cert, '--no-delete-after-revoke']) + # Sometimes in oldest tests (using openssl binary and not cryptography), the OCSP status is + # not seen immediately by Certbot as invalid. Waiting few seconds solves this transient issue. + time.sleep(5) + output = context.certbot(['certificates']) + + assert output.count('INVALID') == 1, 'Expected {0} to be INVALID'.format(cert) + assert output.count('REVOKED') == 1, 'Expected {0} to be REVOKED'.format(cert) + + +def test_dry_run_deactivate_authzs(context): + """Test that Certbot deactivates authorizations when performing a dry run""" + + name = context.get_domain('dry-run-authz-deactivation') + args = ['certonly', '--cert-name', name, '-d', name, '--dry-run'] + log_line = 'Recreating order after authz deactivation' + + # First order will not need deactivation + context.certbot(args) + with open(join(context.workspace, 'logs', 'letsencrypt.log'), 'r') as f: + assert log_line not in f.read(), 'First order should not have had any authz reuse' + + # Second order will require deactivation + context.certbot(args) + with open(join(context.workspace, 'logs', 'letsencrypt.log'), 'r') as f: + assert log_line in f.read(), 'Second order should have been recreated due to authz reuse' diff --git a/certbot-ci/certbot_integration_tests/conftest.py b/certbot-ci/certbot_integration_tests/conftest.py new file mode 100644 index 000000000..bb1d76e57 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/conftest.py @@ -0,0 +1,96 @@ +""" +General conftest for pytest execution of all integration tests lying +in the certbot_integration tests package. +As stated by pytest documentation, conftest module is used to set on +for a directory a specific configuration using built-in pytest hooks. + +See https://docs.pytest.org/en/latest/reference.html#hook-reference +""" +from __future__ import print_function +import contextlib +import subprocess +import sys + +from certbot_integration_tests.utils import acme_server as acme_lib + + +def pytest_addoption(parser): + """ + Standard pytest hook to add options to the pytest parser. + :param parser: current pytest parser that will be used on the CLI + """ + parser.addoption('--acme-server', default='pebble', + choices=['boulder-v1', 'boulder-v2', 'pebble'], + help='select the ACME server to use (boulder-v1, boulder-v2, ' + 'pebble), defaulting to pebble') + + +def pytest_configure(config): + """ + Standard pytest hook used to add a configuration logic for each node of a pytest run. + :param config: the current pytest configuration + """ + if not hasattr(config, 'slaveinput'): # If true, this is the primary node + with _print_on_err(): + config.acme_xdist = _setup_primary_node(config) + + +def pytest_configure_node(node): + """ + Standard pytest-xdist hook used to configure a worker node. + :param node: current worker node + """ + node.slaveinput['acme_xdist'] = node.config.acme_xdist + + +@contextlib.contextmanager +def _print_on_err(): + """ + During pytest-xdist setup, stdout is used for nodes communication, so print is useless. + However, stderr is still available. This context manager transfers stdout to stderr + for the duration of the context, allowing to display prints to the user. + """ + old_stdout = sys.stdout + sys.stdout = sys.stderr + try: + yield + finally: + sys.stdout = old_stdout + + +def _setup_primary_node(config): + """ + Setup the environment for integration tests. + Will: + - check runtime compatibility (Docker, docker-compose, Nginx) + - create a temporary workspace and the persistent GIT repositories space + - configure and start paralleled ACME CA servers using Docker + - transfer ACME CA servers configurations to pytest nodes using env variables + :param config: Configuration of the pytest primary node + """ + # Check for runtime compatibility: some tools are required to be available in PATH + if 'boulder' in config.option.acme_server: + try: + subprocess.check_output(['docker', '-v'], stderr=subprocess.STDOUT) + except (subprocess.CalledProcessError, OSError): + raise ValueError('Error: docker is required in PATH to launch the integration tests on' + 'boulder, but is not installed or not available for current user.') + + try: + subprocess.check_output(['docker-compose', '-v'], stderr=subprocess.STDOUT) + except (subprocess.CalledProcessError, OSError): + raise ValueError('Error: docker-compose is required in PATH to launch the integration tests, ' + 'but is not installed or not available for current user.') + + # Parameter numprocesses is added to option by pytest-xdist + workers = ['primary'] if not config.option.numprocesses\ + else ['gw{0}'.format(i) for i in range(config.option.numprocesses)] + + # By calling setup_acme_server we ensure that all necessary acme server instances will be + # fully started. This runtime is reflected by the acme_xdist returned. + acme_server = acme_lib.ACMEServer(config.option.acme_server, workers) + config.add_cleanup(acme_server.stop) + print('ACME xdist config:\n{0}'.format(acme_server.acme_xdist)) + acme_server.start() + + return acme_server.acme_xdist diff --git a/certbot-apache/docs/_static/.gitignore b/certbot-ci/certbot_integration_tests/nginx_tests/__init__.py similarity index 100% rename from certbot-apache/docs/_static/.gitignore rename to certbot-ci/certbot_integration_tests/nginx_tests/__init__.py diff --git a/certbot-ci/certbot_integration_tests/nginx_tests/context.py b/certbot-ci/certbot_integration_tests/nginx_tests/context.py new file mode 100644 index 000000000..3a769840c --- /dev/null +++ b/certbot-ci/certbot_integration_tests/nginx_tests/context.py @@ -0,0 +1,62 @@ +import os +import subprocess + +from certbot_integration_tests.certbot_tests import context as certbot_context +from certbot_integration_tests.nginx_tests import nginx_config as config +from certbot_integration_tests.utils import certbot_call +from certbot_integration_tests.utils import misc + + +class IntegrationTestsContext(certbot_context.IntegrationTestsContext): + """General fixture describing a certbot-nginx integration tests context""" + def __init__(self, request): + super(IntegrationTestsContext, self).__init__(request) + + self.nginx_root = os.path.join(self.workspace, 'nginx') + os.mkdir(self.nginx_root) + + self.webroot = os.path.join(self.nginx_root, 'webroot') + os.mkdir(self.webroot) + with open(os.path.join(self.webroot, 'index.html'), 'w') as file_handler: + file_handler.write('Hello World!') + + self.nginx_config_path = os.path.join(self.nginx_root, 'nginx.conf') + self.nginx_config = None + + default_server = request.param['default_server'] + self.process = self._start_nginx(default_server) + + def cleanup(self): + self._stop_nginx() + super(IntegrationTestsContext, self).cleanup() + + def certbot_test_nginx(self, args): + """ + Main command to execute certbot using the nginx plugin. + :param list args: list of arguments to pass to nginx + :param bool force_renew: set to False to not renew by default + """ + command = ['--authenticator', 'nginx', '--installer', 'nginx', + '--nginx-server-root', self.nginx_root] + command.extend(args) + return certbot_call.certbot_test( + command, self.directory_url, self.http_01_port, self.tls_alpn_01_port, + self.config_dir, self.workspace, force_renew=True) + + def _start_nginx(self, default_server): + self.nginx_config = config.construct_nginx_config( + self.nginx_root, self.webroot, self.http_01_port, self.tls_alpn_01_port, + self.other_port, default_server, wtf_prefix=self.worker_id) + with open(self.nginx_config_path, 'w') as file: + file.write(self.nginx_config) + + process = subprocess.Popen(['nginx', '-c', self.nginx_config_path, '-g', 'daemon off;']) + + assert process.poll() is None + misc.check_until_timeout('http://localhost:{0}'.format(self.http_01_port)) + return process + + def _stop_nginx(self): + assert self.process.poll() is None + self.process.terminate() + self.process.wait() diff --git a/certbot-ci/certbot_integration_tests/nginx_tests/nginx_config.py b/certbot-ci/certbot_integration_tests/nginx_tests/nginx_config.py new file mode 100644 index 000000000..18991ae62 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/nginx_tests/nginx_config.py @@ -0,0 +1,126 @@ +"""General purpose nginx test configuration generator.""" +import getpass + +import pkg_resources + + +def construct_nginx_config(nginx_root, nginx_webroot, http_port, https_port, other_port, + default_server, key_path=None, cert_path=None, wtf_prefix='le'): + """ + This method returns a full nginx configuration suitable for integration tests. + :param str nginx_root: nginx root configuration path + :param str nginx_webroot: nginx webroot path + :param int http_port: HTTP port to listen on + :param int https_port: HTTPS port to listen on + :param int other_port: other HTTP port to listen on + :param bool default_server: True to set a default server in nginx config, False otherwise + :param str key_path: the path to a SSL key + :param str cert_path: the path to a SSL certificate + :param str wtf_prefix: the prefix to use in all domains handled by this nginx config + :return: a string containing the full nginx configuration + :rtype: str + """ + key_path = key_path if key_path \ + else pkg_resources.resource_filename('certbot_integration_tests', 'assets/key.pem') + cert_path = cert_path if cert_path \ + else pkg_resources.resource_filename('certbot_integration_tests', 'assets/cert.pem') + return '''\ +# This error log will be written regardless of server scope error_log +# definitions, so we have to set this here in the main scope. +# +# Even doing this, Nginx will still try to create the default error file, and +# log a non-fatal error when it fails. After that things will work, however. +error_log {nginx_root}/error.log; + +# The pidfile will be written to /var/run unless this is set. +pid {nginx_root}/nginx.pid; + +user {user}; +worker_processes 1; + +events {{ + worker_connections 1024; +}} + +http {{ + # Set an array of temp, cache and log file options that will otherwise default to + # restricted locations accessible only to root. + client_body_temp_path {nginx_root}/client_body; + fastcgi_temp_path {nginx_root}/fastcgi_temp; + proxy_temp_path {nginx_root}/proxy_temp; + #scgi_temp_path {nginx_root}/scgi_temp; + #uwsgi_temp_path {nginx_root}/uwsgi_temp; + access_log {nginx_root}/error.log; + + # This should be turned off in a Virtualbox VM, as it can cause some + # interesting issues with data corruption in delivered files. + sendfile off; + + tcp_nopush on; + tcp_nodelay on; + keepalive_timeout 65; + types_hash_max_size 2048; + + #include /etc/nginx/mime.types; + index index.html index.htm index.php; + + log_format main '$remote_addr - $remote_user [$time_local] $status ' + '"$request" $body_bytes_sent "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for"'; + + default_type application/octet-stream; + + server {{ + # IPv4. + listen {http_port} {default_server}; + # IPv6. + listen [::]:{http_port} {default_server}; + server_name nginx.{wtf_prefix}.wtf nginx2.{wtf_prefix}.wtf; + + root {nginx_webroot}; + + location / {{ + # First attempt to serve request as file, then as directory, then fall + # back to index.html. + try_files $uri $uri/ /index.html; + }} + }} + + server {{ + listen {http_port}; + listen [::]:{http_port}; + server_name nginx3.{wtf_prefix}.wtf; + + root {nginx_webroot}; + + location /.well-known/ {{ + return 404; + }} + + return 301 https://$host$request_uri; + }} + + server {{ + listen {other_port}; + listen [::]:{other_port}; + server_name nginx4.{wtf_prefix}.wtf nginx5.{wtf_prefix}.wtf; + }} + + server {{ + listen {http_port}; + listen [::]:{http_port}; + listen {https_port} ssl; + listen [::]:{https_port} ssl; + if ($scheme != "https") {{ + return 301 https://$host$request_uri; + }} + server_name nginx6.{wtf_prefix}.wtf nginx7.{wtf_prefix}.wtf; + + ssl_certificate {cert_path}; + ssl_certificate_key {key_path}; + }} +}} +'''.format(nginx_root=nginx_root, nginx_webroot=nginx_webroot, user=getpass.getuser(), + http_port=http_port, https_port=https_port, other_port=other_port, + default_server='default_server' if default_server else '', wtf_prefix=wtf_prefix, + key_path=key_path, cert_path=cert_path) diff --git a/certbot-ci/certbot_integration_tests/nginx_tests/test_main.py b/certbot-ci/certbot_integration_tests/nginx_tests/test_main.py new file mode 100644 index 000000000..1a62ea8d7 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/nginx_tests/test_main.py @@ -0,0 +1,54 @@ +"""Module executing integration tests against certbot with nginx plugin.""" +import os +import ssl + +import pytest + +from certbot_integration_tests.nginx_tests import context as nginx_context + + +@pytest.fixture() +def context(request): + # Fixture request is a built-in pytest fixture describing current test request. + integration_test_context = nginx_context.IntegrationTestsContext(request) + try: + yield integration_test_context + finally: + integration_test_context.cleanup() + + +@pytest.mark.parametrize('certname_pattern, params, context', [ + ('nginx.{0}.wtf', ['run'], {'default_server': True}), + ('nginx2.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': True}), + # Overlapping location block and server-block-level return 301 + ('nginx3.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': True}), + # No matching server block; default_server exists + ('nginx4.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': True}), + # No matching server block; default_server does not exist + ('nginx5.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': False}), + # Multiple domains, mix of matching and not + ('nginx6.{0}.wtf,nginx7.{0}.wtf', ['--preferred-challenges', 'http'], {'default_server': False}), +], indirect=['context']) +def test_certificate_deployment(certname_pattern, params, context): + # type: (str, list, nginx_context.IntegrationTestsContext) -> None + """ + Test various scenarios to deploy a certificate to nginx using certbot. + """ + domains = certname_pattern.format(context.worker_id) + command = ['--domains', domains] + command.extend(params) + context.certbot_test_nginx(command) + + lineage = domains.split(',')[0] + server_cert = ssl.get_server_certificate(('localhost', context.tls_alpn_01_port)) + with open(os.path.join(context.workspace, 'conf/live/{0}/cert.pem'.format(lineage)), 'r') as file: + certbot_cert = file.read() + + assert server_cert == certbot_cert + + context.certbot_test_nginx(['rollback', '--checkpoints', '1']) + + with open(context.nginx_config_path, 'r') as file_h: + current_nginx_config = file_h.read() + + assert context.nginx_config == current_nginx_config diff --git a/certbot-apache/docs/_templates/.gitignore b/certbot-ci/certbot_integration_tests/utils/__init__.py similarity index 100% rename from certbot-apache/docs/_templates/.gitignore rename to certbot-ci/certbot_integration_tests/utils/__init__.py diff --git a/certbot-ci/certbot_integration_tests/utils/acme_server.py b/certbot-ci/certbot_integration_tests/utils/acme_server.py new file mode 100755 index 000000000..5483251e6 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/utils/acme_server.py @@ -0,0 +1,223 @@ +#!/usr/bin/env python +"""Module to setup an ACME CA server environment able to run multiple tests in parallel""" +from __future__ import print_function + +import errno +import json +import os +from os.path import join +import shutil +import subprocess +import sys +import tempfile +import time + +import requests + +from certbot_integration_tests.utils import misc +from certbot_integration_tests.utils import pebble_artifacts +from certbot_integration_tests.utils import proxy +from certbot_integration_tests.utils.constants import * + + +class ACMEServer(object): + """ + ACMEServer configures and handles the lifecycle of an ACME CA server and an HTTP reverse proxy + instance, to allow parallel execution of integration tests against the unique http-01 port + expected by the ACME CA server. + Typically all pytest integration tests will be executed in this context. + ACMEServer gives access the acme_xdist parameter, listing the ports and directory url to use + for each pytest node. It exposes also start and stop methods in order to start the stack, and + stop it with proper resources cleanup. + ACMEServer is also a context manager, and so can be used to ensure ACME server is started/stopped + upon context enter/exit. + """ + def __init__(self, acme_server, nodes, http_proxy=True, stdout=False): + """ + Create an ACMEServer instance. + :param str acme_server: the type of acme server used (boulder-v1, boulder-v2 or pebble) + :param list nodes: list of node names that will be setup by pytest xdist + :param bool http_proxy: if False do not start the HTTP proxy + :param bool stdout: if True stream subprocesses stdout to standard stdout + """ + self._construct_acme_xdist(acme_server, nodes) + + self._acme_type = 'pebble' if acme_server == 'pebble' else 'boulder' + self._proxy = http_proxy + self._workspace = tempfile.mkdtemp() + self._processes = [] + self._stdout = sys.stdout if stdout else open(os.devnull, 'w') + + def start(self): + """Start the test stack""" + try: + if self._proxy: + self._prepare_http_proxy() + if self._acme_type == 'pebble': + self._prepare_pebble_server() + if self._acme_type == 'boulder': + self._prepare_boulder_server() + except BaseException as e: + self.stop() + raise e + + def stop(self): + """Stop the test stack, and clean its resources""" + print('=> Tear down the test infrastructure...') + try: + for process in self._processes: + try: + process.terminate() + except OSError as e: + # Process may be not started yet, so no PID and terminate fails. + # Then the process never started, and the situation is acceptable. + if e.errno != errno.ESRCH: + raise + for process in self._processes: + process.wait() + + if os.path.exists(os.path.join(self._workspace, 'boulder')): + # Boulder docker generates build artifacts owned by root with 0o744 permissions. + # If we started the acme server from a normal user that has access to the Docker + # daemon, this user will not be able to delete these artifacts from the host. + # We need to do it through a docker. + process = self._launch_process(['docker', 'run', '--rm', '-v', + '{0}:/workspace'.format(self._workspace), + 'alpine', 'rm', '-rf', '/workspace/boulder']) + process.wait() + finally: + shutil.rmtree(self._workspace) + if self._stdout != sys.stdout: + self._stdout.close() + print('=> Test infrastructure stopped and cleaned up.') + + def __enter__(self): + self.start() + return self.acme_xdist + + def __exit__(self, exc_type, exc_val, exc_tb): + self.stop() + + def _construct_acme_xdist(self, acme_server, nodes): + """Generate and return the acme_xdist dict""" + acme_xdist = {'acme_server': acme_server, 'challtestsrv_port': CHALLTESTSRV_PORT} + + # Directory and ACME port are set implicitly in the docker-compose.yml files of Boulder/Pebble. + if acme_server == 'pebble': + acme_xdist['directory_url'] = PEBBLE_DIRECTORY_URL + else: # boulder + acme_xdist['directory_url'] = BOULDER_V2_DIRECTORY_URL \ + if acme_server == 'boulder-v2' else BOULDER_V1_DIRECTORY_URL + + acme_xdist['http_port'] = {node: port for (node, port) + in zip(nodes, range(5200, 5200 + len(nodes)))} + acme_xdist['https_port'] = {node: port for (node, port) + in zip(nodes, range(5100, 5100 + len(nodes)))} + acme_xdist['other_port'] = {node: port for (node, port) + in zip(nodes, range(5300, 5300 + len(nodes)))} + + self.acme_xdist = acme_xdist + + def _prepare_pebble_server(self): + """Configure and launch the Pebble server""" + print('=> Starting pebble instance deployment...') + pebble_path, challtestsrv_path, pebble_config_path = pebble_artifacts.fetch(self._workspace) + + # Configure Pebble at full speed (PEBBLE_VA_NOSLEEP=1) and not randomly refusing valid + # nonce (PEBBLE_WFE_NONCEREJECT=0) to have a stable test environment. + environ = os.environ.copy() + environ['PEBBLE_VA_NOSLEEP'] = '1' + environ['PEBBLE_WFE_NONCEREJECT'] = '0' + environ['PEBBLE_AUTHZREUSE'] = '100' + + self._launch_process( + [pebble_path, '-config', pebble_config_path, '-dnsserver', '127.0.0.1:8053'], + env=environ) + + self._launch_process( + [challtestsrv_path, '-management', ':{0}'.format(CHALLTESTSRV_PORT), '-defaultIPv6', '""', + '-defaultIPv4', '127.0.0.1', '-http01', '""', '-tlsalpn01', '""', '-https01', '""']) + + # pebble_ocsp_server is imported here and not at the top of module in order to avoid a useless + # ImportError, in the case where cryptography dependency is too old to support ocsp, but + # Boulder is used instead of Pebble, so pebble_ocsp_server is not used. This is the typical + # situation of integration-certbot-oldest tox testenv. + from certbot_integration_tests.utils import pebble_ocsp_server + self._launch_process([sys.executable, pebble_ocsp_server.__file__]) + + # Wait for the ACME CA server to be up. + print('=> Waiting for pebble instance to respond...') + misc.check_until_timeout(self.acme_xdist['directory_url']) + + print('=> Finished pebble instance deployment.') + + def _prepare_boulder_server(self): + """Configure and launch the Boulder server""" + print('=> Starting boulder instance deployment...') + instance_path = join(self._workspace, 'boulder') + + # Load Boulder from git, that includes a docker-compose.yml ready for production. + process = self._launch_process(['git', 'clone', 'https://github.com/letsencrypt/boulder', + '--single-branch', '--depth=1', instance_path]) + process.wait() + + # Allow Boulder to ignore usual limit rate policies, useful for tests. + os.rename(join(instance_path, 'test/rate-limit-policies-b.yml'), + join(instance_path, 'test/rate-limit-policies.yml')) + + # Launch the Boulder server + self._launch_process(['docker-compose', 'up', '--force-recreate'], cwd=instance_path) + + # Wait for the ACME CA server to be up. + print('=> Waiting for boulder instance to respond...') + misc.check_until_timeout(self.acme_xdist['directory_url'], attempts=240) + + # Configure challtestsrv to answer any A record request with ip of the docker host. + response = requests.post('http://localhost:{0}/set-default-ipv4'.format(CHALLTESTSRV_PORT), + json={'ip': '10.77.77.1'}) + response.raise_for_status() + + print('=> Finished boulder instance deployment.') + + def _prepare_http_proxy(self): + """Configure and launch an HTTP proxy""" + print('=> Configuring the HTTP proxy...') + mapping = {r'.+\.{0}\.wtf'.format(node): 'http://127.0.0.1:{0}'.format(port) + for node, port in self.acme_xdist['http_port'].items()} + command = [sys.executable, proxy.__file__, str(HTTP_01_PORT), json.dumps(mapping)] + self._launch_process(command) + print('=> Finished configuring the HTTP proxy.') + + def _launch_process(self, command, cwd=os.getcwd(), env=None): + """Launch silently a subprocess OS command""" + if not env: + env = os.environ + process = subprocess.Popen(command, stdout=self._stdout, stderr=subprocess.STDOUT, cwd=cwd, env=env) + self._processes.append(process) + return process + + +def main(): + args = sys.argv[1:] + server_type = args[0] if args else 'pebble' + possible_values = ('pebble', 'boulder-v1', 'boulder-v2') + if server_type not in possible_values: + raise ValueError('Invalid server value {0}, should be one of {1}' + .format(server_type, possible_values)) + + acme_server = ACMEServer(server_type, [], http_proxy=False, stdout=True) + + try: + with acme_server as acme_xdist: + print('--> Instance of {0} is running, directory URL is {0}' + .format(acme_xdist['directory_url'])) + print('--> Press CTRL+C to stop the ACME server.') + + while True: + time.sleep(3600) + except KeyboardInterrupt: + pass + + +if __name__ == '__main__': + main() diff --git a/certbot-ci/certbot_integration_tests/utils/certbot_call.py b/certbot-ci/certbot_integration_tests/utils/certbot_call.py new file mode 100755 index 000000000..2ddaa41c8 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/utils/certbot_call.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python +"""Module to call certbot in test mode""" +from __future__ import absolute_import + +from distutils.version import LooseVersion +import os +import subprocess +import sys + +import certbot_integration_tests +from certbot_integration_tests.utils.constants import * + + +def certbot_test(certbot_args, directory_url, http_01_port, tls_alpn_01_port, + config_dir, workspace, force_renew=True): + """ + Invoke the certbot executable available in PATH in a test context for the given args. + The test context consists in running certbot in debug mode, with various flags suitable + for tests (eg. no ssl check, customizable ACME challenge ports and config directory ...). + This command captures stdout and returns it to the caller. + :param list certbot_args: the arguments to pass to the certbot executable + :param str directory_url: URL of the ACME directory server to use + :param int http_01_port: port for the HTTP-01 challenges + :param int tls_alpn_01_port: port for the TLS-ALPN-01 challenges + :param str config_dir: certbot configuration directory to use + :param str workspace: certbot current directory to use + :param bool force_renew: set False to not force renew existing certificates (default: True) + :return: stdout as string + :rtype: str + """ + command, env = _prepare_args_env(certbot_args, directory_url, http_01_port, tls_alpn_01_port, + config_dir, workspace, force_renew) + + return subprocess.check_output(command, universal_newlines=True, cwd=workspace, env=env) + + +def _prepare_environ(workspace): + new_environ = os.environ.copy() + new_environ['TMPDIR'] = workspace + + # So, pytest is nice, and a little too nice for our usage. + # In order to help user to call seamlessly any piece of python code without requiring to + # install it as a full-fledged setuptools distribution for instance, it may inject the path + # to the test files into the PYTHONPATH. This allows the python interpreter to import + # as modules any python file available at this path. + # See https://docs.pytest.org/en/3.2.5/pythonpath.html for the explanation and description. + # However this behavior is not good in integration tests, in particular the nginx oldest ones. + # Indeed during these kind of tests certbot is installed as a transitive dependency to + # certbot-nginx. Here is the trick: this certbot version is not necessarily the same as + # the certbot codebase lying in current working directory. For instance in oldest tests + # certbot==0.36.0 may be installed while the codebase corresponds to certbot==0.37.0.dev0. + # Then during a pytest run, PYTHONPATH contains the path to the Certbot codebase, so invoking + # certbot will import the modules from the codebase (0.37.0.dev0), not from the + # required/installed version (0.36.0). + # This will lead to funny and totally incomprehensible errors. To avoid that, we ensure that + # if PYTHONPATH is set, it does not contain the path to the root of the codebase. + if new_environ.get('PYTHONPATH'): + # certbot_integration_tests.__file__ is: + # '/path/to/certbot/certbot-ci/certbot_integration_tests/__init__.pyc' + # ... and we want '/path/to/certbot' + certbot_root = os.path.dirname(os.path.dirname(os.path.dirname(certbot_integration_tests.__file__))) + python_paths = [path for path in new_environ['PYTHONPATH'].split(':') if path != certbot_root] + new_environ['PYTHONPATH'] = ':'.join(python_paths) + + return new_environ + + +def _compute_additional_args(workspace, environ, force_renew): + additional_args = [] + output = subprocess.check_output(['certbot', '--version'], + universal_newlines=True, stderr=subprocess.STDOUT, + cwd=workspace, env=environ) + version_str = output.split(' ')[1].strip() # Typical response is: output = 'certbot 0.31.0.dev0' + if LooseVersion(version_str) >= LooseVersion('0.30.0'): + additional_args.append('--no-random-sleep-on-renew') + + if force_renew: + additional_args.append('--renew-by-default') + + return additional_args + + +def _prepare_args_env(certbot_args, directory_url, http_01_port, tls_alpn_01_port, + config_dir, workspace, force_renew): + + new_environ = _prepare_environ(workspace) + additional_args = _compute_additional_args(workspace, new_environ, force_renew) + + command = [ + 'certbot', + '--server', directory_url, + '--no-verify-ssl', + '--http-01-port', str(http_01_port), + '--https-port', str(tls_alpn_01_port), + '--manual-public-ip-logging-ok', + '--config-dir', config_dir, + '--work-dir', os.path.join(workspace, 'work'), + '--logs-dir', os.path.join(workspace, 'logs'), + '--non-interactive', + '--no-redirect', + '--agree-tos', + '--register-unsafely-without-email', + '--debug', + '-vv' + ] + + command.extend(certbot_args) + command.extend(additional_args) + + print('--> Invoke command:\n=====\n{0}\n====='.format(subprocess.list2cmdline(command))) + + return command, new_environ + + +def main(): + args = sys.argv[1:] + + # Default config is pebble + directory_url = os.environ.get('SERVER', PEBBLE_DIRECTORY_URL) + http_01_port = int(os.environ.get('HTTP_01_PORT', HTTP_01_PORT)) + tls_alpn_01_port = int(os.environ.get('TLS_ALPN_01_PORT', TLS_ALPN_01_PORT)) + + # Execution of certbot in a self-contained workspace + workspace = os.environ.get('WORKSPACE', os.path.join(os.getcwd(), '.certbot_test_workspace')) + if not os.path.exists(workspace): + print('--> Creating a workspace for certbot_test: {0}'.format(workspace)) + os.mkdir(workspace) + else: + print('--> Using an existing workspace for certbot_test: {0}'.format(workspace)) + config_dir = os.path.join(workspace, 'conf') + + # Invoke certbot in test mode, without capturing output so users see directly the outcome. + command, env = _prepare_args_env(args, directory_url, http_01_port, tls_alpn_01_port, + config_dir, workspace, True) + subprocess.check_call(command, universal_newlines=True, cwd=workspace, env=env) + + +if __name__ == '__main__': + main() diff --git a/certbot-ci/certbot_integration_tests/utils/constants.py b/certbot-ci/certbot_integration_tests/utils/constants.py new file mode 100644 index 000000000..dfdeda411 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/utils/constants.py @@ -0,0 +1,9 @@ +"""Some useful constants to use throughout certbot-ci integration tests""" +HTTP_01_PORT = 5002 +TLS_ALPN_01_PORT = 5001 +CHALLTESTSRV_PORT = 8055 +BOULDER_V1_DIRECTORY_URL = 'http://localhost:4000/directory' +BOULDER_V2_DIRECTORY_URL = 'http://localhost:4001/directory' +PEBBLE_DIRECTORY_URL = 'https://localhost:14000/dir' +PEBBLE_MANAGEMENT_URL = 'https://localhost:15000' +MOCK_OCSP_SERVER_PORT = 4002 diff --git a/certbot-ci/certbot_integration_tests/utils/misc.py b/certbot-ci/certbot_integration_tests/utils/misc.py new file mode 100644 index 000000000..b08f11e89 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/utils/misc.py @@ -0,0 +1,302 @@ +""" +Misc module contains stateless functions that could be used during pytest execution, +or outside during setup/teardown of the integration tests environment. +""" +import contextlib +import errno +import multiprocessing +import os +import re +import shutil +import stat +import sys +import tempfile +import time +import warnings + +from cryptography.hazmat.backends import default_backend +from cryptography.hazmat.primitives.asymmetric import ec +from cryptography.hazmat.primitives.serialization import Encoding +from cryptography.hazmat.primitives.serialization import NoEncryption +from cryptography.hazmat.primitives.serialization import PrivateFormat +from OpenSSL import crypto +import pkg_resources +import requests +from six.moves import SimpleHTTPServer +from six.moves import socketserver + +RSA_KEY_TYPE = 'rsa' +ECDSA_KEY_TYPE = 'ecdsa' + + +def check_until_timeout(url, attempts=30): + """ + Wait and block until given url responds with status 200, or raise an exception + after the specified number of attempts. + :param str url: the URL to test + :param int attempts: the number of times to try to connect to the URL + :raise ValueError: exception raised if unable to reach the URL + """ + try: + import urllib3 + urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + except ImportError: + # Handle old versions of request with vendorized urllib3 + from requests.packages.urllib3.exceptions import InsecureRequestWarning + requests.packages.urllib3.disable_warnings(InsecureRequestWarning) + + for _ in range(attempts): + time.sleep(1) + try: + if requests.get(url, verify=False).status_code == 200: + return + except requests.exceptions.ConnectionError: + pass + + raise ValueError('Error, url did not respond after {0} attempts: {1}'.format(attempts, url)) + + +class GracefulTCPServer(socketserver.TCPServer): + """ + This subclass of TCPServer allows graceful reuse of an address that has + just been released by another instance of TCPServer. + """ + allow_reuse_address = True + + +def _run_server(port): + GracefulTCPServer(('', port), SimpleHTTPServer.SimpleHTTPRequestHandler).serve_forever() + + +@contextlib.contextmanager +def create_http_server(port): + """ + Setup and start an HTTP server for the given TCP port. + This server stays active for the lifetime of the context, and is automatically + stopped with context exit, while its temporary webroot is deleted. + :param int port: the TCP port to use + :return str: the temporary webroot attached to this server + """ + current_cwd = os.getcwd() + webroot = tempfile.mkdtemp() + + process = multiprocessing.Process(target=_run_server, args=(port,)) + + try: + # SimpleHTTPServer is designed to serve files from the current working directory at the + # time it starts. So we temporarily change the cwd to our crafted webroot before launch. + try: + os.chdir(webroot) + process.start() + finally: + os.chdir(current_cwd) + + check_until_timeout('http://localhost:{0}/'.format(port)) + + yield webroot + finally: + try: + if process.is_alive(): + process.terminate() + process.join() # Block until process is effectively terminated + finally: + shutil.rmtree(webroot) + + +def list_renewal_hooks_dirs(config_dir): + """ + Find and return paths of all hook directories for the given certbot config directory + :param str config_dir: path to the certbot config directory + :return str[]: list of path to the standard hooks directory for this certbot instance + """ + renewal_hooks_root = os.path.join(config_dir, 'renewal-hooks') + return [os.path.join(renewal_hooks_root, item) for item in ['pre', 'deploy', 'post']] + + +def generate_test_file_hooks(config_dir, hook_probe): + """ + Create a suite of certbot hook scripts and put them in the relevant hook directory + for the given certbot configuration directory. These scripts, when executed, will write + specific verbs in the given hook_probe file to allow asserting they have effectively + been executed. The deploy hook also checks that the renewal environment variables are set. + :param str config_dir: current certbot config directory + :param hook_probe: path to the hook probe to test hook scripts execution + """ + hook_path = pkg_resources.resource_filename('certbot_integration_tests', 'assets/hook.py') + + for hook_dir in list_renewal_hooks_dirs(config_dir): + # We want an equivalent of bash `chmod -p $HOOK_DIR, that does not fail if one folder of + # the hierarchy already exists. It is not the case of os.makedirs. Python 3 has an + # optional parameter `exists_ok` to not fail on existing dir, but Python 2.7 does not. + # So we pass through a try except pass for it. To be removed with dropped support on py27. + try: + os.makedirs(hook_dir) + except OSError as error: + if error.errno != errno.EEXIST: + raise + + if os.name != 'nt': + entrypoint_script_path = os.path.join(hook_dir, 'entrypoint.sh') + entrypoint_script = '''\ +#!/usr/bin/env bash +set -e +"{0}" "{1}" "{2}" "{3}" +'''.format(sys.executable, hook_path, entrypoint_script_path, hook_probe) + else: + entrypoint_script_path = os.path.join(hook_dir, 'entrypoint.bat') + entrypoint_script = '''\ +@echo off +"{0}" "{1}" "{2}" "{3}" + '''.format(sys.executable, hook_path, entrypoint_script_path, hook_probe) + + with open(entrypoint_script_path, 'w') as file_h: + file_h.write(entrypoint_script) + + os.chmod(entrypoint_script_path, os.stat(entrypoint_script_path).st_mode | stat.S_IEXEC) + + +@contextlib.contextmanager +def manual_http_hooks(http_server_root, http_port): + """ + Generate suitable http-01 hooks command for test purpose in the given HTTP + server webroot directory. These hooks command use temporary python scripts + that are deleted upon context exit. + :param str http_server_root: path to the HTTP server configured to serve http-01 challenges + :param int http_port: HTTP port that the HTTP server listen on + :return (str, str): a tuple containing the authentication hook and cleanup hook commands + """ + tempdir = tempfile.mkdtemp() + try: + auth_script_path = os.path.join(tempdir, 'auth.py') + with open(auth_script_path, 'w') as file_h: + file_h.write('''\ +#!/usr/bin/env python +import os +import requests +import time +import sys +challenge_dir = os.path.join('{0}', '.well-known', 'acme-challenge') +os.makedirs(challenge_dir) +challenge_file = os.path.join(challenge_dir, os.environ.get('CERTBOT_TOKEN')) +with open(challenge_file, 'w') as file_h: + file_h.write(os.environ.get('CERTBOT_VALIDATION')) +url = 'http://localhost:{1}/.well-known/acme-challenge/' + os.environ.get('CERTBOT_TOKEN') +for _ in range(0, 10): + time.sleep(1) + try: + if request.get(url).status_code == 200: + sys.exit(0) + except requests.exceptions.ConnectionError: + pass +raise ValueError('Error, url did not respond after 10 attempts: {{0}}'.format(url)) +'''.format(http_server_root.replace('\\', '\\\\'), http_port)) + os.chmod(auth_script_path, 0o755) + + cleanup_script_path = os.path.join(tempdir, 'cleanup.py') + with open(cleanup_script_path, 'w') as file_h: + file_h.write('''\ +#!/usr/bin/env python +import os +import shutil +well_known = os.path.join('{0}', '.well-known') +shutil.rmtree(well_known) +'''.format(http_server_root.replace('\\', '\\\\'))) + os.chmod(cleanup_script_path, 0o755) + + yield ('{0} {1}'.format(sys.executable, auth_script_path), + '{0} {1}'.format(sys.executable, cleanup_script_path)) + finally: + shutil.rmtree(tempdir) + + +def generate_csr(domains, key_path, csr_path, key_type=RSA_KEY_TYPE): + """ + Generate a private key, and a CSR for the given domains using this key. + :param domains: the domain names to include in the CSR + :type domains: `list` of `str` + :param str key_path: path to the private key that will be generated + :param str csr_path: path to the CSR that will be generated + :param str key_type: type of the key (misc.RSA_KEY_TYPE or misc.ECDSA_KEY_TYPE) + """ + if key_type == RSA_KEY_TYPE: + key = crypto.PKey() + key.generate_key(crypto.TYPE_RSA, 2048) + elif key_type == ECDSA_KEY_TYPE: + with warnings.catch_warnings(): + # Ignore a warning on some old versions of cryptography + warnings.simplefilter('ignore', category=PendingDeprecationWarning) + key = ec.generate_private_key(ec.SECP384R1(), default_backend()) + key = key.private_bytes(encoding=Encoding.PEM, format=PrivateFormat.TraditionalOpenSSL, + encryption_algorithm=NoEncryption()) + key = crypto.load_privatekey(crypto.FILETYPE_PEM, key) + else: + raise ValueError('Invalid key type: {0}'.format(key_type)) + + with open(key_path, 'wb') as file_h: + file_h.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, key)) + + req = crypto.X509Req() + san = ', '.join(['DNS:{0}'.format(item) for item in domains]) + san_constraint = crypto.X509Extension(b'subjectAltName', False, san.encode('utf-8')) + req.add_extensions([san_constraint]) + + req.set_pubkey(key) + req.set_version(2) + req.sign(key, 'sha256') + + with open(csr_path, 'wb') as file_h: + file_h.write(crypto.dump_certificate_request(crypto.FILETYPE_ASN1, req)) + + +def read_certificate(cert_path): + """ + Load the certificate from the provided path, and return a human readable version of it (TEXT mode). + :param str cert_path: the path to the certificate + :returns: the TEXT version of the certificate, as it would be displayed by openssl binary + """ + with open(cert_path, 'rb') as file: + data = file.read() + + cert = crypto.load_certificate(crypto.FILETYPE_PEM, data) + return crypto.dump_certificate(crypto.FILETYPE_TEXT, cert).decode('utf-8') + + +def load_sample_data_path(workspace): + """ + Load the certbot configuration example designed to make OCSP tests, and return its path + :param str workspace: current test workspace directory path + :returns: the path to the loaded sample data directory + :rtype: str + """ + original = pkg_resources.resource_filename('certbot_integration_tests', 'assets/sample-config') + copied = os.path.join(workspace, 'sample-config') + shutil.copytree(original, copied, symlinks=True) + + if os.name == 'nt': + # Fix the symlinks on Windows since GIT is not creating them upon checkout + for lineage in ['a.encryption-example.com', 'b.encryption-example.com']: + current_live = os.path.join(copied, 'live', lineage) + for name in os.listdir(current_live): + if name != 'README': + current_file = os.path.join(current_live, name) + with open(current_file) as file_h: + src = file_h.read() + os.unlink(current_file) + os.symlink(os.path.join(current_live, src), current_file) + + return copied + + +def echo(keyword, path=None): + """ + Generate a platform independent executable command + that echoes the given keyword into the given file. + :param keyword: the keyword to echo (must be a single keyword) + :param path: path to the file were keyword is echoed + :return: the executable command + """ + if not re.match(r'^\w+$', keyword): + raise ValueError('Error, keyword `{0}` is not a single keyword.' + .format(keyword)) + return '{0} -c "from __future__ import print_function; print(\'{1}\')"{2}'.format( + os.path.basename(sys.executable), keyword, ' >> "{0}"'.format(path) if path else '') diff --git a/certbot-ci/certbot_integration_tests/utils/pebble_artifacts.py b/certbot-ci/certbot_integration_tests/utils/pebble_artifacts.py new file mode 100644 index 000000000..2b1557928 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/utils/pebble_artifacts.py @@ -0,0 +1,53 @@ +import json +import os +import stat + +import pkg_resources +import requests + +from certbot_integration_tests.utils.constants import MOCK_OCSP_SERVER_PORT + +PEBBLE_VERSION = 'v2.2.1' +ASSETS_PATH = pkg_resources.resource_filename('certbot_integration_tests', 'assets') + + +def fetch(workspace): + suffix = 'linux-amd64' if os.name != 'nt' else 'windows-amd64.exe' + + pebble_path = _fetch_asset('pebble', suffix) + challtestsrv_path = _fetch_asset('pebble-challtestsrv', suffix) + pebble_config_path = _build_pebble_config(workspace) + + return pebble_path, challtestsrv_path, pebble_config_path + + +def _fetch_asset(asset, suffix): + asset_path = os.path.join(ASSETS_PATH, '{0}_{1}_{2}'.format(asset, PEBBLE_VERSION, suffix)) + if not os.path.exists(asset_path): + asset_url = ('https://github.com/letsencrypt/pebble/releases/download/{0}/{1}_{2}' + .format(PEBBLE_VERSION, asset, suffix)) + response = requests.get(asset_url) + response.raise_for_status() + with open(asset_path, 'wb') as file_h: + file_h.write(response.content) + os.chmod(asset_path, os.stat(asset_path).st_mode | stat.S_IEXEC) + + return asset_path + + +def _build_pebble_config(workspace): + config_path = os.path.join(workspace, 'pebble-config.json') + with open(config_path, 'w') as file_h: + file_h.write(json.dumps({ + 'pebble': { + 'listenAddress': '0.0.0.0:14000', + 'managementListenAddress': '0.0.0.0:15000', + 'certificate': os.path.join(ASSETS_PATH, 'cert.pem'), + 'privateKey': os.path.join(ASSETS_PATH, 'key.pem'), + 'httpPort': 5002, + 'tlsPort': 5001, + 'ocspResponderURL': 'http://127.0.0.1:{0}'.format(MOCK_OCSP_SERVER_PORT), + }, + })) + + return config_path diff --git a/certbot-ci/certbot_integration_tests/utils/pebble_ocsp_server.py b/certbot-ci/certbot_integration_tests/utils/pebble_ocsp_server.py new file mode 100755 index 000000000..9458560e8 --- /dev/null +++ b/certbot-ci/certbot_integration_tests/utils/pebble_ocsp_server.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python +""" +This runnable module interfaces itself with the Pebble management interface in order +to serve a mock OCSP responder during integration tests against Pebble. +""" +import datetime +import re + +from cryptography import x509 +from cryptography.hazmat.backends import default_backend +from cryptography.hazmat.primitives import hashes +from cryptography.hazmat.primitives import serialization +from cryptography.x509 import ocsp +from dateutil import parser +import requests +from six.moves import BaseHTTPServer + +from certbot_integration_tests.utils.constants import MOCK_OCSP_SERVER_PORT +from certbot_integration_tests.utils.constants import PEBBLE_MANAGEMENT_URL +from certbot_integration_tests.utils.misc import GracefulTCPServer + + +class _ProxyHandler(BaseHTTPServer.BaseHTTPRequestHandler): + def do_POST(self): + request = requests.get(PEBBLE_MANAGEMENT_URL + '/intermediate-keys/0', verify=False) + issuer_key = serialization.load_pem_private_key(request.content, None, default_backend()) + + request = requests.get(PEBBLE_MANAGEMENT_URL + '/intermediates/0', verify=False) + issuer_cert = x509.load_pem_x509_certificate(request.content, default_backend()) + + try: + content_len = int(self.headers.getheader('content-length', 0)) + except AttributeError: + content_len = int(self.headers.get('Content-Length')) + + ocsp_request = ocsp.load_der_ocsp_request(self.rfile.read(content_len)) + response = requests.get('{0}/cert-status-by-serial/{1}'.format( + PEBBLE_MANAGEMENT_URL, str(hex(ocsp_request.serial_number)).replace('0x', '')), verify=False) + + if not response.ok: + ocsp_response = ocsp.OCSPResponseBuilder.build_unsuccessful(ocsp.OCSPResponseStatus.UNAUTHORIZED) + else: + data = response.json() + + now = datetime.datetime.utcnow() + cert = x509.load_pem_x509_certificate(data['Certificate'].encode(), default_backend()) + if data['Status'] != 'Revoked': + ocsp_status, revocation_time, revocation_reason = ocsp.OCSPCertStatus.GOOD, None, None + else: + ocsp_status, revocation_reason = ocsp.OCSPCertStatus.REVOKED, x509.ReasonFlags.unspecified + revoked_at = re.sub(r'( \+\d{4}).*$', r'\1', data['RevokedAt']) # "... +0000 UTC" => "+0000" + revocation_time = parser.parse(revoked_at) + + ocsp_response = ocsp.OCSPResponseBuilder().add_response( + cert=cert, issuer=issuer_cert, algorithm=hashes.SHA1(), + cert_status=ocsp_status, + this_update=now, next_update=now + datetime.timedelta(hours=1), + revocation_time=revocation_time, revocation_reason=revocation_reason + ).responder_id( + ocsp.OCSPResponderEncoding.NAME, issuer_cert + ).sign(issuer_key, hashes.SHA256()) + + self.send_response(200) + self.end_headers() + self.wfile.write(ocsp_response.public_bytes(serialization.Encoding.DER)) + + +if __name__ == '__main__': + try: + GracefulTCPServer(('', MOCK_OCSP_SERVER_PORT), _ProxyHandler).serve_forever() + except KeyboardInterrupt: + pass diff --git a/certbot-ci/certbot_integration_tests/utils/proxy.py b/certbot-ci/certbot_integration_tests/utils/proxy.py new file mode 100644 index 000000000..3a16adebf --- /dev/null +++ b/certbot-ci/certbot_integration_tests/utils/proxy.py @@ -0,0 +1,36 @@ +#!/usr/bin/env python +import json +import re +import sys + +import requests +from six.moves import BaseHTTPServer + +from certbot_integration_tests.utils.misc import GracefulTCPServer + + +def _create_proxy(mapping): + class ProxyHandler(BaseHTTPServer.BaseHTTPRequestHandler): + def do_GET(self): + headers = {key.lower(): value for key, value in self.headers.items()} + backend = [backend for pattern, backend in mapping.items() + if re.match(pattern, headers['host'])][0] + response = requests.get(backend + self.path, headers=headers) + + self.send_response(response.status_code) + for key, value in response.headers.items(): + self.send_header(key, value) + self.end_headers() + self.wfile.write(response.content) + + return ProxyHandler + + +if __name__ == '__main__': + http_port = int(sys.argv[1]) + port_mapping = json.loads(sys.argv[2]) + httpd = GracefulTCPServer(('', http_port), _create_proxy(port_mapping)) + try: + httpd.serve_forever() + except KeyboardInterrupt: + pass diff --git a/certbot-ci/setup.py b/certbot-ci/setup.py new file mode 100644 index 000000000..75d2cc96a --- /dev/null +++ b/certbot-ci/setup.py @@ -0,0 +1,70 @@ +from distutils.version import StrictVersion +import sys + +from setuptools import __version__ as setuptools_version +from setuptools import find_packages +from setuptools import setup + +version = '0.32.0.dev0' + +install_requires = [ + 'coverage', + 'cryptography', + 'docker-compose', + 'pyopenssl', + 'pytest', + 'pytest-cov', + 'pytest-xdist', + 'python-dateutil', + 'pyyaml', + 'requests', + 'six', +] + +# Add pywin32 on Windows platforms to handle low-level system calls. +# This dependency needs to be added using environment markers to avoid its installation on Linux. +# However environment markers are supported only with setuptools >= 36.2. +# So this dependency is not added for old Linux distributions with old setuptools, +# in order to allow these systems to build certbot from sources. +if StrictVersion(setuptools_version) >= StrictVersion('36.2'): + install_requires.append("pywin32>=224 ; sys_platform == 'win32'") +elif 'bdist_wheel' in sys.argv[1:]: + raise RuntimeError('Error, you are trying to build certbot wheels using an old version ' + 'of setuptools. Version 36.2+ of setuptools is required.') + +setup( + name='certbot-ci', + version=version, + description="Certbot continuous integration framework", + url='https://github.com/certbot/certbot', + author="Certbot Project", + author_email='client-dev@letsencrypt.org', + license='Apache License 2.0', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', + classifiers=[ + 'Development Status :: 3 - Alpha', + 'Intended Audience :: Developers', + 'License :: OSI Approved :: Apache Software License', + 'Programming Language :: Python', + 'Programming Language :: Python :: 2', + 'Programming Language :: Python :: 2.7', + 'Programming Language :: Python :: 3', + 'Programming Language :: Python :: 3.5', + 'Programming Language :: Python :: 3.6', + 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', + 'Topic :: Internet :: WWW/HTTP', + 'Topic :: Security', + ], + + packages=find_packages(), + include_package_data=True, + install_requires=install_requires, + + entry_points={ + 'console_scripts': [ + 'certbot_test=certbot_integration_tests.utils.certbot_call:main', + 'run_acme_server=certbot_integration_tests.utils.acme_server:main', + ], + } +) diff --git a/certbot-compatibility-test/docs/_static/.gitignore b/certbot-ci/windows_installer_integration_tests/__init__.py similarity index 100% rename from certbot-compatibility-test/docs/_static/.gitignore rename to certbot-ci/windows_installer_integration_tests/__init__.py diff --git a/certbot-ci/windows_installer_integration_tests/conftest.py b/certbot-ci/windows_installer_integration_tests/conftest.py new file mode 100644 index 000000000..e36654f90 --- /dev/null +++ b/certbot-ci/windows_installer_integration_tests/conftest.py @@ -0,0 +1,38 @@ +""" +General conftest for pytest execution of all integration tests lying +in the window_installer_integration tests package. +As stated by pytest documentation, conftest module is used to set on +for a directory a specific configuration using built-in pytest hooks. + +See https://docs.pytest.org/en/latest/reference.html#hook-reference +""" +from __future__ import print_function +import os + +import pytest + +ROOT_PATH = os.path.dirname(os.path.dirname(os.path.dirname(__file__))) + + +def pytest_addoption(parser): + """ + Standard pytest hook to add options to the pytest parser. + :param parser: current pytest parser that will be used on the CLI + """ + parser.addoption('--installer-path', + default=os.path.join(ROOT_PATH, 'windows-installer', 'build', + 'nsis', 'certbot-beta-installer-win32.exe'), + help='set the path of the windows installer to use, default to ' + 'CERTBOT_ROOT_PATH\\windows-installer\\build\\nsis\\certbot-beta-installer-win32.exe') + parser.addoption('--allow-persistent-changes', action='store_true', + help='needs to be set, and confirm that the test will make persistent changes on this machine') + + +def pytest_configure(config): + """ + Standard pytest hook used to add a configuration logic for each node of a pytest run. + :param config: the current pytest configuration + """ + if not config.option.allow_persistent_changes: + raise RuntimeError('This integration test would install Certbot on your machine. ' + 'Please run it again with the `--allow-persistent-changes` flag set to acknowledge.') diff --git a/certbot-ci/windows_installer_integration_tests/test_main.py b/certbot-ci/windows_installer_integration_tests/test_main.py new file mode 100644 index 000000000..c8c347aa8 --- /dev/null +++ b/certbot-ci/windows_installer_integration_tests/test_main.py @@ -0,0 +1,61 @@ +import os +import time +import unittest +import subprocess +import re + + +@unittest.skipIf(os.name != 'nt', reason='Windows installer tests must be run on Windows.') +def test_it(request): + try: + subprocess.check_call(['certbot', '--version']) + except (subprocess.CalledProcessError, OSError): + pass + else: + raise AssertionError('Expect certbot to not be available in the PATH.') + + try: + # Install certbot + subprocess.check_call([request.config.option.installer_path, '/S']) + + # Assert certbot is installed and runnable + output = subprocess.check_output(['certbot', '--version'], universal_newlines=True) + assert re.match(r'^certbot \d+\.\d+\.\d+.*$', output), 'Flag --version does not output a version.' + + # Assert renew task is installed and ready + output = _ps('(Get-ScheduledTask -TaskName "Certbot Renew Task").State', capture_stdout=True) + assert output.strip() == 'Ready' + + # Assert renew task is working + now = time.time() + _ps('Start-ScheduledTask -TaskName "Certbot Renew Task"') + + status = 'Running' + while status != 'Ready': + status = _ps('(Get-ScheduledTask -TaskName "Certbot Renew Task").State', capture_stdout=True).strip() + time.sleep(1) + + log_path = os.path.join('C:\\', 'Certbot', 'log', 'letsencrypt.log') + + modification_time = os.path.getmtime(log_path) + assert now < modification_time, 'Certbot log file has not been modified by the renew task.' + + with open(log_path) as file_h: + data = file_h.read() + assert 'no renewal failures' in data, 'Renew task did not execute properly.' + + finally: + # Sadly this command cannot work in non interactive mode: uninstaller will ask explicitly permission in an UAC prompt + # print('Uninstalling Certbot ...') + # uninstall_path = _ps('(gci "HKLM:\\SOFTWARE\\Wow6432Node\\Microsoft\\Windows\\CurrentVersion\\Uninstall"' + # ' | foreach { gp $_.PSPath }' + # ' | ? { $_ -match "Certbot" }' + # ' | select UninstallString)' + # '.UninstallString', capture_stdout=True) + # subprocess.check_call([uninstall_path, '/S']) + pass + + +def _ps(powershell_str, capture_stdout=False): + fn = subprocess.check_output if capture_stdout else subprocess.check_call + return fn(['powershell.exe', '-c', powershell_str], universal_newlines=True) diff --git a/certbot-compatibility-test/Dockerfile b/certbot-compatibility-test/Dockerfile index 1e9e0d727..a9996f779 100644 --- a/certbot-compatibility-test/Dockerfile +++ b/certbot-compatibility-test/Dockerfile @@ -1,4 +1,4 @@ -FROM debian:jessie +FROM debian:stretch MAINTAINER Brad Warren # no need to mkdir anything: @@ -14,7 +14,7 @@ RUN /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto --os-packages-only # the above is not likely to change, so by putting it further up the # Dockerfile we make sure we cache as much as possible -COPY setup.py README.rst CHANGELOG.md MANIFEST.in linter_plugin.py tox.cover.py tox.ini .pylintrc /opt/certbot/src/ +COPY certbot/setup.py certbot/README.rst certbot/CHANGELOG.md certbot/MANIFEST.in linter_plugin.py tox.cover.py tox.ini .pylintrc /opt/certbot/src/ # all above files are necessary for setup.py, however, package source # code directory has to be copied separately to a subdirectory... @@ -38,7 +38,7 @@ ENV PATH /opt/certbot/venv/bin:$PATH RUN /opt/certbot/venv/bin/python \ /opt/certbot/src/tools/pip_install_editable.py \ /opt/certbot/src/acme \ - /opt/certbot/src \ + /opt/certbot/src/certbot \ /opt/certbot/src/certbot-apache \ /opt/certbot/src/certbot-nginx \ /opt/certbot/src/certbot-compatibility-test diff --git a/certbot-compatibility-test/MANIFEST.in b/certbot-compatibility-test/MANIFEST.in index 11762538a..a9d4f5ce7 100644 --- a/certbot-compatibility-test/MANIFEST.in +++ b/certbot-compatibility-test/MANIFEST.in @@ -1,6 +1,5 @@ include LICENSE.txt include README.rst -recursive-include docs * include certbot_compatibility_test/configurators/apache/a2enmod.sh include certbot_compatibility_test/configurators/apache/a2dismod.sh include certbot_compatibility_test/configurators/apache/Dockerfile diff --git a/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/common.py b/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/common.py index 82195264b..a9b1ce87e 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/common.py +++ b/certbot-compatibility-test/certbot_compatibility_test/configurators/apache/common.py @@ -6,10 +6,10 @@ import subprocess import mock import zope.interface -from certbot import configuration from certbot import errors as le_errors from certbot import util as certbot_util -from certbot_apache import entrypoint +from certbot._internal import configuration +from certbot_apache._internal import entrypoint from certbot_compatibility_test import errors from certbot_compatibility_test import interfaces from certbot_compatibility_test import util @@ -18,7 +18,6 @@ from certbot_compatibility_test.configurators import common as configurators_com @zope.interface.implementer(interfaces.IConfiguratorProxy) class Proxy(configurators_common.Proxy): - # pylint: disable=too-many-instance-attributes """A common base for Apache test configurators""" def __init__(self, args): @@ -28,7 +27,7 @@ class Proxy(configurators_common.Proxy): self.modules = self.server_root = self.test_conf = self.version = None patch = mock.patch( - "certbot_apache.configurator.display_ops.select_vhost") + "certbot_apache._internal.configurator.display_ops.select_vhost") mock_display = patch.start() mock_display.side_effect = le_errors.PluginError( "Unable to determine vhost") diff --git a/certbot-compatibility-test/certbot_compatibility_test/configurators/common.py b/certbot-compatibility-test/certbot_compatibility_test/configurators/common.py index 2a800c1c2..b592d6288 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/configurators/common.py +++ b/certbot-compatibility-test/certbot_compatibility_test/configurators/common.py @@ -4,16 +4,14 @@ import os import shutil import tempfile -from certbot import constants +from certbot._internal import constants from certbot_compatibility_test import errors from certbot_compatibility_test import util - logger = logging.getLogger(__name__) class Proxy(object): - # pylint: disable=too-many-instance-attributes """A common base for compatibility test configurators""" @classmethod @@ -23,6 +21,9 @@ class Proxy(object): def __init__(self, args): """Initializes the plugin with the given command line args""" self._temp_dir = tempfile.mkdtemp() + # tempfile.mkdtemp() creates folders with too restrictive permissions to be accessible + # to an Apache worker, leading to HTTP challenge failures. Let's fix that. + os.chmod(self._temp_dir, 0o755) self.le_config = util.create_le_config(self._temp_dir) config_dir = util.extract_configs(args.configs, self._temp_dir) self._configs = [ @@ -42,8 +43,7 @@ class Proxy(object): method = getattr(self._configurator, name, None) if callable(method): return method - else: - raise AttributeError() + raise AttributeError() def has_more_configs(self): """Returns true if there are more configs to test""" @@ -81,15 +81,13 @@ class Proxy(object): """Returns the set of domain names that the plugin should find""" if self._all_names: return self._all_names - else: - raise errors.Error("No configuration file loaded") + raise errors.Error("No configuration file loaded") def get_testable_domain_names(self): """Returns the set of domain names that can be tested against""" if self._test_names: return self._test_names - else: - return {"example.com"} + return {"example.com"} def deploy_cert(self, domain, cert_path, key_path, chain_path=None, fullchain_path=None): diff --git a/certbot-compatibility-test/certbot_compatibility_test/configurators/nginx/common.py b/certbot-compatibility-test/certbot_compatibility_test/configurators/nginx/common.py index ed5cf750e..3011b9823 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/configurators/nginx/common.py +++ b/certbot-compatibility-test/certbot_compatibility_test/configurators/nginx/common.py @@ -5,24 +5,20 @@ import subprocess import zope.interface -from certbot import configuration -from certbot_nginx import configurator -from certbot_nginx import constants +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module +from certbot._internal import configuration from certbot_compatibility_test import errors from certbot_compatibility_test import interfaces from certbot_compatibility_test import util from certbot_compatibility_test.configurators import common as configurators_common +from certbot_nginx._internal import configurator +from certbot_nginx._internal import constants @zope.interface.implementer(interfaces.IConfiguratorProxy) class Proxy(configurators_common.Proxy): - # pylint: disable=too-many-instance-attributes """A common base for Nginx test configurators""" - def __init__(self, args): - """Initializes the plugin with the given command line args""" - super(Proxy, self).__init__(args) - def load_config(self): """Loads the next configuration for the plugin to test""" config = super(Proxy, self).load_config() @@ -48,7 +44,7 @@ class Proxy(configurators_common.Proxy): def _prepare_configurator(self): """Prepares the Nginx plugin for testing""" - for k in constants.CLI_DEFAULTS.keys(): + for k in constants.CLI_DEFAULTS: setattr(self.le_config, "nginx_" + k, constants.os_constant(k)) conf = configuration.NamespaceConfig(self.le_config) @@ -72,15 +68,23 @@ def _get_server_root(config): def _get_names(config): """Returns all and testable domain names in config""" - all_names = set() + all_names = set() # type: Set[str] for root, _dirs, files in os.walk(config): for this_file in files: - for line in open(os.path.join(root, this_file)): - if line.strip().startswith("server_name"): - names = line.partition("server_name")[2].rpartition(";")[0] - for n in names.split(): - # Filter out wildcards in both all_names and test_names - if not n.startswith("*."): - all_names.add(n) + update_names = _get_server_names(root, this_file) + all_names.update(update_names) non_ip_names = set(n for n in all_names if not util.IP_REGEX.match(n)) return all_names, non_ip_names + + +def _get_server_names(root, filename): + """Returns all names in a config file path""" + all_names = set() + for line in open(os.path.join(root, filename)): + if line.strip().startswith("server_name"): + names = line.partition("server_name")[2].rpartition(";")[0] + for n in names.split(): + # Filter out wildcards in both all_names and test_names + if not n.startswith("*."): + all_names.add(n) + return all_names diff --git a/certbot-compatibility-test/certbot_compatibility_test/interfaces.py b/certbot-compatibility-test/certbot_compatibility_test/interfaces.py index 7d3daee09..0249c2aaa 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/interfaces.py +++ b/certbot-compatibility-test/certbot_compatibility_test/interfaces.py @@ -6,8 +6,9 @@ import certbot.interfaces # pylint: disable=no-self-argument,no-method-argument -class IPluginProxy(zope.interface.Interface): +class IPluginProxy(zope.interface.Interface): # pylint: disable=inherit-non-class """Wraps a Certbot plugin""" + http_port = zope.interface.Attribute( "The port to connect to on localhost for HTTP traffic") @@ -17,7 +18,7 @@ class IPluginProxy(zope.interface.Interface): def add_parser_arguments(cls, parser): """Adds command line arguments needed by the parser""" - def __init__(args): + def __init__(args): # pylint: disable=super-init-not-called """Initializes the plugin with the given command line args""" def cleanup_from_tests(): # type: ignore diff --git a/certbot-compatibility-test/certbot_compatibility_test/test_driver.py b/certbot-compatibility-test/certbot_compatibility_test/test_driver.py index 9eea95e67..2c3f880e0 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/test_driver.py +++ b/certbot-compatibility-test/certbot_compatibility_test/test_driver.py @@ -1,33 +1,31 @@ """Tests Certbot plugins against different server configurations.""" import argparse +import contextlib import filecmp import logging import os import shutil +import sys import tempfile import time -import sys import OpenSSL - -from six.moves import xrange # pylint: disable=import-error,redefined-builtin +from urllib3.util import connection from acme import challenges from acme import crypto_util from acme import messages -from acme.magic_typing import List, Tuple # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Tuple # pylint: disable=unused-import, no-name-in-module from certbot import achallenges from certbot import errors as le_errors from certbot.tests import acme_util - from certbot_compatibility_test import errors from certbot_compatibility_test import util from certbot_compatibility_test import validator - from certbot_compatibility_test.configurators.apache import common as a_common from certbot_compatibility_test.configurators.nginx import common as n_common - DESCRIPTION = """ Tests Certbot plugins against different server configurations. It is assumed that Docker is already installed. If no test type is specified, all @@ -58,25 +56,27 @@ def test_authenticator(plugin, config, temp_dir): return False success = True - for i in xrange(len(responses)): - if not responses[i]: + for i, response in enumerate(responses): + achall = achalls[i] + if not response: logger.error( "Plugin failed to complete %s for %s in %s", - type(achalls[i]), achalls[i].domain, config) + type(achall), achall.domain, config) success = False - elif isinstance(responses[i], challenges.TLSSNI01Response): - verified = responses[i].simple_verify(achalls[i].chall, - achalls[i].domain, - util.JWK.public_key(), - host="127.0.0.1", - port=plugin.https_port) + elif isinstance(response, challenges.HTTP01Response): + # We fake the DNS resolution to ensure that any domain is resolved + # to the local HTTP server setup for the compatibility tests + with _fake_dns_resolution("127.0.0.1"): + verified = response.simple_verify( + achall.chall, achall.domain, + util.JWK.public_key(), port=plugin.http_port) if verified: logger.info( - "tls-sni-01 verification for %s succeeded", achalls[i].domain) + "http-01 verification for %s succeeded", achall.domain) else: logger.error( - "**** tls-sni-01 verification for %s in %s failed", - achalls[i].domain, config) + "**** http-01 verification for %s in %s failed", + achall.domain, config) success = False if success: @@ -89,8 +89,7 @@ def test_authenticator(plugin, config, temp_dir): if _dirs_are_unequal(config, backup): logger.error("Challenge cleanup failed for %s", config) return False - else: - logger.info("Challenge cleanup succeeded") + logger.info("Challenge cleanup succeeded") return success @@ -102,9 +101,9 @@ def _create_achalls(plugin): for domain in names: prefs = plugin.get_chall_pref(domain) for chall_type in prefs: - if chall_type == challenges.TLSSNI01: - chall = challenges.TLSSNI01( - token=os.urandom(challenges.TLSSNI01.TOKEN_SIZE)) + if chall_type == challenges.HTTP01: + chall = challenges.HTTP01( + token=os.urandom(challenges.HTTP01.TOKEN_SIZE)) challb = acme_util.chall_to_challb( chall, messages.STATUS_PENDING) achall = achallenges.KeyAuthorizationAnnotatedChallenge( @@ -235,9 +234,8 @@ def test_rollback(plugin, config, backup): if _dirs_are_unequal(config, backup): logger.error("*** Rollback failed for config `%s`", config) return False - else: - logger.info("Rollback succeeded") - return True + logger.info("Rollback succeeded") + return True def _create_backup(config, temp_dir): @@ -252,7 +250,7 @@ def _create_backup(config, temp_dir): def _dirs_are_unequal(dir1, dir2): """Returns True if dir1 and dir2 are unequal""" dircmps = [filecmp.dircmp(dir1, dir2)] - while len(dircmps): + while dircmps: dircmp = dircmps.pop() if dircmp.left_only or dircmp.right_only: logger.error("The following files and directories are only " @@ -306,7 +304,7 @@ def get_args(): "-e", "--enhance", action="store_true", help="tests the enhancements " "the plugin supports (implicitly includes installer tests)") - for plugin in PLUGINS.itervalues(): + for plugin in PLUGINS.values(): plugin.add_parser_arguments(parser) args = parser.parse_args() @@ -369,5 +367,21 @@ def main(): sys.exit(1) +@contextlib.contextmanager +def _fake_dns_resolution(resolved_ip): + """Monkey patch urllib3 to make any hostname be resolved to the provided IP""" + _original_create_connection = connection.create_connection + + def _patched_create_connection(address, *args, **kwargs): + _, port = address + return _original_create_connection((resolved_ip, port), *args, **kwargs) + + try: + connection.create_connection = _patched_create_connection + yield + finally: + connection.create_connection = _original_create_connection + + if __name__ == "__main__": main() diff --git a/certbot-compatibility-test/certbot_compatibility_test/testdata/rsa1024_key.pem b/certbot-compatibility-test/certbot_compatibility_test/testdata/rsa1024_key.pem deleted file mode 100644 index 8f82146ba..000000000 --- a/certbot-compatibility-test/certbot_compatibility_test/testdata/rsa1024_key.pem +++ /dev/null @@ -1,15 +0,0 @@ ------BEGIN RSA PRIVATE KEY----- -MIICXAIBAAKBgQCsREbM+UcfsgDy2w56AVGyxsO0HVsbEZHHoEzv7qksIwFgRYMp -rowwIxD450RQQqjvw9IoXlMVmr1t5szn5KXn9JRO9T5KNCCy3VPx75WBcp6kzd9Q -2HS1OEOtpilNnDkZ+TJfdgFWPUBYj2o4Md1hPmcvagiIJY5U6speka2bjwIDAQAB -AoGANCMZ9pF/mDUsmlP4Rq69hkkoFAxKdZ/UqkF256so4mXZ1cRUFTpxzWPfkCWW -hGAYdzCiG3uo08IYkPmojIqkN1dk5Hcq5eQAmshaPkQHQCHjmPjjcNvgjIXQoGUf -TpDU2hbY4UAlJlj4ZLh+jGP5Zq8/WrNi8RsI3v9Nagfp/FECQQDgi2q8p1gX0TNh -d1aEKmSXkR3bxkyFk6oS+pBrAG3+yX27ZayN6Rx6DOs/FcBsOu7fX3PYBziDeEWe -Lkf1P743AkEAxGYT/LY3puglSz4iJZZzWmRCrVOg41yhfQ+F1BRX43/2vtoU5GyM -2lUn1vQ2e/rfmnAvfJxc90GeZCIHB1ihaQJBALH8UMLxMtbOMJgVbDKfF9U8ZhqK -+KT5A1q/2jG2yXmoZU1hroFeQgBMtTvwFfK0VBwjIUQflSBA+Y4EyW0Q9ckCQGvd -jHitM1+N/H2YwHRYbz5j9mLvnVuCEod3MQ9LpQGj1Eb5y6OxIqL/RgQ+2HW7UXem -yc3sqvp5pZ5lOesE+JECQETPI64gqxlTIs3nErNMpMynUuTWpaElOcIJTT6icLzB -Xix67kKXjROO5D58GEYkM0Yi5k7YdUPoQBW7MoIrSIA= ------END RSA PRIVATE KEY----- diff --git a/certbot-compatibility-test/certbot_compatibility_test/testdata/rsa1024_key2.pem b/certbot-compatibility-test/certbot_compatibility_test/testdata/rsa1024_key2.pem deleted file mode 100644 index 03f77d903..000000000 --- a/certbot-compatibility-test/certbot_compatibility_test/testdata/rsa1024_key2.pem +++ /dev/null @@ -1,15 +0,0 @@ ------BEGIN RSA PRIVATE KEY----- -MIICXAIBAAKBgQDCzejLjo8wsz0avrylt7HQyF0+vsKritF70EGmc64cV0XfkCTR -o+vMXBXMuUY6Kv3hTXV7klgkNYmL7gXAsFGQ4B9qeMnkYn0GcQdI51u076y/26Fu -37uJg45Q6eApKInJSsyLVMcAT4HUJ6fFnUodFAKR7vTzOQryNW7Et4gA4wIDAQAB -AoGAKiAU40/krwdTg2ETslJS5W8ums7tkeLnAfs69x+02vQUbA/jpmHoL70KCcdW -5GU/mWUCrsIqxUm+gL/sBosaV/TF256qUBt2qQCZTN8MbDaNSYiiMnucOfbWdIqx -Zgls6GUoXQvPic9RUoFSlgfSjo5ezz6el5ihvRMp+wbk24ECQQD3oz4hN029DSZo -Y3+flmBn77gA0BMUvLa6hmt9b3xT5U/ToCLfbmUvpx7zV1g5era2y9qt/o3UtAbW -1zCVETgzAkEAyWHv/+RnSXp8/D4YwTVWyeWi862uNBPkuLGP/0zASdwBfBK3uBls -+VumfSCtp0kt2AXXmScg1fkHdeAVT6AkkQJBAJb2XRnCrRFiwtdAULzo3zx9Vp6o -OfmaUYrEByMgo5pBYLiSFrA+jFDQgH238YCY3mnxPA517+CLHuA5rtQw+yECQCfm -gL/pyFE1tLfhsdPuNpDwL9YqLl7hJis1+zrxQRQhRCYKK16NoxrQ/u7B38ZKaIvp -tGsC5q2elszTJkXNjBECQCVE9QCVx056vHVdPWM8z3GAeV3sJQ01HLLjebTEEz6G -jH54gk+YYPp4kjCvVUykbnB58BY2n88GQt5Jj5eLuMo= ------END RSA PRIVATE KEY----- diff --git a/certbot-compatibility-test/certbot_compatibility_test/util.py b/certbot-compatibility-test/certbot_compatibility_test/util.py index 6051bbc2e..3465b7143 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/util.py +++ b/certbot-compatibility-test/certbot_compatibility_test/util.py @@ -8,13 +8,11 @@ import tarfile import josepy as jose -from acme import test_util -from certbot import constants - +from certbot._internal import constants +from certbot.tests import util as test_util from certbot_compatibility_test import errors - -_KEY_BASE = "rsa1024_key.pem" +_KEY_BASE = "rsa2048_key.pem" KEY_PATH = test_util.vector_path(_KEY_BASE) KEY = test_util.load_pyopenssl_private_key(_KEY_BASE) JWK = jose.JWKRSA(key=test_util.load_rsa_private_key(_KEY_BASE)) @@ -35,7 +33,7 @@ def create_le_config(parent_dir): config["domains"] = None - return argparse.Namespace(**config) # pylint: disable=star-args + return argparse.Namespace(**config) def extract_configs(configs, parent_dir): diff --git a/certbot-compatibility-test/certbot_compatibility_test/validator.py b/certbot-compatibility-test/certbot_compatibility_test/validator.py index 3455ce82d..796ebbe9d 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/validator.py +++ b/certbot-compatibility-test/certbot_compatibility_test/validator.py @@ -1,15 +1,14 @@ """Validators to determine the current webserver configuration""" import logging import socket -import requests +import requests import six -from six.moves import xrange # pylint: disable=import-error,redefined-builtin +from six.moves import xrange # pylint: disable=import-error, redefined-builtin from acme import crypto_util from acme import errors as acme_errors - logger = logging.getLogger(__name__) diff --git a/certbot-compatibility-test/certbot_compatibility_test/validator_test.py b/certbot-compatibility-test/certbot_compatibility_test/validator_test.py index d0552a756..235ce0e3c 100644 --- a/certbot-compatibility-test/certbot_compatibility_test/validator_test.py +++ b/certbot-compatibility-test/certbot_compatibility_test/validator_test.py @@ -1,9 +1,9 @@ """Tests for certbot_compatibility_test.validator.""" -import requests import unittest import mock import OpenSSL +import requests from acme import errors as acme_errors from certbot_compatibility_test import validator @@ -39,7 +39,7 @@ class ValidatorTest(unittest.TestCase): cert, "test.com", "127.0.0.1")) @mock.patch("certbot_compatibility_test.validator.requests.get") - def test_succesful_redirect(self, mock_get_request): + def test_successful_redirect(self, mock_get_request): mock_get_request.return_value = create_response( 301, {"location": "https://test.com"}) self.assertTrue(self.validator.redirect("test.com")) diff --git a/certbot-compatibility-test/docs/.gitignore b/certbot-compatibility-test/docs/.gitignore deleted file mode 100644 index ba65b13af..000000000 --- a/certbot-compatibility-test/docs/.gitignore +++ /dev/null @@ -1 +0,0 @@ -/_build/ diff --git a/certbot-compatibility-test/docs/Makefile b/certbot-compatibility-test/docs/Makefile deleted file mode 100644 index 0c9cf40aa..000000000 --- a/certbot-compatibility-test/docs/Makefile +++ /dev/null @@ -1,192 +0,0 @@ -# Makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -PAPER = -BUILDDIR = _build - -# User-friendly check for sphinx-build -ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) -$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) -endif - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . -# the i18n builder cannot share the environment and doctrees with the others -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext - -help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " singlehtml to make a single large HTML file" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " applehelp to make an Apple Help Book" - @echo " devhelp to make HTML files and a Devhelp project" - @echo " epub to make an epub" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " latexpdf to make LaTeX files and run them through pdflatex" - @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" - @echo " text to make text files" - @echo " man to make manual pages" - @echo " texinfo to make Texinfo files" - @echo " info to make Texinfo files and run them through makeinfo" - @echo " gettext to make PO message catalogs" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " xml to make Docutils-native XML files" - @echo " pseudoxml to make pseudoxml-XML files for display purposes" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - @echo " coverage to run coverage check of the documentation (if enabled)" - -clean: - rm -rf $(BUILDDIR)/* - -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." - -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." - -singlehtml: - $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml - @echo - @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." - -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/certbot-compatibility-test.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/certbot-compatibility-test.qhc" - -applehelp: - $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp - @echo - @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." - @echo "N.B. You won't be able to view it unless you put it in" \ - "~/Library/Documentation/Help or install it in your application" \ - "bundle." - -devhelp: - $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp - @echo - @echo "Build finished." - @echo "To view the help file:" - @echo "# mkdir -p $$HOME/.local/share/devhelp/certbot-compatibility-test" - @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/certbot-compatibility-test" - @echo "# devhelp" - -epub: - $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub - @echo - @echo "Build finished. The epub file is in $(BUILDDIR)/epub." - -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make' in that directory to run these through (pdf)latex" \ - "(use \`make latexpdf' here to do that automatically)." - -latexpdf: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through pdflatex..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -latexpdfja: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through platex and dvipdfmx..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -text: - $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text - @echo - @echo "Build finished. The text files are in $(BUILDDIR)/text." - -man: - $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man - @echo - @echo "Build finished. The manual pages are in $(BUILDDIR)/man." - -texinfo: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo - @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." - @echo "Run \`make' in that directory to run these through makeinfo" \ - "(use \`make info' here to do that automatically)." - -info: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo "Running Texinfo files through makeinfo..." - make -C $(BUILDDIR)/texinfo info - @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." - -gettext: - $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale - @echo - @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." - -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." - -coverage: - $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage - @echo "Testing of coverage in the sources finished, look at the " \ - "results in $(BUILDDIR)/coverage/python.txt." - -xml: - $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml - @echo - @echo "Build finished. The XML files are in $(BUILDDIR)/xml." - -pseudoxml: - $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml - @echo - @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." diff --git a/certbot-compatibility-test/docs/api.rst b/certbot-compatibility-test/docs/api.rst deleted file mode 100644 index 8668ec5d8..000000000 --- a/certbot-compatibility-test/docs/api.rst +++ /dev/null @@ -1,8 +0,0 @@ -================= -API Documentation -================= - -.. toctree:: - :glob: - - api/** diff --git a/certbot-compatibility-test/docs/api/index.rst b/certbot-compatibility-test/docs/api/index.rst deleted file mode 100644 index fea92d2e5..000000000 --- a/certbot-compatibility-test/docs/api/index.rst +++ /dev/null @@ -1,53 +0,0 @@ -:mod:`certbot_compatibility_test` -------------------------------------- - -.. automodule:: certbot_compatibility_test - :members: - -:mod:`certbot_compatibility_test.errors` -============================================ - -.. automodule:: certbot_compatibility_test.errors - :members: - -:mod:`certbot_compatibility_test.interfaces` -================================================ - -.. automodule:: certbot_compatibility_test.interfaces - :members: - -:mod:`certbot_compatibility_test.test_driver` -================================================= - -.. automodule:: certbot_compatibility_test.test_driver - :members: - -:mod:`certbot_compatibility_test.util` -========================================== - -.. automodule:: certbot_compatibility_test.util - :members: - -:mod:`certbot_compatibility_test.configurators` -=================================================== - -.. automodule:: certbot_compatibility_test.configurators - :members: - -:mod:`certbot_compatibility_test.configurators.apache` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. automodule:: certbot_compatibility_test.configurators.apache - :members: - -:mod:`certbot_compatibility_test.configurators.apache.apache24` -------------------------------------------------------------------- - -.. automodule:: certbot_compatibility_test.configurators.apache.apache24 - :members: - -:mod:`certbot_compatibility_test.configurators.apache.common` -------------------------------------------------------------------- - -.. automodule:: certbot_compatibility_test.configurators.apache.common - :members: diff --git a/certbot-compatibility-test/docs/conf.py b/certbot-compatibility-test/docs/conf.py deleted file mode 100644 index f89f4b368..000000000 --- a/certbot-compatibility-test/docs/conf.py +++ /dev/null @@ -1,319 +0,0 @@ -# -*- coding: utf-8 -*- -# -# certbot-compatibility-test documentation build configuration file, created by -# sphinx-quickstart on Sun Oct 18 13:40:53 2015. -# -# This file is execfile()d with the current directory set to its -# containing dir. -# -# Note that not all possible configuration values are present in this -# autogenerated file. -# -# All configuration values have a default; values that are commented out -# serve to show the default. - -import sys -import os -import shlex - - -here = os.path.abspath(os.path.dirname(__file__)) - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -sys.path.insert(0, os.path.abspath(os.path.join(here, '..'))) - -# -- General configuration ------------------------------------------------ - -# If your documentation needs a minimal Sphinx version, state it here. -needs_sphinx = '1.0' - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom -# ones. -extensions = [ - 'sphinx.ext.autodoc', - 'sphinx.ext.intersphinx', - 'sphinx.ext.todo', - 'sphinx.ext.coverage', - 'sphinx.ext.viewcode', - 'repoze.sphinx.autointerface', -] - -autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# The suffix(es) of source filenames. -# You can specify multiple suffix as a list of string: -# source_suffix = ['.rst', '.md'] -source_suffix = '.rst' - -# The encoding of source files. -#source_encoding = 'utf-8-sig' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'certbot-compatibility-test' -copyright = u'2014-2015, Let\'s Encrypt Project' -author = u'Certbot Project' - -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. -# -# The short X.Y version. -version = '0' -# The full version, including alpha/beta/rc tags. -release = '0' - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -# -# This is also used if you do content translation via gettext catalogs. -# Usually you set "language" from the command line for these cases. -language = 'en' - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -exclude_patterns = ['_build'] - -# The reST default role (used for this markup: `text`) to use for all -# documents. -default_role = 'py:obj' - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] - -# If true, keep warnings as "system message" paragraphs in the built documents. -#keep_warnings = False - -# If true, `todo` and `todoList` produce output, else they produce nothing. -todo_include_todos = True - - -# -- Options for HTML output ---------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. - -# http://docs.readthedocs.org/en/latest/theme.html#how-do-i-use-this-locally-and-on-read-the-docs -# on_rtd is whether we are on readthedocs.org -on_rtd = os.environ.get('READTHEDOCS', None) == 'True' -if not on_rtd: # only import and set the theme if we're building docs locally - import sphinx_rtd_theme - html_theme = 'sphinx_rtd_theme' - html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] -# otherwise, readthedocs.org uses their theme by default, so no need to specify it - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -#html_theme_options = {} - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -#html_title = None - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -#html_logo = None - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -#html_favicon = None - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - -# Add any extra paths that contain custom files (such as robots.txt or -# .htaccess) here, relative to this directory. These files are copied -# directly to the root of the documentation. -#html_extra_path = [] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_domain_indices = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True - -# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True - -# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None - -# Language to be used for generating the HTML full-text search index. -# Sphinx supports the following languages: -# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' -# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' -#html_search_language = 'en' - -# A dictionary with options for the search language support, empty by default. -# Now only 'ja' uses this config value -#html_search_options = {'type': 'default'} - -# The name of a javascript file (relative to the configuration directory) that -# implements a search results scorer. If empty, the default will be used. -#html_search_scorer = 'scorer.js' - -# Output file base name for HTML help builder. -htmlhelp_basename = 'certbot-compatibility-testdoc' - -# -- Options for LaTeX output --------------------------------------------- - -latex_elements = { - # The paper size ('letterpaper' or 'a4paper'). - #'papersize': 'letterpaper', - - # The font size ('10pt', '11pt' or '12pt'). - #'pointsize': '10pt', - - # Additional stuff for the LaTeX preamble. - #'preamble': '', - - # Latex figure (float) alignment - #'figure_align': 'htbp', -} - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, -# author, documentclass [howto, manual, or own class]). -latex_documents = [ - (master_doc, 'certbot-compatibility-test.tex', - u'certbot-compatibility-test Documentation', - u'Certbot Project', 'manual'), -] - -# The name of an image file (relative to this directory) to place at the top of -# the title page. -#latex_logo = None - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# If true, show page references after internal links. -#latex_show_pagerefs = False - -# If true, show URL addresses after external links. -#latex_show_urls = False - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_domain_indices = True - - -# -- Options for manual page output --------------------------------------- - -# One entry per manual page. List of tuples -# (source start file, name, description, authors, manual section). -man_pages = [ - (master_doc, 'certbot-compatibility-test', - u'certbot-compatibility-test Documentation', - [author], 1) -] - -# If true, show URL addresses after external links. -#man_show_urls = False - - -# -- Options for Texinfo output ------------------------------------------- - -# Grouping the document tree into Texinfo files. List of tuples -# (source start file, target name, title, author, -# dir menu entry, description, category) -texinfo_documents = [ - (master_doc, 'certbot-compatibility-test', - u'certbot-compatibility-test Documentation', - author, 'certbot-compatibility-test', - 'One line description of project.', 'Miscellaneous'), -] - -# Documents to append as an appendix to all manuals. -#texinfo_appendices = [] - -# If false, no module index is generated. -#texinfo_domain_indices = True - -# How to display URL addresses: 'footnote', 'no', or 'inline'. -#texinfo_show_urls = 'footnote' - -# If true, do not generate a @detailmenu in the "Top" node's menu. -#texinfo_no_detailmenu = False - - -intersphinx_mapping = { - 'python': ('https://docs.python.org/', None), - 'acme': ('https://acme-python.readthedocs.org/en/latest/', None), - 'certbot': ('https://certbot.eff.org/docs/', None), - 'certbot-apache': ( - 'https://letsencrypt-apache.readthedocs.org/en/latest/', None), - 'certbot-nginx': ( - 'https://letsencrypt-nginx.readthedocs.org/en/latest/', None), -} diff --git a/certbot-compatibility-test/docs/index.rst b/certbot-compatibility-test/docs/index.rst deleted file mode 100644 index a5e71e844..000000000 --- a/certbot-compatibility-test/docs/index.rst +++ /dev/null @@ -1,27 +0,0 @@ -.. certbot-compatibility-test documentation master file, created by - sphinx-quickstart on Sun Oct 18 13:40:53 2015. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to certbot-compatibility-test's documentation! -========================================================== - -Contents: - -.. toctree:: - :maxdepth: 2 - - -.. toctree:: - :maxdepth: 1 - - api - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` - diff --git a/certbot-compatibility-test/docs/make.bat b/certbot-compatibility-test/docs/make.bat deleted file mode 100644 index b6c0360f4..000000000 --- a/certbot-compatibility-test/docs/make.bat +++ /dev/null @@ -1,263 +0,0 @@ -@ECHO OFF - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set BUILDDIR=_build -set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . -set I18NSPHINXOPTS=%SPHINXOPTS% . -if NOT "%PAPER%" == "" ( - set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% - set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% -) - -if "%1" == "" goto help - -if "%1" == "help" ( - :help - echo.Please use `make ^` where ^ is one of - echo. html to make standalone HTML files - echo. dirhtml to make HTML files named index.html in directories - echo. singlehtml to make a single large HTML file - echo. pickle to make pickle files - echo. json to make JSON files - echo. htmlhelp to make HTML files and a HTML help project - echo. qthelp to make HTML files and a qthelp project - echo. devhelp to make HTML files and a Devhelp project - echo. epub to make an epub - echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter - echo. text to make text files - echo. man to make manual pages - echo. texinfo to make Texinfo files - echo. gettext to make PO message catalogs - echo. changes to make an overview over all changed/added/deprecated items - echo. xml to make Docutils-native XML files - echo. pseudoxml to make pseudoxml-XML files for display purposes - echo. linkcheck to check all external links for integrity - echo. doctest to run all doctests embedded in the documentation if enabled - echo. coverage to run coverage check of the documentation if enabled - goto end -) - -if "%1" == "clean" ( - for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i - del /q /s %BUILDDIR%\* - goto end -) - - -REM Check if sphinx-build is available and fallback to Python version if any -%SPHINXBUILD% 2> nul -if errorlevel 9009 goto sphinx_python -goto sphinx_ok - -:sphinx_python - -set SPHINXBUILD=python -m sphinx.__init__ -%SPHINXBUILD% 2> nul -if errorlevel 9009 ( - echo. - echo.The 'sphinx-build' command was not found. Make sure you have Sphinx - echo.installed, then set the SPHINXBUILD environment variable to point - echo.to the full path of the 'sphinx-build' executable. Alternatively you - echo.may add the Sphinx directory to PATH. - echo. - echo.If you don't have Sphinx installed, grab it from - echo.http://sphinx-doc.org/ - exit /b 1 -) - -:sphinx_ok - - -if "%1" == "html" ( - %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/html. - goto end -) - -if "%1" == "dirhtml" ( - %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. - goto end -) - -if "%1" == "singlehtml" ( - %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. - goto end -) - -if "%1" == "pickle" ( - %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the pickle files. - goto end -) - -if "%1" == "json" ( - %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the JSON files. - goto end -) - -if "%1" == "htmlhelp" ( - %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run HTML Help Workshop with the ^ -.hhp project file in %BUILDDIR%/htmlhelp. - goto end -) - -if "%1" == "qthelp" ( - %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run "qcollectiongenerator" with the ^ -.qhcp project file in %BUILDDIR%/qthelp, like this: - echo.^> qcollectiongenerator %BUILDDIR%\qthelp\certbot-compatibility-test.qhcp - echo.To view the help file: - echo.^> assistant -collectionFile %BUILDDIR%\qthelp\certbot-compatibility-test.ghc - goto end -) - -if "%1" == "devhelp" ( - %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. - goto end -) - -if "%1" == "epub" ( - %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The epub file is in %BUILDDIR%/epub. - goto end -) - -if "%1" == "latex" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "latexpdf" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - cd %BUILDDIR%/latex - make all-pdf - cd %~dp0 - echo. - echo.Build finished; the PDF files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "latexpdfja" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - cd %BUILDDIR%/latex - make all-pdf-ja - cd %~dp0 - echo. - echo.Build finished; the PDF files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "text" ( - %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The text files are in %BUILDDIR%/text. - goto end -) - -if "%1" == "man" ( - %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The manual pages are in %BUILDDIR%/man. - goto end -) - -if "%1" == "texinfo" ( - %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. - goto end -) - -if "%1" == "gettext" ( - %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The message catalogs are in %BUILDDIR%/locale. - goto end -) - -if "%1" == "changes" ( - %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes - if errorlevel 1 exit /b 1 - echo. - echo.The overview file is in %BUILDDIR%/changes. - goto end -) - -if "%1" == "linkcheck" ( - %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck - if errorlevel 1 exit /b 1 - echo. - echo.Link check complete; look for any errors in the above output ^ -or in %BUILDDIR%/linkcheck/output.txt. - goto end -) - -if "%1" == "doctest" ( - %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest - if errorlevel 1 exit /b 1 - echo. - echo.Testing of doctests in the sources finished, look at the ^ -results in %BUILDDIR%/doctest/output.txt. - goto end -) - -if "%1" == "coverage" ( - %SPHINXBUILD% -b coverage %ALLSPHINXOPTS% %BUILDDIR%/coverage - if errorlevel 1 exit /b 1 - echo. - echo.Testing of coverage in the sources finished, look at the ^ -results in %BUILDDIR%/coverage/python.txt. - goto end -) - -if "%1" == "xml" ( - %SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The XML files are in %BUILDDIR%/xml. - goto end -) - -if "%1" == "pseudoxml" ( - %SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml. - goto end -) - -:end diff --git a/certbot-compatibility-test/nginx/roundtrip.py b/certbot-compatibility-test/nginx/roundtrip.py index 85d283c78..afc68647d 100644 --- a/certbot-compatibility-test/nginx/roundtrip.py +++ b/certbot-compatibility-test/nginx/roundtrip.py @@ -3,7 +3,8 @@ import os import sys -from certbot_nginx import nginxparser +from certbot_nginx._internal import nginxparser + def roundtrip(stuff): success = True diff --git a/certbot-compatibility-test/readthedocs.org.requirements.txt b/certbot-compatibility-test/readthedocs.org.requirements.txt deleted file mode 100644 index c2a0c1110..000000000 --- a/certbot-compatibility-test/readthedocs.org.requirements.txt +++ /dev/null @@ -1,13 +0,0 @@ -# readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation -# dependencies), but it allows to specify a requirements.txt file at -# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) - -# Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead - --e acme --e . --e certbot-apache --e certbot-compatibility-test[docs] diff --git a/certbot-compatibility-test/setup.py b/certbot-compatibility-test/setup.py index f519ed422..1dbcefa75 100644 --- a/certbot-compatibility-test/setup.py +++ b/certbot-compatibility-test/setup.py @@ -1,10 +1,9 @@ import sys -from setuptools import setup from setuptools import find_packages +from setuptools import setup - -version = '0.31.0.dev0' +version = '1.3.0.dev0' install_requires = [ 'certbot', @@ -20,11 +19,6 @@ if sys.version_info < (2, 7, 9): install_requires.append('ndg-httpsclient') install_requires.append('pyasn1') -docs_extras = [ - 'repoze.sphinx.autointerface', - 'Sphinx>=1.0', # autodoc_member_order = 'bysource', autodoc_default_flags - 'sphinx_rtd_theme', -] setup( name='certbot-compatibility-test', @@ -34,7 +28,7 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ 'Development Status :: 3 - Alpha', 'Intended Audience :: Developers', @@ -43,10 +37,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', ], @@ -54,9 +48,6 @@ setup( packages=find_packages(), include_package_data=True, install_requires=install_requires, - extras_require={ - 'docs': docs_extras, - }, entry_points={ 'console_scripts': [ 'certbot-compatibility-test = certbot_compatibility_test.test_driver:main', diff --git a/certbot-dns-cloudflare/Dockerfile b/certbot-dns-cloudflare/Dockerfile deleted file mode 100644 index 27dcc8751..000000000 --- a/certbot-dns-cloudflare/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-cloudflare - -RUN pip install --no-cache-dir --editable src/certbot-dns-cloudflare diff --git a/certbot-dns-cloudflare/MANIFEST.in b/certbot-dns-cloudflare/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-cloudflare/MANIFEST.in +++ b/certbot-dns-cloudflare/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-cloudflare/certbot_dns_cloudflare/__init__.py b/certbot-dns-cloudflare/certbot_dns_cloudflare/__init__.py index 7e53f83ce..11886ea54 100644 --- a/certbot-dns-cloudflare/certbot_dns_cloudflare/__init__.py +++ b/certbot-dns-cloudflare/certbot_dns_cloudflare/__init__.py @@ -22,15 +22,40 @@ Credentials Use of this plugin requires a configuration file containing Cloudflare API credentials, obtained from your Cloudflare -`account page `_. +`account page `_. + +Previously, Cloudflare's "Global API Key" was used for authentication, however +this key can access the entire Cloudflare API for all domains in your account, +meaning it could cause a lot of damage if leaked. + +Cloudflare's newer API Tokens can be restricted to specific domains and +operations, and are therefore now the recommended authentication option. + +However, due to some shortcomings in Cloudflare's implementation of Tokens, +Tokens created for Certbot currently require ``Zone:Zone:Read`` and ``Zone:DNS:Edit`` +permissions for **all** zones in your account. While this is not ideal, your Token +will still have fewer permission than the Global key, so it's still worth doing. +Hopefully Cloudflare will improve this in the future. + +Using Cloudflare Tokens also requires at least version 2.3.1 of the ``cloudflare`` +python module. If the version that automatically installed with this plugin is +older than that, and you can't upgrade it on your system, you'll have to stick to +the Global key. .. code-block:: ini - :name: credentials.ini - :caption: Example credentials file: + :name: certbot_cloudflare_token.ini + :caption: Example credentials file using restricted API Token (recommended): + + # Cloudflare API token used by Certbot + dns_cloudflare_api_token = 0123456789abcdef0123456789abcdef01234567 + +.. code-block:: ini + :name: certbot_cloudflare_key.ini + :caption: Example credentials file using Global API Key (not recommended): # Cloudflare API credentials used by Certbot dns_cloudflare_email = cloudflare@example.com - dns_cloudflare_api_key = 0123456789abcdef0123456789abcdef01234567 + dns_cloudflare_api_key = 0123456789abcdef0123456789abcdef01234 The path to this file can be provided interactively or using the ``--dns-cloudflare-credentials`` command-line argument. Certbot records the path diff --git a/certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/__init__.py b/certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/__init__.py new file mode 100644 index 000000000..93b0672b5 --- /dev/null +++ b/certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_cloudflare.dns_cloudflare` plugin.""" diff --git a/certbot-dns-cloudflare/certbot_dns_cloudflare/dns_cloudflare.py b/certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/dns_cloudflare.py similarity index 66% rename from certbot-dns-cloudflare/certbot_dns_cloudflare/dns_cloudflare.py rename to certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/dns_cloudflare.py index 7604a8baf..22124ac04 100644 --- a/certbot-dns-cloudflare/certbot_dns_cloudflare/dns_cloudflare.py +++ b/certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/dns_cloudflare.py @@ -4,13 +4,17 @@ import logging import CloudFlare import zope.interface +from acme.magic_typing import Any +from acme.magic_typing import Dict +from acme.magic_typing import List + from certbot import errors from certbot import interfaces from certbot.plugins import dns_common logger = logging.getLogger(__name__) -ACCOUNT_URL = 'https://www.cloudflare.com/a/account/my-account' +ACCOUNT_URL = 'https://dash.cloudflare.com/profile/api-tokens' @zope.interface.implementer(interfaces.IAuthenticator) @@ -38,14 +42,35 @@ class Authenticator(dns_common.DNSAuthenticator): return 'This plugin configures a DNS TXT record to respond to a dns-01 challenge using ' + \ 'the Cloudflare API.' + def _validate_credentials(self, credentials): + token = credentials.conf('api-token') + email = credentials.conf('email') + key = credentials.conf('api-key') + if token: + if email or key: + raise errors.PluginError('{}: dns_cloudflare_email and dns_cloudflare_api_key are ' + 'not needed when using an API Token' + .format(credentials.confobj.filename)) + elif email or key: + if not email: + raise errors.PluginError('{}: dns_cloudflare_email is required when using a Global ' + 'API Key. (should be email address associated with ' + 'Cloudflare account)'.format(credentials.confobj.filename)) + if not key: + raise errors.PluginError('{}: dns_cloudflare_api_key is required when using a ' + 'Global API Key. (see {})' + .format(credentials.confobj.filename, ACCOUNT_URL)) + else: + raise errors.PluginError('{}: Either dns_cloudflare_api_token (recommended), or ' + 'dns_cloudflare_email and dns_cloudflare_api_key are required.' + ' (see {})'.format(credentials.confobj.filename, ACCOUNT_URL)) + def _setup_credentials(self): self.credentials = self._configure_credentials( 'credentials', 'Cloudflare credentials INI file', - { - 'email': 'email address associated with Cloudflare account', - 'api-key': 'API key for Cloudflare account, obtained from {0}'.format(ACCOUNT_URL) - } + None, + self._validate_credentials ) def _perform(self, domain, validation_name, validation): @@ -55,6 +80,8 @@ class Authenticator(dns_common.DNSAuthenticator): self._get_cloudflare_client().del_txt_record(domain, validation_name, validation) def _get_cloudflare_client(self): + if self.credentials.conf('api-token'): + return _CloudflareClient(None, self.credentials.conf('api-token')) return _CloudflareClient(self.credentials.conf('email'), self.credentials.conf('api-key')) @@ -88,8 +115,15 @@ class _CloudflareClient(object): logger.debug('Attempting to add record to zone %s: %s', zone_id, data) self.cf.zones.dns_records.post(zone_id, data=data) # zones | pylint: disable=no-member except CloudFlare.exceptions.CloudFlareAPIError as e: + code = int(e) + hint = None + + if code == 9109: + hint = 'Does your API token have "Zone:DNS:Edit" permissions?' + logger.error('Encountered CloudFlareAPIError adding TXT record: %d %s', e, e) - raise errors.PluginError('Error communicating with the Cloudflare API: {0}'.format(e)) + raise errors.PluginError('Error communicating with the Cloudflare API: {0}{1}' + .format(e, ' ({0})'.format(hint) if hint else '')) record_id = self._find_txt_record_id(zone_id, record_name, record_content) logger.debug('Successfully added TXT record with record_id: %s', record_id) @@ -139,6 +173,8 @@ class _CloudflareClient(object): """ zone_name_guesses = dns_common.base_domain_name_guesses(domain) + zones = [] # type: List[Dict[str, Any]] + code = msg = None for zone_name in zone_name_guesses: params = {'name': zone_name, @@ -148,26 +184,37 @@ class _CloudflareClient(object): zones = self.cf.zones.get(params=params) # zones | pylint: disable=no-member except CloudFlare.exceptions.CloudFlareAPIError as e: code = int(e) + msg = str(e) hint = None if code == 6003: - hint = 'Did you copy your entire API key?' + hint = ('Did you copy your entire API token/key? To use Cloudflare tokens, ' + 'you\'ll need the python package cloudflare>=2.3.1.{}' + .format(' This certbot is running cloudflare ' + str(CloudFlare.__version__) + if hasattr(CloudFlare, '__version__') else '')) elif code == 9103: - hint = 'Did you enter the correct email address?' + hint = 'Did you enter the correct email address and Global key?' + elif code == 9109: + hint = 'Did you enter a valid Cloudflare Token?' - raise errors.PluginError('Error determining zone_id: {0} {1}. Please confirm that ' - 'you have supplied valid Cloudflare API credentials.{2}' - .format(code, e, ' ({0})'.format(hint) if hint else '')) + if hint: + raise errors.PluginError('Error determining zone_id: {0} {1}. Please confirm ' + 'that you have supplied valid Cloudflare API credentials. ({2})' + .format(code, msg, hint)) + else: + logger.debug('Unrecognised CloudFlareAPIError while finding zone_id: %d %s. ' + 'Continuing with next zone guess...', e, e) - if len(zones) > 0: + if zones: zone_id = zones[0]['id'] logger.debug('Found zone_id of %s for %s using name %s', zone_id, domain, zone_name) return zone_id raise errors.PluginError('Unable to determine zone_id for {0} using zone names: {1}. ' - 'Please confirm that the domain name has been entered correctly ' - 'and is already associated with the supplied Cloudflare account.' - .format(domain, zone_name_guesses)) + 'Please confirm that the domain name has been entered correctly ' + 'and is already associated with the supplied Cloudflare account.{2}' + .format(domain, zone_name_guesses, ' The error from Cloudflare was:' + ' {0} {1}'.format(code, msg) if code is not None else '')) def _find_txt_record_id(self, zone_id, record_name, record_content): """ @@ -191,9 +238,9 @@ class _CloudflareClient(object): logger.debug('Encountered CloudFlareAPIError getting TXT record_id: %s', e) records = [] - if len(records) > 0: + if records: # Cleanup is returning the system to the state we found it. If, for some reason, # there are multiple matching records, we only delete one because we only added one. return records[0]['id'] - else: - logger.debug('Unable to find TXT record.') + logger.debug('Unable to find TXT record.') + return None diff --git a/certbot-dns-cloudflare/docs/api.rst b/certbot-dns-cloudflare/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-cloudflare/docs/api.rst +++ b/certbot-dns-cloudflare/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-cloudflare/docs/api/dns_cloudflare.rst b/certbot-dns-cloudflare/docs/api/dns_cloudflare.rst deleted file mode 100644 index 35f525201..000000000 --- a/certbot-dns-cloudflare/docs/api/dns_cloudflare.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_cloudflare.dns_cloudflare` --------------------------------------------- - -.. automodule:: certbot_dns_cloudflare.dns_cloudflare - :members: diff --git a/certbot-dns-cloudflare/docs/conf.py b/certbot-dns-cloudflare/docs/conf.py index aa7809246..97e54421e 100644 --- a/certbot-dns-cloudflare/docs/conf.py +++ b/certbot-dns-cloudflare/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-cloudflare/local-oldest-requirements.txt b/certbot-dns-cloudflare/local-oldest-requirements.txt index 8368d266e..cf61c15a5 100644 --- a/certbot-dns-cloudflare/local-oldest-requirements.txt +++ b/certbot-dns-cloudflare/local-oldest-requirements.txt @@ -1,2 +1,3 @@ -acme[dev]==0.21.1 -certbot[dev]==0.21.1 +# Remember to update setup.py to match the package versions below. +acme[dev]==0.29.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-cloudflare/readthedocs.org.requirements.txt b/certbot-dns-cloudflare/readthedocs.org.requirements.txt index b18901111..f1df15227 100644 --- a/certbot-dns-cloudflare/readthedocs.org.requirements.txt +++ b/certbot-dns-cloudflare/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-cloudflare[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-cloudflare[docs]" does not work as +# expected and "pip install -e certbot-dns-cloudflare[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-cloudflare[docs] diff --git a/certbot-dns-cloudflare/setup.py b/certbot-dns-cloudflare/setup.py index ff33293fe..9376bc1c4 100644 --- a/certbot-dns-cloudflare/setup.py +++ b/certbot-dns-cloudflare/setup.py @@ -1,14 +1,16 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.21.1', - 'certbot>=0.21.1', + 'acme>=0.29.0', + 'certbot>=1.1.0', 'cloudflare>=1.5.1', 'mock', 'setuptools', @@ -20,6 +22,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-cloudflare', version=version, @@ -28,9 +44,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -39,10 +55,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -59,8 +75,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-cloudflare = certbot_dns_cloudflare.dns_cloudflare:Authenticator', + 'dns-cloudflare = certbot_dns_cloudflare._internal.dns_cloudflare:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_cloudflare', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-cloudflare/certbot_dns_cloudflare/dns_cloudflare_test.py b/certbot-dns-cloudflare/tests/dns_cloudflare_test.py similarity index 65% rename from certbot-dns-cloudflare/certbot_dns_cloudflare/dns_cloudflare_test.py rename to certbot-dns-cloudflare/tests/dns_cloudflare_test.py index e60d6ff8b..d38330191 100644 --- a/certbot-dns-cloudflare/certbot_dns_cloudflare/dns_cloudflare_test.py +++ b/certbot-dns-cloudflare/tests/dns_cloudflare_test.py @@ -1,17 +1,20 @@ -"""Tests for certbot_dns_cloudflare.dns_cloudflare.""" +"""Tests for certbot_dns_cloudflare._internal.dns_cloudflare.""" -import os import unittest import CloudFlare import mock from certbot import errors +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins.dns_test_common import DOMAIN from certbot.tests import util as test_util API_ERROR = CloudFlare.exceptions.CloudFlareAPIError(1000, '', '') + +API_TOKEN = 'an-api-token' + API_KEY = 'an-api-key' EMAIL = 'example@example.com' @@ -19,7 +22,7 @@ EMAIL = 'example@example.com' class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthenticatorTest): def setUp(self): - from certbot_dns_cloudflare.dns_cloudflare import Authenticator + from certbot_dns_cloudflare._internal.dns_cloudflare import Authenticator super(AuthenticatorTest, self).setUp() @@ -49,6 +52,50 @@ class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthentic expected = [mock.call.del_txt_record(DOMAIN, '_acme-challenge.'+DOMAIN, mock.ANY)] self.assertEqual(expected, self.mock_client.mock_calls) + def test_api_token(self): + dns_test_common.write({"cloudflare_api_token": API_TOKEN}, + self.config.cloudflare_credentials) + self.auth.perform([self.achall]) + + expected = [mock.call.add_txt_record(DOMAIN, '_acme-challenge.'+DOMAIN, mock.ANY, mock.ANY)] + self.assertEqual(expected, self.mock_client.mock_calls) + + def test_no_creds(self): + dns_test_common.write({}, self.config.cloudflare_credentials) + self.assertRaises(errors.PluginError, + self.auth.perform, + [self.achall]) + + def test_missing_email_or_key(self): + dns_test_common.write({"cloudflare_api_key": API_KEY}, self.config.cloudflare_credentials) + self.assertRaises(errors.PluginError, + self.auth.perform, + [self.achall]) + + dns_test_common.write({"cloudflare_email": EMAIL}, self.config.cloudflare_credentials) + self.assertRaises(errors.PluginError, + self.auth.perform, + [self.achall]) + + def test_email_or_key_with_token(self): + dns_test_common.write({"cloudflare_api_token": API_TOKEN, "cloudflare_email": EMAIL}, + self.config.cloudflare_credentials) + self.assertRaises(errors.PluginError, + self.auth.perform, + [self.achall]) + + dns_test_common.write({"cloudflare_api_token": API_TOKEN, "cloudflare_api_key": API_KEY}, + self.config.cloudflare_credentials) + self.assertRaises(errors.PluginError, + self.auth.perform, + [self.achall]) + + dns_test_common.write({"cloudflare_api_token": API_TOKEN, "cloudflare_email": EMAIL, + "cloudflare_api_key": API_KEY}, self.config.cloudflare_credentials) + self.assertRaises(errors.PluginError, + self.auth.perform, + [self.achall]) + class CloudflareClientTest(unittest.TestCase): record_name = "foo" @@ -58,7 +105,7 @@ class CloudflareClientTest(unittest.TestCase): record_id = 2 def setUp(self): - from certbot_dns_cloudflare.dns_cloudflare import _CloudflareClient + from certbot_dns_cloudflare._internal.dns_cloudflare import _CloudflareClient self.cloudflare_client = _CloudflareClient(EMAIL, API_KEY) @@ -83,7 +130,7 @@ class CloudflareClientTest(unittest.TestCase): def test_add_txt_record_error(self): self.cf.zones.get.return_value = [{'id': self.zone_id}] - self.cf.zones.dns_records.post.side_effect = API_ERROR + self.cf.zones.dns_records.post.side_effect = CloudFlare.exceptions.CloudFlareAPIError(9109, '', '') self.assertRaises( errors.PluginError, @@ -106,6 +153,25 @@ class CloudflareClientTest(unittest.TestCase): self.cloudflare_client.add_txt_record, DOMAIN, self.record_name, self.record_content, self.record_ttl) + def test_add_txt_record_bad_creds(self): + self.cf.zones.get.side_effect = CloudFlare.exceptions.CloudFlareAPIError(6003, '', '') + self.assertRaises( + errors.PluginError, + self.cloudflare_client.add_txt_record, + DOMAIN, self.record_name, self.record_content, self.record_ttl) + + self.cf.zones.get.side_effect = CloudFlare.exceptions.CloudFlareAPIError(9103, '', '') + self.assertRaises( + errors.PluginError, + self.cloudflare_client.add_txt_record, + DOMAIN, self.record_name, self.record_content, self.record_ttl) + + self.cf.zones.get.side_effect = CloudFlare.exceptions.CloudFlareAPIError(9109, '', '') + self.assertRaises( + errors.PluginError, + self.cloudflare_client.add_txt_record, + DOMAIN, self.record_name, self.record_content, self.record_ttl) + def test_del_txt_record(self): self.cf.zones.get.return_value = [{'id': self.zone_id}] self.cf.zones.dns_records.get.return_value = [{'id': self.record_id}] diff --git a/certbot-dns-cloudxns/Dockerfile b/certbot-dns-cloudxns/Dockerfile deleted file mode 100644 index cc84ea65b..000000000 --- a/certbot-dns-cloudxns/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-cloudxns - -RUN pip install --no-cache-dir --editable src/certbot-dns-cloudxns diff --git a/certbot-dns-cloudxns/MANIFEST.in b/certbot-dns-cloudxns/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-cloudxns/MANIFEST.in +++ b/certbot-dns-cloudxns/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-cloudxns/certbot_dns_cloudxns/_internal/__init__.py b/certbot-dns-cloudxns/certbot_dns_cloudxns/_internal/__init__.py new file mode 100644 index 000000000..e2177417d --- /dev/null +++ b/certbot-dns-cloudxns/certbot_dns_cloudxns/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_cloudxns.dns_cloudxns` plugin.""" diff --git a/certbot-dns-cloudxns/certbot_dns_cloudxns/dns_cloudxns.py b/certbot-dns-cloudxns/certbot_dns_cloudxns/_internal/dns_cloudxns.py similarity index 100% rename from certbot-dns-cloudxns/certbot_dns_cloudxns/dns_cloudxns.py rename to certbot-dns-cloudxns/certbot_dns_cloudxns/_internal/dns_cloudxns.py index 5132137f8..2a0f12ea7 100644 --- a/certbot-dns-cloudxns/certbot_dns_cloudxns/dns_cloudxns.py +++ b/certbot-dns-cloudxns/certbot_dns_cloudxns/_internal/dns_cloudxns.py @@ -1,8 +1,8 @@ """DNS Authenticator for CloudXNS DNS.""" import logging -import zope.interface from lexicon.providers import cloudxns +import zope.interface from certbot import errors from certbot import interfaces diff --git a/certbot-dns-cloudxns/docs/api.rst b/certbot-dns-cloudxns/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-cloudxns/docs/api.rst +++ b/certbot-dns-cloudxns/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-cloudxns/docs/api/dns_cloudxns.rst b/certbot-dns-cloudxns/docs/api/dns_cloudxns.rst deleted file mode 100644 index be794d1a0..000000000 --- a/certbot-dns-cloudxns/docs/api/dns_cloudxns.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_cloudxns.dns_cloudxns` ----------------------------------------- - -.. automodule:: certbot_dns_cloudxns.dns_cloudxns - :members: diff --git a/certbot-dns-cloudxns/docs/conf.py b/certbot-dns-cloudxns/docs/conf.py index 9e2f4c0e6..1fc05c94c 100644 --- a/certbot-dns-cloudxns/docs/conf.py +++ b/certbot-dns-cloudxns/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-cloudxns/local-oldest-requirements.txt b/certbot-dns-cloudxns/local-oldest-requirements.txt index 65f5a758e..1307698d4 100644 --- a/certbot-dns-cloudxns/local-oldest-requirements.txt +++ b/certbot-dns-cloudxns/local-oldest-requirements.txt @@ -1,2 +1,3 @@ --e acme[dev] --e .[dev] +# Remember to update setup.py to match the package versions below. +acme[dev]==0.31.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-cloudxns/readthedocs.org.requirements.txt b/certbot-dns-cloudxns/readthedocs.org.requirements.txt index ae2ff8165..a9a4d068b 100644 --- a/certbot-dns-cloudxns/readthedocs.org.requirements.txt +++ b/certbot-dns-cloudxns/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-cloudxns[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-cloudxns[docs]" does not work as +# expected and "pip install -e certbot-dns-cloudxns[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-cloudxns[docs] diff --git a/certbot-dns-cloudxns/setup.py b/certbot-dns-cloudxns/setup.py index 1a6f900d8..4e99ff5ff 100644 --- a/certbot-dns-cloudxns/setup.py +++ b/certbot-dns-cloudxns/setup.py @@ -1,14 +1,16 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.31.0.dev0', - 'certbot>=0.31.0.dev0', + 'acme>=0.31.0', + 'certbot>=1.1.0', 'dns-lexicon>=2.2.1', # Support for >1 TXT record per name 'mock', 'setuptools', @@ -20,6 +22,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-cloudxns', version=version, @@ -28,9 +44,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -39,10 +55,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -59,8 +75,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-cloudxns = certbot_dns_cloudxns.dns_cloudxns:Authenticator', + 'dns-cloudxns = certbot_dns_cloudxns._internal.dns_cloudxns:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_cloudxns', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-cloudxns/certbot_dns_cloudxns/dns_cloudxns_test.py b/certbot-dns-cloudxns/tests/dns_cloudxns_test.py similarity index 80% rename from certbot-dns-cloudxns/certbot_dns_cloudxns/dns_cloudxns_test.py rename to certbot-dns-cloudxns/tests/dns_cloudxns_test.py index c9bad23ab..a1e3cde89 100644 --- a/certbot-dns-cloudxns/certbot_dns_cloudxns/dns_cloudxns_test.py +++ b/certbot-dns-cloudxns/tests/dns_cloudxns_test.py @@ -1,11 +1,12 @@ -"""Tests for certbot_dns_cloudxns.dns_cloudxns.""" +"""Tests for certbot_dns_cloudxns._internal.dns_cloudxns.""" -import os import unittest import mock -from requests.exceptions import HTTPError, RequestException +from requests.exceptions import HTTPError +from requests.exceptions import RequestException +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins import dns_test_common_lexicon from certbot.tests import util as test_util @@ -24,7 +25,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, def setUp(self): super(AuthenticatorTest, self).setUp() - from certbot_dns_cloudxns.dns_cloudxns import Authenticator + from certbot_dns_cloudxns._internal.dns_cloudxns import Authenticator path = os.path.join(self.tempdir, 'file.ini') dns_test_common.write({"cloudxns_api_key": API_KEY, "cloudxns_secret_key": SECRET}, path) @@ -42,7 +43,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, class CloudXNSLexiconClientTest(unittest.TestCase, dns_test_common_lexicon.BaseLexiconClientTest): def setUp(self): - from certbot_dns_cloudxns.dns_cloudxns import _CloudXNSLexiconClient + from certbot_dns_cloudxns._internal.dns_cloudxns import _CloudXNSLexiconClient self.client = _CloudXNSLexiconClient(API_KEY, SECRET, 0) diff --git a/certbot-dns-digitalocean/Dockerfile b/certbot-dns-digitalocean/Dockerfile deleted file mode 100644 index 8bdd0619f..000000000 --- a/certbot-dns-digitalocean/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-digitalocean - -RUN pip install --no-cache-dir --editable src/certbot-dns-digitalocean diff --git a/certbot-dns-digitalocean/MANIFEST.in b/certbot-dns-digitalocean/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-digitalocean/MANIFEST.in +++ b/certbot-dns-digitalocean/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-digitalocean/certbot_dns_digitalocean/_internal/__init__.py b/certbot-dns-digitalocean/certbot_dns_digitalocean/_internal/__init__.py new file mode 100644 index 000000000..0291a9341 --- /dev/null +++ b/certbot-dns-digitalocean/certbot_dns_digitalocean/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_digitalocean.dns_digitalocean` plugin.""" diff --git a/certbot-dns-digitalocean/certbot_dns_digitalocean/dns_digitalocean.py b/certbot-dns-digitalocean/certbot_dns_digitalocean/_internal/dns_digitalocean.py similarity index 99% rename from certbot-dns-digitalocean/certbot_dns_digitalocean/dns_digitalocean.py rename to certbot-dns-digitalocean/certbot_dns_digitalocean/_internal/dns_digitalocean.py index 5a4f22327..7f3abbe31 100644 --- a/certbot-dns-digitalocean/certbot_dns_digitalocean/dns_digitalocean.py +++ b/certbot-dns-digitalocean/certbot_dns_digitalocean/_internal/dns_digitalocean.py @@ -154,7 +154,7 @@ class _DigitalOceanClient(object): for guess in domain_name_guesses: matches = [domain for domain in domains if domain.name == guess] - if len(matches) > 0: + if matches: domain = matches[0] logger.debug('Found base domain for %s using name %s', domain_name, guess) return domain diff --git a/certbot-dns-digitalocean/docs/api.rst b/certbot-dns-digitalocean/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-digitalocean/docs/api.rst +++ b/certbot-dns-digitalocean/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-digitalocean/docs/api/dns_digitalocean.rst b/certbot-dns-digitalocean/docs/api/dns_digitalocean.rst deleted file mode 100644 index 8a787987e..000000000 --- a/certbot-dns-digitalocean/docs/api/dns_digitalocean.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_digitalocean.dns_digitalocean` ------------------------------------------------- - -.. automodule:: certbot_dns_digitalocean.dns_digitalocean - :members: diff --git a/certbot-dns-digitalocean/docs/conf.py b/certbot-dns-digitalocean/docs/conf.py index e223b1535..0741e4cea 100644 --- a/certbot-dns-digitalocean/docs/conf.py +++ b/certbot-dns-digitalocean/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-digitalocean/local-oldest-requirements.txt b/certbot-dns-digitalocean/local-oldest-requirements.txt index 8368d266e..cf61c15a5 100644 --- a/certbot-dns-digitalocean/local-oldest-requirements.txt +++ b/certbot-dns-digitalocean/local-oldest-requirements.txt @@ -1,2 +1,3 @@ -acme[dev]==0.21.1 -certbot[dev]==0.21.1 +# Remember to update setup.py to match the package versions below. +acme[dev]==0.29.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-digitalocean/readthedocs.org.requirements.txt b/certbot-dns-digitalocean/readthedocs.org.requirements.txt index 08d973ab3..d0cc2f74a 100644 --- a/certbot-dns-digitalocean/readthedocs.org.requirements.txt +++ b/certbot-dns-digitalocean/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-digitalocean[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-digitalocean[docs]" does not work as +# expected and "pip install -e certbot-dns-digitalocean[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-digitalocean[docs] diff --git a/certbot-dns-digitalocean/setup.py b/certbot-dns-digitalocean/setup.py index 2f7fa37d6..9c9d1717c 100644 --- a/certbot-dns-digitalocean/setup.py +++ b/certbot-dns-digitalocean/setup.py @@ -1,14 +1,16 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.21.1', - 'certbot>=0.21.1', + 'acme>=0.29.0', + 'certbot>=1.1.0', 'mock', 'python-digitalocean>=1.11', 'setuptools', @@ -21,6 +23,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-digitalocean', version=version, @@ -29,9 +45,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -40,10 +56,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -60,8 +76,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-digitalocean = certbot_dns_digitalocean.dns_digitalocean:Authenticator', + 'dns-digitalocean = certbot_dns_digitalocean._internal.dns_digitalocean:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_digitalocean', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-digitalocean/certbot_dns_digitalocean/dns_digitalocean_test.py b/certbot-dns-digitalocean/tests/dns_digitalocean_test.py similarity index 95% rename from certbot-dns-digitalocean/certbot_dns_digitalocean/dns_digitalocean_test.py rename to certbot-dns-digitalocean/tests/dns_digitalocean_test.py index 0e2043f50..71301a47c 100644 --- a/certbot-dns-digitalocean/certbot_dns_digitalocean/dns_digitalocean_test.py +++ b/certbot-dns-digitalocean/tests/dns_digitalocean_test.py @@ -1,12 +1,12 @@ -"""Tests for certbot_dns_digitalocean.dns_digitalocean.""" +"""Tests for certbot_dns_digitalocean._internal.dns_digitalocean.""" -import os import unittest import digitalocean import mock from certbot import errors +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins.dns_test_common import DOMAIN from certbot.tests import util as test_util @@ -18,7 +18,7 @@ TOKEN = 'a-token' class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthenticatorTest): def setUp(self): - from certbot_dns_digitalocean.dns_digitalocean import Authenticator + from certbot_dns_digitalocean._internal.dns_digitalocean import Authenticator super(AuthenticatorTest, self).setUp() @@ -57,7 +57,7 @@ class DigitalOceanClientTest(unittest.TestCase): record_content = "bar" def setUp(self): - from certbot_dns_digitalocean.dns_digitalocean import _DigitalOceanClient + from certbot_dns_digitalocean._internal.dns_digitalocean import _DigitalOceanClient self.digitalocean_client = _DigitalOceanClient(TOKEN) diff --git a/certbot-dns-dnsimple/Dockerfile b/certbot-dns-dnsimple/Dockerfile deleted file mode 100644 index 38d2be80e..000000000 --- a/certbot-dns-dnsimple/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-dnsimple - -RUN pip install --no-cache-dir --editable src/certbot-dns-dnsimple diff --git a/certbot-dns-dnsimple/MANIFEST.in b/certbot-dns-dnsimple/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-dnsimple/MANIFEST.in +++ b/certbot-dns-dnsimple/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-dnsimple/certbot_dns_dnsimple/_internal/__init__.py b/certbot-dns-dnsimple/certbot_dns_dnsimple/_internal/__init__.py new file mode 100644 index 000000000..070927555 --- /dev/null +++ b/certbot-dns-dnsimple/certbot_dns_dnsimple/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_dnsimple.dns_dnsimple` plugin.""" diff --git a/certbot-dns-dnsimple/certbot_dns_dnsimple/dns_dnsimple.py b/certbot-dns-dnsimple/certbot_dns_dnsimple/_internal/dns_dnsimple.py similarity index 100% rename from certbot-dns-dnsimple/certbot_dns_dnsimple/dns_dnsimple.py rename to certbot-dns-dnsimple/certbot_dns_dnsimple/_internal/dns_dnsimple.py index ad2a3fa30..8c48d31e7 100644 --- a/certbot-dns-dnsimple/certbot_dns_dnsimple/dns_dnsimple.py +++ b/certbot-dns-dnsimple/certbot_dns_dnsimple/_internal/dns_dnsimple.py @@ -1,8 +1,8 @@ """DNS Authenticator for DNSimple DNS.""" import logging -import zope.interface from lexicon.providers import dnsimple +import zope.interface from certbot import errors from certbot import interfaces diff --git a/certbot-dns-dnsimple/docs/api.rst b/certbot-dns-dnsimple/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-dnsimple/docs/api.rst +++ b/certbot-dns-dnsimple/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-dnsimple/docs/api/dns_dnsimple.rst b/certbot-dns-dnsimple/docs/api/dns_dnsimple.rst deleted file mode 100644 index b0544107b..000000000 --- a/certbot-dns-dnsimple/docs/api/dns_dnsimple.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_dnsimple.dns_dnsimple` ----------------------------------------- - -.. automodule:: certbot_dns_dnsimple.dns_dnsimple - :members: diff --git a/certbot-dns-dnsimple/docs/conf.py b/certbot-dns-dnsimple/docs/conf.py index da692fb9e..99cc93135 100644 --- a/certbot-dns-dnsimple/docs/conf.py +++ b/certbot-dns-dnsimple/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-dnsimple/local-oldest-requirements.txt b/certbot-dns-dnsimple/local-oldest-requirements.txt index 65f5a758e..1307698d4 100644 --- a/certbot-dns-dnsimple/local-oldest-requirements.txt +++ b/certbot-dns-dnsimple/local-oldest-requirements.txt @@ -1,2 +1,3 @@ --e acme[dev] --e .[dev] +# Remember to update setup.py to match the package versions below. +acme[dev]==0.31.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-dnsimple/readthedocs.org.requirements.txt b/certbot-dns-dnsimple/readthedocs.org.requirements.txt index fef73916c..04163ff34 100644 --- a/certbot-dns-dnsimple/readthedocs.org.requirements.txt +++ b/certbot-dns-dnsimple/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-dnsimple[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-dnsimple[docs]" does not work as +# expected and "pip install -e certbot-dns-dnsimple[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-dnsimple[docs] diff --git a/certbot-dns-dnsimple/setup.py b/certbot-dns-dnsimple/setup.py index 1a2ce5d92..9cde6214c 100644 --- a/certbot-dns-dnsimple/setup.py +++ b/certbot-dns-dnsimple/setup.py @@ -1,25 +1,53 @@ -from setuptools import setup +import os +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.31.0.dev0', - 'certbot>=0.31.0.dev0', - 'dns-lexicon>=2.2.1', # Support for >1 TXT record per name + 'acme>=0.31.0', + 'certbot>=1.1.0', 'mock', 'setuptools', 'zope.interface', ] +# This package normally depends on dns-lexicon>=3.2.1 to address the +# problem described in https://github.com/AnalogJ/lexicon/issues/387, +# however, the fix there has been backported to older versions of +# lexicon found in various Linux distros. This conditional helps us test +# that we've maintained compatibility with these versions of lexicon +# which allows us to potentially upgrade our packages in these distros +# as necessary. +if os.environ.get('CERTBOT_OLDEST') == '1': + install_requires.append('dns-lexicon>=2.2.1') +else: + install_requires.append('dns-lexicon>=3.2.1') + docs_extras = [ 'Sphinx>=1.0', # autodoc_member_order = 'bysource', autodoc_default_flags 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-dnsimple', version=version, @@ -28,9 +56,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -39,10 +67,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -59,8 +87,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-dnsimple = certbot_dns_dnsimple.dns_dnsimple:Authenticator', + 'dns-dnsimple = certbot_dns_dnsimple._internal.dns_dnsimple:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_dnsimple', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-dnsimple/certbot_dns_dnsimple/dns_dnsimple_test.py b/certbot-dns-dnsimple/tests/dns_dnsimple_test.py similarity index 84% rename from certbot-dns-dnsimple/certbot_dns_dnsimple/dns_dnsimple_test.py rename to certbot-dns-dnsimple/tests/dns_dnsimple_test.py index d8f3a23ea..ca5eb4f36 100644 --- a/certbot-dns-dnsimple/certbot_dns_dnsimple/dns_dnsimple_test.py +++ b/certbot-dns-dnsimple/tests/dns_dnsimple_test.py @@ -1,11 +1,11 @@ -"""Tests for certbot_dns_dnsimple.dns_dnsimple.""" +"""Tests for certbot_dns_dnsimple._internal.dns_dnsimple.""" -import os import unittest import mock from requests.exceptions import HTTPError +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins import dns_test_common_lexicon from certbot.tests import util as test_util @@ -19,7 +19,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, def setUp(self): super(AuthenticatorTest, self).setUp() - from certbot_dns_dnsimple.dns_dnsimple import Authenticator + from certbot_dns_dnsimple._internal.dns_dnsimple import Authenticator path = os.path.join(self.tempdir, 'file.ini') dns_test_common.write({"dnsimple_token": TOKEN}, path) @@ -39,7 +39,7 @@ class DNSimpleLexiconClientTest(unittest.TestCase, dns_test_common_lexicon.BaseL LOGIN_ERROR = HTTPError('401 Client Error: Unauthorized for url: ...') def setUp(self): - from certbot_dns_dnsimple.dns_dnsimple import _DNSimpleLexiconClient + from certbot_dns_dnsimple._internal.dns_dnsimple import _DNSimpleLexiconClient self.client = _DNSimpleLexiconClient(TOKEN, 0) diff --git a/certbot-dns-dnsmadeeasy/Dockerfile b/certbot-dns-dnsmadeeasy/Dockerfile deleted file mode 100644 index ff7936925..000000000 --- a/certbot-dns-dnsmadeeasy/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-dnsmadeeasy - -RUN pip install --no-cache-dir --editable src/certbot-dns-dnsmadeeasy diff --git a/certbot-dns-dnsmadeeasy/MANIFEST.in b/certbot-dns-dnsmadeeasy/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-dnsmadeeasy/MANIFEST.in +++ b/certbot-dns-dnsmadeeasy/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/_internal/__init__.py b/certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/_internal/__init__.py new file mode 100644 index 000000000..37350ce0b --- /dev/null +++ b/certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_dnsmadeeasy.dns_dnsmadeeasy` plugin.""" diff --git a/certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/dns_dnsmadeeasy.py b/certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/_internal/dns_dnsmadeeasy.py similarity index 99% rename from certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/dns_dnsmadeeasy.py rename to certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/_internal/dns_dnsmadeeasy.py index 4b63cb4b5..ed3146dce 100644 --- a/certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/dns_dnsmadeeasy.py +++ b/certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/_internal/dns_dnsmadeeasy.py @@ -1,8 +1,8 @@ """DNS Authenticator for DNS Made Easy DNS.""" import logging -import zope.interface from lexicon.providers import dnsmadeeasy +import zope.interface from certbot import errors from certbot import interfaces @@ -82,7 +82,7 @@ class _DNSMadeEasyLexiconClient(dns_common_lexicon.LexiconClient): def _handle_http_error(self, e, domain_name): if domain_name in str(e) and str(e).startswith('404 Client Error: Not Found for url:'): - return + return None hint = None if str(e).startswith('403 Client Error: Forbidden for url:'): diff --git a/certbot-dns-dnsmadeeasy/docs/api.rst b/certbot-dns-dnsmadeeasy/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-dnsmadeeasy/docs/api.rst +++ b/certbot-dns-dnsmadeeasy/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-dnsmadeeasy/docs/api/dns_dnsmadeeasy.rst b/certbot-dns-dnsmadeeasy/docs/api/dns_dnsmadeeasy.rst deleted file mode 100644 index 81948a77f..000000000 --- a/certbot-dns-dnsmadeeasy/docs/api/dns_dnsmadeeasy.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_dnsmadeeasy.dns_dnsmadeeasy` ------------------------------------- - -.. automodule:: certbot_dns_dnsmadeeasy.dns_dnsmadeeasy - :members: diff --git a/certbot-dns-dnsmadeeasy/docs/conf.py b/certbot-dns-dnsmadeeasy/docs/conf.py index 7d26f9742..1f0c57812 100644 --- a/certbot-dns-dnsmadeeasy/docs/conf.py +++ b/certbot-dns-dnsmadeeasy/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-dnsmadeeasy/local-oldest-requirements.txt b/certbot-dns-dnsmadeeasy/local-oldest-requirements.txt index 65f5a758e..1307698d4 100644 --- a/certbot-dns-dnsmadeeasy/local-oldest-requirements.txt +++ b/certbot-dns-dnsmadeeasy/local-oldest-requirements.txt @@ -1,2 +1,3 @@ --e acme[dev] --e .[dev] +# Remember to update setup.py to match the package versions below. +acme[dev]==0.31.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-dnsmadeeasy/readthedocs.org.requirements.txt b/certbot-dns-dnsmadeeasy/readthedocs.org.requirements.txt index 8f8c6c731..eb205d8f2 100644 --- a/certbot-dns-dnsmadeeasy/readthedocs.org.requirements.txt +++ b/certbot-dns-dnsmadeeasy/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-dnsmadeeasy[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-dnsmadeeasy[docs]" does not work as +# expected and "pip install -e certbot-dns-dnsmadeeasy[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-dnsmadeeasy[docs] diff --git a/certbot-dns-dnsmadeeasy/setup.py b/certbot-dns-dnsmadeeasy/setup.py index 0a99f452d..adaba6851 100644 --- a/certbot-dns-dnsmadeeasy/setup.py +++ b/certbot-dns-dnsmadeeasy/setup.py @@ -1,14 +1,16 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.31.0.dev0', - 'certbot>=0.31.0.dev0', + 'acme>=0.31.0', + 'certbot>=1.1.0', 'dns-lexicon>=2.2.1', # Support for >1 TXT record per name 'mock', 'setuptools', @@ -20,6 +22,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-dnsmadeeasy', version=version, @@ -28,9 +44,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -39,10 +55,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -59,8 +75,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-dnsmadeeasy = certbot_dns_dnsmadeeasy.dns_dnsmadeeasy:Authenticator', + 'dns-dnsmadeeasy = certbot_dns_dnsmadeeasy._internal.dns_dnsmadeeasy:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_dnsmadeeasy', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/dns_dnsmadeeasy_test.py b/certbot-dns-dnsmadeeasy/tests/dns_dnsmadeeasy_test.py similarity index 86% rename from certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/dns_dnsmadeeasy_test.py rename to certbot-dns-dnsmadeeasy/tests/dns_dnsmadeeasy_test.py index 44a777e1b..b94cc7d05 100644 --- a/certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy/dns_dnsmadeeasy_test.py +++ b/certbot-dns-dnsmadeeasy/tests/dns_dnsmadeeasy_test.py @@ -1,11 +1,11 @@ -"""Tests for certbot_dns_dnsmadeeasy.dns_dnsmadeeasy.""" +"""Tests for certbot_dns_dnsmadeeasy._internal.dns_dnsmadeeasy.""" -import os import unittest import mock from requests.exceptions import HTTPError +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins import dns_test_common_lexicon from certbot.plugins.dns_test_common import DOMAIN @@ -21,7 +21,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, def setUp(self): super(AuthenticatorTest, self).setUp() - from certbot_dns_dnsmadeeasy.dns_dnsmadeeasy import Authenticator + from certbot_dns_dnsmadeeasy._internal.dns_dnsmadeeasy import Authenticator path = os.path.join(self.tempdir, 'file.ini') dns_test_common.write({"dnsmadeeasy_api_key": API_KEY, @@ -44,7 +44,7 @@ class DNSMadeEasyLexiconClientTest(unittest.TestCase, LOGIN_ERROR = HTTPError('403 Client Error: Forbidden for url: {0}.'.format(DOMAIN)) def setUp(self): - from certbot_dns_dnsmadeeasy.dns_dnsmadeeasy import _DNSMadeEasyLexiconClient + from certbot_dns_dnsmadeeasy._internal.dns_dnsmadeeasy import _DNSMadeEasyLexiconClient self.client = _DNSMadeEasyLexiconClient(API_KEY, SECRET_KEY, 0) diff --git a/certbot-dns-gehirn/Dockerfile b/certbot-dns-gehirn/Dockerfile deleted file mode 100644 index 48ad902b5..000000000 --- a/certbot-dns-gehirn/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-gehirn - -RUN pip install --no-cache-dir --editable src/certbot-dns-gehirn diff --git a/certbot-dns-gehirn/MANIFEST.in b/certbot-dns-gehirn/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-gehirn/MANIFEST.in +++ b/certbot-dns-gehirn/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-gehirn/README.rst b/certbot-dns-gehirn/README.rst index 16058eff8..7a825bd7e 100644 --- a/certbot-dns-gehirn/README.rst +++ b/certbot-dns-gehirn/README.rst @@ -1 +1 @@ -Gehirn Infrastracture Service DNS Authenticator plugin for Certbot +Gehirn Infrastructure Service DNS Authenticator plugin for Certbot diff --git a/certbot-dns-gehirn/certbot_dns_gehirn/__init__.py b/certbot-dns-gehirn/certbot_dns_gehirn/__init__.py index db54154ac..fdcb8cd48 100644 --- a/certbot-dns-gehirn/certbot_dns_gehirn/__init__.py +++ b/certbot-dns-gehirn/certbot_dns_gehirn/__init__.py @@ -1,14 +1,14 @@ """ The `~certbot_dns_gehirn.dns_gehirn` plugin automates the process of completing a ``dns-01`` challenge (`~acme.challenges.DNS01`) by creating, and subsequently -removing, TXT records using the Gehirn Infrastracture Service DNS API. +removing, TXT records using the Gehirn Infrastructure Service DNS API. Named Arguments --------------- ======================================== ===================================== -``--dns-gehirn-credentials`` Gehirn Infrastracture Service +``--dns-gehirn-credentials`` Gehirn Infrastructure Service credentials_ INI file. (Required) ``--dns-gehirn-propagation-seconds`` The number of seconds to wait for DNS @@ -22,15 +22,15 @@ Credentials ----------- Use of this plugin requires a configuration file containing -Gehirn Infrastracture Service DNS API credentials, -obtained from your Gehirn Infrastracture Service +Gehirn Infrastructure Service DNS API credentials, +obtained from your Gehirn Infrastructure Service `dashboard `_. .. code-block:: ini :name: credentials.ini :caption: Example credentials file: - # Gehirn Infrastracture Service API credentials used by Certbot + # Gehirn Infrastructure Service API credentials used by Certbot dns_gehirn_api_token = 00000000-0000-0000-0000-000000000000 dns_gehirn_api_secret = MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw @@ -40,7 +40,7 @@ to this file for use during renewal, but does not store the file's contents. .. caution:: You should protect these API credentials as you would the password to your - Gehirn Infrastracture Service account. Users who can read this file can use + Gehirn Infrastructure Service account. Users who can read this file can use these credentials to issue arbitrary API calls on your behalf. Users who can cause Certbot to run using these credentials can complete a ``dns-01`` challenge to acquire new certificates or revoke existing certificates for diff --git a/certbot-dns-gehirn/certbot_dns_gehirn/_internal/__init__.py b/certbot-dns-gehirn/certbot_dns_gehirn/_internal/__init__.py new file mode 100644 index 000000000..f8d6485dc --- /dev/null +++ b/certbot-dns-gehirn/certbot_dns_gehirn/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_gehirn.dns_gehirn` plugin.""" diff --git a/certbot-dns-gehirn/certbot_dns_gehirn/dns_gehirn.py b/certbot-dns-gehirn/certbot_dns_gehirn/_internal/dns_gehirn.py similarity index 78% rename from certbot-dns-gehirn/certbot_dns_gehirn/dns_gehirn.py rename to certbot-dns-gehirn/certbot_dns_gehirn/_internal/dns_gehirn.py index edf530072..76c0ed584 100644 --- a/certbot-dns-gehirn/certbot_dns_gehirn/dns_gehirn.py +++ b/certbot-dns-gehirn/certbot_dns_gehirn/_internal/dns_gehirn.py @@ -1,8 +1,8 @@ -"""DNS Authenticator for Gehirn Infrastracture Service DNS.""" +"""DNS Authenticator for Gehirn Infrastructure Service DNS.""" import logging -import zope.interface from lexicon.providers import gehirn +import zope.interface from certbot import interfaces from certbot.plugins import dns_common @@ -15,14 +15,14 @@ DASHBOARD_URL = "https://gis.gehirn.jp/" @zope.interface.implementer(interfaces.IAuthenticator) @zope.interface.provider(interfaces.IPluginFactory) class Authenticator(dns_common.DNSAuthenticator): - """DNS Authenticator for Gehirn Infrastracture Service DNS + """DNS Authenticator for Gehirn Infrastructure Service DNS - This Authenticator uses the Gehirn Infrastracture Service API to fulfill + This Authenticator uses the Gehirn Infrastructure Service API to fulfill a dns-01 challenge. """ description = 'Obtain certificates using a DNS TXT record ' + \ - '(if you are using Gehirn Infrastracture Service for DNS).' + '(if you are using Gehirn Infrastructure Service for DNS).' ttl = 60 def __init__(self, *args, **kwargs): @@ -32,20 +32,20 @@ class Authenticator(dns_common.DNSAuthenticator): @classmethod def add_parser_arguments(cls, add): # pylint: disable=arguments-differ super(Authenticator, cls).add_parser_arguments(add, default_propagation_seconds=30) - add('credentials', help='Gehirn Infrastracture Service credentials file.') + add('credentials', help='Gehirn Infrastructure Service credentials file.') def more_info(self): # pylint: disable=missing-docstring,no-self-use return 'This plugin configures a DNS TXT record to respond to a dns-01 challenge using ' + \ - 'the Gehirn Infrastracture Service API.' + 'the Gehirn Infrastructure Service API.' def _setup_credentials(self): self.credentials = self._configure_credentials( 'credentials', - 'Gehirn Infrastracture Service credentials file', + 'Gehirn Infrastructure Service credentials file', { - 'api-token': 'API token for Gehirn Infrastracture Service ' + \ + 'api-token': 'API token for Gehirn Infrastructure Service ' + \ 'API obtained from {0}'.format(DASHBOARD_URL), - 'api-secret': 'API secret for Gehirn Infrastracture Service ' + \ + 'api-secret': 'API secret for Gehirn Infrastructure Service ' + \ 'API obtained from {0}'.format(DASHBOARD_URL), } ) @@ -66,7 +66,7 @@ class Authenticator(dns_common.DNSAuthenticator): class _GehirnLexiconClient(dns_common_lexicon.LexiconClient): """ - Encapsulates all communication with the Gehirn Infrastracture Service via Lexicon. + Encapsulates all communication with the Gehirn Infrastructure Service via Lexicon. """ def __init__(self, api_token, api_secret, ttl): @@ -83,5 +83,5 @@ class _GehirnLexiconClient(dns_common_lexicon.LexiconClient): def _handle_http_error(self, e, domain_name): if domain_name in str(e) and (str(e).startswith('404 Client Error: Not Found for url:')): - return # Expected errors when zone name guess is wrong + return None # Expected errors when zone name guess is wrong return super(_GehirnLexiconClient, self)._handle_http_error(e, domain_name) diff --git a/certbot-dns-gehirn/docs/api.rst b/certbot-dns-gehirn/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-gehirn/docs/api.rst +++ b/certbot-dns-gehirn/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-gehirn/docs/api/dns_gehirn.rst b/certbot-dns-gehirn/docs/api/dns_gehirn.rst deleted file mode 100644 index 35a13e9c1..000000000 --- a/certbot-dns-gehirn/docs/api/dns_gehirn.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_gehirn.dns_gehirn` ------------------------------------- - -.. automodule:: certbot_dns_gehirn.dns_gehirn - :members: diff --git a/certbot-dns-gehirn/docs/conf.py b/certbot-dns-gehirn/docs/conf.py index a1b2799fb..527bc3d55 100644 --- a/certbot-dns-gehirn/docs/conf.py +++ b/certbot-dns-gehirn/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-gehirn/local-oldest-requirements.txt b/certbot-dns-gehirn/local-oldest-requirements.txt index 65f5a758e..1307698d4 100644 --- a/certbot-dns-gehirn/local-oldest-requirements.txt +++ b/certbot-dns-gehirn/local-oldest-requirements.txt @@ -1,2 +1,3 @@ --e acme[dev] --e .[dev] +# Remember to update setup.py to match the package versions below. +acme[dev]==0.31.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-gehirn/readthedocs.org.requirements.txt b/certbot-dns-gehirn/readthedocs.org.requirements.txt index d9f4f9823..97af343d9 100644 --- a/certbot-dns-gehirn/readthedocs.org.requirements.txt +++ b/certbot-dns-gehirn/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-gehirn[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-gehirn[docs]" does not work as +# expected and "pip install -e certbot-dns-gehirn[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-gehirn[docs] diff --git a/certbot-dns-gehirn/setup.py b/certbot-dns-gehirn/setup.py index f4a75379c..a849cef45 100644 --- a/certbot-dns-gehirn/setup.py +++ b/certbot-dns-gehirn/setup.py @@ -1,13 +1,15 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Please update tox.ini when modifying dependency version requirements install_requires = [ - 'acme>=0.31.0.dev0', - 'certbot>=0.31.0.dev0', + 'acme>=0.31.0', + 'certbot>=1.1.0', 'dns-lexicon>=2.1.22', 'mock', 'setuptools', @@ -19,17 +21,31 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-gehirn', version=version, - description="Gehirn Infrastracture Service DNS Authenticator plugin for Certbot", + description="Gehirn Infrastructure Service DNS Authenticator plugin for Certbot", url='https://github.com/certbot/certbot', author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -38,9 +54,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', + 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -57,8 +74,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-gehirn = certbot_dns_gehirn.dns_gehirn:Authenticator', + 'dns-gehirn = certbot_dns_gehirn._internal.dns_gehirn:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_gehirn', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-gehirn/certbot_dns_gehirn/dns_gehirn_test.py b/certbot-dns-gehirn/tests/dns_gehirn_test.py similarity index 87% rename from certbot-dns-gehirn/certbot_dns_gehirn/dns_gehirn_test.py rename to certbot-dns-gehirn/tests/dns_gehirn_test.py index b771c103e..f5b95b6c3 100644 --- a/certbot-dns-gehirn/certbot_dns_gehirn/dns_gehirn_test.py +++ b/certbot-dns-gehirn/tests/dns_gehirn_test.py @@ -1,11 +1,11 @@ -"""Tests for certbot_dns_gehirn.dns_gehirn.""" +"""Tests for certbot_dns_gehirn._internal.dns_gehirn.""" -import os import unittest import mock from requests.exceptions import HTTPError +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins import dns_test_common_lexicon from certbot.plugins.dns_test_common import DOMAIN @@ -20,7 +20,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, def setUp(self): super(AuthenticatorTest, self).setUp() - from certbot_dns_gehirn.dns_gehirn import Authenticator + from certbot_dns_gehirn._internal.dns_gehirn import Authenticator path = os.path.join(self.tempdir, 'file.ini') dns_test_common.write( @@ -43,7 +43,7 @@ class GehirnLexiconClientTest(unittest.TestCase, dns_test_common_lexicon.BaseLex LOGIN_ERROR = HTTPError('401 Client Error: Unauthorized for url: {0}.'.format(DOMAIN)) def setUp(self): - from certbot_dns_gehirn.dns_gehirn import _GehirnLexiconClient + from certbot_dns_gehirn._internal.dns_gehirn import _GehirnLexiconClient self.client = _GehirnLexiconClient(API_TOKEN, API_SECRET, 0) diff --git a/certbot-dns-google/Dockerfile b/certbot-dns-google/Dockerfile deleted file mode 100644 index 4a258d0ee..000000000 --- a/certbot-dns-google/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-google - -RUN pip install --no-cache-dir --editable src/certbot-dns-google diff --git a/certbot-dns-google/MANIFEST.in b/certbot-dns-google/MANIFEST.in index c91330e38..a7301ee7f 100644 --- a/certbot-dns-google/MANIFEST.in +++ b/certbot-dns-google/MANIFEST.in @@ -2,3 +2,6 @@ include LICENSE.txt include README.rst recursive-include docs * recursive-include certbot_dns_google/testdata * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-google/certbot_dns_google/_internal/__init__.py b/certbot-dns-google/certbot_dns_google/_internal/__init__.py new file mode 100644 index 000000000..f433213ff --- /dev/null +++ b/certbot-dns-google/certbot_dns_google/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_google.dns_google` plugin.""" diff --git a/certbot-dns-google/certbot_dns_google/dns_google.py b/certbot-dns-google/certbot_dns_google/_internal/dns_google.py similarity index 97% rename from certbot-dns-google/certbot_dns_google/dns_google.py rename to certbot-dns-google/certbot_dns_google/_internal/dns_google.py index 0b84dddb0..3aa910b52 100644 --- a/certbot-dns-google/certbot_dns_google/dns_google.py +++ b/certbot-dns-google/certbot_dns_google/_internal/dns_google.py @@ -2,11 +2,11 @@ import json import logging -import httplib2 -import zope.interface from googleapiclient import discovery from googleapiclient import errors as googleapiclient_errors +import httplib2 from oauth2client.service_account import ServiceAccountCredentials +import zope.interface from certbot import errors from certbot import interfaces @@ -235,7 +235,7 @@ class _GoogleClient(object): :rtype: `list` of `string` or `None` """ - rrs_request = self.dns.resourceRecordSets() # pylint: disable=no-member + rrs_request = self.dns.resourceRecordSets() request = rrs_request.list(managedZone=zone_id, project=self.project_id) # Add dot as the API returns absolute domains record_name += "." @@ -274,10 +274,11 @@ class _GoogleClient(object): raise errors.PluginError('Encountered error finding managed zone: {0}' .format(e)) - if len(zones) > 0: - zone_id = zones[0]['id'] - logger.debug('Found id of %s for %s using name %s', zone_id, domain, zone_name) - return zone_id + for zone in zones: + zone_id = zone['id'] + if 'privateVisibilityConfig' not in zone: + logger.debug('Found id of %s for %s using name %s', zone_id, domain, zone_name) + return zone_id raise errors.PluginError('Unable to determine managed zone for {0} using zone names: {1}.' .format(domain, zone_dns_name_guesses)) @@ -303,5 +304,4 @@ class _GoogleClient(object): if isinstance(content, bytes): return content.decode() - else: - return content + return content diff --git a/certbot-dns-google/docs/api.rst b/certbot-dns-google/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-google/docs/api.rst +++ b/certbot-dns-google/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-google/docs/api/dns_google.rst b/certbot-dns-google/docs/api/dns_google.rst deleted file mode 100644 index 6f5459934..000000000 --- a/certbot-dns-google/docs/api/dns_google.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_google.dns_google` ------------------------------------- - -.. automodule:: certbot_dns_google.dns_google - :members: diff --git a/certbot-dns-google/docs/conf.py b/certbot-dns-google/docs/conf.py index bbb343ee8..b2ddcfb34 100644 --- a/certbot-dns-google/docs/conf.py +++ b/certbot-dns-google/docs/conf.py @@ -18,6 +18,7 @@ # import os import sys + sys.path.insert(0, os.path.abspath('_ext')) @@ -38,7 +39,7 @@ extensions = ['sphinx.ext.autodoc', 'jsonlexer'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-google/local-oldest-requirements.txt b/certbot-dns-google/local-oldest-requirements.txt index 8368d266e..cf61c15a5 100644 --- a/certbot-dns-google/local-oldest-requirements.txt +++ b/certbot-dns-google/local-oldest-requirements.txt @@ -1,2 +1,3 @@ -acme[dev]==0.21.1 -certbot[dev]==0.21.1 +# Remember to update setup.py to match the package versions below. +acme[dev]==0.29.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-google/readthedocs.org.requirements.txt b/certbot-dns-google/readthedocs.org.requirements.txt index 6ea393f86..fe97cee94 100644 --- a/certbot-dns-google/readthedocs.org.requirements.txt +++ b/certbot-dns-google/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-google[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-google[docs]" does not work as +# expected and "pip install -e certbot-dns-google[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-google[docs] diff --git a/certbot-dns-google/setup.py b/certbot-dns-google/setup.py index c99ad38aa..51d5b8a3f 100644 --- a/certbot-dns-google/setup.py +++ b/certbot-dns-google/setup.py @@ -1,19 +1,19 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.21.1', - 'certbot>=0.21.1', - # 1.5 is the first version that supports oauth2client>=2.0 - 'google-api-python-client>=1.5', + 'acme>=0.29.0', + 'certbot>=1.1.0', + 'google-api-python-client>=1.5.5', 'mock', - # for oauth2client.service_account.ServiceAccountCredentials - 'oauth2client>=2.0', + 'oauth2client>=4.0', 'setuptools', 'zope.interface', # already a dependency of google-api-python-client, but added for consistency @@ -25,6 +25,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-google', version=version, @@ -33,9 +47,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -44,10 +58,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -64,8 +78,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-google = certbot_dns_google.dns_google:Authenticator', + 'dns-google = certbot_dns_google._internal.dns_google:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_google', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-google/certbot_dns_google/dns_google_test.py b/certbot-dns-google/tests/dns_google_test.py similarity index 89% rename from certbot-dns-google/certbot_dns_google/dns_google_test.py rename to certbot-dns-google/tests/dns_google_test.py index 2b081885b..647a75b05 100644 --- a/certbot-dns-google/certbot_dns_google/dns_google_test.py +++ b/certbot-dns-google/tests/dns_google_test.py @@ -1,15 +1,15 @@ -"""Tests for certbot_dns_google.dns_google.""" +"""Tests for certbot_dns_google._internal.dns_google.""" -import os import unittest -import mock from googleapiclient import discovery from googleapiclient.errors import Error from googleapiclient.http import HttpMock from httplib2 import ServerNotFoundError +import mock from certbot import errors +from certbot.compat import os from certbot.errors import PluginError from certbot.plugins import dns_test_common from certbot.plugins.dns_test_common import DOMAIN @@ -25,7 +25,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthentic def setUp(self): super(AuthenticatorTest, self).setUp() - from certbot_dns_google.dns_google import Authenticator + from certbot_dns_google._internal.dns_google import Authenticator path = os.path.join(self.tempdir, 'file.json') open(path, "wb").close() @@ -68,7 +68,7 @@ class GoogleClientTest(unittest.TestCase): change = "an-id" def _setUp_client_with_mock(self, zone_request_side_effect): - from certbot_dns_google.dns_google import _GoogleClient + from certbot_dns_google._internal.dns_google import _GoogleClient pwd = os.path.dirname(__file__) rel_path = 'testdata/discovery.json' @@ -96,18 +96,18 @@ class GoogleClientTest(unittest.TestCase): @mock.patch('googleapiclient.discovery.build') @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google._GoogleClient.get_project_id') + @mock.patch('certbot_dns_google._internal.dns_google._GoogleClient.get_project_id') def test_client_without_credentials(self, get_project_id_mock, credential_mock, unused_discovery_mock): - from certbot_dns_google.dns_google import _GoogleClient + from certbot_dns_google._internal.dns_google import _GoogleClient _GoogleClient(None) self.assertFalse(credential_mock.called) self.assertTrue(get_project_id_mock.called) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) - @mock.patch('certbot_dns_google.dns_google._GoogleClient.get_project_id') + @mock.patch('certbot_dns_google._internal.dns_google._GoogleClient.get_project_id') def test_add_txt_record(self, get_project_id_mock, credential_mock): client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}]) credential_mock.assert_called_once_with('/not/a/real/path.json', mock.ANY) @@ -133,7 +133,7 @@ class GoogleClientTest(unittest.TestCase): project=PROJECT_ID) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_add_txt_record_and_poll(self, unused_credential_mock): client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}]) @@ -151,12 +151,13 @@ class GoogleClientTest(unittest.TestCase): project=PROJECT_ID) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_add_txt_record_delete_old(self, unused_credential_mock): client, changes = self._setUp_client_with_mock( [{'managedZones': [{'id': self.zone}]}]) - mock_get_rrs = "certbot_dns_google.dns_google._GoogleClient.get_existing_txt_rrset" + # pylint: disable=line-too-long + mock_get_rrs = "certbot_dns_google._internal.dns_google._GoogleClient.get_existing_txt_rrset" with mock.patch(mock_get_rrs) as mock_rrs: mock_rrs.return_value = ["sample-txt-contents"] client.add_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl) @@ -165,7 +166,7 @@ class GoogleClientTest(unittest.TestCase): changes.create.call_args_list[0][1]["body"]["deletions"][0]["rrdatas"]) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_add_txt_record_noop(self, unused_credential_mock): client, changes = self._setUp_client_with_mock( @@ -175,7 +176,7 @@ class GoogleClientTest(unittest.TestCase): self.assertFalse(changes.create.called) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_add_txt_record_error_during_zone_lookup(self, unused_credential_mock): client, unused_changes = self._setUp_client_with_mock(API_ERROR) @@ -184,7 +185,7 @@ class GoogleClientTest(unittest.TestCase): DOMAIN, self.record_name, self.record_content, self.record_ttl) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_add_txt_record_zone_not_found(self, unused_credential_mock): client, unused_changes = self._setUp_client_with_mock([{'managedZones': []}, @@ -194,7 +195,7 @@ class GoogleClientTest(unittest.TestCase): DOMAIN, self.record_name, self.record_content, self.record_ttl) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_add_txt_record_error_during_add(self, unused_credential_mock): client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}]) @@ -204,12 +205,13 @@ class GoogleClientTest(unittest.TestCase): DOMAIN, self.record_name, self.record_content, self.record_ttl) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_del_txt_record(self, unused_credential_mock): client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}]) - mock_get_rrs = "certbot_dns_google.dns_google._GoogleClient.get_existing_txt_rrset" + # pylint: disable=line-too-long + mock_get_rrs = "certbot_dns_google._internal.dns_google._GoogleClient.get_existing_txt_rrset" with mock.patch(mock_get_rrs) as mock_rrs: mock_rrs.return_value = ["\"sample-txt-contents\"", "\"example-txt-contents\""] @@ -243,7 +245,7 @@ class GoogleClientTest(unittest.TestCase): project=PROJECT_ID) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_del_txt_record_error_during_zone_lookup(self, unused_credential_mock): client, unused_changes = self._setUp_client_with_mock(API_ERROR) @@ -251,7 +253,7 @@ class GoogleClientTest(unittest.TestCase): client.del_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_del_txt_record_zone_not_found(self, unused_credential_mock): client, unused_changes = self._setUp_client_with_mock([{'managedZones': []}, @@ -260,7 +262,7 @@ class GoogleClientTest(unittest.TestCase): client.del_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_del_txt_record_error_during_delete(self, unused_credential_mock): client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}]) @@ -269,7 +271,7 @@ class GoogleClientTest(unittest.TestCase): client.del_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_get_existing(self, unused_credential_mock): client, unused_changes = self._setUp_client_with_mock( @@ -281,12 +283,11 @@ class GoogleClientTest(unittest.TestCase): self.assertEqual(not_found, None) @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name') - @mock.patch('certbot_dns_google.dns_google.open', + @mock.patch('certbot_dns_google._internal.dns_google.open', mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True) def test_get_existing_fallback(self, unused_credential_mock): client, unused_changes = self._setUp_client_with_mock( [{'managedZones': [{'id': self.zone}]}]) - # pylint: disable=no-member mock_execute = client.dns.resourceRecordSets.return_value.list.return_value.execute mock_execute.side_effect = API_ERROR @@ -294,7 +295,7 @@ class GoogleClientTest(unittest.TestCase): self.assertFalse(rrset) def test_get_project_id(self): - from certbot_dns_google.dns_google import _GoogleClient + from certbot_dns_google._internal.dns_google import _GoogleClient response = DummyResponse() response.status = 200 diff --git a/certbot-dns-google/certbot_dns_google/testdata/discovery.json b/certbot-dns-google/tests/testdata/discovery.json similarity index 100% rename from certbot-dns-google/certbot_dns_google/testdata/discovery.json rename to certbot-dns-google/tests/testdata/discovery.json diff --git a/certbot-dns-linode/Dockerfile b/certbot-dns-linode/Dockerfile deleted file mode 100644 index 2e237b521..000000000 --- a/certbot-dns-linode/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-linode - -RUN pip install --no-cache-dir --editable src/certbot-dns-linode diff --git a/certbot-dns-linode/MANIFEST.in b/certbot-dns-linode/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-linode/MANIFEST.in +++ b/certbot-dns-linode/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-linode/certbot_dns_linode/__init__.py b/certbot-dns-linode/certbot_dns_linode/__init__.py index 0a6ccec61..107781a13 100644 --- a/certbot-dns-linode/certbot_dns_linode/__init__.py +++ b/certbot-dns-linode/certbot_dns_linode/__init__.py @@ -27,7 +27,8 @@ Credentials Use of this plugin requires a configuration file containing Linode API credentials, obtained from your Linode account's `Applications & API -Tokens page `_. +Tokens page (legacy) `_ or `Applications +& API Tokens page (new) `_. .. code-block:: ini :name: credentials.ini @@ -35,6 +36,7 @@ Tokens page `_. # Linode API credentials used by Certbot dns_linode_key = 0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ64 + dns_linode_version = [|3|4] The path to this file can be provided interactively or using the ``--dns-linode-credentials`` command-line argument. Certbot records the path diff --git a/certbot-dns-linode/certbot_dns_linode/_internal/__init__.py b/certbot-dns-linode/certbot_dns_linode/_internal/__init__.py new file mode 100644 index 000000000..9090d92d3 --- /dev/null +++ b/certbot-dns-linode/certbot_dns_linode/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_linode.dns_linode` plugin.""" diff --git a/certbot-dns-linode/certbot_dns_linode/dns_linode.py b/certbot-dns-linode/certbot_dns_linode/_internal/dns_linode.py similarity index 62% rename from certbot-dns-linode/certbot_dns_linode/dns_linode.py rename to certbot-dns-linode/certbot_dns_linode/_internal/dns_linode.py index 4e0500fa0..ea6046849 100644 --- a/certbot-dns-linode/certbot_dns_linode/dns_linode.py +++ b/certbot-dns-linode/certbot_dns_linode/_internal/dns_linode.py @@ -1,8 +1,10 @@ """DNS Authenticator for Linode.""" import logging +import re -import zope.interface from lexicon.providers import linode +from lexicon.providers import linode4 +import zope.interface from certbot import errors from certbot import interfaces @@ -12,6 +14,7 @@ from certbot.plugins import dns_common_lexicon logger = logging.getLogger(__name__) API_KEY_URL = 'https://manager.linode.com/profile/api' +API_KEY_URL_V4 = 'https://cloud.linode.com/profile/tokens' @zope.interface.implementer(interfaces.IAuthenticator) @zope.interface.provider(interfaces.IPluginFactory) @@ -41,7 +44,8 @@ class Authenticator(dns_common.DNSAuthenticator): 'credentials', 'Linode credentials INI file', { - 'key': 'API key for Linode account, obtained from {0}'.format(API_KEY_URL) + 'key': 'API key for Linode account, obtained from {0} or {1}' + .format(API_KEY_URL, API_KEY_URL_V4) } ) @@ -52,7 +56,23 @@ class Authenticator(dns_common.DNSAuthenticator): self._get_linode_client().del_txt_record(domain, validation_name, validation) def _get_linode_client(self): - return _LinodeLexiconClient(self.credentials.conf('key')) + api_key = self.credentials.conf('key') + api_version = self.credentials.conf('version') + if api_version == '': + api_version = None + + if not api_version: + api_version = 3 + + # Match for v4 api key + regex_v4 = re.compile('^[0-9a-f]{64}$') + regex_match = regex_v4.match(api_key) + if regex_match: + api_version = 4 + else: + api_version = int(api_version) + + return _LinodeLexiconClient(api_key, api_version) class _LinodeLexiconClient(dns_common_lexicon.LexiconClient): @@ -60,17 +80,29 @@ class _LinodeLexiconClient(dns_common_lexicon.LexiconClient): Encapsulates all communication with the Linode API. """ - def __init__(self, api_key): + def __init__(self, api_key, api_version): super(_LinodeLexiconClient, self).__init__() - config = dns_common_lexicon.build_lexicon_config('linode', {}, { - 'auth_token': api_key, - }) + self.api_version = api_version - self.provider = linode.Provider(config) + if api_version == 3: + config = dns_common_lexicon.build_lexicon_config('linode', {}, { + 'auth_token': api_key, + }) + + self.provider = linode.Provider(config) + elif api_version == 4: + config = dns_common_lexicon.build_lexicon_config('linode4', {}, { + 'auth_token': api_key, + }) + + self.provider = linode4.Provider(config) + else: + raise errors.PluginError('Invalid api version specified: {0}. (Supported: 3, 4)' + .format(api_version)) def _handle_general_error(self, e, domain_name): if not str(e).startswith('Domain not found'): return errors.PluginError('Unexpected error determining zone identifier for {0}: {1}' .format(domain_name, e)) - + return None diff --git a/certbot-dns-linode/certbot_dns_linode/dns_linode_test.py b/certbot-dns-linode/certbot_dns_linode/dns_linode_test.py deleted file mode 100644 index 2a0ee49f7..000000000 --- a/certbot-dns-linode/certbot_dns_linode/dns_linode_test.py +++ /dev/null @@ -1,47 +0,0 @@ -"""Tests for certbot_dns_linode.dns_linode.""" - -import os -import unittest - -import mock - -from certbot.plugins import dns_test_common -from certbot.plugins import dns_test_common_lexicon -from certbot.tests import util as test_util - -TOKEN = 'a-token' - -class AuthenticatorTest(test_util.TempDirTestCase, - dns_test_common_lexicon.BaseLexiconAuthenticatorTest): - - def setUp(self): - super(AuthenticatorTest, self).setUp() - - from certbot_dns_linode.dns_linode import Authenticator - - path = os.path.join(self.tempdir, 'file.ini') - dns_test_common.write({"linode_key": TOKEN}, path) - - self.config = mock.MagicMock(linode_credentials=path, - linode_propagation_seconds=0) # don't wait during tests - - self.auth = Authenticator(self.config, "linode") - - self.mock_client = mock.MagicMock() - # _get_linode_client | pylint: disable=protected-access - self.auth._get_linode_client = mock.MagicMock(return_value=self.mock_client) - -class LinodeLexiconClientTest(unittest.TestCase, dns_test_common_lexicon.BaseLexiconClientTest): - - DOMAIN_NOT_FOUND = Exception('Domain not found') - - def setUp(self): - from certbot_dns_linode.dns_linode import _LinodeLexiconClient - - self.client = _LinodeLexiconClient(TOKEN) - - self.provider_mock = mock.MagicMock() - self.client.provider = self.provider_mock - -if __name__ == "__main__": - unittest.main() # pragma: no cover diff --git a/certbot-dns-linode/docs/api.rst b/certbot-dns-linode/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-linode/docs/api.rst +++ b/certbot-dns-linode/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-linode/docs/api/dns_linode.rst b/certbot-dns-linode/docs/api/dns_linode.rst deleted file mode 100644 index 6380b3eba..000000000 --- a/certbot-dns-linode/docs/api/dns_linode.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_linode.dns_linode` ------------------------------------------------- - -.. automodule:: certbot_dns_linode.dns_linode - :members: diff --git a/certbot-dns-linode/docs/conf.py b/certbot-dns-linode/docs/conf.py index 1fb721400..c6d564b7a 100644 --- a/certbot-dns-linode/docs/conf.py +++ b/certbot-dns-linode/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-linode/local-oldest-requirements.txt b/certbot-dns-linode/local-oldest-requirements.txt index 65f5a758e..a8bd7449a 100644 --- a/certbot-dns-linode/local-oldest-requirements.txt +++ b/certbot-dns-linode/local-oldest-requirements.txt @@ -1,2 +1,4 @@ --e acme[dev] --e .[dev] +# Remember to update setup.py to match the package versions below. +acme[dev]==0.31.0 +certbot[dev]==1.1.0 +dns-lexicon==2.2.3 diff --git a/certbot-dns-linode/readthedocs.org.requirements.txt b/certbot-dns-linode/readthedocs.org.requirements.txt index 47449454f..3d28f43bf 100644 --- a/certbot-dns-linode/readthedocs.org.requirements.txt +++ b/certbot-dns-linode/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-linode[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-linode[docs]" does not work as +# expected and "pip install -e certbot-dns-linode[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-linode[docs] diff --git a/certbot-dns-linode/setup.py b/certbot-dns-linode/setup.py index 31c2c20bc..e7e91b929 100644 --- a/certbot-dns-linode/setup.py +++ b/certbot-dns-linode/setup.py @@ -1,13 +1,16 @@ -from setuptools import setup -from setuptools import find_packages +import sys -version = '0.31.0.dev0' +from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand + +version = '1.3.0.dev0' # Please update tox.ini when modifying dependency version requirements install_requires = [ - 'acme>=0.31.0.dev0', - 'certbot>=0.31.0.dev0', - 'dns-lexicon>=2.2.1', + 'acme>=0.31.0', + 'certbot>=1.1.0', + 'dns-lexicon>=2.2.3', 'mock', 'setuptools', 'zope.interface', @@ -18,6 +21,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-linode', version=version, @@ -26,9 +43,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -37,10 +54,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -57,8 +74,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-linode = certbot_dns_linode.dns_linode:Authenticator', + 'dns-linode = certbot_dns_linode._internal.dns_linode:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_linode', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-linode/tests/dns_linode_test.py b/certbot-dns-linode/tests/dns_linode_test.py new file mode 100644 index 000000000..3cf615486 --- /dev/null +++ b/certbot-dns-linode/tests/dns_linode_test.py @@ -0,0 +1,144 @@ +"""Tests for certbot_dns_linode._internal.dns_linode.""" + +import unittest + +import mock + +from certbot import errors +from certbot.compat import os +from certbot.plugins import dns_test_common +from certbot.plugins import dns_test_common_lexicon +from certbot.tests import util as test_util +from certbot_dns_linode._internal.dns_linode import Authenticator + +TOKEN = 'a-token' +TOKEN_V3 = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ64' +TOKEN_V4 = '0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + +class AuthenticatorTest(test_util.TempDirTestCase, + dns_test_common_lexicon.BaseLexiconAuthenticatorTest): + + def setUp(self): + super(AuthenticatorTest, self).setUp() + + path = os.path.join(self.tempdir, 'file.ini') + dns_test_common.write({"linode_key": TOKEN}, path) + + self.config = mock.MagicMock(linode_credentials=path, + linode_propagation_seconds=0) # don't wait during tests + + self.auth = Authenticator(self.config, "linode") + + self.mock_client = mock.MagicMock() + # _get_linode_client | pylint: disable=protected-access + self.auth._get_linode_client = mock.MagicMock(return_value=self.mock_client) + + # pylint: disable=protected-access + def test_api_version_3_detection(self): + path = os.path.join(self.tempdir, 'file_3_auto.ini') + dns_test_common.write({"linode_key": TOKEN_V3}, path) + + config = mock.MagicMock(linode_credentials=path, + linode_propagation_seconds=0) + auth = Authenticator(config, "linode") + auth._setup_credentials() + client = auth._get_linode_client() + self.assertEqual(3, client.api_version) + + # pylint: disable=protected-access + def test_api_version_4_detection(self): + path = os.path.join(self.tempdir, 'file_4_auto.ini') + dns_test_common.write({"linode_key": TOKEN_V4}, path) + + config = mock.MagicMock(linode_credentials=path, + linode_propagation_seconds=0) + auth = Authenticator(config, "linode") + auth._setup_credentials() + client = auth._get_linode_client() + self.assertEqual(4, client.api_version) + + # pylint: disable=protected-access + def test_api_version_3_detection_empty_version(self): + path = os.path.join(self.tempdir, 'file_3_auto_empty.ini') + dns_test_common.write({"linode_key": TOKEN_V3, "linode_version": ""}, path) + + config = mock.MagicMock(linode_credentials=path, + linode_propagation_seconds=0) + auth = Authenticator(config, "linode") + auth._setup_credentials() + client = auth._get_linode_client() + self.assertEqual(3, client.api_version) + + # pylint: disable=protected-access + def test_api_version_4_detection_empty_version(self): + path = os.path.join(self.tempdir, 'file_4_auto_empty.ini') + dns_test_common.write({"linode_key": TOKEN_V4, "linode_version": ""}, path) + + config = mock.MagicMock(linode_credentials=path, + linode_propagation_seconds=0) + auth = Authenticator(config, "linode") + auth._setup_credentials() + client = auth._get_linode_client() + self.assertEqual(4, client.api_version) + + # pylint: disable=protected-access + def test_api_version_3_manual(self): + path = os.path.join(self.tempdir, 'file_3_manual.ini') + dns_test_common.write({"linode_key": TOKEN_V4, "linode_version": 3}, path) + + config = mock.MagicMock(linode_credentials=path, + linode_propagation_seconds=0) + auth = Authenticator(config, "linode") + auth._setup_credentials() + client = auth._get_linode_client() + self.assertEqual(3, client.api_version) + + # pylint: disable=protected-access + def test_api_version_4_manual(self): + path = os.path.join(self.tempdir, 'file_4_manual.ini') + dns_test_common.write({"linode_key": TOKEN_V3, "linode_version": 4}, path) + + config = mock.MagicMock(linode_credentials=path, + linode_propagation_seconds=0) + auth = Authenticator(config, "linode") + auth._setup_credentials() + client = auth._get_linode_client() + self.assertEqual(4, client.api_version) + + # pylint: disable=protected-access + def test_api_version_error(self): + path = os.path.join(self.tempdir, 'file_version_error.ini') + dns_test_common.write({"linode_key": TOKEN_V3, "linode_version": 5}, path) + + config = mock.MagicMock(linode_credentials=path, + linode_propagation_seconds=0) + auth = Authenticator(config, "linode") + auth._setup_credentials() + self.assertRaises(errors.PluginError, auth._get_linode_client) + +class LinodeLexiconClientTest(unittest.TestCase, dns_test_common_lexicon.BaseLexiconClientTest): + + DOMAIN_NOT_FOUND = Exception('Domain not found') + + def setUp(self): + from certbot_dns_linode._internal.dns_linode import _LinodeLexiconClient + + self.client = _LinodeLexiconClient(TOKEN, 3) + + self.provider_mock = mock.MagicMock() + self.client.provider = self.provider_mock + +class Linode4LexiconClientTest(unittest.TestCase, dns_test_common_lexicon.BaseLexiconClientTest): + + DOMAIN_NOT_FOUND = Exception('Domain not found') + + def setUp(self): + from certbot_dns_linode._internal.dns_linode import _LinodeLexiconClient + + self.client = _LinodeLexiconClient(TOKEN, 4) + + self.provider_mock = mock.MagicMock() + self.client.provider = self.provider_mock + +if __name__ == "__main__": + unittest.main() # pragma: no cover diff --git a/certbot-dns-luadns/Dockerfile b/certbot-dns-luadns/Dockerfile deleted file mode 100644 index 6efb4d777..000000000 --- a/certbot-dns-luadns/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-luadns - -RUN pip install --no-cache-dir --editable src/certbot-dns-luadns diff --git a/certbot-dns-luadns/MANIFEST.in b/certbot-dns-luadns/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-luadns/MANIFEST.in +++ b/certbot-dns-luadns/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-luadns/certbot_dns_luadns/_internal/__init__.py b/certbot-dns-luadns/certbot_dns_luadns/_internal/__init__.py new file mode 100644 index 000000000..8ab0a00e2 --- /dev/null +++ b/certbot-dns-luadns/certbot_dns_luadns/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_luadns.dns_luadns` plugin.""" diff --git a/certbot-dns-luadns/certbot_dns_luadns/dns_luadns.py b/certbot-dns-luadns/certbot_dns_luadns/_internal/dns_luadns.py similarity index 100% rename from certbot-dns-luadns/certbot_dns_luadns/dns_luadns.py rename to certbot-dns-luadns/certbot_dns_luadns/_internal/dns_luadns.py index 7cdd4c8e1..7c18c7131 100644 --- a/certbot-dns-luadns/certbot_dns_luadns/dns_luadns.py +++ b/certbot-dns-luadns/certbot_dns_luadns/_internal/dns_luadns.py @@ -1,8 +1,8 @@ """DNS Authenticator for LuaDNS DNS.""" import logging -import zope.interface from lexicon.providers import luadns +import zope.interface from certbot import errors from certbot import interfaces diff --git a/certbot-dns-luadns/docs/api.rst b/certbot-dns-luadns/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-luadns/docs/api.rst +++ b/certbot-dns-luadns/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-luadns/docs/api/dns_luadns.rst b/certbot-dns-luadns/docs/api/dns_luadns.rst deleted file mode 100644 index 9aecbaf05..000000000 --- a/certbot-dns-luadns/docs/api/dns_luadns.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_luadns.dns_luadns` ----------------------------------- - -.. automodule:: certbot_dns_luadns.dns_luadns - :members: diff --git a/certbot-dns-luadns/docs/conf.py b/certbot-dns-luadns/docs/conf.py index bd81d5a5f..8e9d49988 100644 --- a/certbot-dns-luadns/docs/conf.py +++ b/certbot-dns-luadns/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-luadns/local-oldest-requirements.txt b/certbot-dns-luadns/local-oldest-requirements.txt index 65f5a758e..1307698d4 100644 --- a/certbot-dns-luadns/local-oldest-requirements.txt +++ b/certbot-dns-luadns/local-oldest-requirements.txt @@ -1,2 +1,3 @@ --e acme[dev] --e .[dev] +# Remember to update setup.py to match the package versions below. +acme[dev]==0.31.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-luadns/readthedocs.org.requirements.txt b/certbot-dns-luadns/readthedocs.org.requirements.txt index acb51e4ef..6f467dc7c 100644 --- a/certbot-dns-luadns/readthedocs.org.requirements.txt +++ b/certbot-dns-luadns/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-luadns[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-luadns[docs]" does not work as +# expected and "pip install -e certbot-dns-luadns[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-luadns[docs] diff --git a/certbot-dns-luadns/setup.py b/certbot-dns-luadns/setup.py index 31472e8cf..ea64f79a2 100644 --- a/certbot-dns-luadns/setup.py +++ b/certbot-dns-luadns/setup.py @@ -1,14 +1,16 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.31.0.dev0', - 'certbot>=0.31.0.dev0', + 'acme>=0.31.0', + 'certbot>=1.1.0', 'dns-lexicon>=2.2.1', # Support for >1 TXT record per name 'mock', 'setuptools', @@ -20,6 +22,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-luadns', version=version, @@ -28,9 +44,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -39,10 +55,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -59,8 +75,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-luadns = certbot_dns_luadns.dns_luadns:Authenticator', + 'dns-luadns = certbot_dns_luadns._internal.dns_luadns:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_luadns', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-luadns/certbot_dns_luadns/dns_luadns_test.py b/certbot-dns-luadns/tests/dns_luadns_test.py similarity index 85% rename from certbot-dns-luadns/certbot_dns_luadns/dns_luadns_test.py rename to certbot-dns-luadns/tests/dns_luadns_test.py index bf77e03e4..934d3e103 100644 --- a/certbot-dns-luadns/certbot_dns_luadns/dns_luadns_test.py +++ b/certbot-dns-luadns/tests/dns_luadns_test.py @@ -1,11 +1,11 @@ -"""Tests for certbot_dns_luadns.dns_luadns.""" +"""Tests for certbot_dns_luadns._internal.dns_luadns.""" -import os import unittest import mock from requests.exceptions import HTTPError +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins import dns_test_common_lexicon from certbot.tests import util as test_util @@ -20,7 +20,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, def setUp(self): super(AuthenticatorTest, self).setUp() - from certbot_dns_luadns.dns_luadns import Authenticator + from certbot_dns_luadns._internal.dns_luadns import Authenticator path = os.path.join(self.tempdir, 'file.ini') dns_test_common.write({"luadns_email": EMAIL, "luadns_token": TOKEN}, path) @@ -40,7 +40,7 @@ class LuaDNSLexiconClientTest(unittest.TestCase, dns_test_common_lexicon.BaseLex LOGIN_ERROR = HTTPError("401 Client Error: Unauthorized for url: ...") def setUp(self): - from certbot_dns_luadns.dns_luadns import _LuaDNSLexiconClient + from certbot_dns_luadns._internal.dns_luadns import _LuaDNSLexiconClient self.client = _LuaDNSLexiconClient(EMAIL, TOKEN, 0) diff --git a/certbot-dns-nsone/Dockerfile b/certbot-dns-nsone/Dockerfile deleted file mode 100644 index 88fc13c57..000000000 --- a/certbot-dns-nsone/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-nsone - -RUN pip install --no-cache-dir --editable src/certbot-dns-nsone diff --git a/certbot-dns-nsone/MANIFEST.in b/certbot-dns-nsone/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-nsone/MANIFEST.in +++ b/certbot-dns-nsone/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-nsone/certbot_dns_nsone/_internal/__init__.py b/certbot-dns-nsone/certbot_dns_nsone/_internal/__init__.py new file mode 100644 index 000000000..40a095edf --- /dev/null +++ b/certbot-dns-nsone/certbot_dns_nsone/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_nsone.dns_nsone` plugin.""" diff --git a/certbot-dns-nsone/certbot_dns_nsone/dns_nsone.py b/certbot-dns-nsone/certbot_dns_nsone/_internal/dns_nsone.py similarity index 86% rename from certbot-dns-nsone/certbot_dns_nsone/dns_nsone.py rename to certbot-dns-nsone/certbot_dns_nsone/_internal/dns_nsone.py index 3e23df11c..f5af37389 100644 --- a/certbot-dns-nsone/certbot_dns_nsone/dns_nsone.py +++ b/certbot-dns-nsone/certbot_dns_nsone/_internal/dns_nsone.py @@ -1,8 +1,8 @@ """DNS Authenticator for NS1 DNS.""" import logging -import zope.interface from lexicon.providers import nsone +import zope.interface from certbot import errors from certbot import interfaces @@ -76,11 +76,10 @@ class _NS1LexiconClient(dns_common_lexicon.LexiconClient): def _handle_http_error(self, e, domain_name): if domain_name in str(e) and (str(e).startswith('404 Client Error: Not Found for url:') or \ str(e).startswith("400 Client Error: Bad Request for url:")): - return # Expected errors when zone name guess is wrong - else: - hint = None - if str(e).startswith('401 Client Error: Unauthorized for url:'): - hint = 'Is your API key correct?' + return None # Expected errors when zone name guess is wrong + hint = None + if str(e).startswith('401 Client Error: Unauthorized for url:'): + hint = 'Is your API key correct?' - return errors.PluginError('Error determining zone identifier: {0}.{1}' - .format(e, ' ({0})'.format(hint) if hint else '')) + return errors.PluginError('Error determining zone identifier: {0}.{1}' + .format(e, ' ({0})'.format(hint) if hint else '')) diff --git a/certbot-dns-nsone/docs/api.rst b/certbot-dns-nsone/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-nsone/docs/api.rst +++ b/certbot-dns-nsone/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-nsone/docs/api/dns_nsone.rst b/certbot-dns-nsone/docs/api/dns_nsone.rst deleted file mode 100644 index 788ce732a..000000000 --- a/certbot-dns-nsone/docs/api/dns_nsone.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_nsone.dns_nsone` ----------------------------------- - -.. automodule:: certbot_dns_nsone.dns_nsone - :members: diff --git a/certbot-dns-nsone/docs/conf.py b/certbot-dns-nsone/docs/conf.py index cffe2a25c..5531959ed 100644 --- a/certbot-dns-nsone/docs/conf.py +++ b/certbot-dns-nsone/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-nsone/local-oldest-requirements.txt b/certbot-dns-nsone/local-oldest-requirements.txt index 65f5a758e..1307698d4 100644 --- a/certbot-dns-nsone/local-oldest-requirements.txt +++ b/certbot-dns-nsone/local-oldest-requirements.txt @@ -1,2 +1,3 @@ --e acme[dev] --e .[dev] +# Remember to update setup.py to match the package versions below. +acme[dev]==0.31.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-nsone/readthedocs.org.requirements.txt b/certbot-dns-nsone/readthedocs.org.requirements.txt index dbdee4480..bf17eae30 100644 --- a/certbot-dns-nsone/readthedocs.org.requirements.txt +++ b/certbot-dns-nsone/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-nsone[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-nsone[docs]" does not work as +# expected and "pip install -e certbot-dns-nsone[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-nsone[docs] diff --git a/certbot-dns-nsone/setup.py b/certbot-dns-nsone/setup.py index 41b99cc73..d6bedca1c 100644 --- a/certbot-dns-nsone/setup.py +++ b/certbot-dns-nsone/setup.py @@ -1,14 +1,16 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.31.0.dev0', - 'certbot>=0.31.0.dev0', + 'acme>=0.31.0', + 'certbot>=1.1.0', 'dns-lexicon>=2.2.1', # Support for >1 TXT record per name 'mock', 'setuptools', @@ -20,6 +22,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-nsone', version=version, @@ -28,9 +44,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -39,10 +55,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -59,8 +75,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-nsone = certbot_dns_nsone.dns_nsone:Authenticator', + 'dns-nsone = certbot_dns_nsone._internal.dns_nsone:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_nsone', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-nsone/certbot_dns_nsone/dns_nsone_test.py b/certbot-dns-nsone/tests/dns_nsone_test.py similarity index 86% rename from certbot-dns-nsone/certbot_dns_nsone/dns_nsone_test.py rename to certbot-dns-nsone/tests/dns_nsone_test.py index 56668dd01..dd6168f08 100644 --- a/certbot-dns-nsone/certbot_dns_nsone/dns_nsone_test.py +++ b/certbot-dns-nsone/tests/dns_nsone_test.py @@ -1,11 +1,11 @@ -"""Tests for certbot_dns_nsone.dns_nsone.""" +"""Tests for certbot_dns_nsone._internal.dns_nsone.""" -import os import unittest import mock from requests.exceptions import HTTPError +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins import dns_test_common_lexicon from certbot.plugins.dns_test_common import DOMAIN @@ -20,7 +20,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, def setUp(self): super(AuthenticatorTest, self).setUp() - from certbot_dns_nsone.dns_nsone import Authenticator + from certbot_dns_nsone._internal.dns_nsone import Authenticator path = os.path.join(self.tempdir, 'file.ini') dns_test_common.write({"nsone_api_key": API_KEY}, path) @@ -40,7 +40,7 @@ class NS1LexiconClientTest(unittest.TestCase, dns_test_common_lexicon.BaseLexico LOGIN_ERROR = HTTPError('401 Client Error: Unauthorized for url: {0}.'.format(DOMAIN)) def setUp(self): - from certbot_dns_nsone.dns_nsone import _NS1LexiconClient + from certbot_dns_nsone._internal.dns_nsone import _NS1LexiconClient self.client = _NS1LexiconClient(API_KEY, 0) diff --git a/certbot-dns-ovh/Dockerfile b/certbot-dns-ovh/Dockerfile deleted file mode 100644 index e8da96d95..000000000 --- a/certbot-dns-ovh/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-ovh - -RUN pip install --no-cache-dir --editable src/certbot-dns-ovh diff --git a/certbot-dns-ovh/MANIFEST.in b/certbot-dns-ovh/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-ovh/MANIFEST.in +++ b/certbot-dns-ovh/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-ovh/certbot_dns_ovh/_internal/__init__.py b/certbot-dns-ovh/certbot_dns_ovh/_internal/__init__.py new file mode 100644 index 000000000..133694b9e --- /dev/null +++ b/certbot-dns-ovh/certbot_dns_ovh/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_ovh.dns_ovh` plugin.""" diff --git a/certbot-dns-ovh/certbot_dns_ovh/dns_ovh.py b/certbot-dns-ovh/certbot_dns_ovh/_internal/dns_ovh.py similarity index 100% rename from certbot-dns-ovh/certbot_dns_ovh/dns_ovh.py rename to certbot-dns-ovh/certbot_dns_ovh/_internal/dns_ovh.py index 84771b0a8..a495983f2 100644 --- a/certbot-dns-ovh/certbot_dns_ovh/dns_ovh.py +++ b/certbot-dns-ovh/certbot_dns_ovh/_internal/dns_ovh.py @@ -1,8 +1,8 @@ """DNS Authenticator for OVH DNS.""" import logging -import zope.interface from lexicon.providers import ovh +import zope.interface from certbot import errors from certbot import interfaces diff --git a/certbot-dns-ovh/docs/api.rst b/certbot-dns-ovh/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-ovh/docs/api.rst +++ b/certbot-dns-ovh/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-ovh/docs/api/dns_ovh.rst b/certbot-dns-ovh/docs/api/dns_ovh.rst deleted file mode 100644 index 79863d05f..000000000 --- a/certbot-dns-ovh/docs/api/dns_ovh.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_ovh.dns_ovh` ------------------------------- - -.. automodule:: certbot_dns_ovh.dns_ovh - :members: diff --git a/certbot-dns-ovh/docs/conf.py b/certbot-dns-ovh/docs/conf.py index 57194666e..56e24a920 100644 --- a/certbot-dns-ovh/docs/conf.py +++ b/certbot-dns-ovh/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-ovh/local-oldest-requirements.txt b/certbot-dns-ovh/local-oldest-requirements.txt index 01cbcb317..c55e0d570 100644 --- a/certbot-dns-ovh/local-oldest-requirements.txt +++ b/certbot-dns-ovh/local-oldest-requirements.txt @@ -1,3 +1,4 @@ --e acme[dev] --e .[dev] +# Remember to update setup.py to match the package versions below. +acme[dev]==0.31.0 +certbot[dev]==1.1.0 dns-lexicon==2.7.14 diff --git a/certbot-dns-ovh/readthedocs.org.requirements.txt b/certbot-dns-ovh/readthedocs.org.requirements.txt index 0780e12a1..3c21ae0ce 100644 --- a/certbot-dns-ovh/readthedocs.org.requirements.txt +++ b/certbot-dns-ovh/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-ovh[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-ovh[docs]" does not work as +# expected and "pip install -e certbot-dns-ovh[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-ovh[docs] diff --git a/certbot-dns-ovh/setup.py b/certbot-dns-ovh/setup.py index 5b3329568..8f5b052a2 100644 --- a/certbot-dns-ovh/setup.py +++ b/certbot-dns-ovh/setup.py @@ -1,14 +1,16 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.31.0.dev0', - 'certbot>=0.31.0.dev0', + 'acme>=0.31.0', + 'certbot>=1.1.0', 'dns-lexicon>=2.7.14', # Correct proxy use on OVH provider 'mock', 'setuptools', @@ -20,6 +22,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-ovh', version=version, @@ -28,9 +44,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -39,9 +55,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', + 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -58,8 +75,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-ovh = certbot_dns_ovh.dns_ovh:Authenticator', + 'dns-ovh = certbot_dns_ovh._internal.dns_ovh:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_ovh', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-ovh/certbot_dns_ovh/dns_ovh_test.py b/certbot-dns-ovh/tests/dns_ovh_test.py similarity index 89% rename from certbot-dns-ovh/certbot_dns_ovh/dns_ovh_test.py rename to certbot-dns-ovh/tests/dns_ovh_test.py index f2a10485d..a420239ab 100644 --- a/certbot-dns-ovh/certbot_dns_ovh/dns_ovh_test.py +++ b/certbot-dns-ovh/tests/dns_ovh_test.py @@ -1,11 +1,11 @@ -"""Tests for certbot_dns_ovh.dns_ovh.""" +"""Tests for certbot_dns_ovh._internal.dns_ovh.""" -import os import unittest import mock from requests.exceptions import HTTPError +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins import dns_test_common_lexicon from certbot.tests import util as test_util @@ -22,7 +22,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, def setUp(self): super(AuthenticatorTest, self).setUp() - from certbot_dns_ovh.dns_ovh import Authenticator + from certbot_dns_ovh._internal.dns_ovh import Authenticator path = os.path.join(self.tempdir, 'file.ini') credentials = { @@ -48,7 +48,7 @@ class OVHLexiconClientTest(unittest.TestCase, dns_test_common_lexicon.BaseLexico LOGIN_ERROR = HTTPError('403 Client Error: Forbidden for url: https://eu.api.ovh.com/1.0/...') def setUp(self): - from certbot_dns_ovh.dns_ovh import _OVHLexiconClient + from certbot_dns_ovh._internal.dns_ovh import _OVHLexiconClient self.client = _OVHLexiconClient( ENDPOINT, APPLICATION_KEY, APPLICATION_SECRET, CONSUMER_KEY, 0 diff --git a/certbot-dns-rfc2136/Dockerfile b/certbot-dns-rfc2136/Dockerfile deleted file mode 100644 index 1b8feb2f8..000000000 --- a/certbot-dns-rfc2136/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-rfc2136 - -RUN pip install --no-cache-dir --editable src/certbot-dns-rfc2136 diff --git a/certbot-dns-rfc2136/MANIFEST.in b/certbot-dns-rfc2136/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-rfc2136/MANIFEST.in +++ b/certbot-dns-rfc2136/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-rfc2136/certbot_dns_rfc2136/_internal/__init__.py b/certbot-dns-rfc2136/certbot_dns_rfc2136/_internal/__init__.py new file mode 100644 index 000000000..44894bb35 --- /dev/null +++ b/certbot-dns-rfc2136/certbot_dns_rfc2136/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_rfc2136.dns_rfc2136` plugin.""" diff --git a/certbot-dns-rfc2136/certbot_dns_rfc2136/dns_rfc2136.py b/certbot-dns-rfc2136/certbot_dns_rfc2136/_internal/dns_rfc2136.py similarity index 94% rename from certbot-dns-rfc2136/certbot_dns_rfc2136/dns_rfc2136.py rename to certbot-dns-rfc2136/certbot_dns_rfc2136/_internal/dns_rfc2136.py index b8c01cdd3..cb4d5addb 100644 --- a/certbot-dns-rfc2136/certbot_dns_rfc2136/dns_rfc2136.py +++ b/certbot-dns-rfc2136/certbot_dns_rfc2136/_internal/dns_rfc2136.py @@ -57,7 +57,7 @@ class Authenticator(dns_common.DNSAuthenticator): def _validate_algorithm(self, credentials): algorithm = credentials.conf('algorithm') if algorithm: - if not self.ALGORITHMS.get(algorithm): + if not self.ALGORITHMS.get(algorithm.upper()): raise errors.PluginError("Unknown algorithm: {0}.".format(algorithm)) def _setup_credentials(self): @@ -129,7 +129,7 @@ class _RFC2136Client(object): rcode = response.rcode() if rcode == dns.rcode.NOERROR: - logger.debug('Successfully added TXT record') + logger.debug('Successfully added TXT record %s', record_name) else: raise errors.PluginError('Received response from server: {0}' .format(dns.rcode.to_text(rcode))) @@ -164,7 +164,7 @@ class _RFC2136Client(object): rcode = response.rcode() if rcode == dns.rcode.NOERROR: - logger.debug('Successfully deleted TXT record') + logger.debug('Successfully deleted TXT record %s', record_name) else: raise errors.PluginError('Received response from server: {0}' .format(dns.rcode.to_text(rcode))) @@ -206,7 +206,11 @@ class _RFC2136Client(object): request.flags ^= dns.flags.RD try: - response = dns.query.udp(request, self.server, port=self.port) + try: + response = dns.query.tcp(request, self.server, port=self.port) + except OSError as e: + logger.debug('TCP query failed, fallback to UDP: %s', e) + response = dns.query.udp(request, self.server, port=self.port) rcode = response.rcode() # Authoritative Answer bit should be set @@ -220,4 +224,3 @@ class _RFC2136Client(object): except Exception as e: raise errors.PluginError('Encountered error when making query: {0}' .format(e)) - diff --git a/certbot-dns-rfc2136/docs/api.rst b/certbot-dns-rfc2136/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-rfc2136/docs/api.rst +++ b/certbot-dns-rfc2136/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-rfc2136/docs/api/dns_rfc2136.rst b/certbot-dns-rfc2136/docs/api/dns_rfc2136.rst deleted file mode 100644 index f5e98454a..000000000 --- a/certbot-dns-rfc2136/docs/api/dns_rfc2136.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_rfc2136.dns_rfc2136` --------------------------------------- - -.. automodule:: certbot_dns_rfc2136.dns_rfc2136 - :members: diff --git a/certbot-dns-rfc2136/docs/conf.py b/certbot-dns-rfc2136/docs/conf.py index 8cc5d595f..c0d55078e 100644 --- a/certbot-dns-rfc2136/docs/conf.py +++ b/certbot-dns-rfc2136/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-rfc2136/local-oldest-requirements.txt b/certbot-dns-rfc2136/local-oldest-requirements.txt index 8368d266e..cf61c15a5 100644 --- a/certbot-dns-rfc2136/local-oldest-requirements.txt +++ b/certbot-dns-rfc2136/local-oldest-requirements.txt @@ -1,2 +1,3 @@ -acme[dev]==0.21.1 -certbot[dev]==0.21.1 +# Remember to update setup.py to match the package versions below. +acme[dev]==0.29.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-rfc2136/readthedocs.org.requirements.txt b/certbot-dns-rfc2136/readthedocs.org.requirements.txt index df89018ce..2cf4f70f8 100644 --- a/certbot-dns-rfc2136/readthedocs.org.requirements.txt +++ b/certbot-dns-rfc2136/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-rfc2136[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-rfc2136[docs]" does not work as +# expected and "pip install -e certbot-dns-rfc2136[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-rfc2136[docs] diff --git a/certbot-dns-rfc2136/setup.py b/certbot-dns-rfc2136/setup.py index edf7b6ba6..fa51c2108 100644 --- a/certbot-dns-rfc2136/setup.py +++ b/certbot-dns-rfc2136/setup.py @@ -1,14 +1,16 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.21.1', - 'certbot>=0.21.1', + 'acme>=0.29.0', + 'certbot>=1.1.0', 'dnspython', 'mock', 'setuptools', @@ -20,6 +22,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-rfc2136', version=version, @@ -28,9 +44,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -39,10 +55,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -59,8 +75,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-rfc2136 = certbot_dns_rfc2136.dns_rfc2136:Authenticator', + 'dns-rfc2136 = certbot_dns_rfc2136._internal.dns_rfc2136:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_rfc2136', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-rfc2136/certbot_dns_rfc2136/dns_rfc2136_test.py b/certbot-dns-rfc2136/tests/dns_rfc2136_test.py similarity index 87% rename from certbot-dns-rfc2136/certbot_dns_rfc2136/dns_rfc2136_test.py rename to certbot-dns-rfc2136/tests/dns_rfc2136_test.py index 89ce3d93e..c767dba23 100644 --- a/certbot-dns-rfc2136/certbot_dns_rfc2136/dns_rfc2136_test.py +++ b/certbot-dns-rfc2136/tests/dns_rfc2136_test.py @@ -1,6 +1,5 @@ -"""Tests for certbot_dns_rfc2136.dns_rfc2136.""" +"""Tests for certbot_dns_rfc2136._internal.dns_rfc2136.""" -import os import unittest import dns.flags @@ -9,6 +8,7 @@ import dns.tsig import mock from certbot import errors +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins.dns_test_common import DOMAIN from certbot.tests import util as test_util @@ -23,7 +23,7 @@ VALID_CONFIG = {"rfc2136_server": SERVER, "rfc2136_name": NAME, "rfc2136_secret" class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthenticatorTest): def setUp(self): - from certbot_dns_rfc2136.dns_rfc2136 import Authenticator + from certbot_dns_rfc2136._internal.dns_rfc2136 import Authenticator super(AuthenticatorTest, self).setUp() @@ -64,7 +64,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthentic def test_valid_algorithm_passes(self): config = VALID_CONFIG.copy() - config["rfc2136_algorithm"] = "HMAC-SHA512" + config["rfc2136_algorithm"] = "HMAC-sha512" dns_test_common.write(config, self.config.rfc2136_credentials) self.auth.perform([self.achall]) @@ -73,7 +73,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthentic class RFC2136ClientTest(unittest.TestCase): def setUp(self): - from certbot_dns_rfc2136.dns_rfc2136 import _RFC2136Client + from certbot_dns_rfc2136._internal.dns_rfc2136 import _RFC2136Client self.rfc2136_client = _RFC2136Client(SERVER, PORT, NAME, SECRET, dns.tsig.HMAC_MD5) @@ -162,7 +162,7 @@ class RFC2136ClientTest(unittest.TestCase): self.rfc2136_client._find_domain, 'foo.bar.'+DOMAIN) - @mock.patch("dns.query.udp") + @mock.patch("dns.query.tcp") def test_query_soa_found(self, query_mock): query_mock.return_value = mock.MagicMock(answer=[mock.MagicMock()], flags=dns.flags.AA) query_mock.return_value.rcode.return_value = dns.rcode.NOERROR @@ -171,9 +171,9 @@ class RFC2136ClientTest(unittest.TestCase): result = self.rfc2136_client._query_soa(DOMAIN) query_mock.assert_called_with(mock.ANY, SERVER, port=PORT) - self.assertTrue(result == True) + self.assertTrue(result) - @mock.patch("dns.query.udp") + @mock.patch("dns.query.tcp") def test_query_soa_not_found(self, query_mock): query_mock.return_value.rcode.return_value = dns.rcode.NXDOMAIN @@ -181,9 +181,9 @@ class RFC2136ClientTest(unittest.TestCase): result = self.rfc2136_client._query_soa(DOMAIN) query_mock.assert_called_with(mock.ANY, SERVER, port=PORT) - self.assertTrue(result == False) + self.assertFalse(result) - @mock.patch("dns.query.udp") + @mock.patch("dns.query.tcp") def test_query_soa_wraps_errors(self, query_mock): query_mock.side_effect = Exception @@ -193,6 +193,20 @@ class RFC2136ClientTest(unittest.TestCase): self.rfc2136_client._query_soa, DOMAIN) + @mock.patch("dns.query.udp") + @mock.patch("dns.query.tcp") + def test_query_soa_fallback_to_udp(self, tcp_mock, udp_mock): + tcp_mock.side_effect = OSError + udp_mock.return_value = mock.MagicMock(answer=[mock.MagicMock()], flags=dns.flags.AA) + udp_mock.return_value.rcode.return_value = dns.rcode.NOERROR + + # _query_soa | pylint: disable=protected-access + result = self.rfc2136_client._query_soa(DOMAIN) + + tcp_mock.assert_called_with(mock.ANY, SERVER, port=PORT) + udp_mock.assert_called_with(mock.ANY, SERVER, port=PORT) + self.assertTrue(result) + if __name__ == "__main__": unittest.main() # pragma: no cover diff --git a/certbot-dns-route53/Dockerfile b/certbot-dns-route53/Dockerfile deleted file mode 100644 index a1b8d6caf..000000000 --- a/certbot-dns-route53/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-route53 - -RUN pip install --no-cache-dir --editable src/certbot-dns-route53 diff --git a/certbot-dns-route53/LICENSE b/certbot-dns-route53/LICENSE.txt similarity index 100% rename from certbot-dns-route53/LICENSE rename to certbot-dns-route53/LICENSE.txt diff --git a/certbot-dns-route53/MANIFEST.in b/certbot-dns-route53/MANIFEST.in index c48c07e59..fc62028b0 100644 --- a/certbot-dns-route53/MANIFEST.in +++ b/certbot-dns-route53/MANIFEST.in @@ -1,3 +1,6 @@ -include LICENSE +include LICENSE.txt include README recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-route53/certbot_dns_route53/_internal/__init__.py b/certbot-dns-route53/certbot_dns_route53/_internal/__init__.py new file mode 100644 index 000000000..ac9ead791 --- /dev/null +++ b/certbot-dns-route53/certbot_dns_route53/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_route53.dns_route53` plugin.""" diff --git a/certbot-dns-route53/certbot_dns_route53/dns_route53.py b/certbot-dns-route53/certbot_dns_route53/_internal/dns_route53.py similarity index 90% rename from certbot-dns-route53/certbot_dns_route53/dns_route53.py rename to certbot-dns-route53/certbot_dns_route53/_internal/dns_route53.py index f71935de2..637558304 100644 --- a/certbot-dns-route53/certbot_dns_route53/dns_route53.py +++ b/certbot-dns-route53/certbot_dns_route53/_internal/dns_route53.py @@ -4,15 +4,17 @@ import logging import time import boto3 +from botocore.exceptions import ClientError +from botocore.exceptions import NoCredentialsError import zope.interface -from botocore.exceptions import NoCredentialsError, ClientError +from acme.magic_typing import DefaultDict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module from certbot import errors from certbot import interfaces from certbot.plugins import dns_common -from acme.magic_typing import DefaultDict, List, Dict # pylint: disable=unused-import, no-name-in-module - logger = logging.getLogger(__name__) INSTRUCTIONS = ( @@ -20,6 +22,7 @@ INSTRUCTIONS = ( "https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials " # pylint: disable=line-too-long "and add the necessary permissions for Route53 access.") + @zope.interface.implementer(interfaces.IAuthenticator) @zope.interface.provider(interfaces.IPluginFactory) class Authenticator(dns_common.DNSAuthenticator): @@ -44,7 +47,7 @@ class Authenticator(dns_common.DNSAuthenticator): def _setup_credentials(self): pass - def _perform(self, domain, validation_domain_name, validation): # pylint: disable=missing-docstring + def _perform(self, domain, validation_name, validation): # pylint: disable=missing-docstring pass def perform(self, achalls): @@ -65,9 +68,9 @@ class Authenticator(dns_common.DNSAuthenticator): raise errors.PluginError("\n".join([str(e), INSTRUCTIONS])) return [achall.response(achall.account_key) for achall in achalls] - def _cleanup(self, domain, validation_domain_name, validation): + def _cleanup(self, domain, validation_name, validation): try: - self._change_txt_record("DELETE", validation_domain_name, validation) + self._change_txt_record("DELETE", validation_name, validation) except (NoCredentialsError, ClientError) as e: logger.debug('Encountered error during cleanup: %s', e, exc_info=True) diff --git a/certbot-dns-route53/certbot_dns_route53/authenticator.py b/certbot-dns-route53/certbot_dns_route53/authenticator.py index 53215ea1d..2987934a1 100644 --- a/certbot-dns-route53/certbot_dns_route53/authenticator.py +++ b/certbot-dns-route53/certbot_dns_route53/authenticator.py @@ -1,16 +1,17 @@ -"""Shim around `~certbot_dns_route53.dns_route53` for backwards compatibility.""" +"""Shim around `~certbot_dns_route53._internal.dns_route53` for backwards compatibility.""" import warnings import zope.interface from certbot import interfaces -from certbot_dns_route53 import dns_route53 +from certbot_dns_route53._internal import dns_route53 @zope.interface.implementer(interfaces.IAuthenticator) @zope.interface.provider(interfaces.IPluginFactory) class Authenticator(dns_route53.Authenticator): - """Shim around `~certbot_dns_route53.dns_route53.Authenticator` for backwards compatibility.""" + """Shim around `~certbot_dns_route53._internal.dns_route53.Authenticator` + for backwards compatibility.""" hidden = True diff --git a/certbot-dns-route53/docs/api.rst b/certbot-dns-route53/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-route53/docs/api.rst +++ b/certbot-dns-route53/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-route53/docs/api/authenticator.rst b/certbot-dns-route53/docs/api/authenticator.rst deleted file mode 100644 index 2d96a419b..000000000 --- a/certbot-dns-route53/docs/api/authenticator.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_route53.authenticator` ----------------------------------------- - -.. automodule:: certbot_dns_route53.authenticator - :members: diff --git a/certbot-dns-route53/docs/api/dns_route53.rst b/certbot-dns-route53/docs/api/dns_route53.rst deleted file mode 100644 index 7573f2e19..000000000 --- a/certbot-dns-route53/docs/api/dns_route53.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_route53.dns_route53` --------------------------------------- - -.. automodule:: certbot_dns_route53.dns_route53 - :members: diff --git a/certbot-dns-route53/docs/conf.py b/certbot-dns-route53/docs/conf.py index 25a7c6e4d..c2eb880ac 100644 --- a/certbot-dns-route53/docs/conf.py +++ b/certbot-dns-route53/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-route53/local-oldest-requirements.txt b/certbot-dns-route53/local-oldest-requirements.txt index 4e4aadbd8..cf61c15a5 100644 --- a/certbot-dns-route53/local-oldest-requirements.txt +++ b/certbot-dns-route53/local-oldest-requirements.txt @@ -1,2 +1,3 @@ -acme[dev]==0.25.0 -certbot[dev]==0.21.1 +# Remember to update setup.py to match the package versions below. +acme[dev]==0.29.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-route53/readthedocs.org.requirements.txt b/certbot-dns-route53/readthedocs.org.requirements.txt index 660a90d0e..993225eac 100644 --- a/certbot-dns-route53/readthedocs.org.requirements.txt +++ b/certbot-dns-route53/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-route53[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-route53[docs]" does not work as +# expected and "pip install -e certbot-dns-route53[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-route53[docs] diff --git a/certbot-dns-route53/setup.py b/certbot-dns-route53/setup.py index 69c2c7ed3..f25e348ff 100644 --- a/certbot-dns-route53/setup.py +++ b/certbot-dns-route53/setup.py @@ -1,19 +1,36 @@ -from setuptools import setup -from setuptools import find_packages +import sys -version = '0.31.0.dev0' +from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand + +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.25.0', - 'certbot>=0.21.1', + 'acme>=0.29.0', + 'certbot>=1.1.0', 'boto3', 'mock', 'setuptools', 'zope.interface', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-route53', version=version, @@ -22,9 +39,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -33,10 +50,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -50,9 +67,11 @@ setup( keywords=['certbot', 'route53', 'aws'], entry_points={ 'certbot.plugins': [ - 'dns-route53 = certbot_dns_route53.dns_route53:Authenticator', + 'dns-route53 = certbot_dns_route53._internal.dns_route53:Authenticator', 'certbot-route53:auth = certbot_dns_route53.authenticator:Authenticator' ], }, + tests_require=["pytest"], test_suite='certbot_dns_route53', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-route53/certbot_dns_route53/dns_route53_test.py b/certbot-dns-route53/tests/dns_route53_test.py similarity index 96% rename from certbot-dns-route53/certbot_dns_route53/dns_route53_test.py rename to certbot-dns-route53/tests/dns_route53_test.py index 71326c2af..85ec259b1 100644 --- a/certbot-dns-route53/certbot_dns_route53/dns_route53_test.py +++ b/certbot-dns-route53/tests/dns_route53_test.py @@ -1,12 +1,13 @@ -"""Tests for certbot_dns_route53.dns_route53.Authenticator""" +"""Tests for certbot_dns_route53._internal.dns_route53.Authenticator""" -import os import unittest +from botocore.exceptions import ClientError +from botocore.exceptions import NoCredentialsError import mock -from botocore.exceptions import NoCredentialsError, ClientError from certbot import errors +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins.dns_test_common import DOMAIN @@ -15,7 +16,7 @@ class AuthenticatorTest(unittest.TestCase, dns_test_common.BaseAuthenticatorTest # pylint: disable=protected-access def setUp(self): - from certbot_dns_route53.dns_route53 import Authenticator + from certbot_dns_route53._internal.dns_route53 import Authenticator super(AuthenticatorTest, self).setUp() @@ -122,7 +123,7 @@ class ClientTest(unittest.TestCase): } def setUp(self): - from certbot_dns_route53.dns_route53 import Authenticator + from certbot_dns_route53._internal.dns_route53 import Authenticator super(ClientTest, self).setUp() diff --git a/certbot-dns-route53/tools/tester.pkoch-macos_sierra.sh b/certbot-dns-route53/tools/tester.pkoch-macos_sierra.sh deleted file mode 100755 index dbbaa2251..000000000 --- a/certbot-dns-route53/tools/tester.pkoch-macos_sierra.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash -# I just wanted a place to dump the incantations I use for testing. -set -e - -brew install openssl libffi - -rm -rf scratch; mkdir scratch - -virtualenv scratch/venv -p /usr/local/bin/python2.7 -scratch/venv/bin/pip install -U pip setuptools - -CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib scratch/venv/bin/pip install -e . - -scratch/venv/bin/certbot certonly -n --agree-tos --test-cert --email pkoch@lifeonmars.pt -a certbot-route53:auth -d pkoch.lifeonmars.pt --work-dir scratch --config-dir scratch --logs-dir scratch - -rm -rf scratch diff --git a/certbot-dns-sakuracloud/Dockerfile b/certbot-dns-sakuracloud/Dockerfile deleted file mode 100644 index 694773f61..000000000 --- a/certbot-dns-sakuracloud/Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM certbot/certbot - -COPY . src/certbot-dns-sakuracloud - -RUN pip install --no-cache-dir --editable src/certbot-dns-sakuracloud diff --git a/certbot-dns-sakuracloud/MANIFEST.in b/certbot-dns-sakuracloud/MANIFEST.in index 18f018c08..5a661cef6 100644 --- a/certbot-dns-sakuracloud/MANIFEST.in +++ b/certbot-dns-sakuracloud/MANIFEST.in @@ -1,3 +1,6 @@ include LICENSE.txt include README.rst recursive-include docs * +recursive-include tests * +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-dns-sakuracloud/certbot_dns_sakuracloud/_internal/__init__.py b/certbot-dns-sakuracloud/certbot_dns_sakuracloud/_internal/__init__.py new file mode 100644 index 000000000..0c8839024 --- /dev/null +++ b/certbot-dns-sakuracloud/certbot_dns_sakuracloud/_internal/__init__.py @@ -0,0 +1 @@ +"""Internal implementation of `~certbot_dns_sakuracloud.dns_sakuracloud` plugin.""" diff --git a/certbot-dns-sakuracloud/certbot_dns_sakuracloud/dns_sakuracloud.py b/certbot-dns-sakuracloud/certbot_dns_sakuracloud/_internal/dns_sakuracloud.py similarity index 97% rename from certbot-dns-sakuracloud/certbot_dns_sakuracloud/dns_sakuracloud.py rename to certbot-dns-sakuracloud/certbot_dns_sakuracloud/_internal/dns_sakuracloud.py index 7fd6d3ef5..25042bfc6 100644 --- a/certbot-dns-sakuracloud/certbot_dns_sakuracloud/dns_sakuracloud.py +++ b/certbot-dns-sakuracloud/certbot_dns_sakuracloud/_internal/dns_sakuracloud.py @@ -1,8 +1,8 @@ """DNS Authenticator for Sakura Cloud DNS.""" import logging -import zope.interface from lexicon.providers import sakuracloud +import zope.interface from certbot import interfaces from certbot.plugins import dns_common @@ -86,5 +86,5 @@ class _SakuraCloudLexiconClient(dns_common_lexicon.LexiconClient): def _handle_http_error(self, e, domain_name): if domain_name in str(e) and (str(e).startswith('404 Client Error: Not Found for url:')): - return # Expected errors when zone name guess is wrong + return None # Expected errors when zone name guess is wrong return super(_SakuraCloudLexiconClient, self)._handle_http_error(e, domain_name) diff --git a/certbot-dns-sakuracloud/docs/api.rst b/certbot-dns-sakuracloud/docs/api.rst index 8668ec5d8..ac13c3df2 100644 --- a/certbot-dns-sakuracloud/docs/api.rst +++ b/certbot-dns-sakuracloud/docs/api.rst @@ -2,7 +2,4 @@ API Documentation ================= -.. toctree:: - :glob: - - api/** +Certbot plugins implement the Certbot plugins API, and do not otherwise have an external API. diff --git a/certbot-dns-sakuracloud/docs/api/dns_sakuracloud.rst b/certbot-dns-sakuracloud/docs/api/dns_sakuracloud.rst deleted file mode 100644 index 74692e15b..000000000 --- a/certbot-dns-sakuracloud/docs/api/dns_sakuracloud.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_dns_sakuracloud.dns_sakuracloud` ----------------------------------------------- - -.. automodule:: certbot_dns_sakuracloud.dns_sakuracloud - :members: diff --git a/certbot-dns-sakuracloud/docs/conf.py b/certbot-dns-sakuracloud/docs/conf.py index e14fe1d4c..70a4d7434 100644 --- a/certbot-dns-sakuracloud/docs/conf.py +++ b/certbot-dns-sakuracloud/docs/conf.py @@ -17,6 +17,7 @@ # documentation root, use os.path.abspath to make it absolute, like shown here. # import os + # import sys # sys.path.insert(0, os.path.abspath('.')) @@ -37,7 +38,7 @@ extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/certbot-dns-sakuracloud/local-oldest-requirements.txt b/certbot-dns-sakuracloud/local-oldest-requirements.txt index 65f5a758e..1307698d4 100644 --- a/certbot-dns-sakuracloud/local-oldest-requirements.txt +++ b/certbot-dns-sakuracloud/local-oldest-requirements.txt @@ -1,2 +1,3 @@ --e acme[dev] --e .[dev] +# Remember to update setup.py to match the package versions below. +acme[dev]==0.31.0 +certbot[dev]==1.1.0 diff --git a/certbot-dns-sakuracloud/readthedocs.org.requirements.txt b/certbot-dns-sakuracloud/readthedocs.org.requirements.txt index 3f46d95ef..07bc8a289 100644 --- a/certbot-dns-sakuracloud/readthedocs.org.requirements.txt +++ b/certbot-dns-sakuracloud/readthedocs.org.requirements.txt @@ -1,12 +1,12 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot-dns-sakuracloud[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# in --editable mode (-e), just "pip install certbot-dns-sakuracloud[docs]" does not work as +# expected and "pip install -e certbot-dns-sakuracloud[docs]" must be used instead -e acme --e . +-e certbot -e certbot-dns-sakuracloud[docs] diff --git a/certbot-dns-sakuracloud/setup.py b/certbot-dns-sakuracloud/setup.py index 4ebfc6e1d..8df2320ba 100644 --- a/certbot-dns-sakuracloud/setup.py +++ b/certbot-dns-sakuracloud/setup.py @@ -1,13 +1,15 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Please update tox.ini when modifying dependency version requirements install_requires = [ - 'acme>=0.31.0.dev0', - 'certbot>=0.31.0.dev0', + 'acme>=0.31.0', + 'certbot>=1.1.0', 'dns-lexicon>=2.1.23', 'mock', 'setuptools', @@ -19,6 +21,20 @@ docs_extras = [ 'sphinx_rtd_theme', ] +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-dns-sakuracloud', version=version, @@ -27,9 +43,9 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Apache Software License', @@ -38,9 +54,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', + 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -57,8 +74,10 @@ setup( }, entry_points={ 'certbot.plugins': [ - 'dns-sakuracloud = certbot_dns_sakuracloud.dns_sakuracloud:Authenticator', + 'dns-sakuracloud = certbot_dns_sakuracloud._internal.dns_sakuracloud:Authenticator', ], }, + tests_require=["pytest"], test_suite='certbot_dns_sakuracloud', + cmdclass={"test": PyTest}, ) diff --git a/certbot-dns-sakuracloud/certbot_dns_sakuracloud/dns_sakuracloud_test.py b/certbot-dns-sakuracloud/tests/dns_sakuracloud_test.py similarity index 86% rename from certbot-dns-sakuracloud/certbot_dns_sakuracloud/dns_sakuracloud_test.py rename to certbot-dns-sakuracloud/tests/dns_sakuracloud_test.py index 1d9282f9a..16890b5a9 100644 --- a/certbot-dns-sakuracloud/certbot_dns_sakuracloud/dns_sakuracloud_test.py +++ b/certbot-dns-sakuracloud/tests/dns_sakuracloud_test.py @@ -1,11 +1,11 @@ -"""Tests for certbot_dns_sakuracloud.dns_sakuracloud.""" +"""Tests for certbot_dns_sakuracloud._internal.dns_sakuracloud.""" -import os import unittest import mock from requests.exceptions import HTTPError +from certbot.compat import os from certbot.plugins import dns_test_common from certbot.plugins import dns_test_common_lexicon from certbot.plugins.dns_test_common import DOMAIN @@ -20,7 +20,7 @@ class AuthenticatorTest(test_util.TempDirTestCase, def setUp(self): super(AuthenticatorTest, self).setUp() - from certbot_dns_sakuracloud.dns_sakuracloud import Authenticator + from certbot_dns_sakuracloud._internal.dns_sakuracloud import Authenticator path = os.path.join(self.tempdir, 'file.ini') dns_test_common.write( @@ -44,7 +44,7 @@ class SakuraCloudLexiconClientTest(unittest.TestCase, LOGIN_ERROR = HTTPError('401 Client Error: Unauthorized for url: {0}.'.format(DOMAIN)) def setUp(self): - from certbot_dns_sakuracloud.dns_sakuracloud import _SakuraCloudLexiconClient + from certbot_dns_sakuracloud._internal.dns_sakuracloud import _SakuraCloudLexiconClient self.client = _SakuraCloudLexiconClient(API_TOKEN, API_SECRET, 0) diff --git a/certbot-nginx/MANIFEST.in b/certbot-nginx/MANIFEST.in index 2daca6738..65b27877e 100644 --- a/certbot-nginx/MANIFEST.in +++ b/certbot-nginx/MANIFEST.in @@ -1,5 +1,6 @@ include LICENSE.txt include README.rst -recursive-include docs * -recursive-include certbot_nginx/tests/testdata * -include certbot_nginx/options-ssl-nginx.conf +recursive-include tests * +recursive-include certbot_nginx/_internal/tls_configs *.conf +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot-nginx/certbot_nginx/_internal/__init__.py b/certbot-nginx/certbot_nginx/_internal/__init__.py new file mode 100644 index 000000000..71d79b0c2 --- /dev/null +++ b/certbot-nginx/certbot_nginx/_internal/__init__.py @@ -0,0 +1 @@ +"""Certbot nginx plugin internal implementation.""" diff --git a/certbot-nginx/certbot_nginx/configurator.py b/certbot-nginx/certbot_nginx/_internal/configurator.py similarity index 88% rename from certbot-nginx/certbot_nginx/configurator.py rename to certbot-nginx/certbot_nginx/_internal/configurator.py index dd0bf9e8b..fdead036a 100644 --- a/certbot-nginx/certbot_nginx/configurator.py +++ b/certbot-nginx/certbot_nginx/_internal/configurator.py @@ -1,6 +1,7 @@ """Nginx Configuration""" +# https://github.com/PyCQA/pylint/issues/73 +from distutils.version import LooseVersion # pylint: disable=no-name-in-module, import-error import logging -import os import re import socket import subprocess @@ -8,28 +9,26 @@ import tempfile import time import OpenSSL +import pkg_resources import zope.interface from acme import challenges from acme import crypto_util as acme_crypto_util - -from certbot import constants as core_constants +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module from certbot import crypto_util from certbot import errors from certbot import interfaces from certbot import util - +from certbot.compat import os from certbot.plugins import common - -from certbot_nginx import constants -from certbot_nginx import display_ops -from certbot_nginx import nginxparser -from certbot_nginx import parser -from certbot_nginx import tls_sni_01 -from certbot_nginx import http_01 -from certbot_nginx import obj # pylint: disable=unused-import -from acme.magic_typing import List, Dict, Set # pylint: disable=unused-import, no-name-in-module - +from certbot_nginx._internal import constants +from certbot_nginx._internal import display_ops +from certbot_nginx._internal import http_01 +from certbot_nginx._internal import nginxparser +from certbot_nginx._internal import obj # pylint: disable=unused-import +from certbot_nginx._internal import parser NAME_RANK = 0 START_WILDCARD_RANK = 1 @@ -44,7 +43,6 @@ logger = logging.getLogger(__name__) @zope.interface.implementer(interfaces.IAuthenticator, interfaces.IInstaller) @zope.interface.provider(interfaces.IPluginFactory) class NginxConfigurator(common.Installer): - # pylint: disable=too-many-instance-attributes,too-many-public-methods """Nginx configurator. .. todo:: Add proper support for comments in the config. Currently, @@ -54,7 +52,7 @@ class NginxConfigurator(common.Installer): :type config: :class:`~certbot.interfaces.IConfig` :ivar parser: Handles low level parsing - :type parser: :class:`~certbot_nginx.parser` + :type parser: :class:`~certbot_nginx._internal.parser` :ivar str save_notes: Human-readable config change notes @@ -92,13 +90,14 @@ class NginxConfigurator(common.Installer): :param tup version: version of Nginx as a tuple (1, 4, 7) (used mostly for unittesting) + :param tup openssl_version: version of OpenSSL linked to Nginx as a tuple (1, 4, 7) + (used mostly for unittesting) + """ version = kwargs.pop("version", None) + openssl_version = kwargs.pop("openssl_version", None) super(NginxConfigurator, self).__init__(*args, **kwargs) - # Verify that all directories and files exist with proper permissions - self._verify_setup() - # Files to save self.save_notes = "" @@ -116,12 +115,46 @@ class NginxConfigurator(common.Installer): # These will be set in the prepare function self.parser = None self.version = version + self.openssl_version = openssl_version self._enhance_func = {"redirect": self._enable_redirect, "ensure-http-header": self._set_http_header, "staple-ocsp": self._enable_ocsp_stapling} self.reverter.recovery_routine() + @property + def mod_ssl_conf_src(self): + """Full absolute path to SSL configuration file source.""" + + # Why all this complexity? Well, we want to support Mozilla's intermediate + # recommendations. But TLS1.3 is only supported by newer versions of Nginx. + # And as for session tickets, our ideal is to turn them off across the board. + # But! Turning them off at all is only supported with new enough versions of + # Nginx. And older versions of OpenSSL have a bug that leads to browser errors + # given certain configurations. While we'd prefer to have forward secrecy, we'd + # rather fail open than error out. Unfortunately, Nginx can be compiled against + # many versions of OpenSSL. So we have to check both for the two different features, + # leading to four different combinations of options. + # For a complete history, check out https://github.com/certbot/certbot/issues/7322 + + use_tls13 = self.version >= (1, 13, 0) + session_tix_off = self.version >= (1, 5, 9) and self.openssl_version and\ + LooseVersion(self.openssl_version) >= LooseVersion('1.0.2l') + + if use_tls13: + if session_tix_off: + config_filename = "options-ssl-nginx.conf" + else: + config_filename = "options-ssl-nginx-tls13-session-tix-on.conf" + else: + if session_tix_off: + config_filename = "options-ssl-nginx-tls12-only.conf" + else: + config_filename = "options-ssl-nginx-old.conf" + + return pkg_resources.resource_filename( + "certbot_nginx", os.path.join("_internal", "tls_configs", config_filename)) + @property def mod_ssl_conf(self): """Full absolute path to SSL configuration file.""" @@ -132,6 +165,11 @@ class NginxConfigurator(common.Installer): """Full absolute path to digest of updated SSL configuration file.""" return os.path.join(self.config.config_dir, constants.UPDATED_MOD_SSL_CONF_DIGEST) + def install_ssl_options_conf(self, options_ssl, options_ssl_digest): + """Copy Certbot's SSL options file into the system's config dir if required.""" + return common.install_version_controlled_file(options_ssl, options_ssl_digest, + self.mod_ssl_conf_src, constants.ALL_SSL_OPTIONS_HASHES) + # This is called in determine_authenticator and determine_installer def prepare(self): """Prepare the authenticator/installer. @@ -148,30 +186,29 @@ class NginxConfigurator(common.Installer): # Make sure configuration is valid self.config_test() - self.parser = parser.NginxParser(self.conf('server-root')) - install_ssl_options_conf(self.mod_ssl_conf, self.updated_mod_ssl_conf_digest) - - self.install_ssl_dhparams() - # Set Version if self.version is None: self.version = self.get_version() + if self.openssl_version is None: + self.openssl_version = self._get_openssl_version() + + self.install_ssl_options_conf(self.mod_ssl_conf, self.updated_mod_ssl_conf_digest) + + self.install_ssl_dhparams() + # Prevent two Nginx plugins from modifying a config at once try: util.lock_dir_until_exit(self.conf('server-root')) except (OSError, errors.LockError): logger.debug('Encountered error:', exc_info=True) - raise errors.PluginError( - 'Unable to lock %s', self.conf('server-root')) - + raise errors.PluginError('Unable to lock {0}'.format(self.conf('server-root'))) # Entry point in main.py for installing cert def deploy_cert(self, domain, cert_path, key_path, chain_path=None, fullchain_path=None): - # pylint: disable=unused-argument """Deploys certificate to specified virtual host. .. note:: Aborts if the vhost is missing ssl_certificate or @@ -192,8 +229,7 @@ class NginxConfigurator(common.Installer): for vhost in vhosts: self._deploy_cert(vhost, cert_path, key_path, chain_path, fullchain_path) - def _deploy_cert(self, vhost, cert_path, key_path, chain_path, fullchain_path): - # pylint: disable=unused-argument + def _deploy_cert(self, vhost, cert_path, key_path, chain_path, fullchain_path): # pylint: disable=unused-argument """ Helper function for deploy_cert() that handles the actual deployment this exists because we might want to do multiple deployments per @@ -247,7 +283,7 @@ class NginxConfigurator(common.Installer): filtered_vhosts[name] = vhost # Only unique VHost objects - dialog_input = set([vhost for vhost in filtered_vhosts.values()]) + dialog_input = set(filtered_vhosts.values()) # Ask the user which of names to enable, expect list of names back return_vhosts = display_ops.select_vhost_multiple(list(dialog_input)) @@ -286,7 +322,7 @@ class NginxConfigurator(common.Installer): MisconfigurationError. :returns: ssl vhosts associated with name - :rtype: list of :class:`~certbot_nginx.obj.VirtualHost` + :rtype: list of :class:`~certbot_nginx._internal.obj.VirtualHost` """ if util.is_wildcard_domain(target_name): @@ -298,7 +334,7 @@ class NginxConfigurator(common.Installer): if create_if_no_match: # result will not be [None] because it errors on failure vhosts = [self._vhost_from_duplicated_default(target_name, True, - str(self.config.tls_sni_01_port))] + str(self.config.https_port))] else: # No matches. Raise a misconfiguration error. raise errors.MisconfigurationError( @@ -405,7 +441,7 @@ class NginxConfigurator(common.Installer): :param list matches: list of dicts containing the vhost, the matching name, and the numerical rank :returns: the most matching vhost - :rtype: :class:`~certbot_nginx.obj.VirtualHost` + :rtype: :class:`~certbot_nginx._internal.obj.VirtualHost` """ if not matches: @@ -416,9 +452,8 @@ class NginxConfigurator(common.Installer): rank = matches[0]['rank'] wildcards = [x for x in matches if x['rank'] == rank] return max(wildcards, key=lambda x: len(x['name']))['vhost'] - else: - # Exact or regex match - return matches[0]['vhost'] + # Exact or regex match + return matches[0]['vhost'] def _rank_matches_by_name(self, vhost_list, target_name): """Returns a ranked list of vhosts from vhost_list that match target_name. @@ -493,7 +528,7 @@ class NginxConfigurator(common.Installer): MisconfigurationError. :returns: vhosts associated with name - :rtype: list of :class:`~certbot_nginx.obj.VirtualHost` + :rtype: list of :class:`~certbot_nginx._internal.obj.VirtualHost` """ if util.is_wildcard_domain(target_name): @@ -512,25 +547,23 @@ class NginxConfigurator(common.Installer): if matching_port == "" or matching_port is None: # if no port is specified, Nginx defaults to listening on port 80. return test_port == self.DEFAULT_LISTEN_PORT - else: - return test_port == matching_port + return test_port == matching_port def _vhost_listening_on_port_no_ssl(self, vhost, port): found_matching_port = False - if len(vhost.addrs) == 0: + if not vhost.addrs: # if there are no listen directives at all, Nginx defaults to # listening on port 80. found_matching_port = (port == self.DEFAULT_LISTEN_PORT) else: for addr in vhost.addrs: - if self._port_matches(port, addr.get_port()) and addr.ssl == False: + if self._port_matches(port, addr.get_port()) and not addr.ssl: found_matching_port = True if found_matching_port: # make sure we don't have an 'ssl on' directive return not self.parser.has_ssl_on_directive(vhost) - else: - return False + return False def _get_redirect_ranked_matches(self, target_name, port): """Gets a ranked list of plaintextish port-listening vhosts matching target_name @@ -562,7 +595,7 @@ class NginxConfigurator(common.Installer): :rtype: set """ - all_names = set() # type: Set[str] + all_names = set() # type: Set[str] for vhost in self.parser.get_vhosts(): all_names.update(vhost.names) @@ -609,16 +642,17 @@ class NginxConfigurator(common.Installer): Make a server SSL by adding new listen and SSL directives. :param vhost: The vhost to add SSL to. - :type vhost: :class:`~certbot_nginx.obj.VirtualHost` + :type vhost: :class:`~certbot_nginx._internal.obj.VirtualHost` """ - ipv6info = self.ipv6_info(self.config.tls_sni_01_port) + https_port = self.config.https_port + ipv6info = self.ipv6_info(https_port) ipv6_block = [''] ipv4_block = [''] # If the vhost was implicitly listening on the default Nginx port, # have it continue to do so. - if len(vhost.addrs) == 0: + if not vhost.addrs: listen_block = [['\n ', 'listen', ' ', self.DEFAULT_LISTEN_PORT]] self.parser.add_server_directives(vhost, listen_block) @@ -626,7 +660,7 @@ class NginxConfigurator(common.Installer): ipv6_block = ['\n ', 'listen', ' ', - '[::]:{0}'.format(self.config.tls_sni_01_port), + '[::]:{0}'.format(https_port), ' ', 'ssl'] if not ipv6info[1]: @@ -638,7 +672,7 @@ class NginxConfigurator(common.Installer): ipv4_block = ['\n ', 'listen', ' ', - '{0}'.format(self.config.tls_sni_01_port), + '{0}'.format(https_port), ' ', 'ssl'] @@ -668,9 +702,9 @@ class NginxConfigurator(common.Installer): :param str domain: domain to enhance :param str enhancement: enhancement type defined in - :const:`~certbot.constants.ENHANCEMENTS` + :const:`~certbot.plugins.enhancements.ENHANCEMENTS` :param options: options for the enhancement - See :const:`~certbot.constants.ENHANCEMENTS` + See :const:`~certbot.plugins.enhancements.ENHANCEMENTS` documentation for appropriate parameter. """ @@ -736,9 +770,9 @@ class NginxConfigurator(common.Installer): :param vhost: The server block to break up into two. :param list only_directives: If this exists, only duplicate these directives when splitting the block. - :type vhost: :class:`~certbot_nginx.obj.VirtualHost` + :type vhost: :class:`~certbot_nginx._internal.obj.VirtualHost` :returns: tuple (http_vhost, https_vhost) - :rtype: tuple of type :class:`~certbot_nginx.obj.VirtualHost` + :rtype: tuple of type :class:`~certbot_nginx._internal.obj.VirtualHost` """ http_vhost = self.parser.duplicate_vhost(vhost, only_directives=only_directives) @@ -800,8 +834,6 @@ class NginxConfigurator(common.Installer): :param str domain: domain to enable redirect for :param `~obj.Vhost` vhost: vhost to enable redirect for """ - - http_vhost = None if vhost.ssl: http_vhost, _ = self._split_block(vhost, ['listen', 'server_name']) @@ -891,21 +923,27 @@ class NginxConfigurator(common.Installer): except errors.SubprocessError as err: raise errors.MisconfigurationError(str(err)) - def _verify_setup(self): - """Verify the setup to ensure safe operating environment. + def _nginx_version(self): + """Return results of nginx -V - Make sure that files/directories are setup with appropriate permissions - Aim for defensive coding... make sure all input files - have permissions of root. + :returns: version text + :rtype: str + :raises .PluginError: + Unable to run Nginx version command """ - uid = os.geteuid() - util.make_or_verify_dir( - self.config.work_dir, core_constants.CONFIG_DIRS_MODE, uid) - util.make_or_verify_dir( - self.config.backup_dir, core_constants.CONFIG_DIRS_MODE, uid) - util.make_or_verify_dir( - self.config.config_dir, core_constants.CONFIG_DIRS_MODE, uid) + try: + proc = subprocess.Popen( + [self.conf('ctl'), "-c", self.nginx_conf, "-V"], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + universal_newlines=True) + text = proc.communicate()[1] # nginx prints output to stderr + except (OSError, ValueError) as error: + logger.debug(str(error), exc_info=True) + raise errors.PluginError( + "Unable to run %s -V" % self.conf('ctl')) + return text def get_version(self): """Return version of Nginx Server. @@ -919,17 +957,7 @@ class NginxConfigurator(common.Installer): Unable to find Nginx version or version is unsupported """ - try: - proc = subprocess.Popen( - [self.conf('ctl'), "-c", self.nginx_conf, "-V"], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - universal_newlines=True) - text = proc.communicate()[1] # nginx prints output to stderr - except (OSError, ValueError) as error: - logger.debug(str(error), exc_info=True) - raise errors.PluginError( - "Unable to run %s -V" % self.conf('ctl')) + text = self._nginx_version() version_regex = re.compile(r"nginx version: ([^/]+)/([0-9\.]*)", re.IGNORECASE) version_matches = version_regex.findall(text) @@ -962,6 +990,28 @@ class NginxConfigurator(common.Installer): return nginx_version + def _get_openssl_version(self): + """Return version of OpenSSL linked to Nginx. + + Version is returned as string. If no version can be found, empty string is returned. + + :returns: openssl_version + :rtype: str + + :raises .PluginError: + Unable to run Nginx version command + """ + text = self._nginx_version() + + matches = re.findall(r"running with OpenSSL ([^ ]+) ", text) + if not matches: + matches = re.findall(r"built with OpenSSL ([^ ]+) ", text) + if not matches: + logger.warning("NGINX configured with OpenSSL alternatives is not officially" + " supported by Certbot.") + return "" + return matches[0] + def more_info(self): """Human-readable string to help understand the module""" return ( @@ -1039,7 +1089,7 @@ class NginxConfigurator(common.Installer): ########################################################################### def get_chall_pref(self, unused_domain): # pylint: disable=no-self-use """Return list of challenge preferences.""" - return [challenges.HTTP01, challenges.TLSSNI01] + return [challenges.HTTP01] # Entry point in main.py for performing challenges def perform(self, achalls): @@ -1052,19 +1102,14 @@ class NginxConfigurator(common.Installer): """ self._chall_out += len(achalls) responses = [None] * len(achalls) - sni_doer = tls_sni_01.NginxTlsSni01(self) http_doer = http_01.NginxHttp01(self) for i, achall in enumerate(achalls): # Currently also have chall_doer hold associated index of the # challenge. This helps to put all of the responses back together # when they are all complete. - if isinstance(achall.chall, challenges.HTTP01): - http_doer.add_chall(achall, i) - else: # tls-sni-01 - sni_doer.add_chall(achall, i) + http_doer.add_chall(achall, i) - sni_response = sni_doer.perform() http_response = http_doer.perform() # Must restart in order to activate the challenges. # Handled here because we may be able to load up other challenge types @@ -1073,9 +1118,8 @@ class NginxConfigurator(common.Installer): # Go through all of the challenges and assign them to the proper place # in the responses return value. All responses must be in the same order # as the original challenges. - for chall_response, chall_doer in ((sni_response, sni_doer), (http_response, http_doer)): - for i, resp in enumerate(chall_response): - responses[chall_doer.indices[i]] = resp + for i, resp in enumerate(http_response): + responses[http_doer.indices[i]] = resp return responses @@ -1148,11 +1192,6 @@ def nginx_restart(nginx_ctl, nginx_conf): time.sleep(1) -def install_ssl_options_conf(options_ssl, options_ssl_digest): - """Copy Certbot's SSL options file into the system's config dir if required.""" - return common.install_version_controlled_file(options_ssl, options_ssl_digest, - constants.MOD_SSL_CONF_SRC, constants.ALL_SSL_OPTIONS_HASHES) - def _determine_default_server_root(): if os.environ.get("CERTBOT_DOCS") == "1": default_server_root = "%s or %s" % (constants.LINUX_SERVER_ROOT, diff --git a/certbot-nginx/certbot_nginx/constants.py b/certbot-nginx/certbot_nginx/_internal/constants.py similarity index 67% rename from certbot-nginx/certbot_nginx/constants.py rename to certbot-nginx/certbot_nginx/_internal/constants.py index d749b6989..fbf6ed424 100644 --- a/certbot-nginx/certbot_nginx/constants.py +++ b/certbot-nginx/certbot_nginx/_internal/constants.py @@ -1,5 +1,4 @@ """nginx plugin constants.""" -import pkg_resources import platform FREEBSD_DARWIN_SERVER_ROOT = "/usr/local/etc/nginx" @@ -20,15 +19,9 @@ CLI_DEFAULTS = dict( MOD_SSL_CONF_DEST = "options-ssl-nginx.conf" """Name of the mod_ssl config file as saved in `IConfig.config_dir`.""" -MOD_SSL_CONF_SRC = pkg_resources.resource_filename( - "certbot_nginx", "options-ssl-nginx.conf") -"""Path to the nginx mod_ssl config file found in the Certbot -distribution.""" - UPDATED_MOD_SSL_CONF_DIGEST = ".updated-options-ssl-nginx-conf-digest.txt" """Name of the hash of the updated or informed mod_ssl_conf as saved in `IConfig.config_dir`.""" - ALL_SSL_OPTIONS_HASHES = [ '0f81093a1465e3d4eaa8b0c14e77b2a2e93568b0fc1351c2b87893a95f0de87c', '9a7b32c49001fed4cff8ad24353329472a50e86ade1ef9b2b9e43566a619612e', @@ -36,6 +29,18 @@ ALL_SSL_OPTIONS_HASHES = [ '7f95624dd95cf5afc708b9f967ee83a24b8025dc7c8d9df2b556bbc64256b3ff', '394732f2bbe3e5e637c3fb5c6e980a1f1b90b01e2e8d6b7cff41dde16e2a756d', '4b16fec2bcbcd8a2f3296d886f17f9953ffdcc0af54582452ca1e52f5f776f16', + 'c052ffff0ad683f43bffe105f7c606b339536163490930e2632a335c8d191cc4', + '02329eb19930af73c54b3632b3165d84571383b8c8c73361df940cb3894dd426', + '63e2bddebb174a05c9d8a7cf2adf72f7af04349ba59a1a925fe447f73b2f1abf', + '2901debc7ecbc10917edd9084c05464c9c5930b463677571eaf8c94bffd11ae2', + '30baca73ed9a5b0e9a69ea40e30482241d8b1a7343aa79b49dc5d7db0bf53b6c', + '02329eb19930af73c54b3632b3165d84571383b8c8c73361df940cb3894dd426', + '108c4555058a087496a3893aea5d9e1cee0f20a3085d44a52dc1a66522299ac3', + 'd5e021706ecdccc7090111b0ae9a29ef61523e927f020e410caf0a1fd7063981', + 'ef11e3fb17213e74d3e1816cde0ec37b8b95b4167cf21e7b8ff1eaa9c6f918ee', + 'af85f6193808a44789a1d293e6cffa249cad9a21135940800958b8e3c72dbc69', + 'a2a612fd21b02abaa32d9d11ac63d987d6e3054dbfa356de5800eea0d7ce17f3', + '2d9648302e3588a172c318e46bff88ade46fc7a16d6afc85322776a04800d473', ] """SHA256 hashes of the contents of all versions of MOD_SSL_CONF_SRC""" diff --git a/certbot-nginx/certbot_nginx/display_ops.py b/certbot-nginx/certbot_nginx/_internal/display_ops.py similarity index 98% rename from certbot-nginx/certbot_nginx/display_ops.py rename to certbot-nginx/certbot_nginx/_internal/display_ops.py index 5d6bda6b0..bbb47f98a 100644 --- a/certbot-nginx/certbot_nginx/display_ops.py +++ b/certbot-nginx/certbot_nginx/_internal/display_ops.py @@ -4,10 +4,8 @@ import logging import zope.component from certbot import interfaces - import certbot.display.util as display_util - logger = logging.getLogger(__name__) @@ -22,7 +20,7 @@ def select_vhost_multiple(vhosts): return list() tags_list = [vhost.display_repr()+"\n" for vhost in vhosts] # Remove the extra newline from the last entry - if len(tags_list): + if tags_list: tags_list[-1] = tags_list[-1][:-1] code, names = zope.component.getUtility(interfaces.IDisplay).checklist( "Which server blocks would you like to modify?", diff --git a/certbot-nginx/certbot_nginx/http_01.py b/certbot-nginx/certbot_nginx/_internal/http_01.py similarity index 93% rename from certbot-nginx/certbot_nginx/http_01.py rename to certbot-nginx/certbot_nginx/_internal/http_01.py index e46d7b9b9..97b111576 100644 --- a/certbot-nginx/certbot_nginx/http_01.py +++ b/certbot-nginx/certbot_nginx/_internal/http_01.py @@ -1,17 +1,14 @@ """A class that performs HTTP-01 challenges for Nginx""" import logging -import os from acme import challenges - +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module from certbot import errors +from certbot.compat import os from certbot.plugins import common - -from certbot_nginx import obj -from certbot_nginx import nginxparser -from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module - +from certbot_nginx._internal import nginxparser +from certbot_nginx._internal import obj logger = logging.getLogger(__name__) @@ -29,10 +26,10 @@ class NginxHttp01(common.ChallengePerformer): :param list indices: Meant to hold indices of challenges in a larger array. NginxHttp01 is capable of solving many challenges at once which causes an indexing issue within NginxConfigurator - who must return all responses in order. Imagine NginxConfigurator - maintaining state about where all of the http-01 Challenges, - TLS-SNI-01 Challenges belong in the response array. This is an - optional utility. + who must return all responses in order. Imagine + NginxConfigurator maintaining state about where all of the + challenges, possibly of different types, belong in the response + array. This is an optional utility. """ @@ -110,7 +107,7 @@ class NginxHttp01(common.ChallengePerformer): def _default_listen_addresses(self): """Finds addresses for a challenge block to listen on. - :returns: list of :class:`certbot_nginx.obj.Addr` to apply + :returns: list of :class:`certbot_nginx._internal.obj.Addr` to apply :rtype: list """ addresses = [] # type: List[obj.Addr] @@ -204,3 +201,4 @@ class NginxHttp01(common.ChallengePerformer): ' ', '$1', ' ', 'break']] self.configurator.parser.add_server_directives(vhost, rewrite_directive, insert_at_top=True) + return None diff --git a/certbot-nginx/certbot_nginx/nginxparser.py b/certbot-nginx/certbot_nginx/_internal/nginxparser.py similarity index 95% rename from certbot-nginx/certbot_nginx/nginxparser.py rename to certbot-nginx/certbot_nginx/_internal/nginxparser.py index bfb75adcc..4fa1362a0 100644 --- a/certbot-nginx/certbot_nginx/nginxparser.py +++ b/certbot-nginx/certbot_nginx/_internal/nginxparser.py @@ -3,10 +3,18 @@ import copy import logging -from pyparsing import ( - Literal, White, Forward, Group, Optional, OneOrMore, QuotedString, Regex, ZeroOrMore, Combine) -from pyparsing import stringEnd +from pyparsing import Combine +from pyparsing import Forward +from pyparsing import Group +from pyparsing import Literal +from pyparsing import OneOrMore +from pyparsing import Optional +from pyparsing import QuotedString +from pyparsing import Regex from pyparsing import restOfLine +from pyparsing import stringEnd +from pyparsing import White +from pyparsing import ZeroOrMore import six logger = logging.getLogger(__name__) @@ -63,7 +71,6 @@ class RawNginxParser(object): return self.parse().asList() class RawNginxDumper(object): - # pylint: disable=too-few-public-methods """A class that dumps nginx configuration from the provided tree.""" def __init__(self, blocks): self.blocks = blocks @@ -179,12 +186,11 @@ class UnspacedList(list): """ if not isinstance(inbound, list): # str or None - return (inbound, inbound) + return inbound, inbound else: if not hasattr(inbound, "spaced"): inbound = UnspacedList(inbound) - return (inbound, inbound.spaced) - + return inbound, inbound.spaced def insert(self, i, x): item, spaced_item = self._coerce(x) diff --git a/certbot-nginx/certbot_nginx/obj.py b/certbot-nginx/certbot_nginx/_internal/obj.py similarity index 96% rename from certbot-nginx/certbot_nginx/obj.py rename to certbot-nginx/certbot_nginx/_internal/obj.py index 8868fcfad..1a92c8b37 100644 --- a/certbot-nginx/certbot_nginx/obj.py +++ b/certbot-nginx/certbot_nginx/_internal/obj.py @@ -37,7 +37,6 @@ class Addr(common.Addr): CANONICAL_UNSPECIFIED_ADDRESS = UNSPECIFIED_IPV4_ADDRESSES[0] def __init__(self, host, port, ssl, default, ipv6, ipv6only): - # pylint: disable=too-many-arguments super(Addr, self).__init__((host, port)) self.ssl = ssl self.default = default @@ -84,7 +83,7 @@ class Addr(common.Addr): port = tup[2] # The rest of the parts are options; we only care about ssl and default - while len(parts) > 0: + while parts: nextpart = parts.pop() if nextpart == 'ssl': ssl = True @@ -120,9 +119,9 @@ class Addr(common.Addr): def __repr__(self): return "Addr(" + self.__str__() + ")" - def __hash__(self): + def __hash__(self): # pylint: disable=useless-super-delegation # Python 3 requires explicit overridden for __hash__ - # See certbot-apache/certbot_apache/obj.py for more information + # See certbot-apache/certbot_apache/_internal/obj.py for more information return super(Addr, self).__hash__() def super_eq(self, other): @@ -145,7 +144,7 @@ class Addr(common.Addr): return False -class VirtualHost(object): # pylint: disable=too-few-public-methods +class VirtualHost(object): """Represents an Nginx Virtualhost. :ivar str filep: file path of VH @@ -162,7 +161,6 @@ class VirtualHost(object): # pylint: disable=too-few-public-methods """ def __init__(self, filep, addrs, ssl, enabled, names, raw, path): - # pylint: disable=too-many-arguments """Initialize a VH.""" self.filep = filep self.addrs = addrs @@ -224,15 +222,17 @@ class VirtualHost(object): # pylint: disable=too-few-public-methods for a in self.addrs: if a.ipv6: return True + return False def ipv4_enabled(self): """Return true if one or more of the listen directives in vhost are IPv4 only""" - if self.addrs is None or len(self.addrs) == 0: + if not self.addrs: return True for a in self.addrs: if not a.ipv6: return True + return False def display_repr(self): """Return a representation of VHost to be used in dialog""" @@ -250,7 +250,7 @@ def _find_directive(directives, directive_name, match_content=None): """Find a directive of type directive_name in directives. If match_content is given, Searches for `match_content` in the directive arguments. """ - if not directives or isinstance(directives, six.string_types) or len(directives) == 0: + if not directives or isinstance(directives, six.string_types): return None # If match_content is None, just match on directive type. Otherwise, match on diff --git a/certbot-nginx/certbot_nginx/parser.py b/certbot-nginx/certbot_nginx/_internal/parser.py similarity index 93% rename from certbot-nginx/certbot_nginx/parser.py rename to certbot-nginx/certbot_nginx/_internal/parser.py index 622eb8d55..edb77a1c1 100644 --- a/certbot-nginx/certbot_nginx/parser.py +++ b/certbot-nginx/certbot_nginx/_internal/parser.py @@ -3,17 +3,20 @@ import copy import functools import glob import logging -import os -import pyparsing import re +import pyparsing import six +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Tuple # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Union # pylint: disable=unused-import, no-name-in-module from certbot import errors - -from certbot_nginx import obj -from certbot_nginx import nginxparser -from acme.magic_typing import Union, Dict, Set, Any, List, Tuple # pylint: disable=unused-import, no-name-in-module +from certbot.compat import os +from certbot_nginx._internal import nginxparser +from certbot_nginx._internal import obj logger = logging.getLogger(__name__) @@ -52,6 +55,7 @@ class NginxParser(object): :param str filepath: The path to the files to parse, as a glob """ + # pylint: disable=too-many-nested-blocks filepath = self.abs_path(filepath) trees = self._parse_files(filepath) for tree in trees: @@ -81,9 +85,8 @@ class NginxParser(object): """ if not os.path.isabs(path): - return os.path.join(self.root, path) - else: - return path + return os.path.normpath(os.path.join(self.root, path)) + return os.path.normpath(path) def _build_addr_to_ssl(self): """Builds a map from address to whether it listens on ssl in any server block @@ -129,7 +132,7 @@ class NginxParser(object): Technically this is a misnomer because Nginx does not have virtual hosts, it has 'server blocks'. - :returns: List of :class:`~certbot_nginx.obj.VirtualHost` + :returns: List of :class:`~certbot_nginx._internal.obj.VirtualHost` objects found in configuration :rtype: list @@ -262,7 +265,7 @@ class NginxParser(object): def has_ssl_on_directive(self, vhost): """Does vhost have ssl on for all ports? - :param :class:`~certbot_nginx.obj.VirtualHost` vhost: The vhost in question + :param :class:`~certbot_nginx._internal.obj.VirtualHost` vhost: The vhost in question :returns: True if 'ssl on' directive is included :rtype: bool @@ -272,7 +275,7 @@ class NginxParser(object): for directive in server: if not directive: continue - elif _is_ssl_on_directive(directive): + if _is_ssl_on_directive(directive): return True return False @@ -288,7 +291,7 @@ class NginxParser(object): ..todo :: Doesn't match server blocks whose server_name directives are split across multiple conf files. - :param :class:`~certbot_nginx.obj.VirtualHost` vhost: The vhost + :param :class:`~certbot_nginx._internal.obj.VirtualHost` vhost: The vhost whose information we use to match on :param list directives: The directives to add :param bool insert_at_top: True if the directives need to be inserted at the top @@ -310,7 +313,7 @@ class NginxParser(object): ..todo :: Doesn't match server blocks whose server_name directives are split across multiple conf files. - :param :class:`~certbot_nginx.obj.VirtualHost` vhost: The vhost + :param :class:`~certbot_nginx._internal.obj.VirtualHost` vhost: The vhost whose information we use to match on :param list directives: The directives to add :param bool insert_at_top: True if the directives need to be inserted at the top @@ -323,7 +326,7 @@ class NginxParser(object): def remove_server_directives(self, vhost, directive_name, match_func=None): """Remove all directives of type directive_name. - :param :class:`~certbot_nginx.obj.VirtualHost` vhost: The vhost + :param :class:`~certbot_nginx._internal.obj.VirtualHost` vhost: The vhost to remove directives from :param string directive_name: The directive type to remove :param callable match_func: Function of the directive that returns true for directives @@ -359,7 +362,7 @@ class NginxParser(object): only_directives=None): """Duplicate the vhost in the configuration files. - :param :class:`~certbot_nginx.obj.VirtualHost` vhost_template: The vhost + :param :class:`~certbot_nginx._internal.obj.VirtualHost` vhost_template: The vhost whose information we copy :param bool remove_singleton_listen_params: If we should remove parameters from listen directives in the block that can only be used once per address @@ -367,7 +370,7 @@ class NginxParser(object): looks at first level of depth; does not expand includes. :returns: A vhost object for the newly created vhost - :rtype: :class:`~certbot_nginx.obj.VirtualHost` + :rtype: :class:`~certbot_nginx._internal.obj.VirtualHost` """ # TODO: https://github.com/certbot/certbot/issues/5185 # put it in the same file as the template, at the same level @@ -381,7 +384,7 @@ class NginxParser(object): if only_directives is not None: new_directives = nginxparser.UnspacedList([]) for directive in raw_in_parsed[1]: - if len(directive) > 0 and directive[0] in only_directives: + if directive and directive[0] in only_directives: new_directives.append(directive) raw_in_parsed[1] = new_directives @@ -394,7 +397,7 @@ class NginxParser(object): addr.default = False addr.ipv6only = False for directive in enclosing_block[new_vhost.path[-1]][1]: - if len(directive) > 0 and directive[0] == 'listen': + if directive and directive[0] == 'listen': # Exclude one-time use parameters which will cause an error if repeated. # https://nginx.org/en/docs/http/ngx_http_core_module.html#listen exclude = set(('default_server', 'default', 'setfib', 'fastopen', 'backlog', @@ -465,19 +468,19 @@ def get_best_match(target_name, names): elif _regex_match(target_name, name): regex.append(name) - if len(exact) > 0: + if exact: # There can be more than one exact match; e.g. eff.org, .eff.org match = min(exact, key=len) return ('exact', match) - if len(wildcard_start) > 0: + if wildcard_start: # Return the longest wildcard match = max(wildcard_start, key=len) return ('wildcard_start', match) - if len(wildcard_end) > 0: + if wildcard_end: # Return the longest wildcard match = max(wildcard_end, key=len) return ('wildcard_end', match) - if len(regex) > 0: + if regex: # Just return the first one for now match = regex[0] return ('regex', match) @@ -486,7 +489,7 @@ def get_best_match(target_name, names): def _exact_match(target_name, name): - return target_name == name or '.' + target_name == name + return name in (target_name, '.' + target_name) def _wildcard_match(target_name, name, start): @@ -504,7 +507,7 @@ def _wildcard_match(target_name, name, start): # The first part must be a wildcard or blank, e.g. '.eff.org' first = match_parts.pop(0) - if first != '*' and first != '': + if first not in ('*', ''): return False target_name = '.'.join(parts) @@ -522,10 +525,7 @@ def _regex_match(target_name, name): # After tilde is a perl-compatible regex try: regex = re.compile(name[1:]) - if re.match(regex, target_name): - return True - else: - return False + return re.match(regex, target_name) except re.error: # pragma: no cover # perl-compatible regexes are sometimes not recognized by python return False @@ -585,7 +585,7 @@ def comment_directive(block, location): if isinstance(next_entry, list) and next_entry: if len(next_entry) >= 2 and next_entry[-2] == "#" and COMMENT in next_entry[-1]: return - elif isinstance(next_entry, nginxparser.UnspacedList): + if isinstance(next_entry, nginxparser.UnspacedList): next_entry = next_entry.spaced[0] else: next_entry = next_entry[0] @@ -661,13 +661,12 @@ def _add_directive(block, directive, insert_at_top): for included_directive in included_directives: included_dir_loc = _find_location(block, included_directive[0]) included_dir_name = included_directive[0] - if not _is_whitespace_or_comment(included_directive) \ - and not can_append(included_dir_loc, included_dir_name): + if (not _is_whitespace_or_comment(included_directive) + and not can_append(included_dir_loc, included_dir_name)): if block[included_dir_loc] != included_directive: raise errors.MisconfigurationError(err_fmt.format(included_directive, block[included_dir_loc])) - else: - _comment_out_directive(block, included_dir_loc, directive[1]) + _comment_out_directive(block, included_dir_loc, directive[1]) if can_append(location, directive_name): if insert_at_top: diff --git a/certbot-nginx/certbot_nginx/parser_obj.py b/certbot-nginx/certbot_nginx/_internal/parser_obj.py similarity index 92% rename from certbot-nginx/certbot_nginx/parser_obj.py rename to certbot-nginx/certbot_nginx/_internal/parser_obj.py index f01cb2fd3..e03913887 100644 --- a/certbot-nginx/certbot_nginx/parser_obj.py +++ b/certbot-nginx/certbot_nginx/_internal/parser_obj.py @@ -3,16 +3,17 @@ raw lists of tokens from pyparsing. """ import abc import logging + import six +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module from certbot import errors -from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module - logger = logging.getLogger(__name__) COMMENT = " managed by Certbot" COMMENT_BLOCK = ["#", COMMENT] + class Parsable(object): """ Abstract base class for "Parsable" objects whose underlying representation is a tree of lists. @@ -112,6 +113,7 @@ class Parsable(object): """ return [elem.dump(include_spaces) for elem in self._data] + class Statements(Parsable): """ A group or list of "Statements". A Statement is either a Block or a Sentence. @@ -142,24 +144,23 @@ class Statements(Parsable): if self.parent is not None: self._trailing_whitespace = "\n" + self.parent.get_tabs() - def parse(self, parse_this, add_spaces=False): + def parse(self, raw_list, add_spaces=False): """ Parses a list of statements. - Expects all elements in `parse_this` to be parseable by `type(self).parsing_hooks`, - with an optional whitespace string at the last index of `parse_this`. + Expects all elements in `raw_list` to be parseable by `type(self).parsing_hooks`, + with an optional whitespace string at the last index of `raw_list`. """ - if not isinstance(parse_this, list): + if not isinstance(raw_list, list): raise errors.MisconfigurationError("Statements parsing expects a list!") # If there's a trailing whitespace in the list of statements, keep track of it. - if len(parse_this) > 0 and isinstance(parse_this[-1], six.string_types) \ - and parse_this[-1].isspace(): - self._trailing_whitespace = parse_this[-1] - parse_this = parse_this[:-1] - self._data = [parse_raw(elem, self, add_spaces) for elem in parse_this] + if raw_list and isinstance(raw_list[-1], six.string_types) and raw_list[-1].isspace(): + self._trailing_whitespace = raw_list[-1] + raw_list = raw_list[:-1] + self._data = [parse_raw(elem, self, add_spaces) for elem in raw_list] def get_tabs(self): """ Takes a guess at the tabbing of all contained Statements by retrieving the tabbing of the first Statement.""" - if len(self._data) > 0: + if self._data: return self._data[0].get_tabs() return "" @@ -179,6 +180,7 @@ class Statements(Parsable): # ======== End overridden functions + def _space_list(list_): """ Inserts whitespace between adjacent non-whitespace tokens. """ spaced_statement = [] # type: List[str] @@ -188,6 +190,7 @@ def _space_list(list_): spaced_statement.insert(0, " ") return spaced_statement + class Sentence(Parsable): """ A list of words. Non-whitespace words are typically separated with whitespace tokens. """ @@ -205,15 +208,15 @@ class Sentence(Parsable): return isinstance(lists, list) and len(lists) > 0 and \ all([isinstance(elem, six.string_types) for elem in lists]) - def parse(self, parse_this, add_spaces=False): + def parse(self, raw_list, add_spaces=False): """ Parses a list of string types into this object. If add_spaces is set, adds whitespace tokens between adjacent non-whitespace tokens.""" if add_spaces: - parse_this = _space_list(parse_this) - if not isinstance(parse_this, list) or \ - any([not isinstance(elem, six.string_types) for elem in parse_this]): + raw_list = _space_list(raw_list) + if not isinstance(raw_list, list) or \ + any([not isinstance(elem, six.string_types) for elem in raw_list]): raise errors.MisconfigurationError("Sentence parsing expects a list of string types.") - self._data = parse_this + self._data = raw_list def iterate(self, expanded=False, match=None): """ Simply yields itself. """ @@ -255,6 +258,7 @@ class Sentence(Parsable): def __contains__(self, word): return word in self.words + class Block(Parsable): """ Any sort of bloc, denoted by a block name and curly braces, like so: The parsed block: @@ -297,26 +301,26 @@ class Block(Parsable): for elem in self.contents.iterate(expanded, match): yield elem - def parse(self, parse_this, add_spaces=False): + def parse(self, raw_list, add_spaces=False): """ Parses a list that resembles a block. The assumptions that this routine makes are: - 1. the first element of `parse_this` is a valid Sentence. - 2. the second element of `parse_this` is a valid Statement. + 1. the first element of `raw_list` is a valid Sentence. + 2. the second element of `raw_list` is a valid Statement. If add_spaces is set, we call it recursively on `names` and `contents`, and add an extra trailing space to `names` (to separate the block's opening bracket and the block name). """ - if not Block.should_parse(parse_this): + if not Block.should_parse(raw_list): raise errors.MisconfigurationError("Block parsing expects a list of length 2. " "First element should be a list of string types (the bloc names), " "and second should be another list of statements (the bloc content).") self.names = Sentence(self) if add_spaces: - parse_this[0].append(" ") - self.names.parse(parse_this[0], add_spaces) + raw_list[0].append(" ") + self.names.parse(raw_list[0], add_spaces) self.contents = Statements(self) - self.contents.parse(parse_this[1], add_spaces) + self.contents.parse(raw_list[1], add_spaces) self._data = [self.names, self.contents] def get_tabs(self): diff --git a/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx-old.conf b/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx-old.conf new file mode 100644 index 000000000..a678b0507 --- /dev/null +++ b/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx-old.conf @@ -0,0 +1,13 @@ +# This file contains important security parameters. If you modify this file +# manually, Certbot will be unable to automatically provide future security +# updates. Instead, Certbot will print and log an error message with a path to +# the up-to-date file that you will need to refer to when manually updating +# this file. + +ssl_session_cache shared:le_nginx_SSL:10m; +ssl_session_timeout 1440m; + +ssl_protocols TLSv1.2; +ssl_prefer_server_ciphers off; + +ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"; diff --git a/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx-tls12-only.conf b/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx-tls12-only.conf new file mode 100644 index 000000000..1933cbc4f --- /dev/null +++ b/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx-tls12-only.conf @@ -0,0 +1,14 @@ +# This file contains important security parameters. If you modify this file +# manually, Certbot will be unable to automatically provide future security +# updates. Instead, Certbot will print and log an error message with a path to +# the up-to-date file that you will need to refer to when manually updating +# this file. + +ssl_session_cache shared:le_nginx_SSL:10m; +ssl_session_timeout 1440m; +ssl_session_tickets off; + +ssl_protocols TLSv1.2; +ssl_prefer_server_ciphers off; + +ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"; diff --git a/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx-tls13-session-tix-on.conf b/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx-tls13-session-tix-on.conf new file mode 100644 index 000000000..52fdfde24 --- /dev/null +++ b/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx-tls13-session-tix-on.conf @@ -0,0 +1,13 @@ +# This file contains important security parameters. If you modify this file +# manually, Certbot will be unable to automatically provide future security +# updates. Instead, Certbot will print and log an error message with a path to +# the up-to-date file that you will need to refer to when manually updating +# this file. + +ssl_session_cache shared:le_nginx_SSL:10m; +ssl_session_timeout 1440m; + +ssl_protocols TLSv1.2 TLSv1.3; +ssl_prefer_server_ciphers off; + +ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"; diff --git a/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf b/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf new file mode 100644 index 000000000..978e6e8ab --- /dev/null +++ b/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf @@ -0,0 +1,14 @@ +# This file contains important security parameters. If you modify this file +# manually, Certbot will be unable to automatically provide future security +# updates. Instead, Certbot will print and log an error message with a path to +# the up-to-date file that you will need to refer to when manually updating +# this file. + +ssl_session_cache shared:le_nginx_SSL:10m; +ssl_session_timeout 1440m; +ssl_session_tickets off; + +ssl_protocols TLSv1.2 TLSv1.3; +ssl_prefer_server_ciphers off; + +ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"; diff --git a/certbot-nginx/certbot_nginx/options-ssl-nginx.conf b/certbot-nginx/certbot_nginx/options-ssl-nginx.conf deleted file mode 100644 index 292d42984..000000000 --- a/certbot-nginx/certbot_nginx/options-ssl-nginx.conf +++ /dev/null @@ -1,13 +0,0 @@ -# This file contains important security parameters. If you modify this file -# manually, Certbot will be unable to automatically provide future security -# updates. Instead, Certbot will print and log an error message with a path to -# the up-to-date file that you will need to refer to when manually updating -# this file. - -ssl_session_cache shared:le_nginx_SSL:1m; -ssl_session_timeout 1440m; - -ssl_protocols TLSv1 TLSv1.1 TLSv1.2; -ssl_prefer_server_ciphers on; - -ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS"; diff --git a/certbot-nginx/certbot_nginx/tests/__init__.py b/certbot-nginx/certbot_nginx/tests/__init__.py deleted file mode 100644 index 32ca193d9..000000000 --- a/certbot-nginx/certbot_nginx/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Certbot Nginx Tests""" diff --git a/certbot-nginx/certbot_nginx/tests/tls_sni_01_test.py b/certbot-nginx/certbot_nginx/tests/tls_sni_01_test.py deleted file mode 100644 index 72b65911c..000000000 --- a/certbot-nginx/certbot_nginx/tests/tls_sni_01_test.py +++ /dev/null @@ -1,164 +0,0 @@ -"""Tests for certbot_nginx.tls_sni_01""" -import unittest -import shutil - -import mock -import six - -from acme import challenges - -from certbot import achallenges -from certbot import errors - -from certbot.plugins import common_test -from certbot.tests import acme_util - -from certbot_nginx import obj -from certbot_nginx.tests import util - - -class TlsSniPerformTest(util.NginxTest): - """Test the NginxTlsSni01 challenge.""" - - account_key = common_test.AUTH_KEY - achalls = [ - achallenges.KeyAuthorizationAnnotatedChallenge( - challb=acme_util.chall_to_challb( - challenges.TLSSNI01(token=b"kNdwjwOeX0I_A8DXt9Msmg"), "pending"), - domain="www.example.com", account_key=account_key), - achallenges.KeyAuthorizationAnnotatedChallenge( - challb=acme_util.chall_to_challb( - challenges.TLSSNI01( - token=b"\xba\xa9\xda? 1 else None - else: - key = values = "" - if isinstance(key, list): - new = copy.deepcopy(entry) - new[1] = filter_comments(values) - yield new - else: - if key != '#' and spaceless: - yield spaceless - - return list(traverse(tree)) - - -def contains_at_depth(haystack, needle, n): - """Is the needle in haystack at depth n? - - Return true if the needle is present in one of the sub-iterables in haystack - at depth n. Haystack must be an iterable. - """ - # Specifically use hasattr rather than isinstance(..., collections.Iterable) - # because we want to include lists but reject strings. - if not hasattr(haystack, '__iter__') or hasattr(haystack, 'strip'): - return False - if n == 0: - return needle in haystack - else: - for item in haystack: - if contains_at_depth(item, needle, n - 1): - return True - return False diff --git a/certbot-nginx/certbot_nginx/tls_sni_01.py b/certbot-nginx/certbot_nginx/tls_sni_01.py deleted file mode 100644 index 60ec1ed1a..000000000 --- a/certbot-nginx/certbot_nginx/tls_sni_01.py +++ /dev/null @@ -1,177 +0,0 @@ -"""A class that performs TLS-SNI-01 challenges for Nginx""" - -import logging -import os - -import six - -from certbot import errors -from certbot.plugins import common - -from certbot_nginx import obj -from certbot_nginx import nginxparser - - -logger = logging.getLogger(__name__) - - -class NginxTlsSni01(common.TLSSNI01): - """TLS-SNI-01 authenticator for Nginx - - :ivar configurator: NginxConfigurator object - :type configurator: :class:`~nginx.configurator.NginxConfigurator` - - :ivar list achalls: Annotated - class:`~certbot.achallenges.KeyAuthorizationAnnotatedChallenge` - challenges - - :param list indices: Meant to hold indices of challenges in a - larger array. NginxTlsSni01 is capable of solving many challenges - at once which causes an indexing issue within NginxConfigurator - who must return all responses in order. Imagine NginxConfigurator - maintaining state about where all of the http-01 Challenges, - TLS-SNI-01 Challenges belong in the response array. This is an - optional utility. - - :param str challenge_conf: location of the challenge config file - - """ - - def perform(self): - """Perform a challenge on Nginx. - - :returns: list of :class:`certbot.acme.challenges.TLSSNI01Response` - :rtype: list - - """ - if not self.achalls: - return [] - - addresses = [] - default_addr = "{0} ssl".format( - self.configurator.config.tls_sni_01_port) - - for achall in self.achalls: - vhosts = self.configurator.choose_vhosts(achall.domain, create_if_no_match=True) - - # len is max 1 because Nginx doesn't authenticate wildcards - if vhosts and vhosts[0].addrs: - addresses.append(list(vhosts[0].addrs)) - else: - # choose_vhosts might have modified vhosts, so put this after - ipv6, ipv6only = self.configurator.ipv6_info( - self.configurator.config.tls_sni_01_port) - if ipv6: - # If IPv6 is active in Nginx configuration - ipv6_addr = "[::]:{0} ssl".format( - self.configurator.config.tls_sni_01_port) - if not ipv6only: - # If ipv6only=on is not already present in the config - ipv6_addr = ipv6_addr + " ipv6only=on" - addresses.append([obj.Addr.fromstring(default_addr), - obj.Addr.fromstring(ipv6_addr)]) - logger.info(("Using default addresses %s and %s for " + - "TLSSNI01 authentication."), - default_addr, - ipv6_addr) - else: - addresses.append([obj.Addr.fromstring(default_addr)]) - logger.info("Using default address %s for TLSSNI01 authentication.", - default_addr) - - # Create challenge certs - responses = [self._setup_challenge_cert(x) for x in self.achalls] - - # Set up the configuration - self._mod_config(addresses) - - # Save reversible changes - self.configurator.save("SNI Challenge", True) - - return responses - - def _mod_config(self, ll_addrs): - """Modifies Nginx config to include challenge server blocks. - - :param list ll_addrs: list of lists of - :class:`certbot_nginx.obj.Addr` to apply - - :raises .MisconfigurationError: - Unable to find a suitable HTTP block in which to include - authenticator hosts. - - """ - # Add the 'include' statement for the challenges if it doesn't exist - # already in the main config - included = False - include_directive = ['\n', 'include', ' ', self.challenge_conf] - root = self.configurator.parser.config_root - - bucket_directive = ['\n', 'server_names_hash_bucket_size', ' ', '128'] - - main = self.configurator.parser.parsed[root] - for line in main: - if line[0] == ['http']: - body = line[1] - found_bucket = False - posn = 0 - for inner_line in body: - if inner_line[0] == bucket_directive[1]: - if int(inner_line[1]) < int(bucket_directive[3]): - body[posn] = bucket_directive - found_bucket = True - posn += 1 - if not found_bucket: - body.insert(0, bucket_directive) - if include_directive not in body: - body.insert(0, include_directive) - included = True - break - if not included: - raise errors.MisconfigurationError( - 'Certbot could not find an HTTP block to include ' - 'TLS-SNI-01 challenges in %s.' % root) - config = [self._make_server_block(pair[0], pair[1]) - for pair in six.moves.zip(self.achalls, ll_addrs)] - config = nginxparser.UnspacedList(config) - - self.configurator.reverter.register_file_creation( - True, self.challenge_conf) - - with open(self.challenge_conf, "w") as new_conf: - nginxparser.dump(config, new_conf) - - logger.debug("Generated server block:\n%s", str(config)) - - def _make_server_block(self, achall, addrs): - """Creates a server block for a challenge. - - :param achall: Annotated TLS-SNI-01 challenge - :type achall: - :class:`certbot.achallenges.KeyAuthorizationAnnotatedChallenge` - - :param list addrs: addresses of challenged domain - :class:`list` of type :class:`~nginx.obj.Addr` - - :returns: server block for the challenge host - :rtype: list - - """ - document_root = os.path.join( - self.configurator.config.work_dir, "tls_sni_01_page") - - block = [['listen', ' ', addr.to_string(include_default=False)] for addr in addrs] - - block.extend([['server_name', ' ', - achall.response(achall.account_key).z_domain.decode('ascii')], - # access and error logs necessary for - # integration testing (non-root) - ['access_log', ' ', os.path.join( - self.configurator.config.work_dir, 'access.log')], - ['error_log', ' ', os.path.join( - self.configurator.config.work_dir, 'error.log')], - ['ssl_certificate', ' ', self.get_cert_path(achall)], - ['ssl_certificate_key', ' ', self.get_key_path(achall)], - ['include', ' ', self.configurator.mod_ssl_conf], - [['location', ' ', '/'], [['root', ' ', document_root]]]]) - return [['server'], block] diff --git a/certbot-nginx/docs/.gitignore b/certbot-nginx/docs/.gitignore deleted file mode 100644 index ba65b13af..000000000 --- a/certbot-nginx/docs/.gitignore +++ /dev/null @@ -1 +0,0 @@ -/_build/ diff --git a/certbot-nginx/docs/Makefile b/certbot-nginx/docs/Makefile deleted file mode 100644 index 0bd88a347..000000000 --- a/certbot-nginx/docs/Makefile +++ /dev/null @@ -1,192 +0,0 @@ -# Makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -PAPER = -BUILDDIR = _build - -# User-friendly check for sphinx-build -ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) -$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) -endif - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . -# the i18n builder cannot share the environment and doctrees with the others -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext - -help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " singlehtml to make a single large HTML file" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " applehelp to make an Apple Help Book" - @echo " devhelp to make HTML files and a Devhelp project" - @echo " epub to make an epub" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " latexpdf to make LaTeX files and run them through pdflatex" - @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" - @echo " text to make text files" - @echo " man to make manual pages" - @echo " texinfo to make Texinfo files" - @echo " info to make Texinfo files and run them through makeinfo" - @echo " gettext to make PO message catalogs" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " xml to make Docutils-native XML files" - @echo " pseudoxml to make pseudoxml-XML files for display purposes" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - @echo " coverage to run coverage check of the documentation (if enabled)" - -clean: - rm -rf $(BUILDDIR)/* - -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." - -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." - -singlehtml: - $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml - @echo - @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." - -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/certbot-nginx.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/certbot-nginx.qhc" - -applehelp: - $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp - @echo - @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." - @echo "N.B. You won't be able to view it unless you put it in" \ - "~/Library/Documentation/Help or install it in your application" \ - "bundle." - -devhelp: - $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp - @echo - @echo "Build finished." - @echo "To view the help file:" - @echo "# mkdir -p $$HOME/.local/share/devhelp/certbot-nginx" - @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/certbot-nginx" - @echo "# devhelp" - -epub: - $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub - @echo - @echo "Build finished. The epub file is in $(BUILDDIR)/epub." - -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make' in that directory to run these through (pdf)latex" \ - "(use \`make latexpdf' here to do that automatically)." - -latexpdf: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through pdflatex..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -latexpdfja: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through platex and dvipdfmx..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -text: - $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text - @echo - @echo "Build finished. The text files are in $(BUILDDIR)/text." - -man: - $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man - @echo - @echo "Build finished. The manual pages are in $(BUILDDIR)/man." - -texinfo: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo - @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." - @echo "Run \`make' in that directory to run these through makeinfo" \ - "(use \`make info' here to do that automatically)." - -info: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo "Running Texinfo files through makeinfo..." - make -C $(BUILDDIR)/texinfo info - @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." - -gettext: - $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale - @echo - @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." - -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." - -coverage: - $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage - @echo "Testing of coverage in the sources finished, look at the " \ - "results in $(BUILDDIR)/coverage/python.txt." - -xml: - $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml - @echo - @echo "Build finished. The XML files are in $(BUILDDIR)/xml." - -pseudoxml: - $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml - @echo - @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." diff --git a/certbot-nginx/docs/_templates/.gitignore b/certbot-nginx/docs/_templates/.gitignore deleted file mode 100644 index e69de29bb..000000000 diff --git a/certbot-nginx/docs/api.rst b/certbot-nginx/docs/api.rst deleted file mode 100644 index 8668ec5d8..000000000 --- a/certbot-nginx/docs/api.rst +++ /dev/null @@ -1,8 +0,0 @@ -================= -API Documentation -================= - -.. toctree:: - :glob: - - api/** diff --git a/certbot-nginx/docs/api/nginxparser.rst b/certbot-nginx/docs/api/nginxparser.rst deleted file mode 100644 index 6a3be5247..000000000 --- a/certbot-nginx/docs/api/nginxparser.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_nginx.nginxparser` ------------------------------------- - -.. automodule:: certbot_nginx.nginxparser - :members: diff --git a/certbot-nginx/docs/api/obj.rst b/certbot-nginx/docs/api/obj.rst deleted file mode 100644 index a2c94037b..000000000 --- a/certbot-nginx/docs/api/obj.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_nginx.obj` ----------------------------- - -.. automodule:: certbot_nginx.obj - :members: diff --git a/certbot-nginx/docs/api/parser.rst b/certbot-nginx/docs/api/parser.rst deleted file mode 100644 index 0149f99cb..000000000 --- a/certbot-nginx/docs/api/parser.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_nginx.parser` -------------------------------- - -.. automodule:: certbot_nginx.parser - :members: diff --git a/certbot-nginx/docs/api/tls_sni_01.rst b/certbot-nginx/docs/api/tls_sni_01.rst deleted file mode 100644 index 5074f63d9..000000000 --- a/certbot-nginx/docs/api/tls_sni_01.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_nginx.tls_sni_01` ------------------------------------ - -.. automodule:: certbot_nginx.tls_sni_01 - :members: diff --git a/certbot-nginx/docs/conf.py b/certbot-nginx/docs/conf.py deleted file mode 100644 index 167abb4fb..000000000 --- a/certbot-nginx/docs/conf.py +++ /dev/null @@ -1,311 +0,0 @@ -# -*- coding: utf-8 -*- -# -# certbot-nginx documentation build configuration file, created by -# sphinx-quickstart on Sun Oct 18 13:39:39 2015. -# -# This file is execfile()d with the current directory set to its -# containing dir. -# -# Note that not all possible configuration values are present in this -# autogenerated file. -# -# All configuration values have a default; values that are commented out -# serve to show the default. - -import sys -import os -import shlex - - -here = os.path.abspath(os.path.dirname(__file__)) - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -sys.path.insert(0, os.path.abspath(os.path.join(here, '..'))) - -# -- General configuration ------------------------------------------------ - -# If your documentation needs a minimal Sphinx version, state it here. -needs_sphinx = '1.0' - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom -# ones. -extensions = [ - 'sphinx.ext.autodoc', - 'sphinx.ext.intersphinx', - 'sphinx.ext.todo', - 'sphinx.ext.coverage', - 'sphinx.ext.viewcode', -] - -autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# The suffix(es) of source filenames. -# You can specify multiple suffix as a list of string: -# source_suffix = ['.rst', '.md'] -source_suffix = '.rst' - -# The encoding of source files. -#source_encoding = 'utf-8-sig' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'certbot-nginx' -copyright = u'2014-2015, Let\'s Encrypt Project' -author = u'Let\'s Encrypt Project' - -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. -# -# The short X.Y version. -version = '0' -# The full version, including alpha/beta/rc tags. -release = '0' - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -# -# This is also used if you do content translation via gettext catalogs. -# Usually you set "language" from the command line for these cases. -language = 'en' - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -exclude_patterns = ['_build'] - -# The reST default role (used for this markup: `text`) to use for all -# documents. -default_role = 'py:obj' - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] - -# If true, keep warnings as "system message" paragraphs in the built documents. -#keep_warnings = False - -# If true, `todo` and `todoList` produce output, else they produce nothing. -todo_include_todos = True - - -# -- Options for HTML output ---------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. - -# http://docs.readthedocs.org/en/latest/theme.html#how-do-i-use-this-locally-and-on-read-the-docs -# on_rtd is whether we are on readthedocs.org -on_rtd = os.environ.get('READTHEDOCS', None) == 'True' -if not on_rtd: # only import and set the theme if we're building docs locally - import sphinx_rtd_theme - html_theme = 'sphinx_rtd_theme' - html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] -# otherwise, readthedocs.org uses their theme by default, so no need to specify it - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -#html_theme_options = {} - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -#html_title = None - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -#html_logo = None - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -#html_favicon = None - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - -# Add any extra paths that contain custom files (such as robots.txt or -# .htaccess) here, relative to this directory. These files are copied -# directly to the root of the documentation. -#html_extra_path = [] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_domain_indices = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True - -# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True - -# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None - -# Language to be used for generating the HTML full-text search index. -# Sphinx supports the following languages: -# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' -# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' -#html_search_language = 'en' - -# A dictionary with options for the search language support, empty by default. -# Now only 'ja' uses this config value -#html_search_options = {'type': 'default'} - -# The name of a javascript file (relative to the configuration directory) that -# implements a search results scorer. If empty, the default will be used. -#html_search_scorer = 'scorer.js' - -# Output file base name for HTML help builder. -htmlhelp_basename = 'certbot-nginxdoc' - -# -- Options for LaTeX output --------------------------------------------- - -latex_elements = { - # The paper size ('letterpaper' or 'a4paper'). - #'papersize': 'letterpaper', - - # The font size ('10pt', '11pt' or '12pt'). - #'pointsize': '10pt', - - # Additional stuff for the LaTeX preamble. - #'preamble': '', - - # Latex figure (float) alignment - #'figure_align': 'htbp', -} - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, -# author, documentclass [howto, manual, or own class]). -latex_documents = [ - (master_doc, 'certbot-nginx.tex', u'certbot-nginx Documentation', - u'Let\'s Encrypt Project', 'manual'), -] - -# The name of an image file (relative to this directory) to place at the top of -# the title page. -#latex_logo = None - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# If true, show page references after internal links. -#latex_show_pagerefs = False - -# If true, show URL addresses after external links. -#latex_show_urls = False - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_domain_indices = True - - -# -- Options for manual page output --------------------------------------- - -# One entry per manual page. List of tuples -# (source start file, name, description, authors, manual section). -man_pages = [ - (master_doc, 'certbot-nginx', u'certbot-nginx Documentation', - [author], 1) -] - -# If true, show URL addresses after external links. -#man_show_urls = False - - -# -- Options for Texinfo output ------------------------------------------- - -# Grouping the document tree into Texinfo files. List of tuples -# (source start file, target name, title, author, -# dir menu entry, description, category) -texinfo_documents = [ - (master_doc, 'certbot-nginx', u'certbot-nginx Documentation', - author, 'certbot-nginx', 'One line description of project.', - 'Miscellaneous'), -] - -# Documents to append as an appendix to all manuals. -#texinfo_appendices = [] - -# If false, no module index is generated. -#texinfo_domain_indices = True - -# How to display URL addresses: 'footnote', 'no', or 'inline'. -#texinfo_show_urls = 'footnote' - -# If true, do not generate a @detailmenu in the "Top" node's menu. -#texinfo_no_detailmenu = False - - -intersphinx_mapping = { - 'python': ('https://docs.python.org/', None), - 'acme': ('https://acme-python.readthedocs.org/en/latest/', None), - 'certbot': ('https://certbot.eff.org/docs/', None), -} diff --git a/certbot-nginx/docs/index.rst b/certbot-nginx/docs/index.rst deleted file mode 100644 index 488a7ab9c..000000000 --- a/certbot-nginx/docs/index.rst +++ /dev/null @@ -1,31 +0,0 @@ -.. certbot-nginx documentation master file, created by - sphinx-quickstart on Sun Oct 18 13:39:39 2015. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to certbot-nginx's documentation! -============================================= - -Contents: - -.. toctree:: - :maxdepth: 2 - - -.. toctree:: - :maxdepth: 1 - - api - - -.. automodule:: certbot_nginx - :members: - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` - diff --git a/certbot-nginx/docs/make.bat b/certbot-nginx/docs/make.bat deleted file mode 100644 index b12255d4c..000000000 --- a/certbot-nginx/docs/make.bat +++ /dev/null @@ -1,263 +0,0 @@ -@ECHO OFF - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set BUILDDIR=_build -set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . -set I18NSPHINXOPTS=%SPHINXOPTS% . -if NOT "%PAPER%" == "" ( - set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% - set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% -) - -if "%1" == "" goto help - -if "%1" == "help" ( - :help - echo.Please use `make ^` where ^ is one of - echo. html to make standalone HTML files - echo. dirhtml to make HTML files named index.html in directories - echo. singlehtml to make a single large HTML file - echo. pickle to make pickle files - echo. json to make JSON files - echo. htmlhelp to make HTML files and a HTML help project - echo. qthelp to make HTML files and a qthelp project - echo. devhelp to make HTML files and a Devhelp project - echo. epub to make an epub - echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter - echo. text to make text files - echo. man to make manual pages - echo. texinfo to make Texinfo files - echo. gettext to make PO message catalogs - echo. changes to make an overview over all changed/added/deprecated items - echo. xml to make Docutils-native XML files - echo. pseudoxml to make pseudoxml-XML files for display purposes - echo. linkcheck to check all external links for integrity - echo. doctest to run all doctests embedded in the documentation if enabled - echo. coverage to run coverage check of the documentation if enabled - goto end -) - -if "%1" == "clean" ( - for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i - del /q /s %BUILDDIR%\* - goto end -) - - -REM Check if sphinx-build is available and fallback to Python version if any -%SPHINXBUILD% 2> nul -if errorlevel 9009 goto sphinx_python -goto sphinx_ok - -:sphinx_python - -set SPHINXBUILD=python -m sphinx.__init__ -%SPHINXBUILD% 2> nul -if errorlevel 9009 ( - echo. - echo.The 'sphinx-build' command was not found. Make sure you have Sphinx - echo.installed, then set the SPHINXBUILD environment variable to point - echo.to the full path of the 'sphinx-build' executable. Alternatively you - echo.may add the Sphinx directory to PATH. - echo. - echo.If you don't have Sphinx installed, grab it from - echo.http://sphinx-doc.org/ - exit /b 1 -) - -:sphinx_ok - - -if "%1" == "html" ( - %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/html. - goto end -) - -if "%1" == "dirhtml" ( - %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. - goto end -) - -if "%1" == "singlehtml" ( - %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. - goto end -) - -if "%1" == "pickle" ( - %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the pickle files. - goto end -) - -if "%1" == "json" ( - %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the JSON files. - goto end -) - -if "%1" == "htmlhelp" ( - %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run HTML Help Workshop with the ^ -.hhp project file in %BUILDDIR%/htmlhelp. - goto end -) - -if "%1" == "qthelp" ( - %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run "qcollectiongenerator" with the ^ -.qhcp project file in %BUILDDIR%/qthelp, like this: - echo.^> qcollectiongenerator %BUILDDIR%\qthelp\certbot-nginx.qhcp - echo.To view the help file: - echo.^> assistant -collectionFile %BUILDDIR%\qthelp\certbot-nginx.ghc - goto end -) - -if "%1" == "devhelp" ( - %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. - goto end -) - -if "%1" == "epub" ( - %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The epub file is in %BUILDDIR%/epub. - goto end -) - -if "%1" == "latex" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "latexpdf" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - cd %BUILDDIR%/latex - make all-pdf - cd %~dp0 - echo. - echo.Build finished; the PDF files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "latexpdfja" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - cd %BUILDDIR%/latex - make all-pdf-ja - cd %~dp0 - echo. - echo.Build finished; the PDF files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "text" ( - %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The text files are in %BUILDDIR%/text. - goto end -) - -if "%1" == "man" ( - %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The manual pages are in %BUILDDIR%/man. - goto end -) - -if "%1" == "texinfo" ( - %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. - goto end -) - -if "%1" == "gettext" ( - %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The message catalogs are in %BUILDDIR%/locale. - goto end -) - -if "%1" == "changes" ( - %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes - if errorlevel 1 exit /b 1 - echo. - echo.The overview file is in %BUILDDIR%/changes. - goto end -) - -if "%1" == "linkcheck" ( - %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck - if errorlevel 1 exit /b 1 - echo. - echo.Link check complete; look for any errors in the above output ^ -or in %BUILDDIR%/linkcheck/output.txt. - goto end -) - -if "%1" == "doctest" ( - %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest - if errorlevel 1 exit /b 1 - echo. - echo.Testing of doctests in the sources finished, look at the ^ -results in %BUILDDIR%/doctest/output.txt. - goto end -) - -if "%1" == "coverage" ( - %SPHINXBUILD% -b coverage %ALLSPHINXOPTS% %BUILDDIR%/coverage - if errorlevel 1 exit /b 1 - echo. - echo.Testing of coverage in the sources finished, look at the ^ -results in %BUILDDIR%/coverage/python.txt. - goto end -) - -if "%1" == "xml" ( - %SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The XML files are in %BUILDDIR%/xml. - goto end -) - -if "%1" == "pseudoxml" ( - %SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml. - goto end -) - -:end diff --git a/certbot-nginx/local-oldest-requirements.txt b/certbot-nginx/local-oldest-requirements.txt index bcd02d197..cee142934 100644 --- a/certbot-nginx/local-oldest-requirements.txt +++ b/certbot-nginx/local-oldest-requirements.txt @@ -1,2 +1,3 @@ -acme[dev]==0.26.0 -certbot[dev]==0.22.0 +# Remember to update setup.py to match the package versions below. +acme[dev]==1.0.0 +certbot[dev]==1.1.0 diff --git a/certbot-nginx/readthedocs.org.requirements.txt b/certbot-nginx/readthedocs.org.requirements.txt deleted file mode 100644 index ca5f33363..000000000 --- a/certbot-nginx/readthedocs.org.requirements.txt +++ /dev/null @@ -1,12 +0,0 @@ -# readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation -# dependencies), but it allows to specify a requirements.txt file at -# https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) - -# Although ReadTheDocs certainly doesn't need to install the project -# in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead - --e acme --e . --e certbot-nginx[docs] diff --git a/certbot-nginx/setup.py b/certbot-nginx/setup.py index 70e11f62b..3b75a3424 100644 --- a/certbot-nginx/setup.py +++ b/certbot-nginx/setup.py @@ -1,14 +1,16 @@ -from setuptools import setup +import sys + from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand - -version = '0.31.0.dev0' +version = '1.3.0.dev0' # Remember to update local-oldest-requirements.txt when changing the minimum # acme/certbot version. install_requires = [ - 'acme>=0.26.0', - 'certbot>=0.22.0', + 'acme>=1.0.0', + 'certbot>=1.1.0', 'mock', 'PyOpenSSL', 'pyparsing>=1.5.5', # Python3 support; perhaps unnecessary? @@ -16,10 +18,21 @@ install_requires = [ 'zope.interface', ] -docs_extras = [ - 'Sphinx>=1.0', # autodoc_member_order = 'bysource', autodoc_default_flags - 'sphinx_rtd_theme', -] + +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + setup( name='certbot-nginx', @@ -29,7 +42,7 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Environment :: Plugins', @@ -40,10 +53,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -55,13 +68,12 @@ setup( packages=find_packages(), include_package_data=True, install_requires=install_requires, - extras_require={ - 'docs': docs_extras, - }, entry_points={ 'certbot.plugins': [ - 'nginx = certbot_nginx.configurator:NginxConfigurator', + 'nginx = certbot_nginx._internal.configurator:NginxConfigurator', ], }, test_suite='certbot_nginx', + tests_require=["pytest"], + cmdclass={"test": PyTest}, ) diff --git a/certbot-nginx/tests/boulder-integration.conf.sh b/certbot-nginx/tests/boulder-integration.conf.sh index 470eab28e..35cedf5ed 100755 --- a/certbot-nginx/tests/boulder-integration.conf.sh +++ b/certbot-nginx/tests/boulder-integration.conf.sh @@ -3,16 +3,22 @@ # https://www.exratione.com/2014/03/running-nginx-as-a-non-root-user/ # https://github.com/exratione/non-root-nginx/blob/9a77f62e5d5cb9c9026fd62eece76b9514011019/nginx.conf +# USAGE: ./boulder-integration.conf.sh /path/to/root cert.key cert.pem >> nginx.conf + +ROOT=$1 +CERT_KEY_PATH=$2 +CERT_PATH=$3 + cat < $nginx_conf - - killall nginx || true - nginx -c $nginx_root/nginx.conf -} - -certbot_test_nginx () { - certbot_test \ - --authenticator nginx \ - --installer nginx \ - --nginx-server-root $nginx_root \ - "$@" -} - -test_deployment_and_rollback() { - # Arguments: certname - echo | openssl s_client -connect localhost:5001 \ - | openssl x509 -out $root/nginx.pem - diff -q $root/nginx.pem "$root/conf/live/$1/cert.pem" - - certbot_test_nginx rollback --checkpoints 9001 - diff -q <(echo "$original") $nginx_conf -} - -export default_server="default_server" -nginx -v -reload_nginx -certbot_test_nginx --domains nginx.wtf run -test_deployment_and_rollback nginx.wtf -certbot_test_nginx --domains nginx-tls.wtf run --preferred-challenges tls-sni -test_deployment_and_rollback nginx-tls.wtf -certbot_test_nginx --domains nginx2.wtf --preferred-challenges http -test_deployment_and_rollback nginx2.wtf -# Overlapping location block and server-block-level return 301 -certbot_test_nginx --domains nginx3.wtf --preferred-challenges http -test_deployment_and_rollback nginx3.wtf -# No matching server block; default_server exists -certbot_test_nginx --domains nginx4.wtf --preferred-challenges http -test_deployment_and_rollback nginx4.wtf -# No matching server block; default_server does not exist -export default_server="" -reload_nginx -if nginx -c $nginx_root/nginx.conf -T 2>/dev/null | grep "default_server"; then - echo "Failed to remove default_server" - exit 1 -fi -certbot_test_nginx --domains nginx5.wtf --preferred-challenges http -test_deployment_and_rollback nginx5.wtf -# Mutiple domains, mix of matching and not -certbot_test_nginx --domains nginx6.wtf,nginx7.wtf --preferred-challenges http -test_deployment_and_rollback nginx6.wtf - -# note: not reached if anything above fails, hence "killall" at the -# top -nginx -c $nginx_root/nginx.conf -s stop - -coverage report --fail-under 75 --include 'certbot-nginx/*' --show-missing diff --git a/certbot-nginx/certbot_nginx/tests/configurator_test.py b/certbot-nginx/tests/configurator_test.py similarity index 80% rename from certbot-nginx/certbot_nginx/tests/configurator_test.py rename to certbot-nginx/tests/configurator_test.py index 957588e2a..ef5593395 100644 --- a/certbot-nginx/certbot_nginx/tests/configurator_test.py +++ b/certbot-nginx/tests/configurator_test.py @@ -1,7 +1,4 @@ -# pylint: disable=too-many-public-methods -"""Test for certbot_nginx.configurator.""" -import os -import shutil +"""Test for certbot_nginx._internal.configurator.""" import unittest import mock @@ -9,18 +6,16 @@ import OpenSSL from acme import challenges from acme import messages - from certbot import achallenges from certbot import crypto_util from certbot import errors +from certbot.compat import os from certbot.tests import util as certbot_test_util - -from certbot_nginx import constants -from certbot_nginx import obj -from certbot_nginx import parser -from certbot_nginx.configurator import _redirect_block_for_domain -from certbot_nginx.nginxparser import UnspacedList -from certbot_nginx.tests import util +from certbot_nginx._internal import obj +from certbot_nginx._internal import parser +from certbot_nginx._internal.configurator import _redirect_block_for_domain +from certbot_nginx._internal.nginxparser import UnspacedList +import test_util as util class NginxConfiguratorTest(util.NginxTest): @@ -30,16 +25,10 @@ class NginxConfiguratorTest(util.NginxTest): def setUp(self): super(NginxConfiguratorTest, self).setUp() - self.config = util.get_nginx_configurator( + self.config = self.get_nginx_configurator( self.config_path, self.config_dir, self.work_dir, self.logs_dir) - def tearDown(self): - shutil.rmtree(self.temp_dir) - shutil.rmtree(self.config_dir) - shutil.rmtree(self.work_dir) - shutil.rmtree(self.logs_dir) - - @mock.patch("certbot_nginx.configurator.util.exe_exists") + @mock.patch("certbot_nginx._internal.configurator.util.exe_exists") def test_prepare_no_install(self, mock_exe_exists): mock_exe_exists.return_value = False self.assertRaises( @@ -49,8 +38,8 @@ class NginxConfiguratorTest(util.NginxTest): self.assertEqual((1, 6, 2), self.config.version) self.assertEqual(11, len(self.config.parser.parsed)) - @mock.patch("certbot_nginx.configurator.util.exe_exists") - @mock.patch("certbot_nginx.configurator.subprocess.Popen") + @mock.patch("certbot_nginx._internal.configurator.util.exe_exists") + @mock.patch("certbot_nginx._internal.configurator.subprocess.Popen") def test_prepare_initializes_version(self, mock_popen, mock_exe_exists): mock_popen().communicate.return_value = ( "", "\n".join(["nginx version: nginx/1.6.2", @@ -69,11 +58,14 @@ class NginxConfiguratorTest(util.NginxTest): def test_prepare_locked(self): server_root = self.config.conf("server-root") + + from certbot import util as certbot_util + certbot_util._LOCKS[server_root].release() # pylint: disable=protected-access + self.config.config_test = mock.Mock() - os.remove(os.path.join(server_root, ".certbot.lock")) certbot_test_util.lock_and_call(self._test_prepare_locked, server_root) - @mock.patch("certbot_nginx.configurator.util.exe_exists") + @mock.patch("certbot_nginx._internal.configurator.util.exe_exists") def _test_prepare_locked(self, unused_exe_exists): try: self.config.prepare() @@ -84,15 +76,15 @@ class NginxConfiguratorTest(util.NginxTest): else: # pragma: no cover self.fail("Exception wasn't raised!") - @mock.patch("certbot_nginx.configurator.socket.gethostbyaddr") + @mock.patch("certbot_nginx._internal.configurator.socket.gethostbyaddr") def test_get_all_names(self, mock_gethostbyaddr): mock_gethostbyaddr.return_value = ('155.225.50.69.nephoscale.net', [], []) names = self.config.get_all_names() - self.assertEqual(names, set( - ["155.225.50.69.nephoscale.net", "www.example.org", "another.alias", + self.assertEqual(names, { + "155.225.50.69.nephoscale.net", "www.example.org", "another.alias", "migration.com", "summer.com", "geese.com", "sslon.com", "globalssl.com", "globalsslsetssl.com", "ipv6.com", "ipv6ssl.com", - "headers.com"])) + "headers.com"}) def test_supported_enhancements(self): self.assertEqual(['redirect', 'ensure-http-header', 'staple-ocsp'], @@ -103,7 +95,7 @@ class NginxConfiguratorTest(util.NginxTest): errors.PluginError, self.config.enhance, 'myhost', 'unknown_enhancement') def test_get_chall_pref(self): - self.assertEqual([challenges.HTTP01, challenges.TLSSNI01], + self.assertEqual([challenges.HTTP01], self.config.get_chall_pref('myhost')) def test_save(self): @@ -171,6 +163,7 @@ class NginxConfiguratorTest(util.NginxTest): 'abc.www.foo.com': "etc_nginx/foo.conf", 'www.bar.co.uk': "etc_nginx/nginx.conf", 'ipv6.com': "etc_nginx/sites-enabled/ipv6.com"} + conf_path = {key: os.path.normpath(value) for key, value in conf_path.items()} vhost = self.config.choose_vhosts(name)[0] path = os.path.relpath(vhost.filep, self.temp_dir) @@ -223,7 +216,7 @@ class NginxConfiguratorTest(util.NginxTest): "example/chain.pem", None) - @mock.patch('certbot_nginx.parser.NginxParser.update_or_add_server_directives') + @mock.patch('certbot_nginx._internal.parser.NginxParser.update_or_add_server_directives') def test_deploy_cert_raise_on_add_error(self, mock_update_or_add_server_directives): mock_update_or_add_server_directives.side_effect = errors.MisconfigurationError() self.assertRaises( @@ -321,21 +314,13 @@ class NginxConfiguratorTest(util.NginxTest): ]], parsed_migration_conf[0]) - @mock.patch("certbot_nginx.configurator.tls_sni_01.NginxTlsSni01.perform") - @mock.patch("certbot_nginx.configurator.http_01.NginxHttp01.perform") - @mock.patch("certbot_nginx.configurator.NginxConfigurator.restart") - @mock.patch("certbot_nginx.configurator.NginxConfigurator.revert_challenge_config") - def test_perform_and_cleanup(self, mock_revert, mock_restart, mock_http_perform, - mock_tls_perform): + @mock.patch("certbot_nginx._internal.configurator.http_01.NginxHttp01.perform") + @mock.patch("certbot_nginx._internal.configurator.NginxConfigurator.restart") + @mock.patch("certbot_nginx._internal.configurator.NginxConfigurator.revert_challenge_config") + def test_perform_and_cleanup(self, mock_revert, mock_restart, mock_http_perform): # Only tests functionality specific to configurator.perform # Note: As more challenges are offered this will have to be expanded - achall1 = achallenges.KeyAuthorizationAnnotatedChallenge( - challb=messages.ChallengeBody( - chall=challenges.TLSSNI01(token=b"kNdwjwOeX0I_A8DXt9Msmg"), - uri="https://ca.org/chall0_uri", - status=messages.Status("pending"), - ), domain="localhost", account_key=self.rsa512jwk) - achall2 = achallenges.KeyAuthorizationAnnotatedChallenge( + achall = achallenges.KeyAuthorizationAnnotatedChallenge( challb=messages.ChallengeBody( chall=challenges.HTTP01(token=b"m8TdO1qik4JVFtgPPurJmg"), uri="https://ca.org/chall1_uri", @@ -343,24 +328,21 @@ class NginxConfiguratorTest(util.NginxTest): ), domain="example.com", account_key=self.rsa512jwk) expected = [ - achall1.response(self.rsa512jwk), - achall2.response(self.rsa512jwk), + achall.response(self.rsa512jwk), ] - mock_tls_perform.return_value = expected[:1] - mock_http_perform.return_value = expected[1:] - responses = self.config.perform([achall1, achall2]) + mock_http_perform.return_value = expected[:] + responses = self.config.perform([achall]) - self.assertEqual(mock_tls_perform.call_count, 1) self.assertEqual(mock_http_perform.call_count, 1) self.assertEqual(responses, expected) - self.config.cleanup([achall1, achall2]) + self.config.cleanup([achall]) self.assertEqual(0, self.config._chall_out) # pylint: disable=protected-access self.assertEqual(mock_revert.call_count, 1) self.assertEqual(mock_restart.call_count, 2) - @mock.patch("certbot_nginx.configurator.subprocess.Popen") + @mock.patch("certbot_nginx._internal.configurator.subprocess.Popen") def test_get_version(self, mock_popen): mock_popen().communicate.return_value = ( "", "\n".join(["nginx version: nginx/1.4.2", @@ -410,21 +392,83 @@ class NginxConfiguratorTest(util.NginxTest): mock_popen.side_effect = OSError("Can't find program") self.assertRaises(errors.PluginError, self.config.get_version) - @mock.patch("certbot_nginx.configurator.subprocess.Popen") + @mock.patch("certbot_nginx._internal.configurator.subprocess.Popen") + def test_get_openssl_version(self, mock_popen): + # pylint: disable=protected-access + mock_popen().communicate.return_value = ( + "", """ + nginx version: nginx/1.15.5 + built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) + built with OpenSSL 1.0.2g 1 Mar 2016 + TLS SNI support enabled + configure arguments: + """) + self.assertEqual(self.config._get_openssl_version(), "1.0.2g") + + mock_popen().communicate.return_value = ( + "", """ + nginx version: nginx/1.15.5 + built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) + built with OpenSSL 1.0.2-beta1 1 Mar 2016 + TLS SNI support enabled + configure arguments: + """) + self.assertEqual(self.config._get_openssl_version(), "1.0.2-beta1") + + mock_popen().communicate.return_value = ( + "", """ + nginx version: nginx/1.15.5 + built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) + built with OpenSSL 1.0.2 1 Mar 2016 + TLS SNI support enabled + configure arguments: + """) + self.assertEqual(self.config._get_openssl_version(), "1.0.2") + + mock_popen().communicate.return_value = ( + "", """ + nginx version: nginx/1.15.5 + built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) + built with OpenSSL 1.0.2g 1 Mar 2016 (running with OpenSSL 1.0.2a 1 Mar 2016) + TLS SNI support enabled + configure arguments: + """) + self.assertEqual(self.config._get_openssl_version(), "1.0.2a") + + mock_popen().communicate.return_value = ( + "", """ + nginx version: nginx/1.15.5 + built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) + built with LibreSSL 2.2.2 + TLS SNI support enabled + configure arguments: + """) + self.assertEqual(self.config._get_openssl_version(), "") + + mock_popen().communicate.return_value = ( + "", """ + nginx version: nginx/1.15.5 + built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) + TLS SNI support enabled + configure arguments: + """) + self.assertEqual(self.config._get_openssl_version(), "") + + @mock.patch("certbot_nginx._internal.configurator.subprocess.Popen") def test_nginx_restart(self, mock_popen): mocked = mock_popen() mocked.communicate.return_value = ('', '') mocked.returncode = 0 self.config.restart() - @mock.patch("certbot_nginx.configurator.subprocess.Popen") + @mock.patch("certbot_nginx._internal.configurator.subprocess.Popen") def test_nginx_restart_fail(self, mock_popen): mocked = mock_popen() mocked.communicate.return_value = ('', '') mocked.returncode = 1 self.assertRaises(errors.MisconfigurationError, self.config.restart) - @mock.patch("certbot_nginx.configurator.subprocess.Popen") + @mock.patch("certbot_nginx._internal.configurator.subprocess.Popen") def test_no_nginx_start(self, mock_popen): mock_popen.side_effect = OSError("Can't find program") self.assertRaises(errors.MisconfigurationError, self.config.restart) @@ -443,11 +487,6 @@ class NginxConfiguratorTest(util.NginxTest): mock_recovery_routine.side_effect = errors.ReverterError("foo") self.assertRaises(errors.PluginError, self.config.recovery_routine) - @mock.patch("certbot.reverter.Reverter.view_config_changes") - def test_view_config_changes_throws_error_from_reverter(self, mock_view_config_changes): - mock_view_config_changes.side_effect = errors.ReverterError("foo") - self.assertRaises(errors.PluginError, self.config.view_config_changes) - @mock.patch("certbot.reverter.Reverter.rollback_checkpoints") def test_rollback_checkpoints_throws_error_from_reverter(self, mock_rollback_checkpoints): mock_rollback_checkpoints.side_effect = errors.ReverterError("foo") @@ -583,20 +622,20 @@ class NginxConfiguratorTest(util.NginxTest): "ensure-http-header", "Strict-Transport-Security") - @mock.patch('certbot_nginx.obj.VirtualHost.contains_list') + @mock.patch('certbot_nginx._internal.obj.VirtualHost.contains_list') def test_certbot_redirect_exists(self, mock_contains_list): # Test that we add no redirect statement if there is already a # redirect in the block that is managed by certbot # Has a certbot redirect mock_contains_list.return_value = True - with mock.patch("certbot_nginx.configurator.logger") as mock_logger: + with mock.patch("certbot_nginx._internal.configurator.logger") as mock_logger: self.config.enhance("www.example.com", "redirect") self.assertEqual(mock_logger.info.call_args[0][0], "Traffic on port %s already redirecting to ssl in %s") def test_redirect_dont_enhance(self): # Test that we don't accidentally add redirect to ssl-only block - with mock.patch("certbot_nginx.configurator.logger") as mock_logger: + with mock.patch("certbot_nginx._internal.configurator.logger") as mock_logger: self.config.enhance("geese.com", "redirect") self.assertEqual(mock_logger.info.call_args[0][0], 'No matching insecure server blocks listening on port %s found.') @@ -787,7 +826,7 @@ class NginxConfiguratorTest(util.NginxTest): self.assertTrue(util.contains_at_depth(generated_conf, expected, 2)) @mock.patch('certbot.reverter.logger') - @mock.patch('certbot_nginx.parser.NginxParser.load') + @mock.patch('certbot_nginx._internal.parser.NginxParser.load') def test_parser_reload_after_config_changes(self, mock_parser_load, unused_mock_logger): self.config.recovery_routine() self.config.revert_challenge_config() @@ -796,7 +835,7 @@ class NginxConfiguratorTest(util.NginxTest): def test_choose_vhosts_wildcard(self): # pylint: disable=protected-access - mock_path = "certbot_nginx.display_ops.select_vhost_multiple" + mock_path = "certbot_nginx._internal.display_ops.select_vhost_multiple" with mock.patch(mock_path) as mock_select_vhs: vhost = [x for x in self.config.parser.get_vhosts() if 'summer.com' in x.names][0] @@ -812,7 +851,7 @@ class NginxConfiguratorTest(util.NginxTest): def test_choose_vhosts_wildcard_redirect(self): # pylint: disable=protected-access - mock_path = "certbot_nginx.display_ops.select_vhost_multiple" + mock_path = "certbot_nginx._internal.display_ops.select_vhost_multiple" with mock.patch(mock_path) as mock_select_vhs: vhost = [x for x in self.config.parser.get_vhosts() if 'summer.com' in x.names][0] @@ -833,7 +872,7 @@ class NginxConfiguratorTest(util.NginxTest): if 'geese.com' in x.names][0] mock_choose_vhosts.return_value = [vhost] self.config._choose_vhosts_wildcard = mock_choose_vhosts - mock_d = "certbot_nginx.configurator.NginxConfigurator._deploy_cert" + mock_d = "certbot_nginx._internal.configurator.NginxConfigurator._deploy_cert" with mock.patch(mock_d) as mock_dep: self.config.deploy_cert("*.com", "/tmp/path", "/tmp/path", "/tmp/path", "/tmp/path") @@ -841,7 +880,7 @@ class NginxConfiguratorTest(util.NginxTest): self.assertEqual(len(mock_dep.call_args_list), 1) self.assertEqual(vhost, mock_dep.call_args_list[0][0][0]) - @mock.patch("certbot_nginx.display_ops.select_vhost_multiple") + @mock.patch("certbot_nginx._internal.display_ops.select_vhost_multiple") def test_deploy_cert_wildcard_no_vhosts(self, mock_dialog): # pylint: disable=protected-access mock_dialog.return_value = [] @@ -850,7 +889,7 @@ class NginxConfiguratorTest(util.NginxTest): "*.wild.cat", "/tmp/path", "/tmp/path", "/tmp/path", "/tmp/path") - @mock.patch("certbot_nginx.display_ops.select_vhost_multiple") + @mock.patch("certbot_nginx._internal.display_ops.select_vhost_multiple") def test_enhance_wildcard_ocsp_after_install(self, mock_dialog): # pylint: disable=protected-access vhost = [x for x in self.config.parser.get_vhosts() @@ -859,7 +898,7 @@ class NginxConfiguratorTest(util.NginxTest): self.config.enhance("*.com", "staple-ocsp", "example/chain.pem") self.assertFalse(mock_dialog.called) - @mock.patch("certbot_nginx.display_ops.select_vhost_multiple") + @mock.patch("certbot_nginx._internal.display_ops.select_vhost_multiple") def test_enhance_wildcard_redirect_or_ocsp_no_install(self, mock_dialog): vhost = [x for x in self.config.parser.get_vhosts() if 'summer.com' in x.names][0] @@ -867,7 +906,7 @@ class NginxConfiguratorTest(util.NginxTest): self.config.enhance("*.com", "staple-ocsp", "example/chain.pem") self.assertTrue(mock_dialog.called) - @mock.patch("certbot_nginx.display_ops.select_vhost_multiple") + @mock.patch("certbot_nginx._internal.display_ops.select_vhost_multiple") def test_enhance_wildcard_double_redirect(self, mock_dialog): # pylint: disable=protected-access vhost = [x for x in self.config.parser.get_vhosts() @@ -878,7 +917,7 @@ class NginxConfiguratorTest(util.NginxTest): def test_choose_vhosts_wildcard_no_ssl_filter_port(self): # pylint: disable=protected-access - mock_path = "certbot_nginx.display_ops.select_vhost_multiple" + mock_path = "certbot_nginx._internal.display_ops.select_vhost_multiple" with mock.patch(mock_path) as mock_select_vhs: mock_select_vhs.return_value = [] self.config._choose_vhosts_wildcard("*.com", @@ -894,16 +933,15 @@ class InstallSslOptionsConfTest(util.NginxTest): def setUp(self): super(InstallSslOptionsConfTest, self).setUp() - self.config = util.get_nginx_configurator( + self.config = self.get_nginx_configurator( self.config_path, self.config_dir, self.work_dir, self.logs_dir) def _call(self): - from certbot_nginx.configurator import install_ssl_options_conf - install_ssl_options_conf(self.config.mod_ssl_conf, self.config.updated_mod_ssl_conf_digest) + self.config.install_ssl_options_conf(self.config.mod_ssl_conf, + self.config.updated_mod_ssl_conf_digest) def _current_ssl_options_hash(self): - from certbot_nginx.constants import MOD_SSL_CONF_SRC - return crypto_util.sha256sum(MOD_SSL_CONF_SRC) + return crypto_util.sha256sum(self.config.mod_ssl_conf_src) def _assert_current_file(self): self.assertTrue(os.path.isfile(self.config.mod_ssl_conf)) @@ -923,10 +961,29 @@ class InstallSslOptionsConfTest(util.NginxTest): self._call() self._assert_current_file() + def _mock_hash_except_ssl_conf_src(self, fake_hash): + # Write a bad file in place so that update tests fail if no update occurs. + # We're going to pretend this file (the currently installed conf file) + # actually hashes to `fake_hash` for the update tests. + with open(self.config.mod_ssl_conf, "w") as f: + f.write("bogus") + sha256 = crypto_util.sha256sum + def _hash(filename): + return sha256(filename) if filename == self.config.mod_ssl_conf_src else fake_hash + return _hash + def test_prev_file_updates_to_current(self): - from certbot_nginx.constants import ALL_SSL_OPTIONS_HASHES - with mock.patch('certbot.crypto_util.sha256sum') as mock_sha256: - mock_sha256.return_value = ALL_SSL_OPTIONS_HASHES[0] + from certbot_nginx._internal.constants import ALL_SSL_OPTIONS_HASHES + with mock.patch('certbot.crypto_util.sha256sum', + new=self._mock_hash_except_ssl_conf_src(ALL_SSL_OPTIONS_HASHES[0])): + self._call() + self._assert_current_file() + + def test_prev_file_updates_to_current_old_nginx(self): + from certbot_nginx._internal.constants import ALL_SSL_OPTIONS_HASHES + self.config.version = (1, 5, 8) + with mock.patch('certbot.crypto_util.sha256sum', + new=self._mock_hash_except_ssl_conf_src(ALL_SSL_OPTIONS_HASHES[0])): self._call() self._assert_current_file() @@ -937,7 +994,7 @@ class InstallSslOptionsConfTest(util.NginxTest): self._call() self.assertFalse(mock_logger.warning.called) self.assertTrue(os.path.isfile(self.config.mod_ssl_conf)) - self.assertEqual(crypto_util.sha256sum(constants.MOD_SSL_CONF_SRC), + self.assertEqual(crypto_util.sha256sum(self.config.mod_ssl_conf_src), self._current_ssl_options_hash()) self.assertNotEqual(crypto_util.sha256sum(self.config.mod_ssl_conf), self._current_ssl_options_hash()) @@ -952,7 +1009,7 @@ class InstallSslOptionsConfTest(util.NginxTest): self.assertEqual(mock_logger.warning.call_args[0][0], "%s has been manually modified; updated file " "saved to %s. We recommend updating %s for security purposes.") - self.assertEqual(crypto_util.sha256sum(constants.MOD_SSL_CONF_SRC), + self.assertEqual(crypto_util.sha256sum(self.config.mod_ssl_conf_src), self._current_ssl_options_hash()) # only print warning once with mock.patch("certbot.plugins.common.logger") as mock_logger: @@ -960,17 +1017,63 @@ class InstallSslOptionsConfTest(util.NginxTest): self.assertFalse(mock_logger.warning.called) def test_current_file_hash_in_all_hashes(self): - from certbot_nginx.constants import ALL_SSL_OPTIONS_HASHES + from certbot_nginx._internal.constants import ALL_SSL_OPTIONS_HASHES self.assertTrue(self._current_ssl_options_hash() in ALL_SSL_OPTIONS_HASHES, "Constants.ALL_SSL_OPTIONS_HASHES must be appended" " with the sha256 hash of self.config.mod_ssl_conf when it is updated.") + def test_ssl_config_files_hash_in_all_hashes(self): + """ + It is really critical that all TLS Nginx config files have their SHA256 hash registered in + constants.ALL_SSL_OPTIONS_HASHES. Otherwise Certbot will mistakenly assume that the config + file has been manually edited by the user, and will refuse to update it. + This test ensures that all necessary hashes are present. + """ + from certbot_nginx._internal.constants import ALL_SSL_OPTIONS_HASHES + import pkg_resources + all_files = [ + pkg_resources.resource_filename("certbot_nginx", + os.path.join("_internal", "tls_configs", x)) + for x in ("options-ssl-nginx.conf", + "options-ssl-nginx-old.conf", + "options-ssl-nginx-tls12-only.conf") + ] + self.assertTrue(all_files) + for one_file in all_files: + file_hash = crypto_util.sha256sum(one_file) + self.assertTrue(file_hash in ALL_SSL_OPTIONS_HASHES, + "Constants.ALL_SSL_OPTIONS_HASHES must be appended with the sha256 " + "hash of {0} when it is updated.".format(one_file)) + + def test_nginx_version_uses_correct_config(self): + self.config.version = (1, 5, 8) + self.config.openssl_version = "1.0.2g" # shouldn't matter + self.assertEqual(os.path.basename(self.config.mod_ssl_conf_src), + "options-ssl-nginx-old.conf") + self._call() + self._assert_current_file() + self.config.version = (1, 5, 9) + self.config.openssl_version = "1.0.2l" + self.assertEqual(os.path.basename(self.config.mod_ssl_conf_src), + "options-ssl-nginx-tls12-only.conf") + self._call() + self._assert_current_file() + self.config.version = (1, 13, 0) + self.assertEqual(os.path.basename(self.config.mod_ssl_conf_src), + "options-ssl-nginx.conf") + self._call() + self._assert_current_file() + self.config.version = (1, 13, 0) + self.config.openssl_version = "1.0.2k" + self.assertEqual(os.path.basename(self.config.mod_ssl_conf_src), + "options-ssl-nginx-tls13-session-tix-on.conf") + class DetermineDefaultServerRootTest(certbot_test_util.ConfigTestCase): - """Tests for certbot_nginx.configurator._determine_default_server_root.""" + """Tests for certbot_nginx._internal.configurator._determine_default_server_root.""" def _call(self): - from certbot_nginx.configurator import _determine_default_server_root + from certbot_nginx._internal.configurator import _determine_default_server_root return _determine_default_server_root() @mock.patch.dict(os.environ, {"CERTBOT_DOCS": "1"}) @@ -988,7 +1091,7 @@ class DetermineDefaultServerRootTest(certbot_test_util.ConfigTestCase): self.assertIn("/usr/local/etc/nginx", server_root) self.assertIn("/etc/nginx", server_root) else: - self.assertTrue(server_root == "/etc/nginx" or server_root == "/usr/local/etc/nginx") + self.assertTrue(server_root in ("/etc/nginx", "/usr/local/etc/nginx")) if __name__ == "__main__": diff --git a/certbot-nginx/certbot_nginx/tests/display_ops_test.py b/certbot-nginx/tests/display_ops_test.py similarity index 83% rename from certbot-nginx/certbot_nginx/tests/display_ops_test.py rename to certbot-nginx/tests/display_ops_test.py index e3c6fb66b..377255441 100644 --- a/certbot-nginx/certbot_nginx/tests/display_ops_test.py +++ b/certbot-nginx/tests/display_ops_test.py @@ -1,18 +1,15 @@ -"""Test certbot_apache.display_ops.""" +"""Test certbot_nginx._internal.display_ops.""" import unittest from certbot.display import util as display_util - from certbot.tests import util as certbot_util - -from certbot_nginx import parser - -from certbot_nginx.display_ops import select_vhost_multiple -from certbot_nginx.tests import util +from certbot_nginx._internal import parser +from certbot_nginx._internal.display_ops import select_vhost_multiple +import test_util as util class SelectVhostMultiTest(util.NginxTest): - """Tests for certbot_nginx.display_ops.select_vhost_multiple.""" + """Tests for certbot_nginx._internal.display_ops.select_vhost_multiple.""" def setUp(self): super(SelectVhostMultiTest, self).setUp() diff --git a/certbot-nginx/certbot_nginx/tests/http_01_test.py b/certbot-nginx/tests/http_01_test.py similarity index 83% rename from certbot-nginx/certbot_nginx/tests/http_01_test.py rename to certbot-nginx/tests/http_01_test.py index ed3c257ee..6418a8841 100644 --- a/certbot-nginx/certbot_nginx/tests/http_01_test.py +++ b/certbot-nginx/tests/http_01_test.py @@ -1,25 +1,24 @@ -"""Tests for certbot_nginx.http_01""" +"""Tests for certbot_nginx._internal.http_01""" import unittest -import shutil +import josepy as jose import mock import six from acme import challenges - from certbot import achallenges - -from certbot.plugins import common_test from certbot.tests import acme_util +from certbot.tests import util as test_util +from certbot_nginx._internal.obj import Addr +import test_util as util -from certbot_nginx.obj import Addr -from certbot_nginx.tests import util +AUTH_KEY = jose.JWKRSA.load(test_util.load_vector("rsa512_key.pem")) class HttpPerformTest(util.NginxTest): """Test the NginxHttp01 challenge.""" - account_key = common_test.AUTH_KEY + account_key = AUTH_KEY achalls = [ achallenges.KeyAuthorizationAnnotatedChallenge( challb=acme_util.chall_to_challb( @@ -48,22 +47,17 @@ class HttpPerformTest(util.NginxTest): def setUp(self): super(HttpPerformTest, self).setUp() - config = util.get_nginx_configurator( + config = self.get_nginx_configurator( self.config_path, self.config_dir, self.work_dir, self.logs_dir) - from certbot_nginx import http_01 + from certbot_nginx._internal import http_01 self.http01 = http_01.NginxHttp01(config) - def tearDown(self): - shutil.rmtree(self.temp_dir) - shutil.rmtree(self.config_dir) - shutil.rmtree(self.work_dir) - def test_perform0(self): responses = self.http01.perform() self.assertEqual([], responses) - @mock.patch("certbot_nginx.configurator.NginxConfigurator.save") + @mock.patch("certbot_nginx._internal.configurator.NginxConfigurator.save") def test_perform1(self, mock_save): self.http01.add_chall(self.achalls[0]) response = self.achalls[0].response(self.account_key) @@ -79,11 +73,11 @@ class HttpPerformTest(util.NginxTest): self.http01.add_chall(achall) acme_responses.append(achall.response(self.account_key)) - sni_responses = self.http01.perform() + http_responses = self.http01.perform() - self.assertEqual(len(sni_responses), 4) + self.assertEqual(len(http_responses), 4) for i in six.moves.range(4): - self.assertEqual(sni_responses[i], acme_responses[i]) + self.assertEqual(http_responses[i], acme_responses[i]) def test_mod_config(self): self.http01.add_chall(self.achalls[0]) @@ -109,7 +103,7 @@ class HttpPerformTest(util.NginxTest): # self.assertEqual(vhost.addrs, set(v_addr2_print)) # self.assertEqual(vhost.names, set([response.z_domain.decode('ascii')])) - @mock.patch("certbot_nginx.configurator.NginxConfigurator.ipv6_info") + @mock.patch("certbot_nginx._internal.configurator.NginxConfigurator.ipv6_info") def test_default_listen_addresses_no_memoization(self, ipv6_info): # pylint: disable=protected-access ipv6_info.return_value = (True, True) @@ -119,7 +113,7 @@ class HttpPerformTest(util.NginxTest): self.http01._default_listen_addresses() self.assertEqual(ipv6_info.call_count, 2) - @mock.patch("certbot_nginx.configurator.NginxConfigurator.ipv6_info") + @mock.patch("certbot_nginx._internal.configurator.NginxConfigurator.ipv6_info") def test_default_listen_addresses_t_t(self, ipv6_info): # pylint: disable=protected-access ipv6_info.return_value = (True, True) @@ -128,7 +122,7 @@ class HttpPerformTest(util.NginxTest): http_ipv6_addr = Addr.fromstring("[::]:80") self.assertEqual(addrs, [http_addr, http_ipv6_addr]) - @mock.patch("certbot_nginx.configurator.NginxConfigurator.ipv6_info") + @mock.patch("certbot_nginx._internal.configurator.NginxConfigurator.ipv6_info") def test_default_listen_addresses_t_f(self, ipv6_info): # pylint: disable=protected-access ipv6_info.return_value = (True, False) @@ -137,7 +131,7 @@ class HttpPerformTest(util.NginxTest): http_ipv6_addr = Addr.fromstring("[::]:80 ipv6only=on") self.assertEqual(addrs, [http_addr, http_ipv6_addr]) - @mock.patch("certbot_nginx.configurator.NginxConfigurator.ipv6_info") + @mock.patch("certbot_nginx._internal.configurator.NginxConfigurator.ipv6_info") def test_default_listen_addresses_f_f(self, ipv6_info): # pylint: disable=protected-access ipv6_info.return_value = (False, False) diff --git a/certbot-nginx/certbot_nginx/tests/nginxparser_test.py b/certbot-nginx/tests/nginxparser_test.py similarity index 97% rename from certbot-nginx/certbot_nginx/tests/nginxparser_test.py rename to certbot-nginx/tests/nginxparser_test.py index 7fc4576c3..a5212078f 100644 --- a/certbot-nginx/certbot_nginx/tests/nginxparser_test.py +++ b/certbot-nginx/tests/nginxparser_test.py @@ -1,4 +1,4 @@ -"""Test for certbot_nginx.nginxparser.""" +"""Test for certbot_nginx._internal.nginxparser.""" import copy import operator import tempfile @@ -6,10 +6,13 @@ import unittest from pyparsing import ParseException -from certbot_nginx.nginxparser import ( - RawNginxParser, loads, load, dumps, dump, UnspacedList) -from certbot_nginx.tests import util - +from certbot_nginx._internal.nginxparser import dump +from certbot_nginx._internal.nginxparser import dumps +from certbot_nginx._internal.nginxparser import load +from certbot_nginx._internal.nginxparser import loads +from certbot_nginx._internal.nginxparser import RawNginxParser +from certbot_nginx._internal.nginxparser import UnspacedList +import test_util as util FIRST = operator.itemgetter(0) diff --git a/certbot-nginx/certbot_nginx/tests/obj_test.py b/certbot-nginx/tests/obj_test.py similarity index 92% rename from certbot-nginx/certbot_nginx/tests/obj_test.py rename to certbot-nginx/tests/obj_test.py index 9e5853c4a..db808229f 100644 --- a/certbot-nginx/certbot_nginx/tests/obj_test.py +++ b/certbot-nginx/tests/obj_test.py @@ -1,12 +1,12 @@ -"""Test the helper objects in certbot_nginx.obj.""" -import unittest +"""Test the helper objects in certbot_nginx._internal.obj.""" import itertools +import unittest class AddrTest(unittest.TestCase): """Test the Addr class.""" def setUp(self): - from certbot_nginx.obj import Addr + from certbot_nginx._internal.obj import Addr self.addr1 = Addr.fromstring("192.168.1.1") self.addr2 = Addr.fromstring("192.168.1.1:* ssl") self.addr3 = Addr.fromstring("192.168.1.1:80") @@ -71,14 +71,14 @@ class AddrTest(unittest.TestCase): self.assertEqual(self.addr6.to_string(include_default=False), "80") def test_eq(self): - from certbot_nginx.obj import Addr + from certbot_nginx._internal.obj import Addr new_addr1 = Addr.fromstring("192.168.1.1 spdy") self.assertEqual(self.addr1, new_addr1) self.assertNotEqual(self.addr1, self.addr2) self.assertFalse(self.addr1 == 3333) def test_equivalent_any_addresses(self): - from certbot_nginx.obj import Addr + from certbot_nginx._internal.obj import Addr any_addresses = ("0.0.0.0:80 default_server ssl", "80 default_server ssl", "*:80 default_server ssl", @@ -97,7 +97,7 @@ class AddrTest(unittest.TestCase): Addr.fromstring(any_address)) def test_set_inclusion(self): - from certbot_nginx.obj import Addr + from certbot_nginx._internal.obj import Addr set_a = set([self.addr1, self.addr2]) addr1b = Addr.fromstring("192.168.1.1") addr2b = Addr.fromstring("192.168.1.1:* ssl") @@ -109,8 +109,8 @@ class AddrTest(unittest.TestCase): class VirtualHostTest(unittest.TestCase): """Test the VirtualHost class.""" def setUp(self): - from certbot_nginx.obj import VirtualHost - from certbot_nginx.obj import Addr + from certbot_nginx._internal.obj import VirtualHost + from certbot_nginx._internal.obj import Addr raw1 = [ ['listen', '69.50.225.155:9000'], [['if', '($scheme', '!=', '"https") '], @@ -159,8 +159,8 @@ class VirtualHostTest(unittest.TestCase): set(['localhost']), raw_has_hsts, []) def test_eq(self): - from certbot_nginx.obj import Addr - from certbot_nginx.obj import VirtualHost + from certbot_nginx._internal.obj import Addr + from certbot_nginx._internal.obj import VirtualHost vhost1b = VirtualHost( "filep", set([Addr.fromstring("localhost blah")]), False, False, @@ -183,9 +183,9 @@ class VirtualHostTest(unittest.TestCase): self.assertFalse(self.vhost1.has_header('Bogus-Header')) def test_contains_list(self): - from certbot_nginx.obj import VirtualHost - from certbot_nginx.obj import Addr - from certbot_nginx.configurator import _test_block_from_block + from certbot_nginx._internal.obj import VirtualHost + from certbot_nginx._internal.obj import Addr + from certbot_nginx._internal.configurator import _test_block_from_block test_block = [ ['\n ', 'return', ' ', '301', ' ', 'https://$host$request_uri'], ['\n'] diff --git a/certbot-nginx/certbot_nginx/tests/parser_obj_test.py b/certbot-nginx/tests/parser_obj_test.py similarity index 92% rename from certbot-nginx/certbot_nginx/tests/parser_obj_test.py rename to certbot-nginx/tests/parser_obj_test.py index 2217be54f..132f83771 100644 --- a/certbot-nginx/certbot_nginx/tests/parser_obj_test.py +++ b/certbot-nginx/tests/parser_obj_test.py @@ -1,20 +1,22 @@ """ Tests for functions and classes in parser_obj.py """ import unittest + import mock -from certbot_nginx.parser_obj import parse_raw -from certbot_nginx.parser_obj import COMMENT_BLOCK +from certbot_nginx._internal.parser_obj import COMMENT_BLOCK +from certbot_nginx._internal.parser_obj import parse_raw + class CommentHelpersTest(unittest.TestCase): def test_is_comment(self): - from certbot_nginx.parser_obj import _is_comment + from certbot_nginx._internal.parser_obj import _is_comment self.assertTrue(_is_comment(parse_raw(['#']))) self.assertTrue(_is_comment(parse_raw(['#', ' literally anything else']))) self.assertFalse(_is_comment(parse_raw(['not', 'even', 'a', 'comment']))) def test_is_certbot_comment(self): - from certbot_nginx.parser_obj import _is_certbot_comment + from certbot_nginx._internal.parser_obj import _is_certbot_comment self.assertTrue(_is_certbot_comment( parse_raw(COMMENT_BLOCK))) self.assertFalse(_is_certbot_comment( @@ -25,7 +27,7 @@ class CommentHelpersTest(unittest.TestCase): parse_raw(['not', 'even', 'a', 'comment']))) def test_certbot_comment(self): - from certbot_nginx.parser_obj import _certbot_comment, _is_certbot_comment + from certbot_nginx._internal.parser_obj import _certbot_comment, _is_certbot_comment comment = _certbot_comment(None) self.assertTrue(_is_certbot_comment(comment)) self.assertEqual(comment.dump(), COMMENT_BLOCK) @@ -35,7 +37,7 @@ class CommentHelpersTest(unittest.TestCase): class ParsingHooksTest(unittest.TestCase): def test_is_sentence(self): - from certbot_nginx.parser_obj import Sentence + from certbot_nginx._internal.parser_obj import Sentence self.assertFalse(Sentence.should_parse([])) self.assertTrue(Sentence.should_parse([''])) self.assertTrue(Sentence.should_parse(['word'])) @@ -44,7 +46,7 @@ class ParsingHooksTest(unittest.TestCase): self.assertFalse(Sentence.should_parse(['word', []])) def test_is_block(self): - from certbot_nginx.parser_obj import Block + from certbot_nginx._internal.parser_obj import Block self.assertFalse(Block.should_parse([])) self.assertFalse(Block.should_parse([''])) self.assertFalse(Block.should_parse(['two', 'words'])) @@ -71,7 +73,7 @@ class ParsingHooksTest(unittest.TestCase): fake_parser1.not_called() fake_parser2.called_once() - @mock.patch("certbot_nginx.parser_obj.Parsable.parsing_hooks") + @mock.patch("certbot_nginx._internal.parser_obj.Parsable.parsing_hooks") def test_parse_raw_no_match(self, parsing_hooks): from certbot import errors fake_parser1 = mock.Mock() @@ -91,7 +93,7 @@ class ParsingHooksTest(unittest.TestCase): class SentenceTest(unittest.TestCase): def setUp(self): - from certbot_nginx.parser_obj import Sentence + from certbot_nginx._internal.parser_obj import Sentence self.sentence = Sentence(None) def test_parse_bad_sentence_raises_error(self): @@ -137,7 +139,7 @@ class SentenceTest(unittest.TestCase): class BlockTest(unittest.TestCase): def setUp(self): - from certbot_nginx.parser_obj import Block + from certbot_nginx._internal.parser_obj import Block self.bloc = Block(None) self.name = ['server', 'name'] self.contents = [['thing', '1'], ['thing', '2'], ['another', 'one']] @@ -153,7 +155,7 @@ class BlockTest(unittest.TestCase): def test_iterate_match(self): # can match on contents while expanded - from certbot_nginx.parser_obj import Block, Sentence + from certbot_nginx._internal.parser_obj import Block, Sentence expected = [['thing', '1'], ['thing', '2']] for i, elem in enumerate(self.bloc.iterate(expanded=True, match=lambda x: isinstance(x, Sentence) and 'thing' in x.words)): @@ -192,7 +194,7 @@ class BlockTest(unittest.TestCase): class StatementsTest(unittest.TestCase): def setUp(self): - from certbot_nginx.parser_obj import Statements + from certbot_nginx._internal.parser_obj import Statements self.statements = Statements(None) self.raw = [ ['sentence', 'one'], diff --git a/certbot-nginx/certbot_nginx/tests/parser_test.py b/certbot-nginx/tests/parser_test.py similarity index 90% rename from certbot-nginx/certbot_nginx/tests/parser_test.py rename to certbot-nginx/tests/parser_test.py index f6f28e42b..2f3b260ca 100644 --- a/certbot-nginx/certbot_nginx/tests/parser_test.py +++ b/certbot-nginx/tests/parser_test.py @@ -1,25 +1,21 @@ -"""Tests for certbot_nginx.parser.""" +"""Tests for certbot_nginx._internal.parser.""" import glob -import os import re import shutil import unittest +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module from certbot import errors - -from certbot_nginx import nginxparser -from certbot_nginx import obj -from certbot_nginx import parser -from certbot_nginx.tests import util -from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from certbot.compat import os +from certbot_nginx._internal import nginxparser +from certbot_nginx._internal import obj +from certbot_nginx._internal import parser +import test_util as util -class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods +class NginxParserTest(util.NginxTest): """Nginx Parser Test.""" - def setUp(self): - super(NginxParserTest, self).setUp() - def tearDown(self): shutil.rmtree(self.temp_dir) shutil.rmtree(self.config_dir) @@ -32,8 +28,16 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods self.assertEqual(nparser.root, self.config_path) def test_root_absolute(self): - nparser = parser.NginxParser(os.path.relpath(self.config_path)) - self.assertEqual(nparser.root, self.config_path) + curr_dir = os.getcwd() + try: + # On Windows current directory may be on a different drive than self.tempdir. + # However a relative path between two different drives is invalid. So we move to + # self.tempdir to ensure that we stay on the same drive. + os.chdir(self.temp_dir) + nparser = parser.NginxParser(os.path.relpath(self.config_path)) + self.assertEqual(nparser.root, self.config_path) + finally: + os.chdir(curr_dir) def test_root_no_trailing_slash(self): nparser = parser.NginxParser(self.config_path + os.path.sep) @@ -45,16 +49,16 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods """ nparser = parser.NginxParser(self.config_path) nparser.load() - self.assertEqual(set([nparser.abs_path(x) for x in - ['foo.conf', 'nginx.conf', 'server.conf', - 'sites-enabled/default', - 'sites-enabled/example.com', - 'sites-enabled/headers.com', - 'sites-enabled/migration.com', - 'sites-enabled/sslon.com', - 'sites-enabled/globalssl.com', - 'sites-enabled/ipv6.com', - 'sites-enabled/ipv6ssl.com']]), + self.assertEqual({nparser.abs_path(x) for x in + ['foo.conf', 'nginx.conf', 'server.conf', + 'sites-enabled/default', + 'sites-enabled/example.com', + 'sites-enabled/headers.com', + 'sites-enabled/migration.com', + 'sites-enabled/sslon.com', + 'sites-enabled/globalssl.com', + 'sites-enabled/ipv6.com', + 'sites-enabled/ipv6ssl.com']}, set(nparser.parsed.keys())) self.assertEqual([['server_name', 'somename', 'alias', 'another.alias']], nparser.parsed[nparser.abs_path('server.conf')]) @@ -67,9 +71,15 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods def test_abs_path(self): nparser = parser.NginxParser(self.config_path) - self.assertEqual('/etc/nginx/*', nparser.abs_path('/etc/nginx/*')) - self.assertEqual(os.path.join(self.config_path, 'foo/bar/'), - nparser.abs_path('foo/bar/')) + if os.name != 'nt': + self.assertEqual('/etc/nginx/*', nparser.abs_path('/etc/nginx/*')) + self.assertEqual(os.path.join(self.config_path, 'foo/bar'), + nparser.abs_path('foo/bar')) + else: + self.assertEqual('C:\\etc\\nginx\\*', nparser.abs_path('C:\\etc\\nginx\\*')) + self.assertEqual(os.path.join(self.config_path, 'foo\\bar'), + nparser.abs_path('foo\\bar')) + def test_filedump(self): nparser = parser.NginxParser(self.config_path) @@ -241,7 +251,7 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods [['foo', 'bar'], ['ssl_certificate', '/etc/ssl/cert2.pem']]) nparser.add_server_directives(mock_vhost, [['foo', 'bar']]) - from certbot_nginx.parser import COMMENT + from certbot_nginx._internal.parser import COMMENT self.assertEqual(nparser.parsed[example_com], [[['server'], [['listen', '69.50.225.155:9000'], ['listen', '127.0.0.1'], @@ -276,7 +286,7 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods nparser.add_server_directives(mock_vhost, [['\n ', 'include', ' ', nparser.abs_path('comment_in_file.conf')]]) - from certbot_nginx.parser import COMMENT + from certbot_nginx._internal.parser import COMMENT self.assertEqual(nparser.parsed[example_com], [[['server'], [['listen', '69.50.225.155:9000'], ['listen', '127.0.0.1'], @@ -296,7 +306,7 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods mock_vhost = obj.VirtualHost(filep, None, None, None, target, None, [0]) nparser.update_or_add_server_directives( mock_vhost, [['server_name', 'foobar.com']]) - from certbot_nginx.parser import COMMENT + from certbot_nginx._internal.parser import COMMENT self.assertEqual( nparser.parsed[filep], [[['server'], [['listen', '69.50.225.155:9000'], @@ -355,7 +365,7 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods ["\n", "a", " ", "b", "\n"], ["c", " ", "d"], ["\n", "e", " ", "f"]]) - from certbot_nginx.parser import comment_directive, COMMENT_BLOCK + from certbot_nginx._internal.parser import comment_directive, COMMENT_BLOCK comment_directive(block, 1) comment_directive(block, 0) self.assertEqual(block.spaced, [ @@ -379,7 +389,7 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods ssl_prefer_server_ciphers on; }""") block = server_block[0][1] - from certbot_nginx.parser import _comment_out_directive + from certbot_nginx._internal.parser import _comment_out_directive _comment_out_directive(block, 4, "blah1") _comment_out_directive(block, 5, "blah2") _comment_out_directive(block, 6, "blah3") diff --git a/certbot-nginx/tests/test_util.py b/certbot-nginx/tests/test_util.py new file mode 100644 index 000000000..8dfd18637 --- /dev/null +++ b/certbot-nginx/tests/test_util.py @@ -0,0 +1,130 @@ +"""Common utilities for certbot_nginx.""" +import copy +import shutil +import tempfile + +import josepy as jose +import mock +import pkg_resources +import zope.component + +from certbot import util +from certbot.compat import os +from certbot.plugins import common +from certbot.tests import util as test_util +from certbot_nginx._internal import configurator +from certbot_nginx._internal import nginxparser + + +class NginxTest(test_util.ConfigTestCase): + + def setUp(self): + super(NginxTest, self).setUp() + + self.configuration = self.config + self.config = None + + self.temp_dir, self.config_dir, self.work_dir = common.dir_setup( + "etc_nginx", __name__) + self.logs_dir = tempfile.mkdtemp('logs') + + self.config_path = os.path.join(self.temp_dir, "etc_nginx") + + self.rsa512jwk = jose.JWKRSA.load(test_util.load_vector( + "rsa512_key.pem")) + + def tearDown(self): + # Cleanup opened resources after a test. This is usually done through atexit handlers in + # Certbot, but during tests, atexit will not run registered functions before tearDown is + # called and instead will run them right before the entire test process exits. + # It is a problem on Windows, that does not accept to clean resources before closing them. + util._release_locks() # pylint: disable=protected-access + + shutil.rmtree(self.temp_dir) + shutil.rmtree(self.config_dir) + shutil.rmtree(self.work_dir) + shutil.rmtree(self.logs_dir) + + def get_nginx_configurator(self, config_path, config_dir, work_dir, logs_dir, + version=(1, 6, 2), openssl_version="1.0.2g"): + """Create an Nginx Configurator with the specified options.""" + + backups = os.path.join(work_dir, "backups") + + self.configuration.nginx_server_root = config_path + self.configuration.le_vhost_ext = "-le-ssl.conf" + self.configuration.config_dir = config_dir + self.configuration.work_dir = work_dir + self.configuration.logs_dir = logs_dir + self.configuration.backup_dir = backups + self.configuration.temp_checkpoint_dir = os.path.join(work_dir, "temp_checkpoints") + self.configuration.in_progress_dir = os.path.join(backups, "IN_PROGRESS") + self.configuration.server = "https://acme-server.org:443/new" + self.configuration.http01_port = 80 + self.configuration.https_port = 5001 + + with mock.patch("certbot_nginx._internal.configurator.NginxConfigurator." + "config_test"): + with mock.patch("certbot_nginx._internal.configurator.util." + "exe_exists") as mock_exe_exists: + mock_exe_exists.return_value = True + config = configurator.NginxConfigurator( + self.configuration, + name="nginx", + version=version, + openssl_version=openssl_version) + config.prepare() + + # Provide general config utility. + zope.component.provideUtility(self.configuration) + + return config + + +def get_data_filename(filename): + """Gets the filename of a test data file.""" + return pkg_resources.resource_filename( + __name__, os.path.join( + "testdata", "etc_nginx", filename)) + + +def filter_comments(tree): + """Filter comment nodes from parsed configurations.""" + + def traverse(tree): + """Generator dropping comment nodes""" + for entry in tree: + # key, values = entry + spaceless = [e for e in entry if not nginxparser.spacey(e)] + if spaceless: + key = spaceless[0] + values = spaceless[1] if len(spaceless) > 1 else None + else: + key = values = "" + if isinstance(key, list): + new = copy.deepcopy(entry) + new[1] = filter_comments(values) + yield new + else: + if key != '#' and spaceless: + yield spaceless + + return list(traverse(tree)) + + +def contains_at_depth(haystack, needle, n): + """Is the needle in haystack at depth n? + + Return true if the needle is present in one of the sub-iterables in haystack + at depth n. Haystack must be an iterable. + """ + # Specifically use hasattr rather than isinstance(..., collections.Iterable) + # because we want to include lists but reject strings. + if not hasattr(haystack, '__iter__') or hasattr(haystack, 'strip'): + return False + if n == 0: + return needle in haystack + for item in haystack: + if contains_at_depth(item, needle, n - 1): + return True + return False diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/broken.conf b/certbot-nginx/tests/testdata/etc_nginx/broken.conf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/broken.conf rename to certbot-nginx/tests/testdata/etc_nginx/broken.conf diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/comment_in_file.conf b/certbot-nginx/tests/testdata/etc_nginx/comment_in_file.conf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/comment_in_file.conf rename to certbot-nginx/tests/testdata/etc_nginx/comment_in_file.conf diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/edge_cases.conf b/certbot-nginx/tests/testdata/etc_nginx/edge_cases.conf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/edge_cases.conf rename to certbot-nginx/tests/testdata/etc_nginx/edge_cases.conf diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/foo.conf b/certbot-nginx/tests/testdata/etc_nginx/foo.conf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/foo.conf rename to certbot-nginx/tests/testdata/etc_nginx/foo.conf diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/mime.types b/certbot-nginx/tests/testdata/etc_nginx/mime.types similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/mime.types rename to certbot-nginx/tests/testdata/etc_nginx/mime.types diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/minimalistic_comments.conf b/certbot-nginx/tests/testdata/etc_nginx/minimalistic_comments.conf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/minimalistic_comments.conf rename to certbot-nginx/tests/testdata/etc_nginx/minimalistic_comments.conf diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/multiline_quotes.conf b/certbot-nginx/tests/testdata/etc_nginx/multiline_quotes.conf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/multiline_quotes.conf rename to certbot-nginx/tests/testdata/etc_nginx/multiline_quotes.conf diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/nginx.conf b/certbot-nginx/tests/testdata/etc_nginx/nginx.conf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/nginx.conf rename to certbot-nginx/tests/testdata/etc_nginx/nginx.conf diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/server.conf b/certbot-nginx/tests/testdata/etc_nginx/server.conf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/server.conf rename to certbot-nginx/tests/testdata/etc_nginx/server.conf diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/default b/certbot-nginx/tests/testdata/etc_nginx/sites-enabled/default similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/default rename to certbot-nginx/tests/testdata/etc_nginx/sites-enabled/default diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/example.com b/certbot-nginx/tests/testdata/etc_nginx/sites-enabled/example.com similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/example.com rename to certbot-nginx/tests/testdata/etc_nginx/sites-enabled/example.com diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/globalssl.com b/certbot-nginx/tests/testdata/etc_nginx/sites-enabled/globalssl.com similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/globalssl.com rename to certbot-nginx/tests/testdata/etc_nginx/sites-enabled/globalssl.com diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/headers.com b/certbot-nginx/tests/testdata/etc_nginx/sites-enabled/headers.com similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/headers.com rename to certbot-nginx/tests/testdata/etc_nginx/sites-enabled/headers.com diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/ipv6.com b/certbot-nginx/tests/testdata/etc_nginx/sites-enabled/ipv6.com similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/ipv6.com rename to certbot-nginx/tests/testdata/etc_nginx/sites-enabled/ipv6.com diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/ipv6ssl.com b/certbot-nginx/tests/testdata/etc_nginx/sites-enabled/ipv6ssl.com similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/ipv6ssl.com rename to certbot-nginx/tests/testdata/etc_nginx/sites-enabled/ipv6ssl.com diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/migration.com b/certbot-nginx/tests/testdata/etc_nginx/sites-enabled/migration.com similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/migration.com rename to certbot-nginx/tests/testdata/etc_nginx/sites-enabled/migration.com diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/sslon.com b/certbot-nginx/tests/testdata/etc_nginx/sites-enabled/sslon.com similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/sslon.com rename to certbot-nginx/tests/testdata/etc_nginx/sites-enabled/sslon.com diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/fastcgi_params b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/fastcgi_params similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/fastcgi_params rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/fastcgi_params diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/koi-utf b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/koi-utf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/koi-utf rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/koi-utf diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/koi-win b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/koi-win similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/koi-win rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/koi-win diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/mime.types b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/mime.types similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/mime.types rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/mime.types diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi-ui.conf.1.4.1 b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi-ui.conf.1.4.1 similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi-ui.conf.1.4.1 rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi-ui.conf.1.4.1 diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi.rules b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi.rules similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi.rules rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi.rules diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi_core.rules b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi_core.rules similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi_core.rules rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/naxsi_core.rules diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/nginx.conf b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/nginx.conf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/nginx.conf rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/nginx.conf diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/proxy_params b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/proxy_params similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/proxy_params rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/proxy_params diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/scgi_params b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/scgi_params similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/scgi_params rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/scgi_params diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/sites-available/default b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/sites-available/default similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/sites-available/default rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/sites-available/default diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/sites-enabled/default b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/sites-enabled/default similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/sites-enabled/default rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/sites-enabled/default diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/uwsgi_params b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/uwsgi_params similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/uwsgi_params rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/uwsgi_params diff --git a/certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/win-utf b/certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/win-utf similarity index 100% rename from certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/win-utf rename to certbot-nginx/tests/testdata/etc_nginx/ubuntu_nginx_1_4_6/default_vhost/nginx/win-utf diff --git a/certbot-postfix/MANIFEST.in b/certbot-postfix/MANIFEST.in deleted file mode 100644 index 273381403..000000000 --- a/certbot-postfix/MANIFEST.in +++ /dev/null @@ -1,4 +0,0 @@ -include LICENSE.txt -include README.rst -recursive-include certbot_postfix/testdata * -recursive-include certbot_postfix/docs * diff --git a/certbot-postfix/README.rst b/certbot-postfix/README.rst deleted file mode 100644 index 1ae9cb980..000000000 --- a/certbot-postfix/README.rst +++ /dev/null @@ -1,23 +0,0 @@ -========================== -Postfix plugin for Certbot -========================== - -Note: this MTA installer is in **developer beta**-- we appreciate any testing, feedback, or -feature requests for this plugin. - -To install this plugin, in the root of this repo, run:: - - python tools/venv.py - source venv/bin/activate - -You can use this installer with any `authenticator plugin -`_. -For instance, with the `standalone authenticator -`_, which requires no extra server -software, you might run:: - - sudo ./venv/bin/certbot run --standalone -i postfix -d - -To just install existing certs with this plugin, run:: - - sudo ./venv/bin/certbot install -i postfix --cert-path --key-path -d diff --git a/certbot-postfix/certbot_postfix/__init__.py b/certbot-postfix/certbot_postfix/__init__.py deleted file mode 100644 index 122c54bc6..000000000 --- a/certbot-postfix/certbot_postfix/__init__.py +++ /dev/null @@ -1,3 +0,0 @@ -"""Certbot Postfix plugin.""" - -from certbot_postfix.installer import Installer diff --git a/certbot-postfix/certbot_postfix/constants.py b/certbot-postfix/certbot_postfix/constants.py deleted file mode 100644 index 40a263a53..000000000 --- a/certbot-postfix/certbot_postfix/constants.py +++ /dev/null @@ -1,63 +0,0 @@ -"""Postfix plugin constants.""" - -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import Dict, Tuple, Union -# pylint: enable=unused-import, no-name-in-module - -MINIMUM_VERSION = (2, 11,) - -# If the value of a default VAR is a tuple, then the values which -# come LATER in the tuple are more strict/more secure. -# Certbot will default to the first value in the tuple, but will -# not override "more secure" settings. - -ACCEPTABLE_SERVER_SECURITY_LEVELS = ("may", "encrypt") -ACCEPTABLE_CLIENT_SECURITY_LEVELS = ("may", "encrypt", - "dane", "dane-only", - "fingerprint", - "verify", "secure") -ACCEPTABLE_CIPHER_LEVELS = ("medium", "high") - -# Exporting certain ciphers to prevent logjam: https://weakdh.org/sysadmin.html -EXCLUDE_CIPHERS = ("aNULL, eNULL, EXPORT, DES, RC4, MD5, PSK, aECDH, " - "EDH-DSS-DES-CBC3-SHA, EDH-RSA-DES-CBC3-SHA, KRB5-DES, CBC3-SHA",) - - -TLS_VERSIONS = ("SSLv2", "SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2") -# Should NOT use SSLv2/3. -ACCEPTABLE_TLS_VERSIONS = ("TLSv1", "TLSv1.1", "TLSv1.2") - -# Variables associated with enabling opportunistic TLS. -TLS_SERVER_VARS = { - "smtpd_tls_security_level": ACCEPTABLE_SERVER_SECURITY_LEVELS, -} # type:Dict[str, Tuple[str, ...]] -TLS_CLIENT_VARS = { - "smtp_tls_security_level": ACCEPTABLE_CLIENT_SECURITY_LEVELS, -} # type:Dict[str, Tuple[str, ...]] -# Default variables for a secure MTA server [receiver]. -DEFAULT_SERVER_VARS = { - "smtpd_tls_auth_only": ("yes",), - "smtpd_tls_mandatory_protocols": ("!SSLv2, !SSLv3",), - "smtpd_tls_protocols": ("!SSLv2, !SSLv3",), - "smtpd_tls_ciphers": ACCEPTABLE_CIPHER_LEVELS, - "smtpd_tls_mandatory_ciphers": ACCEPTABLE_CIPHER_LEVELS, - "smtpd_tls_exclude_ciphers": EXCLUDE_CIPHERS, - "smtpd_tls_eecdh_grade": ("strong",), -} # type:Dict[str, Tuple[str, ...]] - -# Default variables for a secure MTA client [sender]. -DEFAULT_CLIENT_VARS = { - "smtp_tls_ciphers": ACCEPTABLE_CIPHER_LEVELS, - "smtp_tls_exclude_ciphers": EXCLUDE_CIPHERS, - "smtp_tls_mandatory_ciphers": ACCEPTABLE_CIPHER_LEVELS, -} # type:Dict[str, Tuple[str, ...]] - -CLI_DEFAULTS = dict( - config_dir="/etc/postfix", - ctl="postfix", - config_utility="postconf", - tls_only=False, - ignore_master_overrides=False, - server_only=False, -) -"""CLI defaults.""" diff --git a/certbot-postfix/certbot_postfix/installer.py b/certbot-postfix/certbot_postfix/installer.py deleted file mode 100644 index 9ba92ef8f..000000000 --- a/certbot-postfix/certbot_postfix/installer.py +++ /dev/null @@ -1,288 +0,0 @@ -"""certbot installer plugin for postfix.""" -import logging -import os - -import zope.interface -import zope.component -import six - -from certbot import errors -from certbot import interfaces -from certbot import util as certbot_util -from certbot.plugins import common as plugins_common - -from certbot_postfix import constants -from certbot_postfix import postconf -from certbot_postfix import util - -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import Callable, Dict, List -# pylint: enable=unused-import, no-name-in-module - -logger = logging.getLogger(__name__) - -@zope.interface.implementer(interfaces.IInstaller) -@zope.interface.provider(interfaces.IPluginFactory) -class Installer(plugins_common.Installer): - """Certbot installer plugin for Postfix. - - :ivar str config_dir: Postfix configuration directory to modify - :ivar list save_notes: documentation for proposed changes. This is - cleared and stored in Certbot checkpoints when save() is called - - :ivar postconf: Wrapper for Postfix configuration command-line tool. - :type postconf: :class: `certbot_postfix.postconf.ConfigMain` - :ivar postfix: Wrapper for Postfix command-line tool. - :type postfix: :class: `certbot_postfix.util.PostfixUtil` - """ - - description = "Configure TLS with the Postfix MTA" - - @classmethod - def add_parser_arguments(cls, add): - add("ctl", default=constants.CLI_DEFAULTS["ctl"], - help="Path to the 'postfix' control program.") - # This directory points to Postfix's configuration directory. - add("config-dir", default=constants.CLI_DEFAULTS["config_dir"], - help="Path to the directory containing the " - "Postfix main.cf file to modify instead of using the " - "default configuration paths.") - add("config-utility", default=constants.CLI_DEFAULTS["config_utility"], - help="Path to the 'postconf' executable.") - add("tls-only", action="store_true", default=constants.CLI_DEFAULTS["tls_only"], - help="Only set params to enable opportunistic TLS and install certificates.") - add("server-only", action="store_true", default=constants.CLI_DEFAULTS["server_only"], - help="Only set server params (prefixed with smtpd*)") - add("ignore-master-overrides", action="store_true", - default=constants.CLI_DEFAULTS["ignore_master_overrides"], - help="Ignore errors reporting overridden TLS parameters in master.cf.") - - def __init__(self, *args, **kwargs): - super(Installer, self).__init__(*args, **kwargs) - # Wrapper around postconf commands - self.postfix = None - self.postconf = None - - # Files to save - self.save_notes = [] # type: List[str] - - self._enhance_func = {} # type: Dict[str, Callable[[str, str], None]] - # Since we only need to enable TLS once for all domains, - # keep track of whether this enhancement was already called. - self._tls_enabled = False - - def prepare(self): - """Prepare the installer. - - :raises errors.PluginError: when an unexpected error occurs - :raises errors.MisconfigurationError: when the config is invalid - :raises errors.NoInstallationError: when can't find installation - :raises errors.NotSupportedError: when version is not supported - """ - # Verify postfix and postconf are installed - for param in ("ctl", "config_utility",): - util.verify_exe_exists(self.conf(param), - "Cannot find executable '{0}'. You can provide the " - "path to this command with --{1}".format( - self.conf(param), - self.option_name(param))) - - # Set up CLI tools - self.postfix = util.PostfixUtil(self.conf('config-dir')) - self.postconf = postconf.ConfigMain(self.conf('config-utility'), - self.conf('ignore-master-overrides'), - self.conf('config-dir')) - - # Ensure current configuration is valid. - self.config_test() - - # Check Postfix version - self._check_version() - self._lock_config_dir() - self.install_ssl_dhparams() - - def config_test(self): - """Test to see that the current Postfix configuration is valid. - - :raises errors.MisconfigurationError: If the configuration is invalid. - """ - self.postfix.test() - - def _check_version(self): - """Verifies that the installed Postfix version is supported. - - :raises errors.NotSupportedError: if the version is unsupported - """ - if self._get_version() < constants.MINIMUM_VERSION: - version_string = '.'.join([str(n) for n in constants.MINIMUM_VERSION]) - raise errors.NotSupportedError('Postfix version must be at least %s' % version_string) - - def _lock_config_dir(self): - """Stop two Postfix plugins from modifying the config at once. - - :raises .PluginError: if unable to acquire the lock - """ - try: - certbot_util.lock_dir_until_exit(self.conf('config-dir')) - except (OSError, errors.LockError): - logger.debug("Encountered error:", exc_info=True) - raise errors.PluginError( - "Unable to lock %s" % self.conf('config-dir')) - - def more_info(self): - """Human-readable string to help the user. Describes steps taken and any relevant - info to help the user decide which plugin to use. - - :rtype: str - """ - return ( - "Configures Postfix to try to authenticate mail servers, use " - "installed certificates and disable weak ciphers and protocols.{0}" - "Server root: {root}{0}" - "Version: {version}".format( - os.linesep, - root=self.conf('config-dir'), - version='.'.join([str(i) for i in self._get_version()])) - ) - - def _get_version(self): - """Return the version of Postfix, as a tuple. (e.g. '2.11.3' is (2, 11, 3)) - - :returns: version - :rtype: tuple - - :raises errors.PluginError: Unable to find Postfix version. - """ - mail_version = self.postconf.get_default("mail_version") - return tuple(int(i) for i in mail_version.split('.')) - - def get_all_names(self): - """Returns all names that may be authenticated. - - :rtype: `set` of `str` - - """ - return certbot_util.get_filtered_names(self.postconf.get(var) - for var in ('mydomain', 'myhostname', 'myorigin',)) - - def _set_vars(self, var_dict): - """Sets all parameters in var_dict to config file. If current value is already set - as more secure (acceptable), then don't set/overwrite it. - """ - for param, acceptable in six.iteritems(var_dict): - if not util.is_acceptable_value(param, self.postconf.get(param), acceptable): - self.postconf.set(param, acceptable[0], acceptable) - - def _confirm_changes(self): - """Confirming outstanding updates for configuration parameters. - - :raises errors.PluginError: when user rejects the configuration changes. - """ - updates = self.postconf.get_changes() - output_string = "Postfix TLS configuration parameters to update in main.cf:\n" - for name, value in six.iteritems(updates): - output_string += "{0} = {1}\n".format(name, value) - output_string += "Is this okay?\n" - if not zope.component.getUtility(interfaces.IDisplay).yesno(output_string, - force_interactive=True, default=True): - raise errors.PluginError( - "Manually rejected configuration changes.\n" - "Try using --tls-only or --server-only to change a particular" - "subset of configuration parameters.") - - def deploy_cert(self, domain, cert_path, - key_path, chain_path, fullchain_path): - """Configure the Postfix SMTP server to use the given TLS cert. - - :param str domain: domain to deploy certificate file - :param str cert_path: absolute path to the certificate file - :param str key_path: absolute path to the private key file - :param str chain_path: absolute path to the certificate chain file - :param str fullchain_path: absolute path to the certificate fullchain - file (cert plus chain) - - :raises .PluginError: when cert cannot be deployed - - """ - # pylint: disable=unused-argument - if self._tls_enabled: - return - self._tls_enabled = True - self.save_notes.append("Configuring TLS for {0}".format(domain)) - self.postconf.set("smtpd_tls_cert_file", cert_path) - self.postconf.set("smtpd_tls_key_file", key_path) - self._set_vars(constants.TLS_SERVER_VARS) - if not self.conf('server_only'): - self._set_vars(constants.TLS_CLIENT_VARS) - if not self.conf('tls_only'): - self._set_vars(constants.DEFAULT_SERVER_VARS) - if not self.conf('server_only'): - self._set_vars(constants.DEFAULT_CLIENT_VARS) - # Despite the name, this option also supports 2048-bit DH params. - # http://www.postfix.org/FORWARD_SECRECY_README.html#server_fs - self.postconf.set("smtpd_tls_dh1024_param_file", self.ssl_dhparams) - self._confirm_changes() - - def enhance(self, domain, enhancement, options=None): - """Raises an exception since this installer doesn't support any enhancements. - """ - # pylint: disable=unused-argument - raise errors.PluginError( - "Unsupported enhancement: {0}".format(enhancement)) - - def supported_enhancements(self): - """Returns a list of supported enhancements. - - :rtype: list - - """ - return [] - - def save(self, title=None, temporary=False): - """Creates backups and writes changes to configuration files. - - :param str title: The title of the save. If a title is given, the - configuration will be saved as a new checkpoint and put in a - timestamped directory. `title` has no effect if temporary is true. - - :param bool temporary: Indicates whether the changes made will - be quickly reversed in the future (challenges) - - :raises errors.PluginError: when save is unsuccessful - """ - save_files = set((os.path.join(self.conf('config-dir'), "main.cf"),)) - self.add_to_checkpoint(save_files, - "\n".join(self.save_notes), temporary) - self.postconf.flush() - - del self.save_notes[:] - - if title and not temporary: - self.finalize_checkpoint(title) - - def recovery_routine(self): - super(Installer, self).recovery_routine() - self.postconf = postconf.ConfigMain(self.conf('config-utility'), - self.conf('ignore-master-overrides'), - self.conf('config-dir')) - - def rollback_checkpoints(self, rollback=1): - """Rollback saved checkpoints. - - :param int rollback: Number of checkpoints to revert - - :raises .errors.PluginError: If there is a problem with the input or - the function is unable to correctly revert the configuration - """ - super(Installer, self).rollback_checkpoints(rollback) - self.postconf = postconf.ConfigMain(self.conf('config-utility'), - self.conf('ignore-master-overrides'), - self.conf('config-dir')) - - def restart(self): - """Restart or refresh the server content. - - :raises .PluginError: when server cannot be restarted - """ - self.postfix.restart() - diff --git a/certbot-postfix/certbot_postfix/postconf.py b/certbot-postfix/certbot_postfix/postconf.py deleted file mode 100644 index 466e0e63e..000000000 --- a/certbot-postfix/certbot_postfix/postconf.py +++ /dev/null @@ -1,152 +0,0 @@ -"""Classes that wrap the postconf command line utility. -""" -import six -from certbot import errors -from certbot_postfix import util - -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import Dict, List, Tuple -# pylint: enable=unused-import, no-name-in-module - -class ConfigMain(util.PostfixUtilBase): - """A parser for Postfix's main.cf file.""" - - def __init__(self, executable, ignore_master_overrides=False, config_dir=None): - super(ConfigMain, self).__init__(executable, config_dir) - # Whether to ignore overrides from master. - self._ignore_master_overrides = ignore_master_overrides - # List of all current Postfix parameters, from `postconf` command. - self._db = {} # type: Dict[str, str] - # List of current master.cf overrides from Postfix config. Dictionary - # of parameter name => list of tuples (service name, paramter value) - # Note: We should never modify master without explicit permission. - self._master_db = {} # type: Dict[str, List[Tuple[str, str]]] - # List of all changes requested to the Postfix parameters as they are now - # in _db. These changes are flushed to `postconf` on `flush`. - self._updated = {} # type: Dict[str, str] - self._read_from_conf() - - def _read_from_conf(self): - """Reads initial parameter state from `main.cf` into this object. - """ - out = self._get_output() - for name, value in _parse_main_output(out): - self._db[name] = value - out = self._get_output_master() - for name, value in _parse_main_output(out): - service, param_name = name.rsplit("/", 1) - if param_name not in self._master_db: - self._master_db[param_name] = [] - self._master_db[param_name].append((service, value)) - - def _get_output_master(self): - """Retrieves output for `master.cf` parameters.""" - return self._get_output('-P') - - def get_default(self, name): - """Retrieves default value of parameter `name` from postfix parameters. - - :param str name: The name of the parameter to fetch. - :returns: The default value of parameter `name`. - :rtype: str - """ - out = self._get_output(['-d', name]) - _, value = next(_parse_main_output(out), (None, None)) - return value - - def get(self, name): - """Retrieves working value of parameter `name` from postfix parameters. - - :param str name: The name of the parameter to fetch. - :returns: The value of parameter `name`. - :rtype: str - """ - if name in self._updated: - return self._updated[name] - return self._db[name] - - def get_master_overrides(self, name): - """Retrieves list of overrides for parameter `name` in postfix's Master config - file. - - :returns: List of tuples (service, value), meaning that parameter `name` - is overridden as `value` for `service`. - :rtype: `list` of `tuple` of `str` - """ - if name in self._master_db: - return self._master_db[name] - return None - - def set(self, name, value, acceptable_overrides=None): - """Sets parameter `name` to `value`. If `name` is overridden by a particular service in - `master.cf`, reports any of these parameter conflicts as long as - `ignore_master_overrides` was not set. - - .. note:: that this function does not flush these parameter values to main.cf; - To do that, use `flush`. - - :param str name: The name of the parameter to set. - :param str value: The value of the parameter. - :param tuple acceptable_overrides: If the master configuration file overrides `value` - with a value in acceptable_overrides. - """ - if name not in self._db: - raise KeyError("Parameter name %s is not a valid Postfix parameter name.", name) - # Check to see if this parameter is overridden by master. - overrides = self.get_master_overrides(name) - if not self._ignore_master_overrides and overrides is not None: - util.report_master_overrides(name, overrides, acceptable_overrides) - if value != self._db[name]: - # _db contains the "original" state of parameters. We only care about - # writes if they cause a delta from the original state. - self._updated[name] = value - elif name in self._updated: - # If this write reverts a previously updated parameter back to the - # original DB's state, we don't have to keep track of it in _updated. - del self._updated[name] - - def flush(self): - """Flushes all parameter changes made using `self.set`, to `main.cf` - - :raises error.PluginError: When flush to main.cf fails for some reason. - """ - if len(self._updated) == 0: - return - args = ['-e'] - for name, value in six.iteritems(self._updated): - args.append('{0}={1}'.format(name, value)) - try: - self._get_output(args) - except IOError as e: - raise errors.PluginError("Unable to save to Postfix config: %v", e) - for name, value in six.iteritems(self._updated): - self._db[name] = value - self._updated = {} - - def get_changes(self): - """ Return queued changes to main.cf. - - :rtype: dict[str, str] - """ - return self._updated - -def _parse_main_output(output): - """Parses the raw output from Postconf about main.cf. - - Expects the output to look like: - - .. code-block:: none - - name1 = value1 - name2 = value2 - - :param str output: data postconf wrote to stdout about main.cf - - :returns: generator providing key-value pairs from main.cf - :rtype: Iterator[tuple(str, str)] - """ - for line in output.splitlines(): - name, _, value = line.partition(" =") - yield name, value.strip() - - diff --git a/certbot-postfix/certbot_postfix/tests/__init__.py b/certbot-postfix/certbot_postfix/tests/__init__.py deleted file mode 100644 index 7316b5888..000000000 --- a/certbot-postfix/certbot_postfix/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -""" Certbot Postfix Tests """ diff --git a/certbot-postfix/certbot_postfix/tests/installer_test.py b/certbot-postfix/certbot_postfix/tests/installer_test.py deleted file mode 100644 index 37b78bdca..000000000 --- a/certbot-postfix/certbot_postfix/tests/installer_test.py +++ /dev/null @@ -1,314 +0,0 @@ -"""Tests for certbot_postfix.installer.""" -from contextlib import contextmanager -import copy -import functools -import os -import pkg_resources -import six -import unittest - -import mock - -from certbot import errors -from certbot.tests import util as certbot_test_util - -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import Dict, Tuple, Union -# pylint: enable=unused-import, no-name-in-module - -DEFAULT_MAIN_CF = { - "smtpd_tls_cert_file": "", - "smtpd_tls_key_file": "", - "smtpd_tls_dh1024_param_file": "", - "smtpd_tls_security_level": "none", - "smtpd_tls_auth_only": "", - "smtpd_tls_mandatory_protocols": "", - "smtpd_tls_protocols": "", - "smtpd_tls_ciphers": "", - "smtpd_tls_exclude_ciphers": "", - "smtpd_tls_mandatory_ciphers": "", - "smtpd_tls_eecdh_grade": "medium", - "smtp_tls_security_level": "", - "smtp_tls_ciphers": "", - "smtp_tls_exclude_ciphers": "", - "smtp_tls_mandatory_ciphers": "", - "mail_version": "3.2.3" -} - -def _main_cf_with(obj): - main_cf = copy.copy(DEFAULT_MAIN_CF) - main_cf.update(obj) - return main_cf - -class InstallerTest(certbot_test_util.ConfigTestCase): - # pylint: disable=too-many-public-methods - - def setUp(self): - super(InstallerTest, self).setUp() - _config_file = pkg_resources.resource_filename("certbot_postfix.tests", - os.path.join("testdata", "config.json")) - self.config.postfix_ctl = "postfix" - self.config.postfix_config_dir = self.tempdir - self.config.postfix_config_utility = "postconf" - self.config.postfix_tls_only = False - self.config.postfix_server_only = False - self.config.config_dir = self.tempdir - - @mock.patch("certbot_postfix.installer.util.is_acceptable_value") - def test_set_vars(self, mock_is_acceptable_value): - mock_is_acceptable_value.return_value = True - with create_installer(self.config) as installer: - installer.prepare() - mock_is_acceptable_value.return_value = False - - @mock.patch("certbot_postfix.installer.util.is_acceptable_value") - def test_acceptable_value(self, mock_is_acceptable_value): - mock_is_acceptable_value.return_value = True - with create_installer(self.config) as installer: - installer.prepare() - mock_is_acceptable_value.return_value = False - - @certbot_test_util.patch_get_utility() - def test_confirm_changes_no_raises_error(self, mock_util): - mock_util().yesno.return_value = False - with create_installer(self.config) as installer: - installer.prepare() - self.assertRaises(errors.PluginError, installer.deploy_cert, - "example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - - @certbot_test_util.patch_get_utility() - def test_save(self, mock_util): - mock_util().yesno.return_value = True - with create_installer(self.config) as installer: - installer.prepare() - installer.postconf.flush = mock.Mock() - installer.reverter = mock.Mock() - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - installer.save() - self.assertEqual(installer.save_notes, []) - self.assertEqual(installer.postconf.flush.call_count, 1) - self.assertEqual(installer.reverter.add_to_checkpoint.call_count, 1) - - @certbot_test_util.patch_get_utility() - def test_save_with_title(self, mock_util): - mock_util().yesno.return_value = True - with create_installer(self.config) as installer: - installer.prepare() - installer.postconf.flush = mock.Mock() - installer.reverter = mock.Mock() - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - installer.save(title="new_file!") - self.assertEqual(installer.reverter.finalize_checkpoint.call_count, 1) - - @certbot_test_util.patch_get_utility() - def test_rollback_checkpoints_resets_postconf(self, mock_util): - mock_util().yesno.return_value = True - with create_installer(self.config) as installer: - installer.prepare() - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - installer.rollback_checkpoints() - self.assertEqual(installer.postconf.get_changes(), {}) - - @certbot_test_util.patch_get_utility() - def test_recovery_routine_resets_postconf(self, mock_util): - mock_util().yesno.return_value = True - with create_installer(self.config) as installer: - installer.prepare() - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - installer.recovery_routine() - self.assertEqual(installer.postconf.get_changes(), {}) - - def test_restart(self): - with create_installer(self.config) as installer: - installer.prepare() - installer.restart() - self.assertEqual(installer.postfix.restart.call_count, 1) - - def test_add_parser_arguments(self): - options = set(("ctl", "config-dir", "config-utility", - "tls-only", "server-only", "ignore-master-overrides")) - mock_add = mock.MagicMock() - - from certbot_postfix import installer - installer.Installer.add_parser_arguments(mock_add) - - for call in mock_add.call_args_list: - self.assertTrue(call[0][0] in options) - - def test_no_postconf_prepare(self): - with create_installer(self.config) as installer: - installer_path = "certbot_postfix.installer" - exe_exists_path = installer_path + ".certbot_util.exe_exists" - path_surgery_path = "certbot_postfix.util.plugins_util.path_surgery" - with mock.patch(path_surgery_path, return_value=False): - with mock.patch(exe_exists_path, return_value=False): - self.assertRaises(errors.NoInstallationError, - installer.prepare) - - def test_old_version(self): - with create_installer(self.config, main_cf=_main_cf_with({"mail_version": "0.0.1"}))\ - as installer: - self.assertRaises(errors.NotSupportedError, installer.prepare) - - def test_lock_error(self): - with create_installer(self.config) as installer: - assert_raises = functools.partial(self.assertRaises, - errors.PluginError, - installer.prepare) - certbot_test_util.lock_and_call(assert_raises, self.tempdir) - - - @mock.patch('certbot.util.lock_dir_until_exit') - def test_dir_locked(self, lock_dir): - with create_installer(self.config) as installer: - lock_dir.side_effect = errors.LockError - self.assertRaises(errors.PluginError, installer.prepare) - - def test_more_info(self): - with create_installer(self.config) as installer: - installer.prepare() - output = installer.more_info() - self.assertTrue("Postfix" in output) - self.assertTrue(self.tempdir in output) - self.assertTrue(DEFAULT_MAIN_CF["mail_version"] in output) - - def test_get_all_names(self): - config = {"mydomain": "example.org", - "myhostname": "mail.example.org", - "myorigin": "example.org"} - with create_installer(self.config, main_cf=_main_cf_with(config)) as installer: - installer.prepare() - result = installer.get_all_names() - self.assertEqual(result, set(config.values())) - - @certbot_test_util.patch_get_utility() - def test_deploy(self, mock_util): - mock_util().yesno.return_value = True - from certbot_postfix import constants - with create_installer(self.config) as installer: - installer.prepare() - - # pylint: disable=protected-access - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - changes = installer.postconf.get_changes() - expected = {} # type: Dict[str, Tuple[str, ...]] - expected.update(constants.TLS_SERVER_VARS) - expected.update(constants.DEFAULT_SERVER_VARS) - expected.update(constants.DEFAULT_CLIENT_VARS) - self.assertEqual(changes["smtpd_tls_key_file"], "key_path") - self.assertEqual(changes["smtpd_tls_cert_file"], "cert_path") - for name, value in six.iteritems(expected): - self.assertEqual(changes[name], value[0]) - - @certbot_test_util.patch_get_utility() - def test_tls_only(self, mock_util): - mock_util().yesno.return_value = True - with create_installer(self.config) as installer: - installer.prepare() - installer.conf = lambda x: x == "tls_only" - installer.postconf.set = mock.Mock() - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - self.assertEqual(installer.postconf.set.call_count, 4) - - @certbot_test_util.patch_get_utility() - def test_server_only(self, mock_util): - mock_util().yesno.return_value = True - with create_installer(self.config) as installer: - installer.prepare() - installer.conf = lambda x: x == "server_only" - installer.postconf.set = mock.Mock() - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - self.assertEqual(installer.postconf.set.call_count, 11) - - @certbot_test_util.patch_get_utility() - def test_tls_and_server_only(self, mock_util): - mock_util().yesno.return_value = True - with create_installer(self.config) as installer: - installer.prepare() - installer.conf = lambda x: True - installer.postconf.set = mock.Mock() - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - self.assertEqual(installer.postconf.set.call_count, 3) - - @certbot_test_util.patch_get_utility() - def test_deploy_twice(self, mock_util): - # Deploying twice on the same installer shouldn't do anything! - mock_util().yesno.return_value = True - with create_installer(self.config) as installer: - installer.prepare() - from certbot_postfix.postconf import ConfigMain - with mock.patch.object(ConfigMain, "set", wraps=installer.postconf.set) as fake_set: - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - self.assertEqual(fake_set.call_count, 15) - fake_set.reset_mock() - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - self.assertFalse(fake_set.called) - - @certbot_test_util.patch_get_utility() - def test_deploy_already_secure(self, mock_util): - # Should not overwrite "more-secure" parameters - mock_util().yesno.return_value = True - more_secure = { - "smtpd_tls_security_level": "encrypt", - "smtpd_tls_protocols": "!SSLv3, !SSLv2, !TLSv1", - "smtpd_tls_eecdh_grade": "strong" - } - with create_installer(self.config,\ - main_cf=_main_cf_with(more_secure)) as installer: - installer.prepare() - installer.deploy_cert("example.com", "cert_path", "key_path", - "chain_path", "fullchain_path") - for param in more_secure.keys(): - self.assertFalse(param in installer.postconf.get_changes()) - - def test_enhance(self): - with create_installer(self.config) as installer: - installer.prepare() - self.assertRaises(errors.PluginError, - installer.enhance, - "example.org", "redirect") - - def test_supported_enhancements(self): - with create_installer(self.config) as installer: - installer.prepare() - self.assertEqual(installer.supported_enhancements(), []) - -@contextmanager -def create_installer(config, main_cf=DEFAULT_MAIN_CF): -# pylint: disable=dangerous-default-value - """Creates a Postfix installer with calls to `postconf` and `postfix` mocked out. - - In particular, creates a ConfigMain object that does regular things, but seeds it - with values from `main_cf` and `master_cf` dicts. - """ - from certbot_postfix.postconf import ConfigMain - from certbot_postfix import installer - def _mock_init_postconf(postconf, executable, ignore_master_overrides=False, config_dir=None): - # pylint: disable=protected-access,unused-argument - postconf._ignore_master_overrides = ignore_master_overrides - postconf._db = main_cf - postconf._master_db = {} - postconf._updated = {} - # override get_default to get from main - postconf.get_default = lambda name: main_cf[name] - with mock.patch.object(ConfigMain, "__init__", _mock_init_postconf): - exe_exists_path = "certbot_postfix.installer.certbot_util.exe_exists" - with mock.patch(exe_exists_path, return_value=True): - with mock.patch("certbot_postfix.installer.util.PostfixUtil", - return_value=mock.Mock()): - yield installer.Installer(config, "postfix") - -if __name__ == "__main__": - unittest.main() # pragma: no cover - diff --git a/certbot-postfix/certbot_postfix/tests/postconf_test.py b/certbot-postfix/certbot_postfix/tests/postconf_test.py deleted file mode 100644 index 01a43773d..000000000 --- a/certbot-postfix/certbot_postfix/tests/postconf_test.py +++ /dev/null @@ -1,107 +0,0 @@ -"""Tests for certbot_postfix.postconf.""" - -import mock -import unittest - -from certbot import errors - -class PostConfTest(unittest.TestCase): - """Tests for certbot_postfix.util.PostConf.""" - def setUp(self): - from certbot_postfix.postconf import ConfigMain - super(PostConfTest, self).setUp() - with mock.patch('certbot_postfix.util.PostfixUtilBase._get_output') as mock_call: - with mock.patch('certbot_postfix.postconf.ConfigMain._get_output_master') as \ - mock_master_call: - with mock.patch('certbot_postfix.postconf.util.verify_exe_exists') as verify_exe: - verify_exe.return_value = True - mock_call.return_value = ('default_parameter = value\n' - 'extra_param =\n' - 'overridden_by_master = default\n') - mock_master_call.return_value = ( - 'service/type/overridden_by_master = master_value\n' - 'service2/type/overridden_by_master = master_value2\n' - ) - self.config = ConfigMain('postconf', False) - - @mock.patch('certbot_postfix.util.PostfixUtilBase._get_output') - @mock.patch('certbot_postfix.postconf.util.verify_exe_exists') - def test_get_output_master(self, mock_verify_exe, mock_get_output): - from certbot_postfix.postconf import ConfigMain - mock_verify_exe.return_value = True - ConfigMain('postconf', lambda x, y, z: None) - mock_get_output.assert_called_with('-P') - - @mock.patch('certbot_postfix.util.PostfixUtilBase._get_output') - def test_read_default(self, mock_get_output): - mock_get_output.return_value = 'param = default_value' - self.assertEqual(self.config.get_default('param'), 'default_value') - - @mock.patch('certbot_postfix.util.PostfixUtilBase._call') - def test_set(self, mock_call): - self.config.set('extra_param', 'other_value') - self.assertEqual(self.config.get('extra_param'), 'other_value') - self.config.flush() - mock_call.assert_called_with(['-e', 'extra_param=other_value']) - - def test_set_bad_param_name(self): - self.assertRaises(KeyError, self.config.set, 'nonexistent_param', 'some_value') - - @mock.patch('certbot_postfix.util.PostfixUtilBase._call') - def test_write_revert(self, mock_call): - self.config.set('default_parameter', 'fake_news') - # revert config set - self.config.set('default_parameter', 'value') - self.config.flush() - mock_call.assert_not_called() - - @mock.patch('certbot_postfix.util.PostfixUtilBase._call') - def test_write_default(self, mock_call): - self.config.set('default_parameter', 'value') - self.config.flush() - mock_call.assert_not_called() - - def test_master_overrides(self): - self.assertEqual(self.config.get_master_overrides('overridden_by_master'), - [('service/type', 'master_value'), - ('service2/type', 'master_value2')]) - - def test_set_check_override(self): - self.assertRaises(errors.PluginError, self.config.set, - 'overridden_by_master', 'new_value') - - def test_ignore_check_override(self): - # pylint: disable=protected-access - self.config._ignore_master_overrides = True - self.config.set('overridden_by_master', 'new_value') - - def test_check_acceptable_overrides(self): - self.config.set('overridden_by_master', 'new_value', - ('master_value', 'master_value2')) - - @mock.patch('certbot_postfix.util.PostfixUtilBase._get_output') - def test_flush(self, mock_out): - self.config.set('default_parameter', 'new_value') - self.config.set('extra_param', 'another_value') - self.config.flush() - arguments = mock_out.call_args_list[-1][0][0] - self.assertEqual('-e', arguments[0]) - self.assertTrue('default_parameter=new_value' in arguments) - self.assertTrue('extra_param=another_value' in arguments) - - @mock.patch('certbot_postfix.util.PostfixUtilBase._get_output') - def test_flush_updates_object(self, mock_out): - self.config.set('default_parameter', 'new_value') - self.config.flush() - mock_out.reset_mock() - self.config.set('default_parameter', 'new_value') - mock_out.assert_not_called() - - @mock.patch('certbot_postfix.util.PostfixUtilBase._get_output') - def test_flush_throws_error_on_fail(self, mock_out): - mock_out.side_effect = [IOError("oh no!")] - self.config.set('default_parameter', 'new_value') - self.assertRaises(errors.PluginError, self.config.flush) - -if __name__ == '__main__': # pragma: no cover - unittest.main() diff --git a/certbot-postfix/certbot_postfix/tests/util_test.py b/certbot-postfix/certbot_postfix/tests/util_test.py deleted file mode 100644 index fa38f83ab..000000000 --- a/certbot-postfix/certbot_postfix/tests/util_test.py +++ /dev/null @@ -1,205 +0,0 @@ -"""Tests for certbot_postfix.util.""" - -import subprocess -import unittest - -import mock - -from certbot import errors - - -class PostfixUtilBaseTest(unittest.TestCase): - """Tests for certbot_postfix.util.PostfixUtilBase.""" - - @classmethod - def _create_object(cls, *args, **kwargs): - from certbot_postfix.util import PostfixUtilBase - return PostfixUtilBase(*args, **kwargs) - - @mock.patch('certbot_postfix.util.verify_exe_exists') - def test_no_exe(self, mock_verify): - expected_error = errors.NoInstallationError - mock_verify.side_effect = expected_error - self.assertRaises(expected_error, self._create_object, 'nonexistent') - - def test_object_creation(self): - with mock.patch('certbot_postfix.util.verify_exe_exists'): - self._create_object('existent') - - @mock.patch('certbot_postfix.util.check_all_output') - def test_call_extends_args(self, mock_output): - # pylint: disable=protected-access - with mock.patch('certbot_postfix.util.verify_exe_exists'): - mock_output.return_value = 'expected' - postfix = self._create_object('executable') - postfix._call(['many', 'extra', 'args']) - mock_output.assert_called_with(['executable', 'many', 'extra', 'args']) - postfix._call() - mock_output.assert_called_with(['executable']) - - def test_create_with_config(self): - # pylint: disable=protected-access - with mock.patch('certbot_postfix.util.verify_exe_exists'): - postfix = self._create_object('exec', 'config_dir') - self.assertEqual(postfix._base_command, ['exec', '-c', 'config_dir']) - -class PostfixUtilTest(unittest.TestCase): - def setUp(self): - # pylint: disable=protected-access - from certbot_postfix.util import PostfixUtil - with mock.patch('certbot_postfix.util.verify_exe_exists'): - self.postfix = PostfixUtil() - self.postfix._call = mock.Mock() - self.mock_call = self.postfix._call - - def test_test(self): - self.postfix.test() - self.mock_call.assert_called_with(['check']) - - def test_test_raises_error_when_check_fails(self): - self.mock_call.side_effect = [subprocess.CalledProcessError(1, "")] - self.assertRaises(errors.MisconfigurationError, self.postfix.test) - self.mock_call.assert_called_with(['check']) - - def test_restart_while_running(self): - self.mock_call.side_effect = [subprocess.CalledProcessError(1, ""), None] - self.postfix.restart() - self.mock_call.assert_called_with(['start']) - - def test_restart_while_not_running(self): - self.postfix.restart() - self.mock_call.assert_called_with(['reload']) - - def test_restart_raises_error_when_reload_fails(self): - self.mock_call.side_effect = [None, subprocess.CalledProcessError(1, "")] - self.assertRaises(errors.PluginError, self.postfix.restart) - self.mock_call.assert_called_with(['reload']) - - def test_restart_raises_error_when_start_fails(self): - self.mock_call.side_effect = [ - subprocess.CalledProcessError(1, ""), - subprocess.CalledProcessError(1, "")] - self.assertRaises(errors.PluginError, self.postfix.restart) - self.mock_call.assert_called_with(['start']) - -class CheckAllOutputTest(unittest.TestCase): - """Tests for certbot_postfix.util.check_all_output.""" - - @classmethod - def _call(cls, *args, **kwargs): - from certbot_postfix.util import check_all_output - return check_all_output(*args, **kwargs) - - @mock.patch('certbot_postfix.util.logger') - @mock.patch('certbot_postfix.util.subprocess.Popen') - def test_command_error(self, mock_popen, mock_logger): - command = 'foo' - retcode = 42 - output = 'bar' - err = 'baz' - - mock_popen().communicate.return_value = (output, err) - mock_popen().poll.return_value = 42 - - self.assertRaises(subprocess.CalledProcessError, self._call, command) - log_args = mock_logger.debug.call_args[0] - for value in (command, retcode, output, err,): - self.assertTrue(value in log_args) - - @mock.patch('certbot_postfix.util.subprocess.Popen') - def test_success(self, mock_popen): - command = 'foo' - expected = ('bar', '') - mock_popen().communicate.return_value = expected - mock_popen().poll.return_value = 0 - - self.assertEqual(self._call(command), expected) - - def test_stdout_error(self): - self.assertRaises(ValueError, self._call, stdout=None) - - def test_stderr_error(self): - self.assertRaises(ValueError, self._call, stderr=None) - - def test_universal_newlines_error(self): - self.assertRaises(ValueError, self._call, universal_newlines=False) - - -class VerifyExeExistsTest(unittest.TestCase): - """Tests for certbot_postfix.util.verify_exe_exists.""" - - @classmethod - def _call(cls, *args, **kwargs): - from certbot_postfix.util import verify_exe_exists - return verify_exe_exists(*args, **kwargs) - - @mock.patch('certbot_postfix.util.certbot_util.exe_exists') - @mock.patch('certbot_postfix.util.plugins_util.path_surgery') - def test_failure(self, mock_exe_exists, mock_path_surgery): - mock_exe_exists.return_value = mock_path_surgery.return_value = False - self.assertRaises(errors.NoInstallationError, self._call, 'foo') - - @mock.patch('certbot_postfix.util.certbot_util.exe_exists') - def test_simple_success(self, mock_exe_exists): - mock_exe_exists.return_value = True - self._call('foo') - - @mock.patch('certbot_postfix.util.certbot_util.exe_exists') - @mock.patch('certbot_postfix.util.plugins_util.path_surgery') - def test_successful_surgery(self, mock_exe_exists, mock_path_surgery): - mock_exe_exists.return_value = False - mock_path_surgery.return_value = True - self._call('foo') - -class TestUtils(unittest.TestCase): - """ Testing random utility functions in util.py - """ - def test_report_master_overrides(self): - from certbot_postfix.util import report_master_overrides - self.assertRaises(errors.PluginError, report_master_overrides, 'name', - [('service/type', 'value')]) - # Shouldn't raise error - report_master_overrides('name', [('service/type', 'value')], - acceptable_overrides=('value',)) - - def test_no_acceptable_value(self): - from certbot_postfix.util import is_acceptable_value - self.assertFalse(is_acceptable_value('name', 'value', None)) - - def test_is_acceptable_value(self): - from certbot_postfix.util import is_acceptable_value - self.assertTrue(is_acceptable_value('name', 'value', ('value',))) - self.assertFalse(is_acceptable_value('name', 'bad', ('value',))) - - def test_is_acceptable_tuples(self): - from certbot_postfix.util import is_acceptable_value - self.assertTrue(is_acceptable_value('name', 'value', ('value', 'value1'))) - self.assertFalse(is_acceptable_value('name', 'bad', ('value', 'value1'))) - - def test_is_acceptable_protocols(self): - from certbot_postfix.util import is_acceptable_value - # SSLv2 and SSLv3 are both not supported, unambiguously - self.assertFalse(is_acceptable_value('tls_mandatory_protocols_lol', - 'SSLv2, SSLv3', None)) - self.assertFalse(is_acceptable_value('tls_protocols_lol', - 'SSLv2, SSLv3', None)) - self.assertFalse(is_acceptable_value('tls_protocols_lol', - '!SSLv2, !TLSv1', None)) - self.assertFalse(is_acceptable_value('tls_protocols_lol', - '!SSLv2, SSLv3, !SSLv3, ', None)) - self.assertTrue(is_acceptable_value('tls_protocols_lol', - '!SSLv2, !SSLv3', None)) - self.assertTrue(is_acceptable_value('tls_protocols_lol', - '!SSLv3, !TLSv1, !SSLv2', None)) - # TLSv1.2 is supported unambiguously - self.assertFalse(is_acceptable_value('tls_protocols_lol', - 'TLSv1, TLSv1.1,', None)) - self.assertFalse(is_acceptable_value('tls_protocols_lol', - 'TLSv1.2, !TLSv1.2,', None)) - self.assertTrue(is_acceptable_value('tls_protocols_lol', - 'TLSv1.2, ', None)) - self.assertTrue(is_acceptable_value('tls_protocols_lol', - 'TLSv1, TLSv1.1, TLSv1.2', None)) - -if __name__ == '__main__': # pragma: no cover - unittest.main() diff --git a/certbot-postfix/certbot_postfix/util.py b/certbot-postfix/certbot_postfix/util.py deleted file mode 100644 index f06989903..000000000 --- a/certbot-postfix/certbot_postfix/util.py +++ /dev/null @@ -1,292 +0,0 @@ -"""Utility functions for use in the Postfix installer.""" -import logging -import re -import subprocess - -from certbot import errors -from certbot import util as certbot_util -from certbot.plugins import util as plugins_util -from certbot_postfix import constants - -logger = logging.getLogger(__name__) - -COMMAND = "postfix" - -class PostfixUtilBase(object): - """A base class for wrapping Postfix command line utilities.""" - - def __init__(self, executable, config_dir=None): - """Sets up the Postfix utility class. - - :param str executable: name or path of the Postfix utility - :param str config_dir: path to an alternative Postfix config - - :raises .NoInstallationError: when the executable isn't found - - """ - self.executable = executable - verify_exe_exists(executable) - self._set_base_command(config_dir) - self.config_dir = None - - def _set_base_command(self, config_dir): - self._base_command = [self.executable] - if config_dir is not None: - self._base_command.extend(('-c', config_dir,)) - - def _call(self, extra_args=None): - """Runs the Postfix utility and returns the result. - - :param list extra_args: additional arguments for the command - - :returns: data written to stdout and stderr - :rtype: `tuple` of `str` - - :raises subprocess.CalledProcessError: if the command fails - - """ - args = list(self._base_command) - if extra_args is not None: - args.extend(extra_args) - return check_all_output(args) - - def _get_output(self, extra_args=None): - """Runs the Postfix utility and returns only stdout output. - - This function relies on self._call for running the utility. - - :param list extra_args: additional arguments for the command - - :returns: data written to stdout - :rtype: str - - :raises subprocess.CalledProcessError: if the command fails - - """ - return self._call(extra_args)[0] - -class PostfixUtil(PostfixUtilBase): - """Wrapper around Postfix CLI tool. - """ - - def __init__(self, config_dir=None): - super(PostfixUtil, self).__init__(COMMAND, config_dir) - - def test(self): - """Make sure the configuration is valid. - - :raises .MisconfigurationError: if the config is invalid - """ - try: - self._call(["check"]) - except subprocess.CalledProcessError as e: - logger.debug("Could not check postfix configuration:\n%s", e) - raise errors.MisconfigurationError( - "Postfix failed internal configuration check.") - - def restart(self): - """Restart or refresh the server content. - - :raises .PluginError: when server cannot be restarted - - """ - logger.info("Reloading Postfix configuration...") - if self._is_running(): - self._reload() - else: - self._start() - - - def _is_running(self): - """Is Postfix currently running? - - Uses the 'postfix status' command to determine if Postfix is - currently running using the specified configuration files. - - :returns: True if Postfix is running, otherwise, False - :rtype: bool - - """ - try: - self._call(["status"]) - except subprocess.CalledProcessError: - return False - return True - - def _start(self): - """Instructions Postfix to start running. - - :raises .PluginError: when Postfix cannot start - - """ - try: - self._call(["start"]) - except subprocess.CalledProcessError: - raise errors.PluginError("Postfix failed to start") - - def _reload(self): - """Instructs Postfix to reload its configuration. - - If Postfix isn't currently running, this method will fail. - - :raises .PluginError: when Postfix cannot reload - """ - try: - self._call(["reload"]) - except subprocess.CalledProcessError: - raise errors.PluginError( - "Postfix failed to reload its configuration") - -def check_all_output(*args, **kwargs): - """A version of subprocess.check_output that also captures stderr. - - This is the same as :func:`subprocess.check_output` except output - written to stderr is also captured and returned to the caller. The - return value is a tuple of two strings (rather than byte strings). - To accomplish this, the caller cannot set the stdout, stderr, or - universal_newlines parameters to :class:`subprocess.Popen`. - - Additionally, if the command exits with a nonzero status, output is - not included in the raised :class:`subprocess.CalledProcessError` - because Python 2.6 does not support this. Instead, the failure - including the output is logged. - - :param tuple args: positional arguments for Popen - :param dict kwargs: keyword arguments for Popen - - :returns: data written to stdout and stderr - :rtype: `tuple` of `str` - - :raises ValueError: if arguments are invalid - :raises subprocess.CalledProcessError: if the command fails - - """ - for keyword in ('stdout', 'stderr', 'universal_newlines',): - if keyword in kwargs: - raise ValueError( - keyword + ' argument not allowed, it will be overridden.') - - kwargs['stdout'] = subprocess.PIPE - kwargs['stderr'] = subprocess.PIPE - kwargs['universal_newlines'] = True - - process = subprocess.Popen(*args, **kwargs) - output, err = process.communicate() - retcode = process.poll() - if retcode: - cmd = kwargs.get('args') - if cmd is None: - cmd = args[0] - logger.debug( - "'%s' exited with %d. stdout output was:\n%s\nstderr output was:\n%s", - cmd, retcode, output, err) - raise subprocess.CalledProcessError(retcode, cmd) - return (output, err) - - -def verify_exe_exists(exe, message=None): - """Ensures an executable with the given name is available. - - If an executable isn't found for the given path or name, extra - directories are added to the user's PATH to help find system - utilities that may not be available in the default cron PATH. - - :param str exe: executable path or name - :param str message: Error message to print. - - :raises .NoInstallationError: when the executable isn't found - - """ - if message is None: - message = "Cannot find executable '{0}'.".format(exe) - if not (certbot_util.exe_exists(exe) or plugins_util.path_surgery(exe)): - raise errors.NoInstallationError(message) - -def report_master_overrides(name, overrides, acceptable_overrides=None): - """If the value for a parameter `name` is overridden by other services, - report a warning to notify the user. If `parameter` is a TLS version parameter - (i.e., `parameter` contains 'tls_protocols' or 'tls_mandatory_protocols'), then - `acceptable_overrides` isn't used each value in overrides is inspected for secure TLS - versions. - - :param str name: The name of the parameter that is being overridden. - :param list overrides: The values that other services are setting for `name`. - Each override is a tuple: (service name, value) - :param tuple acceptable_overrides: Override values that are acceptable. For instance, if - another service is overriding our parameter with a more secure option, we don't have - to warn. If this is set to None, errors are raised for *any* overrides of `name`! - """ - error_string = "" - for override in overrides: - service, value = override - # If this override is acceptable: - if acceptable_overrides is not None and \ - is_acceptable_value(name, value, acceptable_overrides): - continue - error_string += " {0}: {1}\n".format(service, value) - if error_string: - raise errors.PluginError("{0} is overridden with less secure options by the " - "following services in master.cf:\n".format(name) + error_string) - - -def is_acceptable_value(parameter, value, acceptable=None): - """ Returns whether the `value` for this `parameter` is acceptable, - given a tuple of `acceptable` values. If `parameter` is a TLS version parameter - (i.e., `parameter` contains 'tls_protocols' or 'tls_mandatory_protocols'), then - `acceptable` isn't used and `value` is inspected for secure TLS versions. - - :param str parameter: The name of the parameter being set. - :param str value: Proposed new value for parameter. - :param tuple acceptable: List of acceptable values for parameter. - """ - # Check if param value is a comma-separated list of protocols. - # Otherwise, just check whether the value is in the acceptable list. - if 'tls_protocols' in parameter or 'tls_mandatory_protocols' in parameter: - return _has_acceptable_tls_versions(value) - if acceptable is not None: - return value in acceptable - return False - - -def _has_acceptable_tls_versions(parameter_string): - """ - Checks to see if the list of TLS protocols is acceptable. - This requires that TLSv1.2 is supported, and neither SSLv2 nor SSLv3 are supported. - - Should be a string of protocol names delimited by commas, spaces, or colons. - - Postfix's documents suggest listing protocols to exclude, like "!SSLv2, !SSLv3". - Listing the protocols to include, like "TLSv1, TLSv1.1, TLSv1.2" is okay as well, - though not recommended - - When these two modes are interspersed, the presence of a single non-negated protocol name - (i.e. "TLSv1" rather than "!TLSv1") automatically excludes all other unnamed protocols. - - In addition, the presence of both a protocol name inclusion and exclusion isn't explicitly - documented, so this method should return False if it encounters contradicting statements - about TLSv1.2, SSLv2, or SSLv3. (for instance, "SSLv3, !SSLv3"). - """ - if not parameter_string: - return False - bad_versions = list(constants.TLS_VERSIONS) - for version in constants.ACCEPTABLE_TLS_VERSIONS: - del bad_versions[bad_versions.index(version)] - supported_version_list = re.split("[, :]+", parameter_string) - # The presence of any non-"!" protocol listing excludes the others by default. - inclusion_list = False - for version in supported_version_list: - if not version: - continue - if version in bad_versions: # short-circuit if we recognize any bad version - return False - if version[0] != "!": - inclusion_list = True - if inclusion_list: # For any inclusion list, we still require TLS 1.2. - if "TLSv1.2" not in supported_version_list or "!TLSv1.2" in supported_version_list: - return False - else: - for bad_version in bad_versions: - if "!" + bad_version not in supported_version_list: - return False - return True - diff --git a/certbot-postfix/docs/.gitignore b/certbot-postfix/docs/.gitignore deleted file mode 100644 index ba65b13af..000000000 --- a/certbot-postfix/docs/.gitignore +++ /dev/null @@ -1 +0,0 @@ -/_build/ diff --git a/certbot-postfix/docs/Makefile b/certbot-postfix/docs/Makefile deleted file mode 100644 index 717ff654f..000000000 --- a/certbot-postfix/docs/Makefile +++ /dev/null @@ -1,20 +0,0 @@ -# Minimal makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -SPHINXPROJ = certbot-postfix -SOURCEDIR = . -BUILDDIR = _build - -# Put it first so that "make" without argument is like "make help". -help: - @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) - -.PHONY: help Makefile - -# Catch-all target: route all unknown targets to Sphinx using the new -# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). -%: Makefile - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) \ No newline at end of file diff --git a/certbot-postfix/docs/api.rst b/certbot-postfix/docs/api.rst deleted file mode 100644 index 8668ec5d8..000000000 --- a/certbot-postfix/docs/api.rst +++ /dev/null @@ -1,8 +0,0 @@ -================= -API Documentation -================= - -.. toctree:: - :glob: - - api/** diff --git a/certbot-postfix/docs/api/installer.rst b/certbot-postfix/docs/api/installer.rst deleted file mode 100644 index 121d58d5b..000000000 --- a/certbot-postfix/docs/api/installer.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_postfix.installer` --------------------------------------- - -.. automodule:: certbot_postfix.installer - :members: diff --git a/certbot-postfix/docs/api/postconf.rst b/certbot-postfix/docs/api/postconf.rst deleted file mode 100644 index 917150e45..000000000 --- a/certbot-postfix/docs/api/postconf.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot_postfix.postconf` -------------------------------- - -.. automodule:: certbot_postfix.postconf - :members: diff --git a/certbot-postfix/docs/conf.py b/certbot-postfix/docs/conf.py deleted file mode 100644 index 51d99aab5..000000000 --- a/certbot-postfix/docs/conf.py +++ /dev/null @@ -1,190 +0,0 @@ -# -*- coding: utf-8 -*- -# -# Configuration file for the Sphinx documentation builder. -# -# This file does only contain a selection of the most common options. For a -# full list see the documentation: -# http://www.sphinx-doc.org/en/master/config - -# -- Path setup -------------------------------------------------------------- - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -# -import os -# import sys -# sys.path.insert(0, os.path.abspath('.')) - - -# -- Project information ----------------------------------------------------- - -project = u'certbot-postfix' -copyright = u'2018, Certbot Project' -author = u'Certbot Project' - -# The short X.Y version -version = u'0' -# The full version, including alpha/beta/rc tags -release = u'0' - - -# -- General configuration --------------------------------------------------- - -# If your documentation needs a minimal Sphinx version, state it here. -# -needs_sphinx = '1.0' - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom -# ones. -extensions = [ - 'sphinx.ext.autodoc', - 'sphinx.ext.intersphinx', - 'sphinx.ext.todo', - 'sphinx.ext.coverage', - 'sphinx.ext.viewcode', -] - -autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# The suffix(es) of source filenames. -# You can specify multiple suffix as a list of string: -# -# source_suffix = ['.rst', '.md'] -source_suffix = '.rst' - -# The master toctree document. -master_doc = 'index' - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -# -# This is also used if you do content translation via gettext catalogs. -# Usually you set "language" from the command line for these cases. -language = u'en' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -# This pattern also affects html_static_path and html_extra_path . -exclude_patterns = [u'_build', 'Thumbs.db', '.DS_Store'] - -default_role = 'py:obj' - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - - -# -- Options for HTML output ------------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. -# - -# http://docs.readthedocs.org/en/latest/theme.html#how-do-i-use-this-locally-and-on-read-the-docs -# on_rtd is whether we are on readthedocs.org -on_rtd = os.environ.get('READTHEDOCS', None) == 'True' -if not on_rtd: # only import and set the theme if we're building docs locally - import sphinx_rtd_theme - html_theme = 'sphinx_rtd_theme' - html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] -# otherwise, readthedocs.org uses their theme by default, so no need to specify it - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -# -# html_theme_options = {} - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -# html_static_path = ['_static'] - -# Custom sidebar templates, must be a dictionary that maps document names -# to template names. -# -# The default sidebars (for documents that don't match any pattern) are -# defined by theme itself. Builtin themes are using these templates by -# default: ``['localtoc.html', 'relations.html', 'sourcelink.html', -# 'searchbox.html']``. -# -# html_sidebars = {} - - -# -- Options for HTMLHelp output --------------------------------------------- - -# Output file base name for HTML help builder. -htmlhelp_basename = 'certbot-postfixdoc' - - -# -- Options for LaTeX output ------------------------------------------------ - -latex_elements = { - # The paper size ('letterpaper' or 'a4paper'). - # - # 'papersize': 'letterpaper', - - # The font size ('10pt', '11pt' or '12pt'). - # - # 'pointsize': '10pt', - - # Additional stuff for the LaTeX preamble. - # - # 'preamble': '', - - # Latex figure (float) alignment - # - # 'figure_align': 'htbp', -} - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, -# author, documentclass [howto, manual, or own class]). -latex_documents = [ - (master_doc, 'certbot-postfix.tex', u'certbot-postfix Documentation', - u'Certbot Project', 'manual'), -] - - -# -- Options for manual page output ------------------------------------------ - -# One entry per manual page. List of tuples -# (source start file, name, description, authors, manual section). -man_pages = [ - (master_doc, 'certbot-postfix', u'certbot-postfix Documentation', - [author], 1) -] - - -# -- Options for Texinfo output ---------------------------------------------- - -# Grouping the document tree into Texinfo files. List of tuples -# (source start file, target name, title, author, -# dir menu entry, description, category) -texinfo_documents = [ - (master_doc, 'certbot-postfix', u'certbot-postfix Documentation', - author, 'certbot-postfix', 'One line description of project.', - 'Miscellaneous'), -] - - -# -- Extension configuration ------------------------------------------------- - -# -- Options for intersphinx extension --------------------------------------- - -# Example configuration for intersphinx: refer to the Python standard library. -intersphinx_mapping = { - 'python': ('https://docs.python.org/', None), - 'acme': ('https://acme-python.readthedocs.org/en/latest/', None), - 'certbot': ('https://certbot.eff.org/docs/', None), -} - -# -- Options for todo extension ---------------------------------------------- - -# If true, `todo` and `todoList` produce output, else they produce nothing. -todo_include_todos = True diff --git a/certbot-postfix/docs/index.rst b/certbot-postfix/docs/index.rst deleted file mode 100644 index 3d6697bcb..000000000 --- a/certbot-postfix/docs/index.rst +++ /dev/null @@ -1,28 +0,0 @@ -.. certbot-postfix documentation master file, created by - sphinx-quickstart on Wed May 2 16:01:06 2018. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to certbot-postfix's documentation! -=========================================== - -.. toctree:: - :maxdepth: 2 - :caption: Contents: - -.. automodule:: certbot_postfix - :members: - -.. toctree:: - :maxdepth: 1 - - api - - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` diff --git a/certbot-postfix/docs/make.bat b/certbot-postfix/docs/make.bat deleted file mode 100644 index 23fbdc93c..000000000 --- a/certbot-postfix/docs/make.bat +++ /dev/null @@ -1,36 +0,0 @@ -@ECHO OFF - -pushd %~dp0 - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set SOURCEDIR=. -set BUILDDIR=_build -set SPHINXPROJ=certbot-postfix - -if "%1" == "" goto help - -%SPHINXBUILD% >NUL 2>NUL -if errorlevel 9009 ( - echo. - echo.The 'sphinx-build' command was not found. Make sure you have Sphinx - echo.installed, then set the SPHINXBUILD environment variable to point - echo.to the full path of the 'sphinx-build' executable. Alternatively you - echo.may add the Sphinx directory to PATH. - echo. - echo.If you don't have Sphinx installed, grab it from - echo.http://sphinx-doc.org/ - exit /b 1 -) - -%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% -goto end - -:help -%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% - -:end -popd diff --git a/certbot-postfix/local-oldest-requirements.txt b/certbot-postfix/local-oldest-requirements.txt deleted file mode 100644 index bc0cdbf00..000000000 --- a/certbot-postfix/local-oldest-requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -acme[dev]==0.25.0 -certbot[dev]==0.23.0 diff --git a/certbot-postfix/setup.cfg b/certbot-postfix/setup.cfg deleted file mode 100644 index 2a9acf13d..000000000 --- a/certbot-postfix/setup.cfg +++ /dev/null @@ -1,2 +0,0 @@ -[bdist_wheel] -universal = 1 diff --git a/certbot-postfix/setup.py b/certbot-postfix/setup.py deleted file mode 100644 index 0ff2908df..000000000 --- a/certbot-postfix/setup.py +++ /dev/null @@ -1,64 +0,0 @@ -from setuptools import setup -from setuptools import find_packages - - -version = '0.26.0.dev0' - -install_requires = [ - 'acme>=0.25.0', - 'certbot>=0.23.0', - 'setuptools', - 'six', - 'zope.component', - 'zope.interface', -] - -docs_extras = [ - 'Sphinx>=1.0', # autodoc_member_order = 'bysource', autodoc_default_flags - 'sphinx_rtd_theme', -] - -setup( - name='certbot-postfix', - version=version, - description="Postfix plugin for Certbot", - url='https://github.com/certbot/certbot', - author="Certbot Project", - author_email='client-dev@letsencrypt.org', - license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', - classifiers=[ - 'Development Status :: 3 - Alpha', - 'Environment :: Plugins', - 'Intended Audience :: System Administrators', - 'License :: OSI Approved :: Apache Software License', - 'Operating System :: POSIX :: Linux', - 'Programming Language :: Python', - 'Programming Language :: Python :: 2', - 'Programming Language :: Python :: 2.7', - 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', - 'Programming Language :: Python :: 3.5', - 'Programming Language :: Python :: 3.6', - 'Programming Language :: Python :: 3.7', - 'Topic :: Communications :: Email :: Mail Transport Agents', - 'Topic :: Security', - 'Topic :: System :: Installation/Setup', - 'Topic :: System :: Networking', - 'Topic :: System :: Systems Administration', - 'Topic :: Utilities', - ], - - packages=find_packages(), - include_package_data=True, - install_requires=install_requires, - extras_require={ - 'docs': docs_extras, - }, - entry_points={ - 'certbot.plugins': [ - 'postfix = certbot_postfix:Installer', - ], - }, - test_suite='certbot_postfix', -) diff --git a/CHANGELOG.md b/certbot/CHANGELOG.md similarity index 77% rename from CHANGELOG.md rename to certbot/CHANGELOG.md index ef91f1a8f..7367c929f 100644 --- a/CHANGELOG.md +++ b/certbot/CHANGELOG.md @@ -2,21 +2,496 @@ Certbot adheres to [Semantic Versioning](https://semver.org/). -## 0.31.0 - master +## 1.3.0 - master ### Added -* Avoid to process again challenges that are already validated - when a certificate is issued. -* Support for initiating (but not solving end-to-end) TLS-ALPN-01 challenges - with the `acme` module. * Don't verify the existing certificate in HTTP01Response.simple_verify, for compatibility with the real-world ACME challenge checks. ### Changed +* + +### Fixed + +* + +More details about these changes can be found on our GitHub repo. + +## 1.2.0 - 2020-02-04 + +### Added + +* Added support for Cloudflare's limited-scope API Tokens + +### Changed + +* Add directory field to error message when field is missing. +* If MD5 hasher is not available, try it in non-security mode (fix for FIPS systems) -- [#1948](https://github.com/certbot/certbot/issues/1948) +* Disable old SSL versions and ciphersuites and remove `SSLCompression off` setting to follow Mozilla recommendations in Apache. +* Remove ECDHE-RSA-AES128-SHA from NGINX ciphers list now that Windows 2008 R2 and Windows 7 are EOLed +* Support for Python 3.4 has been removed. + +### Fixed + +* Fix collections.abc imports for Python 3.9. + +More details about these changes can be found on our GitHub repo. + +## 1.1.0 - 2020-01-14 + +### Added + +* + +### Changed + +* Removed the fallback introduced with 0.34.0 in `acme` to retry a POST-as-GET + request as a GET request when the targeted ACME CA server seems to not support + POST-as-GET requests. +* certbot-auto no longer supports architectures other than x86_64 on RHEL 6 + based systems. Existing certbot-auto installations affected by this will + continue to work, but they will no longer receive updates. To install a + newer version of Certbot on these systems, you should update your OS. +* Support for Python 3.4 in Certbot and its ACME library is deprecated and will be + removed in the next release of Certbot. certbot-auto users on x86_64 systems running + RHEL 6 or derivatives will be asked to enable Software Collections (SCL) repository + so Python 3.6 can be installed. certbot-auto can enable the SCL repo for you on CentOS 6 + while users on other RHEL 6 based systems will be asked to do this manually. + +### Fixed + +* + +More details about these changes can be found on our GitHub repo. + +## 1.0.0 - 2019-12-03 + +### Added + +* + +### Removed + +* The `docs` extras for the `certbot-apache` and `certbot-nginx` packages + have been removed. + +### Changed + +* certbot-auto has deprecated support for systems using OpenSSL 1.0.1 that are + not running on x86-64. This primarily affects RHEL 6 based systems. +* Certbot's `config_changes` subcommand has been removed +* `certbot.plugins.common.TLSSNI01` has been removed. +* Deprecated attributes related to the TLS-SNI-01 challenge in + `acme.challenges` and `acme.standalone` + have been removed. +* The functions `certbot.client.view_config_changes`, + `certbot.main.config_changes`, + `certbot.plugins.common.Installer.view_config_changes`, + `certbot.reverter.Reverter.view_config_changes`, and + `certbot.util.get_systemd_os_info` have been removed +* Certbot's `register --update-registration` subcommand has been removed +* When possible, default to automatically configuring the webserver so all requests + redirect to secure HTTPS access. This is mostly relevant when running Certbot + in non-interactive mode. Previously, the default was to not redirect all requests. + +### Fixed + +* + +More details about these changes can be found on our GitHub repo. + +## 0.40.1 - 2019-11-05 + +### Changed + +* Added back support for Python 3.4 to Certbot components and certbot-auto due + to a bug when requiring Python 2.7 or 3.5+ on RHEL 6 based systems. + +More details about these changes can be found on our GitHub repo. + +## 0.40.0 - 2019-11-05 + +### Added + +* + +### Changed + +* We deprecated support for Python 3.4 in Certbot and its ACME library. Support + for Python 3.4 will be removed in the next major release of Certbot. + certbot-auto users on RHEL 6 based systems will be asked to enable Software + Collections (SCL) repository so Python 3.6 can be installed. certbot-auto can + enable the SCL repo for you on CentOS 6 while users on other RHEL 6 based + systems will be asked to do this manually. +* `--server` may now be combined with `--dry-run`. Certbot will, as before, use the + staging server instead of the live server when `--dry-run` is used. +* `--dry-run` now requests fresh authorizations every time, fixing the issue + where it was prone to falsely reporting success. +* Updated certbot-dns-google to depend on newer versions of + google-api-python-client and oauth2client. +* The OS detection logic again uses distro library for Linux OSes +* certbot.plugins.common.TLSSNI01 has been deprecated and will be removed in a + future release. +* CLI flags --tls-sni-01-port and --tls-sni-01-address have been removed. +* The values tls-sni and tls-sni-01 for the --preferred-challenges flag are no + longer accepted. +* Removed the flags: `--agree-dev-preview`, `--dialog`, and `--apache-init-script` +* acme.standalone.BaseRequestHandlerWithLogging and + acme.standalone.simple_tls_sni_01_server have been deprecated and will be + removed in a future release of the library. +* certbot-dns-rfc2136 now use TCP to query SOA records. + +### Fixed + +* + +More details about these changes can be found on our GitHub repo. + +## 0.39.0 - 2019-10-01 + +### Added + +* Support for Python 3.8 was added to Certbot and all of its components. +* Support for CentOS 8 was added to certbot-auto. + +### Changed + +* Don't send OCSP requests for expired certificates +* Return to using platform.linux_distribution instead of distro.linux_distribution in OS fingerprinting for Python < 3.8 +* Updated the Nginx plugin's TLS configuration to keep support for some versions of IE11. + +### Fixed + +* Fixed OS detection in the Apache plugin on RHEL 6. + +More details about these changes can be found on our GitHub repo. + +## 0.38.0 - 2019-09-03 + +### Added + +* Disable session tickets for Nginx users when appropriate. + +### Changed + +* If Certbot fails to rollback your server configuration, the error message + links to the Let's Encrypt forum. Change the link to the Help category now + that the Server category has been closed. +* Replace platform.linux_distribution with distro.linux_distribution as a step + towards Python 3.8 support in Certbot. + +### Fixed + +* Fixed OS detection in the Apache plugin on Scientific Linux. + +More details about these changes can be found on our GitHub repo. + +## 0.37.2 - 2019-08-21 + +* Stop disabling TLS session tickets in Nginx as it caused TLS failures on + some systems. + +More details about these changes can be found on our GitHub repo. + +## 0.37.1 - 2019-08-08 + +### Fixed + +* Stop disabling TLS session tickets in Apache as it caused TLS failures on + some systems. + +More details about these changes can be found on our GitHub repo. + +## 0.37.0 - 2019-08-07 + +### Added + +* Turn off session tickets for apache plugin by default +* acme: Authz deactivation added to `acme` module. + +### Changed + +* Follow updated Mozilla recommendations for Nginx ssl_protocols, ssl_ciphers, + and ssl_prefer_server_ciphers + +### Fixed + +* Fix certbot-auto failures on RHEL 8. + +More details about these changes can be found on our GitHub repo. + +## 0.36.0 - 2019-07-11 + +### Added + +* Turn off session tickets for nginx plugin by default +* Added missing error types from RFC8555 to acme + +### Changed + +* Support for Ubuntu 14.04 Trusty has been removed. +* Update the 'manage your account' help to be more generic. +* The error message when Certbot's Apache plugin is unable to modify your + Apache configuration has been improved. +* Certbot's config_changes subcommand has been deprecated and will be + removed in a future release. +* `certbot config_changes` no longer accepts a --num parameter. +* The functions `certbot.plugins.common.Installer.view_config_changes` and + `certbot.reverter.Reverter.view_config_changes` have been deprecated and will + be removed in a future release. + +### Fixed + +* Replace some unnecessary platform-specific line separation. + +More details about these changes can be found on our GitHub repo. + +## 0.35.1 - 2019-06-10 + +### Fixed + +* Support for specifying an authoritative base domain in our dns-rfc2136 plugin + has been removed. This feature was added in our last release but had a bug + which caused the plugin to fail so the feature has been removed until it can + be added properly. + +Despite us having broken lockstep, we are continuing to release new versions of +all Certbot components during releases for the time being, however, the only +package with changes other than its version number was: + +* certbot-dns-rfc2136 + +More details about these changes can be found on our GitHub repo. + +## 0.35.0 - 2019-06-05 + +### Added + +* dns_rfc2136 plugin now supports explicitly specifying an authoritative + base domain for cases when the automatic method does not work (e.g. + Split horizon DNS) + +### Changed + +* + +### Fixed + +* Renewal parameter `webroot_path` is always saved, avoiding some regressions + when `webroot` authenticator plugin is invoked with no challenge to perform. +* Certbot now accepts OCSP responses when an explicit authorized + responder, different from the issuer, is used to sign OCSP + responses. +* Scripts in Certbot hook directories are no longer executed when their + filenames end in a tilde. + +Despite us having broken lockstep, we are continuing to release new versions of +all Certbot components during releases for the time being, however, the only +package with changes other than its version number was: + +* certbot +* certbot-dns-rfc2136 + +More details about these changes can be found on our GitHub repo. + +## 0.34.2 - 2019-05-07 + +### Fixed + +* certbot-auto no longer writes a check_permissions.py script at the root + of the filesystem. + +Despite us having broken lockstep, we are continuing to release new versions of +all Certbot components during releases for the time being, however, the only +changes in this release were to certbot-auto. + +More details about these changes can be found on our GitHub repo. + +## 0.34.1 - 2019-05-06 + +### Fixed + +* certbot-auto no longer prints a blank line when there are no permissions + problems. + +Despite us having broken lockstep, we are continuing to release new versions of +all Certbot components during releases for the time being, however, the only +changes in this release were to certbot-auto. + +More details about these changes can be found on our GitHub repo. + +## 0.34.0 - 2019-05-01 + +### Changed + +* Apache plugin now tries to restart httpd on Fedora using systemctl if a + configuration test error is detected. This has to be done due to the way + Fedora now generates the self signed certificate files upon first + restart. +* Updated Certbot and its plugins to improve the handling of file system permissions + on Windows as a step towards adding proper Windows support to Certbot. +* Updated urllib3 to 1.24.2 in certbot-auto. +* Removed the fallback introduced with 0.32.0 in `acme` to retry a challenge response + with a `keyAuthorization` if sending the response without this field caused a + `malformed` error to be received from the ACME server. +* Linode DNS plugin now supports api keys created from their new panel + at [cloud.linode.com](https://cloud.linode.com) + +### Fixed + +* Fixed Google DNS Challenge issues when private zones exist +* Adding a warning noting that future versions of Certbot will automatically configure the + webserver so that all requests redirect to secure HTTPS access. You can control this + behavior and disable this warning with the --redirect and --no-redirect flags. +* certbot-auto now prints warnings when run as root with insecure file system + permissions. If you see these messages, you should fix the problem by + following the instructions at + https://community.letsencrypt.org/t/certbot-auto-deployment-best-practices/91979/, + however, these warnings can be disabled as necessary with the flag + --no-permissions-check. +* `acme` module uses now a POST-as-GET request to retrieve the registration + from an ACME v2 server +* Convert the tsig algorithm specified in the certbot_dns_rfc2136 configuration file to + all uppercase letters before validating. This makes the value in the config case + insensitive. + +Despite us having broken lockstep, we are continuing to release new versions of +all Certbot components during releases for the time being, however, the only +package with changes other than its version number was: + +* acme +* certbot +* certbot-apache +* certbot-dns-cloudflare +* certbot-dns-cloudxns +* certbot-dns-digitalocean +* certbot-dns-dnsimple +* certbot-dns-dnsmadeeasy +* certbot-dns-gehirn +* certbot-dns-google +* certbot-dns-linode +* certbot-dns-luadns +* certbot-dns-nsone +* certbot-dns-ovh +* certbot-dns-rfc2136 +* certbot-dns-route53 +* certbot-dns-sakuracloud +* certbot-nginx + +More details about these changes can be found on our GitHub repo. + +## 0.33.1 - 2019-04-04 + +### Fixed + +* A bug causing certbot-auto to print warnings or crash on some RHEL based + systems has been resolved. + +Despite us having broken lockstep, we are continuing to release new versions of +all Certbot components during releases for the time being, however, the only +changes in this release were to certbot-auto. + +More details about these changes can be found on our GitHub repo. + +## 0.33.0 - 2019-04-03 + +### Added + +* Fedora 29+ is now supported by certbot-auto. Since Python 2.x is on a deprecation + path in Fedora, certbot-auto will install and use Python 3.x on Fedora 29+. +* CLI flag `--https-port` has been added for Nginx plugin exclusively, and replaces + `--tls-sni-01-port`. It defines the HTTPS port the Nginx plugin will use while + setting up a new SSL vhost. By default the HTTPS port is 443. + +### Changed + +* Support for TLS-SNI-01 has been removed from all official Certbot plugins. +* Attributes related to the TLS-SNI-01 challenge in `acme.challenges` and `acme.standalone` + modules are deprecated and will be removed soon. +* CLI flags `--tls-sni-01-port` and `--tls-sni-01-address` are now no-op, will + generate a deprecation warning if used, and will be removed soon. +* Options `tls-sni` and `tls-sni-01` in `--preferred-challenges` flag are now no-op, + will generate a deprecation warning if used, and will be removed soon. +* CLI flag `--standalone-supported-challenges` has been removed. + +### Fixed + +* Certbot uses the Python library cryptography for OCSP when cryptography>=2.5 + is installed. We fixed a bug in Certbot causing it to interpret timestamps in + the OCSP response as being in the local timezone rather than UTC. +* Issue causing the default CentOS 6 TLS configuration to ignore some of the + HTTPS VirtualHosts created by Certbot. mod_ssl loading is now moved to main + http.conf for this environment where possible. + +Despite us having broken lockstep, we are continuing to release new versions of +all Certbot components during releases for the time being, however, the only +package with changes other than its version number was: + +* acme +* certbot +* certbot-apache +* certbot-nginx + +More details about these changes can be found on our GitHub repo. + +## 0.32.0 - 2019-03-06 + +### Added + +* If possible, Certbot uses built-in support for OCSP from recent cryptography + versions instead of the OpenSSL binary: as a consequence Certbot does not need + the OpenSSL binary to be installed anymore if cryptography>=2.5 is installed. + +### Changed + +* Certbot and its acme module now depend on josepy>=1.1.0 to avoid printing the + warnings described at https://github.com/certbot/josepy/issues/13. +* Apache plugin now respects CERTBOT_DOCS environment variable when adding + command line defaults. +* The running of manual plugin hooks is now always included in Certbot's log + output. +* Tests execution for certbot, certbot-apache and certbot-nginx packages now relies on pytest. +* An ACME CA server may return a "Retry-After" HTTP header on authorization polling, as + specified in the ACME protocol, to indicate when the next polling should occur. Certbot now + reads this header if set and respect its value. +* The `acme` module avoids sending the `keyAuthorization` field in the JWS + payload when responding to a challenge as the field is not included in the + current ACME protocol. To ease the migration path for ACME CA servers, + Certbot and its `acme` module will first try the request without the + `keyAuthorization` field but will temporarily retry the request with the + field included if a `malformed` error is received. This fallback will be + removed in version 0.34.0. + +Despite us having broken lockstep, we are continuing to release new versions of +all Certbot components during releases for the time being, however, the only +package with changes other than its version number was: + +* acme +* certbot +* certbot-apache +* certbot-nginx + +More details about these changes can be found on our GitHub repo. + +## 0.31.0 - 2019-02-07 + +### Added + +* Avoid reprocessing challenges that are already validated + when a certificate is issued. +* Support for initiating (but not solving end-to-end) TLS-ALPN-01 challenges + with the `acme` module. + +### Changed + +* Certbot's official Docker images are now based on Alpine Linux 3.9 rather + than 3.7. The new version comes with OpenSSL 1.1.1. * Lexicon-based DNS plugins are now fully compatible with Lexicon 3.x (support on 2.x branch is maintained). +* Apache plugin now attempts to configure all VirtualHosts matching requested + domain name instead of only a single one when answering the HTTP-01 challenge. ### Fixed @@ -30,6 +505,7 @@ package with changes other than its version number was: * acme * certbot +* certbot-apache * certbot-dns-cloudxns * certbot-dns-dnsimple * certbot-dns-dnsmadeeasy @@ -177,7 +653,7 @@ https://github.com/certbot/certbot/milestone/62?closed=1 * Log warning about TLS-SNI deprecation in Certbot * Stop preferring TLS-SNI in the Apache, Nginx, and standalone plugins * OVH DNS plugin now relies on Lexicon>=2.7.14 to support HTTP proxies -* Default time the Linode plugin waits for DNS changes to propogate is now 1200 seconds. +* Default time the Linode plugin waits for DNS changes to propagate is now 1200 seconds. ### Fixed @@ -296,7 +772,7 @@ https://github.com/certbot/certbot/milestone/58?closed=1 increased over time. The max-age value is not increased to a large value until you've successfully managed to renew your certificate. This enhancement can be requested with the --auto-hsts flag. -* New official DNS plugins have been created for Gehirn Infrastracture Service, +* New official DNS plugins have been created for Gehirn Infrastructure Service, Linode, OVH, and Sakura Cloud. These plugins can be found on our Docker Hub page at https://hub.docker.com/u/certbot and on PyPI. * The ability to reuse ACME accounts from Let's Encrypt's ACMEv1 endpoint on diff --git a/certbot-postfix/LICENSE.txt b/certbot/LICENSE.txt similarity index 88% rename from certbot-postfix/LICENSE.txt rename to certbot/LICENSE.txt index c8314fd1c..b905dd120 100644 --- a/certbot-postfix/LICENSE.txt +++ b/certbot/LICENSE.txt @@ -1,17 +1,14 @@ - Copyright 2017 Electronic Frontier Foundation and others +Certbot ACME Client +Copyright (c) Electronic Frontier Foundation and others +Licensed Apache Version 2.0 - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at +The nginx plugin incorporates code from nginxparser +Copyright (c) 2014 Fatih Erikli +Licensed MIT - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. +Text of Apache License +====================== Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ @@ -187,4 +184,22 @@ incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. - END OF TERMS AND CONDITIONS + +Text of MIT License +=================== +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/MANIFEST.in b/certbot/MANIFEST.in similarity index 70% rename from MANIFEST.in rename to certbot/MANIFEST.in index 7f529c7a7..ef91a3e7c 100644 --- a/MANIFEST.in +++ b/certbot/MANIFEST.in @@ -1,9 +1,10 @@ include README.rst include CHANGELOG.md -include CONTRIBUTING.md include LICENSE.txt -include linter_plugin.py recursive-include docs * recursive-include examples * recursive-include certbot/tests/testdata * +recursive-include tests *.py include certbot/ssl-dhparams.pem +global-exclude __pycache__ +global-exclude *.py[cod] diff --git a/certbot/README.rst b/certbot/README.rst new file mode 100644 index 000000000..2c934ce59 --- /dev/null +++ b/certbot/README.rst @@ -0,0 +1,131 @@ +.. This file contains a series of comments that are used to include sections of this README in other files. Do not modify these comments unless you know what you are doing. tag:intro-begin + +Certbot is part of EFF’s effort to encrypt the entire Internet. Secure communication over the Web relies on HTTPS, which requires the use of a digital certificate that lets browsers verify the identity of web servers (e.g., is that really google.com?). Web servers obtain their certificates from trusted third parties called certificate authorities (CAs). Certbot is an easy-to-use client that fetches a certificate from Let’s Encrypt—an open certificate authority launched by the EFF, Mozilla, and others—and deploys it to a web server. + +Anyone who has gone through the trouble of setting up a secure website knows what a hassle getting and maintaining a certificate is. Certbot and Let’s Encrypt can automate away the pain and let you turn on and manage HTTPS with simple commands. Using Certbot and Let's Encrypt is free, so there’s no need to arrange payment. + +How you use Certbot depends on the configuration of your web server. The best way to get started is to use our `interactive guide `_. It generates instructions based on your configuration settings. In most cases, you’ll need `root or administrator access `_ to your web server to run Certbot. + +Certbot is meant to be run directly on your web server, not on your personal computer. If you’re using a hosted service and don’t have direct access to your web server, you might not be able to use Certbot. Check with your hosting provider for documentation about uploading certificates or using certificates issued by Let’s Encrypt. + +Certbot is a fully-featured, extensible client for the Let's +Encrypt CA (or any other CA that speaks the `ACME +`_ +protocol) that can automate the tasks of obtaining certificates and +configuring webservers to use them. This client runs on Unix-based operating +systems. + +To see the changes made to Certbot between versions please refer to our +`changelog `_. + +Until May 2016, Certbot was named simply ``letsencrypt`` or ``letsencrypt-auto``, +depending on install method. Instructions on the Internet, and some pieces of the +software, may still refer to this older name. + +Contributing +------------ + +If you'd like to contribute to this project please read `Developer Guide +`_. + +This project is governed by `EFF's Public Projects Code of Conduct `_. + +.. _installation: + +How to run the client +--------------------- + +The easiest way to install and run Certbot is by visiting `certbot.eff.org`_, +where you can find the correct instructions for many web server and OS +combinations. For more information, see `Get Certbot +`_. + +.. _certbot.eff.org: https://certbot.eff.org/ + +Understanding the client in more depth +-------------------------------------- + +To understand what the client is doing in detail, it's important to +understand the way it uses plugins. Please see the `explanation of +plugins `_ in +the User Guide. + +Links +===== + +.. Do not modify this comment unless you know what you're doing. tag:links-begin + +Documentation: https://certbot.eff.org/docs + +Software project: https://github.com/certbot/certbot + +Notes for developers: https://certbot.eff.org/docs/contributing.html + +Main Website: https://certbot.eff.org + +Let's Encrypt Website: https://letsencrypt.org + +Community: https://community.letsencrypt.org + +ACME spec: http://ietf-wg-acme.github.io/acme/ + +ACME working area in github: https://github.com/ietf-wg-acme/acme + +|build-status| |coverage| |docs| |container| + +.. |build-status| image:: https://travis-ci.com/certbot/certbot.svg?branch=master + :target: https://travis-ci.com/certbot/certbot + :alt: Travis CI status + +.. |coverage| image:: https://codecov.io/gh/certbot/certbot/branch/master/graph/badge.svg + :target: https://codecov.io/gh/certbot/certbot + :alt: Coverage status + +.. |docs| image:: https://readthedocs.org/projects/letsencrypt/badge/ + :target: https://readthedocs.org/projects/letsencrypt/ + :alt: Documentation status + +.. |container| image:: https://quay.io/repository/letsencrypt/letsencrypt/status + :target: https://quay.io/repository/letsencrypt/letsencrypt + :alt: Docker Repository on Quay.io + +.. Do not modify this comment unless you know what you're doing. tag:links-end + +System Requirements +=================== + +See https://certbot.eff.org/docs/install.html#system-requirements. + +.. Do not modify this comment unless you know what you're doing. tag:intro-end + +.. Do not modify this comment unless you know what you're doing. tag:features-begin + +Current Features +===================== + +* Supports multiple web servers: + + - apache/2.x + - nginx/0.8.48+ + - webroot (adds files to webroot directories in order to prove control of + domains and obtain certs) + - standalone (runs its own simple webserver to prove you control a domain) + - other server software via `third party plugins `_ + +* The private key is generated locally on your system. +* Can talk to the Let's Encrypt CA or optionally to other ACME + compliant services. +* Can get domain-validated (DV) certificates. +* Can revoke certificates. +* Adjustable RSA key bit-length (2048 (default), 4096, ...). +* Can optionally install a http -> https redirect, so your site effectively + runs https only (Apache only) +* Fully automated. +* Configuration changes are logged and can be reverted. +* Supports an interactive text UI, or can be driven entirely from the + command line. +* Free and Open Source Software, made with Python. + +.. Do not modify this comment unless you know what you're doing. tag:features-end + +For extensive documentation on using and contributing to Certbot, go to https://certbot.eff.org/docs. If you would like to contribute to the project or run the latest code from git, you should read our `developer guide `_. diff --git a/certbot/auth_handler.py b/certbot/auth_handler.py deleted file mode 100644 index 3dfaaf26f..000000000 --- a/certbot/auth_handler.py +++ /dev/null @@ -1,576 +0,0 @@ -"""ACME AuthHandler.""" -import collections -import logging -import time - -import six -import zope.component - -from acme import challenges -from acme import messages -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import DefaultDict, Dict, List, Set, Collection -# pylint: enable=unused-import, no-name-in-module -from certbot import achallenges -from certbot import errors -from certbot import error_handler -from certbot import interfaces - - -logger = logging.getLogger(__name__) - - -AnnotatedAuthzr = collections.namedtuple("AnnotatedAuthzr", ["authzr", "achalls"]) -"""Stores an authorization resource and its active annotated challenges.""" - - -class AuthHandler(object): - """ACME Authorization Handler for a client. - - :ivar auth: Authenticator capable of solving - :class:`~acme.challenges.Challenge` types - :type auth: :class:`certbot.interfaces.IAuthenticator` - - :ivar acme.client.BackwardsCompatibleClientV2 acme_client: ACME client API. - - :ivar account: Client's Account - :type account: :class:`certbot.account.Account` - - :ivar list pref_challs: sorted user specified preferred challenges - type strings with the most preferred challenge listed first - - """ - def __init__(self, auth, acme_client, account, pref_challs): - self.auth = auth - self.acme = acme_client - - self.account = account - self.pref_challs = pref_challs - - def handle_authorizations(self, orderr, best_effort=False): - """Retrieve all authorizations for challenges. - - :param acme.messages.OrderResource orderr: must have - authorizations filled in - :param bool best_effort: Whether or not all authorizations are - required (this is useful in renewal) - - :returns: List of authorization resources - :rtype: list - - :raises .AuthorizationError: If unable to retrieve all - authorizations - - """ - aauthzrs = [AnnotatedAuthzr(authzr, []) - for authzr in orderr.authorizations] - - self._choose_challenges(aauthzrs) - config = zope.component.getUtility(interfaces.IConfig) - notify = zope.component.getUtility(interfaces.IDisplay).notification - - # While there are still challenges remaining... - while self._has_challenges(aauthzrs): - with error_handler.ExitHandler(self._cleanup_challenges, aauthzrs): - resp = self._solve_challenges(aauthzrs) - logger.info("Waiting for verification...") - if config.debug_challenges: - notify('Challenges loaded. Press continue to submit to CA. ' - 'Pass "-v" for more info about challenges.', pause=True) - - # Send all Responses - this modifies achalls - self._respond(aauthzrs, resp, best_effort) - - # Just make sure all decisions are complete. - self.verify_authzr_complete(aauthzrs) - - # Only return valid authorizations - ret_val = [aauthzr.authzr for aauthzr in aauthzrs - if aauthzr.authzr.body.status == messages.STATUS_VALID] - - if not ret_val: - raise errors.AuthorizationError( - "Challenges failed for all domains") - - return ret_val - - def _choose_challenges(self, aauthzrs): - """ - Retrieve necessary and pending challenges to satisfy server. - NB: Necessary and already validated challenges are not retrieved, - as they can be reused for a certificate issuance. - """ - pending_authzrs = [aauthzr for aauthzr in aauthzrs - if aauthzr.authzr.body.status != messages.STATUS_VALID] - if pending_authzrs: - logger.info("Performing the following challenges:") - for aauthzr in pending_authzrs: - aauthzr_challenges = aauthzr.authzr.body.challenges - if self.acme.acme_version == 1: - combinations = aauthzr.authzr.body.combinations - else: - combinations = tuple((i,) for i in range(len(aauthzr_challenges))) - - path = gen_challenge_path( - aauthzr_challenges, - self._get_chall_pref(aauthzr.authzr.body.identifier.value), - combinations) - - aauthzr_achalls = self._challenge_factory( - aauthzr.authzr, path) - aauthzr.achalls.extend(aauthzr_achalls) - - for aauthzr in aauthzrs: - for achall in aauthzr.achalls: - if isinstance(achall.chall, challenges.TLSSNI01): - logger.warning("TLS-SNI-01 is deprecated, and will stop working soon.") - return - - def _has_challenges(self, aauthzrs): - """Do we have any challenges to perform?""" - return any(aauthzr.achalls for aauthzr in aauthzrs) - - def _solve_challenges(self, aauthzrs): - """Get Responses for challenges from authenticators.""" - resp = [] # type: Collection[challenges.ChallengeResponse] - all_achalls = self._get_all_achalls(aauthzrs) - try: - if all_achalls: - resp = self.auth.perform(all_achalls) - except errors.AuthorizationError: - logger.critical("Failure in setting up challenges.") - logger.info("Attempting to clean up outstanding challenges...") - raise - - assert len(resp) == len(all_achalls) - - return resp - - def _get_all_achalls(self, aauthzrs): - """Return all active challenges.""" - all_achalls = [] # type: Collection[challenges.ChallengeResponse] - for aauthzr in aauthzrs: - all_achalls.extend(aauthzr.achalls) - return all_achalls - - def _respond(self, aauthzrs, resp, best_effort): - """Send/Receive confirmation of all challenges. - - .. note:: This method also cleans up the auth_handler state. - - """ - # TODO: chall_update is a dirty hack to get around acme-spec #105 - chall_update = dict() \ - # type: Dict[int, List[achallenges.KeyAuthorizationAnnotatedChallenge]] - self._send_responses(aauthzrs, resp, chall_update) - - # Check for updated status... - self._poll_challenges(aauthzrs, chall_update, best_effort) - - def _send_responses(self, aauthzrs, resps, chall_update): - """Send responses and make sure errors are handled. - - :param aauthzrs: authorizations and the selected annotated challenges - to try and perform - :type aauthzrs: `list` of `AnnotatedAuthzr` - :param resps: challenge responses from the authenticator where - each response at index i corresponds to the annotated - challenge at index i in the list returned by - :func:`_get_all_achalls` - :type resps: `collections.abc.Iterable` of - :class:`~acme.challenges.ChallengeResponse` or `False` or - `None` - :param dict chall_update: parameter that is updated to hold - aauthzr index to list of outstanding solved annotated challenges - - """ - active_achalls = [] - resps_iter = iter(resps) - for i, aauthzr in enumerate(aauthzrs): - for achall in aauthzr.achalls: - # This line needs to be outside of the if block below to - # ensure failed challenges are cleaned up correctly - active_achalls.append(achall) - - resp = next(resps_iter) - # Don't send challenges for None and False authenticator responses - if resp: - self.acme.answer_challenge(achall.challb, resp) - # TODO: answer_challenge returns challr, with URI, - # that can be used in _find_updated_challr - # comparisons... - chall_update.setdefault(i, []).append(achall) - - return active_achalls - - def _poll_challenges(self, aauthzrs, chall_update, - best_effort, min_sleep=3, max_rounds=30): - """Wait for all challenge results to be determined.""" - indices_to_check = set(chall_update.keys()) - comp_indices = set() - rounds = 0 - - while indices_to_check and rounds < max_rounds: - # TODO: Use retry-after... - time.sleep(min_sleep) - all_failed_achalls = set() # type: Set[achallenges.KeyAuthorizationAnnotatedChallenge] - for index in indices_to_check: - comp_achalls, failed_achalls = self._handle_check( - aauthzrs, index, chall_update[index]) - - if len(comp_achalls) == len(chall_update[index]): - comp_indices.add(index) - elif not failed_achalls: - for achall, _ in comp_achalls: - chall_update[index].remove(achall) - # We failed some challenges... damage control - else: - if best_effort: - comp_indices.add(index) - logger.warning( - "Challenge failed for domain %s", - aauthzrs[index].authzr.body.identifier.value) - else: - all_failed_achalls.update( - updated for _, updated in failed_achalls) - - if all_failed_achalls: - _report_failed_challs(all_failed_achalls) - raise errors.FailedChallenges(all_failed_achalls) - - indices_to_check -= comp_indices - comp_indices.clear() - rounds += 1 - - def _handle_check(self, aauthzrs, index, achalls): - """Returns tuple of ('completed', 'failed').""" - completed = [] - failed = [] - - original_aauthzr = aauthzrs[index] - updated_authzr, _ = self.acme.poll(original_aauthzr.authzr) - aauthzrs[index] = AnnotatedAuthzr(updated_authzr, original_aauthzr.achalls) - if updated_authzr.body.status == messages.STATUS_VALID: - return achalls, [] - - # Note: if the whole authorization is invalid, the individual failed - # challenges will be determined here... - for achall in achalls: - updated_achall = achall.update(challb=self._find_updated_challb( - updated_authzr, achall)) - - # This does nothing for challenges that have yet to be decided yet. - if updated_achall.status == messages.STATUS_VALID: - completed.append((achall, updated_achall)) - elif updated_achall.status == messages.STATUS_INVALID: - failed.append((achall, updated_achall)) - - return completed, failed - - def _find_updated_challb(self, authzr, achall): # pylint: disable=no-self-use - """Find updated challenge body within Authorization Resource. - - .. warning:: This assumes only one instance of type of challenge in - each challenge resource. - - :param .AuthorizationResource authzr: Authorization Resource - :param .AnnotatedChallenge achall: Annotated challenge for which - to get status - - """ - for authzr_challb in authzr.body.challenges: - if type(authzr_challb.chall) is type(achall.challb.chall): # noqa - return authzr_challb - raise errors.AuthorizationError( - "Target challenge not found in authorization resource") - - def _get_chall_pref(self, domain): - """Return list of challenge preferences. - - :param str domain: domain for which you are requesting preferences - - """ - chall_prefs = [] - # Make sure to make a copy... - plugin_pref = self.auth.get_chall_pref(domain) - if self.pref_challs: - plugin_pref_types = set(chall.typ for chall in plugin_pref) - for typ in self.pref_challs: - if typ in plugin_pref_types: - chall_prefs.append(challenges.Challenge.TYPES[typ]) - if chall_prefs: - return chall_prefs - raise errors.AuthorizationError( - "None of the preferred challenges " - "are supported by the selected plugin") - chall_prefs.extend(plugin_pref) - return chall_prefs - - def _cleanup_challenges(self, aauthzrs, achalls=None): - """Cleanup challenges. - - :param aauthzrs: authorizations and their selected annotated - challenges - :type aauthzrs: `list` of `AnnotatedAuthzr` - :param achalls: annotated challenges to cleanup - :type achalls: `list` of :class:`certbot.achallenges.AnnotatedChallenge` - - """ - logger.info("Cleaning up challenges") - if achalls is None: - achalls = self._get_all_achalls(aauthzrs) - if achalls: - self.auth.cleanup(achalls) - for achall in achalls: - for aauthzr in aauthzrs: - if achall in aauthzr.achalls: - aauthzr.achalls.remove(achall) - break - - def verify_authzr_complete(self, aauthzrs): - """Verifies that all authorizations have been decided. - - :param aauthzrs: authorizations and their selected annotated - challenges - :type aauthzrs: `list` of `AnnotatedAuthzr` - - :returns: Whether all authzr are complete - :rtype: bool - - """ - for aauthzr in aauthzrs: - authzr = aauthzr.authzr - if (authzr.body.status != messages.STATUS_VALID and - authzr.body.status != messages.STATUS_INVALID): - raise errors.AuthorizationError("Incomplete authorizations") - - def _challenge_factory(self, authzr, path): - """Construct Namedtuple Challenges - - :param messages.AuthorizationResource authzr: authorization - - :param list path: List of indices from `challenges`. - - :returns: achalls, list of challenge type - :class:`certbot.achallenges.Indexed` - :rtype: list - - :raises .errors.Error: if challenge type is not recognized - - """ - achalls = [] - - for index in path: - challb = authzr.body.challenges[index] - achalls.append(challb_to_achall( - challb, self.account.key, authzr.body.identifier.value)) - - return achalls - - -def challb_to_achall(challb, account_key, domain): - """Converts a ChallengeBody object to an AnnotatedChallenge. - - :param .ChallengeBody challb: ChallengeBody - :param .JWK account_key: Authorized Account Key - :param str domain: Domain of the challb - - :returns: Appropriate AnnotatedChallenge - :rtype: :class:`certbot.achallenges.AnnotatedChallenge` - - """ - chall = challb.chall - logger.info("%s challenge for %s", chall.typ, domain) - - if isinstance(chall, challenges.KeyAuthorizationChallenge): - return achallenges.KeyAuthorizationAnnotatedChallenge( - challb=challb, domain=domain, account_key=account_key) - elif isinstance(chall, challenges.DNS): - return achallenges.DNS(challb=challb, domain=domain) - else: - raise errors.Error( - "Received unsupported challenge of type: %s", chall.typ) - - -def gen_challenge_path(challbs, preferences, combinations): - """Generate a plan to get authority over the identity. - - .. todo:: This can be possibly be rewritten to use resolved_combinations. - - :param tuple challbs: A tuple of challenges - (:class:`acme.messages.Challenge`) from - :class:`acme.messages.AuthorizationResource` to be - fulfilled by the client in order to prove possession of the - identifier. - - :param list preferences: List of challenge preferences for domain - (:class:`acme.challenges.Challenge` subclasses) - - :param tuple combinations: A collection of sets of challenges from - :class:`acme.messages.Challenge`, each of which would - be sufficient to prove possession of the identifier. - - :returns: tuple of indices from ``challenges``. - :rtype: tuple - - :raises certbot.errors.AuthorizationError: If a - path cannot be created that satisfies the CA given the preferences and - combinations. - - """ - if combinations: - return _find_smart_path(challbs, preferences, combinations) - else: - return _find_dumb_path(challbs, preferences) - - -def _find_smart_path(challbs, preferences, combinations): - """Find challenge path with server hints. - - Can be called if combinations is included. Function uses a simple - ranking system to choose the combo with the lowest cost. - - """ - chall_cost = {} - max_cost = 1 - for i, chall_cls in enumerate(preferences): - chall_cost[chall_cls] = i - max_cost += i - - # max_cost is now equal to sum(indices) + 1 - - best_combo = None - # Set above completing all of the available challenges - best_combo_cost = max_cost - - combo_total = 0 - for combo in combinations: - for challenge_index in combo: - combo_total += chall_cost.get(challbs[ - challenge_index].chall.__class__, max_cost) - - if combo_total < best_combo_cost: - best_combo = combo - best_combo_cost = combo_total - - combo_total = 0 - - if not best_combo: - _report_no_chall_path(challbs) - - return best_combo - - -def _find_dumb_path(challbs, preferences): - """Find challenge path without server hints. - - Should be called if the combinations hint is not included by the - server. This function either returns a path containing all - challenges provided by the CA or raises an exception. - - """ - path = [] - for i, challb in enumerate(challbs): - # supported is set to True if the challenge type is supported - supported = next((True for pref_c in preferences - if isinstance(challb.chall, pref_c)), False) - if supported: - path.append(i) - else: - _report_no_chall_path(challbs) - - return path - - -def _report_no_chall_path(challbs): - """Logs and raises an error that no satisfiable chall path exists. - - :param challbs: challenges from the authorization that can't be satisfied - - """ - msg = ("Client with the currently selected authenticator does not support " - "any combination of challenges that will satisfy the CA.") - if len(challbs) == 1 and isinstance(challbs[0].chall, challenges.DNS01): - msg += ( - " You may need to use an authenticator " - "plugin that can do challenges over DNS.") - logger.critical(msg) - raise errors.AuthorizationError(msg) - - -_ERROR_HELP_COMMON = ( - "To fix these errors, please make sure that your domain name was entered " - "correctly and the DNS A/AAAA record(s) for that domain contain(s) the " - "right IP address.") - - -_ERROR_HELP = { - "connection": - _ERROR_HELP_COMMON + " Additionally, please check that your computer " - "has a publicly routable IP address and that no firewalls are preventing " - "the server from communicating with the client. If you're using the " - "webroot plugin, you should also verify that you are serving files " - "from the webroot path you provided.", - "dnssec": - _ERROR_HELP_COMMON + " Additionally, if you have DNSSEC enabled for " - "your domain, please ensure that the signature is valid.", - "malformed": - "To fix these errors, please make sure that you did not provide any " - "invalid information to the client, and try running Certbot " - "again.", - "serverInternal": - "Unfortunately, an error on the ACME server prevented you from completing " - "authorization. Please try again later.", - "tls": - _ERROR_HELP_COMMON + " Additionally, please check that you have an " - "up-to-date TLS configuration that allows the server to communicate " - "with the Certbot client.", - "unauthorized": _ERROR_HELP_COMMON, - "unknownHost": _ERROR_HELP_COMMON, -} - - -def _report_failed_challs(failed_achalls): - """Notifies the user about failed challenges. - - :param set failed_achalls: A set of failed - :class:`certbot.achallenges.AnnotatedChallenge`. - - """ - problems = collections.defaultdict(list)\ - # type: DefaultDict[str, List[achallenges.KeyAuthorizationAnnotatedChallenge]] - for achall in failed_achalls: - if achall.error: - problems[achall.error.typ].append(achall) - reporter = zope.component.getUtility(interfaces.IReporter) - for achalls in six.itervalues(problems): - reporter.add_message( - _generate_failed_chall_msg(achalls), reporter.MEDIUM_PRIORITY) - - -def _generate_failed_chall_msg(failed_achalls): - """Creates a user friendly error message about failed challenges. - - :param list failed_achalls: A list of failed - :class:`certbot.achallenges.AnnotatedChallenge` with the same error - type. - - :returns: A formatted error message for the client. - :rtype: str - - """ - error = failed_achalls[0].error - typ = error.typ - if messages.is_acme_error(error): - typ = error.code - msg = ["The following errors were reported by the server:"] - - for achall in failed_achalls: - msg.append("\n\nDomain: %s\nType: %s\nDetail: %s" % ( - achall.domain, typ, achall.error.detail)) - - if typ in _ERROR_HELP: - msg.append("\n\n") - msg.append(_ERROR_HELP[typ]) - - return "".join(msg) diff --git a/certbot/__init__.py b/certbot/certbot/__init__.py similarity index 76% rename from certbot/__init__.py rename to certbot/certbot/__init__.py index bf68034c8..84ade6b08 100644 --- a/certbot/__init__.py +++ b/certbot/certbot/__init__.py @@ -1,4 +1,4 @@ """Certbot client.""" # version number like 1.2.3a0, must have at least 2 parts, like 1.2 -__version__ = '0.31.0.dev0' +__version__ = '1.3.0.dev0' diff --git a/certbot/certbot/_internal/__init__.py b/certbot/certbot/_internal/__init__.py new file mode 100644 index 000000000..45ec6ac9c --- /dev/null +++ b/certbot/certbot/_internal/__init__.py @@ -0,0 +1,6 @@ +""" +Modules internal to Certbot. + +This package contains modules that are not considered part of Certbot's public +API. They may be changed without updating Certbot's major version. +""" diff --git a/certbot/account.py b/certbot/certbot/_internal/account.py similarity index 91% rename from certbot/account.py rename to certbot/certbot/_internal/account.py index 0c653f6dd..61f63bda6 100644 --- a/certbot/account.py +++ b/certbot/certbot/_internal/account.py @@ -3,7 +3,6 @@ import datetime import functools import hashlib import logging -import os import shutil import socket @@ -16,18 +15,16 @@ import zope.component from acme import fields as acme_fields from acme import messages - -from certbot import compat -from certbot import constants from certbot import errors from certbot import interfaces from certbot import util - +from certbot._internal import constants +from certbot.compat import os logger = logging.getLogger(__name__) -class Account(object): # pylint: disable=too-few-public-methods +class Account(object): """ACME protocol registration. :ivar .RegistrationResource regr: Registration Resource @@ -59,11 +56,18 @@ class Account(object): # pylint: disable=too-few-public-methods tz=pytz.UTC).replace(microsecond=0), creation_host=socket.getfqdn()) if meta is None else meta - self.id = hashlib.md5( - self.key.key.public_key().public_bytes( - encoding=serialization.Encoding.PEM, - format=serialization.PublicFormat.SubjectPublicKeyInfo) - ).hexdigest() + # try MD5, else use MD5 in non-security mode (e.g. for FIPS systems / RHEL) + try: + hasher = hashlib.md5() + except ValueError: + hasher = hashlib.new('md5', usedforsecurity=False) # type: ignore + + hasher.update(self.key.key.public_key().public_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PublicFormat.SubjectPublicKeyInfo) + ) + + self.id = hasher.hexdigest() # Implementation note: Email? Multiple accounts can have the # same email address. Registration URI? Assigned by the # server, not guaranteed to be stable over time, nor @@ -110,8 +114,7 @@ class AccountMemoryStorage(interfaces.AccountStorage): def find_all(self): return list(six.itervalues(self.accounts)) - def save(self, account, acme): - # pylint: disable=unused-argument + def save(self, account, client): if account.id in self.accounts: logger.debug("Overwriting account: %s", account.id) self.accounts[account.id] = account @@ -141,8 +144,7 @@ class AccountFileStorage(interfaces.AccountStorage): """ def __init__(self, config): self.config = config - util.make_or_verify_dir(config.accounts_dir, 0o700, compat.os_geteuid(), - self.config.strict_permissions) + util.make_or_verify_dir(config.accounts_dir, 0o700, self.config.strict_permissions) def _account_dir_path(self, account_id): return self._account_dir_path_for_server_path(account_id, self.config.server_path) @@ -221,9 +223,8 @@ class AccountFileStorage(interfaces.AccountStorage): else: self._symlink_to_accounts_dir(prev_server_path, server_path) return prev_loaded_account - else: - raise errors.AccountNotFound( - "Account at %s does not exist" % account_dir_path) + raise errors.AccountNotFound( + "Account at %s does not exist" % account_dir_path) try: with open(self._regr_path(account_dir_path)) as regr_file: @@ -235,18 +236,13 @@ class AccountFileStorage(interfaces.AccountStorage): except IOError as error: raise errors.AccountStorageError(error) - acc = Account(regr, key, meta) - if acc.id != account_id: - raise errors.AccountStorageError( - "Account ids mismatch (expected: {0}, found: {1}".format( - account_id, acc.id)) - return acc + return Account(regr, key, meta) def load(self, account_id): return self._load_for_server_path(account_id, self.config.server_path) - def save(self, account, acme): - self._save(account, acme, regr_only=False) + def save(self, account, client): + self._save(account, client, regr_only=False) def save_regr(self, account, acme): """Save the registration resource. @@ -324,8 +320,7 @@ class AccountFileStorage(interfaces.AccountStorage): def _save(self, account, acme, regr_only): account_dir_path = self._account_dir_path(account.id) - util.make_or_verify_dir(account_dir_path, 0o700, compat.os_geteuid(), - self.config.strict_permissions) + util.make_or_verify_dir(account_dir_path, 0o700, self.config.strict_permissions) try: with open(self._regr_path(account_dir_path), "w") as regr_file: regr = account.regr diff --git a/certbot/certbot/_internal/auth_handler.py b/certbot/certbot/_internal/auth_handler.py new file mode 100644 index 000000000..e4ad91247 --- /dev/null +++ b/certbot/certbot/_internal/auth_handler.py @@ -0,0 +1,469 @@ +"""ACME AuthHandler.""" +import datetime +import logging +import time + +import zope.component + +from acme import challenges +from acme import errors as acme_errors +from acme import messages +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Tuple # pylint: disable=unused-import, no-name-in-module +from certbot import achallenges +from certbot import errors +from certbot import interfaces +from certbot._internal import error_handler + +logger = logging.getLogger(__name__) + + +class AuthHandler(object): + """ACME Authorization Handler for a client. + + :ivar auth: Authenticator capable of solving + :class:`~acme.challenges.Challenge` types + :type auth: :class:`certbot.interfaces.IAuthenticator` + + :ivar acme.client.BackwardsCompatibleClientV2 acme_client: ACME client API. + + :ivar account: Client's Account + :type account: :class:`certbot._internal.account.Account` + + :ivar list pref_challs: sorted user specified preferred challenges + type strings with the most preferred challenge listed first + + """ + def __init__(self, auth, acme_client, account, pref_challs): + self.auth = auth + self.acme = acme_client + + self.account = account + self.pref_challs = pref_challs + + def handle_authorizations(self, orderr, best_effort=False, max_retries=30): + """ + Retrieve all authorizations, perform all challenges required to validate + these authorizations, then poll and wait for the authorization to be checked. + :param acme.messages.OrderResource orderr: must have authorizations filled in + :param bool best_effort: if True, not all authorizations need to be validated (eg. renew) + :param int max_retries: maximum number of retries to poll authorizations + :returns: list of all validated authorizations + :rtype: List + + :raises .AuthorizationError: If unable to retrieve all authorizations + """ + authzrs = orderr.authorizations[:] + if not authzrs: + raise errors.AuthorizationError('No authorization to handle.') + + # Retrieve challenges that need to be performed to validate authorizations. + achalls = self._choose_challenges(authzrs) + if not achalls: + return authzrs + + # Starting now, challenges will be cleaned at the end no matter what. + with error_handler.ExitHandler(self._cleanup_challenges, achalls): + # To begin, let's ask the authenticator plugin to perform all challenges. + try: + resps = self.auth.perform(achalls) + + # If debug is on, wait for user input before starting the verification process. + logger.info('Waiting for verification...') + config = zope.component.getUtility(interfaces.IConfig) + if config.debug_challenges: + notify = zope.component.getUtility(interfaces.IDisplay).notification + notify('Challenges loaded. Press continue to submit to CA. ' + 'Pass "-v" for more info about challenges.', pause=True) + except errors.AuthorizationError as error: + logger.critical('Failure in setting up challenges.') + logger.info('Attempting to clean up outstanding challenges...') + raise error + # All challenges should have been processed by the authenticator. + assert len(resps) == len(achalls), 'Some challenges have not been performed.' + + # Inform the ACME CA server that challenges are available for validation. + for achall, resp in zip(achalls, resps): + self.acme.answer_challenge(achall.challb, resp) + + # Wait for authorizations to be checked. + self._poll_authorizations(authzrs, max_retries, best_effort) + + # Keep validated authorizations only. If there is none, no certificate can be issued. + authzrs_validated = [authzr for authzr in authzrs + if authzr.body.status == messages.STATUS_VALID] + if not authzrs_validated: + raise errors.AuthorizationError('All challenges have failed.') + + return authzrs_validated + + def deactivate_valid_authorizations(self, orderr): + # type: (messages.OrderResource) -> Tuple[List, List] + """ + Deactivate all `valid` authorizations in the order, so that they cannot be re-used + in subsequent orders. + :param messages.OrderResource orderr: must have authorizations filled in + :returns: tuple of list of successfully deactivated authorizations, and + list of unsuccessfully deactivated authorizations. + :rtype: tuple + """ + to_deactivate = [authzr for authzr in orderr.authorizations + if authzr.body.status == messages.STATUS_VALID] + deactivated = [] + failed = [] + + for authzr in to_deactivate: + try: + authzr = self.acme.deactivate_authorization(authzr) + deactivated.append(authzr) + except acme_errors.Error as e: + failed.append(authzr) + logger.debug('Failed to deactivate authorization %s: %s', authzr.uri, e) + + return (deactivated, failed) + + def _poll_authorizations(self, authzrs, max_retries, best_effort): + """ + Poll the ACME CA server, to wait for confirmation that authorizations have their challenges + all verified. The poll may occur several times, until all authorizations are checked + (valid or invalid), or after a maximum of retries. + """ + authzrs_to_check = {index: (authzr, None) + for index, authzr in enumerate(authzrs)} + authzrs_failed_to_report = [] + # Give an initial second to the ACME CA server to check the authorizations + sleep_seconds = 1 + for _ in range(max_retries): + # Wait for appropriate time (from Retry-After, initial wait, or no wait) + if sleep_seconds > 0: + time.sleep(sleep_seconds) + # Poll all updated authorizations. + authzrs_to_check = {index: self.acme.poll(authzr) for index, (authzr, _) + in authzrs_to_check.items()} + # Update the original list of authzr with the updated authzrs from server. + for index, (authzr, _) in authzrs_to_check.items(): + authzrs[index] = authzr + + # Gather failed authorizations + authzrs_failed = [authzr for authzr, _ in authzrs_to_check.values() + if authzr.body.status == messages.STATUS_INVALID] + for authzr_failed in authzrs_failed: + logger.warning('Challenge failed for domain %s', + authzr_failed.body.identifier.value) + # Accumulating all failed authzrs to build a consolidated report + # on them at the end of the polling. + authzrs_failed_to_report.extend(authzrs_failed) + + # Extract out the authorization already checked for next poll iteration. + # Poll may stop here because there is no pending authorizations anymore. + authzrs_to_check = {index: (authzr, resp) for index, (authzr, resp) + in authzrs_to_check.items() + if authzr.body.status == messages.STATUS_PENDING} + if not authzrs_to_check: + # Polling process is finished, we can leave the loop + break + + # Be merciful with the ACME server CA, check the Retry-After header returned, + # and wait this time before polling again in next loop iteration. + # From all the pending authorizations, we take the greatest Retry-After value + # to avoid polling an authorization before its relevant Retry-After value. + retry_after = max(self.acme.retry_after(resp, 3) + for _, resp in authzrs_to_check.values()) + sleep_seconds = (retry_after - datetime.datetime.now()).total_seconds() + + # In case of failed authzrs, create a report to the user. + if authzrs_failed_to_report: + _report_failed_authzrs(authzrs_failed_to_report, self.account.key) + if not best_effort: + # Without best effort, having failed authzrs is critical and fail the process. + raise errors.AuthorizationError('Some challenges have failed.') + + if authzrs_to_check: + # Here authzrs_to_check is still not empty, meaning we exceeded the max polling attempt. + raise errors.AuthorizationError('All authorizations were not finalized by the CA.') + + def _choose_challenges(self, authzrs): + """ + Retrieve necessary and pending challenges to satisfy server. + NB: Necessary and already validated challenges are not retrieved, + as they can be reused for a certificate issuance. + """ + pending_authzrs = [authzr for authzr in authzrs + if authzr.body.status != messages.STATUS_VALID] + achalls = [] # type: List[achallenges.AnnotatedChallenge] + if pending_authzrs: + logger.info("Performing the following challenges:") + for authzr in pending_authzrs: + authzr_challenges = authzr.body.challenges + if self.acme.acme_version == 1: + combinations = authzr.body.combinations + else: + combinations = tuple((i,) for i in range(len(authzr_challenges))) + + path = gen_challenge_path( + authzr_challenges, + self._get_chall_pref(authzr.body.identifier.value), + combinations) + + achalls.extend(self._challenge_factory(authzr, path)) + + return achalls + + def _get_chall_pref(self, domain): + """Return list of challenge preferences. + + :param str domain: domain for which you are requesting preferences + + """ + chall_prefs = [] + # Make sure to make a copy... + plugin_pref = self.auth.get_chall_pref(domain) + if self.pref_challs: + plugin_pref_types = set(chall.typ for chall in plugin_pref) + for typ in self.pref_challs: + if typ in plugin_pref_types: + chall_prefs.append(challenges.Challenge.TYPES[typ]) + if chall_prefs: + return chall_prefs + raise errors.AuthorizationError( + "None of the preferred challenges " + "are supported by the selected plugin") + chall_prefs.extend(plugin_pref) + return chall_prefs + + def _cleanup_challenges(self, achalls): + """Cleanup challenges. + + :param achalls: annotated challenges to cleanup + :type achalls: `list` of :class:`certbot.achallenges.AnnotatedChallenge` + + """ + logger.info("Cleaning up challenges") + self.auth.cleanup(achalls) + + def _challenge_factory(self, authzr, path): + """Construct Namedtuple Challenges + + :param messages.AuthorizationResource authzr: authorization + + :param list path: List of indices from `challenges`. + + :returns: achalls, list of challenge type + :class:`certbot.achallenges.Indexed` + :rtype: list + + :raises .errors.Error: if challenge type is not recognized + + """ + achalls = [] + + for index in path: + challb = authzr.body.challenges[index] + achalls.append(challb_to_achall( + challb, self.account.key, authzr.body.identifier.value)) + + return achalls + + +def challb_to_achall(challb, account_key, domain): + """Converts a ChallengeBody object to an AnnotatedChallenge. + + :param .ChallengeBody challb: ChallengeBody + :param .JWK account_key: Authorized Account Key + :param str domain: Domain of the challb + + :returns: Appropriate AnnotatedChallenge + :rtype: :class:`certbot.achallenges.AnnotatedChallenge` + + """ + chall = challb.chall + logger.info("%s challenge for %s", chall.typ, domain) + + if isinstance(chall, challenges.KeyAuthorizationChallenge): + return achallenges.KeyAuthorizationAnnotatedChallenge( + challb=challb, domain=domain, account_key=account_key) + elif isinstance(chall, challenges.DNS): + return achallenges.DNS(challb=challb, domain=domain) + raise errors.Error( + "Received unsupported challenge of type: {0}".format(chall.typ)) + + +def gen_challenge_path(challbs, preferences, combinations): + """Generate a plan to get authority over the identity. + + .. todo:: This can be possibly be rewritten to use resolved_combinations. + + :param tuple challbs: A tuple of challenges + (:class:`acme.messages.Challenge`) from + :class:`acme.messages.AuthorizationResource` to be + fulfilled by the client in order to prove possession of the + identifier. + + :param list preferences: List of challenge preferences for domain + (:class:`acme.challenges.Challenge` subclasses) + + :param tuple combinations: A collection of sets of challenges from + :class:`acme.messages.Challenge`, each of which would + be sufficient to prove possession of the identifier. + + :returns: tuple of indices from ``challenges``. + :rtype: tuple + + :raises certbot.errors.AuthorizationError: If a + path cannot be created that satisfies the CA given the preferences and + combinations. + + """ + if combinations: + return _find_smart_path(challbs, preferences, combinations) + return _find_dumb_path(challbs, preferences) + + +def _find_smart_path(challbs, preferences, combinations): + """Find challenge path with server hints. + + Can be called if combinations is included. Function uses a simple + ranking system to choose the combo with the lowest cost. + + """ + chall_cost = {} + max_cost = 1 + for i, chall_cls in enumerate(preferences): + chall_cost[chall_cls] = i + max_cost += i + + # max_cost is now equal to sum(indices) + 1 + + best_combo = None + # Set above completing all of the available challenges + best_combo_cost = max_cost + + combo_total = 0 + for combo in combinations: + for challenge_index in combo: + combo_total += chall_cost.get(challbs[ + challenge_index].chall.__class__, max_cost) + + if combo_total < best_combo_cost: + best_combo = combo + best_combo_cost = combo_total + + combo_total = 0 + + if not best_combo: + _report_no_chall_path(challbs) + + return best_combo + + +def _find_dumb_path(challbs, preferences): + """Find challenge path without server hints. + + Should be called if the combinations hint is not included by the + server. This function either returns a path containing all + challenges provided by the CA or raises an exception. + + """ + path = [] + for i, challb in enumerate(challbs): + # supported is set to True if the challenge type is supported + supported = next((True for pref_c in preferences + if isinstance(challb.chall, pref_c)), False) + if supported: + path.append(i) + else: + _report_no_chall_path(challbs) + + return path + + +def _report_no_chall_path(challbs): + """Logs and raises an error that no satisfiable chall path exists. + + :param challbs: challenges from the authorization that can't be satisfied + + """ + msg = ("Client with the currently selected authenticator does not support " + "any combination of challenges that will satisfy the CA.") + if len(challbs) == 1 and isinstance(challbs[0].chall, challenges.DNS01): + msg += ( + " You may need to use an authenticator " + "plugin that can do challenges over DNS.") + logger.critical(msg) + raise errors.AuthorizationError(msg) + + +_ERROR_HELP_COMMON = ( + "To fix these errors, please make sure that your domain name was entered " + "correctly and the DNS A/AAAA record(s) for that domain contain(s) the " + "right IP address.") + + +_ERROR_HELP = { + "connection": + _ERROR_HELP_COMMON + " Additionally, please check that your computer " + "has a publicly routable IP address and that no firewalls are preventing " + "the server from communicating with the client. If you're using the " + "webroot plugin, you should also verify that you are serving files " + "from the webroot path you provided.", + "dnssec": + _ERROR_HELP_COMMON + " Additionally, if you have DNSSEC enabled for " + "your domain, please ensure that the signature is valid.", + "malformed": + "To fix these errors, please make sure that you did not provide any " + "invalid information to the client, and try running Certbot " + "again.", + "serverInternal": + "Unfortunately, an error on the ACME server prevented you from completing " + "authorization. Please try again later.", + "tls": + _ERROR_HELP_COMMON + " Additionally, please check that you have an " + "up-to-date TLS configuration that allows the server to communicate " + "with the Certbot client.", + "unauthorized": _ERROR_HELP_COMMON, + "unknownHost": _ERROR_HELP_COMMON, +} + + +def _report_failed_authzrs(failed_authzrs, account_key): + """Notifies the user about failed authorizations.""" + problems = {} # type: Dict[str, List[achallenges.AnnotatedChallenge]] + failed_achalls = [challb_to_achall(challb, account_key, authzr.body.identifier.value) + for authzr in failed_authzrs for challb in authzr.body.challenges + if challb.error] + + for achall in failed_achalls: + problems.setdefault(achall.error.typ, []).append(achall) + + reporter = zope.component.getUtility(interfaces.IReporter) + for achalls in problems.values(): + reporter.add_message(_generate_failed_chall_msg(achalls), reporter.MEDIUM_PRIORITY) + + +def _generate_failed_chall_msg(failed_achalls): + """Creates a user friendly error message about failed challenges. + + :param list failed_achalls: A list of failed + :class:`certbot.achallenges.AnnotatedChallenge` with the same error + type. + + :returns: A formatted error message for the client. + :rtype: str + + """ + error = failed_achalls[0].error + typ = error.typ + if messages.is_acme_error(error): + typ = error.code + msg = ["The following errors were reported by the server:"] + + for achall in failed_achalls: + msg.append("\n\nDomain: %s\nType: %s\nDetail: %s" % ( + achall.domain, typ, achall.error.detail)) + + if typ in _ERROR_HELP: + msg.append("\n\n") + msg.append(_ERROR_HELP[typ]) + + return "".join(msg) diff --git a/certbot/cert_manager.py b/certbot/certbot/_internal/cert_manager.py similarity index 95% rename from certbot/cert_manager.py rename to certbot/certbot/_internal/cert_manager.py index 2a67f8765..1def76a3d 100644 --- a/certbot/cert_manager.py +++ b/certbot/certbot/_internal/cert_manager.py @@ -1,21 +1,20 @@ """Tools for managing certificates.""" import datetime import logging -import os -import pytz import re import traceback + +import pytz import zope.component from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module -from certbot import compat from certbot import crypto_util from certbot import errors from certbot import interfaces -from certbot import ocsp -from certbot import storage from certbot import util - +from certbot._internal import ocsp +from certbot._internal import storage +from certbot.compat import os from certbot.display import util as display_util logger = logging.getLogger(__name__) @@ -33,7 +32,7 @@ def update_live_symlinks(config): .. note:: This assumes that the installation is using a Reverter object. :param config: Configuration. - :type config: :class:`certbot.configuration.NamespaceConfig` + :type config: :class:`certbot._internal.configuration.NamespaceConfig` """ for renewal_file in storage.renewal_conf_files(config): @@ -43,7 +42,7 @@ def rename_lineage(config): """Rename the specified lineage to the new name. :param config: Configuration. - :type config: :class:`certbot.configuration.NamespaceConfig` + :type config: :class:`certbot._internal.configuration.NamespaceConfig` """ disp = zope.component.getUtility(interfaces.IDisplay) @@ -70,7 +69,7 @@ def certificates(config): """Display information about certs configured with Certbot :param config: Configuration. - :type config: :class:`certbot.configuration.NamespaceConfig` + :type config: :class:`certbot._internal.configuration.NamespaceConfig` """ parsed_certs = [] parse_failures = [] @@ -105,7 +104,7 @@ def lineage_for_certname(cli_config, certname): """Find a lineage object with name certname.""" configs_dir = cli_config.renewal_configs_dir # Verify the directory is there - util.make_or_verify_dir(configs_dir, mode=0o755, uid=compat.os_geteuid()) + util.make_or_verify_dir(configs_dir, mode=0o755) try: renewal_file = storage.renewal_file_for_certname(cli_config, certname) except errors.CertStorageError: @@ -136,7 +135,7 @@ def find_duplicative_certs(config, domains): undefined. :param config: Configuration. - :type config: :class:`certbot.configuration.NamespaceConfig` + :type config: :class:`certbot._internal.configuration.NamespaceConfig` :param domains: List of domain names :type domains: `list` of `str` @@ -183,10 +182,9 @@ def _archive_files(candidate_lineage, filetype): archive_dir = candidate_lineage.archive_dir pattern = [os.path.join(archive_dir, f) for f in os.listdir(archive_dir) if re.match("{0}[0-9]*.pem".format(filetype), f)] - if len(pattern) > 0: + if pattern: return pattern - else: - return None + return None def _acceptable_matches(): """ Generates the list that's passed to match_and_check_overlaps. Is its own function to @@ -244,8 +242,8 @@ def match_and_check_overlaps(cli_config, acceptable_matches, match_func, rv_func raise errors.Error("No match found for cert-path {0}!".format(cli_config.cert_path[0])) elif len(matched) > 1: raise errors.OverlappingMatchFound() - else: - return matched + return matched + def human_readable_cert_info(config, cert, skip_filter_checks=False): """ Returns a human readable description of info about a RenewableCert object""" @@ -263,7 +261,7 @@ def human_readable_cert_info(config, cert, skip_filter_checks=False): reasons.append('TEST_CERT') if cert.target_expiry <= now: reasons.append('EXPIRED') - if checker.ocsp_revoked(cert.cert, cert.chain): + elif checker.ocsp_revoked(cert): reasons.append('REVOKED') if reasons: @@ -375,7 +373,7 @@ def _search_lineages(cli_config, func, initial_rv, *args): """ configs_dir = cli_config.renewal_configs_dir # Verify the directory is there - util.make_or_verify_dir(configs_dir, mode=0o755, uid=compat.os_geteuid()) + util.make_or_verify_dir(configs_dir, mode=0o755) rv = initial_rv for renewal_file in storage.renewal_conf_files(cli_config): diff --git a/certbot/cli.py b/certbot/certbot/_internal/cli.py similarity index 93% rename from certbot/cli.py rename to certbot/certbot/_internal/cli.py index 31f55711f..d5d498b4d 100644 --- a/certbot/cli.py +++ b/certbot/certbot/_internal/cli.py @@ -1,39 +1,35 @@ """Certbot command line argument & config processing.""" # pylint: disable=too-many-lines from __future__ import print_function + import argparse import copy import glob -import logging import logging.handlers -import os import sys import configargparse import six import zope.component import zope.interface - from zope.interface import interfaces as zope_interfaces from acme import challenges -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import Any, Dict, Optional -# pylint: enable=unused-import, no-name-in-module - +from acme.magic_typing import Any # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Optional # pylint: disable=unused-import, no-name-in-module import certbot - -from certbot import constants from certbot import crypto_util from certbot import errors -from certbot import hooks from certbot import interfaces from certbot import util - +from certbot._internal import constants +from certbot._internal import hooks +from certbot._internal.plugins import disco as plugins_disco +import certbot._internal.plugins.selection as plugin_selection +from certbot.compat import os from certbot.display import util as display_util -from certbot.plugins import disco as plugins_disco import certbot.plugins.enhancements as enhancements -import certbot.plugins.selection as plugin_selection logger = logging.getLogger(__name__) @@ -96,19 +92,20 @@ obtain, install, and renew certificates: manage certificates: certificates Display information about certificates you have from Certbot - revoke Revoke a certificate (supply --cert-path or --cert-name) - delete Delete a certificate + revoke Revoke a certificate (supply --cert-name or --cert-path) + delete Delete a certificate (supply --cert-name) -manage your account with Let's Encrypt: - register Create a Let's Encrypt ACME account - update_account Update a Let's Encrypt ACME account +manage your account: + register Create an ACME account + unregister Deactivate an ACME account + update_account Update an ACME account --agree-tos Agree to the ACME server's Subscriber Agreement -m EMAIL Email address for important account notifications """ # This is the short help for certbot --help, where we disable argparse # altogether -HELP_USAGE = """ +HELP_AND_VERSION_USAGE = """ More detailed help: -h, --help [TOPIC] print this message, or detailed help on a topic; @@ -117,6 +114,8 @@ More detailed help: all, automation, commands, paths, security, testing, or any of the subcommands or plugins (certonly, renew, install, register, nginx, apache, standalone, webroot, etc.) + -h all print a detailed help page including all topics + --version print the version number """ @@ -163,24 +162,6 @@ def report_config_interaction(modified, modifiers): VAR_MODIFIERS.setdefault(var, set()).update(modifiers) -def possible_deprecation_warning(config): - "A deprecation warning for users with the old, not-self-upgrading letsencrypt-auto." - if cli_command != LEAUTO: - return - if config.no_self_upgrade: - # users setting --no-self-upgrade might be hanging on a client version like 0.3.0 - # or 0.5.0 which is the new script, but doesn't set CERTBOT_AUTO; they don't - # need warnings - return - if "CERTBOT_AUTO" not in os.environ: - logger.warning("You are running with an old copy of letsencrypt-auto" - " that does not receive updates, and is less reliable than more" - " recent versions. The letsencrypt client has also been renamed" - " to Certbot. We recommend upgrading to the latest certbot-auto" - " script, or using native OS packages.") - logger.debug("Deprecation warning circumstances: %s / %s", sys.argv[0], os.environ) - - class _Default(object): """A class to use as a default to detect if a value is set by a user""" @@ -306,12 +287,10 @@ def flag_default(name): def config_help(name, hidden=False): """Extract the help message for an `.IConfig` attribute.""" - # pylint: disable=no-member if hidden: return argparse.SUPPRESS - else: - field = interfaces.IConfig.__getitem__(name) # type: zope.interface.interface.Attribute - return field.__doc__ + field = interfaces.IConfig.__getitem__(name) # type: zope.interface.interface.Attribute + return field.__doc__ class HelpfulArgumentGroup(object): @@ -417,11 +396,6 @@ VERB_HELP = [ "usage": "\n\n certbot install --cert-path /path/to/fullchain.pem " " --key-path /path/to/private-key [options]\n\n" }), - ("config_changes", { - "short": "Show changes that Certbot has made to server configurations", - "opts": "Options for controlling which changes are displayed", - "usage": "\n\n certbot config_changes --num NUM [options]\n\n" - }), ("rollback", { "short": "Roll back server conf changes made during certificate installation", "opts": "Options for rolling back server configuration changes", @@ -429,7 +403,7 @@ VERB_HELP = [ }), ("plugins", { "short": "List plugins that are installed and available on your system", - "opts": 'Options for for the "plugins" subcommand', + "opts": 'Options for the "plugins" subcommand', "usage": "\n\n certbot plugins [options]\n\n" }), ("update_symlinks", { @@ -462,11 +436,10 @@ class HelpfulArgumentParser(object): def __init__(self, args, plugins, detect_defaults=False): - from certbot import main + from certbot._internal import main self.VERBS = { "auth": main.certonly, "certonly": main.certonly, - "config_changes": main.config_changes, "run": main.run, "install": main.install, "plugins": main.plugins_cmd, @@ -537,7 +510,7 @@ class HelpfulArgumentParser(object): # Help that are synonyms for --help subcommands COMMANDS_TOPICS = ["command", "commands", "subcommand", "subcommands", "verbs"] def _list_subcommands(self): - longest = max(len(v) for v in VERB_HELP_MAP.keys()) + longest = max(len(v) for v in VERB_HELP_MAP) text = "The full list of available SUBCOMMANDS is:\n\n" for verb, props in sorted(VERB_HELP): @@ -565,8 +538,8 @@ class HelpfulArgumentParser(object): apache_doc = "(the certbot apache plugin is not installed)" usage = SHORT_USAGE - if help_arg == True: - self.notify(usage + COMMAND_OVERVIEW % (apache_doc, nginx_doc) + HELP_USAGE) + if help_arg is True: + self.notify(usage + COMMAND_OVERVIEW % (apache_doc, nginx_doc) + HELP_AND_VERSION_USAGE) sys.exit(0) elif help_arg in self.COMMANDS_TOPICS: self.notify(usage + self._list_subcommands()) @@ -643,20 +616,25 @@ class HelpfulArgumentParser(object): raise errors.Error( "Parameters --hsts and --auto-hsts cannot be used simultaneously.") - possible_deprecation_warning(parsed_args) - return parsed_args def set_test_server(self, parsed_args): """We have --staging/--dry-run; perform sanity check and set config.server""" - if parsed_args.server not in (flag_default("server"), constants.STAGING_URI): - conflicts = ["--staging"] if parsed_args.staging else [] - conflicts += ["--dry-run"] if parsed_args.dry_run else [] - raise errors.Error("--server value conflicts with {0}".format( - " and ".join(conflicts))) + # Flag combinations should produce these results: + # | --staging | --dry-run | + # ------------------------------------------------------------ + # | --server acme-v02 | Use staging | Use staging | + # | --server acme-staging-v02 | Use staging | Use staging | + # | --server | Conflict error | Use | - parsed_args.server = constants.STAGING_URI + default_servers = (flag_default("server"), constants.STAGING_URI) + + if parsed_args.staging and parsed_args.server not in default_servers: + raise errors.Error("--server value conflicts with --staging") + + if parsed_args.server in default_servers: + parsed_args.server = constants.STAGING_URI if parsed_args.dry_run: if self.verb not in ["certonly", "renew"]: @@ -695,7 +673,7 @@ class HelpfulArgumentParser(object): parsed_args.actual_csr = (csr, typ) - csr_domains = set([d.lower() for d in domains]) + csr_domains = {d.lower() for d in domains} config_domains = set(parsed_args.domains) if csr_domains != config_domains: raise errors.ConfigurationError( @@ -752,9 +730,10 @@ class HelpfulArgumentParser(object): """Add a new command line argument. :param topics: str or [str] help topic(s) this should be listed under, - or None for "always documented". The first entry - determines where the flag lives in the "--help all" - output (None -> "optional arguments"). + or None for options that don't fit under a specific + topic which will only be shown in "--help all" output. + The first entry determines where the flag lives in the + "--help all" output (None -> "optional arguments"). :param list *args: the names of this argument flag :param dict **kwargs: various argparse settings for this argument @@ -867,12 +846,11 @@ class HelpfulArgumentParser(object): chosen_topic = "run" if chosen_topic == "all": # Addition of condition closes #6209 (removal of duplicate route53 option). - return dict([(t, True) if t != 'certbot-route53:auth' else (t, False) - for t in self.help_topics]) + return {t: t != 'certbot-route53:auth' for t in self.help_topics} elif not chosen_topic: - return dict([(t, False) for t in self.help_topics]) - else: - return dict([(t, t == chosen_topic) for t in self.help_topics]) + return {t: False for t in self.help_topics} + return {t: t == chosen_topic for t in self.help_topics} + def _add_all_groups(helpful): helpful.add_group("automation", description="Flags for automating execution & other tweaks") @@ -890,7 +868,7 @@ def _add_all_groups(helpful): helpful.add_group(name, description=docs["opts"]) -def prepare_and_parse_args(plugins, args, detect_defaults=False): # pylint: disable=too-many-statements +def prepare_and_parse_args(plugins, args, detect_defaults=False): """Returns parsed command line arguments. :param .PluginsRegistry plugins: available plugins @@ -901,7 +879,6 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False): # pylint: dis """ - # pylint: disable=too-many-statements helpful = HelpfulArgumentParser(args, plugins, detect_defaults) _add_all_groups(helpful) @@ -1000,12 +977,6 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False): # pylint: dis "certificates. Updates to the Subscriber Agreement will still " "affect you, and will be effective 14 days after posting an " "update to the web site.") - # TODO: When `certbot register --update-registration` is fully deprecated, - # delete following helpful.add - helpful.add( - "register", "--update-registration", action="store_true", - default=flag_default("update_registration"), dest="update_registration", - help=argparse.SUPPRESS) helpful.add( ["register", "update_account", "unregister", "automation"], "-m", "--email", default=flag_default("email"), @@ -1091,6 +1062,11 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False): # pylint: dis help="(certbot-auto only) prevent the certbot-auto script from" " installing OS-level dependencies (default: Prompt to install " " OS-wide dependencies, but exit if the user says 'No')") + helpful.add( + "automation", "--no-permissions-check", action="store_true", + default=flag_default("no_permissions_check"), + help="(certbot-auto only) skip the check on the file system" + " permissions of the certbot-auto script") helpful.add( ["automation", "renew", "certonly", "run"], "-q", "--quiet", dest="quiet", action="store_true", @@ -1115,14 +1091,6 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False): # pylint: dis "testing", "--no-verify-ssl", action="store_true", help=config_help("no_verify_ssl"), default=flag_default("no_verify_ssl")) - helpful.add( - ["testing", "standalone", "apache", "nginx"], "--tls-sni-01-port", type=int, - default=flag_default("tls_sni_01_port"), - help=config_help("tls_sni_01_port")) - helpful.add( - ["testing", "standalone"], "--tls-sni-01-address", - default=flag_default("tls_sni_01_address"), - help=config_help("tls_sni_01_address")) helpful.add( ["testing", "standalone", "manual"], "--http-01-port", type=int, dest="http01_port", @@ -1131,6 +1099,10 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False): # pylint: dis ["testing", "standalone"], "--http-01-address", dest="http01_address", default=flag_default("http01_address"), help=config_help("http01_address")) + helpful.add( + ["testing", "nginx"], "--https-port", type=int, + default=flag_default("https_port"), + help=config_help("https_port")) helpful.add( "testing", "--break-my-certs", action="store_true", default=flag_default("break_my_certs"), @@ -1191,7 +1163,7 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False): # pylint: dis action=_PrefChallAction, default=flag_default("pref_challs"), help='A sorted, comma delimited list of the preferred challenge to ' 'use during authorization with the most preferred challenge ' - 'listed first (Eg, "dns" or "tls-sni-01,http,dns"). ' + 'listed first (Eg, "dns" or "http,dns"). ' 'Not all plugins support all challenges. See ' 'https://certbot.eff.org/docs/using.html#plugins for details. ' 'ACME Challenges are versioned, but if you pick "http" rather ' @@ -1259,9 +1231,6 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False): # pylint: dis default=flag_default("autorenew"), dest="autorenew", help="Disable auto renewal of certificates.") - helpful.add_deprecated_argument("--agree-dev-preview", 0) - helpful.add_deprecated_argument("--dialog", 0) - # Populate the command line parameters for new style enhancements enhancements.populate_cli(helpful.add) @@ -1278,10 +1247,7 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False): # pylint: dis def _create_subparsers(helpful): - helpful.add("config_changes", "--num", type=int, default=flag_default("num"), - help="How many past revisions you want to be displayed") - - from certbot.client import sample_user_agent # avoid import loops + from certbot._internal.client import sample_user_agent # avoid import loops helpful.add( None, "--user-agent", default=flag_default("user_agent"), help='Set a custom user agent string for the client. User agent strings allow ' @@ -1410,10 +1376,10 @@ def _plugins_parsing(helpful, plugins): help="Authenticator plugin name.") helpful.add("plugins", "-i", "--installer", default=flag_default("installer"), help="Installer plugin name (also used to find domains).") - helpful.add(["plugins", "certonly", "run", "install", "config_changes"], + helpful.add(["plugins", "certonly", "run", "install"], "--apache", action="store_true", default=flag_default("apache"), help="Obtain and install certificates using Apache") - helpful.add(["plugins", "certonly", "run", "install", "config_changes"], + helpful.add(["plugins", "certonly", "run", "install"], "--nginx", action="store_true", default=flag_default("nginx"), help="Obtain and install certificates using Nginx") helpful.add(["plugins", "certonly"], "--standalone", action="store_true", @@ -1443,12 +1409,12 @@ def _plugins_parsing(helpful, plugins): "using DNSimple for DNS).")) helpful.add(["plugins", "certonly"], "--dns-dnsmadeeasy", action="store_true", default=flag_default("dns_dnsmadeeasy"), - help=("Obtain certificates using a DNS TXT record (if you are" + help=("Obtain certificates using a DNS TXT record (if you are " "using DNS Made Easy for DNS).")) helpful.add(["plugins", "certonly"], "--dns-gehirn", action="store_true", default=flag_default("dns_gehirn"), help=("Obtain certificates using a DNS TXT record " - "(if you are using Gehirn Infrastracture Service for DNS).")) + "(if you are using Gehirn Infrastructure Service for DNS).")) helpful.add(["plugins", "certonly"], "--dns-google", action="store_true", default=flag_default("dns_google"), help=("Obtain certificates using a DNS TXT record (if you are " @@ -1551,9 +1517,10 @@ def parse_preferred_challenges(pref_challs): :raises errors.Error: if pref_challs is invalid """ - aliases = {"dns": "dns-01", "http": "http-01", "tls-sni": "tls-sni-01"} + aliases = {"dns": "dns-01", "http": "http-01"} challs = [c.strip() for c in pref_challs] challs = [aliases.get(c, c) for c in challs] + unrecognized = ", ".join(name for name in challs if name not in challenges.Challenge.TYPES) if unrecognized: @@ -1561,11 +1528,13 @@ def parse_preferred_challenges(pref_challs): "Unrecognized challenges: {0}".format(unrecognized)) return challs + def _user_agent_comment_type(value): if "(" in value or ")" in value: raise argparse.ArgumentTypeError("may not contain parentheses") return value + class _DeployHookAction(argparse.Action): """Action class for parsing deploy hooks.""" diff --git a/certbot/client.py b/certbot/certbot/_internal/client.py similarity index 92% rename from certbot/client.py rename to certbot/certbot/_internal/client.py index 38b77a772..9ce741e38 100644 --- a/certbot/client.py +++ b/certbot/certbot/_internal/client.py @@ -1,13 +1,10 @@ """Certbot client API.""" import datetime import logging -import os import platform - from cryptography.hazmat.backends import default_backend -# https://github.com/python/typeshed/blob/master/third_party/ -# 2/cryptography/hazmat/primitives/asymmetric/rsa.pyi +# See https://github.com/pyca/cryptography/issues/4275 from cryptography.hazmat.primitives.asymmetric.rsa import generate_private_key # type: ignore import josepy as jose import OpenSSL @@ -17,28 +14,24 @@ from acme import client as acme_client from acme import crypto_util as acme_crypto_util from acme import errors as acme_errors from acme import messages -from acme.magic_typing import Optional # pylint: disable=unused-import,no-name-in-module - +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Optional # pylint: disable=unused-import, no-name-in-module import certbot - -from certbot import account -from certbot import auth_handler -from certbot import cli -from certbot import compat -from certbot import constants from certbot import crypto_util -from certbot import eff -from certbot import error_handler from certbot import errors from certbot import interfaces -from certbot import reverter -from certbot import storage from certbot import util - +from certbot._internal import account +from certbot._internal import auth_handler +from certbot._internal import cli +from certbot._internal import constants +from certbot._internal import eff +from certbot._internal import error_handler +from certbot._internal import storage +from certbot._internal.display import enhancements +from certbot._internal.plugins import selection as plugin_selection +from certbot.compat import os from certbot.display import ops as display_ops -from certbot.display import enhancements -from certbot.plugins import selection as plugin_selection - logger = logging.getLogger(__name__) @@ -231,11 +224,9 @@ def perform_registration(acme, config, tos_cb): "Please ensure it is a valid email and attempt " "registration again." % config.email) raise errors.Error(msg) - else: - config.email = display_ops.get_email(invalid=True) - return perform_registration(acme, config, tos_cb) - else: - raise + config.email = display_ops.get_email(invalid=True) + return perform_registration(acme, config, tos_cb) + raise class Client(object): @@ -352,7 +343,7 @@ class Client(object): orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names) authzr = orderr.authorizations - auth_domains = set(a.body.identifier.value for a in authzr) + auth_domains = set(a.body.identifier.value for a in authzr) # pylint: disable=not-an-iterable successful_domains = [d for d in domains if d in auth_domains] # allow_subset_of_names is currently disabled for wildcard @@ -367,10 +358,10 @@ class Client(object): return self.obtain_certificate(successful_domains) else: cert, chain = self.obtain_certificate_from_csr(csr, orderr) - return cert, chain, key, csr def _get_order_and_authorizations(self, csr_pem, best_effort): + # type: (str, bool) -> List[messages.OrderResource] """Request a new order and complete its authorizations. :param str csr_pem: A CSR in PEM format. @@ -386,10 +377,19 @@ class Client(object): except acme_errors.WildcardUnsupportedError: raise errors.Error("The currently selected ACME CA endpoint does" " not support issuing wildcard certificates.") + + # For a dry run, ensure we have an order with fresh authorizations + if orderr and self.config.dry_run: + deactivated, failed = self.auth_handler.deactivate_valid_authorizations(orderr) + if deactivated: + logger.debug("Recreating order after authz deactivations") + orderr = self.acme.new_order(csr_pem) + if failed: + logger.warning("Certbot was unable to obtain fresh authorizations for every domain" + ". The dry run will continue, but results may not be accurate.") + authzr = self.auth_handler.handle_authorizations(orderr, best_effort) return orderr.update(authorizations=authzr) - - # pylint: disable=no-member def obtain_and_enroll_certificate(self, domains, certname): """Obtain and enroll certificate. @@ -402,7 +402,7 @@ class Client(object): :param certname: requested name of lineage :type certname: `str` or `None` - :returns: A new :class:`certbot.storage.RenewableCert` instance + :returns: A new :class:`certbot._internal.storage.RenewableCert` instance referred to the enrolled cert lineage, False if the cert could not be obtained, or None if doing a successful dry run. @@ -421,11 +421,10 @@ class Client(object): logger.debug("Dry run: Skipping creating new lineage for %s", new_name) return None - else: - return storage.RenewableCert.new_lineage( - new_name, cert, - key.pem, chain, - self.config) + return storage.RenewableCert.new_lineage( + new_name, cert, + key.pem, chain, + self.config) def _choose_lineagename(self, domains, certname): """Chooses a name for the new lineage. @@ -444,8 +443,7 @@ class Client(object): elif util.is_wildcard_domain(domains[0]): # Don't make files and directories starting with *. return domains[0][2:] - else: - return domains[0] + return domains[0] def save_certificate(self, cert_pem, chain_pem, cert_path, chain_path, fullchain_path): @@ -465,9 +463,7 @@ class Client(object): """ for path in cert_path, chain_path, fullchain_path: - util.make_or_verify_dir( - os.path.dirname(path), 0o755, compat.os_geteuid(), - self.config.strict_permissions) + util.make_or_verify_dir(os.path.dirname(path), 0o755, self.config.strict_permissions) cert_file, abs_cert_path = _open_pem_file('cert_path', cert_path) @@ -555,6 +551,11 @@ class Client(object): if ask_redirect: if config_name == "redirect" and config_value is None: config_value = enhancements.ask(enhancement_name) + if not config_value: + logger.warning("Future versions of Certbot will automatically " + "configure the webserver so that all requests redirect to secure " + "HTTPS access. You can control this behavior and disable this " + "warning with the --redirect and --no-redirect flags.") if config_value: self.apply_enhancement(domains, enhancement_name, option) enhanced = True @@ -628,7 +629,7 @@ class Client(object): reporter.add_message( "An error occurred and we failed to restore your config and " "restart your server. Please post to " - "https://community.letsencrypt.org/c/server-config " + "https://community.letsencrypt.org/c/help " "with details about your configuration and this error you received.", reporter.HIGH_PRIORITY) raise @@ -702,20 +703,6 @@ def rollback(default_installer, checkpoints, config, plugins): installer.rollback_checkpoints(checkpoints) installer.restart() - -def view_config_changes(config, num=None): - """View checkpoints and associated configuration changes. - - .. note:: This assumes that the installation is using a Reverter object. - - :param config: Configuration. - :type config: :class:`certbot.interfaces.IConfig` - - """ - rev = reverter.Reverter(config) - rev.recovery_routine() - rev.view_config_changes(num) - def _open_pem_file(cli_arg_path, pem_path): """Open a pem file. @@ -731,9 +718,8 @@ def _open_pem_file(cli_arg_path, pem_path): if cli.set_by_cli(cli_arg_path): return util.safe_open(pem_path, chmod=0o644, mode="wb"),\ os.path.abspath(pem_path) - else: - uniq = util.unique_file(pem_path, 0o644, "wb") - return uniq[0], os.path.abspath(uniq[1]) + uniq = util.unique_file(pem_path, 0o644, "wb") + return uniq[0], os.path.abspath(uniq[1]) def _save_chain(chain_pem, chain_file): """Saves chain_pem at a unique path based on chain_path. diff --git a/certbot/configuration.py b/certbot/certbot/_internal/configuration.py similarity index 91% rename from certbot/configuration.py rename to certbot/certbot/_internal/configuration.py index 15f9fa3e0..f3db207db 100644 --- a/certbot/configuration.py +++ b/certbot/certbot/_internal/configuration.py @@ -1,15 +1,15 @@ """Certbot user-supplied configuration.""" import copy -import os -from six.moves.urllib import parse # pylint: disable=import-error +from six.moves.urllib import parse import zope.interface -from certbot import compat -from certbot import constants from certbot import errors from certbot import interfaces from certbot import util +from certbot._internal import constants +from certbot.compat import misc +from certbot.compat import os @zope.interface.implementer(interfaces.IConfig) @@ -20,7 +20,7 @@ class NamespaceConfig(object): :class:`certbot.interfaces.IConfig`. However, note that the following attributes are dynamically resolved using :attr:`~certbot.interfaces.IConfig.work_dir` and relative - paths defined in :py:mod:`certbot.constants`: + paths defined in :py:mod:`certbot._internal.constants`: - `accounts_dir` - `csr_dir` @@ -30,7 +30,7 @@ class NamespaceConfig(object): And the following paths are dynamically resolved using :attr:`~certbot.interfaces.IConfig.config_dir` and relative - paths defined in :py:mod:`certbot.constants`: + paths defined in :py:mod:`certbot._internal.constants`: - `default_archive_dir` - `live_dir` @@ -70,7 +70,7 @@ class NamespaceConfig(object): def accounts_dir_for_server_path(self, server_path): """Path to accounts directory based on server_path""" - server_path = compat.underscores_for_unsupported_characters_in_path(server_path) + server_path = misc.underscores_for_unsupported_characters_in_path(server_path) return os.path.join( self.namespace.config_dir, constants.ACCOUNTS_DIR, server_path) @@ -148,10 +148,10 @@ def check_config_sanity(config): """ # Port check - if config.http01_port == config.tls_sni_01_port: + if config.http01_port == config.https_port: raise errors.ConfigurationError( - "Trying to run http-01 and tls-sni-01 " - "on the same port ({0})".format(config.tls_sni_01_port)) + "Trying to run http-01 and https-port " + "on the same port ({0})".format(config.https_port)) # Domain checks if config.namespace.domains is not None: diff --git a/certbot/constants.py b/certbot/certbot/_internal/constants.py similarity index 86% rename from certbot/constants.py rename to certbot/certbot/_internal/constants.py index c6a80747e..9a2220e0b 100644 --- a/certbot/constants.py +++ b/certbot/certbot/_internal/constants.py @@ -1,10 +1,11 @@ """Certbot constants.""" import logging -import os + import pkg_resources from acme import challenges -from certbot import compat +from certbot.compat import misc +from certbot.compat import os SETUPTOOLS_PLUGINS_ENTRY_POINT = "certbot.plugins" """Setuptools entry point group name for plugins.""" @@ -14,7 +15,7 @@ OLD_SETUPTOOLS_PLUGINS_ENTRY_POINT = "letsencrypt.plugins" CLI_DEFAULTS = dict( config_files=[ - os.path.join(compat.get_default_folder('config'), 'cli.ini'), + os.path.join(misc.get_default_folder('config'), 'cli.ini'), # http://freedesktop.org/wiki/Software/xdg-user-dirs/ os.path.join(os.environ.get("XDG_CONFIG_HOME", "~/.config"), "letsencrypt", "cli.ini"), @@ -30,7 +31,6 @@ CLI_DEFAULTS = dict( certname=None, dry_run=False, register_unsafely_without_email=False, - update_registration=False, email=None, eff_email=None, reinstall=False, @@ -44,16 +44,16 @@ CLI_DEFAULTS = dict( duplicate=False, os_packages_only=False, no_self_upgrade=False, + no_permissions_check=False, no_bootstrap=False, quiet=False, staging=False, debug=False, debug_challenges=False, no_verify_ssl=False, - tls_sni_01_port=challenges.TLSSNI01Response.PORT, - tls_sni_01_address="", http01_port=challenges.HTTP01Response.PORT, http01_address="", + https_port=443, break_my_certs=False, rsa_key_size=2048, must_staple=False, @@ -88,9 +88,9 @@ CLI_DEFAULTS = dict( auth_cert_path="./cert.pem", auth_chain_path="./chain.pem", key_path=None, - config_dir=compat.get_default_folder('config'), - work_dir=compat.get_default_folder('work'), - logs_dir=compat.get_default_folder('logs'), + config_dir=misc.get_default_folder('config'), + work_dir=misc.get_default_folder('work'), + logs_dir=misc.get_default_folder('logs'), server="https://acme-v02.api.letsencrypt.org/directory", # Plugins parsers @@ -145,18 +145,6 @@ RENEWER_DEFAULTS = dict( ) """Defaults for renewer script.""" - -ENHANCEMENTS = ["redirect", "ensure-http-header", "ocsp-stapling"] -"""List of possible :class:`certbot.interfaces.IInstaller` -enhancements. - -List of expected options parameters: -- redirect: None -- ensure-http-header: name of header (i.e. Strict-Transport-Security) -- ocsp-stapling: certificate chain file path - -""" - ARCHIVE_DIR = "archive" """Archive directory, relative to `IConfig.config_dir`.""" @@ -167,9 +155,10 @@ ACCOUNTS_DIR = "accounts" """Directory where all accounts are saved.""" LE_REUSE_SERVERS = { - 'acme-v02.api.letsencrypt.org/directory': 'acme-v01.api.letsencrypt.org/directory', - 'acme-staging-v02.api.letsencrypt.org/directory': - 'acme-staging.api.letsencrypt.org/directory' + os.path.normpath('acme-v02.api.letsencrypt.org/directory'): + os.path.normpath('acme-v01.api.letsencrypt.org/directory'), + os.path.normpath('acme-staging-v02.api.letsencrypt.org/directory'): + os.path.normpath('acme-staging.api.letsencrypt.org/directory') } """Servers that can reuse accounts from other servers.""" diff --git a/certbot/display/__init__.py b/certbot/certbot/_internal/display/__init__.py similarity index 100% rename from certbot/display/__init__.py rename to certbot/certbot/_internal/display/__init__.py diff --git a/certbot/display/completer.py b/certbot/certbot/_internal/display/completer.py similarity index 96% rename from certbot/display/completer.py rename to certbot/certbot/_internal/display/completer.py index 509a1051a..03719862b 100644 --- a/certbot/display/completer.py +++ b/certbot/certbot/_internal/display/completer.py @@ -1,10 +1,11 @@ """Provides Tab completion when prompting users for a path.""" import glob + # readline module is not available on all systems try: import readline except ImportError: - import certbot.display.dummy_readline as readline # type: ignore + import certbot._internal.display.dummy_readline as readline # type: ignore class Completer(object): diff --git a/certbot/display/dummy_readline.py b/certbot/certbot/_internal/display/dummy_readline.py similarity index 100% rename from certbot/display/dummy_readline.py rename to certbot/certbot/_internal/display/dummy_readline.py diff --git a/certbot/display/enhancements.py b/certbot/certbot/_internal/display/enhancements.py similarity index 94% rename from certbot/display/enhancements.py rename to certbot/certbot/_internal/display/enhancements.py index 0f6b6c57d..ce6470708 100644 --- a/certbot/display/enhancements.py +++ b/certbot/certbot/_internal/display/enhancements.py @@ -7,7 +7,6 @@ from certbot import errors from certbot import interfaces from certbot.display import util as display_util - logger = logging.getLogger(__name__) # Define a helper function to avoid verbose code @@ -18,7 +17,7 @@ def ask(enhancement): """Display the enhancement to the user. :param str enhancement: One of the - :class:`certbot.CONFIG.ENHANCEMENTS` enhancements + :const:`~certbot.plugins.enhancements.ENHANCEMENTS` enhancements :returns: True if feature is desired, False otherwise :rtype: bool @@ -50,7 +49,7 @@ def redirect_by_default(): code, selection = util(interfaces.IDisplay).menu( "Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.", - choices, default=0, + choices, default=1, cli_flag="--redirect / --no-redirect", force_interactive=True) if code != display_util.OK: diff --git a/certbot/eff.py b/certbot/certbot/_internal/eff.py similarity index 97% rename from certbot/eff.py rename to certbot/certbot/_internal/eff.py index 388ae986b..586697dbb 100644 --- a/certbot/eff.py +++ b/certbot/certbot/_internal/eff.py @@ -4,9 +4,8 @@ import logging import requests import zope.component -from certbot import constants from certbot import interfaces - +from certbot._internal import constants logger = logging.getLogger(__name__) @@ -73,7 +72,7 @@ def _check_response(response): logger.debug('Received response:\n%s', response.content) try: response.raise_for_status() - if response.json()['status'] == False: + if not response.json()['status']: _report_failure('your e-mail address appears to be invalid') except requests.exceptions.HTTPError: _report_failure() diff --git a/certbot/error_handler.py b/certbot/certbot/_internal/error_handler.py similarity index 76% rename from certbot/error_handler.py rename to certbot/certbot/_internal/error_handler.py index 5e72f8153..5ca3cc57e 100644 --- a/certbot/error_handler.py +++ b/certbot/certbot/_internal/error_handler.py @@ -1,15 +1,16 @@ """Registers functions to be called if an exception or signal occurs.""" import functools import logging -import os import signal import traceback -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import Any, Callable, Dict, List, Union -# pylint: enable=unused-import, no-name-in-module - +from acme.magic_typing import Any # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Callable # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Union # pylint: disable=unused-import, no-name-in-module from certbot import errors +from certbot.compat import os logger = logging.getLogger(__name__) @@ -19,14 +20,30 @@ logger = logging.getLogger(__name__) # potentially occur from inside Python. Signals such as SIGILL were not # included as they could be a sign of something devious and we should terminate # immediately. -_SIGNALS = [signal.SIGTERM] if os.name != "nt": + _SIGNALS = [signal.SIGTERM] for signal_code in [signal.SIGHUP, signal.SIGQUIT, signal.SIGXCPU, signal.SIGXFSZ]: # Adding only those signals that their default action is not Ignore. # This is platform-dependent, so we check it dynamically. if signal.getsignal(signal_code) != signal.SIG_IGN: _SIGNALS.append(signal_code) +else: + # POSIX signals are not implemented natively in Windows, but emulated from the C runtime. + # As consumed by CPython, most of handlers on theses signals are useless, in particular + # SIGTERM: for instance, os.kill(pid, signal.SIGTERM) will call TerminateProcess, that stops + # immediately the process without calling the attached handler. Besides, non-POSIX signals + # (CTRL_C_EVENT and CTRL_BREAK_EVENT) are implemented in a console context to handle the + # CTRL+C event to a process launched from the console. Only CTRL_C_EVENT has a reliable + # behavior in fact, and maps to the handler to SIGINT. However in this case, a + # KeyboardInterrupt is raised, that will be handled by ErrorHandler through the context manager + # protocol. Finally, no signal on Windows is electable to be handled using ErrorHandler. + # + # Refs: https://stackoverflow.com/a/35792192, https://maruel.ca/post/python_windows_signal, + # https://docs.python.org/2/library/os.html#os.kill, + # https://www.reddit.com/r/Python/comments/1dsblt/windows_command_line_automation_ctrlc_question + _SIGNALS = [] + class ErrorHandler(object): """Context manager for running code that must be cleaned up on failure. @@ -57,7 +74,7 @@ class ErrorHandler(object): deferred until they finish. """ - def __init__(self, func=None, *args, **kwargs): + def __init__(self, func, *args, **kwargs): self.call_on_regular_exit = False self.body_executed = False self.funcs = [] # type: List[Callable[[], Any]] @@ -76,7 +93,7 @@ class ErrorHandler(object): # SystemExit is ignored to properly handle forks that don't exec if exec_type is SystemExit: return retval - elif exec_type is None: + if exec_type is None: if not self.call_on_regular_exit: return retval elif exec_type is errors.SignalExit: @@ -151,7 +168,6 @@ class ExitHandler(ErrorHandler): In addition to cleaning up on all signals, also cleans up on regular exit. """ - def __init__(self, func=None, *args, **kwargs): + def __init__(self, func, *args, **kwargs): ErrorHandler.__init__(self, func, *args, **kwargs) self.call_on_regular_exit = True - diff --git a/certbot/hooks.py b/certbot/certbot/_internal/hooks.py similarity index 84% rename from certbot/hooks.py rename to certbot/certbot/_internal/hooks.py index d5239a437..25addd915 100644 --- a/certbot/hooks.py +++ b/certbot/certbot/_internal/hooks.py @@ -2,14 +2,15 @@ from __future__ import print_function import logging -import os +from subprocess import PIPE +from subprocess import Popen -from subprocess import Popen, PIPE - -from acme.magic_typing import Set, List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module from certbot import errors from certbot import util - +from certbot.compat import filesystem +from certbot.compat import os from certbot.plugins import util as plug_util logger = logging.getLogger(__name__) @@ -93,8 +94,7 @@ def _run_pre_hook_if_necessary(command): if command in executed_pre_hooks: logger.info("Pre-hook command already run, skipping: %s", command) else: - logger.info("Running pre-hook command: %s", command) - _run_hook(command) + _run_hook("pre-hook", command) executed_pre_hooks.add(command) @@ -126,8 +126,7 @@ def post_hook(config): _run_eventually(cmd) # certonly / run elif cmd: - logger.info("Running post-hook command: %s", cmd) - _run_hook(cmd) + _run_hook("post-hook", cmd) post_hooks = [] # type: List[str] @@ -149,8 +148,7 @@ def _run_eventually(command): def run_saved_post_hooks(): """Run any post hooks that were saved up in the course of the 'renew' verb""" for cmd in post_hooks: - logger.info("Running post-hook command: %s", cmd) - _run_hook(cmd) + _run_hook("post-hook", cmd) def deploy_hook(config, domains, lineage_path): @@ -220,23 +218,30 @@ def _run_deploy_hook(command, domains, lineage_path, dry_run): os.environ["RENEWED_DOMAINS"] = " ".join(domains) os.environ["RENEWED_LINEAGE"] = lineage_path - logger.info("Running deploy-hook command: %s", command) - _run_hook(command) + _run_hook("deploy-hook", command) -def _run_hook(shell_cmd): +def _run_hook(cmd_name, shell_cmd): """Run a hook command. - :returns: stderr if there was any""" + :param str cmd_name: the user facing name of the hook being run + :param shell_cmd: shell command to execute + :type shell_cmd: `list` of `str` or `str` - err, _ = execute(shell_cmd) + :returns: stderr if there was any""" + err, _ = execute(cmd_name, shell_cmd) return err -def execute(shell_cmd): +def execute(cmd_name, shell_cmd): """Run a command. + :param str cmd_name: the user facing name of the hook being run + :param shell_cmd: shell command to execute + :type shell_cmd: `list` of `str` or `str` + :returns: `tuple` (`str` stderr, `str` stdout)""" + logger.info("Running %s command: %s", cmd_name, shell_cmd) # universal_newlines causes Popen.communicate() # to return str objects instead of bytes in Python 3 @@ -245,13 +250,13 @@ def execute(shell_cmd): out, err = cmd.communicate() base_cmd = os.path.basename(shell_cmd.split(None, 1)[0]) if out: - logger.info('Output from %s:\n%s', base_cmd, out) + logger.info('Output from %s command %s:\n%s', cmd_name, base_cmd, out) if cmd.returncode != 0: - logger.error('Hook command "%s" returned error code %d', - shell_cmd, cmd.returncode) + logger.error('%s command "%s" returned error code %d', + cmd_name, shell_cmd, cmd.returncode) if err: - logger.error('Error output from %s:\n%s', base_cmd, err) - return (err, out) + logger.error('Error output from %s command %s:\n%s', cmd_name, base_cmd, err) + return err, out def list_hooks(dir_path): @@ -263,5 +268,6 @@ def list_hooks(dir_path): :rtype: sorted list of absolute paths to executables in dir_path """ - paths = (os.path.join(dir_path, f) for f in os.listdir(dir_path)) - return sorted(path for path in paths if util.is_exe(path)) + allpaths = (os.path.join(dir_path, f) for f in os.listdir(dir_path)) + hooks = [path for path in allpaths if filesystem.is_executable(path) and not path.endswith('~')] + return sorted(hooks) diff --git a/certbot/certbot/_internal/lock.py b/certbot/certbot/_internal/lock.py new file mode 100644 index 000000000..7823eaac3 --- /dev/null +++ b/certbot/certbot/_internal/lock.py @@ -0,0 +1,263 @@ +"""Implements file locks compatible with Linux and Windows for locking files and directories.""" +import errno +import logging + +from acme.magic_typing import Optional # pylint: disable=unused-import, no-name-in-module +from certbot import errors +from certbot.compat import filesystem +from certbot.compat import os + +try: + import fcntl # pylint: disable=import-error +except ImportError: + import msvcrt # pylint: disable=import-error + POSIX_MODE = False +else: + POSIX_MODE = True + + + +logger = logging.getLogger(__name__) + + +def lock_dir(dir_path): + # type: (str) -> LockFile + """Place a lock file on the directory at dir_path. + + The lock file is placed in the root of dir_path with the name + .certbot.lock. + + :param str dir_path: path to directory + + :returns: the locked LockFile object + :rtype: LockFile + + :raises errors.LockError: if unable to acquire the lock + + """ + return LockFile(os.path.join(dir_path, '.certbot.lock')) + + +class LockFile(object): + """ + Platform independent file lock system. + LockFile accepts a parameter, the path to a file acting as a lock. Once the LockFile, + instance is created, the associated file is 'locked from the point of view of the OS, + meaning that if another instance of Certbot try at the same time to acquire the same lock, + it will raise an Exception. Calling release method will release the lock, and make it + available to every other instance. + Upon exit, Certbot will also release all the locks. + This allows us to protect a file or directory from being concurrently accessed + or modified by two Certbot instances. + LockFile is platform independent: it will proceed to the appropriate OS lock mechanism + depending on Linux or Windows. + """ + def __init__(self, path): + # type: (str) -> None + """ + Create a LockFile instance on the given file path, and acquire lock. + :param str path: the path to the file that will hold a lock + """ + self._path = path + mechanism = _UnixLockMechanism if POSIX_MODE else _WindowsLockMechanism + self._lock_mechanism = mechanism(path) + + self.acquire() + + def __repr__(self): + # type: () -> str + repr_str = '{0}({1}) <'.format(self.__class__.__name__, self._path) + if self.is_locked(): + repr_str += 'acquired>' + else: + repr_str += 'released>' + return repr_str + + def acquire(self): + # type: () -> None + """ + Acquire the lock on the file, forbidding any other Certbot instance to acquire it. + :raises errors.LockError: if unable to acquire the lock + """ + self._lock_mechanism.acquire() + + def release(self): + # type: () -> None + """ + Release the lock on the file, allowing any other Certbot instance to acquire it. + """ + self._lock_mechanism.release() + + def is_locked(self): + # type: () -> bool + """ + Check if the file is currently locked. + :return: True if the file is locked, False otherwise + """ + return self._lock_mechanism.is_locked() + + +class _BaseLockMechanism(object): + def __init__(self, path): + # type: (str) -> None + """ + Create a lock file mechanism for Unix. + :param str path: the path to the lock file + """ + self._path = path + self._fd = None # type: Optional[int] + + def is_locked(self): + # type: () -> bool + """Check if lock file is currently locked. + :return: True if the lock file is locked + :rtype: bool + """ + return self._fd is not None + + def acquire(self): # pylint: disable=missing-docstring + pass # pragma: no cover + + def release(self): # pylint: disable=missing-docstring + pass # pragma: no cover + + +class _UnixLockMechanism(_BaseLockMechanism): + """ + A UNIX lock file mechanism. + This lock file is released when the locked file is closed or the + process exits. It cannot be used to provide synchronization between + threads. It is based on the lock_file package by Martin Horcicka. + """ + def acquire(self): + # type: () -> None + """Acquire the lock.""" + while self._fd is None: + # Open the file + fd = filesystem.open(self._path, os.O_CREAT | os.O_WRONLY, 0o600) + try: + self._try_lock(fd) + if self._lock_success(fd): + self._fd = fd + finally: + # Close the file if it is not the required one + if self._fd is None: + os.close(fd) + + def _try_lock(self, fd): + # type: (int) -> None + """ + Try to acquire the lock file without blocking. + :param int fd: file descriptor of the opened file to lock + """ + try: + fcntl.lockf(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + except IOError as err: + if err.errno in (errno.EACCES, errno.EAGAIN): + logger.debug('A lock on %s is held by another process.', self._path) + raise errors.LockError('Another instance of Certbot is already running.') + raise + + def _lock_success(self, fd): + # type: (int) -> bool + """ + Did we successfully grab the lock? + Because this class deletes the locked file when the lock is + released, it is possible another process removed and recreated + the file between us opening the file and acquiring the lock. + :param int fd: file descriptor of the opened file to lock + :returns: True if the lock was successfully acquired + :rtype: bool + """ + # Normally os module should not be imported in certbot codebase except in certbot.compat + # for the sake of compatibility over Windows and Linux. + # We make an exception here, since _lock_success is private and called only on Linux. + from os import stat, fstat # pylint: disable=os-module-forbidden + try: + stat1 = stat(self._path) + except OSError as err: + if err.errno == errno.ENOENT: + return False + raise + + stat2 = fstat(fd) + # If our locked file descriptor and the file on disk refer to + # the same device and inode, they're the same file. + return stat1.st_dev == stat2.st_dev and stat1.st_ino == stat2.st_ino + + def release(self): + # type: () -> None + """Remove, close, and release the lock file.""" + # It is important the lock file is removed before it's released, + # otherwise: + # + # process A: open lock file + # process B: release lock file + # process A: lock file + # process A: check device and inode + # process B: delete file + # process C: open and lock a different file at the same path + try: + os.remove(self._path) + finally: + # Following check is done to make mypy happy: it ensure that self._fd, marked + # as Optional[int] is effectively int to make it compatible with os.close signature. + if self._fd is None: # pragma: no cover + raise TypeError('Error, self._fd is None.') + try: + os.close(self._fd) + finally: + self._fd = None + + +class _WindowsLockMechanism(_BaseLockMechanism): + """ + A Windows lock file mechanism. + By default on Windows, acquiring a file handler gives exclusive access to the process + and results in an effective lock. However, it is possible to explicitly acquire the + file handler in shared access in terms of read and write, and this is done by os.open + and io.open in Python. So an explicit lock needs to be done through the call of + msvcrt.locking, that will lock the first byte of the file. In theory, it is also + possible to access a file in shared delete access, allowing other processes to delete an + opened file. But this needs also to be done explicitly by all processes using the Windows + low level APIs, and Python does not do it. As of Python 3.7 and below, Python developers + state that deleting a file opened by a process from another process is not possible with + os.open and io.open. + Consequently, mscvrt.locking is sufficient to obtain an effective lock, and the race + condition encountered on Linux is not possible on Windows, leading to a simpler workflow. + """ + def acquire(self): + """Acquire the lock""" + open_mode = os.O_RDWR | os.O_CREAT | os.O_TRUNC + + fd = None + try: + # Under Windows, filesystem.open will raise directly an EACCES error + # if the lock file is already locked. + fd = filesystem.open(self._path, open_mode, 0o600) + msvcrt.locking(fd, msvcrt.LK_NBLCK, 1) + except (IOError, OSError) as err: + if fd: + os.close(fd) + # Anything except EACCES is unexpected. Raise directly the error in that case. + if err.errno != errno.EACCES: + raise + logger.debug('A lock on %s is held by another process.', self._path) + raise errors.LockError('Another instance of Certbot is already running.') + + self._fd = fd + + def release(self): + """Release the lock.""" + try: + msvcrt.locking(self._fd, msvcrt.LK_UNLCK, 1) + os.close(self._fd) + + try: + os.remove(self._path) + except OSError as e: + # If the lock file cannot be removed, it is not a big deal. + # Likely another instance is acquiring the lock we just released. + logger.debug(str(e)) + finally: + self._fd = None diff --git a/certbot/log.py b/certbot/certbot/_internal/log.py similarity index 94% rename from certbot/log.py rename to certbot/certbot/_internal/log.py index b883936f3..0a492ba55 100644 --- a/certbot/log.py +++ b/certbot/certbot/_internal/log.py @@ -13,20 +13,20 @@ and properly flushed before program exit. """ from __future__ import print_function + import functools import logging import logging.handlers -import os +import shutil import sys import tempfile import traceback from acme import messages - -from certbot import compat -from certbot import constants from certbot import errors from certbot import util +from certbot._internal import constants +from certbot.compat import os # Logging format CLI_FMT = "%(message)s" @@ -40,7 +40,7 @@ def pre_arg_parse_setup(): """Setup logging before command line arguments are parsed. Terminal logging is setup using - `certbot.constants.QUIET_LOGGING_LEVEL` so Certbot is as quiet as + `certbot._internal.constants.QUIET_LOGGING_LEVEL` so Certbot is as quiet as possible. File logging is setup so that logging messages are buffered in memory. If Certbot exits before `post_arg_parse_setup` is called, these buffered messages are written to a temporary file. @@ -102,9 +102,9 @@ def post_arg_parse_setup(config): root_logger.addHandler(file_handler) root_logger.removeHandler(memory_handler) - temp_handler = memory_handler.target - memory_handler.setTarget(file_handler) - memory_handler.flush(force=True) + temp_handler = memory_handler.target # pylint: disable=no-member + memory_handler.setTarget(file_handler) # pylint: disable=no-member + memory_handler.flush(force=True) # pylint: disable=unexpected-keyword-arg memory_handler.close() temp_handler.close() @@ -133,8 +133,7 @@ def setup_log_file_handler(config, logfile, fmt): """ # TODO: logs might contain sensitive data such as contents of the # private key! #525 - util.set_up_core_dir( - config.logs_dir, 0o700, compat.os_geteuid(), config.strict_permissions) + util.set_up_core_dir(config.logs_dir, 0o700, config.strict_permissions) log_file_path = os.path.join(config.logs_dir, logfile) try: handler = logging.handlers.RotatingFileHandler( @@ -181,8 +180,7 @@ class ColoredStreamHandler(logging.StreamHandler): out = super(ColoredStreamHandler, self).format(record) if self.colored and record.levelno >= self.red_level: return ''.join((util.ANSI_SGR_RED, out, util.ANSI_SGR_RESET)) - else: - return out + return out class MemoryHandler(logging.handlers.MemoryHandler): @@ -240,9 +238,10 @@ class TempHandler(logging.StreamHandler): """ def __init__(self): - stream = tempfile.NamedTemporaryFile('w', delete=False) + self._workdir = tempfile.mkdtemp() + self.path = os.path.join(self._workdir, 'log') + stream = util.safe_open(self.path, mode='w', chmod=0o600) super(TempHandler, self).__init__(stream) - self.path = stream.name self._delete = True def emit(self, record): @@ -266,7 +265,7 @@ class TempHandler(logging.StreamHandler): # stream like stderr to be used self.stream.close() if self._delete: - os.remove(self.path) + shutil.rmtree(self._workdir) self._delete = False super(TempHandler, self).close() finally: diff --git a/certbot/main.py b/certbot/certbot/_internal/main.py similarity index 90% rename from certbot/main.py rename to certbot/certbot/_internal/main.py index a0c0ab64d..72fcfca71 100644 --- a/certbot/main.py +++ b/certbot/certbot/_internal/main.py @@ -1,9 +1,9 @@ """Certbot main entry point.""" # pylint: disable=too-many-lines from __future__ import print_function + import functools import logging.handlers -import os import sys import configobj @@ -12,31 +12,31 @@ import zope.component from acme import errors as acme_errors from acme.magic_typing import Union # pylint: disable=unused-import, no-name-in-module - import certbot - -from certbot import account -from certbot import cert_manager -from certbot import cli -from certbot import client -from certbot import compat -from certbot import configuration -from certbot import constants from certbot import crypto_util -from certbot import eff from certbot import errors -from certbot import hooks from certbot import interfaces -from certbot import log -from certbot import renewal -from certbot import reporter -from certbot import storage -from certbot import updater from certbot import util - -from certbot.display import util as display_util, ops as display_ops -from certbot.plugins import disco as plugins_disco -from certbot.plugins import selection as plug_sel +from certbot._internal import account +from certbot._internal import cert_manager +from certbot._internal import cli +from certbot._internal import client +from certbot._internal import configuration +from certbot._internal import constants +from certbot._internal import eff +from certbot._internal import hooks +from certbot._internal import log +from certbot._internal import renewal +from certbot._internal import reporter +from certbot._internal import storage +from certbot._internal import updater +from certbot._internal.plugins import disco as plugins_disco +from certbot._internal.plugins import selection as plug_sel +from certbot.compat import filesystem +from certbot.compat import misc +from certbot.compat import os +from certbot.display import ops as display_ops +from certbot.display import util as display_util from certbot.plugins import enhancements USER_CANCELLED = ("User chose to cancel the operation and may " @@ -121,7 +121,7 @@ def _get_and_save_cert(le_client, config, domains=None, certname=None, lineage=N lineage = le_client.obtain_and_enroll_certificate(domains, certname) if lineage is False: raise errors.Error("Certificate could not be obtained") - elif lineage is not None: + if lineage is not None: hooks.deploy_hook(config, lineage.names(), lineage.live_dir) finally: hooks.post_hook(config) @@ -162,19 +162,18 @@ def _handle_subset_cert_request(config, domains, cert): cli_flag="--expand", force_interactive=True): return "renew", cert - else: - reporter_util = zope.component.getUtility(interfaces.IReporter) - reporter_util.add_message( - "To obtain a new certificate that contains these names without " - "replacing your existing certificate for {0}, you must use the " - "--duplicate option.{br}{br}" - "For example:{br}{br}{1} --duplicate {2}".format( - existing, - sys.argv[0], " ".join(sys.argv[1:]), - br=os.linesep - ), - reporter_util.HIGH_PRIORITY) - raise errors.Error(USER_CANCELLED) + reporter_util = zope.component.getUtility(interfaces.IReporter) + reporter_util.add_message( + "To obtain a new certificate that contains these names without " + "replacing your existing certificate for {0}, you must use the " + "--duplicate option.{br}{br}" + "For example:{br}{br}{1} --duplicate {2}".format( + existing, + sys.argv[0], " ".join(sys.argv[1:]), + br=os.linesep + ), + reporter_util.HIGH_PRIORITY) + raise errors.Error(USER_CANCELLED) def _handle_identical_cert_request(config, lineage): @@ -220,12 +219,12 @@ def _handle_identical_cert_request(config, lineage): # skipping the menu for this case. raise errors.Error( "Operation canceled. You may re-run the client.") - elif response[1] == 0: + if response[1] == 0: return "reinstall", lineage elif response[1] == 1: return "renew", lineage - else: - assert False, "This is impossible" + raise AssertionError('This is impossible') + def _find_lineage_for_domains(config, domains): """Determine whether there are duplicated names and how to handle @@ -264,6 +263,7 @@ def _find_lineage_for_domains(config, domains): return _handle_identical_cert_request(config, ident_names_cert) elif subset_names_cert is not None: return _handle_subset_cert_request(config, domains, subset_names_cert) + return None, None def _find_cert(config, domains, certname): """Finds an existing certificate object given domains and/or a certificate name. @@ -311,23 +311,20 @@ def _find_lineage_for_domains_and_certname(config, domains, certname): """ if not certname: return _find_lineage_for_domains(config, domains) - else: - lineage = cert_manager.lineage_for_certname(config, certname) - if lineage: - if domains: - if set(cert_manager.domains_for_certname(config, certname)) != set(domains): - _ask_user_to_confirm_new_names(config, domains, certname, - lineage.names()) # raises if no - return "renew", lineage - # unnecessarily specified domains or no domains specified - return _handle_identical_cert_request(config, lineage) - else: - if domains: - return "newcert", None - else: - raise errors.ConfigurationError("No certificate with name {0} found. " - "Use -d to specify domains, or run certbot certificates to see " - "possible certificate names.".format(certname)) + lineage = cert_manager.lineage_for_certname(config, certname) + if lineage: + if domains: + if set(cert_manager.domains_for_certname(config, certname)) != set(domains): + _ask_user_to_confirm_new_names(config, domains, certname, + lineage.names()) # raises if no + return "renew", lineage + # unnecessarily specified domains or no domains specified + return _handle_identical_cert_request(config, lineage) + elif domains: + return "newcert", None + raise errors.ConfigurationError("No certificate with name {0} found. " + "Use -d to specify domains, or run certbot certificates to see " + "possible certificate names.".format(certname)) def _get_added_removed(after, before): """Get lists of items removed from `before` @@ -342,7 +339,7 @@ def _get_added_removed(after, before): def _format_list(character, strings): """Format list with given character """ - if len(strings) == 0: + if not strings: formatted = "{br}(None)" else: formatted = "{br}{ch} " + "{br}{ch} ".join(strings) @@ -482,7 +479,7 @@ def _determine_account(config): :returns: Account and optionally ACME client API (biproduct of new registration). - :rtype: tuple of :class:`certbot.account.Account` and :class:`acme.client.Client` + :rtype: tuple of :class:`certbot._internal.account.Account` and :class:`acme.client.Client` :raises errors.Error: If unable to register an account with ACME server @@ -501,6 +498,7 @@ def _determine_account(config): raise errors.Error( "Registration cannot proceed without accepting " "Terms of Service.") + return None account_storage = account.AccountFileStorage(config) acme = None @@ -530,7 +528,7 @@ def _determine_account(config): return acc, acme -def _delete_if_appropriate(config): # pylint: disable=too-many-locals,too-many-branches +def _delete_if_appropriate(config): """Does the user want to delete their now-revoked certs? If run in non-interactive mode, deleting happens automatically. @@ -650,6 +648,7 @@ def unregister(config, unused_plugins): account_files.delete(config.account) reporter_util.add_message("Account deactivated.", reporter_util.MEDIUM_PRIORITY) + return None def register(config, unused_plugins): @@ -665,20 +664,12 @@ def register(config, unused_plugins): :rtype: None or str """ - # TODO: When `certbot register --update-registration` is fully deprecated, - # delete the true case of if block - if config.update_registration: - msg = ("Usage 'certbot register --update-registration' is deprecated.\n" - "Please use 'cerbot update_account [options]' instead.\n") - logger.warning(msg) - return update_account(config, unused_plugins) - # Portion of _determine_account logic to see whether accounts already # exist or not. account_storage = account.AccountFileStorage(config) accounts = account_storage.find_all() - if len(accounts) > 0: + if accounts: # TODO: add a flag to register a duplicate account (this will # also require extending _determine_account's behavior # or else extracting the registration code from there) @@ -687,7 +678,7 @@ def register(config, unused_plugins): "unsupported.") # _determine_account will register an account _determine_account(config) - return + return None def update_account(config, unused_plugins): @@ -710,7 +701,7 @@ def update_account(config, unused_plugins): reporter_util = zope.component.getUtility(interfaces.IReporter) add_msg = lambda m: reporter_util.add_message(m, reporter_util.MEDIUM_PRIORITY) - if len(accounts) == 0: + if not accounts: return "Could not find an existing account to update." if config.email is None: if config.register_unsafely_without_email: @@ -733,6 +724,7 @@ def update_account(config, unused_plugins): account_storage.save_regr(acc, cb_client.acme) eff.handle_subscription(config) add_msg("Your e-mail address was updated to {0}.".format(config.email)) + return None def _install_cert(config, le_client, domains, lineage=None): """Install a cert @@ -760,6 +752,7 @@ def _install_cert(config, le_client, domains, lineage=None): path_provider.cert_path, path_provider.chain_path, path_provider.fullchain_path) le_client.enhance_config(domains, path_provider.chain_path) + def install(config, plugins): """Install a previously obtained cert in a server. @@ -817,6 +810,8 @@ def install(config, plugins): lineage = cert_manager.lineage_for_certname(config, config.certname) enhancements.enable(lineage, domains, installer, config) + return None + def _populate_from_certname(config): """Helper function for install to populate missing config values from lineage defined by --cert-name.""" @@ -835,12 +830,12 @@ def _populate_from_certname(config): return config def _check_certificate_and_key(config): - if not os.path.isfile(os.path.realpath(config.cert_path)): + if not os.path.isfile(filesystem.realpath(config.cert_path)): raise errors.ConfigurationError("Error while reading certificate from path " - "{0}".format(config.cert_path)) - if not os.path.isfile(os.path.realpath(config.key_path)): + "{0}".format(config.cert_path)) + if not os.path.isfile(filesystem.realpath(config.key_path)): raise errors.ConfigurationError("Error while reading private key from path " - "{0}".format(config.key_path)) + "{0}".format(config.key_path)) def plugins_cmd(config, plugins): """List server software plugins. @@ -879,6 +874,7 @@ def plugins_cmd(config, plugins): logger.debug("Prepared plugins: %s", available) notify(str(available)) + def enhance(config, plugins): """Add security enhancements to existing configuration @@ -935,6 +931,8 @@ def enhance(config, plugins): if enhancements.are_requested(config): enhancements.enable(lineage, domains, installer, config) + return None + def rollback(config, plugins): """Rollback server configuration changes made during install. @@ -951,24 +949,6 @@ def rollback(config, plugins): """ client.rollback(config.installer, config.checkpoints, config, plugins) - -def config_changes(config, unused_plugins): - """Show changes made to server config during installation - - View checkpoints and associated configuration changes. - - :param config: Configuration object - :type config: interfaces.IConfig - - :param unused_plugins: List of plugins (deprecated) - :type unused_plugins: `list` of `str` - - :returns: `None` - :rtype: None - - """ - client.view_config_changes(config, num=config.num) - def update_symlinks(config, unused_plugins): """Update the certificate file family symlinks @@ -1038,7 +1018,8 @@ def certificates(config, unused_plugins): """ cert_manager.certificates(config) -def revoke(config, unused_plugins): # TODO: coop with renewal config +# TODO: coop with renewal config +def revoke(config, unused_plugins): """Revoke a previously obtained certificate. :param config: Configuration object @@ -1080,9 +1061,10 @@ def revoke(config, unused_plugins): # TODO: coop with renewal config return str(e) display_ops.success_revocation(config.cert_path[0]) + return None -def run(config, plugins): # pylint: disable=too-many-branches,too-many-locals +def run(config, plugins): """Obtain a certificate and install. :param config: Configuration object @@ -1134,6 +1116,7 @@ def run(config, plugins): # pylint: disable=too-many-branches,too-many-locals display_ops.success_renewal(domains) _suggest_donation_if_appropriate(config) + return None def _csr_get_and_save_cert(config, le_client): @@ -1284,18 +1267,14 @@ def make_or_verify_needed_dirs(config): :rtype: None """ - util.set_up_core_dir(config.config_dir, constants.CONFIG_DIRS_MODE, - compat.os_geteuid(), config.strict_permissions) - util.set_up_core_dir(config.work_dir, constants.CONFIG_DIRS_MODE, - compat.os_geteuid(), config.strict_permissions) + util.set_up_core_dir(config.config_dir, constants.CONFIG_DIRS_MODE, config.strict_permissions) + util.set_up_core_dir(config.work_dir, constants.CONFIG_DIRS_MODE, config.strict_permissions) hook_dirs = (config.renewal_pre_hooks_dir, config.renewal_deploy_hooks_dir, config.renewal_post_hooks_dir,) for hook_dir in hook_dirs: - util.make_or_verify_dir(hook_dir, - uid=compat.os_geteuid(), - strict=config.strict_permissions) + util.make_or_verify_dir(hook_dir, strict=config.strict_permissions) def set_displayer(config): @@ -1320,15 +1299,18 @@ def set_displayer(config): zope.component.provideUtility(displayer) -def main(cli_args=sys.argv[1:]): - """Command line argument parsing and main script execution. +def main(cli_args=None): + """Run Certbot. - :returns: result of requested command + :param cli_args: command line to Certbot, defaults to ``sys.argv[1:]`` + :type cli_args: `list` of `str` - :raises errors.Error: OS errors triggered by wrong permissions - :raises errors.Error: error if plugin command is not supported + :returns: value for `sys.exit` about the exit status of Certbot + :rtype: `str` or `int` or `None` """ + if not cli_args: + cli_args = sys.argv[1:] log.pre_arg_parse_setup() @@ -1345,14 +1327,14 @@ def main(cli_args=sys.argv[1:]): # On windows, shell without administrative right cannot create symlinks required by certbot. # So we check the rights before continuing. - compat.raise_for_non_administrative_windows_rights(config.verb) + misc.raise_for_non_administrative_windows_rights() try: log.post_arg_parse_setup(config) make_or_verify_needed_dirs(config) except errors.Error: # Let plugins_cmd be run as un-privileged user. - if config.func != plugins_cmd: + if config.func != plugins_cmd: # pylint: disable=comparison-with-callable raise set_displayer(config) @@ -1363,10 +1345,3 @@ def main(cli_args=sys.argv[1:]): util.atexit_register(report.print_messages) return config.func(config, plugins) - - -if __name__ == "__main__": - err_string = main() - if err_string: - logger.warning("Exiting with message %s", err_string) - sys.exit(err_string) # pragma: no cover diff --git a/certbot/notify.py b/certbot/certbot/_internal/notify.py similarity index 100% rename from certbot/notify.py rename to certbot/certbot/_internal/notify.py diff --git a/certbot/certbot/_internal/ocsp.py b/certbot/certbot/_internal/ocsp.py new file mode 100644 index 000000000..2f6543e5d --- /dev/null +++ b/certbot/certbot/_internal/ocsp.py @@ -0,0 +1,300 @@ +"""Tools for checking certificate revocation.""" +from datetime import datetime +from datetime import timedelta +import logging +import re +from subprocess import PIPE +from subprocess import Popen + +from cryptography import x509 +from cryptography.exceptions import InvalidSignature +from cryptography.exceptions import UnsupportedAlgorithm +from cryptography.hazmat.backends import default_backend +# See https://github.com/pyca/cryptography/issues/4275 +from cryptography.hazmat.primitives import hashes # type: ignore +from cryptography.hazmat.primitives import serialization +import pytz +import requests + +from acme.magic_typing import Optional # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Tuple # pylint: disable=unused-import, no-name-in-module +from certbot import crypto_util +from certbot import errors +from certbot import util +from certbot._internal.storage import RenewableCert # pylint: disable=unused-import + +try: + # Only cryptography>=2.5 has ocsp module + # and signature_hash_algorithm attribute in OCSPResponse class + from cryptography.x509 import ocsp # pylint: disable=import-error, ungrouped-imports + getattr(ocsp.OCSPResponse, 'signature_hash_algorithm') +except (ImportError, AttributeError): # pragma: no cover + ocsp = None # type: ignore + + + +logger = logging.getLogger(__name__) + + +class RevocationChecker(object): + """This class figures out OCSP checking on this system, and performs it.""" + + def __init__(self, enforce_openssl_binary_usage=False): + self.broken = False + self.use_openssl_binary = enforce_openssl_binary_usage or not ocsp + + if self.use_openssl_binary: + if not util.exe_exists("openssl"): + logger.info("openssl not installed, can't check revocation") + self.broken = True + return + + # New versions of openssl want -header var=val, old ones want -header var val + test_host_format = Popen(["openssl", "ocsp", "-header", "var", "val"], + stdout=PIPE, stderr=PIPE, universal_newlines=True) + _out, err = test_host_format.communicate() + if "Missing =" in err: + self.host_args = lambda host: ["Host=" + host] + else: + self.host_args = lambda host: ["Host", host] + + def ocsp_revoked(self, cert): + # type: (RenewableCert) -> bool + """Get revoked status for a particular cert version. + + .. todo:: Make this a non-blocking call + + :param `.storage.RenewableCert` cert: Certificate object + :returns: True if revoked; False if valid or the check failed or cert is expired. + :rtype: bool + + """ + cert_path, chain_path = cert.cert, cert.chain + + if self.broken: + return False + + # Let's Encrypt doesn't update OCSP for expired certificates, + # so don't check OCSP if the cert is expired. + # https://github.com/certbot/certbot/issues/7152 + now = pytz.UTC.fromutc(datetime.utcnow()) + if cert.target_expiry <= now: + return False + + url, host = _determine_ocsp_server(cert_path) + if not host or not url: + return False + + if self.use_openssl_binary: + return self._check_ocsp_openssl_bin(cert_path, chain_path, host, url) + return _check_ocsp_cryptography(cert_path, chain_path, url) + + def _check_ocsp_openssl_bin(self, cert_path, chain_path, host, url): + # type: (str, str, str, str) -> bool + # jdkasten thanks "Bulletproof SSL and TLS - Ivan Ristic" for documenting this! + cmd = ["openssl", "ocsp", + "-no_nonce", + "-issuer", chain_path, + "-cert", cert_path, + "-url", url, + "-CAfile", chain_path, + "-verify_other", chain_path, + "-trust_other", + "-header"] + self.host_args(host) + logger.debug("Querying OCSP for %s", cert_path) + logger.debug(" ".join(cmd)) + try: + output, err = util.run_script(cmd, log=logger.debug) + except errors.SubprocessError: + logger.info("OCSP check failed for %s (are we offline?)", cert_path) + return False + return _translate_ocsp_query(cert_path, output, err) + + +def _determine_ocsp_server(cert_path): + # type: (str) -> Tuple[Optional[str], Optional[str]] + """Extract the OCSP server host from a certificate. + + :param str cert_path: Path to the cert we're checking OCSP for + :rtype tuple: + :returns: (OCSP server URL or None, OCSP server host or None) + + """ + with open(cert_path, 'rb') as file_handler: + cert = x509.load_pem_x509_certificate(file_handler.read(), default_backend()) + try: + extension = cert.extensions.get_extension_for_class(x509.AuthorityInformationAccess) + ocsp_oid = x509.AuthorityInformationAccessOID.OCSP + descriptions = [description for description in extension.value + if description.access_method == ocsp_oid] + + url = descriptions[0].access_location.value + except (x509.ExtensionNotFound, IndexError): + logger.info("Cannot extract OCSP URI from %s", cert_path) + return None, None + + url = url.rstrip() + host = url.partition("://")[2].rstrip("/") + + if host: + return url, host + logger.info("Cannot process OCSP host from URL (%s) in cert at %s", url, cert_path) + return None, None + + +def _check_ocsp_cryptography(cert_path, chain_path, url): + # type: (str, str, str) -> bool + # Retrieve OCSP response + with open(chain_path, 'rb') as file_handler: + issuer = x509.load_pem_x509_certificate(file_handler.read(), default_backend()) + with open(cert_path, 'rb') as file_handler: + cert = x509.load_pem_x509_certificate(file_handler.read(), default_backend()) + builder = ocsp.OCSPRequestBuilder() + builder = builder.add_certificate(cert, issuer, hashes.SHA1()) + request = builder.build() + request_binary = request.public_bytes(serialization.Encoding.DER) + try: + response = requests.post(url, data=request_binary, + headers={'Content-Type': 'application/ocsp-request'}) + except requests.exceptions.RequestException: + logger.info("OCSP check failed for %s (are we offline?)", cert_path, exc_info=True) + return False + if response.status_code != 200: + logger.info("OCSP check failed for %s (HTTP status: %d)", cert_path, response.status_code) + return False + + response_ocsp = ocsp.load_der_ocsp_response(response.content) + + # Check OCSP response validity + if response_ocsp.response_status != ocsp.OCSPResponseStatus.SUCCESSFUL: + logger.error("Invalid OCSP response status for %s: %s", + cert_path, response_ocsp.response_status) + return False + + # Check OCSP signature + try: + _check_ocsp_response(response_ocsp, request, issuer, cert_path) + except UnsupportedAlgorithm as e: + logger.error(str(e)) + except errors.Error as e: + logger.error(str(e)) + except InvalidSignature: + logger.error('Invalid signature on OCSP response for %s', cert_path) + except AssertionError as error: + logger.error('Invalid OCSP response for %s: %s.', cert_path, str(error)) + else: + # Check OCSP certificate status + logger.debug("OCSP certificate status for %s is: %s", + cert_path, response_ocsp.certificate_status) + return response_ocsp.certificate_status == ocsp.OCSPCertStatus.REVOKED + + return False + + +def _check_ocsp_response(response_ocsp, request_ocsp, issuer_cert, cert_path): + """Verify that the OCSP is valid for serveral criteria""" + # Assert OCSP response corresponds to the certificate we are talking about + if response_ocsp.serial_number != request_ocsp.serial_number: + raise AssertionError('the certificate in response does not correspond ' + 'to the certificate in request') + + # Assert signature is valid + _check_ocsp_response_signature(response_ocsp, issuer_cert, cert_path) + + # Assert issuer in response is the expected one + if (not isinstance(response_ocsp.hash_algorithm, type(request_ocsp.hash_algorithm)) + or response_ocsp.issuer_key_hash != request_ocsp.issuer_key_hash + or response_ocsp.issuer_name_hash != request_ocsp.issuer_name_hash): + raise AssertionError('the issuer does not correspond to issuer of the certificate.') + + # In following checks, two situations can occur: + # * nextUpdate is set, and requirement is thisUpdate < now < nextUpdate + # * nextUpdate is not set, and requirement is thisUpdate < now + # NB1: We add a validity period tolerance to handle clock time inconsistencies, + # value is 5 min like for OpenSSL. + # NB2: Another check is to verify that thisUpdate is not too old, it is optional + # for OpenSSL, so we do not do it here. + # See OpenSSL implementation as a reference: + # https://github.com/openssl/openssl/blob/ef45aa14c5af024fcb8bef1c9007f3d1c115bd85/crypto/ocsp/ocsp_cl.c#L338-L391 + now = datetime.utcnow() # thisUpdate/nextUpdate are expressed in UTC/GMT time zone + if not response_ocsp.this_update: + raise AssertionError('param thisUpdate is not set.') + if response_ocsp.this_update > now + timedelta(minutes=5): + raise AssertionError('param thisUpdate is in the future.') + if response_ocsp.next_update and response_ocsp.next_update < now - timedelta(minutes=5): + raise AssertionError('param nextUpdate is in the past.') + + +def _check_ocsp_response_signature(response_ocsp, issuer_cert, cert_path): + """Verify an OCSP response signature against certificate issuer or responder""" + if response_ocsp.responder_name == issuer_cert.subject: + # Case where the OCSP responder is also the certificate issuer + logger.debug('OCSP response for certificate %s is signed by the certificate\'s issuer.', + cert_path) + responder_cert = issuer_cert + else: + # Case where the OCSP responder is not the certificate issuer + logger.debug('OCSP response for certificate %s is delegated to an external responder.', + cert_path) + + responder_certs = [cert for cert in response_ocsp.certificates + if cert.subject == response_ocsp.responder_name] + if not responder_certs: + raise AssertionError('no matching responder certificate could be found') + + # We suppose here that the ACME server support only one certificate in the OCSP status + # request. This is currently the case for LetsEncrypt servers. + # See https://github.com/letsencrypt/boulder/issues/2331 + responder_cert = responder_certs[0] + + if responder_cert.issuer != issuer_cert.subject: + raise AssertionError('responder certificate is not signed ' + 'by the certificate\'s issuer') + + try: + extension = responder_cert.extensions.get_extension_for_class(x509.ExtendedKeyUsage) + delegate_authorized = x509.oid.ExtendedKeyUsageOID.OCSP_SIGNING in extension.value + except (x509.ExtensionNotFound, IndexError): + delegate_authorized = False + if not delegate_authorized: + raise AssertionError('responder is not authorized by issuer to sign OCSP responses') + + # Following line may raise UnsupportedAlgorithm + chosen_hash = responder_cert.signature_hash_algorithm + # For a delegate OCSP responder, we need first check that its certificate is effectively + # signed by the certificate issuer. + crypto_util.verify_signed_payload(issuer_cert.public_key(), responder_cert.signature, + responder_cert.tbs_certificate_bytes, chosen_hash) + + # Following line may raise UnsupportedAlgorithm + chosen_hash = response_ocsp.signature_hash_algorithm + # We check that the OSCP response is effectively signed by the responder + # (an authorized delegate one or the certificate issuer itself). + crypto_util.verify_signed_payload(responder_cert.public_key(), response_ocsp.signature, + response_ocsp.tbs_response_bytes, chosen_hash) + + +def _translate_ocsp_query(cert_path, ocsp_output, ocsp_errors): + """Parse openssl's weird output to work out what it means.""" + + states = ("good", "revoked", "unknown") + patterns = [r"{0}: (WARNING.*)?{1}".format(cert_path, s) for s in states] + good, revoked, unknown = (re.search(p, ocsp_output, flags=re.DOTALL) for p in patterns) + + warning = good.group(1) if good else None + + if ("Response verify OK" not in ocsp_errors) or (good and warning) or unknown: + logger.info("Revocation status for %s is unknown", cert_path) + logger.debug("Uncertain output:\n%s\nstderr:\n%s", ocsp_output, ocsp_errors) + return False + elif good and not warning: + return False + elif revoked: + warning = revoked.group(1) + if warning: + logger.info("OCSP revocation warning: %s", warning) + return True + else: + logger.warning("Unable to properly parse OCSP output: %s\nstderr:%s", + ocsp_output, ocsp_errors) + return False diff --git a/certbot/certbot/_internal/plugins/__init__.py b/certbot/certbot/_internal/plugins/__init__.py new file mode 100644 index 000000000..7831eab61 --- /dev/null +++ b/certbot/certbot/_internal/plugins/__init__.py @@ -0,0 +1 @@ +"""Certbot plugins.""" diff --git a/certbot/plugins/disco.py b/certbot/certbot/_internal/plugins/disco.py similarity index 95% rename from certbot/plugins/disco.py rename to certbot/certbot/_internal/plugins/disco.py index 7be320efc..d7d6390f7 100644 --- a/certbot/plugins/disco.py +++ b/certbot/certbot/_internal/plugins/disco.py @@ -2,19 +2,22 @@ import collections import itertools import logging + import pkg_resources import six - -from collections import OrderedDict - import zope.interface import zope.interface.verify from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module -from certbot import constants from certbot import errors from certbot import interfaces +from certbot._internal import constants +try: + # Python 3.3+ + from collections.abc import Mapping +except ImportError: # pragma: no cover + from collections import Mapping logger = logging.getLogger(__name__) @@ -40,7 +43,6 @@ class PluginEntryPoint(object): "certbot-dns-route53", "certbot-dns-sakuracloud", "certbot-nginx", - "certbot-postfix", ] """Distributions for which prefix will be omitted.""" @@ -182,7 +184,7 @@ class PluginEntryPoint(object): return "\n".join(lines) -class PluginsRegistry(collections.Mapping): +class PluginsRegistry(Mapping): """Plugins registry.""" def __init__(self, plugins): @@ -190,7 +192,11 @@ class PluginsRegistry(collections.Mapping): # This prevents deadlock caused by plugins acquiring a lock # and ensures at least one concurrent Certbot instance will run # successfully. - self._plugins = OrderedDict(sorted(six.iteritems(plugins))) + + # Pylint checks for super init, but also claims the super + # has no __init__member + # pylint: disable=super-init-not-called + self._plugins = collections.OrderedDict(sorted(six.iteritems(plugins))) @classmethod def find_all(cls): @@ -239,7 +245,6 @@ class PluginsRegistry(collections.Mapping): def ifaces(self, *ifaces_groups): """Filter plugins based on interfaces.""" - # pylint: disable=star-args return self.filter(lambda p_ep: p_ep.ifaces(*ifaces_groups)) def verify(self, ifaces): @@ -275,8 +280,7 @@ class PluginsRegistry(collections.Mapping): assert len(candidates) <= 1 if candidates: return candidates[0] - else: - return None + return None def __repr__(self): return "{0}({1})".format( diff --git a/certbot/plugins/manual.py b/certbot/certbot/_internal/plugins/manual.py similarity index 68% rename from certbot/plugins/manual.py rename to certbot/certbot/_internal/plugins/manual.py index 8723a1c62..be6abaad4 100644 --- a/certbot/plugins/manual.py +++ b/certbot/certbot/_internal/plugins/manual.py @@ -1,48 +1,18 @@ """Manual authenticator plugin""" -import os - import zope.component import zope.interface from acme import challenges from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module - from certbot import achallenges # pylint: disable=unused-import -from certbot import interfaces from certbot import errors -from certbot import hooks +from certbot import interfaces from certbot import reverter +from certbot._internal import hooks +from certbot.compat import os from certbot.plugins import common -class ManualTlsSni01(common.TLSSNI01): - """TLS-SNI-01 authenticator for the Manual plugin - - :ivar configurator: Authenticator object - :type configurator: :class:`~certbot.plugins.manual.Authenticator` - - :ivar list achalls: Annotated - class:`~certbot.achallenges.KeyAuthorizationAnnotatedChallenge` - challenges - - :param list indices: Meant to hold indices of challenges in a - larger array. NginxTlsSni01 is capable of solving many challenges - at once which causes an indexing issue within NginxConfigurator - who must return all responses in order. Imagine NginxConfigurator - maintaining state about where all of the http-01 Challenges, - TLS-SNI-01 Challenges belong in the response array. This is an - optional utility. - - :param str challenge_conf: location of the challenge config file - """ - - def perform(self): - """Create the SSL certificates and private keys""" - - for achall in self.achalls: - self._setup_challenge_cert(achall) - - @zope.interface.implementer(interfaces.IAuthenticator) @zope.interface.provider(interfaces.IPluginFactory) class Authenticator(common.Plugin): @@ -63,14 +33,9 @@ class Authenticator(common.Plugin): 'type of challenge. $CERTBOT_DOMAIN will always contain the domain ' 'being authenticated. For HTTP-01 and DNS-01, $CERTBOT_VALIDATION ' 'is the validation string, and $CERTBOT_TOKEN is the filename of the ' - 'resource requested when performing an HTTP-01 challenge. When ' - 'performing a TLS-SNI-01 challenge, $CERTBOT_SNI_DOMAIN will contain ' - 'the SNI name for which the ACME server expects to be presented with ' - 'the self-signed certificate located at $CERTBOT_CERT_PATH. The ' - 'secret key needed to complete the TLS handshake is located at ' - '$CERTBOT_KEY_PATH. An additional cleanup script can also be ' - 'provided and can use the additional variable $CERTBOT_AUTH_OUTPUT ' - 'which contains the stdout output from the auth script.') + 'resource requested when performing an HTTP-01 challenge. An additional ' + 'cleanup script can also be provided and can use the additional variable ' + '$CERTBOT_AUTH_OUTPUT which contains the stdout output from the auth script.') _DNS_INSTRUCTIONS = """\ Please deploy a DNS TXT record under the name {domain} with the following value: @@ -86,14 +51,6 @@ Create a file containing just this data: And make it available on your web server at this URL: {uri} -""" - _TLSSNI_INSTRUCTIONS = """\ -Configure the service listening on port {port} to present the certificate -{cert} -using the secret key -{key} -when it receives a TLS ClientHello with the SNI extension set to -{sni_domain} """ _SUBSEQUENT_CHALLENGE_INSTRUCTIONS = """ (This must be set up in addition to the previous challenges; do not remove, @@ -112,7 +69,6 @@ permitted by DNS standards.) self.reverter.recovery_routine() self.env = dict() \ # type: Dict[achallenges.KeyAuthorizationAnnotatedChallenge, Dict[str, str]] - self.tls_sni_01 = None self.subsequent_dns_challenge = False self.subsequent_any_challenge = False @@ -149,7 +105,7 @@ permitted by DNS standards.) def get_chall_pref(self, domain): # pylint: disable=missing-docstring,no-self-use,unused-argument - return [challenges.HTTP01, challenges.DNS01, challenges.TLSSNI01] + return [challenges.HTTP01, challenges.DNS01] def perform(self, achalls): # pylint: disable=missing-docstring self._verify_ip_logging_ok() @@ -160,12 +116,6 @@ permitted by DNS standards.) responses = [] for achall in achalls: - if isinstance(achall.chall, challenges.TLSSNI01): - # Make a new ManualTlsSni01 instance for each challenge - # because the manual plugin deals with one challenge at a time. - self.tls_sni_01 = ManualTlsSni01(self) - self.tls_sni_01.add_chall(achall) - self.tls_sni_01.perform() perform_achall(achall) responses.append(achall.response(achall.account_key)) return responses @@ -191,18 +141,8 @@ permitted by DNS standards.) env['CERTBOT_TOKEN'] = achall.chall.encode('token') else: os.environ.pop('CERTBOT_TOKEN', None) - if isinstance(achall.chall, challenges.TLSSNI01): - env['CERTBOT_CERT_PATH'] = self.tls_sni_01.get_cert_path(achall) - env['CERTBOT_KEY_PATH'] = self.tls_sni_01.get_key_path(achall) - env['CERTBOT_SNI_DOMAIN'] = self.tls_sni_01.get_z_domain(achall) - os.environ.pop('CERTBOT_VALIDATION', None) - env.pop('CERTBOT_VALIDATION') - else: - os.environ.pop('CERTBOT_CERT_PATH', None) - os.environ.pop('CERTBOT_KEY_PATH', None) - os.environ.pop('CERTBOT_SNI_DOMAIN', None) os.environ.update(env) - _, out = hooks.execute(self.conf('auth-hook')) + _, out = self._execute_hook('auth-hook') env['CERTBOT_AUTH_OUTPUT'] = out.strip() self.env[achall] = env @@ -213,17 +153,11 @@ permitted by DNS standards.) achall=achall, encoded_token=achall.chall.encode('token'), port=self.config.http01_port, uri=achall.chall.uri(achall.domain), validation=validation) - elif isinstance(achall.chall, challenges.DNS01): + else: + assert isinstance(achall.chall, challenges.DNS01) msg = self._DNS_INSTRUCTIONS.format( domain=achall.validation_domain_name(achall.domain), validation=validation) - else: - assert isinstance(achall.chall, challenges.TLSSNI01) - msg = self._TLSSNI_INSTRUCTIONS.format( - cert=self.tls_sni_01.get_cert_path(achall), - key=self.tls_sni_01.get_key_path(achall), - port=self.config.tls_sni_01_port, - sni_domain=self.tls_sni_01.get_z_domain(achall)) if isinstance(achall.chall, challenges.DNS01): if self.subsequent_dns_challenge: # 2nd or later dns-01 challenge @@ -243,5 +177,8 @@ permitted by DNS standards.) if 'CERTBOT_TOKEN' not in env: os.environ.pop('CERTBOT_TOKEN', None) os.environ.update(env) - hooks.execute(self.conf('cleanup-hook')) + self._execute_hook('cleanup-hook') self.reverter.recovery_routine() + + def _execute_hook(self, hook_name): + return hooks.execute(self.option_name(hook_name), self.conf(hook_name)) diff --git a/certbot/plugins/null.py b/certbot/certbot/_internal/plugins/null.py similarity index 94% rename from certbot/plugins/null.py rename to certbot/certbot/_internal/plugins/null.py index 87c0737a5..bf4615497 100644 --- a/certbot/plugins/null.py +++ b/certbot/certbot/_internal/plugins/null.py @@ -7,7 +7,6 @@ import zope.interface from certbot import interfaces from certbot.plugins import common - logger = logging.getLogger(__name__) @@ -49,9 +48,6 @@ class Installer(common.Plugin): def recovery_routine(self): pass # pragma: no cover - def view_config_changes(self): - pass # pragma: no cover - def config_test(self): pass # pragma: no cover diff --git a/certbot/plugins/selection.py b/certbot/certbot/_internal/plugins/selection.py similarity index 95% rename from certbot/plugins/selection.py rename to certbot/certbot/_internal/plugins/selection.py index 9c2138247..6d87e4b07 100644 --- a/certbot/plugins/selection.py +++ b/certbot/certbot/_internal/plugins/selection.py @@ -1,7 +1,6 @@ """Decide which plugins to use for authentication & installation""" from __future__ import print_function -import os import logging import six @@ -9,7 +8,7 @@ import zope.component from certbot import errors from certbot import interfaces - +from certbot.compat import os from certbot.display import util as display_util logger = logging.getLogger(__name__) @@ -44,7 +43,7 @@ def get_unprepared_installer(config, plugins): Get an unprepared interfaces.IInstaller object. :param certbot.interfaces.IConfig config: Configuration - :param certbot.plugins.disco.PluginsRegistry plugins: + :param certbot._internal.plugins.disco.PluginsRegistry plugins: All plugins registered as entry points. :returns: Unprepared installer plugin or None @@ -65,16 +64,15 @@ def get_unprepared_installer(config, plugins): inst = list(installers.values())[0] logger.debug("Selecting plugin: %s", inst) return inst.init(config) - else: - raise errors.PluginSelectionError( - "Could not select or initialize the requested installer %s." % req_inst) + raise errors.PluginSelectionError( + "Could not select or initialize the requested installer %s." % req_inst) def pick_plugin(config, default, plugins, question, ifaces): """Pick plugin. :param certbot.interfaces.IConfig: Configuration :param str default: Plugin name supplied by user or ``None``. - :param certbot.plugins.disco.PluginsRegistry plugins: + :param certbot._internal.plugins.disco.PluginsRegistry plugins: All plugins registered as entry points. :param str question: Question to be presented to the user in case multiple candidates are found. @@ -111,8 +109,7 @@ def pick_plugin(config, default, plugins, question, ifaces): plugin_ep = choose_plugin(list(six.itervalues(prepared)), question) if plugin_ep is None: return None - else: - return plugin_ep.init() + return plugin_ep.init() elif len(prepared) == 1: plugin_ep = list(prepared.values())[0] logger.debug("Single candidate plugin: %s", plugin_ep) @@ -177,7 +174,6 @@ def record_chosen_plugins(config, plugins, auth, inst): def choose_configurator_plugins(config, plugins, verb): - # pylint: disable=too-many-branches """ Figure out which configurator we're going to use, modifies config.authenticator and config.installer strings to reflect that choice if @@ -199,7 +195,7 @@ def choose_configurator_plugins(config, plugins, verb): # Which plugins do we need? if verb == "run": need_inst = need_auth = True - from certbot.cli import cli_command + from certbot._internal.cli import cli_command if req_auth in noninstaller_plugins and not req_inst: msg = ('With the {0} plugin, you probably want to use the "certonly" command, eg:{1}' '{1} {2} certonly --{0}{1}{1}' @@ -212,7 +208,7 @@ def choose_configurator_plugins(config, plugins, verb): need_inst = need_auth = False if verb == "certonly": need_auth = True - if verb == "install" or verb == "enhance": + elif verb in ("install", "enhance"): need_inst = True if config.authenticator: logger.warning("Specifying an authenticator doesn't make sense when " @@ -256,7 +252,7 @@ def set_configurator(previously, now): return now -def cli_plugin_requests(config): # pylint: disable=too-many-branches +def cli_plugin_requests(config): """ Figure out which plugins the user requested with CLI and config options @@ -330,7 +326,7 @@ def diagnose_configurator_problem(cfg_type, requested, plugins): "your existing configuration.\nThe error was: {1!r}" .format(requested, plugins[requested].problem)) elif cfg_type == "installer": - from certbot.cli import cli_command + from certbot._internal.cli import cli_command msg = ('Certbot doesn\'t know how to automatically configure the web ' 'server on this system. However, it can still get a certificate for ' 'you. Please run "{0} certonly" to do so. You\'ll need to ' diff --git a/certbot/plugins/standalone.py b/certbot/certbot/_internal/plugins/standalone.py similarity index 60% rename from certbot/plugins/standalone.py rename to certbot/certbot/_internal/plugins/standalone.py index 16f872a3f..80421299e 100644 --- a/certbot/plugins/standalone.py +++ b/certbot/certbot/_internal/plugins/standalone.py @@ -1,24 +1,24 @@ """Standalone Authenticator.""" -import argparse import collections import logging import socket # https://github.com/python/typeshed/blob/master/stdlib/2and3/socket.pyi from socket import errno as socket_errors # type: ignore -import OpenSSL +import OpenSSL # pylint: disable=unused-import import six import zope.interface from acme import challenges from acme import standalone as acme_standalone -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import DefaultDict, Dict, Set, Tuple, List, Type, TYPE_CHECKING - +from acme.magic_typing import DefaultDict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Tuple # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import TYPE_CHECKING # pylint: disable=unused-import, no-name-in-module from certbot import achallenges # pylint: disable=unused-import from certbot import errors from certbot import interfaces - from certbot.plugins import common logger = logging.getLogger(__name__) @@ -55,32 +55,27 @@ class ServerManager(object): :param int port: Port to run the server on. :param challenge_type: Subclass of `acme.challenges.Challenge`, - either `acme.challenge.HTTP01` or `acme.challenges.TLSSNI01`. + currently only `acme.challenge.HTTP01`. :param str listenaddr: (optional) The address to listen on. Defaults to all addrs. :returns: DualNetworkedServers instance. :rtype: ACMEServerMixin """ - assert challenge_type in (challenges.TLSSNI01, challenges.HTTP01) + assert challenge_type == challenges.HTTP01 if port in self._instances: return self._instances[port] address = (listenaddr, port) try: - if challenge_type is challenges.TLSSNI01: - servers = acme_standalone.TLSSNI01DualNetworkedServers( - address, self.certs) # type: acme_standalone.BaseDualNetworkedServers - else: # challenges.HTTP01 - servers = acme_standalone.HTTP01DualNetworkedServers( - address, self.http_01_resources) + servers = acme_standalone.HTTP01DualNetworkedServers( + address, self.http_01_resources) except socket.error as error: raise errors.StandaloneBindError(error, port) servers.serve_forever() # if port == 0, then random free port on OS is taken - # pylint: disable=no-member # both servers, if they exist, have the same port real_port = servers.getsocknames()[0][1] self._instances[real_port] = servers @@ -96,8 +91,6 @@ class ServerManager(object): for sockname in instance.getsocknames(): logger.debug("Stopping server at %s:%d...", *sockname[:2]) - # Not calling server_close causes problems when renewing multiple - # certs with `certbot renew` using TLSSNI01 and PyOpenSSL 0.13 instance.shutdown_and_server_close() del self._instances[port] @@ -114,70 +107,13 @@ class ServerManager(object): return self._instances.copy() -SUPPORTED_CHALLENGES = [challenges.HTTP01, challenges.TLSSNI01] \ -# type: List[Type[challenges.KeyAuthorizationChallenge]] - - -class SupportedChallengesAction(argparse.Action): - """Action class for parsing standalone_supported_challenges.""" - - def __call__(self, parser, namespace, values, option_string=None): - logger.warning( - "The standalone specific supported challenges flag is " - "deprecated. Please use the --preferred-challenges flag " - "instead.") - converted_values = self._convert_and_validate(values) - namespace.standalone_supported_challenges = converted_values - - def _convert_and_validate(self, data): - """Validate the value of supported challenges provided by the user. - - References to "dvsni" are automatically converted to "tls-sni-01". - - :param str data: comma delimited list of challenge types - - :returns: validated and converted list of challenge types - :rtype: str - - """ - challs = data.split(",") - - # tls-sni-01 was dvsni during private beta - if "dvsni" in challs: - logger.info( - "Updating legacy standalone_supported_challenges value") - challs = [challenges.TLSSNI01.typ if chall == "dvsni" else chall - for chall in challs] - data = ",".join(challs) - - unrecognized = [name for name in challs - if name not in challenges.Challenge.TYPES] - - # argparse.ArgumentErrors raised out of argparse.Action objects - # are caught by argparse which prints usage information and the - # error that occurred before calling sys.exit. - if unrecognized: - raise argparse.ArgumentError( - self, - "Unrecognized challenges: {0}".format(", ".join(unrecognized))) - - choices = set(chall.typ for chall in SUPPORTED_CHALLENGES) - if not set(challs).issubset(choices): - raise argparse.ArgumentError( - self, - "Plugin does not support the following (valid) " - "challenges: {0}".format(", ".join(set(challs) - choices))) - - return data - - @zope.interface.implementer(interfaces.IAuthenticator) @zope.interface.provider(interfaces.IPluginFactory) class Authenticator(common.Plugin): """Standalone Authenticator. This authenticator creates its own ephemeral TCP listener on the - necessary port in order to respond to incoming tls-sni-01 and http-01 + necessary port in order to respond to incoming http-01 challenges from the certificate authority. Therefore, it does not rely on any existing server program. """ @@ -187,10 +123,6 @@ class Authenticator(common.Plugin): def __init__(self, *args, **kwargs): super(Authenticator, self).__init__(*args, **kwargs) - # one self-signed key for all tls-sni-01 certificates - self.key = OpenSSL.crypto.PKey() - self.key.generate_key(OpenSSL.crypto.TYPE_RSA, 2048) - self.served = collections.defaultdict(set) # type: ServedType # Stuff below is shared across threads (i.e. servers read @@ -205,30 +137,20 @@ class Authenticator(common.Plugin): @classmethod def add_parser_arguments(cls, add): - add("supported-challenges", - help=argparse.SUPPRESS, - action=SupportedChallengesAction, - default=",".join(chall.typ for chall in SUPPORTED_CHALLENGES)) - - @property - def supported_challenges(self): - """Challenges supported by this plugin.""" - return [challenges.Challenge.TYPES[name] for name in - self.conf("supported-challenges").split(",")] + pass # No additional argument for the standalone plugin parser def more_info(self): # pylint: disable=missing-docstring return("This authenticator creates its own ephemeral TCP listener " "on the necessary port in order to respond to incoming " - "tls-sni-01 and http-01 challenges from the certificate " - "authority. Therefore, it does not rely on any existing " - "server program.") + "http-01 challenges from the certificate authority. Therefore, " + "it does not rely on any existing server program.") def prepare(self): # pylint: disable=missing-docstring pass def get_chall_pref(self, domain): # pylint: disable=unused-argument,missing-docstring - return self.supported_challenges + return [challenges.HTTP01] def perform(self, achalls): # pylint: disable=missing-docstring return [self._try_perform_single(achall) for achall in achalls] @@ -241,10 +163,7 @@ class Authenticator(common.Plugin): _handle_perform_error(error) def _perform_single(self, achall): - if isinstance(achall.chall, challenges.HTTP01): - servers, response = self._perform_http_01(achall) - else: # tls-sni-01 - servers, response = self._perform_tls_sni_01(achall) + servers, response = self._perform_http_01(achall) self.served[servers].add(achall) return response @@ -258,14 +177,6 @@ class Authenticator(common.Plugin): self.http_01_resources.add(resource) return servers, response - def _perform_tls_sni_01(self, achall): - port = self.config.tls_sni_01_port - addr = self.config.tls_sni_01_address - servers = self.servers.run(port, challenges.TLSSNI01, listenaddr=addr) - response, (cert, _) = achall.response_and_validation(cert_key=self.key) - self.certs[response.z_domain] = (self.key, cert) - return servers, response - def cleanup(self, achalls): # pylint: disable=missing-docstring # reduce self.served and close servers if no challenges are served for unused_servers, server_achalls in self.served.items(): @@ -284,7 +195,7 @@ def _handle_perform_error(error): "the appropriate permissions (for example, you " "aren't running this program as " "root).".format(error.port)) - elif error.socket_error.errno == socket_errors.EADDRINUSE: + if error.socket_error.errno == socket_errors.EADDRINUSE: display = zope.component.getUtility(interfaces.IDisplay) msg = ( "Could not bind TCP port {0} because it is already in " @@ -296,4 +207,4 @@ def _handle_perform_error(error): if not should_retry: raise errors.PluginError(msg) else: - raise + raise error diff --git a/certbot/plugins/webroot.py b/certbot/certbot/_internal/plugins/webroot.py similarity index 89% rename from certbot/plugins/webroot.py rename to certbot/certbot/_internal/plugins/webroot.py index 529094705..c7737e0d1 100644 --- a/certbot/plugins/webroot.py +++ b/certbot/certbot/_internal/plugins/webroot.py @@ -4,26 +4,27 @@ import collections import errno import json import logging -import os import six import zope.component import zope.interface from acme import challenges # pylint: disable=unused-import -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import Dict, Set, DefaultDict, List -# pylint: enable=unused-import, no-name-in-module - +from acme.magic_typing import DefaultDict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module from certbot import achallenges # pylint: disable=unused-import -from certbot import cli from certbot import errors from certbot import interfaces -from certbot.display import util as display_util +from certbot._internal import cli +from certbot.compat import filesystem +from certbot.compat import os from certbot.display import ops +from certbot.display import util as display_util from certbot.plugins import common from certbot.plugins import util - +from certbot.util import safe_open logger = logging.getLogger(__name__) @@ -134,8 +135,7 @@ to serve all files under specified web root ({0}).""" raise errors.PluginError( "Every requested domain must have a " "webroot when using the webroot plugin.") - else: # code == display_util.OK - return None if index == 0 else known_webroots[index - 1] + return None if index == 0 else known_webroots[index - 1] # code == display_util.OK def _prompt_for_new_webroot(self, domain, allowraise=False): code, webroot = ops.validated_directory( @@ -145,12 +145,10 @@ to serve all files under specified web root ({0}).""" if code == display_util.CANCEL: if not allowraise: return None - else: - raise errors.PluginError( - "Every requested domain must have a " - "webroot when using the webroot plugin.") - else: # code == display_util.OK - return _validate_webroot(webroot) + raise errors.PluginError( + "Every requested domain must have a " + "webroot when using the webroot plugin.") + return _validate_webroot(webroot) # code == display_util.OK def _create_challenge_dirs(self): path_map = self.conf("map") @@ -169,19 +167,19 @@ to serve all files under specified web root ({0}).""" # run as non-root (GH #1795) old_umask = os.umask(0o022) try: - stat_path = os.stat(path) # We ignore the last prefix in the next iteration, # as it does not correspond to a folder path ('/' or 'C:') for prefix in sorted(util.get_prefixes(self.full_roots[name])[:-1], key=len): try: - # This is coupled with the "umask" call above because + # Set owner as parent directory if possible, apply mode for Linux/Windows. + # For Linux, this is coupled with the "umask" call above because # os.mkdir's "mode" parameter may not always work: # https://docs.python.org/3/library/os.html#os.mkdir - os.mkdir(prefix, 0o0755) + filesystem.mkdir(prefix, 0o755) self._created_dirs.append(prefix) - # Set owner as parent directory if possible try: - os.chown(prefix, stat_path.st_uid, stat_path.st_gid) + filesystem.copy_ownership_and_apply_mode( + path, prefix, 0o755, copy_user=True, copy_group=True) except (OSError, AttributeError) as exception: logger.info("Unable to change owner and uid of webroot directory") logger.debug("Error was: %s", exception) @@ -207,7 +205,7 @@ to serve all files under specified web root ({0}).""" old_umask = os.umask(0o022) try: - with open(validation_path, "wb") as validation_file: + with safe_open(validation_path, mode="wb", chmod=0o644) as validation_file: validation_file.write(validation.encode()) finally: os.umask(old_umask) @@ -225,7 +223,7 @@ to serve all files under specified web root ({0}).""" self.performed[root_path].remove(achall) not_removed = [] # type: List[str] - while len(self._created_dirs) > 0: + while self._created_dirs: path = self._created_dirs.pop() try: os.rmdir(path) diff --git a/certbot/renewal.py b/certbot/certbot/_internal/renewal.py similarity index 93% rename from certbot/renewal.py rename to certbot/certbot/_internal/renewal.py index 4c592b27f..bf30404f5 100644 --- a/certbot/renewal.py +++ b/certbot/certbot/_internal/renewal.py @@ -1,31 +1,29 @@ """Functionality for autorenewal and associated juggling of configurations""" from __future__ import print_function + import copy import itertools import logging -import os -import traceback +import random import sys import time -import random +import traceback +import OpenSSL import six import zope.component -import OpenSSL - from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module - -from certbot import cli from certbot import crypto_util from certbot import errors from certbot import interfaces from certbot import util -from certbot import hooks -from certbot import storage -from certbot import updater - -from certbot.plugins import disco as plugins_disco +from certbot._internal import cli +from certbot._internal import hooks +from certbot._internal import storage +from certbot._internal import updater +from certbot._internal.plugins import disco as plugins_disco +from certbot.compat import os logger = logging.getLogger(__name__) @@ -35,10 +33,8 @@ logger = logging.getLogger(__name__) # the renewal configuration process loses this information. STR_CONFIG_ITEMS = ["config_dir", "logs_dir", "work_dir", "user_agent", "server", "account", "authenticator", "installer", - "standalone_supported_challenges", "renew_hook", - "pre_hook", "post_hook", "tls_sni_01_address", - "http01_address"] -INT_CONFIG_ITEMS = ["rsa_key_size", "tls_sni_01_port", "http01_port"] + "renew_hook", "pre_hook", "post_hook", "http01_address"] +INT_CONFIG_ITEMS = ["rsa_key_size", "http01_port"] BOOL_CONFIG_ITEMS = ["must_staple", "allow_subset_of_names", "reuse_key", "autorenew"] @@ -109,11 +105,11 @@ def _restore_webroot_config(config, renewalparams): restoring logic is not able to correctly parse it from the serialized form. """ - if "webroot_map" in renewalparams: - if not cli.set_by_cli("webroot_map"): - config.webroot_map = renewalparams["webroot_map"] - elif "webroot_path" in renewalparams: - logger.debug("Ancient renewal conf file without webroot-map, restoring webroot-path") + if "webroot_map" in renewalparams and not cli.set_by_cli("webroot_map"): + config.webroot_map = renewalparams["webroot_map"] + # To understand why webroot_path and webroot_map processing are not mutually exclusive, + # see https://github.com/certbot/certbot/pull/7095 + if "webroot_path" in renewalparams and not cli.set_by_cli("webroot_path"): wp = renewalparams["webroot_path"] if isinstance(wp, six.string_types): # prior to 0.1.0, webroot_path was a string wp = [wp] @@ -196,7 +192,7 @@ def _restore_pref_challs(unused_name, value): :returns: converted option value to be stored in the runtime config :rtype: `list` of `str` - :raises errors.Error: if value can't be converted to an bool + :raises errors.Error: if value can't be converted to a bool """ # If pref_challs has only one element, configobj saves the value @@ -207,7 +203,7 @@ def _restore_pref_challs(unused_name, value): def _restore_bool(name, value): - """Restores an boolean key-value pair from a renewal config file. + """Restores a boolean key-value pair from a renewal config file. :param str name: option name :param str value: option value @@ -215,7 +211,7 @@ def _restore_bool(name, value): :returns: converted option value to be stored in the runtime config :rtype: bool - :raises errors.Error: if value can't be converted to an bool + :raises errors.Error: if value can't be converted to a bool """ lowercase_value = value.lower() @@ -248,7 +244,7 @@ def _restore_int(name, value): def _restore_str(unused_name, value): - """Restores an string key-value pair from a renewal config file. + """Restores a string key-value pair from a renewal config file. :param str unused_name: option name :param str value: option value @@ -375,7 +371,7 @@ def _renew_describe_results(config, renew_successes, renew_failures, disp.notification("\n".join(out), wrap=False) -def handle_renewal_request(config): # pylint: disable=too-many-locals,too-many-branches,too-many-statements +def handle_renewal_request(config): """Examine each lineage; renew if due and report results""" # This is trivially False if config.domains is empty @@ -432,12 +428,12 @@ def handle_renewal_request(config): # pylint: disable=too-many-locals,too-many- # XXX: ensure that each call here replaces the previous one zope.component.provideUtility(lineage_config) renewal_candidate.ensure_deployed() - from certbot import main + from certbot._internal import main plugins = plugins_disco.PluginsRegistry.find_all() if should_renew(lineage_config, renewal_candidate): # Apply random sleep upon first renewal if needed if apply_random_sleep: - sleep_time = random.randint(1, 60 * 8) + sleep_time = random.uniform(1, 60 * 8) logger.info("Non-interactive renewal: random delay of %s seconds", sleep_time) time.sleep(sleep_time) @@ -475,5 +471,7 @@ def handle_renewal_request(config): # pylint: disable=too-many-locals,too-many- if renew_failures or parse_failures: raise errors.Error("{0} renew failure(s), {1} parse failure(s)".format( len(renew_failures), len(parse_failures))) - else: - logger.debug("no renewal failures") + + # Windows installer integration tests rely on handle_renewal_request behavior here. + # If the text below changes, these tests will need to be updated accordingly. + logger.debug("no renewal failures") diff --git a/certbot/reporter.py b/certbot/certbot/_internal/reporter.py similarity index 99% rename from certbot/reporter.py rename to certbot/certbot/_internal/reporter.py index e0063d8e5..947f343d4 100644 --- a/certbot/reporter.py +++ b/certbot/certbot/_internal/reporter.py @@ -12,7 +12,6 @@ import zope.interface from certbot import interfaces from certbot import util - logger = logging.getLogger(__name__) diff --git a/certbot/storage.py b/certbot/certbot/_internal/storage.py similarity index 95% rename from certbot/storage.py rename to certbot/certbot/_internal/storage.py index d17a0f29d..964515eee 100644 --- a/certbot/storage.py +++ b/certbot/certbot/_internal/storage.py @@ -2,27 +2,27 @@ import datetime import glob import logging -import os import re +import shutil import stat import configobj import parsedatetime import pytz -import shutil import six import certbot -from certbot import cli -from certbot import compat -from certbot import constants from certbot import crypto_util from certbot import errors -from certbot import error_handler +from certbot import interfaces from certbot import util - +from certbot._internal import cli +from certbot._internal import constants +from certbot._internal import error_handler +from certbot._internal.plugins import disco as plugins_disco +from certbot.compat import filesystem +from certbot.compat import os from certbot.plugins import common as plugins_common -from certbot.plugins import disco as plugins_disco logger = logging.getLogger(__name__) @@ -144,7 +144,7 @@ def write_renewal_config(o_filename, n_filename, archive_dir, target, relevant_d # Copy permissions from the old version of the file, if it exists. if os.path.exists(o_filename): current_permissions = stat.S_IMODE(os.lstat(o_filename).st_mode) - os.chmod(n_filename, current_permissions) + filesystem.chmod(n_filename, current_permissions) with open(n_filename, "wb") as f: config.write(outfile=f) @@ -163,7 +163,7 @@ def rename_renewal_config(prev_name, new_name, cli_config): raise errors.ConfigurationError("The new certificate name " "is already in use.") try: - os.rename(prev_filename, new_filename) + filesystem.replace(prev_filename, new_filename) except OSError: raise errors.ConfigurationError("Please specify a valid filename " "for the new certificate name.") @@ -192,7 +192,7 @@ def update_configuration(lineagename, archive_dir, target, cli_config): # Save only the config items that are relevant to renewal values = relevant_values(vars(cli_config.namespace)) write_renewal_config(config_filename, temp_filename, archive_dir, target, values) - compat.os_rename(temp_filename, config_filename) + filesystem.replace(temp_filename, config_filename) return configobj.ConfigObj(config_filename) @@ -239,16 +239,17 @@ def _write_live_readme_to(readme_path, is_base_dir=False): "certificates.\n".format(prefix=prefix)) -def _relevant(option): +def _relevant(namespaces, option): """ Is this option one that could be restored for future renewal purposes? + + :param namespaces: plugin namespaces for configuration options + :type namespaces: `list` of `str` :param str option: the name of the option :rtype: bool """ - from certbot import renewal - plugins = plugins_disco.PluginsRegistry.find_all() - namespaces = [plugins_common.dest_namespace(plugin) for plugin in plugins] + from certbot._internal import renewal return (option in renewal.CONFIG_ITEMS or any(option.startswith(namespace) for namespace in namespaces)) @@ -263,10 +264,13 @@ def relevant_values(all_values): :rtype dict: """ + plugins = plugins_disco.PluginsRegistry.find_all() + namespaces = [plugins_common.dest_namespace(plugin) for plugin in plugins] + rv = dict( (option, value) for option, value in six.iteritems(all_values) - if _relevant(option) and cli.option_was_set(option, value)) + if _relevant(namespaces, option) and cli.option_was_set(option, value)) # We always save the server value to help with forward compatibility # and behavioral consistency when versions of Certbot with different # server defaults are used. @@ -301,8 +305,7 @@ def full_archive_path(config_obj, cli_config, lineagename): """ if config_obj and "archive_dir" in config_obj: return config_obj["archive_dir"] - else: - return os.path.join(cli_config.default_archive_dir, lineagename) + return os.path.join(cli_config.default_archive_dir, lineagename) def _full_live_path(cli_config, lineagename): """Returns the full default live path for a lineagename""" @@ -374,8 +377,7 @@ def delete_files(config, certname): logger.debug("Unable to remove %s", archive_path) -class RenewableCert(object): - # pylint: disable=too-many-instance-attributes,too-many-public-methods +class RenewableCert(interfaces.RenewableCert): """Renewable certificate. Represents a lineage of certificates that is under the management of @@ -422,7 +424,7 @@ class RenewableCert(object): """ self.cli_config = cli_config - self.lineagename = lineagename_for_filename(config_filename) + self._lineagename = lineagename_for_filename(config_filename) # self.configuration should be used to read parameters that # may have been chosen based on default values from the @@ -482,6 +484,15 @@ class RenewableCert(object): """Duck type for self.fullchain""" return self.fullchain + @property + def lineagename(self): + """Name given to the certificate lineage. + + :rtype: str + + """ + return self._lineagename + @property def target_expiry(self): """The current target certificate's expiration datetime @@ -510,8 +521,7 @@ class RenewableCert(object): server = self.configuration["renewalparams"].get("server", None) if server: return util.is_staging(server) - else: - return False + return False def _check_symlinks(self): """Raises an exception if a symlink doesn't exist""" @@ -700,9 +710,8 @@ class RenewableCert(object): matches = pattern.match(os.path.basename(target)) if matches: return int(matches.groups()[0]) - else: - logger.debug("No matches for target %s.", kind) - return None + logger.debug("No matches for target %s.", kind) + return None def version(self, kind, version): """The filename that corresponds to the specified version and kind. @@ -859,21 +868,15 @@ class RenewableCert(object): for _, link in previous_links: os.unlink(link) - def names(self, version=None): + def names(self): """What are the subject names of this certificate? - (If no version is specified, use the current version.) - - :param int version: the desired version number :returns: the subject names :rtype: `list` of `str` :raises .CertStorageError: if could not find cert file. """ - if version is None: - target = self.current_target("cert") - else: - target = self.version("cert", version) + target = self.current_target("cert") if target is None: raise errors.CertStorageError("could not find cert file") with open(target) as f: @@ -952,7 +955,6 @@ class RenewableCert(object): @classmethod def new_lineage(cls, lineagename, cert, privkey, chain, cli_config): - # pylint: disable=too-many-locals """Create a new certificate lineage. Attempts to create a certificate lineage -- enrolled for @@ -984,7 +986,7 @@ class RenewableCert(object): for i in (cli_config.renewal_configs_dir, cli_config.default_archive_dir, cli_config.live_dir): if not os.path.exists(i): - os.makedirs(i, 0o700) + filesystem.makedirs(i, 0o700) logger.debug("Creating directory %s.", i) config_file, config_filename = util.unique_lineage_name( cli_config.renewal_configs_dir, lineagename) @@ -1006,16 +1008,14 @@ class RenewableCert(object): config_file.close() raise errors.CertStorageError( "live directory exists for " + lineagename) - os.mkdir(archive) - os.mkdir(live_dir) + filesystem.mkdir(archive) + filesystem.mkdir(live_dir) logger.debug("Archive directory %s and live " "directory %s created.", archive, live_dir) # Put the data into the appropriate files on disk - target = dict([(kind, os.path.join(live_dir, kind + ".pem")) - for kind in ALL_FOUR]) - archive_target = dict([(kind, os.path.join(archive, kind + "1.pem")) - for kind in ALL_FOUR]) + target = {kind: os.path.join(live_dir, kind + ".pem") for kind in ALL_FOUR} + archive_target = {kind: os.path.join(archive, kind + "1.pem") for kind in ALL_FOUR} for kind in ALL_FOUR: os.symlink(_relpath_from_file(archive_target[kind], target[kind]), target[kind]) with open(target["cert"], "wb") as f: @@ -1080,10 +1080,8 @@ class RenewableCert(object): self.cli_config = cli_config target_version = self.next_free_version() - target = dict( - [(kind, - os.path.join(self.archive_dir, "{0}{1}.pem".format(kind, target_version))) - for kind in ALL_FOUR]) + target = {kind: os.path.join(self.archive_dir, "{0}{1}.pem".format(kind, target_version)) + for kind in ALL_FOUR} old_privkey = os.path.join( self.archive_dir, "privkey{0}.pem".format(prior_version)) @@ -1104,13 +1102,11 @@ class RenewableCert(object): with util.safe_open(target["privkey"], "wb", chmod=BASE_PRIVKEY_MODE) as f: logger.debug("Writing new private key to %s.", target["privkey"]) f.write(new_privkey) - # Preserve gid and (mode & 074) from previous privkey in this lineage. - old_mode = stat.S_IMODE(os.stat(old_privkey).st_mode) & \ - (stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP | \ - stat.S_IROTH) - mode = BASE_PRIVKEY_MODE | old_mode - os.chown(target["privkey"], -1, os.stat(old_privkey).st_gid) - os.chmod(target["privkey"], mode) + # Preserve gid and (mode & MASK_FOR_PRIVATE_KEY_PERMISSIONS) + # from previous privkey in this lineage. + mode = filesystem.compute_private_key_mode(old_privkey, BASE_PRIVKEY_MODE) + filesystem.copy_ownership_and_apply_mode( + old_privkey, target["privkey"], mode, copy_user=False, copy_group=True) # Save everything else with open(target["cert"], "wb") as f: diff --git a/certbot/updater.py b/certbot/certbot/_internal/updater.py similarity index 98% rename from certbot/updater.py rename to certbot/certbot/_internal/updater.py index 58df6fcb4..961436ca5 100644 --- a/certbot/updater.py +++ b/certbot/certbot/_internal/updater.py @@ -3,8 +3,7 @@ import logging from certbot import errors from certbot import interfaces - -from certbot.plugins import selection as plug_sel +from certbot._internal.plugins import selection as plug_sel import certbot.plugins.enhancements as enhancements logger = logging.getLogger(__name__) diff --git a/certbot/achallenges.py b/certbot/certbot/achallenges.py similarity index 97% rename from certbot/achallenges.py rename to certbot/certbot/achallenges.py index 6535a6b63..70588683d 100644 --- a/certbot/achallenges.py +++ b/certbot/certbot/achallenges.py @@ -23,11 +23,9 @@ import josepy as jose from acme import challenges - logger = logging.getLogger(__name__) -# pylint: disable=too-few-public-methods class AnnotatedChallenge(jose.ImmutableMap): """Client annotated challenge. diff --git a/certbot/certbot/compat/__init__.py b/certbot/certbot/compat/__init__.py new file mode 100644 index 000000000..74451131a --- /dev/null +++ b/certbot/certbot/compat/__init__.py @@ -0,0 +1,6 @@ +""" +Compatibility layer to run certbot both on Linux and Windows. + +This package contains all logic that needs to be implemented specifically for Linux and for Windows. +Then the rest of certbot code relies on this module to be platform agnostic. +""" diff --git a/certbot/certbot/compat/_path.py b/certbot/certbot/compat/_path.py new file mode 100644 index 000000000..5c5fe460e --- /dev/null +++ b/certbot/certbot/compat/_path.py @@ -0,0 +1,35 @@ +""" +This compat module wraps os.path to forbid some functions. + +isort:skip_file +""" +# pylint: disable=function-redefined +from __future__ import absolute_import + +# First round of wrapping: we import statically all public attributes exposed by the os.path +# module. This allows in particular to have pylint, mypy, IDEs be aware that most of os.path +# members are available in certbot.compat.path. +from os.path import * # type: ignore # pylint: disable=wildcard-import,unused-wildcard-import,redefined-builtin,os-module-forbidden + +# Second round of wrapping: we import dynamically all attributes from the os.path module that have +# not yet been imported by the first round (static star import). +import os.path as std_os_path # pylint: disable=os-module-forbidden +import sys as std_sys + +ourselves = std_sys.modules[__name__] +for attribute in dir(std_os_path): + # Check if the attribute does not already exist in our module. It could be internal attributes + # of the module (__name__, __doc__), or attributes from standard os.path already imported with + # `from os.path import *`. + if not hasattr(ourselves, attribute): + setattr(ourselves, attribute, getattr(std_os_path, attribute)) + +# Clean all remaining importables that are not from the core os.path module. +del ourselves, std_os_path, std_sys + + +# Function os.path.realpath is broken on some versions of Python for Windows. +def realpath(*unused_args, **unused_kwargs): + """Method os.path.realpath() is forbidden""" + raise RuntimeError('Usage of os.path.realpath() is forbidden. ' + 'Use certbot.compat.filesystem.realpath() instead.') diff --git a/certbot/certbot/compat/filesystem.py b/certbot/certbot/compat/filesystem.py new file mode 100644 index 000000000..65bb53f38 --- /dev/null +++ b/certbot/certbot/compat/filesystem.py @@ -0,0 +1,610 @@ +"""Compat module to handle files security on Windows and Linux""" +from __future__ import absolute_import + +import errno +import os # pylint: disable=os-module-forbidden +import stat + +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Tuple # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Union # pylint: disable=unused-import, no-name-in-module + +try: + # pylint: disable=import-error + import ntsecuritycon + import win32security + import win32con + import win32api + import win32file + import pywintypes + import winerror + # pylint: enable=import-error +except ImportError: + POSIX_MODE = True +else: + POSIX_MODE = False + + +def chmod(file_path, mode): + # type: (str, int) -> None + """ + Apply a POSIX mode on given file_path: + * for Linux, the POSIX mode will be directly applied using chmod, + * for Windows, the POSIX mode will be translated into a Windows DACL that make sense for + Certbot context, and applied to the file using kernel calls. + + The definition of the Windows DACL that correspond to a POSIX mode, in the context of Certbot, + is explained at https://github.com/certbot/certbot/issues/6356 and is implemented by the + method _generate_windows_flags(). + + :param str file_path: Path of the file + :param int mode: POSIX mode to apply + """ + if POSIX_MODE: + os.chmod(file_path, mode) + else: + _apply_win_mode(file_path, mode) + + +# One could ask why there is no copy_ownership() function, or even a reimplementation +# of os.chown() that would modify the ownership of file without touching the mode itself. +# This is because on Windows, it would require recalculating the existing DACL against +# the new owner, since the DACL is composed of ACEs that targets a specific user, not dynamically +# the current owner of a file. This action would be necessary to keep consistency between +# the POSIX mode applied to the file and the current owner of this file. +# Since copying and editing arbitrary DACL is very difficult, and since we actually know +# the mode to apply at the time the owner of a file should change, it is easier to just +# change the owner, then reapply the known mode, as copy_ownership_and_apply_mode() does. +def copy_ownership_and_apply_mode(src, dst, mode, copy_user, copy_group): + # type: (str, str, int, bool, bool) -> None + """ + Copy ownership (user and optionally group on Linux) from the source to the + destination, then apply given mode in compatible way for Linux and Windows. + This replaces the os.chown command. + :param str src: Path of the source file + :param str dst: Path of the destination file + :param int mode: Permission mode to apply on the destination file + :param bool copy_user: Copy user if `True` + :param bool copy_group: Copy group if `True` on Linux (has no effect on Windows) + """ + if POSIX_MODE: + stats = os.stat(src) + user_id = stats.st_uid if copy_user else -1 + group_id = stats.st_gid if copy_group else -1 + # On Windows, os.chown does not exist. This is checked through POSIX_MODE value, + # but MyPy/PyLint does not know it and raises an error here on Windows. + # We disable specifically the check to fix the issue. + os.chown(dst, user_id, group_id) + elif copy_user: + # There is no group handling in Windows + _copy_win_ownership(src, dst) + + chmod(dst, mode) + + +def check_mode(file_path, mode): + # type: (str, int) -> bool + """ + Check if the given mode matches the permissions of the given file. + On Linux, will make a direct comparison, on Windows, mode will be compared against + the security model. + :param str file_path: Path of the file + :param int mode: POSIX mode to test + :rtype: bool + :return: True if the POSIX mode matches the file permissions + """ + if POSIX_MODE: + return stat.S_IMODE(os.stat(file_path).st_mode) == mode + + return _check_win_mode(file_path, mode) + + +def check_owner(file_path): + # type: (str) -> bool + """ + Check if given file is owned by current user. + :param str file_path: File path to check + :rtype: bool + :return: True if given file is owned by current user, False otherwise. + """ + if POSIX_MODE: + # On Windows, os.getuid does not exist. This is checked through POSIX_MODE value, + # but MyPy/PyLint does not know it and raises an error here on Windows. + # We disable specifically the check to fix the issue. + return os.stat(file_path).st_uid == os.getuid() # type: ignore + + # Get owner sid of the file + security = win32security.GetFileSecurity(file_path, win32security.OWNER_SECURITY_INFORMATION) + user = security.GetSecurityDescriptorOwner() + + # Compare sids + return _get_current_user() == user + + +def check_permissions(file_path, mode): + # type: (str, int) -> bool + """ + Check if given file has the given mode and is owned by current user. + :param str file_path: File path to check + :param int mode: POSIX mode to check + :rtype: bool + :return: True if file has correct mode and owner, False otherwise. + """ + return check_owner(file_path) and check_mode(file_path, mode) + + +def open(file_path, flags, mode=0o777): # pylint: disable=redefined-builtin + # type: (str, int, int) -> int + """ + Wrapper of original os.open function, that will ensure on Windows that given mode + is correctly applied. + :param str file_path: The file path to open + :param int flags: Flags to apply on file while opened + :param int mode: POSIX mode to apply on file when opened, + Python defaults will be applied if ``None`` + :returns: the file descriptor to the opened file + :rtype: int + :raise: OSError(errno.EEXIST) if the file already exists and os.O_CREAT & os.O_EXCL are set, + OSError(errno.EACCES) on Windows if the file already exists and is a directory, and + os.O_CREAT is set. + """ + if POSIX_MODE: + # On Linux, invoke os.open directly. + return os.open(file_path, flags, mode) + + # Windows: handle creation of the file atomically with proper permissions. + if flags & os.O_CREAT: + # If os.O_EXCL is set, we will use the "CREATE_NEW", that will raise an exception if + # file exists, matching the API contract of this bit flag. Otherwise, we use + # "CREATE_ALWAYS" that will always create the file whether it exists or not. + disposition = win32con.CREATE_NEW if flags & os.O_EXCL else win32con.CREATE_ALWAYS + + attributes = win32security.SECURITY_ATTRIBUTES() + security = attributes.SECURITY_DESCRIPTOR + user = _get_current_user() + dacl = _generate_dacl(user, mode) + # We set second parameter to 0 (`False`) to say that this security descriptor is + # NOT constructed from a default mechanism, but is explicitly set by the user. + # See https://docs.microsoft.com/en-us/windows/desktop/api/securitybaseapi/nf-securitybaseapi-setsecuritydescriptorowner # pylint: disable=line-too-long + security.SetSecurityDescriptorOwner(user, 0) + # We set first parameter to 1 (`True`) to say that this security descriptor contains + # a DACL. Otherwise second and third parameters are ignored. + # We set third parameter to 0 (`False`) to say that this security descriptor is + # NOT constructed from a default mechanism, but is explicitly set by the user. + # See https://docs.microsoft.com/en-us/windows/desktop/api/securitybaseapi/nf-securitybaseapi-setsecuritydescriptordacl # pylint: disable=line-too-long + security.SetSecurityDescriptorDacl(1, dacl, 0) + + handle = None + try: + handle = win32file.CreateFile(file_path, win32file.GENERIC_READ, + win32file.FILE_SHARE_READ & win32file.FILE_SHARE_WRITE, + attributes, disposition, 0, None) + except pywintypes.error as err: + # Handle native windows errors into python errors to be consistent with the API + # of os.open in the situation of a file already existing or locked. + if err.winerror == winerror.ERROR_FILE_EXISTS: + raise OSError(errno.EEXIST, err.strerror) + if err.winerror == winerror.ERROR_SHARING_VIOLATION: + raise OSError(errno.EACCES, err.strerror) + raise err + finally: + if handle: + handle.Close() + + # At this point, the file that did not exist has been created with proper permissions, + # so os.O_CREAT and os.O_EXCL are not needed anymore. We remove them from the flags to + # avoid a FileExists exception before calling os.open. + return os.open(file_path, flags ^ os.O_CREAT ^ os.O_EXCL) + + # Windows: general case, we call os.open, let exceptions be thrown, then chmod if all is fine. + handle = os.open(file_path, flags) + chmod(file_path, mode) + return handle + + +def makedirs(file_path, mode=0o777): + # type: (str, int) -> None + """ + Rewrite of original os.makedirs function, that will ensure on Windows that given mode + is correctly applied. + :param str file_path: The file path to open + :param int mode: POSIX mode to apply on leaf directory when created, Python defaults + will be applied if ``None`` + """ + if POSIX_MODE: + return os.makedirs(file_path, mode) + + orig_mkdir_fn = os.mkdir + try: + # As we know that os.mkdir is called internally by os.makedirs, we will swap the function in + # os module for the time of makedirs execution on Windows. + os.mkdir = mkdir # type: ignore + return os.makedirs(file_path, mode) + finally: + os.mkdir = orig_mkdir_fn + + +def mkdir(file_path, mode=0o777): + # type: (str, int) -> None + """ + Rewrite of original os.mkdir function, that will ensure on Windows that given mode + is correctly applied. + :param str file_path: The file path to open + :param int mode: POSIX mode to apply on directory when created, Python defaults + will be applied if ``None`` + """ + if POSIX_MODE: + return os.mkdir(file_path, mode) + + attributes = win32security.SECURITY_ATTRIBUTES() + security = attributes.SECURITY_DESCRIPTOR + user = _get_current_user() + dacl = _generate_dacl(user, mode) + security.SetSecurityDescriptorOwner(user, False) + security.SetSecurityDescriptorDacl(1, dacl, 0) + + try: + win32file.CreateDirectory(file_path, attributes) + except pywintypes.error as err: + # Handle native windows error into python error to be consistent with the API + # of os.mkdir in the situation of a directory already existing. + if err.winerror == winerror.ERROR_ALREADY_EXISTS: + raise OSError(errno.EEXIST, err.strerror, file_path, err.winerror) + raise err + + return None + + +def replace(src, dst): + # type: (str, str) -> None + """ + Rename a file to a destination path and handles situations where the destination exists. + :param str src: The current file path. + :param str dst: The new file path. + """ + if hasattr(os, 'replace'): + # Use replace if possible. On Windows, only Python >= 3.5 is supported + # so we can assume that os.replace() is always available for this platform. + getattr(os, 'replace')(src, dst) + else: + # Otherwise, use os.rename() that behaves like os.replace() on Linux. + os.rename(src, dst) + + +def realpath(file_path): + # type: (str) -> str + """ + Find the real path for the given path. This method resolves symlinks, including + recursive symlinks, and is protected against symlinks that creates an infinite loop. + """ + original_path = file_path + + if POSIX_MODE: + path = os.path.realpath(file_path) + if os.path.islink(path): + # If path returned by realpath is still a link, it means that it failed to + # resolve the symlink because of a loop. + # See realpath code: https://github.com/python/cpython/blob/master/Lib/posixpath.py + raise RuntimeError('Error, link {0} is a loop!'.format(original_path)) + return path + + inspected_paths = [] # type: List[str] + while os.path.islink(file_path): + link_path = file_path + file_path = os.readlink(file_path) + if not os.path.isabs(file_path): + file_path = os.path.join(os.path.dirname(link_path), file_path) + if file_path in inspected_paths: + raise RuntimeError('Error, link {0} is a loop!'.format(original_path)) + inspected_paths.append(file_path) + + return os.path.abspath(file_path) + + +# On Windows is_executable run from an unprivileged shell may claim that a path is +# executable when it is excutable only if run from a privileged shell. This result +# is due to the fact that GetEffectiveRightsFromAcl calculate effective rights +# without taking into consideration if the target user has currently required the +# elevated privileges or not. However this is not a problem since certbot always +# requires to be run under a privileged shell, so the user will always benefit +# from the highest (privileged one) set of permissions on a given file. +def is_executable(path): + # type: (str) -> bool + """ + Is path an executable file? + :param str path: path to test + :return: True if path is an executable file + :rtype: bool + """ + if POSIX_MODE: + return os.path.isfile(path) and os.access(path, os.X_OK) + + return _win_is_executable(path) + + +def has_world_permissions(path): + # type: (str) -> bool + """ + Check if everybody/world has any right (read/write/execute) on a file given its path + :param str path: path to test + :return: True if everybody/world has any right to the file + :rtype: bool + """ + if POSIX_MODE: + return bool(stat.S_IMODE(os.stat(path).st_mode) & stat.S_IRWXO) + + security = win32security.GetFileSecurity(path, win32security.DACL_SECURITY_INFORMATION) + dacl = security.GetSecurityDescriptorDacl() + + return bool(dacl.GetEffectiveRightsFromAcl({ + 'TrusteeForm': win32security.TRUSTEE_IS_SID, + 'TrusteeType': win32security.TRUSTEE_IS_USER, + 'Identifier': win32security.ConvertStringSidToSid('S-1-1-0'), + })) + + +def compute_private_key_mode(old_key, base_mode): + # type: (str, int) -> int + """ + Calculate the POSIX mode to apply to a private key given the previous private key + :param str old_key: path to the previous private key + :param int base_mode: the minimum modes to apply to a private key + :return: the POSIX mode to apply + :rtype: int + """ + if POSIX_MODE: + # On Linux, we keep read/write/execute permissions + # for group and read permissions for everybody. + old_mode = (stat.S_IMODE(os.stat(old_key).st_mode) & + (stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP | stat.S_IROTH)) + return base_mode | old_mode + + # On Windows, the mode returned by os.stat is not reliable, + # so we do not keep any permission from the previous private key. + return base_mode + + +def has_same_ownership(path1, path2): + # type: (str, str) -> bool + """ + Return True if the ownership of two files given their respective path is the same. + On Windows, ownership is checked against owner only, since files do not have a group owner. + :param str path1: path to the first file + :param str path2: path to the second file + :return: True if both files have the same ownership, False otherwise + :rtype: bool + + """ + if POSIX_MODE: + stats1 = os.stat(path1) + stats2 = os.stat(path2) + return (stats1.st_uid, stats1.st_gid) == (stats2.st_uid, stats2.st_gid) + + security1 = win32security.GetFileSecurity(path1, win32security.OWNER_SECURITY_INFORMATION) + user1 = security1.GetSecurityDescriptorOwner() + + security2 = win32security.GetFileSecurity(path2, win32security.OWNER_SECURITY_INFORMATION) + user2 = security2.GetSecurityDescriptorOwner() + + return user1 == user2 + + +def has_min_permissions(path, min_mode): + # type: (str, int) -> bool + """ + Check if a file given its path has at least the permissions defined by the given minimal mode. + On Windows, group permissions are ignored since files do not have a group owner. + :param str path: path to the file to check + :param int min_mode: the minimal permissions expected + :return: True if the file matches the minimal permissions expectations, False otherwise + :rtype: bool + """ + if POSIX_MODE: + st_mode = os.stat(path).st_mode + return st_mode == st_mode | min_mode + + # Resolve symlinks, to get a consistent result with os.stat on Linux, + # that follows symlinks by default. + path = realpath(path) + + # Get owner sid of the file + security = win32security.GetFileSecurity( + path, win32security.OWNER_SECURITY_INFORMATION | win32security.DACL_SECURITY_INFORMATION) + user = security.GetSecurityDescriptorOwner() + dacl = security.GetSecurityDescriptorDacl() + min_dacl = _generate_dacl(user, min_mode) + + for index in range(min_dacl.GetAceCount()): + min_ace = min_dacl.GetAce(index) + + # On a given ACE, index 0 is the ACE type, 1 is the permission mask, and 2 is the SID. + # See: http://timgolden.me.uk/pywin32-docs/PyACL__GetAce_meth.html + mask = min_ace[1] + user = min_ace[2] + + effective_mask = dacl.GetEffectiveRightsFromAcl({ + 'TrusteeForm': win32security.TRUSTEE_IS_SID, + 'TrusteeType': win32security.TRUSTEE_IS_USER, + 'Identifier': user, + }) + + if effective_mask != effective_mask | mask: + return False + + return True + + +def _win_is_executable(path): + if not os.path.isfile(path): + return False + + security = win32security.GetFileSecurity(path, win32security.DACL_SECURITY_INFORMATION) + dacl = security.GetSecurityDescriptorDacl() + + mode = dacl.GetEffectiveRightsFromAcl({ + 'TrusteeForm': win32security.TRUSTEE_IS_SID, + 'TrusteeType': win32security.TRUSTEE_IS_USER, + 'Identifier': _get_current_user(), + }) + + return mode & ntsecuritycon.FILE_GENERIC_EXECUTE == ntsecuritycon.FILE_GENERIC_EXECUTE + + +def _apply_win_mode(file_path, mode): + """ + This function converts the given POSIX mode into a Windows ACL list, and applies it to the + file given its path. If the given path is a symbolic link, it will resolved to apply the + mode on the targeted file. + """ + file_path = realpath(file_path) + # Get owner sid of the file + security = win32security.GetFileSecurity(file_path, win32security.OWNER_SECURITY_INFORMATION) + user = security.GetSecurityDescriptorOwner() + + # New DACL, that will overwrite existing one (including inherited permissions) + dacl = _generate_dacl(user, mode) + + # Apply the new DACL + security.SetSecurityDescriptorDacl(1, dacl, 0) + win32security.SetFileSecurity(file_path, win32security.DACL_SECURITY_INFORMATION, security) + + +def _generate_dacl(user_sid, mode): + analysis = _analyze_mode(mode) + + # Get standard accounts from "well-known" sid + # See the list here: + # https://support.microsoft.com/en-us/help/243330/well-known-security-identifiers-in-windows-operating-systems + system = win32security.ConvertStringSidToSid('S-1-5-18') + admins = win32security.ConvertStringSidToSid('S-1-5-32-544') + everyone = win32security.ConvertStringSidToSid('S-1-1-0') + + # New dacl, without inherited permissions + dacl = win32security.ACL() + + # If user is already system or admins, any ACE defined here would be superseded by + # the full control ACE that will be added after. + if user_sid not in [system, admins]: + # Handle user rights + user_flags = _generate_windows_flags(analysis['user']) + if user_flags: + dacl.AddAccessAllowedAce(win32security.ACL_REVISION, user_flags, user_sid) + + # Handle everybody rights + everybody_flags = _generate_windows_flags(analysis['all']) + if everybody_flags: + dacl.AddAccessAllowedAce(win32security.ACL_REVISION, everybody_flags, everyone) + + # Handle administrator rights + full_permissions = _generate_windows_flags({'read': True, 'write': True, 'execute': True}) + dacl.AddAccessAllowedAce(win32security.ACL_REVISION, full_permissions, system) + dacl.AddAccessAllowedAce(win32security.ACL_REVISION, full_permissions, admins) + + return dacl + + +def _analyze_mode(mode): + return { + 'user': { + 'read': mode & stat.S_IRUSR, + 'write': mode & stat.S_IWUSR, + 'execute': mode & stat.S_IXUSR, + }, + 'all': { + 'read': mode & stat.S_IROTH, + 'write': mode & stat.S_IWOTH, + 'execute': mode & stat.S_IXOTH, + }, + } + + +def _copy_win_ownership(src, dst): + security_src = win32security.GetFileSecurity(src, win32security.OWNER_SECURITY_INFORMATION) + user_src = security_src.GetSecurityDescriptorOwner() + + security_dst = win32security.GetFileSecurity(dst, win32security.OWNER_SECURITY_INFORMATION) + # Second parameter indicates, if `False`, that the owner of the file is not provided by some + # default mechanism, but is explicitly set instead. This is obviously what we are doing here. + security_dst.SetSecurityDescriptorOwner(user_src, False) + + win32security.SetFileSecurity(dst, win32security.OWNER_SECURITY_INFORMATION, security_dst) + + +def _generate_windows_flags(rights_desc): + # Some notes about how each POSIX right is interpreted. + # + # For the rights read and execute, we have a pretty bijective relation between + # POSIX flags and their generic counterparts on Windows, so we use them directly + # (respectively ntsecuritycon.FILE_GENERIC_READ and ntsecuritycon.FILE_GENERIC_EXECUTE). + # + # But ntsecuritycon.FILE_GENERIC_WRITE does not correspond to what one could expect from a + # write access on Linux: for Windows, FILE_GENERIC_WRITE does not include delete, move or + # rename. This is something that requires ntsecuritycon.FILE_ALL_ACCESS. + # So to reproduce the write right as POSIX, we will apply ntsecuritycon.FILE_ALL_ACCESS + # subtracted of the rights corresponding to POSIX read and POSIX execute. + # + # Finally, having read + write + execute gives a ntsecuritycon.FILE_ALL_ACCESS, + # so a "Full Control" on the file. + # + # A complete list of the rights defined on NTFS can be found here: + # https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2003/cc783530(v=ws.10)#permissions-for-files-and-folders + flag = 0 + if rights_desc['read']: + flag = flag | ntsecuritycon.FILE_GENERIC_READ + if rights_desc['write']: + flag = flag | (ntsecuritycon.FILE_ALL_ACCESS + ^ ntsecuritycon.FILE_GENERIC_READ + ^ ntsecuritycon.FILE_GENERIC_EXECUTE) + if rights_desc['execute']: + flag = flag | ntsecuritycon.FILE_GENERIC_EXECUTE + + return flag + + +def _check_win_mode(file_path, mode): + # Resolve symbolic links + file_path = realpath(file_path) + # Get current dacl file + security = win32security.GetFileSecurity(file_path, win32security.OWNER_SECURITY_INFORMATION + | win32security.DACL_SECURITY_INFORMATION) + dacl = security.GetSecurityDescriptorDacl() + + # Get current file owner sid + user = security.GetSecurityDescriptorOwner() + + if not dacl: + # No DACL means full control to everyone + # This is not a deterministic permissions set. + return False + + # Calculate the target dacl + ref_dacl = _generate_dacl(user, mode) + + return _compare_dacls(dacl, ref_dacl) + + +def _compare_dacls(dacl1, dacl2): + """ + This method compare the two given DACLs to check if they are identical. + Identical means here that they contains the same set of ACEs in the same order. + """ + return ([dacl1.GetAce(index) for index in range(dacl1.GetAceCount())] == + [dacl2.GetAce(index) for index in range(dacl2.GetAceCount())]) + + +def _get_current_user(): + """ + Return the pySID corresponding to the current user. + """ + # We craft the account_name ourselves instead of calling for instance win32api.GetUserNameEx, + # because this function returns nonsense values when Certbot is run under NT AUTHORITY\SYSTEM. + # To run Certbot under NT AUTHORITY\SYSTEM, you can open a shell using the instructions here: + # https://blogs.technet.microsoft.com/ben_parker/2010/10/27/how-do-i-run-powershell-execommand-prompt-as-the-localsystem-account-on-windows-7/ + account_name = r"{0}\{1}".format(win32api.GetDomainName(), win32api.GetUserName()) + # LookupAccountName() expects the system name as first parameter. By passing None to it, + # we instruct Windows to first search the matching account in the machine local accounts, + # then into the primary domain accounts, if the machine has joined a domain, then finally + # into the trusted domains accounts. This is the preferred lookup mechanism to use in Windows + # if there is no reason to use a specific lookup mechanism. + # See https://docs.microsoft.com/en-us/windows/desktop/api/winbase/nf-winbase-lookupaccountnamea + return win32security.LookupAccountName(None, account_name)[0] diff --git a/certbot/certbot/compat/misc.py b/certbot/certbot/compat/misc.py new file mode 100644 index 000000000..ffe611edb --- /dev/null +++ b/certbot/certbot/compat/misc.py @@ -0,0 +1,111 @@ +""" +This compat module handles various platform specific calls that do not fall into one +particular category. +""" +from __future__ import absolute_import + +import select +import sys + +from certbot import errors +from certbot.compat import os + +try: + from win32com.shell import shell as shellwin32 # pylint: disable=import-error + POSIX_MODE = False +except ImportError: # pragma: no cover + POSIX_MODE = True + + + +# For Linux: define OS specific standard binary directories +STANDARD_BINARY_DIRS = ["/usr/sbin", "/usr/local/bin", "/usr/local/sbin"] if POSIX_MODE else [] + + +def raise_for_non_administrative_windows_rights(): + # type: () -> None + """ + On Windows, raise if current shell does not have the administrative rights. + Do nothing on Linux. + + :raises .errors.Error: If the current shell does not have administrative rights on Windows. + """ + if not POSIX_MODE and shellwin32.IsUserAnAdmin() == 0: # pragma: no cover + raise errors.Error('Error, certbot must be run on a shell with administrative rights.') + + +def readline_with_timeout(timeout, prompt): + # type: (float, str) -> str + """ + Read user input to return the first line entered, or raise after specified timeout. + + :param float timeout: The timeout in seconds given to the user. + :param str prompt: The prompt message to display to the user. + + :returns: The first line entered by the user. + :rtype: str + + """ + try: + # Linux specific + # + # Call to select can only be done like this on UNIX + rlist, _, _ = select.select([sys.stdin], [], [], timeout) + if not rlist: + raise errors.Error( + "Timed out waiting for answer to prompt '{0}'".format(prompt)) + return rlist[0].readline() + except OSError: + # Windows specific + # + # No way with select to make a timeout to the user input on Windows, + # as select only supports socket in this case. + # So no timeout on Windows for now. + return sys.stdin.readline() + + +WINDOWS_DEFAULT_FOLDERS = { + 'config': 'C:\\Certbot', + 'work': 'C:\\Certbot\\lib', + 'logs': 'C:\\Certbot\\log', +} +LINUX_DEFAULT_FOLDERS = { + 'config': '/etc/letsencrypt', + 'work': '/var/lib/letsencrypt', + 'logs': '/var/log/letsencrypt', +} + + +def get_default_folder(folder_type): + # type: (str) -> str + """ + Return the relevant default folder for the current OS + + :param str folder_type: The type of folder to retrieve (config, work or logs) + + :returns: The relevant default folder. + :rtype: str + + """ + if os.name != 'nt': + # Linux specific + return LINUX_DEFAULT_FOLDERS[folder_type] + # Windows specific + return WINDOWS_DEFAULT_FOLDERS[folder_type] + + +def underscores_for_unsupported_characters_in_path(path): + # type: (str) -> str + """ + Replace unsupported characters in path for current OS by underscores. + :param str path: the path to normalize + :return: the normalized path + :rtype: str + """ + if os.name != 'nt': + # Linux specific + return path + + # Windows specific + drive, tail = os.path.splitdrive(path) + return drive + tail.replace(':', '_') diff --git a/certbot/certbot/compat/os.py b/certbot/certbot/compat/os.py new file mode 100644 index 000000000..0231dd51a --- /dev/null +++ b/certbot/certbot/compat/os.py @@ -0,0 +1,138 @@ +""" +This compat modules is a wrapper of the core os module that forbids usage of specific operations +(e.g. chown, chmod, getuid) that would be harmful to the Windows file security model of Certbot. +This module is intended to replace standard os module throughout certbot projects (except acme). + +isort:skip_file +""" +# pylint: disable=function-redefined +from __future__ import absolute_import + +# First round of wrapping: we import statically all public attributes exposed by the os module +# This allows in particular to have pylint, mypy, IDEs be aware that most of os members are +# available in certbot.compat.os. +from os import * # type: ignore # pylint: disable=wildcard-import,unused-wildcard-import,redefined-builtin,os-module-forbidden + +# Second round of wrapping: we import dynamically all attributes from the os module that have not +# yet been imported by the first round (static import). This covers in particular the case of +# specific python 3.x versions where not all public attributes are in the special __all__ of os, +# and so not in `from os import *`. +import os as std_os # pylint: disable=os-module-forbidden +import sys as std_sys + +ourselves = std_sys.modules[__name__] +for attribute in dir(std_os): + # Check if the attribute does not already exist in our module. It could be internal attributes + # of the module (__name__, __doc__), or attributes from standard os already imported with + # `from os import *`. + if not hasattr(ourselves, attribute): + setattr(ourselves, attribute, getattr(std_os, attribute)) + +# Import our internal path module, then allow certbot.compat.os.path +# to behave as a module (similarly to os.path). +from certbot.compat import _path as path # type: ignore # pylint: disable=wrong-import-position +std_sys.modules[__name__ + '.path'] = path + +# Clean all remaining importables that are not from the core os module. +del ourselves, std_os, std_sys + + +# Chmod is the root of all evil for our security model on Windows. With the default implementation +# of os.chmod on Windows, almost all bits on mode will be ignored, and only a general RO or RW will +# be applied. The DACL, the inner mechanism to control file access on Windows, will stay on its +# default definition, giving effectively at least read permissions to any one, as the default +# permissions on root path will be inherit by the file (as NTFS state), and root path can be read +# by anyone. So the given mode needs to be translated into a secured and not inherited DACL that +# will be applied to this file using filesystem.chmod, calling internally the win32security +# module to construct and apply the DACL. Complete security model to translate a POSIX mode into +# a suitable DACL on Windows for Certbot can be found here: +# https://github.com/certbot/certbot/issues/6356 +# Basically, it states that appropriate permissions will be set for the owner, nothing for the +# group, appropriate permissions for the "Everyone" group, and all permissions to the +# "Administrators" group + "System" user, as they can do everything anyway. +def chmod(*unused_args, **unused_kwargs): + """Method os.chmod() is forbidden""" + raise RuntimeError('Usage of os.chmod() is forbidden. ' + 'Use certbot.compat.filesystem.chmod() instead.') + + +# Because uid is not a concept on Windows, chown is useless. In fact, it is not even available +# on Python for Windows. So to be consistent on both platforms for Certbot, this method is +# always forbidden. +def chown(*unused_args, **unused_kwargs): + """Method os.chown() is forbidden""" + raise RuntimeError('Usage of os.chown() is forbidden.' + 'Use certbot.compat.filesystem.copy_ownership_and_apply_mode() instead.') + + +# The os.open function on Windows has the same effect as a call to os.chown concerning the file +# modes: these modes lack the correct control over the permissions given to the file. Instead, +# filesystem.open invokes the Windows native API `CreateFile` to ensure that permissions are +# atomically set in case of file creation, or invokes filesystem.chmod to properly set the +# permissions for the other cases. +def open(*unused_args, **unused_kwargs): + """Method os.open() is forbidden""" + raise RuntimeError('Usage of os.open() is forbidden. ' + 'Use certbot.compat.filesystem.open() instead.') + + +# Very similarly to os.open, os.mkdir has the same effects on Windows and creates an unsecured +# folder. So a similar mitigation to security.chmod is provided on this platform. +def mkdir(*unused_args, **unused_kwargs): + """Method os.mkdir() is forbidden""" + raise RuntimeError('Usage of os.mkdir() is forbidden. ' + 'Use certbot.compat.filesystem.mkdir() instead.') + + +# As said above, os.makedirs would call the original os.mkdir function recursively on Windows, +# creating the same flaws for every actual folder created. This method is modified to ensure +# that our modified os.mkdir is called on Windows, by monkey patching temporarily the mkdir method +# on the original os module, executing the modified logic to correctly protect newly created +# folders, then restoring original mkdir method in the os module. +def makedirs(*unused_args, **unused_kwargs): + """Method os.makedirs() is forbidden""" + raise RuntimeError('Usage of os.makedirs() is forbidden. ' + 'Use certbot.compat.filesystem.makedirs() instead.') + + +# Because of the blocking strategy on file handlers on Windows, rename does not behave as expected +# with POSIX systems: an exception will be raised if dst already exists. +def rename(*unused_args, **unused_kwargs): + """Method os.rename() is forbidden""" + raise RuntimeError('Usage of os.rename() is forbidden. ' + 'Use certbot.compat.filesystem.replace() instead.') + + +# Behavior of os.replace is consistent between Windows and Linux. However, it is not supported on +# Python 2.x. So, as for os.rename, we forbid it in favor of filesystem.replace. +def replace(*unused_args, **unused_kwargs): + """Method os.replace() is forbidden""" + raise RuntimeError('Usage of os.replace() is forbidden. ' + 'Use certbot.compat.filesystem.replace() instead.') + + +# Results given by os.access are inconsistent or partial on Windows, because this platform is not +# following the POSIX approach. +def access(*unused_args, **unused_kwargs): + """Method os.access() is forbidden""" + raise RuntimeError('Usage of os.access() is forbidden. ' + 'Use certbot.compat.filesystem.check_mode() or ' + 'certbot.compat.filesystem.is_executable() instead.') + + +# On Windows os.stat call result is inconsistent, with a lot of flags that are not set or +# meaningless. We need to use specialized functions from the certbot.compat.filesystem module. +def stat(*unused_args, **unused_kwargs): + """Method os.stat() is forbidden""" + raise RuntimeError('Usage of os.stat() is forbidden. ' + 'Use certbot.compat.filesystem functions instead ' + '(eg. has_min_permissions, has_same_ownership).') + + +# Method os.fstat has the same problem than os.stat, since it is the same function, +# but accepting a file descriptor instead of a path. +def fstat(*unused_args, **unused_kwargs): + """Method os.stat() is forbidden""" + raise RuntimeError('Usage of os.fstat() is forbidden. ' + 'Use certbot.compat.filesystem functions instead ' + '(eg. has_min_permissions, has_same_ownership).') diff --git a/certbot/crypto_util.py b/certbot/certbot/crypto_util.py similarity index 83% rename from certbot/crypto_util.py rename to certbot/certbot/crypto_util.py index c4a389cd5..9aae75991 100644 --- a/certbot/crypto_util.py +++ b/certbot/certbot/crypto_util.py @@ -6,30 +6,28 @@ """ import hashlib import logging -import os import warnings -import pyrfc3339 -import six -import zope.component +# See https://github.com/pyca/cryptography/issues/4275 +from cryptography import x509 # type: ignore from cryptography.exceptions import InvalidSignature from cryptography.hazmat.backends import default_backend from cryptography.hazmat.primitives.asymmetric.ec import ECDSA from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurvePublicKey from cryptography.hazmat.primitives.asymmetric.padding import PKCS1v15 from cryptography.hazmat.primitives.asymmetric.rsa import RSAPublicKey -# https://github.com/python/typeshed/tree/master/third_party/2/cryptography -from cryptography import x509 # type: ignore from OpenSSL import crypto from OpenSSL import SSL # type: ignore +import pyrfc3339 +import six +import zope.component from acme import crypto_util as acme_crypto_util from acme.magic_typing import IO # pylint: disable=unused-import, no-name-in-module -from certbot import compat from certbot import errors from certbot import interfaces from certbot import util - +from certbot.compat import os logger = logging.getLogger(__name__) @@ -61,8 +59,7 @@ def init_save_key(key_size, key_dir, keyname="key-certbot.pem"): config = zope.component.getUtility(interfaces.IConfig) # Save file - util.make_or_verify_dir(key_dir, 0o700, compat.os_geteuid(), - config.strict_permissions) + util.make_or_verify_dir(key_dir, 0o700, config.strict_permissions) key_f, key_path = util.unique_file( os.path.join(key_dir, keyname), 0o600, "wb") with key_f: @@ -92,8 +89,7 @@ def init_save_csr(privkey, names, path): privkey.pem, names, must_staple=config.must_staple) # Save CSR - util.make_or_verify_dir(path, 0o755, compat.os_geteuid(), - config.strict_permissions) + util.make_or_verify_dir(path, 0o755, config.strict_permissions) csr_f, csr_filename = util.unique_file( os.path.join(path, "csr-certbot.pem"), 0o644, "wb") with csr_f: @@ -216,52 +212,71 @@ def verify_renewable_cert(renewable_cert): 2. That fullchain matches cert and chain when concatenated. 3. Check that the private key matches the certificate. - :param `.storage.RenewableCert` renewable_cert: cert to verify + :param renewable_cert: cert to verify + :type renewable_cert: certbot.interfaces.RenewableCert :raises errors.Error: If verification fails. """ verify_renewable_cert_sig(renewable_cert) verify_fullchain(renewable_cert) - verify_cert_matches_priv_key(renewable_cert.cert, renewable_cert.privkey) + verify_cert_matches_priv_key(renewable_cert.cert_path, renewable_cert.key_path) def verify_renewable_cert_sig(renewable_cert): - """ Verifies the signature of a `.storage.RenewableCert` object. + """Verifies the signature of a RenewableCert object. - :param `.storage.RenewableCert` renewable_cert: cert to verify + :param renewable_cert: cert to verify + :type renewable_cert: certbot.interfaces.RenewableCert :raises errors.Error: If signature verification fails. """ try: - with open(renewable_cert.chain, 'rb') as chain_file: # type: IO[bytes] + with open(renewable_cert.chain_path, 'rb') as chain_file: # type: IO[bytes] chain = x509.load_pem_x509_certificate(chain_file.read(), default_backend()) - with open(renewable_cert.cert, 'rb') as cert_file: # type: IO[bytes] + with open(renewable_cert.cert_path, 'rb') as cert_file: # type: IO[bytes] cert = x509.load_pem_x509_certificate(cert_file.read(), default_backend()) pk = chain.public_key() with warnings.catch_warnings(): - warnings.simplefilter("ignore") - if isinstance(pk, RSAPublicKey): - # https://github.com/python/typeshed/blob/master/third_party/2/cryptography/hazmat/primitives/asymmetric/rsa.pyi - verifier = pk.verifier( # type: ignore - cert.signature, PKCS1v15(), cert.signature_hash_algorithm - ) - verifier.update(cert.tbs_certificate_bytes) - verifier.verify() - elif isinstance(pk, EllipticCurvePublicKey): - verifier = pk.verifier( - cert.signature, ECDSA(cert.signature_hash_algorithm) - ) - verifier.update(cert.tbs_certificate_bytes) - verifier.verify() - else: - raise errors.Error("Unsupported public key type") + verify_signed_payload(pk, cert.signature, cert.tbs_certificate_bytes, + cert.signature_hash_algorithm) except (IOError, ValueError, InvalidSignature) as e: error_str = "verifying the signature of the cert located at {0} has failed. \ - Details: {1}".format(renewable_cert.cert, e) + Details: {1}".format(renewable_cert.cert_path, e) logger.exception(error_str) raise errors.Error(error_str) +def verify_signed_payload(public_key, signature, payload, signature_hash_algorithm): + """Check the signature of a payload. + + :param RSAPublicKey/EllipticCurvePublicKey public_key: the public_key to check signature + :param bytes signature: the signature bytes + :param bytes payload: the payload bytes + :param cryptography.hazmat.primitives.hashes.HashAlgorithm + signature_hash_algorithm: algorithm used to hash the payload + + :raises InvalidSignature: If signature verification fails. + :raises errors.Error: If public key type is not supported + """ + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + if isinstance(public_key, RSAPublicKey): + # https://github.com/python/typeshed/blob/master/third_party/2/cryptography/hazmat/primitives/asymmetric/rsa.pyi + verifier = public_key.verifier( # type: ignore + signature, PKCS1v15(), signature_hash_algorithm + ) + verifier.update(payload) + verifier.verify() + elif isinstance(public_key, EllipticCurvePublicKey): + verifier = public_key.verifier( + signature, ECDSA(signature_hash_algorithm) + ) + verifier.update(payload) + verifier.verify() + else: + raise errors.Error("Unsupported public key type") + + def verify_cert_matches_priv_key(cert_path, key_path): """ Verifies that the private key and cert match. @@ -287,16 +302,17 @@ def verify_cert_matches_priv_key(cert_path, key_path): def verify_fullchain(renewable_cert): """ Verifies that fullchain is indeed cert concatenated with chain. - :param `.storage.RenewableCert` renewable_cert: cert to verify + :param renewable_cert: cert to verify + :type renewable_cert: certbot.interfaces.RenewableCert :raises errors.Error: If cert and chain do not combine to fullchain. """ try: - with open(renewable_cert.chain) as chain_file: # type: IO[str] + with open(renewable_cert.chain_path) as chain_file: # type: IO[str] chain = chain_file.read() - with open(renewable_cert.cert) as cert_file: # type: IO[str] + with open(renewable_cert.cert_path) as cert_file: # type: IO[str] cert = cert_file.read() - with open(renewable_cert.fullchain) as fullchain_file: # type: IO[str] + with open(renewable_cert.fullchain_path) as fullchain_file: # type: IO[str] fullchain = fullchain_file.read() if (cert + chain) != fullchain: error_str = "fullchain does not match cert + chain for {0}!" diff --git a/certbot/certbot/display/__init__.py b/certbot/certbot/display/__init__.py new file mode 100644 index 000000000..9d39dce92 --- /dev/null +++ b/certbot/certbot/display/__init__.py @@ -0,0 +1 @@ +"""Certbot display utilities.""" diff --git a/certbot/display/ops.py b/certbot/certbot/display/ops.py similarity index 94% rename from certbot/display/ops.py rename to certbot/certbot/display/ops.py index 3dae1070b..eab9d251d 100644 --- a/certbot/display/ops.py +++ b/certbot/certbot/display/ops.py @@ -1,14 +1,13 @@ """Contains UI methods for LE user operations.""" import logging -import os import zope.component -from certbot import compat from certbot import errors from certbot import interfaces from certbot import util - +from certbot.compat import misc +from certbot.compat import os from certbot.display import util as display_util logger = logging.getLogger(__name__) @@ -36,7 +35,7 @@ def get_email(invalid=False, optional=True): "the client with --register-unsafely-without-email " "but make sure you then backup your account key from " "{0}\n\n".format(os.path.join( - compat.get_default_folder('config'), 'accounts'))) + misc.get_default_folder('config'), 'accounts'))) if optional: if invalid: msg += unsafe_suggestion @@ -61,11 +60,10 @@ def get_email(invalid=False, optional=True): raise errors.Error( "An e-mail address or " "--register-unsafely-without-email must be provided.") - else: - raise errors.Error("An e-mail address must be provided.") - elif util.safe_email(email): + raise errors.Error("An e-mail address must be provided.") + if util.safe_email(email): return email - elif suggest_unsafe: + if suggest_unsafe: msg += unsafe_suggestion suggest_unsafe = False # add this message at most once @@ -76,7 +74,7 @@ def choose_account(accounts): """Choose an account. :param list accounts: Containing at least one - :class:`~certbot.account.Account` + :class:`~certbot._internal.account.Account` """ # Note this will get more complicated once we start recording authorizations @@ -86,8 +84,7 @@ def choose_account(accounts): "Please choose an account", labels, force_interactive=True) if code == display_util.OK: return accounts[index] - else: - return None + return None def choose_values(values, question=None): """Display screen to let user pick one or multiple values from the provided @@ -102,8 +99,7 @@ def choose_values(values, question=None): question, tags=values, force_interactive=True) if code == display_util.OK and items: return items - else: - return [] + return [] def choose_names(installer, question=None): """Display screen to select domains to validate. @@ -132,8 +128,7 @@ def choose_names(installer, question=None): code, names = _filter_names(names, question) if code == display_util.OK and names: return names - else: - return [] + return [] def get_valid_domains(domains): @@ -217,7 +212,7 @@ def _choose_names_manually(prompt_prefix=""): except errors.ConfigurationError as e: invalid_domains[domain] = str(e) - if len(invalid_domains): + if invalid_domains: retry_message = ( "One or more of the entered domain names was not valid:" "{0}{0}").format(os.linesep) @@ -296,7 +291,7 @@ def _gen_ssl_lab_urls(domains): def _gen_https_names(domains): """Returns a string of the https domains. - Domains are formatted nicely with https:// prepended to each. + Domains are formatted nicely with ``https://`` prepended to each. :param list domains: Each domain is a 'str' @@ -345,7 +340,7 @@ def validated_input(validator, *args, **kwargs): """Like `~certbot.interfaces.IDisplay.input`, but with validation. :param callable validator: A method which will be called on the - supplied input. If the method raises a `errors.Error`, its + supplied input. If the method raises an `errors.Error`, its text will be displayed and the user will be re-prompted. :param list `*args`: Arguments to be passed to `~certbot.interfaces.IDisplay.input`. :param dict `**kwargs`: Arguments to be passed to `~certbot.interfaces.IDisplay.input`. @@ -360,7 +355,7 @@ def validated_directory(validator, *args, **kwargs): """Like `~certbot.interfaces.IDisplay.directory_select`, but with validation. :param callable validator: A method which will be called on the - supplied input. If the method raises a `errors.Error`, its + supplied input. If the method raises an `errors.Error`, its text will be displayed and the user will be re-prompted. :param list `*args`: Arguments to be passed to `~certbot.interfaces.IDisplay.directory_select`. :param dict `**kwargs`: Arguments to be passed to diff --git a/certbot/display/util.py b/certbot/certbot/display/util.py similarity index 94% rename from certbot/display/util.py rename to certbot/certbot/display/util.py index 772b67d74..ba2dd4ecf 100644 --- a/certbot/display/util.py +++ b/certbot/certbot/display/util.py @@ -1,16 +1,16 @@ """Certbot display.""" import logging -import os import sys import textwrap import zope.interface -from certbot import compat -from certbot import constants -from certbot import interfaces from certbot import errors -from certbot.display import completer +from certbot import interfaces +from certbot._internal import constants +from certbot._internal.display import completer +from certbot.compat import misc +from certbot.compat import os logger = logging.getLogger(__name__) @@ -79,7 +79,7 @@ def input_with_timeout(prompt=None, timeout=36000.0): sys.stdout.write(prompt) sys.stdout.flush() - line = compat.readline_with_timeout(timeout, prompt) + line = misc.readline_with_timeout(timeout, prompt) if not line: raise EOFError @@ -89,7 +89,6 @@ def input_with_timeout(prompt=None, timeout=36000.0): @zope.interface.implementer(interfaces.IDisplay) class FileDisplay(object): """File-based display.""" - # pylint: disable=too-many-arguments # see https://github.com/certbot/certbot/issues/3915 def __init__(self, outfile, force_interactive): @@ -114,7 +113,7 @@ class FileDisplay(object): message = _wrap_lines(message) self.outfile.write( "{line}{frame}{line}{msg}{line}{frame}{line}".format( - line=os.linesep, frame=SIDE_FRAME, msg=message)) + line='\n', frame=SIDE_FRAME, msg=message)) self.outfile.flush() if pause: if self._can_interact(force_interactive): @@ -122,10 +121,9 @@ class FileDisplay(object): else: logger.debug("Not pausing for user confirmation") - def menu(self, message, choices, ok_label=None, cancel_label=None, - help_label=None, default=None, + def menu(self, message, choices, ok_label=None, cancel_label=None, # pylint: disable=unused-argument + help_label=None, default=None, # pylint: disable=unused-argument cli_flag=None, force_interactive=False, **unused_kwargs): - # pylint: disable=unused-argument """Display a menu. .. todo:: This doesn't enable the help label/button (I wasn't sold on @@ -179,10 +177,9 @@ class FileDisplay(object): message = _wrap_lines("%s (Enter 'c' to cancel):" % message) + " " ans = input_with_timeout(message) - if ans == "c" or ans == "C": + if ans in ("c", "C"): return CANCEL, "-1" - else: - return OK, ans + return OK, ans def yesno(self, message, yes_label="Yes", no_label="No", default=None, cli_flag=None, force_interactive=False, **unused_kwargs): @@ -228,7 +225,6 @@ class FileDisplay(object): def checklist(self, message, tags, default=None, cli_flag=None, force_interactive=False, **unused_kwargs): - # pylint: disable=unused-argument """Display a checklist. :param str message: Message to display to user @@ -256,16 +252,15 @@ class FileDisplay(object): force_interactive=True) if code == OK: - if len(ans.strip()) == 0: + if not ans.strip(): ans = " ".join(str(x) for x in range(1, len(tags)+1)) indices = separate_list_input(ans) selected_tags = self._scrub_checklist_input(indices, tags) if selected_tags: return code, selected_tags - else: - self.outfile.write( - "** Error - Invalid selection **%s" % os.linesep) - self.outfile.flush() + self.outfile.write( + "** Error - Invalid selection **%s" % os.linesep) + self.outfile.flush() else: return code, [] @@ -286,18 +281,17 @@ class FileDisplay(object): # assert_valid_call(prompt, default, cli_flag, force_interactive) if self._can_interact(force_interactive): return False - elif default is None: + if default is None: msg = "Unable to get an answer for the question:\n{0}".format(prompt) if cli_flag: msg += ( "\nYou can provide an answer on the " "command line with the {0} flag.".format(cli_flag)) raise errors.Error(msg) - else: - logger.debug( - "Falling back to default %s for the prompt:\n%s", - default, prompt) - return True + logger.debug( + "Falling back to default %s for the prompt:\n%s", + default, prompt) + return True def _can_interact(self, force_interactive): """Can we safely interact with the user? @@ -312,7 +306,7 @@ class FileDisplay(object): if (self.force_interactive or force_interactive or sys.stdin.isatty() and self.outfile.isatty()): return True - elif not self.skipped_interaction: + if not self.skipped_interaction: logger.warning( "Skipped user interaction because Certbot doesn't appear to " "be running in a terminal. You should probably include " @@ -468,8 +462,7 @@ class NoninteractiveDisplay(object): msg += "\n\n(You can set this with the {0} flag)".format(cli_flag) raise errors.MissingCommandlineFlag(msg) - def notification(self, message, pause=False, wrap=True, **unused_kwargs): - # pylint: disable=unused-argument + def notification(self, message, pause=False, wrap=True, **unused_kwargs): # pylint: disable=unused-argument """Displays a notification without waiting for user acceptance. :param str message: Message to display to stdout @@ -486,7 +479,7 @@ class NoninteractiveDisplay(object): def menu(self, message, choices, ok_label=None, cancel_label=None, help_label=None, default=None, cli_flag=None, **unused_kwargs): - # pylint: disable=unused-argument,too-many-arguments + # pylint: disable=unused-argument """Avoid displaying a menu. :param str message: title of menu @@ -522,12 +515,10 @@ class NoninteractiveDisplay(object): """ if default is None: self._interaction_fail(message, cli_flag) - else: - return OK, default + return OK, default - def yesno(self, message, yes_label=None, no_label=None, + def yesno(self, message, yes_label=None, no_label=None, # pylint: disable=unused-argument default=None, cli_flag=None, **unused_kwargs): - # pylint: disable=unused-argument """Decide Yes or No, without asking anybody :param str message: question for the user @@ -540,8 +531,7 @@ class NoninteractiveDisplay(object): """ if default is None: self._interaction_fail(message, cli_flag) - else: - return default + return default def checklist(self, message, tags, default=None, cli_flag=None, **unused_kwargs): @@ -559,8 +549,7 @@ class NoninteractiveDisplay(object): """ if default is None: self._interaction_fail(message, cli_flag, "? ".join(tags)) - else: - return OK, default + return OK, default def directory_select(self, message, default=None, cli_flag=None, **unused_kwargs): diff --git a/certbot/errors.py b/certbot/certbot/errors.py similarity index 100% rename from certbot/errors.py rename to certbot/certbot/errors.py diff --git a/certbot/interfaces.py b/certbot/certbot/interfaces.py similarity index 92% rename from certbot/interfaces.py rename to certbot/certbot/interfaces.py index bb9a91b0f..e96712d23 100644 --- a/certbot/interfaces.py +++ b/certbot/certbot/interfaces.py @@ -1,10 +1,10 @@ """Certbot client interfaces.""" import abc + import six import zope.interface # pylint: disable=no-self-argument,no-method-argument,no-init,inherit-non-class -# pylint: disable=too-few-public-methods @six.add_metaclass(abc.ABCMeta) @@ -73,7 +73,7 @@ class IPluginFactory(zope.interface.Interface): description = zope.interface.Attribute("Short plugin description") - def __call__(config, name): + def __call__(config, name): # pylint: disable=signature-differs """Create new `IPlugin`. :param IConfig config: Configuration. @@ -159,21 +159,14 @@ class IAuthenticator(IPlugin): :func:`get_chall_pref` only. :returns: `collections.Iterable` of ACME - :class:`~acme.challenges.ChallengeResponse` instances - or if the :class:`~acme.challenges.Challenge` cannot - be fulfilled then: - - ``None`` - Authenticator can perform challenge, but not at this time. - ``False`` - Authenticator will never be able to perform (error). - + :class:`~acme.challenges.ChallengeResponse` instances corresponding to each provided + :class:`~acme.challenges.Challenge`. :rtype: :class:`collections.Iterable` of :class:`acme.challenges.ChallengeResponse`, where responses are required to be returned in the same order as corresponding input challenges - :raises .PluginError: If challenges cannot be performed + :raises .PluginError: If some or all challenges cannot be performed """ @@ -227,12 +220,6 @@ class IConfig(zope.interface.Interface): no_verify_ssl = zope.interface.Attribute( "Disable verification of the ACME server's certificate.") - tls_sni_01_port = zope.interface.Attribute( - "Port used during tls-sni-01 challenge. " - "This only affects the port Certbot listens on. " - "A conforming ACME server will still attempt to connect on port 443.") - tls_sni_01_address = zope.interface.Attribute( - "The address the server listens to during tls-sni-01 challenge.") http01_port = zope.interface.Attribute( "Port used in the http-01 challenge. " @@ -242,6 +229,11 @@ class IConfig(zope.interface.Interface): http01_address = zope.interface.Attribute( "The address the server listens to during http-01 challenge.") + https_port = zope.interface.Attribute( + "Port used to serve HTTPS. " + "This affects which port Nginx will listen on after a LE certificate " + "is installed.") + pref_challs = zope.interface.Attribute( "Sorted user specified preferred challenges" "type strings with the most preferred challenge listed first") @@ -303,10 +295,10 @@ class IInstaller(IPlugin): :param str domain: domain for which to provide enhancement :param str enhancement: An enhancement as defined in - :const:`~certbot.constants.ENHANCEMENTS` + :const:`~certbot.plugins.enhancements.ENHANCEMENTS` :param options: Flexible options parameter for enhancement. Check documentation of - :const:`~certbot.constants.ENHANCEMENTS` + :const:`~certbot.plugins.enhancements.ENHANCEMENTS` for expected options for each enhancement. :raises .PluginError: If Enhancement is not supported, or if @@ -318,7 +310,7 @@ class IInstaller(IPlugin): """Returns a `collections.Iterable` of supported enhancements. :returns: supported enhancements which should be a subset of - :const:`~certbot.constants.ENHANCEMENTS` + :const:`~certbot.plugins.enhancements.ENHANCEMENTS` :rtype: :class:`collections.Iterable` of :class:`str` """ @@ -363,13 +355,6 @@ class IInstaller(IPlugin): """ - def view_config_changes(): # type: ignore - """Display all of the LE config changes. - - :raises .PluginError: when config changes cannot be parsed - - """ - def config_test(): # type: ignore """Make sure the configuration is valid. @@ -387,7 +372,6 @@ class IInstaller(IPlugin): class IDisplay(zope.interface.Interface): """Generic display.""" - # pylint: disable=too-many-arguments # see https://github.com/certbot/certbot/issues/3915 def notification(message, pause, wrap=True, force_interactive=False): @@ -549,6 +533,62 @@ class IReporter(zope.interface.Interface): """Prints messages to the user and clears the message queue.""" +@six.add_metaclass(abc.ABCMeta) +class RenewableCert(object): + """Interface to a certificate lineage.""" + + @abc.abstractproperty + def cert_path(self): + """Path to the certificate file. + + :rtype: str + + """ + + @abc.abstractproperty + def key_path(self): + """Path to the private key file. + + :rtype: str + + """ + + @abc.abstractproperty + def chain_path(self): + """Path to the certificate chain file. + + :rtype: str + + """ + + @abc.abstractproperty + def fullchain_path(self): + """Path to the full chain file. + + The full chain is the certificate file plus the chain file. + + :rtype: str + + """ + + @abc.abstractproperty + def lineagename(self): + """Name given to the certificate lineage. + + :rtype: str + + """ + + @abc.abstractmethod + def names(self): + """What are the subject names of this certificate? + + :returns: the subject names + :rtype: `list` of `str` + :raises .CertStorageError: if could not find cert file. + + """ + # Updater interfaces # # When "certbot renew" is run, Certbot will iterate over each lineage and check @@ -587,7 +627,7 @@ class GenericUpdater(object): This method is called once for each lineage. :param lineage: Certificate lineage object - :type lineage: storage.RenewableCert + :type lineage: RenewableCert """ @@ -616,6 +656,6 @@ class RenewDeployer(object): This method is called once for each lineage renewed :param lineage: Certificate lineage object - :type lineage: storage.RenewableCert + :type lineage: RenewableCert """ diff --git a/certbot/certbot/main.py b/certbot/certbot/main.py new file mode 100644 index 000000000..b2fb1dbb7 --- /dev/null +++ b/certbot/certbot/main.py @@ -0,0 +1,15 @@ +"""Certbot main public entry point.""" +from certbot._internal import main as internal_main + + +def main(cli_args=None): + """Run Certbot. + + :param cli_args: command line to Certbot, defaults to ``sys.argv[1:]`` + :type cli_args: `list` of `str` + + :returns: value for `sys.exit` about the exit status of Certbot + :rtype: `str` or `int` or `None` + + """ + return internal_main.main(cli_args) diff --git a/certbot/certbot/plugins/__init__.py b/certbot/certbot/plugins/__init__.py new file mode 100644 index 000000000..7831eab61 --- /dev/null +++ b/certbot/certbot/plugins/__init__.py @@ -0,0 +1 @@ +"""Certbot plugins.""" diff --git a/certbot/plugins/common.py b/certbot/certbot/plugins/common.py similarity index 81% rename from certbot/plugins/common.py rename to certbot/certbot/plugins/common.py index ee1af4978..6fa1e76f8 100644 --- a/certbot/plugins/common.py +++ b/certbot/certbot/plugins/common.py @@ -1,25 +1,24 @@ """Plugin common functions.""" import logging -import os import re import shutil +import sys import tempfile +import warnings -import OpenSSL +from josepy import util as jose_util import pkg_resources import zope.interface -from josepy import util as jose_util - from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module from certbot import achallenges # pylint: disable=unused-import -from certbot import constants from certbot import crypto_util from certbot import errors from certbot import interfaces from certbot import reverter -from certbot import util - +from certbot._internal import constants +from certbot.compat import filesystem +from certbot.compat import os from certbot.plugins.storage import PluginStorage logger = logging.getLogger(__name__) @@ -34,6 +33,7 @@ def dest_namespace(name): """ArgumentParser dest namespace (prefix of all destinations).""" return name.replace("-", "_") + "_" + private_ips_regex = re.compile( r"(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|" r"(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)") @@ -189,18 +189,6 @@ class Installer(Plugin): except errors.ReverterError as err: raise errors.PluginError(str(err)) - def view_config_changes(self): - """Show all of the configuration changes that have taken place. - - :raises .errors.PluginError: If there is a problem while processing - the checkpoints directories. - - """ - try: - self.reverter.view_config_changes() - except errors.ReverterError as err: - raise errors.PluginError(str(err)) - @property def ssl_dhparams(self): """Full absolute path to ssl_dhparams file.""" @@ -301,14 +289,13 @@ class Addr(object): # too long, truncate addr_list = addr_list[0:len(result)] append_to_end = False - for i in range(0, len(addr_list)): - block = addr_list[i] - if len(block) == 0: + for i, block in enumerate(addr_list): + if not block: # encountered ::, so rest of the blocks should be # appended to the end append_to_end = True continue - elif len(block) > 1: + if len(block) > 1: # remove leading zeros block = block.lstrip("0") if not append_to_end: @@ -351,7 +338,7 @@ class ChallengePerformer(object): def perform(self): """Perform all added challenges. - :returns: challenge respones + :returns: challenge responses :rtype: `list` of `acme.challenges.KeyAuthorizationChallengeResponse` @@ -359,63 +346,6 @@ class ChallengePerformer(object): raise NotImplementedError() -class TLSSNI01(ChallengePerformer): - # pylint: disable=abstract-method - """Abstract base for TLS-SNI-01 challenge performers""" - - def __init__(self, configurator): - super(TLSSNI01, self).__init__(configurator) - self.challenge_conf = os.path.join( - configurator.config.config_dir, "le_tls_sni_01_cert_challenge.conf") - # self.completed = 0 - - def get_cert_path(self, achall): - """Returns standardized name for challenge certificate. - - :param .KeyAuthorizationAnnotatedChallenge achall: Annotated - tls-sni-01 challenge. - - :returns: certificate file name - :rtype: str - - """ - return os.path.join(self.configurator.config.work_dir, - achall.chall.encode("token") + ".crt") - - def get_key_path(self, achall): - """Get standardized path to challenge key.""" - return os.path.join(self.configurator.config.work_dir, - achall.chall.encode("token") + '.pem') - - def get_z_domain(self, achall): - """Returns z_domain (SNI) name for the challenge.""" - return achall.response(achall.account_key).z_domain.decode("utf-8") - - def _setup_challenge_cert(self, achall, cert_key=None): - - """Generate and write out challenge certificate.""" - cert_path = self.get_cert_path(achall) - key_path = self.get_key_path(achall) - # Register the path before you write out the file - self.configurator.reverter.register_file_creation(True, key_path) - self.configurator.reverter.register_file_creation(True, cert_path) - - response, (cert, key) = achall.response_and_validation( - cert_key=cert_key) - cert_pem = OpenSSL.crypto.dump_certificate( - OpenSSL.crypto.FILETYPE_PEM, cert) - key_pem = OpenSSL.crypto.dump_privatekey( - OpenSSL.crypto.FILETYPE_PEM, key) - - # Write out challenge cert and key - with open(cert_path, "wb") as cert_chall_fd: - cert_chall_fd.write(cert_pem) - with util.safe_open(key_path, 'wb', chmod=0o400) as key_file: - key_file.write(key_pem) - - return response - - def install_version_controlled_file(dest_path, digest_path, src_path, all_hashes): """Copy a file into an active location (likely the system's config dir) if required. @@ -444,9 +374,9 @@ def install_version_controlled_file(dest_path, digest_path, src_path, all_hashes active_file_digest = crypto_util.sha256sum(dest_path) if active_file_digest == current_hash: # already up to date return - elif active_file_digest in all_hashes: # safe to update + if active_file_digest in all_hashes: # safe to update _install_current_file() - else: # has been manually modified, not safe to update + else: # has been manually modified, not safe to update # did they modify the current version or an old version? if os.path.isfile(digest_path): with open(digest_path, "r") as f: @@ -477,15 +407,15 @@ def dir_setup(test_dir, pkg): # pragma: no cover link, (ex: OS X) such plugins will be confused. This function prevents such a case. """ - return os.path.realpath(tempfile.mkdtemp(prefix)) + return filesystem.realpath(tempfile.mkdtemp(prefix)) temp_dir = expanded_tempdir("temp") config_dir = expanded_tempdir("config") work_dir = expanded_tempdir("work") - os.chmod(temp_dir, constants.CONFIG_DIRS_MODE) - os.chmod(config_dir, constants.CONFIG_DIRS_MODE) - os.chmod(work_dir, constants.CONFIG_DIRS_MODE) + filesystem.chmod(temp_dir, constants.CONFIG_DIRS_MODE) + filesystem.chmod(config_dir, constants.CONFIG_DIRS_MODE) + filesystem.chmod(work_dir, constants.CONFIG_DIRS_MODE) test_configs = pkg_resources.resource_filename( pkg, os.path.join("testdata", test_dir)) @@ -494,3 +424,34 @@ def dir_setup(test_dir, pkg): # pragma: no cover test_configs, os.path.join(temp_dir, test_dir), symlinks=True) return temp_dir, config_dir, work_dir + + +# This class takes a similar approach to the cryptography project to deprecate attributes +# in public modules. See the _ModuleWithDeprecation class here: +# https://github.com/pyca/cryptography/blob/91105952739442a74582d3e62b3d2111365b0dc7/src/cryptography/utils.py#L129 +class _TLSSNI01DeprecationModule(object): + """ + Internal class delegating to a module, and displaying warnings when + attributes related to TLS-SNI-01 are accessed. + """ + def __init__(self, module): + self.__dict__['_module'] = module + + def __getattr__(self, attr): + if attr == 'TLSSNI01': + warnings.warn('TLSSNI01 is deprecated and will be removed soon.', + DeprecationWarning, stacklevel=2) + return getattr(self._module, attr) + + def __setattr__(self, attr, value): # pragma: no cover + setattr(self._module, attr, value) + + def __delattr__(self, attr): # pragma: no cover + delattr(self._module, attr) + + def __dir__(self): # pragma: no cover + return ['_module'] + dir(self._module) + + +# Patching ourselves to warn about TLS-SNI challenge deprecation and removal. +sys.modules[__name__] = _TLSSNI01DeprecationModule(sys.modules[__name__]) diff --git a/certbot/plugins/dns_common.py b/certbot/certbot/plugins/dns_common.py similarity index 94% rename from certbot/plugins/dns_common.py rename to certbot/certbot/plugins/dns_common.py index ba88b7aef..d31266434 100644 --- a/certbot/plugins/dns_common.py +++ b/certbot/certbot/plugins/dns_common.py @@ -2,16 +2,16 @@ import abc import logging -import os -import stat from time import sleep import configobj import zope.interface -from acme import challenges +from acme import challenges from certbot import errors from certbot import interfaces +from certbot.compat import filesystem +from certbot.compat import os from certbot.display import ops from certbot.display import util as display_util from certbot.plugins import common @@ -37,7 +37,7 @@ class DNSAuthenticator(common.Plugin): help='The number of seconds to wait for DNS to propagate before asking the ACME server ' 'to verify the DNS record.') - def get_chall_pref(self, unused_domain): # pylint: disable=missing-docstring,no-self-use + def get_chall_pref(self, unused_domain): # pylint: disable=missing-docstring,no-self-use return [challenges.DNS01] def prepare(self): # pylint: disable=missing-docstring @@ -83,7 +83,7 @@ class DNSAuthenticator(common.Plugin): raise NotImplementedError() @abc.abstractmethod - def _perform(self, domain, validation_domain_name, validation): # pragma: no cover + def _perform(self, domain, validation_name, validation): # pragma: no cover """ Performs a dns-01 challenge by creating a DNS TXT record. @@ -95,7 +95,7 @@ class DNSAuthenticator(common.Plugin): raise NotImplementedError() @abc.abstractmethod - def _cleanup(self, domain, validation_domain_name, validation): # pragma: no cover + def _cleanup(self, domain, validation_name, validation): # pragma: no cover """ Deletes the DNS TXT record which would have been created by `_perform_achall`. @@ -197,8 +197,7 @@ class DNSAuthenticator(common.Plugin): if code == display_util.OK: return response - else: - raise errors.PluginError('{0} required to proceed.'.format(label)) + raise errors.PluginError('{0} required to proceed.'.format(label)) @staticmethod def _prompt_for_file(label, validator=None): @@ -231,8 +230,7 @@ class DNSAuthenticator(common.Plugin): if code == display_util.OK: return response - else: - raise errors.PluginError('{0} required to proceed.'.format(label)) + raise errors.PluginError('{0} required to proceed.'.format(label)) class CredentialsConfiguration(object): @@ -302,8 +300,8 @@ def validate_file(filename): if not os.path.exists(filename): raise errors.PluginError('File not found: {0}'.format(filename)) - if not os.path.isfile(filename): - raise errors.PluginError('Path is not a file: {0}'.format(filename)) + if os.path.isdir(filename): + raise errors.PluginError('Path is a directory: {0}'.format(filename)) def validate_file_permissions(filename): @@ -311,8 +309,7 @@ def validate_file_permissions(filename): validate_file(filename) - permissions = stat.S_IMODE(os.stat(filename).st_mode) - if permissions & stat.S_IRWXO: + if filesystem.has_world_permissions(filename): logger.warning('Unsafe permissions on credentials configuration file: %s', filename) diff --git a/certbot/plugins/dns_common_lexicon.py b/certbot/certbot/plugins/dns_common_lexicon.py similarity index 92% rename from certbot/plugins/dns_common_lexicon.py rename to certbot/certbot/plugins/dns_common_lexicon.py index 5b50cc285..3e28a291b 100644 --- a/certbot/plugins/dns_common_lexicon.py +++ b/certbot/certbot/plugins/dns_common_lexicon.py @@ -1,9 +1,12 @@ """Common code for DNS Authenticator Plugins built on Lexicon.""" import logging -from requests.exceptions import HTTPError, RequestException +from requests.exceptions import HTTPError +from requests.exceptions import RequestException -from acme.magic_typing import Union, Dict, Any # pylint: disable=unused-import,no-name-in-module +from acme.magic_typing import Any # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Union # pylint: disable=unused-import, no-name-in-module from certbot import errors from certbot.plugins import dns_common @@ -97,7 +100,7 @@ class LexiconClient(object): result = self._handle_general_error(e, domain_name) if result: - raise result + raise result # pylint: disable=raising-bad-type raise errors.PluginError('Unable to determine zone identifier for {0} using zone names: {1}' .format(domain, domain_name_guesses)) @@ -110,6 +113,7 @@ class LexiconClient(object): if not str(e).startswith('No domain found'): return errors.PluginError('Unexpected error determining zone identifier for {0}: {1}' .format(domain_name, e)) + return None def build_lexicon_config(lexicon_provider_name, lexicon_options, provider_options): diff --git a/certbot/plugins/dns_test_common.py b/certbot/certbot/plugins/dns_test_common.py similarity index 86% rename from certbot/plugins/dns_test_common.py rename to certbot/certbot/plugins/dns_test_common.py index 54b656b20..9ef76c2c3 100644 --- a/certbot/plugins/dns_test_common.py +++ b/certbot/certbot/plugins/dns_test_common.py @@ -1,14 +1,13 @@ """Base test class for DNS authenticators.""" -import os - import configobj import josepy as jose import mock import six -from acme import challenges +from acme import challenges from certbot import achallenges +from certbot.compat import filesystem from certbot.tests import acme_util from certbot.tests import util as test_util @@ -29,18 +28,14 @@ class BaseAuthenticatorTest(object): challb=acme_util.DNS01, domain=DOMAIN, account_key=KEY) def test_more_info(self): - # pylint: disable=no-member - self.assertTrue(isinstance(self.auth.more_info(), six.string_types)) + self.assertTrue(isinstance(self.auth.more_info(), six.string_types)) # pylint: disable=no-member def test_get_chall_pref(self): - # pylint: disable=no-member - self.assertEqual(self.auth.get_chall_pref(None), [challenges.DNS01]) + self.assertEqual(self.auth.get_chall_pref(None), [challenges.DNS01]) # pylint: disable=no-member def test_parser_arguments(self): m = mock.MagicMock() - - # pylint: disable=no-member - self.auth.add_parser_arguments(m) + self.auth.add_parser_arguments(m) # pylint: disable=no-member m.assert_any_call('propagation-seconds', type=int, default=mock.ANY, help=mock.ANY) @@ -60,4 +55,4 @@ def write(values, path): with open(path, "wb") as f: config.write(outfile=f) - os.chmod(path, 0o600) + filesystem.chmod(path, 0o600) diff --git a/certbot/plugins/dns_test_common_lexicon.py b/certbot/certbot/plugins/dns_test_common_lexicon.py similarity index 98% rename from certbot/plugins/dns_test_common_lexicon.py rename to certbot/certbot/plugins/dns_test_common_lexicon.py index a221cf1bf..c77d6da9e 100644 --- a/certbot/plugins/dns_test_common_lexicon.py +++ b/certbot/certbot/plugins/dns_test_common_lexicon.py @@ -2,7 +2,8 @@ import josepy as jose import mock -from requests.exceptions import HTTPError, RequestException +from requests.exceptions import HTTPError +from requests.exceptions import RequestException from certbot import errors from certbot.plugins import dns_test_common diff --git a/certbot/plugins/enhancements.py b/certbot/certbot/plugins/enhancements.py similarity index 84% rename from certbot/plugins/enhancements.py rename to certbot/certbot/plugins/enhancements.py index 7ca096610..f8d9db7dc 100644 --- a/certbot/plugins/enhancements.py +++ b/certbot/certbot/plugins/enhancements.py @@ -1,10 +1,23 @@ """New interface style Certbot enhancements""" import abc + import six -from certbot import constants +from acme.magic_typing import Any # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from certbot._internal import constants -from acme.magic_typing import Dict, List, Any # pylint: disable=unused-import, no-name-in-module +ENHANCEMENTS = ["redirect", "ensure-http-header", "ocsp-stapling"] +"""List of possible :class:`certbot.interfaces.IInstaller` +enhancements. + +List of expected options parameters: +- redirect: None +- ensure-http-header: name of header (i.e. Strict-Transport-Security) +- ocsp-stapling: certificate chain file path + +""" def enabled_enhancements(config): """ @@ -51,7 +64,7 @@ def enable(lineage, domains, installer, config): Run enable method for each requested enhancement that is supported. :param lineage: Certificate lineage object - :type lineage: certbot.storage.RenewableCert + :type lineage: certbot.interfaces.RenewableCert :param domains: List of domains in certificate to enhance :type domains: str @@ -67,9 +80,9 @@ def enable(lineage, domains, installer, config): def populate_cli(add): """ - Populates the command line flags for certbot.cli.HelpfulParser + Populates the command line flags for certbot._internal.cli.HelpfulParser - :param add: Add function of certbot.cli.HelpfulParser + :param add: Add function of certbot._internal.cli.HelpfulParser :type add: func """ for enh in _INDEX: @@ -112,7 +125,7 @@ class AutoHSTSEnhancement(object): Implementation of this method should increase the max-age value. :param lineage: Certificate lineage object - :type lineage: certbot.storage.RenewableCert + :type lineage: certbot.interfaces.RenewableCert .. note:: prepare() method inherited from `interfaces.IPlugin` might need to be called manually within implementation of this interface method @@ -126,7 +139,7 @@ class AutoHSTSEnhancement(object): Long max-age value should be set in implementation of this method. :param lineage: Certificate lineage object - :type lineage: certbot.storage.RenewableCert + :type lineage: certbot.interfaces.RenewableCert """ @abc.abstractmethod @@ -137,7 +150,7 @@ class AutoHSTSEnhancement(object): over the subsequent runs of Certbot renew. :param lineage: Certificate lineage object - :type lineage: certbot.storage.RenewableCert + :type lineage: certbot.interfaces.RenewableCert :param domains: List of domains in certificate to enhance :type domains: str diff --git a/certbot/plugins/storage.py b/certbot/certbot/plugins/storage.py similarity index 90% rename from certbot/plugins/storage.py rename to certbot/certbot/plugins/storage.py index ae3ca1889..7956295d2 100644 --- a/certbot/plugins/storage.py +++ b/certbot/certbot/plugins/storage.py @@ -1,13 +1,16 @@ """Plugin storage class.""" import json import logging -import os -from acme.magic_typing import Any, Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Any # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module from certbot import errors +from certbot.compat import filesystem +from certbot.compat import os logger = logging.getLogger(__name__) + class PluginStorage(object): """Class implementing storage functionality for plugins""" @@ -83,9 +86,10 @@ class PluginStorage(object): logger.error(errmsg) raise errors.PluginStorageError(errmsg) try: - with os.fdopen(os.open(self._storagepath, - os.O_WRONLY | os.O_CREAT | os.O_TRUNC, - 0o600), 'w') as fh: + with os.fdopen(filesystem.open( + self._storagepath, + os.O_WRONLY | os.O_CREAT | os.O_TRUNC, + 0o600), 'w') as fh: fh.write(serialized) except IOError as e: errmsg = "Could not write PluginStorage data to file {0} : {1}".format( diff --git a/certbot/plugins/util.py b/certbot/certbot/plugins/util.py similarity index 82% rename from certbot/plugins/util.py rename to certbot/certbot/plugins/util.py index 5c682c3ff..87eb45fe9 100644 --- a/certbot/plugins/util.py +++ b/certbot/certbot/plugins/util.py @@ -1,11 +1,13 @@ """Plugin utilities.""" import logging -import os from certbot import util +from certbot.compat import os +from certbot.compat.misc import STANDARD_BINARY_DIRS logger = logging.getLogger(__name__) + def get_prefixes(path): """Retrieves all possible path prefixes of a path, in descending order of length. For instance, @@ -18,7 +20,7 @@ def get_prefixes(path): """ prefix = os.path.normpath(path) prefixes = [] - while len(prefix) > 0: + while prefix: prefixes.append(prefix) prefix, _ = os.path.split(prefix) # break once we hit the root path @@ -26,6 +28,7 @@ def get_prefixes(path): break return prefixes + def path_surgery(cmd): """Attempt to perform PATH surgery to find cmd @@ -35,10 +38,9 @@ def path_surgery(cmd): :returns: True if the operation succeeded, False otherwise """ - dirs = ("/usr/sbin", "/usr/local/bin", "/usr/local/sbin") path = os.environ["PATH"] added = [] - for d in dirs: + for d in STANDARD_BINARY_DIRS: if d not in path: path += os.pathsep + d added.append(d) @@ -50,8 +52,7 @@ def path_surgery(cmd): if util.exe_exists(cmd): return True - else: - expanded = " expanded" if any(added) else "" - logger.debug("Failed to find executable %s in%s PATH: %s", cmd, - expanded, path) - return False + expanded = " expanded" if any(added) else "" + logger.debug("Failed to find executable %s in%s PATH: %s", cmd, + expanded, path) + return False diff --git a/certbot/reverter.py b/certbot/certbot/reverter.py similarity index 86% rename from certbot/reverter.py rename to certbot/certbot/reverter.py index 919037358..47a77c80a 100644 --- a/certbot/reverter.py +++ b/certbot/certbot/reverter.py @@ -2,20 +2,18 @@ import csv import glob import logging -import os import shutil +import sys import time import traceback import six -import zope.component -from certbot import compat -from certbot import constants from certbot import errors -from certbot import interfaces from certbot import util - +from certbot._internal import constants +from certbot.compat import filesystem +from certbot.compat import os logger = logging.getLogger(__name__) @@ -66,8 +64,7 @@ class Reverter(object): self.config = config util.make_or_verify_dir( - config.backup_dir, constants.CONFIG_DIRS_MODE, compat.os_geteuid(), - self.config.strict_permissions) + config.backup_dir, constants.CONFIG_DIRS_MODE, self.config.strict_permissions) def revert_temporary_config(self): """Reload users original configuration files after a temporary save. @@ -131,61 +128,6 @@ class Reverter(object): "Unable to load checkpoint during rollback") rollback -= 1 - def view_config_changes(self, for_logging=False, num=None): - """Displays all saved checkpoints. - - All checkpoints are printed by - :meth:`certbot.interfaces.IDisplay.notification`. - - .. todo:: Decide on a policy for error handling, OSError IOError... - - :raises .errors.ReverterError: If invalid directory structure. - - """ - backups = os.listdir(self.config.backup_dir) - backups.sort(reverse=True) - if num: - backups = backups[:num] - if not backups: - logger.info("Certbot has not saved backups of your configuration") - - return - # Make sure there isn't anything unexpected in the backup folder - # There should only be timestamped (float) directories - try: - for bkup in backups: - float(bkup) - except ValueError: - raise errors.ReverterError( - "Invalid directories in {0}".format(self.config.backup_dir)) - - output = [] - for bkup in backups: - output.append(time.ctime(float(bkup))) - cur_dir = os.path.join(self.config.backup_dir, bkup) - with open(os.path.join(cur_dir, "CHANGES_SINCE")) as changes_fd: - output.append(changes_fd.read()) - - output.append("Affected files:") - with open(os.path.join(cur_dir, "FILEPATHS")) as paths_fd: - filepaths = paths_fd.read().splitlines() - for path in filepaths: - output.append(" {0}".format(path)) - - if os.path.isfile(os.path.join(cur_dir, "NEW_FILES")): - with open(os.path.join(cur_dir, "NEW_FILES")) as new_fd: - output.append("New Configuration Files:") - filepaths = new_fd.read().splitlines() - for path in filepaths: - output.append(" {0}".format(path)) - - output.append(os.linesep) - - if for_logging: - return os.linesep.join(output) - zope.component.getUtility(interfaces.IDisplay).notification( - os.linesep.join(output), force_interactive=True, pause=False) - def add_to_temp_checkpoint(self, save_files, save_notes): """Add files to temporary checkpoint. @@ -220,8 +162,7 @@ class Reverter(object): """ util.make_or_verify_dir( - cp_dir, constants.CONFIG_DIRS_MODE, compat.os_geteuid(), - self.config.strict_permissions) + cp_dir, constants.CONFIG_DIRS_MODE, self.config.strict_permissions) op_fd, existing_filepaths = self._read_and_append( os.path.join(cp_dir, "FILEPATHS")) @@ -238,7 +179,7 @@ class Reverter(object): try: shutil.copy2(filename, os.path.join( cp_dir, os.path.basename(filename) + "_" + str(idx))) - op_fd.write(filename + os.linesep) + op_fd.write('{0}\n'.format(filename)) # http://stackoverflow.com/questions/4726260/effective-use-of-python-shutil-copy2 except IOError: op_fd.close() @@ -313,7 +254,10 @@ class Reverter(object): """Run all commands in a file.""" # NOTE: csv module uses native strings. That is, bytes on Python 2 and # unicode on Python 3 - with open(filepath, 'r') as csvfile: + # It is strongly advised to set newline = '' on Python 3 with CSV, + # and it fixes problems on Windows. + kwargs = {'newline': ''} if sys.version_info[0] > 2 else {} + with open(filepath, 'r', **kwargs) as csvfile: # type: ignore csvreader = csv.reader(csvfile) for command in reversed(list(csvreader)): try: @@ -382,7 +326,7 @@ class Reverter(object): for path in files: if path not in ex_files: - new_fd.write("{0}{1}".format(path, os.linesep)) + new_fd.write("{0}\n".format(path)) except (IOError, OSError): logger.error("Unable to register file creation(s) - %s", files) raise errors.ReverterError( @@ -409,11 +353,14 @@ class Reverter(object): """ commands_fp = os.path.join(self._get_cp_dir(temporary), "COMMANDS") command_file = None + # It is strongly advised to set newline = '' on Python 3 with CSV, + # and it fixes problems on Windows. + kwargs = {'newline': ''} if sys.version_info[0] > 2 else {} try: if os.path.isfile(commands_fp): - command_file = open(commands_fp, "a") + command_file = open(commands_fp, "a", **kwargs) # type: ignore else: - command_file = open(commands_fp, "w") + command_file = open(commands_fp, "w", **kwargs) # type: ignore csvwriter = csv.writer(command_file) csvwriter.writerow(command) @@ -434,8 +381,7 @@ class Reverter(object): cp_dir = self.config.in_progress_dir util.make_or_verify_dir( - cp_dir, constants.CONFIG_DIRS_MODE, compat.os_geteuid(), - self.config.strict_permissions) + cp_dir, constants.CONFIG_DIRS_MODE, self.config.strict_permissions) return cp_dir @@ -493,9 +439,9 @@ class Reverter(object): os.remove(path) else: logger.warning( - "File: %s - Could not be found to be deleted %s - " - "Certbot probably shut down unexpectedly", - os.linesep, path) + "File: %s - Could not be found to be deleted\n" + " - Certbot probably shut down unexpectedly", + path) except (IOError, OSError): logger.critical( "Unable to remove filepaths contained within %s", file_list) @@ -576,7 +522,7 @@ class Reverter(object): timestamp = self._checkpoint_timestamp() final_dir = os.path.join(self.config.backup_dir, timestamp) try: - compat.os_rename(self.config.in_progress_dir, final_dir) + filesystem.replace(self.config.in_progress_dir, final_dir) return except OSError: logger.warning("Extreme, unexpected race condition, retrying (%s)", timestamp) diff --git a/certbot/ssl-dhparams.pem b/certbot/certbot/ssl-dhparams.pem similarity index 100% rename from certbot/ssl-dhparams.pem rename to certbot/certbot/ssl-dhparams.pem diff --git a/certbot/certbot/tests/__init__.py b/certbot/certbot/tests/__init__.py new file mode 100644 index 000000000..82290ca0b --- /dev/null +++ b/certbot/certbot/tests/__init__.py @@ -0,0 +1 @@ +"""Utilities for running Certbot tests""" diff --git a/certbot/tests/acme_util.py b/certbot/certbot/tests/acme_util.py similarity index 84% rename from certbot/tests/acme_util.py rename to certbot/certbot/tests/acme_util.py index 2f9445694..3d560dcbc 100644 --- a/certbot/tests/acme_util.py +++ b/certbot/certbot/tests/acme_util.py @@ -6,24 +6,19 @@ import six from acme import challenges from acme import messages - -from certbot import auth_handler - +from certbot._internal import auth_handler from certbot.tests import util - JWK = jose.JWK.load(util.load_vector('rsa512_key.pem')) KEY = util.load_rsa_private_key('rsa512_key.pem') # Challenges HTTP01 = challenges.HTTP01( token=b"evaGxfADs6pSRb2LAv9IZf17Dt3juxGJ+PCt92wr+oA") -TLSSNI01 = challenges.TLSSNI01( - token=jose.b64decode(b"evaGxfADs6pSRb2LAv9IZf17Dt3juxGJyPCt92wrDoA")) DNS01 = challenges.DNS01(token=b"17817c66b60ce2e4012dfad92657527a") DNS01_2 = challenges.DNS01(token=b"cafecafecafecafecafecafe0feedbac") -CHALLENGES = [HTTP01, TLSSNI01, DNS01] +CHALLENGES = [HTTP01, DNS01] def gen_combos(challbs): @@ -43,25 +38,23 @@ def chall_to_challb(chall, status): # pylint: disable=redefined-outer-name if status == messages.STATUS_VALID: kwargs.update({"validated": datetime.datetime.now()}) - return messages.ChallengeBody(**kwargs) # pylint: disable=star-args + return messages.ChallengeBody(**kwargs) # Pending ChallengeBody objects -TLSSNI01_P = chall_to_challb(TLSSNI01, messages.STATUS_PENDING) HTTP01_P = chall_to_challb(HTTP01, messages.STATUS_PENDING) DNS01_P = chall_to_challb(DNS01, messages.STATUS_PENDING) DNS01_P_2 = chall_to_challb(DNS01_2, messages.STATUS_PENDING) -CHALLENGES_P = [HTTP01_P, TLSSNI01_P, DNS01_P] +CHALLENGES_P = [HTTP01_P, DNS01_P] # AnnotatedChallenge objects HTTP01_A = auth_handler.challb_to_achall(HTTP01_P, JWK, "example.com") -TLSSNI01_A = auth_handler.challb_to_achall(TLSSNI01_P, JWK, "example.net") DNS01_A = auth_handler.challb_to_achall(DNS01_P, JWK, "example.org") DNS01_A_2 = auth_handler.challb_to_achall(DNS01_P_2, JWK, "esimerkki.example.org") -ACHALLENGES = [HTTP01_A, TLSSNI01_A, DNS01_A] +ACHALLENGES = [HTTP01_A, DNS01_A] def gen_authzr(authz_status, domain, challs, statuses, combos=True): @@ -96,7 +89,6 @@ def gen_authzr(authz_status, domain, challs, statuses, combos=True): "status": authz_status, }) - # pylint: disable=star-args return messages.AuthorizationResource( uri="https://trusted.ca/new-authz-resource", body=messages.Authorization(**authz_kwargs) diff --git a/certbot/tests/testdata/README b/certbot/certbot/tests/testdata/README similarity index 100% rename from certbot/tests/testdata/README rename to certbot/certbot/tests/testdata/README diff --git a/certbot/tests/testdata/cert-5sans_512.pem b/certbot/certbot/tests/testdata/cert-5sans_512.pem similarity index 100% rename from certbot/tests/testdata/cert-5sans_512.pem rename to certbot/certbot/tests/testdata/cert-5sans_512.pem diff --git a/certbot/tests/testdata/cert-nosans_nistp256.pem b/certbot/certbot/tests/testdata/cert-nosans_nistp256.pem similarity index 100% rename from certbot/tests/testdata/cert-nosans_nistp256.pem rename to certbot/certbot/tests/testdata/cert-nosans_nistp256.pem diff --git a/certbot/tests/testdata/cert-san_512.pem b/certbot/certbot/tests/testdata/cert-san_512.pem similarity index 100% rename from certbot/tests/testdata/cert-san_512.pem rename to certbot/certbot/tests/testdata/cert-san_512.pem diff --git a/certbot/tests/testdata/cert_2048.pem b/certbot/certbot/tests/testdata/cert_2048.pem similarity index 100% rename from certbot/tests/testdata/cert_2048.pem rename to certbot/certbot/tests/testdata/cert_2048.pem diff --git a/certbot/tests/testdata/cert_512.pem b/certbot/certbot/tests/testdata/cert_512.pem similarity index 100% rename from certbot/tests/testdata/cert_512.pem rename to certbot/certbot/tests/testdata/cert_512.pem diff --git a/certbot/tests/testdata/cert_512_bad.pem b/certbot/certbot/tests/testdata/cert_512_bad.pem similarity index 100% rename from certbot/tests/testdata/cert_512_bad.pem rename to certbot/certbot/tests/testdata/cert_512_bad.pem diff --git a/certbot/tests/testdata/cert_fullchain_2048.pem b/certbot/certbot/tests/testdata/cert_fullchain_2048.pem similarity index 100% rename from certbot/tests/testdata/cert_fullchain_2048.pem rename to certbot/certbot/tests/testdata/cert_fullchain_2048.pem diff --git a/certbot/tests/testdata/cli.ini b/certbot/certbot/tests/testdata/cli.ini similarity index 100% rename from certbot/tests/testdata/cli.ini rename to certbot/certbot/tests/testdata/cli.ini diff --git a/certbot/tests/testdata/csr-6sans_512.conf b/certbot/certbot/tests/testdata/csr-6sans_512.conf similarity index 100% rename from certbot/tests/testdata/csr-6sans_512.conf rename to certbot/certbot/tests/testdata/csr-6sans_512.conf diff --git a/certbot/tests/testdata/csr-6sans_512.pem b/certbot/certbot/tests/testdata/csr-6sans_512.pem similarity index 100% rename from certbot/tests/testdata/csr-6sans_512.pem rename to certbot/certbot/tests/testdata/csr-6sans_512.pem diff --git a/certbot/tests/testdata/csr-nonames_512.pem b/certbot/certbot/tests/testdata/csr-nonames_512.pem similarity index 100% rename from certbot/tests/testdata/csr-nonames_512.pem rename to certbot/certbot/tests/testdata/csr-nonames_512.pem diff --git a/certbot/tests/testdata/csr-nosans_512.conf b/certbot/certbot/tests/testdata/csr-nosans_512.conf similarity index 100% rename from certbot/tests/testdata/csr-nosans_512.conf rename to certbot/certbot/tests/testdata/csr-nosans_512.conf diff --git a/certbot/tests/testdata/csr-nosans_512.pem b/certbot/certbot/tests/testdata/csr-nosans_512.pem similarity index 100% rename from certbot/tests/testdata/csr-nosans_512.pem rename to certbot/certbot/tests/testdata/csr-nosans_512.pem diff --git a/certbot/tests/testdata/csr-nosans_nistp256.pem b/certbot/certbot/tests/testdata/csr-nosans_nistp256.pem similarity index 100% rename from certbot/tests/testdata/csr-nosans_nistp256.pem rename to certbot/certbot/tests/testdata/csr-nosans_nistp256.pem diff --git a/certbot/tests/testdata/csr-san_512.pem b/certbot/certbot/tests/testdata/csr-san_512.pem similarity index 100% rename from certbot/tests/testdata/csr-san_512.pem rename to certbot/certbot/tests/testdata/csr-san_512.pem diff --git a/certbot/tests/testdata/csr_512.der b/certbot/certbot/tests/testdata/csr_512.der similarity index 100% rename from certbot/tests/testdata/csr_512.der rename to certbot/certbot/tests/testdata/csr_512.der diff --git a/certbot/tests/testdata/csr_512.pem b/certbot/certbot/tests/testdata/csr_512.pem similarity index 100% rename from certbot/tests/testdata/csr_512.pem rename to certbot/certbot/tests/testdata/csr_512.pem diff --git a/certbot/tests/testdata/nistp256_key.pem b/certbot/certbot/tests/testdata/nistp256_key.pem similarity index 100% rename from certbot/tests/testdata/nistp256_key.pem rename to certbot/certbot/tests/testdata/nistp256_key.pem diff --git a/certbot/certbot/tests/testdata/ocsp_certificate.pem b/certbot/certbot/tests/testdata/ocsp_certificate.pem new file mode 100644 index 000000000..471844859 --- /dev/null +++ b/certbot/certbot/tests/testdata/ocsp_certificate.pem @@ -0,0 +1,37 @@ +-----BEGIN CERTIFICATE----- +MIIGYDCCBEigAwIBAgIKcjrC4hZcebbtODANBgkqhkiG9w0BAQsFADBRMQswCQYD +VQQGEwJOTzEdMBsGA1UECgwUQnV5cGFzcyBBUy05ODMxNjMzMjcxIzAhBgNVBAMM +GkJ1eXBhc3MgQ2xhc3MgMiBUZXN0NCBDQSA1MB4XDTE5MDUxMjE1NTgyMVoXDTE5 +MTEwODIyNTkwMFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK9P +b+YhJPypm4ui+AZUHPrJ6IsB9R/6Wvgec2G/GuW/UNQFktIhU10HOHAbiJeYLqNZ +1Cia8JD6NXXGbprOjIbZWvjulYTaLSlClcK0H7HZrcgrK60OeIGEtur27ga68RML +hs1FG7TNyWVysifOtwW9Oo1mZQQtxViiE2Yb+Q4QqIxitnbrnFmKrVJSUHVXi8/I +BK1yLrJiRBZMIw0wvAWcWEG2Gpp9PAbemlb11Zx8sm/RSGh7u60rmETbB2Pu941s +XJCSQRtq5yKdtjIJTIgbe12SPkknqTqa3aUh7hgho0IymlDSeeocL60SUiUAsPEr +QRWleodOR1ChXz5mFokCAwEAAaOCAokwggKFMAkGA1UdEwQCMAAwHwYDVR0jBBgw +FoAUd9nQBpFm2N0ZJo1JrNowL2p7YrEwHQYDVR0OBBYEFExS23I6sLCeO6KIxzoc +tr9s+HmiMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYB +BQUHAwIwIAYDVR0gBBkwFzALBglghEIBGgEAAgcwCAYGZ4EMAQIBMEIGA1UdHwQ7 +MDkwN6A1oDOGMWh0dHA6Ly9jcmwudGVzdDQuYnV5cGFzcy5uby9jcmwvQlBDbGFz +czJUNENBNS5jcmwwIQYDVR0RAQH/BBcwFYITYnV5cGFzcy5wYWNhbGlzLm5ldDB4 +BggrBgEFBQcBAQRsMGowKQYIKwYBBQUHMAGGHWh0dHA6Ly9vY3NwLnRlc3Q0LmJ1 +eXBhc3MuY29tMD0GCCsGAQUFBzAChjFodHRwOi8vY3J0LnRlc3Q0LmJ1eXBhc3Mu +bm8vY3J0L0JQQ2xhc3MyVDRDQTUuY2VyMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDw +AHYAsMyD5aX5fWuvfAnMKEkEhyrH6IsTLGNQt8b9JuFsbHcAAAFqrMQ/cQAABAMA +RzBFAiEA1oWB4c6q7+tqGA4HhLNACOemr9c2aIUuWxeQE7/PlSYCIEolZ7pWVs1J +VyQW/AqeuXGB7qScwUgLh9C1uOJoeRe6AHYAsMyD5aX5fWuvfAnMKEkEhyrH6IsT +LGNQt8b9JuFsbHcAAAFqrMQ/cQAABAMARzBFAiAoLaNvIwMDifsDAXJBsAKHlYx7 +QPLXL8onYKm8f+Sf1wIhAMepo2GX84UR7WtooqzkBZLG+PaBy1zMuUAG6mwnroF9 +MA0GCSqGSIb3DQEBCwUAA4ICAQAPWLdjNS5lLL5SEtghYebtDmNj2968NYSDvb1L +1/uFwg3LCVRR1Xb3z1Hc/sc1W0IFXU0zOqEQiuP8jkVP7UqkaWuK5Eu0eP0zPI83 +WBZM0+eBwxwzIMK/Q7fYKTu1+vg/FlH0WhtV43DQSik66366zvPi2Tfag9IPvRei +DOjbSOBF0o4er2oCrtI0lK5YrHOdWtD7xwQIuA606P9ucuufMf+JcmduRJsVZ2Zu +3K32SMDdAnyjvQWZNbt1ex3G8vuFQEi690UBhPcha/SO8QvLS89wcaLJnyMIWdv7 +54cbw+fa1nLKM7qph6Mk1yb0qpomPqLmKw4T6WX36c0vDlFSpexJLGgWDFqLUxPN +qV7cJz4mi1qaYfdWXRrnyU4bl55pHTTgEzbohV7apsmytkCe1uFNrpcTh8jzAhGN +PQqarX9UoESR56B/ufbBGlBWi0pkV49BFks6Ue0GVKo7djoxuV6+SsmYSE+6MNPv +IUsm54TSnwxjA8WyG7pl14g1hkGFQ4NRYJMiVqK3DMABaPxVmT7NRxUQQiM0mmM7 +EKNzLBeWHJF5ecdDR1MiIF3ayn+RiZb0r8aSQBMLwN1YwUZw+hSYz1eCd7bHN1gC +1ksxP61f8LBz0SwDoyOTr8wY++wqF26KfoYuKQ3LjLeHvuUtL3EMnAhiyuej8ZOZ +22spng== +-----END CERTIFICATE----- diff --git a/certbot/certbot/tests/testdata/ocsp_issuer_certificate.pem b/certbot/certbot/tests/testdata/ocsp_issuer_certificate.pem new file mode 100644 index 000000000..4f894ae4b --- /dev/null +++ b/certbot/certbot/tests/testdata/ocsp_issuer_certificate.pem @@ -0,0 +1,38 @@ +22spng== +-----END CERTIFICATE----- +-----BEGIN CERTIFICATE----- +MIIGMzCCBBugAwIBAgIJMvsa+ZFQCj8nMA0GCSqGSIb3DQEBCwUAMFQxCzAJBgNV +BAYTAk5PMR0wGwYDVQQKDBRCdXlwYXNzIEFTLTk4MzE2MzMyNzEmMCQGA1UEAwwd +QnV5cGFzcyBDbGFzcyAyIFRlc3Q0IFJvb3QgQ0EwHhcNMTcwMjEzMTY1MjQ2WhcN +MjcwMjEzMTY1MjQ2WjBRMQswCQYDVQQGEwJOTzEdMBsGA1UECgwUQnV5cGFzcyBB +Uy05ODMxNjMzMjcxIzAhBgNVBAMMGkJ1eXBhc3MgQ2xhc3MgMiBUZXN0NCBDQSA1 +MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAi/vpgO2sbUQZsoxWd6us +QvT/59kvw5ehoJABBXFs1J1AV1/K2hjhDXit/sNGKjzDvkfE9PJqXMnhKpPFkUzC +z/NmDK++d6aRflnDvJrxlPVpp0QGbe3qOErByFjWiHoobuVItlpRO/BaBdlgGvmQ +LeZFBXs/ZrLNFUKBcE+DZIyJH7vy2EB5dNNVn2mx0n+371InpKsYUaHNlxPpp+uj +TOL+e4OjWTBwDaI7rVzpavozb8SPzFxjpxLLVH/j+8VPwoe3lmxr8ATyI178iRdA +uxYfaKURSfu7PWjnDNTnq26E3pwW3E5zUbsADgUMh/PzoJAcszL1eHGUQaAGBP85 +PlLmHr+nsPMHXOUyl7Ts6KGkZlvjnVshKwUxYAqjAC7/BY0iI0xc406NK9heeVDk +NiFA8/To6mQ09vO/TBxQtkfNk2yuxiixa101peSg4/+E4VhwYv6MJxS/oVqBd2d3 +wemYW/JUVeJg9wXGq1e/c09/UjGwUGwU9s5LNFEgj4v1tcvWnONzWNXkyMrs5g4e +U8L/DQ3XgNrcA9zrfFq0cQhSJonj/VI/jbBYyB2yEuQAIjAN6eDIOoLmHGIIvZtE +0LL5jaZC3W518jB1OF7QSvaFtaFl0VqDy6LMXL50elMVC+hr9KpDnN0t8gaSiPyZ +wEC9SMdQ7SLVOUK1Xdh3dh0CAwEAAaOCAQkwggEFMA8GA1UdEwEB/wQFMAMBAf8w +HwYDVR0jBBgwFoAU0aT+MaGsc75ZynH0up0oH+tVHh4wHQYDVR0OBBYEFHfZ0AaR +ZtjdGSaNSazaMC9qe2KxMA4GA1UdDwEB/wQEAwIBBjAgBgNVHSAEGTAXMAsGCWCE +QgEaAQACBzAIBgZngQwBAgEwRQYDVR0fBD4wPDA6oDigNoY0aHR0cDovL2NybC50 +ZXN0NC5idXlwYXNzLm5vL2NybC9CUENsYXNzMlQ0Um9vdENBLmNybDA5BggrBgEF +BQcBAQQtMCswKQYIKwYBBQUHMAGGHWh0dHA6Ly9vY3NwLnRlc3Q0LmJ1eXBhc3Mu +Y29tMA0GCSqGSIb3DQEBCwUAA4ICAQBOgxedV31NCpZQRc8yFxoqQNgBnY1UeH/h +/s/9fGQzyGnTWZldEi5MGJKF6ulcYnklitlg/jic9au3xSoqP/i2smUHByX2wMrC +mDpLCwio2x2p/0Wscj5asqzJE2cCWqob2iHxo36nsr3Jdd2GIlzhZ0wm8rMZxsQG +FgbgHYIer79S+PIdHoZuUnCJhsJ+1PRUmm2t7vcmZpu8l4CeL0XJX98l2L8kbBds +MGo1EazGAEirZnSfQKCARhUcEdavsKl067+irsGGcK4+L78Vl9S1/QPfKG30L5fv +nM1X1qAdhsbjwVdrhLkjpzabT0icsW6W17HLh8UBYdA7k4GclA6h+mNrXAt7JAeZ +PzMFq0I7vVJNEdolZHTVCqT0sdJiTj+phS1ztK86Wb1R/5d5B1VSb789zSdJfrwV +ppXgPtZq5x3GQi6ooteWyuWj3cBcNu9TU1D8u1F0XI5gw4Y0VpxlDxysUgFQJlo4 +VYmMpgr442o/35UgwzkIC7x/6dkvMZvM4jYB5JZJXjynR35XawXB/hzybermJ8BB +DsY0MCOwxhpsTbyEC4wfxZ08B4JtORkToOt4OWuejovsr68Ht6ytOPj7dquoPPNM +9eGNSp94nEIiZ2n75ZMg0gIQArXU9OCV6B2TXxB7w2YB0y0teDgVhoM3IY/ltqJ/ +PJrUUjM8OQ== +-----END CERTIFICATE----- diff --git a/certbot/certbot/tests/testdata/ocsp_responder_certificate.pem b/certbot/certbot/tests/testdata/ocsp_responder_certificate.pem new file mode 100644 index 000000000..53bc4a92a --- /dev/null +++ b/certbot/certbot/tests/testdata/ocsp_responder_certificate.pem @@ -0,0 +1,27 @@ +-----BEGIN CERTIFICATE----- +MIIEpjCCAo6gAwIBAgINARMIGYlEsD1LTt6D7zANBgkqhkiG9w0BAQsFADBRMQsw +CQYDVQQGEwJOTzEdMBsGA1UECgwUQnV5cGFzcyBBUy05ODMxNjMzMjcxIzAhBgNV +BAMMGkJ1eXBhc3MgQ2xhc3MgMiBUZXN0NCBDQSA1MB4XDTE5MDQwNTEwMDAwMFoX +DTE5MDcwNDEwMDAwMFowSTELMAkGA1UEBhMCTk8xHTAbBgNVBAoMFEJ1eXBhc3Mg +QVMtOTgzMTYzMzI3MRswGQYDVQQDDBJCdXlwYXNzIFRlc3Q0IE9DU1AwggEiMA0G +CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKGF+kYNd1fbhYT7Vf9xouZlx+4w45 +Y5EowPoaSKFo4uUDDxkj4PwmMiH4w9Q2bGrCbZRrDrvlNVY/kwzLu4CIk6Ip0dgm +VZGNFB3Xo9nai7rI5pn/YVvVnDIQXh1LRbekzLVyHvhRgMpRb19xN/iYsxaOJDph +8eAgbTKf6eitvfbvn/zXHj4KGKycuULI4+mwlfV3uioT4ulbT7PTVJetgi/XXFDO +xMjbqx6I1ZMmzKJ6LNaFlfx6GdZsaLRDCidHzGp8Fm4ZdV+UPvMZcVDQO6rvQ3wU +iGyCqgfE5e0aFvfeLoBPBtaoT0Ht1CvGdTfVet6PXrF6gh40fdEH5Ob5AgMBAAGj +gYQwgYEwCQYDVR0TBAIwADAfBgNVHSMEGDAWgBR32dAGkWbY3RkmjUms2jAvanti +sTAdBgNVHQ4EFgQU3VlR+sSIVpmXklieP7IlpVUcXIowDgYDVR0PAQH/BAQDAgeA +MBMGA1UdJQQMMAoGCCsGAQUFBwMJMA8GCSsGAQUFBzABBQQCBQAwDQYJKoZIhvcN +AQELBQADggIBAFBRLVsBadNFAoFi0HOrfxYsiqggZGJLlgxGyi/0NBIgduG4kcpM +THvplwBwMQEqyp5511pSEbLPAFj8EqC5c46hXZXmT49xlfRvr2Bo+qtTPV9szuWr +8muEIejwRrkATpqWPZWR2zVTXfB90mU2oGuRvxUVmnW4v+FrCChJo7+9yTocZJKx +p4vxYfPMeggomdGAAUz94+0ppSjOLDzs3MA8uOcR0zJ2Y7UHb7PBf/HiM3GO2uKB +sRgdDaGIf/PNpav0xJ/abGNNNwvXzHiMgqqImsuv/JoncPQWbClNurhXpdN7xt9C +HcLX2AdggabcogjWm4guBFuFTsL1i0l8Bsu/6iPJ7ddCeANfYzf7h6AcQq12uFl3 +070F29DtPh8D3FPWgRZZsxoANFjXErxfj4a4+DR+jhhkb9YM/wI0vCOM7W6PKxVn +ZK5kHGOQTcQMj7RCX52gEf27M33zC7HVam+kKhGvwq7D9Bs5hZclzcbjpR4eIxT7 +tzuiy5VpPh1DRLPrphPUB4xsA1dy6zbkg8OqddG6NxD++ja/iZyzSB3SeWyO02qA +QoK2FzDasxpZ9rT3ioAcms3wVNe4lcd4OP8gHZONuat/gvxk6OZvAld6cnIrQZYB +Tbu89ZWvhsyI3p4YC/15pUvA95j9Y0te+G+CF22Eoyb+rtz6mMletnUB +-----END CERTIFICATE----- diff --git a/certbot/tests/testdata/os-release b/certbot/certbot/tests/testdata/os-release similarity index 100% rename from certbot/tests/testdata/os-release rename to certbot/certbot/tests/testdata/os-release diff --git a/certbot/tests/testdata/rsa2048_key.pem b/certbot/certbot/tests/testdata/rsa2048_key.pem similarity index 100% rename from certbot/tests/testdata/rsa2048_key.pem rename to certbot/certbot/tests/testdata/rsa2048_key.pem diff --git a/certbot/tests/testdata/rsa256_key.pem b/certbot/certbot/tests/testdata/rsa256_key.pem similarity index 100% rename from certbot/tests/testdata/rsa256_key.pem rename to certbot/certbot/tests/testdata/rsa256_key.pem diff --git a/certbot/tests/testdata/rsa512_key.pem b/certbot/certbot/tests/testdata/rsa512_key.pem similarity index 100% rename from certbot/tests/testdata/rsa512_key.pem rename to certbot/certbot/tests/testdata/rsa512_key.pem diff --git a/certbot/tests/testdata/sample-archive/cert1.pem b/certbot/certbot/tests/testdata/sample-archive/cert1.pem similarity index 100% rename from certbot/tests/testdata/sample-archive/cert1.pem rename to certbot/certbot/tests/testdata/sample-archive/cert1.pem diff --git a/certbot/tests/testdata/sample-archive/chain1.pem b/certbot/certbot/tests/testdata/sample-archive/chain1.pem similarity index 100% rename from certbot/tests/testdata/sample-archive/chain1.pem rename to certbot/certbot/tests/testdata/sample-archive/chain1.pem diff --git a/certbot/tests/testdata/sample-archive/fullchain1.pem b/certbot/certbot/tests/testdata/sample-archive/fullchain1.pem similarity index 100% rename from certbot/tests/testdata/sample-archive/fullchain1.pem rename to certbot/certbot/tests/testdata/sample-archive/fullchain1.pem diff --git a/certbot/tests/testdata/sample-archive/privkey1.pem b/certbot/certbot/tests/testdata/sample-archive/privkey1.pem similarity index 100% rename from certbot/tests/testdata/sample-archive/privkey1.pem rename to certbot/certbot/tests/testdata/sample-archive/privkey1.pem diff --git a/certbot/tests/testdata/sample-renewal-ancient.conf b/certbot/certbot/tests/testdata/sample-renewal-ancient.conf similarity index 96% rename from certbot/tests/testdata/sample-renewal-ancient.conf rename to certbot/certbot/tests/testdata/sample-renewal-ancient.conf index 333bcaa18..9586d5492 100644 --- a/certbot/tests/testdata/sample-renewal-ancient.conf +++ b/certbot/certbot/tests/testdata/sample-renewal-ancient.conf @@ -62,14 +62,12 @@ break_my_certs = False standalone = True manual = False server = https://acme-staging.api.letsencrypt.org/directory -standalone_supported_challenges = "tls-sni-01,http-01" webroot = True os_packages_only = False apache_init_script = None user_agent = None apache_le_vhost_ext = -le-ssl.conf debug = False -tls_sni_01_port = 443 logs_dir = /var/log/letsencrypt apache_vhost_root = /etc/apache2/sites-available configurator = None diff --git a/certbot/tests/testdata/sample-renewal.conf b/certbot/certbot/tests/testdata/sample-renewal.conf similarity index 96% rename from certbot/tests/testdata/sample-renewal.conf rename to certbot/certbot/tests/testdata/sample-renewal.conf index 04f9ae8ca..936c5c0e0 100644 --- a/certbot/tests/testdata/sample-renewal.conf +++ b/certbot/certbot/tests/testdata/sample-renewal.conf @@ -62,14 +62,12 @@ break_my_certs = False standalone = True manual = False server = https://acme-staging-v02.api.letsencrypt.org/directory -standalone_supported_challenges = "tls-sni-01,http-01" webroot = False os_packages_only = False apache_init_script = None user_agent = None apache_le_vhost_ext = -le-ssl.conf debug = False -tls_sni_01_port = 443 logs_dir = /var/log/letsencrypt apache_vhost_root = /etc/apache2/sites-available configurator = None diff --git a/certbot/tests/testdata/webrootconftest.ini b/certbot/certbot/tests/testdata/webrootconftest.ini similarity index 100% rename from certbot/tests/testdata/webrootconftest.ini rename to certbot/certbot/tests/testdata/webrootconftest.ini diff --git a/certbot/tests/util.py b/certbot/certbot/tests/util.py similarity index 79% rename from certbot/tests/util.py rename to certbot/certbot/tests/util.py index 38a9075c1..02abe0a31 100644 --- a/certbot/tests/util.py +++ b/certbot/certbot/tests/util.py @@ -4,29 +4,30 @@ """ import logging -import multiprocessing -import os -import pkg_resources +from multiprocessing import Event +from multiprocessing import Process import shutil -import stat +import sys import tempfile import unittest -import sys from cryptography.hazmat.backends import default_backend from cryptography.hazmat.primitives import serialization +import josepy as jose import mock import OpenSSL -import josepy as jose +import pkg_resources import six from six.moves import reload_module # pylint: disable=import-error -from certbot import constants from certbot import interfaces -from certbot import storage from certbot import util -from certbot import configuration - +from certbot._internal import configuration +from certbot._internal import constants +from certbot._internal import lock +from certbot._internal import storage +from certbot.compat import filesystem +from certbot.compat import os from certbot.display import util as display_util @@ -56,8 +57,7 @@ def _guess_loader(filename, loader_pem, loader_der): return loader_pem elif ext.lower() == '.der': return loader_der - else: # pragma: no cover - raise ValueError("Loader could not be recognized based on extension") + raise ValueError("Loader could not be recognized based on extension") # pragma: no cover def load_cert(*names): @@ -94,27 +94,6 @@ def load_pyopenssl_private_key(*names): return OpenSSL.crypto.load_privatekey(loader, load_vector(*names)) -def skip_unless(condition, reason): # pragma: no cover - """Skip tests unless a condition holds. - - This implements the basic functionality of unittest.skipUnless - which is only available on Python 2.7+. - - :param bool condition: If ``False``, the test will be skipped - :param str reason: the reason for skipping the test - - :rtype: callable - :returns: decorator that hides tests unless condition is ``True`` - - """ - if hasattr(unittest, "skipUnless"): - return unittest.skipUnless(condition, reason) - elif condition: - return lambda cls: cls - else: - return lambda cls: None - - def make_lineage(config_dir, testfile): """Creates a lineage defined by testfile. @@ -139,7 +118,7 @@ def make_lineage(config_dir, testfile): for directory in (archive_dir, conf_dir, live_dir,): if not os.path.exists(directory): - os.makedirs(directory) + filesystem.makedirs(directory) sample_archive = vector_path('sample-archive') for kind in os.listdir(sample_archive): @@ -212,7 +191,7 @@ class FreezableMock(object): """ def __init__(self, frozen=False, func=None, return_value=mock.sentinel.DEFAULT): - self._frozen_set = set() if frozen else set(('freeze',)) + self._frozen_set = set() if frozen else {'freeze', } self._func = func self._mock = mock.MagicMock() if return_value != mock.sentinel.DEFAULT: @@ -255,8 +234,7 @@ class FreezableMock(object): if self._frozen: if name in self._frozen_set: raise AttributeError('Cannot change frozen attribute ' + name) - else: - return setattr(self._mock, name, value) + return setattr(self._mock, name, value) if name != '_frozen_set': self._frozen_set.add(name) @@ -264,13 +242,13 @@ class FreezableMock(object): if name in ('return_value', 'side_effect'): return setattr(self._mock, name, value) - else: - return object.__setattr__(self, name, value) + return object.__setattr__(self, name, value) def _create_get_utility_mock(): display = FreezableMock() - for name in interfaces.IDisplay.names(): # pylint: disable=no-member + # Use pylint code for disable to keep on single line under line length limit + for name in interfaces.IDisplay.names(): # pylint: E1120 if name != 'notification': frozen_mock = FreezableMock(frozen=True, func=_assert_valid_call) setattr(display, name, frozen_mock) @@ -294,7 +272,8 @@ def _create_get_utility_mock_with_stdout(stdout): display = FreezableMock() - for name in interfaces.IDisplay.names(): # pylint: disable=no-member + # Use pylint code for disable to keep on single line under line length limit + for name in interfaces.IDisplay.names(): # pylint: E1120 if name == 'notification': frozen_mock = FreezableMock(frozen=True, func=_write_msg) @@ -316,7 +295,6 @@ def _assert_valid_call(*args, **kwargs): assert_kwargs['cli_flag'] = kwargs.get('cli_flag', None) assert_kwargs['force_interactive'] = kwargs.get('force_interactive', False) - # pylint: disable=star-args display_util.assert_valid_call(*assert_args, **assert_kwargs) @@ -334,13 +312,12 @@ class TempDirTestCase(unittest.TestCase): # called and instead will run them right before the entire test process exits. # It is a problem on Windows, that does not accept to clean resources before closing them. logging.shutdown() + # Remove logging handlers that have been closed so they won't be + # accidentally used in future tests. + logging.getLogger().handlers = [] util._release_locks() # pylint: disable=protected-access - def handle_rw_files(_, path, __): - """Handle read-only files, that will fail to be removed on Windows.""" - os.chmod(path, stat.S_IWRITE) - os.remove(path) - shutil.rmtree(self.tempdir, onerror=handle_rw_files) + shutil.rmtree(self.tempdir) class ConfigTestCase(TempDirTestCase): @@ -359,47 +336,51 @@ class ConfigTestCase(TempDirTestCase): self.config.chain_path = constants.CLI_DEFAULTS['auth_chain_path'] self.config.server = "https://example.com" -def lock_and_call(func, lock_path): - """Grab a lock for lock_path and call func. - - :param callable func: object to call after acquiring the lock - :param str lock_path: path to file or directory to lock +def _handle_lock(event_in, event_out, path): """ - # Reload module to reset internal _LOCKS dictionary + Acquire a file lock on given path, then wait to release it. This worker is coordinated + using events to signal when the lock should be acquired and released. + :param multiprocessing.Event event_in: event object to signal when to release the lock + :param multiprocessing.Event event_out: event object to signal when the lock is acquired + :param path: the path to lock + """ + if os.path.isdir(path): + my_lock = lock.lock_dir(path) + else: + my_lock = lock.LockFile(path) + try: + event_out.set() + assert event_in.wait(timeout=20), 'Timeout while waiting to release the lock.' + finally: + my_lock.release() + + +def lock_and_call(callback, path_to_lock): + """ + Grab a lock on path_to_lock from a foreign process then execute the callback. + :param callable callback: object to call after acquiring the lock + :param str path_to_lock: path to file or directory to lock + """ + # Reload certbot.util module to reset internal _LOCKS dictionary. reload_module(util) - # start child and wait for it to grab the lock - cv = multiprocessing.Condition() - cv.acquire() - child_args = (cv, lock_path,) - child = multiprocessing.Process(target=hold_lock, args=child_args) - child.start() - cv.wait() + emit_event = Event() + receive_event = Event() + process = Process(target=_handle_lock, args=(emit_event, receive_event, path_to_lock)) + process.start() - # call func and terminate the child - func() - cv.notify() - cv.release() - child.join() - assert child.exitcode == 0 + # Wait confirmation that lock is acquired + assert receive_event.wait(timeout=10), 'Timeout while waiting to acquire the lock.' + # Execute the callback + callback() + # Trigger unlock from foreign process + emit_event.set() -def hold_lock(cv, lock_path): # pragma: no cover - """Acquire a file lock at lock_path and wait to release it. + # Wait for process termination + process.join(timeout=10) + assert process.exitcode == 0 - :param multiprocessing.Condition cv: condition for synchronization - :param str lock_path: path to the file lock - - """ - from certbot import lock - if os.path.isdir(lock_path): - my_lock = lock.lock_dir(lock_path) - else: - my_lock = lock.LockFile(lock_path) - cv.acquire() - cv.notify() - cv.wait() - my_lock.release() def skip_on_windows(reason): """Decorator to skip permanently a test on Windows. A reason is required.""" @@ -408,17 +389,10 @@ def skip_on_windows(reason): return unittest.skipIf(sys.platform == 'win32', reason)(function) return wrapper -def broken_on_windows(function): - """Decorator to skip temporarily a broken test on Windows.""" - reason = 'Test is broken and ignored on windows but should be fixed.' - return unittest.skipIf( - sys.platform == 'win32' - and os.environ.get('SKIP_BROKEN_TESTS_ON_WINDOWS', 'true') == 'true', - reason)(function) def temp_join(path): """ Return the given path joined to the tempdir path for the current platform Eg.: 'cert' => /tmp/cert (Linux) or 'C:\\Users\\currentuser\\AppData\\Temp\\cert' (Windows) """ - return os.path.join(tempfile.gettempdir(), path) + return os.path.join(tempfile.gettempdir(), path) diff --git a/certbot/util.py b/certbot/certbot/util.py similarity index 81% rename from certbot/util.py rename to certbot/certbot/util.py index 416075ce8..0a47cd87a 100644 --- a/certbot/util.py +++ b/certbot/certbot/util.py @@ -1,30 +1,35 @@ """Utilities for all Certbot.""" +# distutils.version under virtualenv confuses pylint +# For more info, see: https://github.com/PyCQA/pylint/issues/73 import argparse import atexit import collections -# distutils.version under virtualenv confuses pylint -# For more info, see: https://github.com/PyCQA/pylint/issues/73 +from collections import OrderedDict import distutils.version # pylint: disable=import-error,no-name-in-module import errno import logging -import os import platform import re -import six import socket import subprocess import sys -from collections import OrderedDict - import configargparse +import six -from acme.magic_typing import Tuple, Union # pylint: disable=unused-import, no-name-in-module -from certbot import compat -from certbot import constants +from acme.magic_typing import Tuple # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Union # pylint: disable=unused-import, no-name-in-module from certbot import errors -from certbot import lock +from certbot._internal import constants +from certbot._internal import lock +from certbot.compat import filesystem +from certbot.compat import os +if sys.platform.startswith('linux'): + import distro # pylint: disable=import-error + _USE_DISTRO = True +else: + _USE_DISTRO = False logger = logging.getLogger(__name__) @@ -62,7 +67,7 @@ def run_script(params, log=logger.error): """Run the script with the given params. :param list params: List of parameters to pass to Popen - :param logging.Logger log: Logger to use for errors + :param callable log: Logger method to use for errors """ try: @@ -88,18 +93,6 @@ def run_script(params, log=logger.error): return stdout, stderr -def is_exe(path): - """Is path an executable file? - - :param str path: path to test - - :returns: True iff path is an executable file - :rtype: bool - - """ - return os.path.isfile(path) and os.access(path, os.X_OK) - - def exe_exists(exe): """Determine whether path/name refers to an executable. @@ -111,11 +104,10 @@ def exe_exists(exe): """ path, _ = os.path.split(exe) if path: - return is_exe(exe) - else: - for path in os.environ["PATH"].split(os.pathsep): - if is_exe(os.path.join(path, exe)): - return True + return filesystem.is_executable(exe) + for path in os.environ["PATH"].split(os.pathsep): + if filesystem.is_executable(os.path.join(path, exe)): + return True return False @@ -145,12 +137,11 @@ def _release_locks(): _LOCKS.clear() -def set_up_core_dir(directory, mode, uid, strict): +def set_up_core_dir(directory, mode, strict): """Ensure directory exists with proper permissions and is locked. :param str directory: Path to a directory. :param int mode: Directory mode. - :param int uid: Directory owner. :param bool strict: require directory to be owned by current user :raises .errors.LockError: if the directory cannot be locked @@ -158,19 +149,18 @@ def set_up_core_dir(directory, mode, uid, strict): """ try: - make_or_verify_dir(directory, mode, uid, strict) + make_or_verify_dir(directory, mode, strict) lock_dir_until_exit(directory) except OSError as error: logger.debug("Exception was:", exc_info=True) raise errors.Error(PERM_ERR_FMT.format(error)) -def make_or_verify_dir(directory, mode=0o755, uid=0, strict=False): +def make_or_verify_dir(directory, mode=0o755, strict=False): """Make sure directory exists with proper permissions. :param str directory: Path to a directory. :param int mode: Directory mode. - :param int uid: Directory owner. :param bool strict: require directory to be owned by current user :raises .errors.Error: if a directory already exists, @@ -182,51 +172,31 @@ def make_or_verify_dir(directory, mode=0o755, uid=0, strict=False): """ try: - os.makedirs(directory, mode) + filesystem.makedirs(directory, mode) except OSError as exception: if exception.errno == errno.EEXIST: - if strict and not check_permissions(directory, mode, uid): + if strict and not filesystem.check_permissions(directory, mode): raise errors.Error( - "%s exists, but it should be owned by user %d with" - "permissions %s" % (directory, uid, oct(mode))) + "%s exists, but it should be owned by current user with" + " permissions %s" % (directory, oct(mode))) else: raise -def check_permissions(filepath, mode, uid=0): - """Check file or directory permissions. - - :param str filepath: Path to the tested file (or directory). - :param int mode: Expected file mode. - :param int uid: Expected file owner. - - :returns: True if `mode` and `uid` match, False otherwise. - :rtype: bool - - """ - file_stat = os.stat(filepath) - return compat.compare_file_modes(file_stat.st_mode, mode) and file_stat.st_uid == uid - - -def safe_open(path, mode="w", chmod=None, buffering=None): +def safe_open(path, mode="w", chmod=None): """Safely open a file. :param str path: Path to a file. :param str mode: Same os `mode` for `open`. - :param int chmod: Same as `mode` for `os.open`, uses Python defaults + :param int chmod: Same as `mode` for `filesystem.open`, uses Python defaults if ``None``. - :param int buffering: Same as `bufsize` for `os.fdopen`, uses Python - defaults if ``None``. """ - # pylint: disable=star-args open_args = () # type: Union[Tuple[()], Tuple[int]] if chmod is not None: open_args = (chmod,) fdopen_args = () # type: Union[Tuple[()], Tuple[int]] - if buffering is not None: - fdopen_args = (buffering,) - fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_RDWR, *open_args) + fd = filesystem.open(path, os.O_CREAT | os.O_EXCL | os.O_RDWR, *open_args) return os.fdopen(fd, mode, *fdopen_args) @@ -312,77 +282,46 @@ def get_filtered_names(all_names): logger.debug('Not suggesting name "%s"', name, exc_info=True) return filtered_names - -def get_os_info(filepath="/etc/os-release"): +def get_os_info(): """ Get OS name and version - :param str filepath: File path of os-release file :returns: (os_name, os_version) :rtype: `tuple` of `str` """ - if os.path.isfile(filepath): - # Systemd os-release parsing might be viable - os_name, os_version = get_systemd_os_info(filepath=filepath) - if os_name: - return (os_name, os_version) + return get_python_os_info(pretty=False) - # Fallback to platform module - return get_python_os_info() - - -def get_os_info_ua(filepath="/etc/os-release"): +def get_os_info_ua(): """ Get OS name and version string for User Agent - :param str filepath: File path of os-release file :returns: os_ua :rtype: `str` """ + if _USE_DISTRO: + os_info = distro.name(pretty=True) - if os.path.isfile(filepath): - os_ua = get_var_from_file("PRETTY_NAME", filepath=filepath) - if not os_ua: - os_ua = get_var_from_file("NAME", filepath=filepath) - if os_ua: - return os_ua + if not _USE_DISTRO or not os_info: + return " ".join(get_python_os_info(pretty=True)) + return os_info - # Fallback - return " ".join(get_python_os_info()) - - -def get_systemd_os_info(filepath="/etc/os-release"): - """ - Parse systemd /etc/os-release for distribution information - - :param str filepath: File path of os-release file - :returns: (os_name, os_version) - :rtype: `tuple` of `str` - """ - - os_name = get_var_from_file("ID", filepath=filepath) - os_version = get_var_from_file("VERSION_ID", filepath=filepath) - - return (os_name, os_version) - - -def get_systemd_os_like(filepath="/etc/os-release"): +def get_systemd_os_like(): """ Get a list of strings that indicate the distribution likeness to other distributions. - :param str filepath: File path of os-release file :returns: List of distribution acronyms :rtype: `list` of `str` """ - return get_var_from_file("ID_LIKE", filepath).split(" ") - + if _USE_DISTRO: + return distro.like().split(" ") + return [] def get_var_from_file(varname, filepath="/etc/os-release"): """ - Get single value from systemd /etc/os-release + Get single value from a file formatted like systemd /etc/os-release :param str varname: Name of variable to fetch :param str filepath: File path of os-release file @@ -402,7 +341,6 @@ def get_var_from_file(varname, filepath="/etc/os-release"): return _normalize_string(line.strip()[len(var_string):]) return "" - def _normalize_string(orig): """ Helper function for get_var_from_file() to remove quotes @@ -410,12 +348,13 @@ def _normalize_string(orig): """ return orig.replace('"', '').replace("'", "").strip() - -def get_python_os_info(): +def get_python_os_info(pretty=False): """ Get Operating System type/distribution and major version using python platform module + :param bool pretty: If the returned OS name should be in longer (pretty) form + :returns: (os_name, os_version) :rtype: `tuple` of `str` """ @@ -426,9 +365,9 @@ def get_python_os_info(): ) os_type, os_ver, _ = info os_type = os_type.lower() - if os_type.startswith('linux'): - info = platform.linux_distribution() - # On arch, platform.linux_distribution() is reportedly ('','',''), + if os_type.startswith('linux') and _USE_DISTRO: + info = distro.linux_distribution(pretty) + # On arch, distro.linux_distribution() is reportedly ('','',''), # so handle it defensively if info[0]: os_type = info[0] @@ -469,16 +408,14 @@ def safe_email(email): """Scrub email address before using it.""" if EMAIL_REGEX.match(email) is not None: return not email.startswith(".") and ".." not in email - else: - logger.warning("Invalid email address: %s.", email) - return False + logger.warning("Invalid email address: %s.", email) + return False class _ShowWarning(argparse.Action): """Action to log a warning when an argument is used.""" def __call__(self, unused1, unused2, unused3, option_string=None): - sys.stderr.write( - "Use of {0} is deprecated.\n".format(option_string)) + logger.warning("Use of %s is deprecated.", option_string) def add_deprecated_argument(add_argument, argument_name, nargs): @@ -498,7 +435,6 @@ def add_deprecated_argument(add_argument, argument_name, nargs): # In version 0.12.0 ACTION_TYPES_THAT_DONT_NEED_A_VALUE was # changed from a set to a tuple. if isinstance(configargparse.ACTION_TYPES_THAT_DONT_NEED_A_VALUE, set): - # pylint: disable=no-member configargparse.ACTION_TYPES_THAT_DONT_NEED_A_VALUE.add( _ShowWarning) else: @@ -599,7 +535,7 @@ def enforce_domain_sanity(domain): for l in labels: if not l: raise errors.ConfigurationError("{0} it contains an empty label.".format(msg)) - elif len(l) > 63: + if len(l) > 63: raise errors.ConfigurationError("{0} label {1} is too long.".format(msg, l)) return domain @@ -633,7 +569,6 @@ def get_strict_version(normalized): """ # strict version ending with "a" and a number designates a pre-release - # pylint: disable=no-member return distutils.version.StrictVersion(normalized.replace(".dev", "a")) diff --git a/certbot/compat.py b/certbot/compat.py deleted file mode 100644 index 3b5d068a6..000000000 --- a/certbot/compat.py +++ /dev/null @@ -1,223 +0,0 @@ -""" -Compatibility layer to run certbot both on Linux and Windows. - -This module contains all required platform specific code, -allowing the rest of Certbot codebase to be platform agnostic. -""" -import os -import select -import sys -import errno -import ctypes -import stat - -from certbot import errors - -try: - # Linux specific - import fcntl # pylint: disable=import-error -except ImportError: - # Windows specific - import msvcrt # pylint: disable=import-error - -UNPRIVILEGED_SUBCOMMANDS_ALLOWED = [ - 'certificates', 'enhance', 'revoke', 'delete', - 'register', 'unregister', 'config_changes', 'plugins'] - - -def raise_for_non_administrative_windows_rights(subcommand): - """ - On Windows, raise if current shell does not have the administrative rights. - Do nothing on Linux. - - :param str subcommand: The subcommand (like 'certonly') passed to the certbot client. - - :raises .errors.Error: If the provided subcommand must be run on a shell with - administrative rights, and current shell does not have these rights. - - """ - # Why not simply try ctypes.windll.shell32.IsUserAnAdmin() and catch AttributeError ? - # Because windll exists only on a Windows runtime, and static code analysis engines - # do not like at all non existent objects when run from Linux (even if we handle properly - # all the cases in the code). - # So we access windll only by reflection to trick theses engines. - if hasattr(ctypes, 'windll') and subcommand not in UNPRIVILEGED_SUBCOMMANDS_ALLOWED: - windll = getattr(ctypes, 'windll') - if windll.shell32.IsUserAnAdmin() == 0: - raise errors.Error( - 'Error, "{0}" subcommand must be run on a shell with administrative rights.' - .format(subcommand)) - - -def os_geteuid(): - """ - Get current user uid - - :returns: The current user uid. - :rtype: int - - """ - try: - # Linux specific - return os.geteuid() - except AttributeError: - # Windows specific - return 0 - - -def os_rename(src, dst): - """ - Rename a file to a destination path and handles situations where the destination exists. - - :param str src: The current file path. - :param str dst: The new file path. - """ - try: - os.rename(src, dst) - except OSError as err: - # Windows specific, renaming a file on an existing path is not possible. - # On Python 3, the best fallback with atomic capabilities we have is os.replace. - if err.errno != errno.EEXIST: - # Every other error is a legitimate exception. - raise - if not hasattr(os, 'replace'): # pragma: no cover - # We should never go on this line. Either we are on Linux and os.rename has succeeded, - # either we are on Windows, and only Python >= 3.4 is supported where os.replace is - # available. - raise RuntimeError('Error: tried to run os_rename on Python < 3.3. ' - 'Certbot supports only Python 3.4 >= on Windows.') - getattr(os, 'replace')(src, dst) - - -def readline_with_timeout(timeout, prompt): - """ - Read user input to return the first line entered, or raise after specified timeout. - - :param float timeout: The timeout in seconds given to the user. - :param str prompt: The prompt message to display to the user. - - :returns: The first line entered by the user. - :rtype: str - - """ - try: - # Linux specific - # - # Call to select can only be done like this on UNIX - rlist, _, _ = select.select([sys.stdin], [], [], timeout) - if not rlist: - raise errors.Error( - "Timed out waiting for answer to prompt '{0}'".format(prompt)) - return rlist[0].readline() - except OSError: - # Windows specific - # - # No way with select to make a timeout to the user input on Windows, - # as select only supports socket in this case. - # So no timeout on Windows for now. - return sys.stdin.readline() - - -def lock_file(fd): - """ - Lock the file linked to the specified file descriptor. - - :param int fd: The file descriptor of the file to lock. - - """ - if 'fcntl' in sys.modules: - # Linux specific - fcntl.lockf(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) - else: - # Windows specific - msvcrt.locking(fd, msvcrt.LK_NBLCK, 1) - - -def release_locked_file(fd, path): - """ - Remove, close, and release a lock file specified by its file descriptor and its path. - - :param int fd: The file descriptor of the lock file. - :param str path: The path of the lock file. - - """ - # Linux specific - # - # It is important the lock file is removed before it's released, - # otherwise: - # - # process A: open lock file - # process B: release lock file - # process A: lock file - # process A: check device and inode - # process B: delete file - # process C: open and lock a different file at the same path - try: - os.remove(path) - except OSError as err: - if err.errno == errno.EACCES: - # Windows specific - # We will not be able to remove a file before closing it. - # To avoid race conditions described for Linux, we will not delete the lockfile, - # just close it to be reused on the next Certbot call. - pass - else: - raise - finally: - os.close(fd) - - -def compare_file_modes(mode1, mode2): - """Return true if the two modes can be considered as equals for this platform""" - if os.name != 'nt': - # Linux specific: standard compare - return oct(stat.S_IMODE(mode1)) == oct(stat.S_IMODE(mode2)) - # Windows specific: most of mode bits are ignored on Windows. Only check user R/W rights. - return (stat.S_IMODE(mode1) & stat.S_IREAD == stat.S_IMODE(mode2) & stat.S_IREAD - and stat.S_IMODE(mode1) & stat.S_IWRITE == stat.S_IMODE(mode2) & stat.S_IWRITE) - - -WINDOWS_DEFAULT_FOLDERS = { - 'config': 'C:\\Certbot', - 'work': 'C:\\Certbot\\lib', - 'logs': 'C:\\Certbot\\log', -} -LINUX_DEFAULT_FOLDERS = { - 'config': '/etc/letsencrypt', - 'work': '/var/lib/letsencrypt', - 'logs': '/var/log/letsencrypt', -} - - -def get_default_folder(folder_type): - """ - Return the relevant default folder for the current OS - - :param str folder_type: The type of folder to retrieve (config, work or logs) - - :returns: The relevant default folder. - :rtype: str - - """ - if os.name != 'nt': - # Linux specific - return LINUX_DEFAULT_FOLDERS[folder_type] - # Windows specific - return WINDOWS_DEFAULT_FOLDERS[folder_type] - - -def underscores_for_unsupported_characters_in_path(path): - # type: (str) -> str - """ - Replace unsupported characters in path for current OS by underscores. - :param str path: the path to normalize - :return: the normalized path - :rtype: str - """ - if os.name != 'nt': - # Linux specific - return path - - # Windows specific - drive, tail = os.path.splitdrive(path) - return drive + tail.replace(':', '_') diff --git a/certbot-apache/docs/.gitignore b/certbot/docs/.gitignore similarity index 100% rename from certbot-apache/docs/.gitignore rename to certbot/docs/.gitignore diff --git a/docs/Makefile b/certbot/docs/Makefile similarity index 100% rename from docs/Makefile rename to certbot/docs/Makefile diff --git a/certbot-compatibility-test/docs/_templates/.gitignore b/certbot/docs/_static/.gitignore similarity index 100% rename from certbot-compatibility-test/docs/_templates/.gitignore rename to certbot/docs/_static/.gitignore diff --git a/docs/_templates/footer.html b/certbot/docs/_templates/footer.html similarity index 100% rename from docs/_templates/footer.html rename to certbot/docs/_templates/footer.html diff --git a/certbot-apache/docs/api.rst b/certbot/docs/api.rst similarity index 69% rename from certbot-apache/docs/api.rst rename to certbot/docs/api.rst index 8668ec5d8..9c8b2f1fe 100644 --- a/certbot-apache/docs/api.rst +++ b/certbot/docs/api.rst @@ -3,6 +3,6 @@ API Documentation ================= .. toctree:: - :glob: + :maxdepth: 4 - api/** + api/certbot diff --git a/certbot/docs/api/certbot.achallenges.rst b/certbot/docs/api/certbot.achallenges.rst new file mode 100644 index 000000000..3fd2f2a42 --- /dev/null +++ b/certbot/docs/api/certbot.achallenges.rst @@ -0,0 +1,7 @@ +certbot.achallenges module +========================== + +.. automodule:: certbot.achallenges + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.compat.filesystem.rst b/certbot/docs/api/certbot.compat.filesystem.rst new file mode 100644 index 000000000..d4f1e2fe0 --- /dev/null +++ b/certbot/docs/api/certbot.compat.filesystem.rst @@ -0,0 +1,7 @@ +certbot.compat.filesystem module +================================ + +.. automodule:: certbot.compat.filesystem + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.compat.misc.rst b/certbot/docs/api/certbot.compat.misc.rst new file mode 100644 index 000000000..35c2913e7 --- /dev/null +++ b/certbot/docs/api/certbot.compat.misc.rst @@ -0,0 +1,7 @@ +certbot.compat.misc module +========================== + +.. automodule:: certbot.compat.misc + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.compat.os.rst b/certbot/docs/api/certbot.compat.os.rst new file mode 100644 index 000000000..3a4c9fe47 --- /dev/null +++ b/certbot/docs/api/certbot.compat.os.rst @@ -0,0 +1,7 @@ +certbot.compat.os module +======================== + +.. automodule:: certbot.compat.os + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.compat.rst b/certbot/docs/api/certbot.compat.rst new file mode 100644 index 000000000..f6f2b3739 --- /dev/null +++ b/certbot/docs/api/certbot.compat.rst @@ -0,0 +1,17 @@ +certbot.compat package +====================== + +.. automodule:: certbot.compat + :members: + :undoc-members: + :show-inheritance: + +Submodules +---------- + +.. toctree:: + + certbot.compat.filesystem + certbot.compat.misc + certbot.compat.os + diff --git a/certbot/docs/api/certbot.crypto_util.rst b/certbot/docs/api/certbot.crypto_util.rst new file mode 100644 index 000000000..34aa665b9 --- /dev/null +++ b/certbot/docs/api/certbot.crypto_util.rst @@ -0,0 +1,7 @@ +certbot.crypto\_util module +=========================== + +.. automodule:: certbot.crypto_util + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.display.ops.rst b/certbot/docs/api/certbot.display.ops.rst new file mode 100644 index 000000000..544b0dad3 --- /dev/null +++ b/certbot/docs/api/certbot.display.ops.rst @@ -0,0 +1,7 @@ +certbot.display.ops module +========================== + +.. automodule:: certbot.display.ops + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.display.rst b/certbot/docs/api/certbot.display.rst new file mode 100644 index 000000000..04bc68b07 --- /dev/null +++ b/certbot/docs/api/certbot.display.rst @@ -0,0 +1,16 @@ +certbot.display package +======================= + +.. automodule:: certbot.display + :members: + :undoc-members: + :show-inheritance: + +Submodules +---------- + +.. toctree:: + + certbot.display.ops + certbot.display.util + diff --git a/certbot/docs/api/certbot.display.util.rst b/certbot/docs/api/certbot.display.util.rst new file mode 100644 index 000000000..22b59dc98 --- /dev/null +++ b/certbot/docs/api/certbot.display.util.rst @@ -0,0 +1,7 @@ +certbot.display.util module +=========================== + +.. automodule:: certbot.display.util + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.errors.rst b/certbot/docs/api/certbot.errors.rst new file mode 100644 index 000000000..731b7695d --- /dev/null +++ b/certbot/docs/api/certbot.errors.rst @@ -0,0 +1,7 @@ +certbot.errors module +===================== + +.. automodule:: certbot.errors + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.interfaces.rst b/certbot/docs/api/certbot.interfaces.rst new file mode 100644 index 000000000..2665aaa01 --- /dev/null +++ b/certbot/docs/api/certbot.interfaces.rst @@ -0,0 +1,7 @@ +certbot.interfaces module +========================= + +.. automodule:: certbot.interfaces + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.main.rst b/certbot/docs/api/certbot.main.rst new file mode 100644 index 000000000..ce0539f5c --- /dev/null +++ b/certbot/docs/api/certbot.main.rst @@ -0,0 +1,7 @@ +certbot.main module +=================== + +.. automodule:: certbot.main + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.plugins.common.rst b/certbot/docs/api/certbot.plugins.common.rst new file mode 100644 index 000000000..e94b2d12e --- /dev/null +++ b/certbot/docs/api/certbot.plugins.common.rst @@ -0,0 +1,7 @@ +certbot.plugins.common module +============================= + +.. automodule:: certbot.plugins.common + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.plugins.dns_common.rst b/certbot/docs/api/certbot.plugins.dns_common.rst new file mode 100644 index 000000000..36c7a6428 --- /dev/null +++ b/certbot/docs/api/certbot.plugins.dns_common.rst @@ -0,0 +1,7 @@ +certbot.plugins.dns\_common module +================================== + +.. automodule:: certbot.plugins.dns_common + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.plugins.dns_common_lexicon.rst b/certbot/docs/api/certbot.plugins.dns_common_lexicon.rst new file mode 100644 index 000000000..1a961accd --- /dev/null +++ b/certbot/docs/api/certbot.plugins.dns_common_lexicon.rst @@ -0,0 +1,7 @@ +certbot.plugins.dns\_common\_lexicon module +=========================================== + +.. automodule:: certbot.plugins.dns_common_lexicon + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.plugins.dns_test_common.rst b/certbot/docs/api/certbot.plugins.dns_test_common.rst new file mode 100644 index 000000000..69e672f0a --- /dev/null +++ b/certbot/docs/api/certbot.plugins.dns_test_common.rst @@ -0,0 +1,7 @@ +certbot.plugins.dns\_test\_common module +======================================== + +.. automodule:: certbot.plugins.dns_test_common + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.plugins.dns_test_common_lexicon.rst b/certbot/docs/api/certbot.plugins.dns_test_common_lexicon.rst new file mode 100644 index 000000000..92d516c99 --- /dev/null +++ b/certbot/docs/api/certbot.plugins.dns_test_common_lexicon.rst @@ -0,0 +1,7 @@ +certbot.plugins.dns\_test\_common\_lexicon module +================================================= + +.. automodule:: certbot.plugins.dns_test_common_lexicon + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.plugins.enhancements.rst b/certbot/docs/api/certbot.plugins.enhancements.rst new file mode 100644 index 000000000..16db737c7 --- /dev/null +++ b/certbot/docs/api/certbot.plugins.enhancements.rst @@ -0,0 +1,7 @@ +certbot.plugins.enhancements module +=================================== + +.. automodule:: certbot.plugins.enhancements + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.plugins.rst b/certbot/docs/api/certbot.plugins.rst new file mode 100644 index 000000000..517a209e6 --- /dev/null +++ b/certbot/docs/api/certbot.plugins.rst @@ -0,0 +1,22 @@ +certbot.plugins package +======================= + +.. automodule:: certbot.plugins + :members: + :undoc-members: + :show-inheritance: + +Submodules +---------- + +.. toctree:: + + certbot.plugins.common + certbot.plugins.dns_common + certbot.plugins.dns_common_lexicon + certbot.plugins.dns_test_common + certbot.plugins.dns_test_common_lexicon + certbot.plugins.enhancements + certbot.plugins.storage + certbot.plugins.util + diff --git a/certbot/docs/api/certbot.plugins.storage.rst b/certbot/docs/api/certbot.plugins.storage.rst new file mode 100644 index 000000000..9ed0fe724 --- /dev/null +++ b/certbot/docs/api/certbot.plugins.storage.rst @@ -0,0 +1,7 @@ +certbot.plugins.storage module +============================== + +.. automodule:: certbot.plugins.storage + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.plugins.util.rst b/certbot/docs/api/certbot.plugins.util.rst new file mode 100644 index 000000000..c5453564e --- /dev/null +++ b/certbot/docs/api/certbot.plugins.util.rst @@ -0,0 +1,7 @@ +certbot.plugins.util module +=========================== + +.. automodule:: certbot.plugins.util + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.reverter.rst b/certbot/docs/api/certbot.reverter.rst new file mode 100644 index 000000000..002b75360 --- /dev/null +++ b/certbot/docs/api/certbot.reverter.rst @@ -0,0 +1,7 @@ +certbot.reverter module +======================= + +.. automodule:: certbot.reverter + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.rst b/certbot/docs/api/certbot.rst new file mode 100644 index 000000000..6f5b4b403 --- /dev/null +++ b/certbot/docs/api/certbot.rst @@ -0,0 +1,31 @@ +certbot package +=============== + +.. automodule:: certbot + :members: + :undoc-members: + :show-inheritance: + +Subpackages +----------- + +.. toctree:: + + certbot.compat + certbot.display + certbot.plugins + certbot.tests + +Submodules +---------- + +.. toctree:: + + certbot.achallenges + certbot.crypto_util + certbot.errors + certbot.interfaces + certbot.main + certbot.reverter + certbot.util + diff --git a/certbot/docs/api/certbot.tests.acme_util.rst b/certbot/docs/api/certbot.tests.acme_util.rst new file mode 100644 index 000000000..908397596 --- /dev/null +++ b/certbot/docs/api/certbot.tests.acme_util.rst @@ -0,0 +1,7 @@ +certbot.tests.acme\_util module +=============================== + +.. automodule:: certbot.tests.acme_util + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.tests.rst b/certbot/docs/api/certbot.tests.rst new file mode 100644 index 000000000..336f0eabc --- /dev/null +++ b/certbot/docs/api/certbot.tests.rst @@ -0,0 +1,16 @@ +certbot.tests package +===================== + +.. automodule:: certbot.tests + :members: + :undoc-members: + :show-inheritance: + +Submodules +---------- + +.. toctree:: + + certbot.tests.acme_util + certbot.tests.util + diff --git a/certbot/docs/api/certbot.tests.util.rst b/certbot/docs/api/certbot.tests.util.rst new file mode 100644 index 000000000..3f0335849 --- /dev/null +++ b/certbot/docs/api/certbot.tests.util.rst @@ -0,0 +1,7 @@ +certbot.tests.util module +========================= + +.. automodule:: certbot.tests.util + :members: + :undoc-members: + :show-inheritance: diff --git a/certbot/docs/api/certbot.util.rst b/certbot/docs/api/certbot.util.rst new file mode 100644 index 000000000..11cb33b09 --- /dev/null +++ b/certbot/docs/api/certbot.util.rst @@ -0,0 +1,7 @@ +certbot.util module +=================== + +.. automodule:: certbot.util + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/challenges.rst b/certbot/docs/challenges.rst similarity index 100% rename from docs/challenges.rst rename to certbot/docs/challenges.rst diff --git a/docs/ciphers.rst b/certbot/docs/ciphers.rst similarity index 98% rename from docs/ciphers.rst rename to certbot/docs/ciphers.rst index 1b320cdf9..04b24b526 100644 --- a/docs/ciphers.rst +++ b/certbot/docs/ciphers.rst @@ -227,7 +227,7 @@ BetterCrypto.org BetterCrypto.org, a collaboration of mostly European IT security experts, has published a draft paper, "Applied Crypto Hardening" -https://bettercrypto.org/static/applied-crypto-hardening.pdf +https://bettercrypto.org/ FF-DHE Internet-Draft ~~~~~~~~~~~~~~~~~~~~~ @@ -248,7 +248,7 @@ Dutch National Cyber Security Centre The Dutch National Cyber Security Centre has published guidance on "ICT-beveiligingsrichtlijnen voor Transport Layer Security (TLS)" ("IT Security Guidelines for Transport Layer Security (TLS)"). These are available only in Dutch at -https://www.ncsc.nl/dienstverlening/expertise-advies/kennisdeling/whitepapers/ict-beveiligingsrichtlijnen-voor-transport-layer-security-tls.html +https://web.archive.org/web/20190516085116/https://www.ncsc.nl/actueel/whitepapers/ict-beveiligingsrichtlijnen-voor-transport-layer-security-tls.html I have access to an English-language summary of the recommendations. @@ -286,7 +286,7 @@ https://weakdh.org/sysadmin.html These lists may have been derived from Mozilla's recommendations. One of the authors clarified his view of the priorities for various changes as a result of the research at -https://www.ietf.org/mail-archive/web/tls/current/msg16496.html +https://web.archive.org/web/20150526022820/https://www.ietf.org/mail-archive/web/tls/current/msg16496.html In particular, he supports ECDHE and also supports the use of the standardized groups in the FF-DHE Internet-Draft mentioned above (which isn't clear from the group's original recommendations). diff --git a/docs/cli-help.txt b/certbot/docs/cli-help.txt similarity index 93% rename from docs/cli-help.txt rename to certbot/docs/cli-help.txt index cd6d431b3..ff49609c4 100644 --- a/docs/cli-help.txt +++ b/certbot/docs/cli-help.txt @@ -24,12 +24,13 @@ obtain, install, and renew certificates: manage certificates: certificates Display information about certificates you have from Certbot - revoke Revoke a certificate (supply --cert-path or --cert-name) - delete Delete a certificate + revoke Revoke a certificate (supply --cert-name or --cert-path) + delete Delete a certificate (supply --cert-name) -manage your account with Let's Encrypt: - register Create a Let's Encrypt ACME account - update_account Update a Let's Encrypt ACME account +manage your account: + register Create an ACME account + unregister Deactivate an ACME account + update_account Update an ACME account --agree-tos Agree to the ACME server's Subscriber Agreement -m EMAIL Email address for important account notifications @@ -99,11 +100,10 @@ optional arguments: --preferred-challenges PREF_CHALLS A sorted, comma delimited list of the preferred challenge to use during authorization with the most - preferred challenge listed first (Eg, "dns" or "tls- - sni-01,http,dns"). Not all plugins support all - challenges. See - https://certbot.eff.org/docs/using.html#plugins for - details. ACME Challenges are versioned, but if you + preferred challenge listed first (Eg, "dns" or + "http,dns"). Not all plugins support all challenges. + See https://certbot.eff.org/docs/using.html#plugins + for details. ACME Challenges are versioned, but if you pick "http" rather than "http-01", Certbot will select the latest version automatically. (default: []) --user-agent USER_AGENT @@ -113,12 +113,12 @@ optional arguments: case, and to know when to deprecate support for past Python versions and flags. If you wish to hide this information from the Let's Encrypt server, set this to - "". (default: CertbotACMEClient/0.30.2 - (certbot(-auto); OS_NAME OS_VERSION) Authenticator/XXX - Installer/YYY (SUBCOMMAND; flags: FLAGS) - Py/major.minor.patchlevel). The flags encoded in the - user agent are: --duplicate, --force-renew, --allow- - subset-of-names, -n, and whether any hooks are set. + "". (default: CertbotACMEClient/1.2.0 (certbot(-auto); + OS_NAME OS_VERSION) Authenticator/XXX Installer/YYY + (SUBCOMMAND; flags: FLAGS) Py/major.minor.patchlevel). + The flags encoded in the user agent are: --duplicate, + --force-renew, --allow-subset-of-names, -n, and + whether any hooks are set. --user-agent-comment USER_AGENT_COMMENT Add a comment to the default user agent string. May be used when repackaging Certbot or calling it from @@ -171,6 +171,10 @@ automation: from installing OS-level dependencies (default: Prompt to install OS-wide dependencies, but exit if the user says 'No') + --no-permissions-check + (certbot-auto only) skip the check on the file system + permissions of the certbot-auto script (default: + False) -q, --quiet Silence all output except errors. Useful for automation via cron. Implies --non-interactive. (default: False) @@ -217,14 +221,6 @@ testing: False) --no-verify-ssl Disable verification of the ACME server's certificate. (default: False) - --tls-sni-01-port TLS_SNI_01_PORT - Port used during tls-sni-01 challenge. This only - affects the port Certbot listens on. A conforming ACME - server will still attempt to connect on port 443. - (default: 443) - --tls-sni-01-address TLS_SNI_01_ADDRESS - The address the server listens to during tls-sni-01 - challenge. (default: ) --http-01-port HTTP01_PORT Port used in the http-01 challenge. This only affects the port Certbot listens on. A conforming ACME server @@ -233,6 +229,10 @@ testing: --http-01-address HTTP01_ADDRESS The address the server listens to during http-01 challenge. (default: ) + --https-port HTTPS_PORT + Port used to serve HTTPS. This affects which port + Nginx will listen on after a LE certificate is + installed. (default: 443) --break-my-certs Be willing to replace or renew valid certificates with invalid (testing/staging) certificates (default: False) @@ -351,8 +351,9 @@ revoke: Specify reason for revoking certificate. (default: unspecified) --delete-after-revoke - Delete certificates after revoking them. (default: - None) + Delete certificates after revoking them, along with + all previous and later versions of those certificates. + (default: None) --no-delete-after-revoke Do not delete certificates after revoking them. This option should be used with caution because the 'renew' @@ -391,12 +392,6 @@ unregister: install: Options for modifying how a certificate is deployed -config_changes: - Options for controlling which changes are displayed - - --num NUM How many past revisions you want to be displayed - (default: None) - rollback: Options for rolling back server configuration changes @@ -404,7 +399,7 @@ rollback: (default: 1) plugins: - Options for for the "plugins" subcommand + Options for the "plugins" subcommand --init Initialize plugins. (default: False) --prepare Initialize and prepare plugins. (default: False) @@ -453,10 +448,10 @@ plugins: using DigitalOcean for DNS). (default: False) --dns-dnsimple Obtain certificates using a DNS TXT record (if you are using DNSimple for DNS). (default: False) - --dns-dnsmadeeasy Obtain certificates using a DNS TXT record (if you - areusing DNS Made Easy for DNS). (default: False) + --dns-dnsmadeeasy Obtain certificates using a DNS TXT record (if you are + using DNS Made Easy for DNS). (default: False) --dns-gehirn Obtain certificates using a DNS TXT record (if you are - using Gehirn Infrastracture Service for DNS). + using Gehirn Infrastructure Service for DNS). (default: False) --dns-google Obtain certificates using a DNS TXT record (if you are using Google Cloud DNS). (default: False) @@ -476,13 +471,14 @@ plugins: using Sakura Cloud for DNS). (default: False) apache: - Apache Web Server plugin + Apache Web Server plugin (Please note that the default values of the + Apache plugin options change depending on the operating system Certbot is + run on.) --apache-enmod APACHE_ENMOD - Path to the Apache 'a2enmod' binary (default: a2enmod) + Path to the Apache 'a2enmod' binary (default: None) --apache-dismod APACHE_DISMOD - Path to the Apache 'a2dismod' binary (default: - a2dismod) + Path to the Apache 'a2dismod' binary (default: None) --apache-le-vhost-ext APACHE_LE_VHOST_EXT SSL vhost configuration extension (default: -le- ssl.conf) @@ -499,10 +495,10 @@ apache: /etc/apache2) --apache-handle-modules APACHE_HANDLE_MODULES Let installer handle enabling required modules for you - (Only Ubuntu/Debian currently) (default: True) + (Only Ubuntu/Debian currently) (default: False) --apache-handle-sites APACHE_HANDLE_SITES Let installer handle enabling sites for you (Only - Ubuntu/Debian currently) (default: True) + Ubuntu/Debian currently) (default: False) --apache-ctl APACHE_CTL Full path to Apache control script (default: apache2ctl) @@ -564,14 +560,14 @@ dns-dnsmadeeasy: dns-gehirn: Obtain certificates using a DNS TXT record (if you are using Gehirn - Infrastracture Service for DNS). + Infrastructure Service for DNS). --dns-gehirn-propagation-seconds DNS_GEHIRN_PROPAGATION_SECONDS The number of seconds to wait for DNS to propagate before asking the ACME server to verify the DNS record. (default: 30) --dns-gehirn-credentials DNS_GEHIRN_CREDENTIALS - Gehirn Infrastracture Service credentials file. + Gehirn Infrastructure Service credentials file. (default: None) dns-google: @@ -670,12 +666,8 @@ manual: challenge. $CERTBOT_DOMAIN will always contain the domain being authenticated. For HTTP-01 and DNS-01, $CERTBOT_VALIDATION is the validation string, and $CERTBOT_TOKEN is the filename of the resource - requested when performing an HTTP-01 challenge. When performing a TLS- - SNI-01 challenge, $CERTBOT_SNI_DOMAIN will contain the SNI name for which - the ACME server expects to be presented with the self-signed certificate - located at $CERTBOT_CERT_PATH. The secret key needed to complete the TLS - handshake is located at $CERTBOT_KEY_PATH. An additional cleanup script - can also be provided and can use the additional variable + requested when performing an HTTP-01 challenge. An additional cleanup + script can also be provided and can use the additional variable $CERTBOT_AUTH_OUTPUT which contains the stdout output from the auth script. diff --git a/certbot/docs/compatibility.rst b/certbot/docs/compatibility.rst new file mode 100644 index 000000000..a511f36a2 --- /dev/null +++ b/certbot/docs/compatibility.rst @@ -0,0 +1,39 @@ +======================= +Backwards Compatibility +======================= + +All Certbot components including `acme `_, +Certbot, and :ref:`non-third party plugins ` follow `Semantic +Versioning `_ both for its Python :doc:`API ` and for the +application itself. This means that we will not change behavior in a backwards +incompatible way except in a new major version of the project. + +.. note:: None of this applies to the behavior of Certbot distribution + mechanisms such as :ref:`certbot-auto ` or OS packages whose + behavior may change at any time. Semantic versioning only applies to the + common Certbot components that are installed by various distribution + methods. + +For Certbot as an application, the command line interface and non-interactive +behavior can be considered stable with two exceptions. The first is that no +aspects of Certbot's console or log output should be considered stable and it +may change at any time. The second is that Certbot's behavior should only be +considered stable with certain files but not all. Files with which users should +expect Certbot to maintain its current behavior with are: + +* ``/etc/letsencrypt/live//{cert,chain,fullchain,privkey}.pem`` where + ```` is the name given to ``--cert-name``. If ``--cert-name`` is not + set by the user, it is the first domain given to ``--domains``. +* :ref:`CLI configuration files ` +* Hook directories in ``/etc/letsencrypt/renewal-hooks`` + +Certbot's behavior with other files may change at any point. + +Another area where Certbot should not be considered stable is its behavior when +not run in non-interactive mode which also may change at any point. + +In general, if we're making a change that we expect will break some users, we +will bump the major version and will have warned about it in a prior release +when possible. For our Python API, we will issue warnings using Python's +warning module. For application level changes, we will print and log warning +messages. diff --git a/docs/conf.py b/certbot/docs/conf.py similarity index 99% rename from docs/conf.py rename to certbot/docs/conf.py index c72d1c1cf..1e57bc224 100644 --- a/docs/conf.py +++ b/certbot/docs/conf.py @@ -19,7 +19,6 @@ import sys import sphinx - here = os.path.abspath(os.path.dirname(__file__)) # read version number (and other metadata) from package init @@ -53,7 +52,7 @@ if sphinx.version_info >= (1, 6): extensions.append('sphinx.ext.imgconverter') autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/docs/contributing.rst b/certbot/docs/contributing.rst similarity index 65% rename from docs/contributing.rst rename to certbot/docs/contributing.rst index 264db630f..25d832761 100644 --- a/docs/contributing.rst +++ b/certbot/docs/contributing.rst @@ -17,6 +17,8 @@ its dependencies, Certbot needs to be run on a UNIX-like OS so if you're using Windows, you'll need to set up a (virtual) machine running an OS such as Linux and continue with these instructions on that UNIX-like OS. +.. _local copy: + Running a local copy of the client ---------------------------------- @@ -34,29 +36,36 @@ run Certbot in Docker. You can find instructions for how to do this :ref:`here install dependencies and set up a virtual environment where you can run Certbot. +Install the OS system dependencies required to run Certbot. + +.. code-block:: shell + + # For APT-based distributions (e.g. Debian, Ubuntu ...) + sudo apt update + sudo apt install python3-dev python3-venv gcc libaugeas0 libssl-dev \ + libffi-dev ca-certificates openssl + # For RPM-based distributions (e.g. Fedora, CentOS ...) + # NB1: old distributions will use yum instead of dnf + # NB2: RHEL-based distributions use python3X-devel instead of python3-devel (e.g. python36-devel) + sudo dnf install python3-devel gcc augeas-libs openssl-devel libffi-devel \ + redhat-rpm-config ca-certificates openssl + +Set up the Python virtual environment that will host your Certbot local instance. + .. code-block:: shell cd certbot - ./certbot-auto --debug --os-packages-only - python tools/venv.py - -If you have Python3 available and want to use it, run the ``venv3.py`` script. - -.. code-block:: shell - python tools/venv3.py .. note:: You may need to repeat this when Certbot's dependencies change or when a new plugin is introduced. You can now run the copy of Certbot from git either by executing -``venv/bin/certbot``, or by activating the virtual environment. You can do the +``venv3/bin/certbot``, or by activating the virtual environment. You can do the latter by running: .. code-block:: shell - source venv/bin/activate - # or source venv3/bin/activate After running this command, ``certbot`` and development tools like ``ipdb``, @@ -89,6 +98,17 @@ tests, and be compliant with the :ref:`coding style `. Testing ------- +You can test your code in several ways: + +- running the `automated unit`_ tests, +- running the `automated integration`_ tests +- running an *ad hoc* `manual integration`_ test + +.. _automated unit: + +Running automated unit tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + When you are working in a file ``foo.py``, there should also be a file ``foo_test.py`` either in the same directory as ``foo.py`` or in the ``tests`` subdirectory (if there isn't, make one). While you are working on your code and tests, run @@ -101,9 +121,9 @@ Once you are done with your code changes, and the tests in ``foo_test.py`` pass, run all of the unittests for Certbot with ``tox -e py27`` (this uses Python 2.7). -Once all the unittests pass, check for sufficient test coverage using -``tox -e cover``, and then check for code style with ``tox -e lint`` (all files) -or ``pylint --rcfile=.pylintrc path/to/file.py`` (single file at a time). +Once all the unittests pass, check for sufficient test coverage using ``tox -e +py27-cover``, and then check for code style with ``tox -e lint`` (all files) or +``pylint --rcfile=.pylintrc path/to/file.py`` (single file at a time). Once all of the above is successful, you may run the full test suite using ``tox --skip-missing-interpreters``. We recommend running the commands above @@ -114,51 +134,100 @@ of output can make it hard to find specific failures when they happen. config if your user has sudo permissions, so it should not be run on a production Apache server. -.. _integration: +.. _automated integration: -Integration testing with the Boulder CA -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Running automated integration tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Generally it is sufficient to open a pull request and let Github and Travis run -integration tests for you, however, if you want to run them locally you need -Docker and docker-compose installed and working. Fetch and start Boulder, Let's -Encrypt's ACME CA software, by using: +integration tests for you. However, you may want to run them locally before submitting +your pull request. You need Docker and docker-compose installed and working. + +The tox environment `integration` will setup `Pebble`_, the Let's Encrypt ACME CA server +for integration testing, then launch the Certbot integration tests. + +With a user allowed to access your local Docker daemon, run: .. code-block:: shell - ./tests/boulder-fetch.sh + tox -e integration -If you have problems with Docker, you may want to try `removing all containers and -volumes`_ and making sure you have at least 1GB of memory. +Tests will be run using pytest. A test report and a code coverage report will be +displayed at the end of the integration tests execution. -Set up a certbot_test alias that enables easily running against the local -Boulder: +.. _Pebble: https://github.com/letsencrypt/pebble + +.. _manual integration: + +Running manual integration tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can also manually execute Certbot against a local instance of the `Pebble`_ ACME server. +This is useful to verify that the modifications done to the code makes Certbot behave as expected. + +To do so you need: + +- Docker installed, and a user with access to the Docker client, +- an available `local copy`_ of Certbot. + +The virtual environment set up with `python tools/venv.py` contains two commands +that can be used once the virtual environment is activated: .. code-block:: shell - export SERVER=http://localhost:4000/directory - source tests/integration/_common.sh + run_acme_server -Run the integration tests using: +- Starts a local instance of Pebble and runs in the foreground printing its logs. +- Press CTRL+C to stop this instance. +- This instance is configured to validate challenges against certbot executed locally. .. code-block:: shell - ./tests/boulder-integration.sh + certbot_test [ARGS...] -.. _removing all containers and volumes: https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes +- Execute certbot with the provided arguments and other arguments useful for testing purposes, + such as: verbose output, full tracebacks in case Certbot crashes, *etc.* +- Execution is preconfigured to interact with the Pebble CA started with ``run_acme_server``. +- Any arguments can be passed as they would be to Certbot (eg. ``certbot_test certonly -d test.example.com``). + +Here is a typical workflow to verify that Certbot successfully issued a certificate +using an HTTP-01 challenge on a machine with Python 3: + +.. code-block:: shell + + python tools/venv3.py + source venv3/bin/activate + run_acme_server & + certbot_test certonly --standalone -d test.example.com + # To stop Pebble, launch `fg` to get back the background job, then press CTRL+C + +Running tests in CI +~~~~~~~~~~~~~~~~~~~ + +Certbot uses both Azure Pipelines and Travis to run continuous integration +tests. If you are using our Azure and Travis setup, a branch whose name starts +with `test-` will run all Azure and Travis tests on that branch. If the branch +name starts with `azure-test-`, it will run all of our Azure tests and none of +our Travis tests. If the branch stats with `travis-test-`, only our Travis +tests will be run. Code components and layout ========================== +The following components of the Certbot repository are distributed to users: + acme contains all protocol specific code certbot main client code certbot-apache and certbot-nginx client code to configure specific web servers -certbot.egg-info - configuration for packaging Certbot - +certbot-dns-* + client code to configure DNS providers +certbot-auto and letsencrypt-auto + shell scripts to install Certbot and its dependencies on UNIX systems +windows installer + Installs Certbot on Windows and is built using the files in windows-installer/ Plugin-architecture ------------------- @@ -187,7 +256,7 @@ Authenticators Authenticators are plugins that prove control of a domain name by solving a challenge provided by the ACME server. ACME currently defines several types of -challenges: HTTP, TLS-SNI (deprecated), TLS-ALPR, and DNS, represented by classes in `acme.challenges`. +challenges: HTTP, TLS-ALPN, and DNS, represented by classes in `acme.challenges`. An authenticator plugin should implement support for at least one challenge type. An Authenticator indicates which challenges it supports by implementing @@ -241,6 +310,16 @@ configuration checkpoints and rollback. Writing your own plugin ~~~~~~~~~~~~~~~~~~~~~~~ +.. note:: The Certbot team is not currently accepting any new DNS plugins + because we want to rethink our approach to the challenge and resolve some + issues like `#6464 `_, + `#6503 `_, and `#6504 + `_ first. + + In the meantime, you're welcome to release it as a third-party plugin. See + `certbot-dns-ispconfig `_ + for one example of that. + Certbot client supports dynamic discovery of plugins through the `setuptools entry points`_ using the `certbot.plugins` group. This way you can, for example, create a custom implementation of @@ -255,7 +334,6 @@ virtualenv like this: .. code-block:: shell . venv/bin/activate - . tests/integration/_common.sh pip install -e examples/plugins/ certbot_test plugins @@ -270,12 +348,6 @@ plugins. It's technically possible to install third-party plugins into the virtualenv used by `certbot-auto`, but they will be wiped away when `certbot-auto` upgrades. -.. warning:: Please be aware though that as this client is still in a - developer-preview stage, the API may undergo a few changes. If you - believe the plugin will be beneficial to the community, please - consider submitting a pull request to the repo and we will update - it with any necessary API changes. - .. _`setuptools entry points`: http://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points @@ -312,6 +384,25 @@ Please: .. _PEP 8 - Style Guide for Python Code: https://www.python.org/dev/peps/pep-0008 +Use ``certbot.compat.os`` instead of ``os`` +=========================================== + + +Python's standard library ``os`` module lacks full support for several Windows +security features about file permissions (eg. DACLs). However several files +handled by Certbot (eg. private keys) need strongly restricted access +on both Linux and Windows. + +To help with this, the ``certbot.compat.os`` module wraps the standard +``os`` module, and forbids usage of methods that lack support for these Windows +security features. + +As a developer, when working on Certbot or its plugins, you must use ``certbot.compat.os`` +in every place you would need ``os`` (eg. ``from certbot.compat import os`` instead of +``import os``). Otherwise the tests will fail when your PR is submitted. + +.. _type annotations: + Mypy type annotations ===================== @@ -332,7 +423,7 @@ Note that instead of just importing ``typing``, due to packaging issues, in Cert .. code-block:: python - from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module + from acme.magic_typing import Dict Also note that OpenSSL, which we rely on, has type definitions for crypto but not SSL. We use both. Those imports should look like this: @@ -351,10 +442,13 @@ Submitting a pull request Steps: -1. Write your code! +1. Write your code! When doing this, you should add :ref:`mypy type annotations + ` for any functions you add or modify. You can check that + you've done this correctly by running ``tox -e mypy`` on a machine that has + Python 3 installed. 2. Make sure your environment is set up properly and that you're in your - virtualenv. You can do this by running ``pip tools/venv.py``. - (this is a **very important** step) + virtualenv. You can do this by following the instructions in the + :ref:`Getting Started ` section. 3. Run ``tox -e lint`` to check for pylint errors. Fix any errors. 4. Run ``tox --skip-missing-interpreters`` to run the entire test suite including coverage. The ``--skip-missing-interpreters`` argument ignores @@ -365,17 +459,39 @@ Steps: rewriting commits makes changes harder to track between reviews. 6. Did your tests pass on Travis? If they didn't, fix any errors. +.. _ask for help: + Asking for help =============== If you have any questions while working on a Certbot issue, don't hesitate to -ask for help! You can do this in the #letsencrypt-dev IRC channel on Freenode. -If you don't already have an IRC client set up, we recommend you join using -`Riot `_. +ask for help! You can do this in the Certbot channel in EFF's Mattermost +instance for its open source projects as described below. + +You can get involved with several of EFF's software projects such as Certbot at +the `EFF Open Source Contributor Chat Platform +`_. +By signing up for the EFF Open Source Contributor Chat Platform, you consent to +share your personal information with the Electronic Frontier Foundation, which +is the operator and data controller for this platform. The channels will be +available both to EFF, and to other users of EFFOSCCP, who may use or disclose +information in these channels outside of EFFOSCCP. EFF will use your +information, according to the `Privacy Policy `_, +to further the mission of EFF, including hosting and moderating the discussions +on this platform. + +Use of EFFOSCCP is subject to the `EFF Code of Conduct +`_. When investigating an alleged Code of +Conduct violation, EFF may review discussion channels or direct messages. Updating certbot-auto and letsencrypt-auto ========================================== +.. note:: We are currently only accepting changes to certbot-auto that fix + regressions on platforms where certbot-auto is the recommended installation + method at https://certbot.eff.org/instructions. If you are unsure if a change + you want to make qualifies, don't hesitate to `ask for help`_! + Updating the scripts -------------------- Developers should *not* modify the ``certbot-auto`` and ``letsencrypt-auto`` files @@ -418,19 +534,22 @@ during the next release. Updating the documentation ========================== -In order to generate the Sphinx documentation, run the following -commands: +Many of the packages in the Certbot repository have documentation in a +``docs/`` directory. This directory is located under the top level directory +for the package. For instance, Certbot's documentation is under +``certbot/docs``. + +To build the documentation of a package, make sure you have followed the +instructions to set up a `local copy`_ of Certbot including activating the +virtual environment. After that, ``cd`` to the docs directory you want to build +and run the command: .. code-block:: shell - make -C docs clean html man - -This should generate documentation in the ``docs/_build/html`` -directory. - -.. note:: If you skipped the "Getting Started" instructions above, - run ``pip install -e ".[docs]"`` to install Certbot's docs extras modules. + make clean html +This would generate the HTML documentation in ``_build/html`` in your current +``docs/`` directory. .. _docker-dev: @@ -477,7 +596,7 @@ OS-level dependencies can be installed like so: In general... * ``sudo`` is required as a suggested way of running privileged process -* `Python`_ 2.7 or 3.4+ is required +* `Python`_ 2.7 or 3.5+ is required * `Augeas`_ is required for the Python bindings * ``virtualenv`` is used for managing other Python library dependencies diff --git a/docs/index.rst b/certbot/docs/index.rst similarity index 95% rename from docs/index.rst rename to certbot/docs/index.rst index 17cde1adf..a7fc75c5b 100644 --- a/docs/index.rst +++ b/certbot/docs/index.rst @@ -10,6 +10,7 @@ Welcome to the Certbot documentation! using contributing packaging + compatibility resources .. toctree:: diff --git a/docs/install.rst b/certbot/docs/install.rst similarity index 77% rename from docs/install.rst rename to certbot/docs/install.rst index 35b262482..11994776c 100644 --- a/docs/install.rst +++ b/certbot/docs/install.rst @@ -11,6 +11,8 @@ About Certbot *Certbot is meant to be run directly on a web server*, normally by a system administrator. In most cases, running Certbot on your personal computer is not a useful option. The instructions below relate to installing and running Certbot on a server. +System administrators can use Certbot directly to request certificates; they should *not* allow unprivileged users to run arbitrary Certbot commands as ``root``, because Certbot allows its user to specify arbitrary file locations and run arbitrary scripts. + Certbot is packaged for many common operating systems and web servers. Check whether ``certbot`` (or ``letsencrypt``) is packaged for your web server's OS by visiting certbot.eff.org_, where you will also find the correct installation instructions for @@ -26,7 +28,7 @@ your system. System Requirements =================== -Certbot currently requires Python 2.7 or 3.4+ running on a UNIX-like operating +Certbot currently requires Python 2.7 or 3.5+ running on a UNIX-like operating system. By default, it requires root access in order to write to ``/etc/letsencrypt``, ``/var/log/letsencrypt``, ``/var/lib/letsencrypt``; to bind to port 80 (if you use the ``standalone`` plugin) and to read and @@ -39,8 +41,8 @@ client as root, either `letsencrypt-nosudo The Apache plugin currently requires an OS with augeas version 1.0; currently `it supports -`_ -modern OSes based on Debian, Fedora, SUSE, Gentoo and Darwin. +`_ +modern OSes based on Debian, Ubuntu, Fedora, SUSE, Gentoo and Darwin. Additional integrity verification of certbot-auto script can be done by verifying its digital signature. @@ -68,9 +70,13 @@ The ``certbot-auto`` wrapper script installs Certbot, obtaining some dependencie from your web server OS and putting others in a python virtual environment. You can download and run it as follows:: - user@webserver:~$ wget https://dl.eff.org/certbot-auto - user@webserver:~$ chmod a+x ./certbot-auto - user@webserver:~$ ./certbot-auto --help + wget https://dl.eff.org/certbot-auto + sudo mv certbot-auto /usr/local/bin/certbot-auto + sudo chown root /usr/local/bin/certbot-auto + sudo chmod 0755 /usr/local/bin/certbot-auto + /usr/local/bin/certbot-auto --help + +To remove certbot-auto, just delete it and the files it places under /opt/eff.org, along with any cronjob or systemd timer you may have created. To check the integrity of the ``certbot-auto`` script, you can use these steps:: @@ -78,7 +84,7 @@ you can use these steps:: user@webserver:~$ wget -N https://dl.eff.org/certbot-auto.asc user@webserver:~$ gpg2 --keyserver pool.sks-keyservers.net --recv-key A2CFB51FA275A7286234E7B24D17C995CD9775F2 - user@webserver:~$ gpg2 --trusted-key 4D17C995CD9775F2 --verify certbot-auto.asc certbot-auto + user@webserver:~$ gpg2 --trusted-key 4D17C995CD9775F2 --verify certbot-auto.asc /usr/local/bin/certbot-auto @@ -104,7 +110,7 @@ the same command line flags and arguments. For more information, see For full command line help, you can type:: - ./certbot-auto --help all + /usr/local/bin/certbot-auto --help all Problems with Python virtual environment ---------------------------------------- @@ -196,23 +202,64 @@ Operating System Packages **Debian** -If you run Debian Stretch or Debian Sid, you can install certbot packages. +If you run Debian Buster or Debian testing/Sid, you can easily install certbot +packages through commands like: .. code-block:: shell sudo apt-get update - sudo apt-get install certbot python-certbot-apache + sudo apt-get install certbot -If you don't want to use the Apache plugin, you can omit the -``python-certbot-apache`` package. Or you can install ``python-certbot-nginx`` instead. - -Packages exist for Debian Jessie via backports. First you'll have to follow the -instructions at http://backports.debian.org/Instructions/ to enable the Jessie backports -repo, if you have not already done so. Then run: +If you run Debian Stretch, we recommend you use the packages in Debian +backports repository. First you'll have to follow the instructions at +https://backports.debian.org/Instructions/ to enable the Stretch backports repo, +if you have not already done so. Then run: .. code-block:: shell - sudo apt-get install certbot python-certbot-apache -t jessie-backports + sudo apt-get install certbot -t stretch-backports + +In all of these cases, there also packages available to help Certbot integrate +with Apache, nginx, or various DNS services. If you are using Apache or nginx, +we strongly recommend that you install the ``python-certbot-apache`` or +``python-certbot-nginx`` package so that Certbot can fully automate HTTPS +configuration for your server. A full list of these packages can be found +through a command like: + +.. code-block:: shell + + apt search 'python-certbot*' + +They can be installed by running the same installation command above but +replacing ``certbot`` with the name of the desired package. + +There are no Certbot packages available for Debian Jessie and Jessie users +should instead use certbot-auto_. + +**Ubuntu** + +If you run Ubuntu Trusty, Xenial, or Bionic, certbot is available through the official PPA, +that can be installed as followed: + +.. code-block:: shell + + sudo apt-get update + sudo apt-get install software-properties-common + sudo add-apt-repository universe + sudo add-apt-repository ppa:certbot/certbot + sudo apt-get update + +Then, certbot can be installed using: + +.. code-block:: shell + + sudo apt-get install certbot + +Optionally to install the Certbot Apache plugin, you can use: + +.. code-block:: shell + + sudo apt-get install python-certbot-apache **Fedora** @@ -283,9 +330,9 @@ Installing from source Installation from source is only supported for developers and the whole process is described in the :doc:`contributing`. -.. warning:: Please do **not** use ``python setup.py install``, ``python pip - install .``, or ``easy_install .``. Please do **not** attempt the +.. warning:: Please do **not** use ``python certbot/setup.py install``, ``python pip + install certbot``, or ``easy_install certbot``. Please do **not** attempt the installation commands as superuser/root and/or without virtual environment, - e.g. ``sudo python setup.py install``, ``sudo pip install``, ``sudo + e.g. ``sudo python certbot/setup.py install``, ``sudo pip install``, ``sudo ./venv/bin/...``. These modes of operation might corrupt your operating system and are **not supported** by the Certbot team! diff --git a/docs/intro.rst b/certbot/docs/intro.rst similarity index 100% rename from docs/intro.rst rename to certbot/docs/intro.rst diff --git a/docs/make.bat b/certbot/docs/make.bat similarity index 100% rename from docs/make.bat rename to certbot/docs/make.bat diff --git a/docs/man/certbot.rst b/certbot/docs/man/certbot.rst similarity index 100% rename from docs/man/certbot.rst rename to certbot/docs/man/certbot.rst diff --git a/certbot/docs/packaging.rst b/certbot/docs/packaging.rst new file mode 100644 index 000000000..7b0b1d41a --- /dev/null +++ b/certbot/docs/packaging.rst @@ -0,0 +1,49 @@ +=============== +Packaging Guide +=============== + +Releases +======== + +We release packages and upload them to PyPI (wheels and source tarballs). + +- https://pypi.python.org/pypi/acme +- https://pypi.python.org/pypi/certbot +- https://pypi.python.org/pypi/certbot-apache +- https://pypi.python.org/pypi/certbot-nginx +- https://pypi.python.org/pypi/certbot-dns-cloudflare +- https://pypi.python.org/pypi/certbot-dns-cloudxns +- https://pypi.python.org/pypi/certbot-dns-digitalocean +- https://pypi.python.org/pypi/certbot-dns-dnsimple +- https://pypi.python.org/pypi/certbot-dns-dnsmadeeasy +- https://pypi.python.org/pypi/certbot-dns-google +- https://pypi.python.org/pypi/certbot-dns-linode +- https://pypi.python.org/pypi/certbot-dns-luadns +- https://pypi.python.org/pypi/certbot-dns-nsone +- https://pypi.python.org/pypi/certbot-dns-ovh +- https://pypi.python.org/pypi/certbot-dns-rfc2136 +- https://pypi.python.org/pypi/certbot-dns-route53 + +The following scripts are used in the process: + +- https://github.com/certbot/certbot/blob/master/tools/release.sh + +We use git tags to identify releases, using `Semantic Versioning`_. For +example: `v0.11.1`. + +.. _`Semantic Versioning`: http://semver.org/ + +Notes for package maintainers +============================= + +0. Please use our tagged releases, not ``master``! + +1. Do not package ``certbot-compatibility-test`` or ``letshelp-certbot`` - it's only used internally. + +2. To run tests on our packages, you should use ``python setup.py test``. Doing things like running ``pytest`` directly on our package files may not work because Certbot relies on setuptools to register and find its plugins. + +3. If you'd like to include automated renewal in your package ``certbot renew -q`` should be added to crontab or systemd timer. Additionally you should include a random per-machine time offset to avoid having a large number of your clients hit Let's Encrypt's servers simultaneously. + +4. ``jws`` is an internal script for ``acme`` module and it doesn't have to be packaged - it's mostly for debugging: you can use it as ``echo foo | jws sign | jws verify``. + +5. Do get in touch with us. We are happy to make any changes that will make packaging easier. If you need to apply some patches don't do it downstream - make a PR here. diff --git a/docs/resources.rst b/certbot/docs/resources.rst similarity index 100% rename from docs/resources.rst rename to certbot/docs/resources.rst diff --git a/docs/using.rst b/certbot/docs/using.rst similarity index 95% rename from docs/using.rst rename to certbot/docs/using.rst index e17e56b64..27ae826bd 100644 --- a/docs/using.rst +++ b/certbot/docs/using.rst @@ -58,8 +58,8 @@ standalone_ Y N | Uses a "standalone" webserver to obtain a certificate. | the only way to obtain wildcard certificates from Let's | Encrypt. manual_ Y N | Helps you obtain a certificate by giving you instructions to http-01_ (80) or - | perform domain validation yourself. Additionally allows you dns-01_ (53) - | to specify scripts to automate the validation task in a + | perform domain validation yourself. Additionally allows you dns-01_ (53) + | to specify scripts to automate the validation task in a | customized way. =========== ==== ==== =============================================================== ============================= @@ -83,7 +83,7 @@ Apache ------ The Apache plugin currently `supports -`_ +`_ modern OSes based on Debian, Fedora, SUSE, Gentoo and Darwin. This automates both obtaining *and* installing certificates on an Apache webserver. To specify this plugin on the command line, simply include @@ -169,9 +169,6 @@ the bound IPv6 port and the failure during the second bind is expected. Use ``---address`` to explicitly tell Certbot which interface (and protocol) to bind. -.. note:: The ``--standalone-supported-challenges`` option has been - deprecated since ``certbot`` version 0.9.0. - .. _dns_plugins: DNS Plugins @@ -271,34 +268,29 @@ There are also a number of third-party plugins for the client, provided by other developers. Many are beta/experimental, but some are already in widespread use: -=========== ==== ==== =============================================================== -Plugin Auth Inst Notes -=========== ==== ==== =============================================================== -plesk_ Y Y Integration with the Plesk web hosting tool -haproxy_ Y Y Integration with the HAProxy load balancer -s3front_ Y Y Integration with Amazon CloudFront distribution of S3 buckets -gandi_ Y Y Integration with Gandi's hosting products and API -varnish_ Y N Obtain certificates via a Varnish server -external_ Y N A plugin for convenient scripting (See also ticket 2782_) -icecast_ N Y Deploy certificates to Icecast 2 streaming media servers -pritunl_ N Y Install certificates in pritunl distributed OpenVPN servers -proxmox_ N Y Install certificates in Proxmox Virtualization servers -postfix_ N Y STARTTLS Everywhere is becoming a Certbot Postfix/Exim plugin -heroku_ Y Y Integration with Heroku SSL -=========== ==== ==== =============================================================== +================== ==== ==== =============================================================== +Plugin Auth Inst Notes +================== ==== ==== =============================================================== +haproxy_ Y Y Integration with the HAProxy load balancer +s3front_ Y Y Integration with Amazon CloudFront distribution of S3 buckets +gandi_ Y N Obtain certificates via the Gandi LiveDNS API +varnish_ Y N Obtain certificates via a Varnish server +external-auth_ Y Y A plugin for convenient scripting +pritunl_ N Y Install certificates in pritunl distributed OpenVPN servers +proxmox_ N Y Install certificates in Proxmox Virtualization servers +dns-standalone_ Y N Obtain certificates via an integrated DNS server +dns-ispconfig_ Y N DNS Authentication using ISPConfig as DNS server +================== ==== ==== =============================================================== -.. _plesk: https://github.com/plesk/letsencrypt-plesk .. _haproxy: https://github.com/greenhost/certbot-haproxy .. _s3front: https://github.com/dlapiduz/letsencrypt-s3front -.. _gandi: https://github.com/Gandi/letsencrypt-gandi -.. _icecast: https://github.com/e00E/lets-encrypt-icecast +.. _gandi: https://github.com/obynio/certbot-plugin-gandi .. _varnish: http://git.sesse.net/?p=letsencrypt-varnish-plugin -.. _2782: https://github.com/certbot/certbot/issues/2782 .. _pritunl: https://github.com/kharkevich/letsencrypt-pritunl .. _proxmox: https://github.com/kharkevich/letsencrypt-proxmox -.. _external: https://github.com/marcan/letsencrypt-external -.. _postfix: https://github.com/EFForg/starttls-everywhere -.. _heroku: https://github.com/gboudreau/certbot-heroku +.. _external-auth: https://github.com/EnigmaBridge/certbot-external-auth +.. _dns-standalone: https://github.com/siilike/certbot-dns-standalone +.. _dns-ispconfig: https://github.com/m42e/certbot-dns-ispconfig If you're interested, you can also :ref:`write your own plugin `. @@ -687,8 +679,8 @@ Where are my certificates? ========================== All generated keys and issued certificates can be found in -``/etc/letsencrypt/live/$domain``. In the case of creating a SAN certificate -with multiple alternative names, ``$domain`` is the first domain passed in +``/etc/letsencrypt/live/$domain``. In the case of creating a SAN certificate +with multiple alternative names, ``$domain`` is the first domain passed in via -d parameter. Rather than copying, please point your (web) server configuration directly to those files (or create symlinks). During the renewal_, ``/etc/letsencrypt/live`` is updated @@ -924,9 +916,10 @@ Certbot accepts a global configuration file that applies its options to all invo of Certbot. Certificate specific configuration choices should be set in the ``.conf`` files that can be found in ``/etc/letsencrypt/renewal``. -By default no cli.ini file is created, after creating one -it is possible to specify the location of this configuration file with -``certbot-auto --config cli.ini`` (or shorter ``-c cli.ini``). An +By default no cli.ini file is created (though it may exist already if you installed Certbot +via a package manager, for instance). +After creating one it is possible to specify the location of this configuration file with +``certbot --config cli.ini`` (or shorter ``-c cli.ini``). An example configuration file is shown below: .. include:: ../examples/cli.ini diff --git a/docs/what.rst b/certbot/docs/what.rst similarity index 100% rename from docs/what.rst rename to certbot/docs/what.rst diff --git a/examples/.gitignore b/certbot/examples/.gitignore similarity index 100% rename from examples/.gitignore rename to certbot/examples/.gitignore diff --git a/examples/cli.ini b/certbot/examples/cli.ini similarity index 95% rename from examples/cli.ini rename to certbot/examples/cli.ini index dbaa9c599..4215fda5b 100644 --- a/examples/cli.ini +++ b/certbot/examples/cli.ini @@ -15,7 +15,6 @@ rsa-key-size = 4096 # Uncomment to use the standalone authenticator on port 443 # authenticator = standalone -# standalone-supported-challenges = tls-sni-01 # Uncomment to use the webroot authenticator. Replace webroot-path with the # path to the public_html / webroot folder being served by your web server. diff --git a/examples/dev-cli.ini b/certbot/examples/dev-cli.ini similarity index 100% rename from examples/dev-cli.ini rename to certbot/examples/dev-cli.ini diff --git a/examples/generate-csr.sh b/certbot/examples/generate-csr.sh similarity index 100% rename from examples/generate-csr.sh rename to certbot/examples/generate-csr.sh diff --git a/examples/openssl.cnf b/certbot/examples/openssl.cnf similarity index 100% rename from examples/openssl.cnf rename to certbot/examples/openssl.cnf diff --git a/examples/plugins/certbot_example_plugins.py b/certbot/examples/plugins/certbot_example_plugins.py similarity index 100% rename from examples/plugins/certbot_example_plugins.py rename to certbot/examples/plugins/certbot_example_plugins.py diff --git a/examples/plugins/setup.py b/certbot/examples/plugins/setup.py similarity index 99% rename from examples/plugins/setup.py rename to certbot/examples/plugins/setup.py index 4538e83b8..ba2b5e4e2 100644 --- a/examples/plugins/setup.py +++ b/certbot/examples/plugins/setup.py @@ -1,6 +1,5 @@ from setuptools import setup - setup( name='certbot-example-plugins', package='certbot_example_plugins.py', diff --git a/certbot/local-oldest-requirements.txt b/certbot/local-oldest-requirements.txt new file mode 100644 index 000000000..f6d158890 --- /dev/null +++ b/certbot/local-oldest-requirements.txt @@ -0,0 +1,2 @@ +# Remember to update setup.py to match the package versions below. +acme[dev]==0.40.0 diff --git a/certbot/lock.py b/certbot/lock.py deleted file mode 100644 index 3ff46518d..000000000 --- a/certbot/lock.py +++ /dev/null @@ -1,124 +0,0 @@ -"""Implements file locks for locking files and directories in UNIX.""" -import errno -import logging -import os - -from certbot import compat -from certbot import errors - -logger = logging.getLogger(__name__) - - -def lock_dir(dir_path): - """Place a lock file on the directory at dir_path. - - The lock file is placed in the root of dir_path with the name - .certbot.lock. - - :param str dir_path: path to directory - - :returns: the locked LockFile object - :rtype: LockFile - - :raises errors.LockError: if unable to acquire the lock - - """ - return LockFile(os.path.join(dir_path, '.certbot.lock')) - - -class LockFile(object): - """A UNIX lock file. - - This lock file is released when the locked file is closed or the - process exits. It cannot be used to provide synchronization between - threads. It is based on the lock_file package by Martin Horcicka. - - """ - def __init__(self, path): - """Initialize and acquire the lock file. - - :param str path: path to the file to lock - - :raises errors.LockError: if unable to acquire the lock - - """ - super(LockFile, self).__init__() - self._path = path - self._fd = None - - self.acquire() - - def acquire(self): - """Acquire the lock file. - - :raises errors.LockError: if lock is already held - :raises OSError: if unable to open or stat the lock file - - """ - while self._fd is None: - # Open the file - fd = os.open(self._path, os.O_CREAT | os.O_WRONLY, 0o600) - try: - self._try_lock(fd) - if self._lock_success(fd): - self._fd = fd - finally: - # Close the file if it is not the required one - if self._fd is None: - os.close(fd) - - def _try_lock(self, fd): - """Try to acquire the lock file without blocking. - - :param int fd: file descriptor of the opened file to lock - - """ - try: - compat.lock_file(fd) - except IOError as err: - if err.errno in (errno.EACCES, errno.EAGAIN): - logger.debug( - "A lock on %s is held by another process.", self._path) - raise errors.LockError( - "Another instance of Certbot is already running.") - raise - - def _lock_success(self, fd): - """Did we successfully grab the lock? - - Because this class deletes the locked file when the lock is - released, it is possible another process removed and recreated - the file between us opening the file and acquiring the lock. - - :param int fd: file descriptor of the opened file to lock - - :returns: True if the lock was successfully acquired - :rtype: bool - - """ - try: - stat1 = os.stat(self._path) - except OSError as err: - if err.errno == errno.ENOENT: - return False - raise - - stat2 = os.fstat(fd) - # If our locked file descriptor and the file on disk refer to - # the same device and inode, they're the same file. - return stat1.st_dev == stat2.st_dev and stat1.st_ino == stat2.st_ino - - def __repr__(self): - repr_str = '{0}({1}) <'.format(self.__class__.__name__, self._path) - if self._fd is None: - repr_str += 'released>' - else: - repr_str += 'acquired>' - return repr_str - - def release(self): - """Remove, close, and release the lock file.""" - try: - compat.release_locked_file(self._fd, self._path) - finally: - self._fd = None diff --git a/certbot/ocsp.py b/certbot/ocsp.py deleted file mode 100644 index 049e14827..000000000 --- a/certbot/ocsp.py +++ /dev/null @@ -1,120 +0,0 @@ -"""Tools for checking certificate revocation.""" -import logging -import re - -from subprocess import Popen, PIPE - -from certbot import errors -from certbot import util - -logger = logging.getLogger(__name__) - -class RevocationChecker(object): - "This class figures out OCSP checking on this system, and performs it." - - def __init__(self): - self.broken = False - - if not util.exe_exists("openssl"): - logger.info("openssl not installed, can't check revocation") - self.broken = True - return - - # New versions of openssl want -header var=val, old ones want -header var val - test_host_format = Popen(["openssl", "ocsp", "-header", "var", "val"], - stdout=PIPE, stderr=PIPE, universal_newlines=True) - _out, err = test_host_format.communicate() - if "Missing =" in err: - self.host_args = lambda host: ["Host=" + host] - else: - self.host_args = lambda host: ["Host", host] - - - def ocsp_revoked(self, cert_path, chain_path): - """Get revoked status for a particular cert version. - - .. todo:: Make this a non-blocking call - - :param str cert_path: Path to certificate - :param str chain_path: Path to intermediate cert - :rtype bool or None: - :returns: True if revoked; False if valid or the check failed - - """ - if self.broken: - return False - - - url, host = self.determine_ocsp_server(cert_path) - if not host: - return False - # jdkasten thanks "Bulletproof SSL and TLS - Ivan Ristic" for documenting this! - cmd = ["openssl", "ocsp", - "-no_nonce", - "-issuer", chain_path, - "-cert", cert_path, - "-url", url, - "-CAfile", chain_path, - "-verify_other", chain_path, - "-trust_other", - "-header"] + self.host_args(host) - logger.debug("Querying OCSP for %s", cert_path) - logger.debug(" ".join(cmd)) - try: - output, err = util.run_script(cmd, log=logger.debug) - except errors.SubprocessError: - logger.info("OCSP check failed for %s (are we offline?)", cert_path) - return False - - return _translate_ocsp_query(cert_path, output, err) - - - def determine_ocsp_server(self, cert_path): - """Extract the OCSP server host from a certificate. - - :param str cert_path: Path to the cert we're checking OCSP for - :rtype tuple: - :returns: (OCSP server URL or None, OCSP server host or None) - - """ - try: - url, _err = util.run_script( - ["openssl", "x509", "-in", cert_path, "-noout", "-ocsp_uri"], - log=logger.debug) - except errors.SubprocessError: - logger.info("Cannot extract OCSP URI from %s", cert_path) - return None, None - - url = url.rstrip() - host = url.partition("://")[2].rstrip("/") - if host: - return url, host - else: - logger.info("Cannot process OCSP host from URL (%s) in cert at %s", url, cert_path) - return None, None - -def _translate_ocsp_query(cert_path, ocsp_output, ocsp_errors): - """Parse openssl's weird output to work out what it means.""" - - states = ("good", "revoked", "unknown") - patterns = [r"{0}: (WARNING.*)?{1}".format(cert_path, s) for s in states] - good, revoked, unknown = (re.search(p, ocsp_output, flags=re.DOTALL) for p in patterns) - - warning = good.group(1) if good else None - - if (not "Response verify OK" in ocsp_errors) or (good and warning) or unknown: - logger.info("Revocation status for %s is unknown", cert_path) - logger.debug("Uncertain output:\n%s\nstderr:\n%s", ocsp_output, ocsp_errors) - return False - elif good and not warning: - return False - elif revoked: - warning = revoked.group(1) - if warning: - logger.info("OCSP revocation warning: %s", warning) - return True - else: - logger.warning("Unable to properly parse OCSP output: %s\nstderr:%s", - ocsp_output, ocsp_errors) - return False - diff --git a/certbot/plugins/__init__.py b/certbot/plugins/__init__.py deleted file mode 100644 index 7b1aca2b4..000000000 --- a/certbot/plugins/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Certbot client.plugins.""" diff --git a/readthedocs.org.requirements.txt b/certbot/readthedocs.org.requirements.txt similarity index 69% rename from readthedocs.org.requirements.txt rename to certbot/readthedocs.org.requirements.txt index 94a81e788..f3964e8a7 100644 --- a/readthedocs.org.requirements.txt +++ b/certbot/readthedocs.org.requirements.txt @@ -1,11 +1,11 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project # in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# expected and "pip install -e certbot[docs]" must be used instead -e acme --e .[docs] +-e certbot[docs] diff --git a/setup.cfg b/certbot/setup.cfg similarity index 100% rename from setup.cfg rename to certbot/setup.cfg diff --git a/setup.py b/certbot/setup.py similarity index 59% rename from setup.py rename to certbot/setup.py index 9e6af2d4f..d19327e5e 100644 --- a/setup.py +++ b/certbot/setup.py @@ -1,8 +1,13 @@ import codecs +from distutils.version import StrictVersion import os import re +import sys -from setuptools import find_packages, setup +from setuptools import __version__ as setuptools_version +from setuptools import find_packages +from setuptools import setup +from setuptools.command.test import test as TestCommand # Workaround for http://bugs.python.org/issue8876, see # http://bugs.python.org/issue8876#msg208792 @@ -31,14 +36,17 @@ version = meta['version'] # specified here to avoid masking the more specific request requirements in # acme. See https://github.com/pypa/pip/issues/988 for more info. install_requires = [ - 'acme>=0.29.0', + 'acme>=0.40.0', # We technically need ConfigArgParse 0.10.0 for Python 2.6 support, but # saying so here causes a runtime error against our temporary fork of 0.9.3 # in which we added 2.6 support (see #2243), so we relax the requirement. 'ConfigArgParse>=0.9.3', 'configobj', 'cryptography>=1.2.3', # load_pem_x509_certificate - 'josepy', + 'distro>=1.0.1', + # 1.1.0+ is required to avoid the warnings described at + # https://github.com/certbot/josepy/issues/13. + 'josepy>=1.1.0', 'mock', 'parsedatetime>=1.3', # Calendar.parseDT 'pyrfc3339', @@ -48,23 +56,38 @@ install_requires = [ 'zope.interface', ] +# Add pywin32 on Windows platforms to handle low-level system calls. +# This dependency needs to be added using environment markers to avoid its installation on Linux. +# However environment markers are supported only with setuptools >= 36.2. +# So this dependency is not added for old Linux distributions with old setuptools, +# in order to allow these systems to build certbot from sources. +pywin32_req = 'pywin32>=227' # do not forget to edit pywin32 dependency accordingly in windows-installer/construct.py +if StrictVersion(setuptools_version) >= StrictVersion('36.2'): + install_requires.append(pywin32_req + " ; sys_platform == 'win32'") +elif 'bdist_wheel' in sys.argv[1:]: + raise RuntimeError('Error, you are trying to build certbot wheels using an old version ' + 'of setuptools. Version 36.2+ of setuptools is required.') +elif os.name == 'nt': + # This branch exists to improve this package's behavior on Windows. Without + # it, if the sdist is installed on Windows with an old version of + # setuptools, pywin32 will not be specified as a dependency. + install_requires.append(pywin32_req) + dev_extras = [ - # Pin astroid==1.3.5, pylint==1.4.2 as a workaround for #289 - 'astroid==1.3.5', 'coverage', 'ipdb', 'pytest', 'pytest-cov', 'pytest-xdist', - 'pylint==1.4.2', # upstream #248 'tox', 'twine', 'wheel', ] dev3_extras = [ + 'astroid', 'mypy', - 'typing', # for python3.4 + 'pylint', ] docs_extras = [ @@ -75,6 +98,22 @@ docs_extras = [ 'sphinx_rtd_theme', ] + +class PyTest(TestCommand): + user_options = [] + + def initialize_options(self): + TestCommand.initialize_options(self) + self.pytest_args = '' + + def run_tests(self): + import shlex + # import here, cause outside the eggs aren't loaded + import pytest + errno = pytest.main(shlex.split(self.pytest_args)) + sys.exit(errno) + + setup( name='certbot', version=version, @@ -84,7 +123,7 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ 'Development Status :: 5 - Production/Stable', 'Environment :: Console', @@ -96,10 +135,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', @@ -118,19 +157,19 @@ setup( 'docs': docs_extras, }, - # to test all packages run "python setup.py test -s - # {acme,certbot_apache,certbot_nginx}" test_suite='certbot', + tests_require=["pytest"], + cmdclass={"test": PyTest}, entry_points={ 'console_scripts': [ 'certbot = certbot.main:main', ], 'certbot.plugins': [ - 'manual = certbot.plugins.manual:Authenticator', - 'null = certbot.plugins.null:Installer', - 'standalone = certbot.plugins.standalone:Authenticator', - 'webroot = certbot.plugins.webroot:Authenticator', + 'manual = certbot._internal.plugins.manual:Authenticator', + 'null = certbot._internal.plugins.null:Installer', + 'standalone = certbot._internal.plugins.standalone:Authenticator', + 'webroot = certbot._internal.plugins.webroot:Authenticator', ], }, ) diff --git a/certbot/tests/__init__.py b/certbot/tests/__init__.py deleted file mode 100644 index 2f4d6e07c..000000000 --- a/certbot/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Certbot Tests""" diff --git a/certbot/tests/account_test.py b/certbot/tests/account_test.py index b062f437b..4a6ed3e01 100644 --- a/certbot/tests/account_test.py +++ b/certbot/tests/account_test.py @@ -1,9 +1,6 @@ -"""Tests for certbot.account.""" +"""Tests for certbot._internal.account.""" import datetime import json -import os -import shutil -import stat import unittest import josepy as jose @@ -11,21 +8,20 @@ import mock import pytz from acme import messages - -from certbot import compat from certbot import errors - +from certbot.compat import filesystem +from certbot.compat import misc +from certbot.compat import os import certbot.tests.util as test_util - KEY = jose.JWKRSA.load(test_util.load_vector("rsa512_key.pem")) class AccountTest(unittest.TestCase): - """Tests for certbot.account.Account.""" + """Tests for certbot._internal.account.Account.""" def setUp(self): - from certbot.account import Account + from certbot._internal.account import Account self.regr = mock.MagicMock() self.meta = Account.Meta( creation_host="test.certbot.org", @@ -34,9 +30,9 @@ class AccountTest(unittest.TestCase): self.acc = Account(self.regr, KEY, self.meta) self.regr.__repr__ = mock.MagicMock(return_value="i_am_a_regr") - with mock.patch("certbot.account.socket") as mock_socket: + with mock.patch("certbot._internal.account.socket") as mock_socket: mock_socket.getfqdn.return_value = "test.certbot.org" - with mock.patch("certbot.account.datetime") as mock_dt: + with mock.patch("certbot._internal.account.datetime") as mock_dt: mock_dt.datetime.now.return_value = self.meta.creation_dt self.acc_no_meta = Account(self.regr, KEY) @@ -58,18 +54,18 @@ class AccountTest(unittest.TestCase): " 3) - self.assertEqual(self.mock_auth.cleanup.call_count, 1) - # Test if list first element is TLSSNI01, use typ because it is an achall - self.assertEqual( - self.mock_auth.cleanup.call_args[0][0][0].typ, "tls-sni-01") + self.assertEqual(self.mock_auth.cleanup.call_count, 1) + # Test if list first element is http-01, use typ because it is an achall + self.assertEqual( + self.mock_auth.cleanup.call_args[0][0][0].typ, "http-01") - self.assertEqual(len(authzr), 1) + self.assertEqual(len(authzr), 1) - def test_name1_tls_sni_01_1_acme_1(self): - self._test_name1_tls_sni_01_1_common(combos=True) + def test_name1_http_01_1_acme_1(self): + self._test_name1_http_01_1_common(combos=True) - def test_name1_tls_sni_01_1_acme_2(self): + def test_name1_http_01_1_acme_2(self): self.mock_net.acme_version = 2 - self._test_name1_tls_sni_01_1_common(combos=False) + self._test_name1_http_01_1_common(combos=False) - @mock.patch("certbot.auth_handler.AuthHandler._poll_challenges") - def test_name1_tls_sni_01_1_http_01_1_dns_1_acme_1(self, mock_poll): - mock_poll.side_effect = self._validate_all - self.mock_auth.get_chall_pref.return_value.append(challenges.HTTP01) + def test_name1_http_01_1_dns_1_acme_1(self): + self.mock_net.poll.side_effect = _gen_mock_on_poll() self.mock_auth.get_chall_pref.return_value.append(challenges.DNS01) authzr = gen_dom_authzr(domain="0", challs=acme_util.CHALLENGES, combos=False) mock_order = mock.MagicMock(authorizations=[authzr]) authzr = self.handler.handle_authorizations(mock_order) - self.assertEqual(self.mock_net.answer_challenge.call_count, 3) + self.assertEqual(self.mock_net.answer_challenge.call_count, 2) - self.assertEqual(mock_poll.call_count, 1) - chall_update = mock_poll.call_args[0][1] - self.assertEqual(list(six.iterkeys(chall_update)), [0]) - self.assertEqual(len(chall_update.values()), 1) + self.assertEqual(self.mock_net.poll.call_count, 1) self.assertEqual(self.mock_auth.cleanup.call_count, 1) - # Test if list first element is TLSSNI01, use typ because it is an achall + # Test if list first element is http-01, use typ because it is an achall for achall in self.mock_auth.cleanup.call_args[0][0]: - self.assertTrue(achall.typ in ["tls-sni-01", "http-01", "dns-01"]) + self.assertTrue(achall.typ in ["http-01", "dns-01"]) # Length of authorizations list self.assertEqual(len(authzr), 1) - @mock.patch("certbot.auth_handler.AuthHandler._poll_challenges") - def test_name1_tls_sni_01_1_http_01_1_dns_1_acme_2(self, mock_poll): + def test_name1_http_01_1_dns_1_acme_2(self): self.mock_net.acme_version = 2 - mock_poll.side_effect = self._validate_all - self.mock_auth.get_chall_pref.return_value.append(challenges.HTTP01) + self.mock_net.poll.side_effect = _gen_mock_on_poll() self.mock_auth.get_chall_pref.return_value.append(challenges.DNS01) authzr = gen_dom_authzr(domain="0", challs=acme_util.CHALLENGES, combos=False) @@ -158,64 +152,51 @@ class HandleAuthorizationsTest(unittest.TestCase): # pylint: disable=too-many-p self.assertEqual(self.mock_net.answer_challenge.call_count, 1) - self.assertEqual(mock_poll.call_count, 1) - chall_update = mock_poll.call_args[0][1] - self.assertEqual(list(six.iterkeys(chall_update)), [0]) - self.assertEqual(len(chall_update.values()), 1) + self.assertEqual(self.mock_net.poll.call_count, 1) self.assertEqual(self.mock_auth.cleanup.call_count, 1) cleaned_up_achalls = self.mock_auth.cleanup.call_args[0][0] self.assertEqual(len(cleaned_up_achalls), 1) - self.assertEqual(cleaned_up_achalls[0].typ, "tls-sni-01") + self.assertEqual(cleaned_up_achalls[0].typ, "http-01") # Length of authorizations list self.assertEqual(len(authzr), 1) - def _test_name3_tls_sni_01_3_common(self, combos): + def _test_name3_http_01_3_common(self, combos): self.mock_net.request_domain_challenges.side_effect = functools.partial( gen_dom_authzr, challs=acme_util.CHALLENGES, combos=combos) - authzrs = [gen_dom_authzr(domain="0", challs=acme_util.CHALLENGES), gen_dom_authzr(domain="1", challs=acme_util.CHALLENGES), gen_dom_authzr(domain="2", challs=acme_util.CHALLENGES)] mock_order = mock.MagicMock(authorizations=authzrs) - with mock.patch("certbot.auth_handler.AuthHandler._poll_challenges") as mock_poll: - mock_poll.side_effect = self._validate_all - authzr = self.handler.handle_authorizations(mock_order) + + self.mock_net.poll.side_effect = _gen_mock_on_poll() + authzr = self.handler.handle_authorizations(mock_order) self.assertEqual(self.mock_net.answer_challenge.call_count, 3) # Check poll call - self.assertEqual(mock_poll.call_count, 1) - chall_update = mock_poll.call_args[0][1] - self.assertEqual(len(list(six.iterkeys(chall_update))), 3) - self.assertTrue(0 in list(six.iterkeys(chall_update))) - self.assertEqual(len(chall_update[0]), 1) - self.assertTrue(1 in list(six.iterkeys(chall_update))) - self.assertEqual(len(chall_update[1]), 1) - self.assertTrue(2 in list(six.iterkeys(chall_update))) - self.assertEqual(len(chall_update[2]), 1) + self.assertEqual(self.mock_net.poll.call_count, 3) self.assertEqual(self.mock_auth.cleanup.call_count, 1) self.assertEqual(len(authzr), 3) - def test_name3_tls_sni_01_3_common_acme_1(self): - self._test_name3_tls_sni_01_3_common(combos=True) + def test_name3_http_01_3_common_acme_1(self): + self._test_name3_http_01_3_common(combos=True) - def test_name3_tls_sni_01_3_common_acme_2(self): + def test_name3_http_01_3_common_acme_2(self): self.mock_net.acme_version = 2 - self._test_name3_tls_sni_01_3_common(combos=False) + self._test_name3_http_01_3_common(combos=False) - @mock.patch("certbot.auth_handler.AuthHandler._poll_challenges") - def test_debug_challenges(self, mock_poll): + def test_debug_challenges(self): zope.component.provideUtility( mock.Mock(debug_challenges=True), interfaces.IConfig) authzrs = [gen_dom_authzr(domain="0", challs=acme_util.CHALLENGES)] mock_order = mock.MagicMock(authorizations=authzrs) - mock_poll.side_effect = self._validate_all + self.mock_net.poll.side_effect = _gen_mock_on_poll() self.handler.handle_authorizations(mock_order) @@ -231,6 +212,18 @@ class HandleAuthorizationsTest(unittest.TestCase): # pylint: disable=too-many-p self.assertRaises( errors.AuthorizationError, self.handler.handle_authorizations, mock_order) + def test_max_retries_exceeded(self): + authzrs = [gen_dom_authzr(domain="0", challs=acme_util.CHALLENGES)] + mock_order = mock.MagicMock(authorizations=authzrs) + + # We will return STATUS_PENDING twice before returning STATUS_VALID. + self.mock_net.poll.side_effect = _gen_mock_on_poll(retry=2) + + with self.assertRaises(errors.AuthorizationError) as error: + # We retry only once, so retries will be exhausted before STATUS_VALID is returned. + self.handler.handle_authorizations(mock_order, False, 1) + self.assertTrue('All authorizations were not finalized by the CA.' in str(error.exception)) + def test_no_domains(self): mock_order = mock.MagicMock(authorizations=[]) self.assertRaises(errors.AuthorizationError, self.handler.handle_authorizations, mock_order) @@ -244,9 +237,8 @@ class HandleAuthorizationsTest(unittest.TestCase): # pylint: disable=too-many-p self.handler.pref_challs.extend((challenges.HTTP01.typ, challenges.DNS01.typ,)) - with mock.patch("certbot.auth_handler.AuthHandler._poll_challenges") as mock_poll: - mock_poll.side_effect = self._validate_all - self.handler.handle_authorizations(mock_order) + self.mock_net.poll.side_effect = _gen_mock_on_poll() + self.handler.handle_authorizations(mock_order) self.assertEqual(self.mock_auth.cleanup.call_count, 1) self.assertEqual( @@ -262,7 +254,7 @@ class HandleAuthorizationsTest(unittest.TestCase): # pylint: disable=too-many-p def _test_preferred_challenges_not_supported_common(self, combos): authzrs = [gen_dom_authzr(domain="0", challs=acme_util.CHALLENGES, combos=combos)] mock_order = mock.MagicMock(authorizations=authzrs) - self.handler.pref_challs.append(challenges.HTTP01.typ) + self.handler.pref_challs.append(challenges.DNS01.typ) self.assertRaises( errors.AuthorizationError, self.handler.handle_authorizations, mock_order) @@ -288,191 +280,142 @@ class HandleAuthorizationsTest(unittest.TestCase): # pylint: disable=too-many-p self.assertEqual(self.mock_auth.cleanup.call_count, 1) self.assertEqual( - self.mock_auth.cleanup.call_args[0][0][0].typ, "tls-sni-01") + self.mock_auth.cleanup.call_args[0][0][0].typ, "http-01") + + def test_answer_error(self): + self.mock_net.answer_challenge.side_effect = errors.AuthorizationError - @mock.patch("certbot.auth_handler.AuthHandler._respond") - def test_respond_error(self, mock_respond): authzrs = [gen_dom_authzr(domain="0", challs=acme_util.CHALLENGES)] mock_order = mock.MagicMock(authorizations=authzrs) - mock_respond.side_effect = errors.AuthorizationError self.assertRaises( errors.AuthorizationError, self.handler.handle_authorizations, mock_order) self.assertEqual(self.mock_auth.cleanup.call_count, 1) self.assertEqual( - self.mock_auth.cleanup.call_args[0][0][0].typ, "tls-sni-01") + self.mock_auth.cleanup.call_args[0][0][0].typ, "http-01") - @mock.patch("certbot.auth_handler.AuthHandler._poll_challenges") - @mock.patch("certbot.auth_handler.AuthHandler.verify_authzr_complete") - def test_incomplete_authzr_error(self, mock_verify, mock_poll): + def test_incomplete_authzr_error(self): authzrs = [gen_dom_authzr(domain="0", challs=acme_util.CHALLENGES)] mock_order = mock.MagicMock(authorizations=authzrs) - mock_verify.side_effect = errors.AuthorizationError - mock_poll.side_effect = self._validate_all + self.mock_net.poll.side_effect = _gen_mock_on_poll(status=messages.STATUS_INVALID) - self.assertRaises( - errors.AuthorizationError, self.handler.handle_authorizations, mock_order) + with test_util.patch_get_utility(): + with self.assertRaises(errors.AuthorizationError) as error: + self.handler.handle_authorizations(mock_order, False) + self.assertTrue('Some challenges have failed.' in str(error.exception)) self.assertEqual(self.mock_auth.cleanup.call_count, 1) self.assertEqual( - self.mock_auth.cleanup.call_args[0][0][0].typ, "tls-sni-01") + self.mock_auth.cleanup.call_args[0][0][0].typ, "http-01") + + def test_best_effort(self): + def _conditional_mock_on_poll(authzr): + """This mock will invalidate one authzr, and invalidate the other one""" + valid_mock = _gen_mock_on_poll(messages.STATUS_VALID) + invalid_mock = _gen_mock_on_poll(messages.STATUS_INVALID) + + if authzr.body.identifier.value == 'will-be-invalid': + return invalid_mock(authzr) + return valid_mock(authzr) + + # Two authzrs. Only one will be valid. + authzrs = [gen_dom_authzr(domain="will-be-valid", challs=acme_util.CHALLENGES), + gen_dom_authzr(domain="will-be-invalid", challs=acme_util.CHALLENGES)] + self.mock_net.poll.side_effect = _conditional_mock_on_poll + + mock_order = mock.MagicMock(authorizations=authzrs) + + with mock.patch('certbot._internal.auth_handler._report_failed_authzrs') as mock_report: + valid_authzr = self.handler.handle_authorizations(mock_order, True) + + # Because best_effort=True, we did not blow up. Instead ... + self.assertEqual(len(valid_authzr), 1) # ... the valid authzr has been processed + self.assertEqual(mock_report.call_count, 1) # ... the invalid authzr has been reported + + self.mock_net.poll.side_effect = _gen_mock_on_poll(status=messages.STATUS_INVALID) + + with test_util.patch_get_utility(): + with self.assertRaises(errors.AuthorizationError) as error: + self.handler.handle_authorizations(mock_order, True) + + # Despite best_effort=True, process will fail because no authzr is valid. + self.assertTrue('All challenges have failed.' in str(error.exception)) def test_validated_challenge_not_rerun(self): - # With pending challenge, we expect the challenge to be tried, and fail. + # With a pending challenge that is not supported by the plugin, we + # expect an exception to be raised. authzr = acme_util.gen_authzr( messages.STATUS_PENDING, "0", - [acme_util.HTTP01], + [acme_util.DNS01], [messages.STATUS_PENDING], False) mock_order = mock.MagicMock(authorizations=[authzr]) self.assertRaises( errors.AuthorizationError, self.handler.handle_authorizations, mock_order) - # With validated challenge; we expect the challenge not be tried again, and succeed. + # With a validated challenge that is not supported by the plugin, we + # expect the challenge to not be solved again and + # handle_authorizations() to succeed. authzr = acme_util.gen_authzr( messages.STATUS_VALID, "0", - [acme_util.HTTP01], + [acme_util.DNS01], [messages.STATUS_VALID], False) mock_order = mock.MagicMock(authorizations=[authzr]) self.handler.handle_authorizations(mock_order) - def _validate_all(self, aauthzrs, unused_1, unused_2): - for i, aauthzr in enumerate(aauthzrs): - azr = aauthzr.authzr - updated_azr = acme_util.gen_authzr( - messages.STATUS_VALID, - azr.body.identifier.value, - [challb.chall for challb in azr.body.challenges], - [messages.STATUS_VALID] * len(azr.body.challenges), - azr.body.combinations) - aauthzrs[i] = type(aauthzr)(updated_azr, aauthzr.achalls) + def test_valid_authzrs_deactivated(self): + """When we deactivate valid authzrs in an orderr, we expect them to become deactivated + and to receive a list of deactivated authzrs in return.""" + def _mock_deactivate(authzr): + if authzr.body.status == messages.STATUS_VALID: + if authzr.body.identifier.value == "is_valid_but_will_fail": + raise acme_errors.Error("Mock deactivation ACME error") + authzb = authzr.body.update(status=messages.STATUS_DEACTIVATED) + authzr = messages.AuthorizationResource(body=authzb) + else: # pragma: no cover + raise errors.Error("Can't deactivate non-valid authz") + return authzr - @mock.patch("certbot.auth_handler.logger") - def test_tls_sni_logs(self, logger): - self._test_name1_tls_sni_01_1_common(combos=True) - self.assertTrue("deprecated" in logger.warning.call_args[0][0]) + to_deactivate = [("is_valid", messages.STATUS_VALID), + ("is_pending", messages.STATUS_PENDING), + ("is_valid_but_will_fail", messages.STATUS_VALID)] + + to_deactivate = [acme_util.gen_authzr(a[1], a[0], [acme_util.HTTP01], + [a[1], False]) for a in to_deactivate] + orderr = mock.MagicMock(authorizations=to_deactivate) + + self.mock_net.deactivate_authorization.side_effect = _mock_deactivate + + authzrs, failed = self.handler.deactivate_valid_authorizations(orderr) + + self.assertEqual(self.mock_net.deactivate_authorization.call_count, 2) + self.assertEqual(len(authzrs), 1) + self.assertEqual(len(failed), 1) + self.assertEqual(authzrs[0].body.identifier.value, "is_valid") + self.assertEqual(authzrs[0].body.status, messages.STATUS_DEACTIVATED) + self.assertEqual(failed[0].body.identifier.value, "is_valid_but_will_fail") + self.assertEqual(failed[0].body.status, messages.STATUS_VALID) -class PollChallengesTest(unittest.TestCase): - # pylint: disable=protected-access - """Test poll challenges.""" +def _gen_mock_on_poll(status=messages.STATUS_VALID, retry=0, wait_value=1): + state = {'count': retry} - def setUp(self): - from certbot.auth_handler import challb_to_achall - from certbot.auth_handler import AuthHandler, AnnotatedAuthzr - - # Account and network are mocked... - self.mock_net = mock.MagicMock() - self.handler = AuthHandler( - None, self.mock_net, mock.Mock(key="mock_key"), []) - - self.doms = ["0", "1", "2"] - self.aauthzrs = [ - AnnotatedAuthzr(acme_util.gen_authzr( - messages.STATUS_PENDING, self.doms[0], - [acme_util.HTTP01, acme_util.TLSSNI01], - [messages.STATUS_PENDING] * 2, False), []), - AnnotatedAuthzr(acme_util.gen_authzr( - messages.STATUS_PENDING, self.doms[1], - acme_util.CHALLENGES, [messages.STATUS_PENDING] * 3, False), []), - AnnotatedAuthzr(acme_util.gen_authzr( - messages.STATUS_PENDING, self.doms[2], - acme_util.CHALLENGES, [messages.STATUS_PENDING] * 3, False), []) - ] - - self.chall_update = {} # type: Dict[int, achallenges.KeyAuthorizationAnnotatedChallenge] - for i, aauthzr in enumerate(self.aauthzrs): - self.chall_update[i] = [ - challb_to_achall(challb, mock.Mock(key="dummy_key"), self.doms[i]) - for challb in aauthzr.authzr.body.challenges] - - - @mock.patch("certbot.auth_handler.time") - def test_poll_challenges(self, unused_mock_time): - self.mock_net.poll.side_effect = self._mock_poll_solve_one_valid - self.handler._poll_challenges(self.aauthzrs, self.chall_update, False) - - for aauthzr in self.aauthzrs: - self.assertEqual(aauthzr.authzr.body.status, messages.STATUS_VALID) - - @mock.patch("certbot.auth_handler.time") - def test_poll_challenges_failure_best_effort(self, unused_mock_time): - self.mock_net.poll.side_effect = self._mock_poll_solve_one_invalid - self.handler._poll_challenges(self.aauthzrs, self.chall_update, True) - - for aauthzr in self.aauthzrs: - self.assertEqual(aauthzr.authzr.body.status, messages.STATUS_PENDING) - - @mock.patch("certbot.auth_handler.time") - @test_util.patch_get_utility() - def test_poll_challenges_failure(self, unused_mock_time, unused_mock_zope): - self.mock_net.poll.side_effect = self._mock_poll_solve_one_invalid - self.assertRaises( - errors.AuthorizationError, self.handler._poll_challenges, - self.aauthzrs, self.chall_update, False) - - @mock.patch("certbot.auth_handler.time") - def test_unable_to_find_challenge_status(self, unused_mock_time): - from certbot.auth_handler import challb_to_achall - self.mock_net.poll.side_effect = self._mock_poll_solve_one_valid - self.chall_update[0].append( - challb_to_achall(acme_util.DNS01_P, "key", self.doms[0])) - self.assertRaises( - errors.AuthorizationError, self.handler._poll_challenges, - self.aauthzrs, self.chall_update, False) - - def test_verify_authzr_failure(self): - self.assertRaises(errors.AuthorizationError, - self.handler.verify_authzr_complete, self.aauthzrs) - - def _mock_poll_solve_one_valid(self, authzr): - # Pending here because my dummy script won't change the full status. - # Basically it didn't raise an error and it stopped earlier than - # Making all challenges invalid which would make mock_poll_solve_one - # change authzr to invalid - return self._mock_poll_solve_one_chall(authzr, messages.STATUS_VALID) - - def _mock_poll_solve_one_invalid(self, authzr): - return self._mock_poll_solve_one_chall(authzr, messages.STATUS_INVALID) - - def _mock_poll_solve_one_chall(self, authzr, desired_status): - # pylint: disable=no-self-use - """Dummy method that solves one chall at a time to desired_status. - - When all are solved.. it changes authzr.status to desired_status - - """ - new_challbs = authzr.body.challenges - for challb in authzr.body.challenges: - if challb.status != desired_status: - new_challbs = tuple( - challb_temp if challb_temp != challb - else acme_util.chall_to_challb(challb.chall, desired_status) - for challb_temp in authzr.body.challenges - ) - break - - if all(test_challb.status == desired_status - for test_challb in new_challbs): - status_ = desired_status - else: - status_ = authzr.body.status - - new_authzr = messages.AuthorizationResource( - uri=authzr.uri, - body=messages.Authorization( - identifier=authzr.body.identifier, - challenges=new_challbs, - combinations=authzr.body.combinations, - status=status_, - ), - ) - return (new_authzr, "response") + def _mock(authzr): + state['count'] = state['count'] - 1 + effective_status = status if state['count'] < 0 else messages.STATUS_PENDING + updated_azr = acme_util.gen_authzr( + effective_status, + authzr.body.identifier.value, + [challb.chall for challb in authzr.body.challenges], + [effective_status] * len(authzr.body.challenges), + authzr.body.combinations) + return updated_azr, mock.MagicMock(headers={'Retry-After': str(wait_value)}) + return _mock class ChallbToAchallTest(unittest.TestCase): - """Tests for certbot.auth_handler.challb_to_achall.""" + """Tests for certbot._internal.auth_handler.challb_to_achall.""" def _call(self, challb): - from certbot.auth_handler import challb_to_achall + from certbot._internal.auth_handler import challb_to_achall return challb_to_achall(challb, "account_key", "domain") def test_it(self): @@ -485,7 +428,7 @@ class ChallbToAchallTest(unittest.TestCase): class GenChallengePathTest(unittest.TestCase): - """Tests for certbot.auth_handler.gen_challenge_path. + """Tests for certbot._internal.auth_handler.gen_challenge_path. .. todo:: Add more tests for dumb_path... depending on what we want to do. @@ -498,13 +441,13 @@ class GenChallengePathTest(unittest.TestCase): @classmethod def _call(cls, challbs, preferences, combinations): - from certbot.auth_handler import gen_challenge_path + from certbot._internal.auth_handler import gen_challenge_path return gen_challenge_path(challbs, preferences, combinations) def test_common_case(self): - """Given TLSSNI01 and HTTP01 with appropriate combos.""" - challbs = (acme_util.TLSSNI01_P, acme_util.HTTP01_P) - prefs = [challenges.TLSSNI01, challenges.HTTP01] + """Given DNS01 and HTTP01 with appropriate combos.""" + challbs = (acme_util.DNS01_P, acme_util.HTTP01_P) + prefs = [challenges.DNS01, challenges.HTTP01] combos = ((0,), (1,)) # Smart then trivial dumb path test @@ -515,8 +458,8 @@ class GenChallengePathTest(unittest.TestCase): self.assertTrue(self._call(challbs[::-1], prefs, None)) def test_not_supported(self): - challbs = (acme_util.DNS01_P, acme_util.TLSSNI01_P) - prefs = [challenges.TLSSNI01] + challbs = (acme_util.DNS01_P, acme_util.HTTP01_P) + prefs = [challenges.HTTP01] combos = ((0, 1),) # smart path fails because no challs in perfs satisfies combos @@ -527,8 +470,8 @@ class GenChallengePathTest(unittest.TestCase): errors.AuthorizationError, self._call, challbs, prefs, None) -class ReportFailedChallsTest(unittest.TestCase): - """Tests for certbot.auth_handler._report_failed_challs.""" +class ReportFailedAuthzrsTest(unittest.TestCase): + """Tests for certbot._internal.auth_handler._report_failed_authzrs.""" # pylint: disable=protected-access def setUp(self): @@ -542,40 +485,36 @@ class ReportFailedChallsTest(unittest.TestCase): # Prevent future regressions if the error type changes self.assertTrue(kwargs["error"].description is not None) - self.http01 = achallenges.KeyAuthorizationAnnotatedChallenge( - # pylint: disable=star-args - challb=messages.ChallengeBody(**kwargs), - domain="example.com", - account_key="key") + http_01 = messages.ChallengeBody(**kwargs) - kwargs["chall"] = acme_util.TLSSNI01 - self.tls_sni_same = achallenges.KeyAuthorizationAnnotatedChallenge( - # pylint: disable=star-args - challb=messages.ChallengeBody(**kwargs), - domain="example.com", - account_key="key") + kwargs["chall"] = acme_util.HTTP01 + http_01 = messages.ChallengeBody(**kwargs) - kwargs["error"] = messages.Error(typ="dnssec", detail="detail") - self.tls_sni_diff = achallenges.KeyAuthorizationAnnotatedChallenge( - # pylint: disable=star-args - challb=messages.ChallengeBody(**kwargs), - domain="foo.bar", - account_key="key") + self.authzr1 = mock.MagicMock() + self.authzr1.body.identifier.value = 'example.com' + self.authzr1.body.challenges = [http_01, http_01] + + kwargs["error"] = messages.Error.with_code("dnssec", detail="detail") + http_01_diff = messages.ChallengeBody(**kwargs) + + self.authzr2 = mock.MagicMock() + self.authzr2.body.identifier.value = 'foo.bar' + self.authzr2.body.challenges = [http_01_diff] @test_util.patch_get_utility() def test_same_error_and_domain(self, mock_zope): - from certbot import auth_handler + from certbot._internal import auth_handler - auth_handler._report_failed_challs([self.http01, self.tls_sni_same]) + auth_handler._report_failed_authzrs([self.authzr1], 'key') call_list = mock_zope().add_message.call_args_list self.assertTrue(len(call_list) == 1) self.assertTrue("Domain: example.com\nType: tls\nDetail: detail" in call_list[0][0][0]) @test_util.patch_get_utility() def test_different_errors_and_domains(self, mock_zope): - from certbot import auth_handler + from certbot._internal import auth_handler - auth_handler._report_failed_challs([self.http01, self.tls_sni_diff]) + auth_handler._report_failed_authzrs([self.authzr1, self.authzr2], 'key') self.assertTrue(mock_zope().add_message.call_count == 2) diff --git a/certbot/tests/cert_manager_test.py b/certbot/tests/cert_manager_test.py index 84774ca77..81134f02f 100644 --- a/certbot/tests/cert_manager_test.py +++ b/certbot/tests/cert_manager_test.py @@ -1,7 +1,6 @@ -"""Tests for certbot.cert_manager.""" +"""Tests for certbot._internal.cert_manager.""" # pylint: disable=protected-access -import os import re import shutil import tempfile @@ -10,14 +9,14 @@ import unittest import configobj import mock -from certbot import configuration from certbot import errors - +from certbot._internal import configuration +from certbot._internal.storage import ALL_FOUR +from certbot.compat import filesystem +from certbot.compat import os from certbot.display import util as display_util -from certbot.storage import ALL_FOUR - -from certbot.tests import storage_test from certbot.tests import util as test_util +import storage_test class BaseCertManagerTest(test_util.ConfigTestCase): @@ -27,7 +26,7 @@ class BaseCertManagerTest(test_util.ConfigTestCase): super(BaseCertManagerTest, self).setUp() self.config.quiet = False - os.makedirs(self.config.renewal_configs_dir) + filesystem.makedirs(self.config.renewal_configs_dir) self.domains = { "example.org": None, @@ -45,14 +44,14 @@ class BaseCertManagerTest(test_util.ConfigTestCase): def _set_up_config(self, domain, custom_archive): # TODO: maybe provide NamespaceConfig.make_dirs? # TODO: main() should create those dirs, c.f. #902 - os.makedirs(os.path.join(self.config.live_dir, domain)) + filesystem.makedirs(os.path.join(self.config.live_dir, domain)) config_file = configobj.ConfigObj() if custom_archive is not None: - os.makedirs(custom_archive) + filesystem.makedirs(custom_archive) config_file["archive_dir"] = custom_archive else: - os.makedirs(os.path.join(self.config.default_archive_dir, domain)) + filesystem.makedirs(os.path.join(self.config.default_archive_dir, domain)) for kind in ALL_FOUR: config_file[kind] = os.path.join(self.config.live_dir, domain, @@ -65,13 +64,12 @@ class BaseCertManagerTest(test_util.ConfigTestCase): class UpdateLiveSymlinksTest(BaseCertManagerTest): - """Tests for certbot.cert_manager.update_live_symlinks + """Tests for certbot._internal.cert_manager.update_live_symlinks """ def test_update_live_symlinks(self): """Test update_live_symlinks""" - # pylint: disable=too-many-statements # create files with incorrect symlinks - from certbot import cert_manager + from certbot._internal import cert_manager archive_paths = {} for domain in self.domains: custom_archive = self.domains[domain] @@ -98,23 +96,23 @@ class UpdateLiveSymlinksTest(BaseCertManagerTest): for kind in ALL_FOUR: os.chdir(os.path.dirname(self.config_files[domain][kind])) self.assertEqual( - os.path.realpath(os.readlink(self.config_files[domain][kind])), - os.path.realpath(archive_paths[domain][kind])) + filesystem.realpath(os.readlink(self.config_files[domain][kind])), + filesystem.realpath(archive_paths[domain][kind])) finally: os.chdir(prev_dir) class DeleteTest(storage_test.BaseRenewableCertTest): - """Tests for certbot.cert_manager.delete + """Tests for certbot._internal.cert_manager.delete """ def _call(self): - from certbot import cert_manager + from certbot._internal import cert_manager cert_manager.delete(self.config) @test_util.patch_get_utility() - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.storage.delete_files') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.storage.delete_files') def test_delete_from_config(self, mock_delete_files, mock_lineage_for_certname, unused_get_utility): """Test delete""" @@ -124,8 +122,8 @@ class DeleteTest(storage_test.BaseRenewableCertTest): mock_delete_files.assert_called_once_with(self.config, "example.org") @test_util.patch_get_utility() - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.storage.delete_files') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.storage.delete_files') def test_delete_interactive_single(self, mock_delete_files, mock_lineage_for_certname, mock_util): """Test delete""" @@ -135,8 +133,8 @@ class DeleteTest(storage_test.BaseRenewableCertTest): mock_delete_files.assert_called_once_with(self.config, "example.org") @test_util.patch_get_utility() - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.storage.delete_files') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.storage.delete_files') def test_delete_interactive_multiple(self, mock_delete_files, mock_lineage_for_certname, mock_util): """Test delete""" @@ -149,20 +147,20 @@ class DeleteTest(storage_test.BaseRenewableCertTest): class CertificatesTest(BaseCertManagerTest): - """Tests for certbot.cert_manager.certificates + """Tests for certbot._internal.cert_manager.certificates """ def _certificates(self, *args, **kwargs): - from certbot.cert_manager import certificates + from certbot._internal.cert_manager import certificates return certificates(*args, **kwargs) - @mock.patch('certbot.cert_manager.logger') + @mock.patch('certbot._internal.cert_manager.logger') @test_util.patch_get_utility() def test_certificates_parse_fail(self, mock_utility, mock_logger): self._certificates(self.config) self.assertTrue(mock_logger.warning.called) #pylint: disable=no-member self.assertTrue(mock_utility.called) - @mock.patch('certbot.cert_manager.logger') + @mock.patch('certbot._internal.cert_manager.logger') @test_util.patch_get_utility() def test_certificates_quiet(self, mock_utility, mock_logger): self.config.quiet = True @@ -171,21 +169,21 @@ class CertificatesTest(BaseCertManagerTest): self.assertTrue(mock_logger.warning.called) #pylint: disable=no-member @mock.patch('certbot.crypto_util.verify_renewable_cert') - @mock.patch('certbot.cert_manager.logger') + @mock.patch('certbot._internal.cert_manager.logger') @test_util.patch_get_utility() - @mock.patch("certbot.storage.RenewableCert") - @mock.patch('certbot.cert_manager._report_human_readable') + @mock.patch("certbot._internal.storage.RenewableCert") + @mock.patch('certbot._internal.cert_manager._report_human_readable') def test_certificates_parse_success(self, mock_report, mock_renewable_cert, mock_utility, mock_logger, mock_verifier): mock_verifier.return_value = None mock_report.return_value = "" self._certificates(self.config) - self.assertFalse(mock_logger.warning.called) #pylint: disable=no-member + self.assertFalse(mock_logger.warning.called) self.assertTrue(mock_report.called) self.assertTrue(mock_utility.called) self.assertTrue(mock_renewable_cert.called) - @mock.patch('certbot.cert_manager.logger') + @mock.patch('certbot._internal.cert_manager.logger') @test_util.patch_get_utility() def test_certificates_no_files(self, mock_utility, mock_logger): empty_tempdir = tempfile.mkdtemp() @@ -196,17 +194,18 @@ class CertificatesTest(BaseCertManagerTest): quiet=False )) - os.makedirs(empty_config.renewal_configs_dir) + filesystem.makedirs(empty_config.renewal_configs_dir) self._certificates(empty_config) - self.assertFalse(mock_logger.warning.called) #pylint: disable=no-member + self.assertFalse(mock_logger.warning.called) self.assertTrue(mock_utility.called) shutil.rmtree(empty_tempdir) - @mock.patch('certbot.cert_manager.ocsp.RevocationChecker.ocsp_revoked') - def test_report_human_readable(self, mock_revoked): #pylint: disable=too-many-statements + @mock.patch('certbot._internal.cert_manager.ocsp.RevocationChecker.ocsp_revoked') + def test_report_human_readable(self, mock_revoked): mock_revoked.return_value = None - from certbot import cert_manager - import datetime, pytz + from certbot._internal import cert_manager + import datetime + import pytz expiry = pytz.UTC.fromutc(datetime.datetime.utcnow()) cert = mock.MagicMock(lineagename="nameone") @@ -228,20 +227,20 @@ class CertificatesTest(BaseCertManagerTest): # pylint: disable=protected-access out = get_report() self.assertTrue('1 hour(s)' in out or '2 hour(s)' in out) - self.assertTrue('VALID' in out and not 'INVALID' in out) + self.assertTrue('VALID' in out and 'INVALID' not in out) cert.target_expiry += datetime.timedelta(days=1) # pylint: disable=protected-access out = get_report() self.assertTrue('1 day' in out) self.assertFalse('under' in out) - self.assertTrue('VALID' in out and not 'INVALID' in out) + self.assertTrue('VALID' in out and 'INVALID' not in out) cert.target_expiry += datetime.timedelta(days=2) # pylint: disable=protected-access out = get_report() self.assertTrue('3 days' in out) - self.assertTrue('VALID' in out and not 'INVALID' in out) + self.assertTrue('VALID' in out and 'INVALID' not in out) cert.is_test_cert = True mock_revoked.return_value = True @@ -271,90 +270,82 @@ class CertificatesTest(BaseCertManagerTest): class SearchLineagesTest(BaseCertManagerTest): - """Tests for certbot.cert_manager._search_lineages.""" + """Tests for certbot._internal.cert_manager._search_lineages.""" @mock.patch('certbot.util.make_or_verify_dir') - @mock.patch('certbot.storage.renewal_conf_files') - @mock.patch('certbot.storage.RenewableCert') + @mock.patch('certbot._internal.storage.renewal_conf_files') + @mock.patch('certbot._internal.storage.RenewableCert') def test_cert_storage_error(self, mock_renewable_cert, mock_renewal_conf_files, - mock_make_or_verify_dir): + mock_make_or_verify_dir): mock_renewal_conf_files.return_value = ["badfile"] mock_renewable_cert.side_effect = errors.CertStorageError - from certbot import cert_manager + from certbot._internal import cert_manager # pylint: disable=protected-access - self.assertEqual(cert_manager._search_lineages(self.config, lambda x: x, "check"), - "check") + self.assertEqual(cert_manager._search_lineages(self.config, lambda x: x, "check"), "check") self.assertTrue(mock_make_or_verify_dir.called) class LineageForCertnameTest(BaseCertManagerTest): - """Tests for certbot.cert_manager.lineage_for_certname""" + """Tests for certbot._internal.cert_manager.lineage_for_certname""" @mock.patch('certbot.util.make_or_verify_dir') - @mock.patch('certbot.storage.renewal_file_for_certname') - @mock.patch('certbot.storage.RenewableCert') + @mock.patch('certbot._internal.storage.renewal_file_for_certname') + @mock.patch('certbot._internal.storage.RenewableCert') def test_found_match(self, mock_renewable_cert, mock_renewal_conf_file, - mock_make_or_verify_dir): + mock_make_or_verify_dir): mock_renewal_conf_file.return_value = "somefile.conf" mock_match = mock.Mock(lineagename="example.com") mock_renewable_cert.return_value = mock_match - from certbot import cert_manager - self.assertEqual(cert_manager.lineage_for_certname(self.config, "example.com"), - mock_match) + from certbot._internal import cert_manager + self.assertEqual(cert_manager.lineage_for_certname(self.config, "example.com"), mock_match) self.assertTrue(mock_make_or_verify_dir.called) @mock.patch('certbot.util.make_or_verify_dir') - @mock.patch('certbot.storage.renewal_file_for_certname') - def test_no_match(self, mock_renewal_conf_file, - mock_make_or_verify_dir): + @mock.patch('certbot._internal.storage.renewal_file_for_certname') + def test_no_match(self, mock_renewal_conf_file, mock_make_or_verify_dir): mock_renewal_conf_file.return_value = "other.com.conf" - from certbot import cert_manager - self.assertEqual(cert_manager.lineage_for_certname(self.config, "example.com"), - None) + from certbot._internal import cert_manager + self.assertEqual(cert_manager.lineage_for_certname(self.config, "example.com"), None) self.assertTrue(mock_make_or_verify_dir.called) @mock.patch('certbot.util.make_or_verify_dir') - @mock.patch('certbot.storage.renewal_file_for_certname') - def test_no_renewal_file(self, mock_renewal_conf_file, - mock_make_or_verify_dir): + @mock.patch('certbot._internal.storage.renewal_file_for_certname') + def test_no_renewal_file(self, mock_renewal_conf_file, mock_make_or_verify_dir): mock_renewal_conf_file.side_effect = errors.CertStorageError() - from certbot import cert_manager - self.assertEqual(cert_manager.lineage_for_certname(self.config, "example.com"), - None) + from certbot._internal import cert_manager + self.assertEqual(cert_manager.lineage_for_certname(self.config, "example.com"), None) self.assertTrue(mock_make_or_verify_dir.called) class DomainsForCertnameTest(BaseCertManagerTest): - """Tests for certbot.cert_manager.domains_for_certname""" + """Tests for certbot._internal.cert_manager.domains_for_certname""" @mock.patch('certbot.util.make_or_verify_dir') - @mock.patch('certbot.storage.renewal_file_for_certname') - @mock.patch('certbot.storage.RenewableCert') + @mock.patch('certbot._internal.storage.renewal_file_for_certname') + @mock.patch('certbot._internal.storage.RenewableCert') def test_found_match(self, mock_renewable_cert, mock_renewal_conf_file, - mock_make_or_verify_dir): + mock_make_or_verify_dir): mock_renewal_conf_file.return_value = "somefile.conf" mock_match = mock.Mock(lineagename="example.com") domains = ["example.com", "example.org"] mock_match.names.return_value = domains mock_renewable_cert.return_value = mock_match - from certbot import cert_manager + from certbot._internal import cert_manager self.assertEqual(cert_manager.domains_for_certname(self.config, "example.com"), domains) self.assertTrue(mock_make_or_verify_dir.called) @mock.patch('certbot.util.make_or_verify_dir') - @mock.patch('certbot.storage.renewal_file_for_certname') - def test_no_match(self, mock_renewal_conf_file, - mock_make_or_verify_dir): + @mock.patch('certbot._internal.storage.renewal_file_for_certname') + def test_no_match(self, mock_renewal_conf_file, mock_make_or_verify_dir): mock_renewal_conf_file.return_value = "somefile.conf" - from certbot import cert_manager - self.assertEqual(cert_manager.domains_for_certname(self.config, "other.com"), - None) + from certbot._internal import cert_manager + self.assertEqual(cert_manager.domains_for_certname(self.config, "other.com"), None) self.assertTrue(mock_make_or_verify_dir.called) class RenameLineageTest(BaseCertManagerTest): - """Tests for certbot.cert_manager.rename_lineage""" + """Tests for certbot._internal.cert_manager.rename_lineage""" def setUp(self): super(RenameLineageTest, self).setUp() @@ -362,10 +353,10 @@ class RenameLineageTest(BaseCertManagerTest): self.config.new_certname = "after" def _call(self, *args, **kwargs): - from certbot import cert_manager + from certbot._internal import cert_manager return cert_manager.rename_lineage(*args, **kwargs) - @mock.patch('certbot.storage.renewal_conf_files') + @mock.patch('certbot._internal.storage.renewal_conf_files') @test_util.patch_get_utility() def test_no_certname(self, mock_get_utility, mock_renewal_conf_files): self.config.certname = None @@ -396,7 +387,7 @@ class RenameLineageTest(BaseCertManagerTest): self.assertRaises(errors.Error, self._call, self.config) @test_util.patch_get_utility() - @mock.patch('certbot.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') def test_no_existing_certname(self, mock_lineage_for_certname, unused_get_utility): self.config.certname = "one" self.config.new_certname = "two" @@ -405,30 +396,30 @@ class RenameLineageTest(BaseCertManagerTest): self._call, self.config) @test_util.patch_get_utility() - @mock.patch("certbot.storage.RenewableCert._check_symlinks") + @mock.patch("certbot._internal.storage.RenewableCert._check_symlinks") def test_rename_cert(self, mock_check, unused_get_utility): mock_check.return_value = True self._call(self.config) - from certbot import cert_manager + from certbot._internal import cert_manager updated_lineage = cert_manager.lineage_for_certname(self.config, self.config.new_certname) self.assertTrue(updated_lineage is not None) self.assertEqual(updated_lineage.lineagename, self.config.new_certname) @test_util.patch_get_utility() - @mock.patch("certbot.storage.RenewableCert._check_symlinks") + @mock.patch("certbot._internal.storage.RenewableCert._check_symlinks") def test_rename_cert_interactive_certname(self, mock_check, mock_get_utility): mock_check.return_value = True self.config.certname = None util_mock = mock_get_utility() util_mock.menu.return_value = (display_util.OK, 0) self._call(self.config) - from certbot import cert_manager + from certbot._internal import cert_manager updated_lineage = cert_manager.lineage_for_certname(self.config, self.config.new_certname) self.assertTrue(updated_lineage is not None) self.assertEqual(updated_lineage.lineagename, self.config.new_certname) @test_util.patch_get_utility() - @mock.patch("certbot.storage.RenewableCert._check_symlinks") + @mock.patch("certbot._internal.storage.RenewableCert._check_symlinks") def test_rename_cert_bad_new_certname(self, mock_check, unused_get_utility): mock_check.return_value = True @@ -450,7 +441,7 @@ class DuplicativeCertsTest(storage_test.BaseRenewableCertTest): @mock.patch('certbot.util.make_or_verify_dir') def test_find_duplicative_names(self, unused_makedir): - from certbot.cert_manager import find_duplicative_certs + from certbot._internal.cert_manager import find_duplicative_certs test_cert = test_util.load_vector('cert-san_512.pem') with open(self.test_rc.cert, 'wb') as f: f.write(test_cert) @@ -479,7 +470,7 @@ class DuplicativeCertsTest(storage_test.BaseRenewableCertTest): class CertPathToLineageTest(storage_test.BaseRenewableCertTest): - """Tests for certbot.cert_manager.cert_path_to_lineage""" + """Tests for certbot._internal.cert_manager.cert_path_to_lineage""" def setUp(self): super(CertPathToLineageTest, self).setUp() @@ -490,11 +481,11 @@ class CertPathToLineageTest(storage_test.BaseRenewableCertTest): self.config.cert_path = (self.fullchain, '') def _call(self, cli_config): - from certbot.cert_manager import cert_path_to_lineage + from certbot._internal.cert_manager import cert_path_to_lineage return cert_path_to_lineage(cli_config) def _archive_files(self, cli_config, filetype): - from certbot.cert_manager import _archive_files + from certbot._internal.cert_manager import _archive_files return _archive_files(cli_config, filetype) def test_basic_match(self): @@ -506,13 +497,13 @@ class CertPathToLineageTest(storage_test.BaseRenewableCertTest): 'SailorMoon', 'fullchain.pem') self.assertRaises(errors.Error, self._call, bad_test_config) - @mock.patch('certbot.cert_manager._acceptable_matches') + @mock.patch('certbot._internal.cert_manager._acceptable_matches') def test_options_fullchain(self, mock_acceptable_matches): mock_acceptable_matches.return_value = [lambda x: x.fullchain_path] self.config.fullchain_path = self.fullchain self.assertEqual('example.org', self._call(self.config)) - @mock.patch('certbot.cert_manager._acceptable_matches') + @mock.patch('certbot._internal.cert_manager._acceptable_matches') def test_options_cert_path(self, mock_acceptable_matches): mock_acceptable_matches.return_value = [lambda x: x.cert_path] test_cert_path = os.path.join(self.config.config_dir, 'live', 'example.org', @@ -520,7 +511,7 @@ class CertPathToLineageTest(storage_test.BaseRenewableCertTest): self.config.cert_path = (test_cert_path, '') self.assertEqual('example.org', self._call(self.config)) - @mock.patch('certbot.cert_manager._acceptable_matches') + @mock.patch('certbot._internal.cert_manager._acceptable_matches') def test_options_archive_cert(self, mock_acceptable_matches): # Also this and the next test check that the regex of _archive_files is working. self.config.cert_path = (os.path.join(self.config.config_dir, 'archive', 'example.org', @@ -528,7 +519,7 @@ class CertPathToLineageTest(storage_test.BaseRenewableCertTest): mock_acceptable_matches.return_value = [lambda x: self._archive_files(x, 'cert')] self.assertEqual('example.org', self._call(self.config)) - @mock.patch('certbot.cert_manager._acceptable_matches') + @mock.patch('certbot._internal.cert_manager._acceptable_matches') def test_options_archive_fullchain(self, mock_acceptable_matches): self.config.cert_path = (os.path.join(self.config.config_dir, 'archive', 'example.org', 'fullchain11.pem'), '') @@ -538,7 +529,8 @@ class CertPathToLineageTest(storage_test.BaseRenewableCertTest): class MatchAndCheckOverlaps(storage_test.BaseRenewableCertTest): - """Tests for certbot.cert_manager.match_and_check_overlaps w/o overlapping archive dirs.""" + """Tests for certbot._internal.cert_manager.match_and_check_overlaps w/o overlapping + archive dirs.""" # A test with real overlapping archive dirs can be found in tests/boulder_integration.sh def setUp(self): super(MatchAndCheckOverlaps, self).setUp() @@ -549,27 +541,27 @@ class MatchAndCheckOverlaps(storage_test.BaseRenewableCertTest): self.config.cert_path = (self.fullchain, '') def _call(self, cli_config, acceptable_matches, match_func, rv_func): - from certbot.cert_manager import match_and_check_overlaps + from certbot._internal.cert_manager import match_and_check_overlaps return match_and_check_overlaps(cli_config, acceptable_matches, match_func, rv_func) def test_basic_match(self): - from certbot.cert_manager import _acceptable_matches + from certbot._internal.cert_manager import _acceptable_matches self.assertEqual(['example.org'], self._call(self.config, _acceptable_matches(), lambda x: self.config.cert_path[0], lambda x: x.lineagename)) - @mock.patch('certbot.cert_manager._search_lineages') + @mock.patch('certbot._internal.cert_manager._search_lineages') def test_no_matches(self, mock_search_lineages): mock_search_lineages.return_value = [] self.assertRaises(errors.Error, self._call, self.config, None, None, None) - @mock.patch('certbot.cert_manager._search_lineages') + @mock.patch('certbot._internal.cert_manager._search_lineages') def test_too_many_matches(self, mock_search_lineages): mock_search_lineages.return_value = ['spider', 'dance'] self.assertRaises(errors.OverlappingMatchFound, self._call, self.config, None, None, None) class GetCertnameTest(unittest.TestCase): - """Tests for certbot.cert_manager.""" + """Tests for certbot._internal.cert_manager.""" def setUp(self): self.get_utility_patch = test_util.patch_get_utility() @@ -580,12 +572,12 @@ class GetCertnameTest(unittest.TestCase): def tearDown(self): self.get_utility_patch.stop() - @mock.patch('certbot.storage.renewal_conf_files') - @mock.patch('certbot.storage.lineagename_for_filename') + @mock.patch('certbot._internal.storage.renewal_conf_files') + @mock.patch('certbot._internal.storage.lineagename_for_filename') def test_get_certnames(self, mock_name, mock_files): mock_files.return_value = ['example.com.conf'] mock_name.return_value = 'example.com' - from certbot import cert_manager + from certbot._internal import cert_manager prompt = "Which certificate would you" self.mock_get_utility().menu.return_value = (display_util.OK, 0) self.assertEqual( @@ -594,12 +586,12 @@ class GetCertnameTest(unittest.TestCase): self.assertTrue( prompt in self.mock_get_utility().menu.call_args[0][0]) - @mock.patch('certbot.storage.renewal_conf_files') - @mock.patch('certbot.storage.lineagename_for_filename') + @mock.patch('certbot._internal.storage.renewal_conf_files') + @mock.patch('certbot._internal.storage.lineagename_for_filename') def test_get_certnames_custom_prompt(self, mock_name, mock_files): mock_files.return_value = ['example.com.conf'] mock_name.return_value = 'example.com' - from certbot import cert_manager + from certbot._internal import cert_manager prompt = "custom prompt" self.mock_get_utility().menu.return_value = (display_util.OK, 0) self.assertEqual( @@ -609,24 +601,24 @@ class GetCertnameTest(unittest.TestCase): self.assertEqual(self.mock_get_utility().menu.call_args[0][0], prompt) - @mock.patch('certbot.storage.renewal_conf_files') - @mock.patch('certbot.storage.lineagename_for_filename') + @mock.patch('certbot._internal.storage.renewal_conf_files') + @mock.patch('certbot._internal.storage.lineagename_for_filename') def test_get_certnames_user_abort(self, mock_name, mock_files): mock_files.return_value = ['example.com.conf'] mock_name.return_value = 'example.com' - from certbot import cert_manager + from certbot._internal import cert_manager self.mock_get_utility().menu.return_value = (display_util.CANCEL, 0) self.assertRaises( errors.Error, cert_manager.get_certnames, self.config, "erroring_anyway", allow_multiple=False) - @mock.patch('certbot.storage.renewal_conf_files') - @mock.patch('certbot.storage.lineagename_for_filename') + @mock.patch('certbot._internal.storage.renewal_conf_files') + @mock.patch('certbot._internal.storage.lineagename_for_filename') def test_get_certnames_allow_multiple(self, mock_name, mock_files): mock_files.return_value = ['example.com.conf'] mock_name.return_value = 'example.com' - from certbot import cert_manager + from certbot._internal import cert_manager prompt = "Which certificate(s) would you" self.mock_get_utility().checklist.return_value = (display_util.OK, ['example.com']) @@ -636,12 +628,12 @@ class GetCertnameTest(unittest.TestCase): self.assertTrue( prompt in self.mock_get_utility().checklist.call_args[0][0]) - @mock.patch('certbot.storage.renewal_conf_files') - @mock.patch('certbot.storage.lineagename_for_filename') + @mock.patch('certbot._internal.storage.renewal_conf_files') + @mock.patch('certbot._internal.storage.lineagename_for_filename') def test_get_certnames_allow_multiple_custom_prompt(self, mock_name, mock_files): mock_files.return_value = ['example.com.conf'] mock_name.return_value = 'example.com' - from certbot import cert_manager + from certbot._internal import cert_manager prompt = "custom prompt" self.mock_get_utility().checklist.return_value = (display_util.OK, ['example.com']) @@ -653,12 +645,12 @@ class GetCertnameTest(unittest.TestCase): self.mock_get_utility().checklist.call_args[0][0], prompt) - @mock.patch('certbot.storage.renewal_conf_files') - @mock.patch('certbot.storage.lineagename_for_filename') + @mock.patch('certbot._internal.storage.renewal_conf_files') + @mock.patch('certbot._internal.storage.lineagename_for_filename') def test_get_certnames_allow_multiple_user_abort(self, mock_name, mock_files): mock_files.return_value = ['example.com.conf'] mock_name.return_value = 'example.com' - from certbot import cert_manager + from certbot._internal import cert_manager self.mock_get_utility().checklist.return_value = (display_util.CANCEL, []) self.assertRaises( errors.Error, diff --git a/certbot/tests/cli_test.py b/certbot/tests/cli_test.py index e16a1bdcf..05da1da4e 100644 --- a/certbot/tests/cli_test.py +++ b/certbot/tests/cli_test.py @@ -1,58 +1,65 @@ -"""Tests for certbot.cli.""" +"""Tests for certbot._internal.cli.""" import argparse -import unittest -import os -import tempfile import copy -import sys +import tempfile +import unittest import mock import six from six.moves import reload_module # pylint: disable=import-error from acme import challenges - -from certbot import cli -from certbot import constants from certbot import errors -from certbot.plugins import disco - +from certbot._internal import cli +from certbot._internal import constants +from certbot._internal.plugins import disco +from certbot.compat import filesystem +from certbot.compat import os import certbot.tests.util as test_util - from certbot.tests.util import TempDirTestCase PLUGINS = disco.PluginsRegistry.find_all() class TestReadFile(TempDirTestCase): - '''Test cli.read_file''' - - + """Test cli.read_file""" def test_read_file(self): - rel_test_path = os.path.relpath(os.path.join(self.tempdir, 'foo')) - self.assertRaises( - argparse.ArgumentTypeError, cli.read_file, rel_test_path) + curr_dir = os.getcwd() + try: + # On Windows current directory may be on a different drive than self.tempdir. + # However a relative path between two different drives is invalid. So we move to + # self.tempdir to ensure that we stay on the same drive. + os.chdir(self.tempdir) + rel_test_path = os.path.relpath(os.path.join(self.tempdir, 'foo')) + self.assertRaises( + argparse.ArgumentTypeError, cli.read_file, rel_test_path) - test_contents = b'bar\n' - with open(rel_test_path, 'wb') as f: - f.write(test_contents) + test_contents = b'bar\n' + with open(rel_test_path, 'wb') as f: + f.write(test_contents) - path, contents = cli.read_file(rel_test_path) - self.assertEqual(path, os.path.abspath(path)) - self.assertEqual(contents, test_contents) + path, contents = cli.read_file(rel_test_path) + self.assertEqual(path, os.path.abspath(path)) + self.assertEqual(contents, test_contents) + finally: + os.chdir(curr_dir) class FlagDefaultTest(unittest.TestCase): """Tests cli.flag_default""" - def test_linux_directories(self): - if 'fcntl' in sys.modules: + def test_default_directories(self): + if os.name != 'nt': self.assertEqual(cli.flag_default('config_dir'), '/etc/letsencrypt') self.assertEqual(cli.flag_default('work_dir'), '/var/lib/letsencrypt') self.assertEqual(cli.flag_default('logs_dir'), '/var/log/letsencrypt') + else: + self.assertEqual(cli.flag_default('config_dir'), 'C:\\Certbot') + self.assertEqual(cli.flag_default('work_dir'), 'C:\\Certbot\\lib') + self.assertEqual(cli.flag_default('logs_dir'), 'C:\\Certbot\\log') -class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods +class ParseTest(unittest.TestCase): '''Test the cli args entrypoint''' @@ -78,34 +85,34 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods def write_msg(message, *args, **kwargs): # pylint: disable=missing-docstring,unused-argument output.write(message) - with mock.patch('certbot.main.sys.stdout', new=output): + with mock.patch('certbot._internal.main.sys.stdout', new=output): with test_util.patch_get_utility() as mock_get_utility: mock_get_utility().notification.side_effect = write_msg - with mock.patch('certbot.main.sys.stderr'): + with mock.patch('certbot._internal.main.sys.stderr'): self.assertRaises(SystemExit, self._unmocked_parse, args, output) return output.getvalue() - @test_util.broken_on_windows - @mock.patch("certbot.cli.flag_default") + @mock.patch("certbot._internal.cli.flag_default") def test_cli_ini_domains(self, mock_flag_default): - tmp_config = tempfile.NamedTemporaryFile() - # use a shim to get ConfigArgParse to pick up tmp_config - shim = ( - lambda v: copy.deepcopy(constants.CLI_DEFAULTS[v]) - if v != "config_files" - else [tmp_config.name] - ) - mock_flag_default.side_effect = shim + with tempfile.NamedTemporaryFile() as tmp_config: + tmp_config.close() # close now because of compatibility issues on Windows + # use a shim to get ConfigArgParse to pick up tmp_config + shim = ( + lambda v: copy.deepcopy(constants.CLI_DEFAULTS[v]) + if v != "config_files" + else [tmp_config.name] + ) + mock_flag_default.side_effect = shim - namespace = self.parse(["certonly"]) - self.assertEqual(namespace.domains, []) - tmp_config.write(b"domains = example.com") - tmp_config.flush() - namespace = self.parse(["certonly"]) - self.assertEqual(namespace.domains, ["example.com"]) - namespace = self.parse(["renew"]) - self.assertEqual(namespace.domains, []) + namespace = self.parse(["certonly"]) + self.assertEqual(namespace.domains, []) + with open(tmp_config.name, 'w') as file_h: + file_h.write("domains = example.com") + namespace = self.parse(["certonly"]) + self.assertEqual(namespace.domains, ["example.com"]) + namespace = self.parse(["renew"]) + self.assertEqual(namespace.domains, []) def test_no_args(self): namespace = self.parse([]) @@ -118,7 +125,7 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods chain = 'chain' fullchain = 'fullchain' - with mock.patch('certbot.main.install'): + with mock.patch('certbot._internal.main.install'): namespace = self.parse(['install', '--cert-path', cert, '--key-path', 'key', '--chain-path', 'chain', '--fullchain-path', 'fullchain']) @@ -178,7 +185,7 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods self.assertTrue("--delete-after-revoke" in out) self.assertTrue("--no-delete-after-revoke" in out) - out = self._help_output(['-h', 'config_changes']) + out = self._help_output(['-h', 'register']) self.assertTrue("--cert-path" not in out) self.assertTrue("--key-path" not in out) @@ -235,11 +242,10 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods self.assertEqual(namespace.domains, ['example.com', 'another.net']) def test_preferred_challenges(self): - short_args = ['--preferred-challenges', 'http, tls-sni-01, dns'] + short_args = ['--preferred-challenges', 'http, dns'] namespace = self.parse(short_args) - expected = [challenges.HTTP01.typ, - challenges.TLSSNI01.typ, challenges.DNS01.typ] + expected = [challenges.HTTP01.typ, challenges.DNS01.typ] self.assertEqual(namespace.pref_challs, expected) short_args = ['--preferred-challenges', 'jumping-over-the-moon'] @@ -258,15 +264,6 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods self.assertTrue(namespace.must_staple) self.assertTrue(namespace.staple) - def test_no_gui(self): - args = ['renew', '--dialog'] - stderr = six.StringIO() - with mock.patch('certbot.main.sys.stderr', new=stderr): - namespace = self.parse(args) - - self.assertTrue(namespace.noninteractive_mode) - self.assertTrue("--dialog is deprecated" in stderr.getvalue()) - def _check_server_conflict_message(self, parser_args, conflicting_args): try: self.parse(parser_args) @@ -313,21 +310,31 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods self.parse(short_args + ['renew']), False) account_dir = os.path.join(config_dir, constants.ACCOUNTS_DIR) - os.mkdir(account_dir) - os.mkdir(os.path.join(account_dir, 'fake_account_dir')) + filesystem.mkdir(account_dir) + filesystem.mkdir(os.path.join(account_dir, 'fake_account_dir')) self._assert_dry_run_flag_worked(self.parse(short_args + ['auth']), True) self._assert_dry_run_flag_worked(self.parse(short_args + ['renew']), True) + self._assert_dry_run_flag_worked(self.parse(short_args + ['certonly']), True) + short_args += ['certonly'] - self._assert_dry_run_flag_worked(self.parse(short_args), True) - short_args += '--server example.com'.split() - conflicts = ['--dry-run'] - self._check_server_conflict_message(short_args, '--dry-run') + # `--dry-run --server example.com` should emit example.com + self.assertEqual(self.parse(short_args + ['--server', 'example.com']).server, + 'example.com') - short_args += ['--staging'] - conflicts += ['--staging'] - self._check_server_conflict_message(short_args, conflicts) + # `--dry-run --server STAGING_URI` should emit STAGING_URI + self.assertEqual(self.parse(short_args + ['--server', constants.STAGING_URI]).server, + constants.STAGING_URI) + + # `--dry-run --server LIVE` should emit STAGING_URI + self.assertEqual(self.parse(short_args + ['--server', cli.flag_default("server")]).server, + constants.STAGING_URI) + + # `--dry-run --server example.com --staging` should emit an error + conflicts = ['--staging'] + self._check_server_conflict_message(short_args + ['--server', 'example.com', '--staging'], + conflicts) def test_option_was_set(self): key_size_option = 'rsa_key_size' @@ -340,6 +347,8 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods config_dir_option = 'config_dir' self.assertFalse(cli.option_was_set( config_dir_option, cli.flag_default(config_dir_option))) + self.assertFalse(cli.option_was_set( + 'authenticator', cli.flag_default('authenticator'))) def test_encode_revocation_reason(self): for reason, code in constants.REVOCATION_REASONS.items(): @@ -356,7 +365,7 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods errors.Error, self.parse, "-n --force-interactive".split()) def test_deploy_hook_conflict(self): - with mock.patch("certbot.cli.sys.stderr"): + with mock.patch("certbot._internal.cli.sys.stderr"): self.assertRaises(SystemExit, self.parse, "--renew-hook foo --deploy-hook bar".split()) @@ -376,7 +385,7 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods self.assertEqual(namespace.renew_hook, value) def test_renew_hook_conflict(self): - with mock.patch("certbot.cli.sys.stderr"): + with mock.patch("certbot._internal.cli.sys.stderr"): self.assertRaises(SystemExit, self.parse, "--deploy-hook foo --renew-hook bar".split()) @@ -396,7 +405,7 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods self.assertEqual(namespace.renew_hook, value) def test_max_log_backups_error(self): - with mock.patch('certbot.cli.sys.stderr'): + with mock.patch('certbot._internal.cli.sys.stderr'): self.assertRaises( SystemExit, self.parse, "--max-log-backups foo".split()) self.assertRaises( @@ -446,9 +455,13 @@ class ParseTest(unittest.TestCase): # pylint: disable=too-many-public-methods for topic in ['all', 'plugins', 'dns-route53']: self.assertFalse('certbot-route53:auth' in self._help_output([help_flag, topic])) + def test_no_permissions_check_accepted(self): + namespace = self.parse(["--no-permissions-check"]) + self.assertTrue(namespace.no_permissions_check) + class DefaultTest(unittest.TestCase): - """Tests for certbot.cli._Default.""" + """Tests for certbot._internal.cli._Default.""" def setUp(self): @@ -522,7 +535,7 @@ class SetByCliTest(unittest.TestCase): def _call_set_by_cli(var, args, verb): - with mock.patch('certbot.cli.helpful_parser') as mock_parser: + with mock.patch('certbot._internal.cli.helpful_parser') as mock_parser: with test_util.patch_get_utility(): mock_parser.args = args mock_parser.verb = verb diff --git a/certbot/tests/client_test.py b/certbot/tests/client_test.py index 330529fc6..7232ed84b 100644 --- a/certbot/tests/client_test.py +++ b/certbot/tests/client_test.py @@ -1,29 +1,29 @@ -"""Tests for certbot.client.""" -import os +"""Tests for certbot._internal.client.""" import platform import shutil import tempfile import unittest +from josepy import interfaces import mock -from certbot import account from certbot import errors from certbot import util - +from certbot._internal import account +from certbot.compat import filesystem +from certbot.compat import os import certbot.tests.util as test_util -from josepy import interfaces - KEY = test_util.load_vector("rsa512_key.pem") CSR_SAN = test_util.load_vector("csr-san_512.pem") +# pylint: disable=line-too-long class DetermineUserAgentTest(test_util.ConfigTestCase): - """Tests for certbot.client.determine_user_agent.""" + """Tests for certbot._internal.client.determine_user_agent.""" def _call(self): - from certbot.client import determine_user_agent + from certbot._internal.client import determine_user_agent return determine_user_agent(self.config) @mock.patch.dict(os.environ, {"CERTBOT_DOCS": "1"}) @@ -52,7 +52,7 @@ class DetermineUserAgentTest(test_util.ConfigTestCase): class RegisterTest(test_util.ConfigTestCase): - """Tests for certbot.client.register.""" + """Tests for certbot._internal.client.register.""" def setUp(self): super(RegisterTest, self).setUp() @@ -62,7 +62,7 @@ class RegisterTest(test_util.ConfigTestCase): self.account_storage = account.AccountMemoryStorage() def _call(self): - from certbot.client import register + from certbot._internal.client import register tos_cb = mock.MagicMock() return register(self.config, self.account_storage, tos_cb) @@ -85,11 +85,11 @@ class RegisterTest(test_util.ConfigTestCase): return False def test_no_tos(self): - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as mock_client: + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as mock_client: mock_client.new_account_and_tos().terms_of_service = "http://tos" mock_client().external_account_required.side_effect = self._false_mock - with mock.patch("certbot.eff.handle_subscription") as mock_handle: - with mock.patch("certbot.account.report_new_account"): + with mock.patch("certbot._internal.eff.handle_subscription") as mock_handle: + with mock.patch("certbot._internal.account.report_new_account"): mock_client().new_account_and_tos.side_effect = errors.Error self.assertRaises(errors.Error, self._call) self.assertFalse(mock_handle.called) @@ -99,36 +99,36 @@ class RegisterTest(test_util.ConfigTestCase): self.assertTrue(mock_handle.called) def test_it(self): - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as mock_client: + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as mock_client: mock_client().external_account_required.side_effect = self._false_mock - with mock.patch("certbot.account.report_new_account"): - with mock.patch("certbot.eff.handle_subscription"): + with mock.patch("certbot._internal.account.report_new_account"): + with mock.patch("certbot._internal.eff.handle_subscription"): self._call() - @mock.patch("certbot.account.report_new_account") - @mock.patch("certbot.client.display_ops.get_email") + @mock.patch("certbot._internal.account.report_new_account") + @mock.patch("certbot._internal.client.display_ops.get_email") def test_email_retry(self, _rep, mock_get_email): from acme import messages self.config.noninteractive_mode = False msg = "DNS problem: NXDOMAIN looking up MX for example.com" mx_err = messages.Error.with_code('invalidContact', detail=msg) - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as mock_client: + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as mock_client: mock_client().external_account_required.side_effect = self._false_mock - with mock.patch("certbot.eff.handle_subscription") as mock_handle: + with mock.patch("certbot._internal.eff.handle_subscription") as mock_handle: mock_client().new_account_and_tos.side_effect = [mx_err, mock.MagicMock()] self._call() self.assertEqual(mock_get_email.call_count, 1) self.assertTrue(mock_handle.called) - @mock.patch("certbot.account.report_new_account") + @mock.patch("certbot._internal.account.report_new_account") def test_email_invalid_noninteractive(self, _rep): from acme import messages self.config.noninteractive_mode = True msg = "DNS problem: NXDOMAIN looking up MX for example.com" mx_err = messages.Error.with_code('invalidContact', detail=msg) - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as mock_client: + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as mock_client: mock_client().external_account_required.side_effect = self._false_mock - with mock.patch("certbot.eff.handle_subscription"): + with mock.patch("certbot._internal.eff.handle_subscription"): mock_client().new_account_and_tos.side_effect = [mx_err, mock.MagicMock()] self.assertRaises(errors.Error, self._call) @@ -136,12 +136,12 @@ class RegisterTest(test_util.ConfigTestCase): self.config.email = None self.assertRaises(errors.Error, self._call) - @mock.patch("certbot.client.logger") + @mock.patch("certbot._internal.client.logger") def test_without_email(self, mock_logger): - with mock.patch("certbot.eff.handle_subscription") as mock_handle: - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as mock_clnt: + with mock.patch("certbot._internal.eff.handle_subscription") as mock_handle: + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as mock_clnt: mock_clnt().external_account_required.side_effect = self._false_mock - with mock.patch("certbot.account.report_new_account"): + with mock.patch("certbot._internal.account.report_new_account"): self.config.email = None self.config.register_unsafely_without_email = True self.config.dry_run = False @@ -149,14 +149,14 @@ class RegisterTest(test_util.ConfigTestCase): mock_logger.info.assert_called_once_with(mock.ANY) self.assertTrue(mock_handle.called) - @mock.patch("certbot.account.report_new_account") - @mock.patch("certbot.client.display_ops.get_email") + @mock.patch("certbot._internal.account.report_new_account") + @mock.patch("certbot._internal.client.display_ops.get_email") def test_dry_run_no_staging_account(self, _rep, mock_get_email): """Tests dry-run for no staging account, expect account created with no email""" - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as mock_client: + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as mock_client: mock_client().external_account_required.side_effect = self._false_mock - with mock.patch("certbot.eff.handle_subscription"): - with mock.patch("certbot.account.report_new_account"): + with mock.patch("certbot._internal.eff.handle_subscription"): + with mock.patch("certbot._internal.account.report_new_account"): self.config.dry_run = True self._call() # check Certbot did not ask the user to provide an email @@ -165,13 +165,13 @@ class RegisterTest(test_util.ConfigTestCase): self.assertFalse(mock_client().new_account_and_tos.call_args[0][0].contact) def test_with_eab_arguments(self): - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as mock_client: + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as mock_client: mock_client().client.directory.__getitem__ = mock.Mock( side_effect=self._new_acct_dir_mock ) mock_client().external_account_required.side_effect = self._false_mock - with mock.patch("certbot.eff.handle_subscription"): - target = "certbot.client.messages.ExternalAccountBinding.from_data" + with mock.patch("certbot._internal.eff.handle_subscription"): + target = "certbot._internal.client.messages.ExternalAccountBinding.from_data" with mock.patch(target) as mock_eab_from_data: self.config.eab_kid = "test-kid" self.config.eab_hmac_key = "J2OAqW4MHXsrHVa_PVg0Y-L_R4SYw0_aL1le6mfblbE" @@ -180,10 +180,10 @@ class RegisterTest(test_util.ConfigTestCase): self.assertTrue(mock_eab_from_data.called) def test_without_eab_arguments(self): - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as mock_client: + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as mock_client: mock_client().external_account_required.side_effect = self._false_mock - with mock.patch("certbot.eff.handle_subscription"): - target = "certbot.client.messages.ExternalAccountBinding.from_data" + with mock.patch("certbot._internal.eff.handle_subscription"): + target = "certbot._internal.client.messages.ExternalAccountBinding.from_data" with mock.patch(target) as mock_eab_from_data: self.config.eab_kid = None self.config.eab_hmac_key = None @@ -192,11 +192,11 @@ class RegisterTest(test_util.ConfigTestCase): self.assertFalse(mock_eab_from_data.called) def test_external_account_required_without_eab_arguments(self): - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as mock_client: + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as mock_client: mock_client().client.net.key.public_key = mock.Mock(side_effect=self._public_key_mock) mock_client().external_account_required.side_effect = self._true_mock - with mock.patch("certbot.eff.handle_subscription"): - with mock.patch("certbot.client.messages.ExternalAccountBinding.from_data"): + with mock.patch("certbot._internal.eff.handle_subscription"): + with mock.patch("certbot._internal.client.messages.ExternalAccountBinding.from_data"): self.config.eab_kid = None self.config.eab_hmac_key = None @@ -205,31 +205,30 @@ class RegisterTest(test_util.ConfigTestCase): def test_unsupported_error(self): from acme import messages msg = "Test" - mx_err = messages.Error(detail=msg, typ="malformed", title="title") - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as mock_client: + mx_err = messages.Error.with_code("malformed", detail=msg, title="title") + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as mock_client: mock_client().client.directory.__getitem__ = mock.Mock( side_effect=self._new_acct_dir_mock ) mock_client().external_account_required.side_effect = self._false_mock - with mock.patch("certbot.eff.handle_subscription") as mock_handle: + with mock.patch("certbot._internal.eff.handle_subscription") as mock_handle: mock_client().new_account_and_tos.side_effect = [mx_err, mock.MagicMock()] self.assertRaises(messages.Error, self._call) self.assertFalse(mock_handle.called) class ClientTestCommon(test_util.ConfigTestCase): - """Common base class for certbot.client.Client tests.""" + """Common base class for certbot._internal.client.Client tests.""" def setUp(self): super(ClientTestCommon, self).setUp() self.config.no_verify_ssl = False self.config.allow_subset_of_names = False - # pylint: disable=star-args self.account = mock.MagicMock(**{"key.pem": KEY}) - from certbot.client import Client - with mock.patch("certbot.client.acme_client.BackwardsCompatibleClientV2") as acme: + from certbot._internal.client import Client + with mock.patch("certbot._internal.client.acme_client.BackwardsCompatibleClientV2") as acme: self.acme_client = acme self.acme = acme.return_value = mock.MagicMock() self.client = Client( @@ -238,7 +237,7 @@ class ClientTestCommon(test_util.ConfigTestCase): class ClientTest(ClientTestCommon): - """Tests for certbot.client.Client.""" + """Tests for certbot._internal.client.Client.""" def setUp(self): super(ClientTest, self).setUp() @@ -257,6 +256,7 @@ class ClientTest(ClientTestCommon): def _mock_obtain_certificate(self): self.client.auth_handler = mock.MagicMock() self.client.auth_handler.handle_authorizations.return_value = [None] + self.client.auth_handler.deactivate_valid_authorizations.return_value = ([], []) self.acme.finalize_order.return_value = self.eg_order self.acme.new_order.return_value = self.eg_order self.eg_order.update.return_value = self.eg_order @@ -272,8 +272,8 @@ class ClientTest(ClientTestCommon): self.acme.finalize_order.assert_called_once_with( self.eg_order, mock.ANY) - @mock.patch("certbot.client.crypto_util") - @mock.patch("certbot.client.logger") + @mock.patch("certbot._internal.client.crypto_util") + @mock.patch("certbot._internal.client.logger") @test_util.patch_get_utility() def test_obtain_certificate_from_csr(self, unused_mock_get_utility, mock_logger, mock_crypto_util): @@ -308,7 +308,7 @@ class ClientTest(ClientTestCommon): test_csr) mock_logger.warning.assert_called_once_with(mock.ANY) - @mock.patch("certbot.client.crypto_util") + @mock.patch("certbot._internal.client.crypto_util") def test_obtain_certificate(self, mock_crypto_util): csr = util.CSR(form="pem", file=None, data=CSR_SAN) mock_crypto_util.init_save_csr.return_value = csr @@ -324,8 +324,8 @@ class ClientTest(ClientTestCommon): mock_crypto_util.cert_and_chain_from_fullchain.assert_called_once_with( self.eg_order.fullchain_pem) - @mock.patch("certbot.client.crypto_util") - @mock.patch("os.remove") + @mock.patch("certbot._internal.client.crypto_util") + @mock.patch("certbot.compat.os.remove") def test_obtain_certificate_partial_success(self, mock_remove, mock_crypto_util): csr = util.CSR(form="pem", file=mock.sentinel.csr_file, data=CSR_SAN) key = util.CSR(form="pem", file=mock.sentinel.key_file, data=CSR_SAN) @@ -342,8 +342,8 @@ class ClientTest(ClientTestCommon): self.assertEqual(mock_remove.call_count, 2) self.assertEqual(mock_crypto_util.cert_and_chain_from_fullchain.call_count, 1) - @mock.patch("certbot.client.crypto_util") - @mock.patch("certbot.client.acme_crypto_util") + @mock.patch("certbot._internal.client.crypto_util") + @mock.patch("certbot._internal.client.acme_crypto_util") def test_obtain_certificate_dry_run(self, mock_acme_crypto, mock_crypto): csr = util.CSR(form="pem", file=None, data=CSR_SAN) mock_acme_crypto.make_csr.return_value = CSR_SAN @@ -361,6 +361,47 @@ class ClientTest(ClientTestCommon): mock_crypto.init_save_csr.assert_not_called() self.assertEqual(mock_crypto.cert_and_chain_from_fullchain.call_count, 1) + @mock.patch("certbot._internal.client.logger") + @mock.patch("certbot._internal.client.crypto_util") + @mock.patch("certbot._internal.client.acme_crypto_util") + def test_obtain_certificate_dry_run_authz_deactivations_failed(self, mock_acme_crypto, + mock_crypto, mock_log): + from acme import messages + csr = util.CSR(form="pem", file=None, data=CSR_SAN) + mock_acme_crypto.make_csr.return_value = CSR_SAN + mock_crypto.make_key.return_value = mock.sentinel.key_pem + key = util.Key(file=None, pem=mock.sentinel.key_pem) + self._set_mock_from_fullchain(mock_crypto.cert_and_chain_from_fullchain) + + self._mock_obtain_certificate() + self.client.config.dry_run = True + + # Two authzs that are already valid and should get deactivated (dry run) + authzrs = self._authzr_from_domains(["example.com", "www.example.com"]) + for authzr in authzrs: + authzr.body.status = messages.STATUS_VALID + + # One deactivation succeeds, one fails + auth_handler = self.client.auth_handler + auth_handler.deactivate_valid_authorizations.return_value = ([authzrs[0]], [authzrs[1]]) + + # Certificate should get issued despite one failed deactivation + self.eg_order.authorizations = authzrs + self.client.auth_handler.handle_authorizations.return_value = authzrs + with test_util.patch_get_utility(): + result = self.client.obtain_certificate(self.eg_domains) + self.assertEqual(result, (mock.sentinel.cert, mock.sentinel.chain, key, csr)) + self._check_obtain_certificate(1) + + # Deactivation success/failure should have been handled properly + self.assertEqual(auth_handler.deactivate_valid_authorizations.call_count, 1, + "Deactivate authorizations should be called") + self.assertEqual(self.acme.new_order.call_count, 2, + "Order should be recreated due to successfully deactivated authorizations") + mock_log.warning.assert_called_with("Certbot was unable to obtain fresh authorizations for" + " every domain. The dry run will continue, but results" + " may not be accurate.") + def _set_mock_from_fullchain(self, mock_from_fullchain): mock_cert = mock.Mock() mock_cert.encode.return_value = mock.sentinel.cert @@ -399,8 +440,8 @@ class ClientTest(ClientTestCommon): (mock.sentinel.cert, mock.sentinel.chain, key, csr)) self._check_obtain_certificate(auth_count) - @mock.patch('certbot.client.Client.obtain_certificate') - @mock.patch('certbot.storage.RenewableCert.new_lineage') + @mock.patch('certbot._internal.client.Client.obtain_certificate') + @mock.patch('certbot._internal.storage.RenewableCert.new_lineage') def test_obtain_and_enroll_certificate(self, mock_storage, mock_obtain_certificate): domains = ["*.example.com", "example.com"] @@ -420,12 +461,11 @@ class ClientTest(ClientTestCommon): names = [call[0][0] for call in mock_storage.call_args_list] self.assertEqual(names, ["example_cert", "example.com", "example.com"]) - @mock.patch("certbot.cli.helpful_parser") + @mock.patch("certbot._internal.cli.helpful_parser") def test_save_certificate(self, mock_parser): - # pylint: disable=too-many-locals certs = ["cert_512.pem", "cert-san_512.pem"] tmp_path = tempfile.mkdtemp() - os.chmod(tmp_path, 0o755) # TODO: really?? + filesystem.chmod(tmp_path, 0o755) # TODO: really?? cert_pem = test_util.load_vector(certs[0]) chain_pem = (test_util.load_vector(certs[0]) + test_util.load_vector(certs[1])) @@ -522,7 +562,7 @@ class ClientTest(ClientTestCommon): class EnhanceConfigTest(ClientTestCommon): - """Tests for certbot.client.Client.enhance_config.""" + """Tests for certbot._internal.client.Client.enhance_config.""" def setUp(self): super(EnhanceConfigTest, self).setUp() @@ -537,20 +577,20 @@ class EnhanceConfigTest(ClientTestCommon): self.assertRaises( errors.Error, self.client.enhance_config, [self.domain], None) - @mock.patch("certbot.client.enhancements") + @mock.patch("certbot._internal.client.enhancements") def test_unsupported(self, mock_enhancements): self.client.installer = mock.MagicMock() self.client.installer.supported_enhancements.return_value = [] self.config.redirect = None self.config.hsts = True - with mock.patch("certbot.client.logger") as mock_logger: + with mock.patch("certbot._internal.client.logger") as mock_logger: self.client.enhance_config([self.domain], None) self.assertEqual(mock_logger.warning.call_count, 1) self.client.installer.enhance.assert_not_called() mock_enhancements.ask.assert_not_called() - @mock.patch("certbot.client.logger") + @mock.patch("certbot._internal.client.logger") def test_already_exists_header(self, mock_log): self.config.hsts = True self._test_with_already_existing() @@ -558,7 +598,7 @@ class EnhanceConfigTest(ClientTestCommon): self.assertEqual(mock_log.warning.call_args[0][1], 'Strict-Transport-Security') - @mock.patch("certbot.client.logger") + @mock.patch("certbot._internal.client.logger") def test_already_exists_redirect(self, mock_log): self.config.redirect = True self._test_with_already_existing() @@ -566,6 +606,21 @@ class EnhanceConfigTest(ClientTestCommon): self.assertEqual(mock_log.warning.call_args[0][1], 'redirect') + @mock.patch("certbot._internal.client.logger") + def test_config_set_no_warning_redirect(self, mock_log): + self.config.redirect = False + self._test_with_already_existing() + self.assertFalse(mock_log.warning.called) + + @mock.patch("certbot._internal.client.enhancements.ask") + @mock.patch("certbot._internal.client.logger") + def test_warn_redirect(self, mock_log, mock_ask): + self.config.redirect = None + mock_ask.return_value = False + self._test_with_already_existing() + self.assertTrue(mock_log.warning.called) + self.assertTrue("disable" in mock_log.warning.call_args[0][0]) + def test_no_ask_hsts(self): self.config.hsts = True self._test_with_all_supported() @@ -615,7 +670,7 @@ class EnhanceConfigTest(ClientTestCommon): self.client.installer = installer self._test_error_with_rollback() - @mock.patch("certbot.client.enhancements.ask") + @mock.patch("certbot._internal.client.enhancements.ask") def test_ask(self, mock_ask): self.config.redirect = None mock_ask.return_value = True @@ -650,15 +705,15 @@ class EnhanceConfigTest(ClientTestCommon): class RollbackTest(unittest.TestCase): - """Tests for certbot.client.rollback.""" + """Tests for certbot._internal.client.rollback.""" def setUp(self): self.m_install = mock.MagicMock() @classmethod def _call(cls, checkpoints, side_effect): - from certbot.client import rollback - with mock.patch("certbot.client.plugin_selection.pick_installer") as mpi: + from certbot._internal.client import rollback + with mock.patch("certbot._internal.client.plugin_selection.pick_installer") as mpi: mpi.side_effect = side_effect rollback(None, checkpoints, {}, mock.MagicMock()) diff --git a/certbot-nginx/docs/_static/.gitignore b/certbot/tests/compat/__init__.py similarity index 100% rename from certbot-nginx/docs/_static/.gitignore rename to certbot/tests/compat/__init__.py diff --git a/certbot/tests/compat/filesystem_test.py b/certbot/tests/compat/filesystem_test.py new file mode 100644 index 000000000..e721bbd48 --- /dev/null +++ b/certbot/tests/compat/filesystem_test.py @@ -0,0 +1,547 @@ +"""Tests for certbot.compat.filesystem""" +import contextlib +import errno +import unittest + +import mock + +from certbot import util +from certbot._internal import lock +from certbot.compat import filesystem +from certbot.compat import os +import certbot.tests.util as test_util +from certbot.tests.util import TempDirTestCase + +try: + # pylint: disable=import-error + import win32api + import win32security + import ntsecuritycon + # pylint: enable=import-error + POSIX_MODE = False +except ImportError: + POSIX_MODE = True + + + +EVERYBODY_SID = 'S-1-1-0' +SYSTEM_SID = 'S-1-5-18' +ADMINS_SID = 'S-1-5-32-544' + + +@unittest.skipIf(POSIX_MODE, reason='Tests specific to Windows security') +class WindowsChmodTests(TempDirTestCase): + """Unit tests for Windows chmod function in filesystem module""" + def setUp(self): + super(WindowsChmodTests, self).setUp() + self.probe_path = _create_probe(self.tempdir) + + def test_symlink_resolution(self): + link_path = os.path.join(self.tempdir, 'link') + os.symlink(self.probe_path, link_path) + + ref_dacl_probe = _get_security_dacl(self.probe_path).GetSecurityDescriptorDacl() + ref_dacl_link = _get_security_dacl(link_path).GetSecurityDescriptorDacl() + + filesystem.chmod(link_path, 0o700) + + # Assert the real file is impacted, not the link. + cur_dacl_probe = _get_security_dacl(self.probe_path).GetSecurityDescriptorDacl() + cur_dacl_link = _get_security_dacl(link_path).GetSecurityDescriptorDacl() + self.assertFalse(filesystem._compare_dacls(ref_dacl_probe, cur_dacl_probe)) # pylint: disable=protected-access + self.assertTrue(filesystem._compare_dacls(ref_dacl_link, cur_dacl_link)) # pylint: disable=protected-access + + def test_world_permission(self): + everybody = win32security.ConvertStringSidToSid(EVERYBODY_SID) + + filesystem.chmod(self.probe_path, 0o700) + dacl = _get_security_dacl(self.probe_path).GetSecurityDescriptorDacl() + + self.assertFalse([dacl.GetAce(index) for index in range(0, dacl.GetAceCount()) + if dacl.GetAce(index)[2] == everybody]) + + filesystem.chmod(self.probe_path, 0o704) + dacl = _get_security_dacl(self.probe_path).GetSecurityDescriptorDacl() + + self.assertTrue([dacl.GetAce(index) for index in range(0, dacl.GetAceCount()) + if dacl.GetAce(index)[2] == everybody]) + + def test_group_permissions_noop(self): + filesystem.chmod(self.probe_path, 0o700) + ref_dacl_probe = _get_security_dacl(self.probe_path).GetSecurityDescriptorDacl() + + filesystem.chmod(self.probe_path, 0o740) + cur_dacl_probe = _get_security_dacl(self.probe_path).GetSecurityDescriptorDacl() + + self.assertTrue(filesystem._compare_dacls(ref_dacl_probe, cur_dacl_probe)) # pylint: disable=protected-access + + def test_admin_permissions(self): + system = win32security.ConvertStringSidToSid(SYSTEM_SID) + admins = win32security.ConvertStringSidToSid(ADMINS_SID) + + filesystem.chmod(self.probe_path, 0o400) + dacl = _get_security_dacl(self.probe_path).GetSecurityDescriptorDacl() + + system_aces = [dacl.GetAce(index) for index in range(0, dacl.GetAceCount()) + if dacl.GetAce(index)[2] == system] + admin_aces = [dacl.GetAce(index) for index in range(0, dacl.GetAceCount()) + if dacl.GetAce(index)[2] == admins] + + self.assertEqual(len(system_aces), 1) + self.assertEqual(len(admin_aces), 1) + + self.assertEqual(system_aces[0][1], ntsecuritycon.FILE_ALL_ACCESS) + self.assertEqual(admin_aces[0][1], ntsecuritycon.FILE_ALL_ACCESS) + + def test_read_flag(self): + self._test_flag(4, ntsecuritycon.FILE_GENERIC_READ) + + def test_execute_flag(self): + self._test_flag(1, ntsecuritycon.FILE_GENERIC_EXECUTE) + + def test_write_flag(self): + self._test_flag(2, (ntsecuritycon.FILE_ALL_ACCESS + ^ ntsecuritycon.FILE_GENERIC_READ + ^ ntsecuritycon.FILE_GENERIC_EXECUTE)) + + def test_full_flag(self): + self._test_flag(7, ntsecuritycon.FILE_ALL_ACCESS) + + def _test_flag(self, everyone_mode, windows_flag): + # Note that flag is tested against `everyone`, not `user`, because practically these unit + # tests are executed with admin privilege, so current user is effectively the admins group, + # and so will always have all rights. + filesystem.chmod(self.probe_path, 0o700 | everyone_mode) + dacl = _get_security_dacl(self.probe_path).GetSecurityDescriptorDacl() + everybody = win32security.ConvertStringSidToSid(EVERYBODY_SID) + + acls_everybody = [dacl.GetAce(index) for index in range(0, dacl.GetAceCount()) + if dacl.GetAce(index)[2] == everybody] + + self.assertEqual(len(acls_everybody), 1) + + acls_everybody = acls_everybody[0] + + self.assertEqual(acls_everybody[1], windows_flag) + + def test_user_admin_dacl_consistency(self): + # Set ownership of target to authenticated user + authenticated_user, _, _ = win32security.LookupAccountName("", win32api.GetUserName()) + security_owner = _get_security_owner(self.probe_path) + _set_owner(self.probe_path, security_owner, authenticated_user) + + filesystem.chmod(self.probe_path, 0o700) + + security_dacl = _get_security_dacl(self.probe_path) + # We expect three ACE: one for admins, one for system, and one for the user + self.assertEqual(security_dacl.GetSecurityDescriptorDacl().GetAceCount(), 3) + + # Set ownership of target to Administrators user group + admin_user = win32security.ConvertStringSidToSid(ADMINS_SID) + security_owner = _get_security_owner(self.probe_path) + _set_owner(self.probe_path, security_owner, admin_user) + + filesystem.chmod(self.probe_path, 0o700) + + security_dacl = _get_security_dacl(self.probe_path) + # We expect only two ACE: one for admins, one for system, + # since the user is also the admins group + self.assertEqual(security_dacl.GetSecurityDescriptorDacl().GetAceCount(), 2) + + +class ComputePrivateKeyModeTest(TempDirTestCase): + def setUp(self): + super(ComputePrivateKeyModeTest, self).setUp() + self.probe_path = _create_probe(self.tempdir) + + def test_compute_private_key_mode(self): + filesystem.chmod(self.probe_path, 0o777) + new_mode = filesystem.compute_private_key_mode(self.probe_path, 0o600) + + if POSIX_MODE: + # On Linux RWX permissions for group and R permission for world + # are persisted from the existing moe + self.assertEqual(new_mode, 0o674) + else: + # On Windows no permission is persisted + self.assertEqual(new_mode, 0o600) + + +@unittest.skipIf(POSIX_MODE, reason='Tests specific to Windows security') +class WindowsOpenTest(TempDirTestCase): + def test_new_file_correct_permissions(self): + path = os.path.join(self.tempdir, 'file') + + desc = filesystem.open(path, os.O_CREAT | os.O_EXCL | os.O_RDWR, 0o700) + os.close(desc) + + dacl = _get_security_dacl(path).GetSecurityDescriptorDacl() + everybody = win32security.ConvertStringSidToSid(EVERYBODY_SID) + + self.assertFalse([dacl.GetAce(index) for index in range(0, dacl.GetAceCount()) + if dacl.GetAce(index)[2] == everybody]) + + def test_existing_file_correct_permissions(self): + path = os.path.join(self.tempdir, 'file') + open(path, 'w').close() + + desc = filesystem.open(path, os.O_EXCL | os.O_RDWR, 0o700) + os.close(desc) + + dacl = _get_security_dacl(path).GetSecurityDescriptorDacl() + everybody = win32security.ConvertStringSidToSid(EVERYBODY_SID) + + self.assertFalse([dacl.GetAce(index) for index in range(0, dacl.GetAceCount()) + if dacl.GetAce(index)[2] == everybody]) + + def test_create_file_on_open(self): + # os.O_CREAT | os.O_EXCL + file not exists = OK + self._test_one_creation(1, file_exist=False, flags=(os.O_CREAT | os.O_EXCL)) + + # os.O_CREAT | os.O_EXCL + file exists = EEXIST OS exception + with self.assertRaises(OSError) as raised: + self._test_one_creation(2, file_exist=True, flags=(os.O_CREAT | os.O_EXCL)) + self.assertEqual(raised.exception.errno, errno.EEXIST) + + # os.O_CREAT + file not exists = OK + self._test_one_creation(3, file_exist=False, flags=os.O_CREAT) + + # os.O_CREAT + file exists = OK + self._test_one_creation(4, file_exist=True, flags=os.O_CREAT) + + # os.O_CREAT + file exists (locked) = EACCES OS exception + path = os.path.join(self.tempdir, '5') + open(path, 'w').close() + filelock = lock.LockFile(path) + try: + with self.assertRaises(OSError) as raised: + self._test_one_creation(5, file_exist=True, flags=os.O_CREAT) + self.assertEqual(raised.exception.errno, errno.EACCES) + finally: + filelock.release() + + # os.O_CREAT not set + file not exists = OS exception + with self.assertRaises(OSError): + self._test_one_creation(6, file_exist=False, flags=os.O_RDONLY) + + def _test_one_creation(self, num, file_exist, flags): + one_file = os.path.join(self.tempdir, str(num)) + if file_exist and not os.path.exists(one_file): + with open(one_file, 'w'): + pass + + handler = None + try: + handler = filesystem.open(one_file, flags) + finally: + if handler: + os.close(handler) + + +@unittest.skipIf(POSIX_MODE, reason='Test specific to Windows security') +class WindowsMkdirTests(test_util.TempDirTestCase): + """Unit tests for Windows mkdir + makedirs functions in filesystem module""" + def test_mkdir_correct_permissions(self): + path = os.path.join(self.tempdir, 'dir') + + filesystem.mkdir(path, 0o700) + + everybody = win32security.ConvertStringSidToSid(EVERYBODY_SID) + + dacl = _get_security_dacl(path).GetSecurityDescriptorDacl() + self.assertFalse([dacl.GetAce(index) for index in range(0, dacl.GetAceCount()) + if dacl.GetAce(index)[2] == everybody]) + + def test_makedirs_correct_permissions(self): + path = os.path.join(self.tempdir, 'dir') + subpath = os.path.join(path, 'subpath') + + filesystem.makedirs(subpath, 0o700) + + everybody = win32security.ConvertStringSidToSid(EVERYBODY_SID) + + dacl = _get_security_dacl(subpath).GetSecurityDescriptorDacl() + self.assertFalse([dacl.GetAce(index) for index in range(0, dacl.GetAceCount()) + if dacl.GetAce(index)[2] == everybody]) + + def test_makedirs_switch_os_mkdir(self): + path = os.path.join(self.tempdir, 'dir') + import os as std_os # pylint: disable=os-module-forbidden + original_mkdir = std_os.mkdir + + filesystem.makedirs(path) + self.assertEqual(original_mkdir, std_os.mkdir) + + try: + filesystem.makedirs(path) # Will fail because path already exists + except OSError: + pass + self.assertEqual(original_mkdir, std_os.mkdir) + + +class OwnershipTest(test_util.TempDirTestCase): + """Tests about copy_ownership_and_apply_mode and has_same_ownership""" + def setUp(self): + super(OwnershipTest, self).setUp() + self.probe_path = _create_probe(self.tempdir) + + @unittest.skipIf(POSIX_MODE, reason='Test specific to Windows security') + def test_copy_ownership_windows(self): + system = win32security.ConvertStringSidToSid(SYSTEM_SID) + security = win32security.SECURITY_ATTRIBUTES().SECURITY_DESCRIPTOR + security.SetSecurityDescriptorOwner(system, False) + + with mock.patch('win32security.GetFileSecurity') as mock_get: + with mock.patch('win32security.SetFileSecurity') as mock_set: + mock_get.return_value = security + filesystem.copy_ownership_and_apply_mode( + 'dummy', self.probe_path, 0o700, copy_user=True, copy_group=False) + + self.assertEqual(mock_set.call_count, 2) + + first_call = mock_set.call_args_list[0] + security = first_call[0][2] + self.assertEqual(system, security.GetSecurityDescriptorOwner()) + + second_call = mock_set.call_args_list[1] + security = second_call[0][2] + dacl = security.GetSecurityDescriptorDacl() + everybody = win32security.ConvertStringSidToSid(EVERYBODY_SID) + self.assertTrue(dacl.GetAceCount()) + self.assertFalse([dacl.GetAce(index) for index in range(0, dacl.GetAceCount()) + if dacl.GetAce(index)[2] == everybody]) + + @unittest.skipUnless(POSIX_MODE, reason='Test specific to Linux security') + def test_copy_ownership_linux(self): + with mock.patch('os.chown') as mock_chown: + with mock.patch('os.chmod') as mock_chmod: + with mock.patch('os.stat') as mock_stat: + mock_stat.return_value.st_uid = 50 + mock_stat.return_value.st_gid = 51 + filesystem.copy_ownership_and_apply_mode( + 'dummy', self.probe_path, 0o700, copy_user=True, copy_group=True) + + mock_chown.assert_called_once_with(self.probe_path, 50, 51) + mock_chmod.assert_called_once_with(self.probe_path, 0o700) + + def test_has_same_ownership(self): + path1 = os.path.join(self.tempdir, 'test1') + path2 = os.path.join(self.tempdir, 'test2') + + util.safe_open(path1, 'w').close() + util.safe_open(path2, 'w').close() + + self.assertTrue(filesystem.has_same_ownership(path1, path2)) + + +class CheckPermissionsTest(test_util.TempDirTestCase): + """Tests relative to functions that check modes.""" + def setUp(self): + super(CheckPermissionsTest, self).setUp() + self.probe_path = _create_probe(self.tempdir) + + def test_check_mode(self): + self.assertTrue(filesystem.check_mode(self.probe_path, 0o744)) + + filesystem.chmod(self.probe_path, 0o700) + self.assertFalse(filesystem.check_mode(self.probe_path, 0o744)) + + @unittest.skipIf(POSIX_MODE, reason='Test specific to Windows security') + def test_check_owner_windows(self): + self.assertTrue(filesystem.check_owner(self.probe_path)) + + system = win32security.ConvertStringSidToSid(SYSTEM_SID) + security = win32security.SECURITY_ATTRIBUTES().SECURITY_DESCRIPTOR + security.SetSecurityDescriptorOwner(system, False) + + with mock.patch('win32security.GetFileSecurity') as mock_get: + mock_get.return_value = security + self.assertFalse(filesystem.check_owner(self.probe_path)) + + @unittest.skipUnless(POSIX_MODE, reason='Test specific to Linux security') + def test_check_owner_linux(self): + self.assertTrue(filesystem.check_owner(self.probe_path)) + + import os as std_os # pylint: disable=os-module-forbidden + # See related inline comment in certbot.compat.filesystem.check_owner method + # that explains why MyPy/PyLint check disable is needed here. + uid = std_os.getuid() + + with mock.patch('os.getuid') as mock_uid: + mock_uid.return_value = uid + 1 + self.assertFalse(filesystem.check_owner(self.probe_path)) + + def test_check_permissions(self): + self.assertTrue(filesystem.check_permissions(self.probe_path, 0o744)) + + with mock.patch('certbot.compat.filesystem.check_mode') as mock_mode: + mock_mode.return_value = False + self.assertFalse(filesystem.check_permissions(self.probe_path, 0o744)) + + with mock.patch('certbot.compat.filesystem.check_owner') as mock_owner: + mock_owner.return_value = False + self.assertFalse(filesystem.check_permissions(self.probe_path, 0o744)) + + def test_check_min_permissions(self): + filesystem.chmod(self.probe_path, 0o744) + self.assertTrue(filesystem.has_min_permissions(self.probe_path, 0o744)) + + filesystem.chmod(self.probe_path, 0o700) + self.assertFalse(filesystem.has_min_permissions(self.probe_path, 0o744)) + + filesystem.chmod(self.probe_path, 0o741) + self.assertFalse(filesystem.has_min_permissions(self.probe_path, 0o744)) + + def test_is_world_reachable(self): + filesystem.chmod(self.probe_path, 0o744) + self.assertTrue(filesystem.has_world_permissions(self.probe_path)) + + filesystem.chmod(self.probe_path, 0o700) + self.assertFalse(filesystem.has_world_permissions(self.probe_path)) + + +class OsReplaceTest(test_util.TempDirTestCase): + """Test to ensure consistent behavior of rename method""" + def test_os_replace_to_existing_file(self): + """Ensure that replace will effectively rename src into dst for all platforms.""" + src = os.path.join(self.tempdir, 'src') + dst = os.path.join(self.tempdir, 'dst') + open(src, 'w').close() + open(dst, 'w').close() + + # On Windows, a direct call to os.rename would fail because dst already exists. + filesystem.replace(src, dst) + + self.assertFalse(os.path.exists(src)) + self.assertTrue(os.path.exists(dst)) + + +class RealpathTest(test_util.TempDirTestCase): + """Tests for realpath method""" + def setUp(self): + super(RealpathTest, self).setUp() + self.probe_path = _create_probe(self.tempdir) + + def test_symlink_resolution(self): + # Remove any symlinks already in probe_path + self.probe_path = filesystem.realpath(self.probe_path) + # Absolute resolution + link_path = os.path.join(self.tempdir, 'link_abs') + os.symlink(self.probe_path, link_path) + + self.assertEqual(self.probe_path, filesystem.realpath(self.probe_path)) + self.assertEqual(self.probe_path, filesystem.realpath(link_path)) + + # Relative resolution + curdir = os.getcwd() + link_path = os.path.join(self.tempdir, 'link_rel') + probe_name = os.path.basename(self.probe_path) + try: + os.chdir(os.path.dirname(self.probe_path)) + os.symlink(probe_name, link_path) + + self.assertEqual(self.probe_path, filesystem.realpath(probe_name)) + self.assertEqual(self.probe_path, filesystem.realpath(link_path)) + finally: + os.chdir(curdir) + + def test_symlink_loop_mitigation(self): + link1_path = os.path.join(self.tempdir, 'link1') + link2_path = os.path.join(self.tempdir, 'link2') + link3_path = os.path.join(self.tempdir, 'link3') + os.symlink(link1_path, link2_path) + os.symlink(link2_path, link3_path) + os.symlink(link3_path, link1_path) + + with self.assertRaises(RuntimeError) as error: + filesystem.realpath(link1_path) + self.assertTrue('link1 is a loop!' in str(error.exception)) + + +class IsExecutableTest(test_util.TempDirTestCase): + """Tests for is_executable method""" + def test_not_executable(self): + file_path = os.path.join(self.tempdir, "foo") + + # On Windows a file created within Certbot will always have all permissions to the + # Administrators group set. Since the unit tests are typically executed under elevated + # privileges, it means that current user will always have effective execute rights on the + # hook script, and so the test will fail. To prevent that and represent a file created + # outside Certbot as typically a hook file is, we mock the _generate_dacl function in + # certbot.compat.filesystem to give rights only to the current user. This implies removing + # all ACEs except the first one from the DACL created by original _generate_dacl function. + + from certbot.compat.filesystem import _generate_dacl + + def _execute_mock(user_sid, mode): + dacl = _generate_dacl(user_sid, mode) + for _ in range(1, dacl.GetAceCount()): + dacl.DeleteAce(1) # DeleteAce dynamically updates the internal index mapping. + return dacl + + # create a non-executable file + with mock.patch("certbot.compat.filesystem._generate_dacl", side_effect=_execute_mock): + os.close(filesystem.open(file_path, os.O_CREAT | os.O_WRONLY, 0o666)) + + self.assertFalse(filesystem.is_executable(file_path)) + + @mock.patch("certbot.compat.filesystem.os.path.isfile") + @mock.patch("certbot.compat.filesystem.os.access") + def test_full_path(self, mock_access, mock_isfile): + with _fix_windows_runtime(): + mock_access.return_value = True + mock_isfile.return_value = True + self.assertTrue(filesystem.is_executable("/path/to/exe")) + + @mock.patch("certbot.compat.filesystem.os.path.isfile") + @mock.patch("certbot.compat.filesystem.os.access") + def test_rel_path(self, mock_access, mock_isfile): + with _fix_windows_runtime(): + mock_access.return_value = True + mock_isfile.return_value = True + self.assertTrue(filesystem.is_executable("exe")) + + @mock.patch("certbot.compat.filesystem.os.path.isfile") + @mock.patch("certbot.compat.filesystem.os.access") + def test_not_found(self, mock_access, mock_isfile): + with _fix_windows_runtime(): + mock_access.return_value = True + mock_isfile.return_value = False + self.assertFalse(filesystem.is_executable("exe")) + + +@contextlib.contextmanager +def _fix_windows_runtime(): + if os.name != 'nt': + yield + else: + with mock.patch('win32security.GetFileSecurity') as mock_get: + dacl_mock = mock_get.return_value.GetSecurityDescriptorDacl + mode_mock = dacl_mock.return_value.GetEffectiveRightsFromAcl + mode_mock.return_value = ntsecuritycon.FILE_GENERIC_EXECUTE + yield + + +def _get_security_dacl(target): + return win32security.GetFileSecurity(target, win32security.DACL_SECURITY_INFORMATION) + + +def _get_security_owner(target): + return win32security.GetFileSecurity(target, win32security.OWNER_SECURITY_INFORMATION) + + +def _set_owner(target, security_owner, user): + security_owner.SetSecurityDescriptorOwner(user, False) + win32security.SetFileSecurity( + target, win32security.OWNER_SECURITY_INFORMATION, security_owner) + + +def _create_probe(tempdir): + filesystem.chmod(tempdir, 0o744) + probe_path = os.path.join(tempdir, 'probe') + util.safe_open(probe_path, 'w', chmod=0o744).close() + return probe_path + + +if __name__ == "__main__": + unittest.main() # pragma: no cover diff --git a/certbot/tests/compat/os_test.py b/certbot/tests/compat/os_test.py new file mode 100644 index 000000000..2fe23f700 --- /dev/null +++ b/certbot/tests/compat/os_test.py @@ -0,0 +1,20 @@ +"""Unit test for os module.""" +import unittest + +from certbot.compat import os + + +class OsTest(unittest.TestCase): + """Unit tests for os module.""" + def test_forbidden_methods(self): + # Checks for os module + for method in ['chmod', 'chown', 'open', 'mkdir', 'makedirs', 'rename', + 'replace', 'access', 'stat', 'fstat']: + self.assertRaises(RuntimeError, getattr(os, method)) + # Checks for os.path module + for method in ['realpath']: + self.assertRaises(RuntimeError, getattr(os.path, method)) + + +if __name__ == "__main__": + unittest.main() # pragma: no cover diff --git a/certbot/tests/compat_test.py b/certbot/tests/compat_test.py deleted file mode 100644 index 552aa5645..000000000 --- a/certbot/tests/compat_test.py +++ /dev/null @@ -1,21 +0,0 @@ -"""Tests for certbot.compat.""" -import os - -from certbot import compat -import certbot.tests.util as test_util - -class OsReplaceTest(test_util.TempDirTestCase): - """Test to ensure consistent behavior of os_rename method""" - - def test_os_rename_to_existing_file(self): - """Ensure that os_rename will effectively rename src into dst for all platforms.""" - src = os.path.join(self.tempdir, 'src') - dst = os.path.join(self.tempdir, 'dst') - open(src, 'w').close() - open(dst, 'w').close() - - # On Windows, a direct call to os.rename will fail because dst already exists. - compat.os_rename(src, dst) - - self.assertFalse(os.path.exists(src)) - self.assertTrue(os.path.exists(dst)) diff --git a/certbot/tests/configuration_test.py b/certbot/tests/configuration_test.py index 10d9059b3..d748b9bfb 100644 --- a/certbot/tests/configuration_test.py +++ b/certbot/tests/configuration_test.py @@ -1,28 +1,28 @@ -"""Tests for certbot.configuration.""" -import os +"""Tests for certbot._internal.configuration.""" import unittest import mock -from certbot import compat -from certbot import constants from certbot import errors - +from certbot._internal import constants +from certbot.compat import misc +from certbot.compat import os from certbot.tests import util as test_util + class NamespaceConfigTest(test_util.ConfigTestCase): - """Tests for certbot.configuration.NamespaceConfig.""" + """Tests for certbot._internal.configuration.NamespaceConfig.""" def setUp(self): super(NamespaceConfigTest, self).setUp() - self.config.foo = 'bar' + self.config.foo = 'bar' # pylint: disable=blacklisted-name self.config.server = 'https://acme-server.org:443/new' - self.config.tls_sni_01_port = 1234 + self.config.https_port = 1234 self.config.http01_port = 4321 def test_init_same_ports(self): - self.config.namespace.tls_sni_01_port = 4321 - from certbot.configuration import NamespaceConfig + self.config.namespace.https_port = 4321 + from certbot._internal.configuration import NamespaceConfig self.assertRaises(errors.Error, NamespaceConfig, self.config.namespace) def test_proxy_getattr(self): @@ -38,7 +38,7 @@ class NamespaceConfigTest(test_util.ConfigTestCase): self.assertEqual(['user:pass@acme.server:443', 'p', 'a', 't', 'h'], self.config.server_path.split(os.path.sep)) - @mock.patch('certbot.configuration.constants') + @mock.patch('certbot._internal.configuration.constants') def test_dynamic_dirs(self, mock_constants): mock_constants.ACCOUNTS_DIR = 'acc' mock_constants.BACKUP_DIR = 'backups' @@ -48,7 +48,7 @@ class NamespaceConfigTest(test_util.ConfigTestCase): mock_constants.KEY_DIR = 'keys' mock_constants.TEMP_CHECKPOINT_DIR = 't' - ref_path = compat.underscores_for_unsupported_characters_in_path( + ref_path = misc.underscores_for_unsupported_characters_in_path( 'acc/acme-server.org:443/new') self.assertEqual( os.path.normpath(self.config.accounts_dir), @@ -70,7 +70,7 @@ class NamespaceConfigTest(test_util.ConfigTestCase): os.path.normpath(os.path.join(self.config.work_dir, 't'))) def test_absolute_paths(self): - from certbot.configuration import NamespaceConfig + from certbot._internal.configuration import NamespaceConfig config_base = "foo" work_base = "bar" @@ -79,7 +79,7 @@ class NamespaceConfigTest(test_util.ConfigTestCase): mock_namespace = mock.MagicMock(spec=['config_dir', 'work_dir', 'logs_dir', 'http01_port', - 'tls_sni_01_port', + 'https_port', 'domains', 'server']) mock_namespace.config_dir = config_base mock_namespace.work_dir = work_base @@ -103,7 +103,7 @@ class NamespaceConfigTest(test_util.ConfigTestCase): self.assertTrue(os.path.isabs(config.key_dir)) self.assertTrue(os.path.isabs(config.temp_checkpoint_dir)) - @mock.patch('certbot.configuration.constants') + @mock.patch('certbot._internal.configuration.constants') def test_renewal_dynamic_dirs(self, mock_constants): mock_constants.ARCHIVE_DIR = 'a' mock_constants.LIVE_DIR = 'l' @@ -118,7 +118,7 @@ class NamespaceConfigTest(test_util.ConfigTestCase): self.config.config_dir, 'renewal_configs')) def test_renewal_absolute_paths(self): - from certbot.configuration import NamespaceConfig + from certbot._internal.configuration import NamespaceConfig config_base = "foo" work_base = "bar" @@ -126,7 +126,7 @@ class NamespaceConfigTest(test_util.ConfigTestCase): mock_namespace = mock.MagicMock(spec=['config_dir', 'work_dir', 'logs_dir', 'http01_port', - 'tls_sni_01_port', + 'https_port', 'domains', 'server']) mock_namespace.config_dir = config_base mock_namespace.work_dir = work_base diff --git a/certbot/tests/crypto_util_test.py b/certbot/tests/crypto_util_test.py index c092efc51..1d642ae9e 100644 --- a/certbot/tests/crypto_util_test.py +++ b/certbot/tests/crypto_util_test.py @@ -1,18 +1,18 @@ """Tests for certbot.crypto_util.""" import logging -import os import unittest -import OpenSSL import mock +import OpenSSL import zope.component from certbot import errors from certbot import interfaces from certbot import util +from certbot.compat import filesystem +from certbot.compat import os import certbot.tests.util as test_util - RSA256_KEY = test_util.load_vector('rsa256_key.pem') RSA256_KEY_PATH = test_util.vector_path('rsa256_key.pem') RSA512_KEY = test_util.load_vector('rsa512_key.pem') @@ -30,6 +30,9 @@ class InitSaveKeyTest(test_util.TempDirTestCase): def setUp(self): super(InitSaveKeyTest, self).setUp() + self.workdir = os.path.join(self.tempdir, 'workdir') + filesystem.mkdir(self.workdir, mode=0o700) + logging.disable(logging.CRITICAL) zope.component.provideUtility( mock.Mock(strict_permissions=True), interfaces.IConfig) @@ -47,15 +50,15 @@ class InitSaveKeyTest(test_util.TempDirTestCase): @mock.patch('certbot.crypto_util.make_key') def test_success(self, mock_make): mock_make.return_value = b'key_pem' - key = self._call(1024, self.tempdir) + key = self._call(1024, self.workdir) self.assertEqual(key.pem, b'key_pem') self.assertTrue('key-certbot.pem' in key.file) - self.assertTrue(os.path.exists(os.path.join(self.tempdir, key.file))) + self.assertTrue(os.path.exists(os.path.join(self.workdir, key.file))) @mock.patch('certbot.crypto_util.make_key') def test_key_failure(self, mock_make): mock_make.side_effect = ValueError - self.assertRaises(ValueError, self._call, 431, self.tempdir) + self.assertRaises(ValueError, self._call, 431, self.workdir) class InitSaveCSRTest(test_util.TempDirTestCase): @@ -161,7 +164,7 @@ class ImportCSRFileTest(unittest.TestCase): test_util.load_vector('cert_512.pem')) -class MakeKeyTest(unittest.TestCase): # pylint: disable=too-few-public-methods +class MakeKeyTest(unittest.TestCase): """Tests for certbot.crypto_util.make_key.""" def test_it(self): # pylint: disable=no-self-use @@ -178,23 +181,20 @@ class VerifyCertSetup(unittest.TestCase): super(VerifyCertSetup, self).setUp() self.renewable_cert = mock.MagicMock() - self.renewable_cert.cert = SS_CERT_PATH - self.renewable_cert.chain = SS_CERT_PATH - self.renewable_cert.privkey = RSA2048_KEY_PATH - self.renewable_cert.fullchain = test_util.vector_path('cert_fullchain_2048.pem') + self.renewable_cert.cert_path = SS_CERT_PATH + self.renewable_cert.chain_path = SS_CERT_PATH + self.renewable_cert.key_path = RSA2048_KEY_PATH + self.renewable_cert.fullchain_path = test_util.vector_path('cert_fullchain_2048.pem') self.bad_renewable_cert = mock.MagicMock() - self.bad_renewable_cert.chain = SS_CERT_PATH - self.bad_renewable_cert.cert = SS_CERT_PATH - self.bad_renewable_cert.fullchain = SS_CERT_PATH + self.bad_renewable_cert.chain_path = SS_CERT_PATH + self.bad_renewable_cert.cert_path = SS_CERT_PATH + self.bad_renewable_cert.fullchain_path = SS_CERT_PATH class VerifyRenewableCertTest(VerifyCertSetup): """Tests for certbot.crypto_util.verify_renewable_cert.""" - def setUp(self): - super(VerifyRenewableCertTest, self).setUp() - def _call(self, renewable_cert): from certbot.crypto_util import verify_renewable_cert return verify_renewable_cert(renewable_cert) @@ -210,9 +210,6 @@ class VerifyRenewableCertTest(VerifyCertSetup): class VerifyRenewableCertSigTest(VerifyCertSetup): """Tests for certbot.crypto_util.verify_renewable_cert.""" - def setUp(self): - super(VerifyRenewableCertSigTest, self).setUp() - def _call(self, renewable_cert): from certbot.crypto_util import verify_renewable_cert_sig return verify_renewable_cert_sig(renewable_cert) @@ -222,22 +219,19 @@ class VerifyRenewableCertSigTest(VerifyCertSetup): def test_cert_sig_match_ec(self): renewable_cert = mock.MagicMock() - renewable_cert.cert = P256_CERT_PATH - renewable_cert.chain = P256_CERT_PATH - renewable_cert.privkey = P256_KEY + renewable_cert.cert_path = P256_CERT_PATH + renewable_cert.chain_path = P256_CERT_PATH + renewable_cert.key_path = P256_KEY self.assertEqual(None, self._call(renewable_cert)) def test_cert_sig_mismatch(self): - self.bad_renewable_cert.cert = test_util.vector_path('cert_512_bad.pem') + self.bad_renewable_cert.cert_path = test_util.vector_path('cert_512_bad.pem') self.assertRaises(errors.Error, self._call, self.bad_renewable_cert) class VerifyFullchainTest(VerifyCertSetup): """Tests for certbot.crypto_util.verify_fullchain.""" - def setUp(self): - super(VerifyFullchainTest, self).setUp() - def _call(self, renewable_cert): from certbot.crypto_util import verify_fullchain return verify_fullchain(renewable_cert) @@ -256,9 +250,6 @@ class VerifyFullchainTest(VerifyCertSetup): class VerifyCertMatchesPrivKeyTest(VerifyCertSetup): """Tests for certbot.crypto_util.verify_cert_matches_priv_key.""" - def setUp(self): - super(VerifyCertMatchesPrivKeyTest, self).setUp() - def _call(self, renewable_cert): from certbot.crypto_util import verify_cert_matches_priv_key return verify_cert_matches_priv_key(renewable_cert.cert, renewable_cert.privkey) diff --git a/certbot/tests/display/completer_test.py b/certbot/tests/display/completer_test.py index 455bf5e1e..5ddf69266 100644 --- a/certbot/tests/display/completer_test.py +++ b/certbot/tests/display/completer_test.py @@ -1,9 +1,8 @@ -"""Test certbot.display.completer.""" -import os +"""Test certbot._internal.display.completer.""" try: - import readline # pylint: disable=import-error + import readline # pylint: disable=import-error except ImportError: - import certbot.display.dummy_readline as readline # type: ignore + import certbot._internal.display.dummy_readline as readline # type: ignore import string import sys import unittest @@ -12,10 +11,13 @@ import mock from six.moves import reload_module # pylint: disable=import-error from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module -import certbot.tests.util as test_util +from certbot.compat import filesystem # pylint: disable=ungrouped-imports +from certbot.compat import os # pylint: disable=ungrouped-imports +import certbot.tests.util as test_util # pylint: disable=ungrouped-imports + class CompleterTest(test_util.TempDirTestCase): - """Test certbot.display.completer.Completer.""" + """Test certbot._internal.display.completer.Completer.""" def setUp(self): super(CompleterTest, self).setUp() @@ -31,13 +33,13 @@ class CompleterTest(test_util.TempDirTestCase): path = os.path.join(self.tempdir, c) self.paths.append(path) if ord(c) % 2: - os.mkdir(path) + filesystem.mkdir(path) else: with open(path, 'w'): pass def test_complete(self): - from certbot.display import completer + from certbot._internal.display import completer my_completer = completer.Completer() num_paths = len(self.paths) @@ -61,7 +63,7 @@ class CompleterTest(test_util.TempDirTestCase): sys.modules['readline'] = original_readline def test_context_manager_with_unmocked_readline(self): - from certbot.display import completer + from certbot._internal.display import completer reload_module(completer) original_completer = readline.get_completer() @@ -73,18 +75,18 @@ class CompleterTest(test_util.TempDirTestCase): self.assertEqual(readline.get_completer(), original_completer) self.assertEqual(readline.get_completer_delims(), original_delims) - @mock.patch('certbot.display.completer.readline', autospec=True) + @mock.patch('certbot._internal.display.completer.readline', autospec=True) def test_context_manager_libedit(self, mock_readline): mock_readline.__doc__ = 'libedit' self._test_context_manager_with_mock_readline(mock_readline) - @mock.patch('certbot.display.completer.readline', autospec=True) + @mock.patch('certbot._internal.display.completer.readline', autospec=True) def test_context_manager_readline(self, mock_readline): mock_readline.__doc__ = 'GNU readline' self._test_context_manager_with_mock_readline(mock_readline) def _test_context_manager_with_mock_readline(self, mock_readline): - from certbot.display import completer + from certbot._internal.display import completer mock_readline.parse_and_bind.side_effect = enable_tab_completion @@ -100,5 +102,6 @@ def enable_tab_completion(unused_command): command = 'bind ^I rl_complete' if libedit else 'tab: complete' readline.parse_and_bind(command) + if __name__ == "__main__": unittest.main() # pragma: no cover diff --git a/certbot/tests/display/enhancements_test.py b/certbot/tests/display/enhancements_test.py index b8321d940..edace29b1 100644 --- a/certbot/tests/display/enhancements_test.py +++ b/certbot/tests/display/enhancements_test.py @@ -18,10 +18,10 @@ class AskTest(unittest.TestCase): @classmethod def _call(cls, enhancement): - from certbot.display.enhancements import ask + from certbot._internal.display.enhancements import ask return ask(enhancement) - @mock.patch("certbot.display.enhancements.util") + @mock.patch("certbot._internal.display.enhancements.util") def test_redirect(self, mock_util): mock_util().menu.return_value = (display_util.OK, 1) self.assertTrue(self._call("redirect")) @@ -34,20 +34,20 @@ class RedirectTest(unittest.TestCase): """Test the redirect_by_default method.""" @classmethod def _call(cls): - from certbot.display.enhancements import redirect_by_default + from certbot._internal.display.enhancements import redirect_by_default return redirect_by_default() - @mock.patch("certbot.display.enhancements.util") + @mock.patch("certbot._internal.display.enhancements.util") def test_secure(self, mock_util): mock_util().menu.return_value = (display_util.OK, 1) self.assertTrue(self._call()) - @mock.patch("certbot.display.enhancements.util") + @mock.patch("certbot._internal.display.enhancements.util") def test_cancel(self, mock_util): mock_util().menu.return_value = (display_util.CANCEL, 1) self.assertFalse(self._call()) - @mock.patch("certbot.display.enhancements.util") + @mock.patch("certbot._internal.display.enhancements.util") def test_easy(self, mock_util): mock_util().menu.return_value = (display_util.OK, 0) self.assertFalse(self._call()) diff --git a/certbot/tests/display/ops_test.py b/certbot/tests/display/ops_test.py index 9ad0ce87a..5df7bfcf8 100644 --- a/certbot/tests/display/ops_test.py +++ b/certbot/tests/display/ops_test.py @@ -1,6 +1,5 @@ # coding=utf-8 """Test certbot.display.ops.""" -import os import sys import unittest @@ -9,16 +8,14 @@ import mock import zope.component from acme import messages - -from certbot import account from certbot import errors - -from certbot.display import util as display_util +from certbot._internal import account +from certbot.compat import filesystem +from certbot.compat import os from certbot.display import ops - +from certbot.display import util as display_util import certbot.tests.util as test_util - KEY = jose.JWKRSA.load(test_util.load_vector("rsa512_key.pem")) @@ -43,7 +40,7 @@ class GetEmailTest(unittest.TestCase): mock_input.return_value = (display_util.OK, "foo@bar.baz") with mock.patch("certbot.display.ops.util.safe_email") as mock_safe_email: mock_safe_email.return_value = True - self.assertTrue(self._call() is "foo@bar.baz") + self.assertTrue(self._call() == "foo@bar.baz") @test_util.patch_get_utility("certbot.display.ops.z_util") def test_ok_not_safe(self, mock_get_utility): @@ -51,7 +48,7 @@ class GetEmailTest(unittest.TestCase): mock_input.return_value = (display_util.OK, "foo@bar.baz") with mock.patch("certbot.display.ops.util.safe_email") as mock_safe_email: mock_safe_email.side_effect = [False, True] - self.assertTrue(self._call() is "foo@bar.baz") + self.assertTrue(self._call() == "foo@bar.baz") @test_util.patch_get_utility("certbot.display.ops.z_util") def test_invalid_flag(self, mock_get_utility): @@ -96,7 +93,7 @@ class ChooseAccountTest(test_util.TempDirTestCase): False)) self.account_keys_dir = os.path.join(self.tempdir, "keys") - os.makedirs(self.account_keys_dir, 0o700) + filesystem.makedirs(self.account_keys_dir, 0o700) self.config = mock.MagicMock( accounts_dir=self.tempdir, @@ -356,7 +353,6 @@ class ChooseNamesTest(unittest.TestCase): class SuccessInstallationTest(unittest.TestCase): - # pylint: disable=too-few-public-methods """Test the success installation message.""" @classmethod def _call(cls, names): @@ -378,7 +374,6 @@ class SuccessInstallationTest(unittest.TestCase): class SuccessRenewalTest(unittest.TestCase): - # pylint: disable=too-few-public-methods """Test the success renewal message.""" @classmethod def _call(cls, names): @@ -399,7 +394,6 @@ class SuccessRenewalTest(unittest.TestCase): self.assertTrue(name in arg) class SuccessRevocationTest(unittest.TestCase): - # pylint: disable=too-few-public-methods """Test the success revocation message.""" @classmethod def _call(cls, path): diff --git a/certbot/tests/display/util_test.py b/certbot/tests/display/util_test.py index 726eb0b0f..615f33406 100644 --- a/certbot/tests/display/util_test.py +++ b/certbot/tests/display/util_test.py @@ -4,14 +4,13 @@ import socket import tempfile import unittest -import six import mock +import six from certbot import errors from certbot import interfaces from certbot.display import util as display_util - CHOICES = [("First", "Description1"), ("Second", "Description2")] TAGS = ["tag1", "tag2", "tag3"] TAGS_CHOICES = [("1", "tag1"), ("2", "tag2"), ("3", "tag3")] @@ -32,7 +31,7 @@ class InputWithTimeoutTest(unittest.TestCase): def test_input(self, prompt=None): expected = "foo bar" stdin = six.StringIO(expected + "\n") - with mock.patch("certbot.compat.select.select") as mock_select: + with mock.patch("certbot.compat.misc.select.select") as mock_select: mock_select.return_value = ([stdin], [], [],) self.assertEqual(self._call(prompt), expected) @@ -59,7 +58,6 @@ class FileOutputDisplayTest(unittest.TestCase): functions look to a user, uncomment the test_visual function. """ - # pylint:disable=too-many-public-methods def setUp(self): super(FileOutputDisplayTest, self).setUp() self.mock_stdout = mock.MagicMock() @@ -225,7 +223,6 @@ class FileOutputDisplayTest(unittest.TestCase): @mock.patch("certbot.display.util.input_with_timeout") def test_directory_select(self, mock_input): - # pylint: disable=star-args args = ["msg", "/var/www/html", "--flag", True] user_input = "/var/www/html" mock_input.return_value = user_input @@ -314,12 +311,12 @@ class FileOutputDisplayTest(unittest.TestCase): def test_methods_take_force_interactive(self): # Every IDisplay method implemented by FileDisplay must take # force_interactive to prevent workflow regressions. - for name in interfaces.IDisplay.names(): # pylint: disable=no-member + for name in interfaces.IDisplay.names(): if six.PY2: - getargspec = inspect.getargspec # pylint: disable=no-member + getargspec = inspect.getargspec else: - getargspec = inspect.getfullargspec # pylint: disable=no-member - arg_spec = getargspec(getattr(self.displayer, name)) + getargspec = inspect.getfullargspec + arg_spec = getargspec(getattr(self.displayer, name)) # pylint: disable=deprecated-method self.assertTrue("force_interactive" in arg_spec.args) @@ -373,14 +370,16 @@ class NoninteractiveDisplayTest(unittest.TestCase): # should take **kwargs because every method of FileDisplay must # take force_interactive which doesn't apply to # NoninteractiveDisplay. - for name in interfaces.IDisplay.names(): # pylint: disable=no-member + + # Use pylint code for disable to keep on single line under line length limit + for name in interfaces.IDisplay.names(): # pylint: disable=E1120 method = getattr(self.displayer, name) # asserts method accepts arbitrary keyword arguments if six.PY2: - result = inspect.getargspec(method).keywords # pylint: disable=no-member + result = inspect.getargspec(method).keywords # pylint:deprecated-method self.assertFalse(result is None) else: - result = inspect.getfullargspec(method).varkw # pylint: disable=no-member + result = inspect.getfullargspec(method).varkw self.assertFalse(result is None) diff --git a/certbot/tests/eff_test.py b/certbot/tests/eff_test.py index 8d0d5778c..cdd7908a3 100644 --- a/certbot/tests/eff_test.py +++ b/certbot/tests/eff_test.py @@ -1,15 +1,15 @@ -"""Tests for certbot.eff.""" -import requests +"""Tests for certbot._internal.eff.""" import unittest import mock +import requests -from certbot import constants +from certbot._internal import constants import certbot.tests.util as test_util class HandleSubscriptionTest(test_util.ConfigTestCase): - """Tests for certbot.eff.handle_subscription.""" + """Tests for certbot._internal.eff.handle_subscription.""" def setUp(self): super(HandleSubscriptionTest, self).setUp() self.email = 'certbot@example.org' @@ -17,11 +17,11 @@ class HandleSubscriptionTest(test_util.ConfigTestCase): self.config.eff_email = None def _call(self): - from certbot.eff import handle_subscription + from certbot._internal.eff import handle_subscription return handle_subscription(self.config) @test_util.patch_get_utility() - @mock.patch('certbot.eff.subscribe') + @mock.patch('certbot._internal.eff.subscribe') def test_failure(self, mock_subscribe, mock_get_utility): self.config.email = None self.config.eff_email = True @@ -32,7 +32,7 @@ class HandleSubscriptionTest(test_util.ConfigTestCase): expected_part = "because you didn't provide an e-mail address" self.assertTrue(expected_part in actual) - @mock.patch('certbot.eff.subscribe') + @mock.patch('certbot._internal.eff.subscribe') def test_no_subscribe_with_no_prompt(self, mock_subscribe): self.config.eff_email = False with test_util.patch_get_utility() as mock_get_utility: @@ -41,7 +41,7 @@ class HandleSubscriptionTest(test_util.ConfigTestCase): self._assert_no_get_utility_calls(mock_get_utility) @test_util.patch_get_utility() - @mock.patch('certbot.eff.subscribe') + @mock.patch('certbot._internal.eff.subscribe') def test_subscribe_with_no_prompt(self, mock_subscribe, mock_get_utility): self.config.eff_email = True self._call() @@ -53,7 +53,7 @@ class HandleSubscriptionTest(test_util.ConfigTestCase): self.assertFalse(mock_get_utility().add_message.called) @test_util.patch_get_utility() - @mock.patch('certbot.eff.subscribe') + @mock.patch('certbot._internal.eff.subscribe') def test_subscribe_with_prompt(self, mock_subscribe, mock_get_utility): mock_get_utility().yesno.return_value = True self._call() @@ -66,7 +66,7 @@ class HandleSubscriptionTest(test_util.ConfigTestCase): self.assertEqual(mock_subscribe.call_args[0][0], self.email) @test_util.patch_get_utility() - @mock.patch('certbot.eff.subscribe') + @mock.patch('certbot._internal.eff.subscribe') def test_no_subscribe_with_prompt(self, mock_subscribe, mock_get_utility): mock_get_utility().yesno.return_value = False self._call() @@ -84,18 +84,18 @@ class HandleSubscriptionTest(test_util.ConfigTestCase): class SubscribeTest(unittest.TestCase): - """Tests for certbot.eff.subscribe.""" + """Tests for certbot._internal.eff.subscribe.""" def setUp(self): self.email = 'certbot@example.org' self.json = {'status': True} self.response = mock.Mock(ok=True) self.response.json.return_value = self.json - @mock.patch('certbot.eff.requests.post') + @mock.patch('certbot._internal.eff.requests.post') def _call(self, mock_post): mock_post.return_value = self.response - from certbot.eff import subscribe + from certbot._internal.eff import subscribe subscribe(self.email) self._check_post_call(mock_post) @@ -111,7 +111,7 @@ class SubscribeTest(unittest.TestCase): @test_util.patch_get_utility() def test_bad_status(self, mock_get_utility): self.json['status'] = False - self._call() # pylint: disable=no-value-for-parameter + self._call() actual = self._get_reported_message(mock_get_utility) expected_part = 'because your e-mail address appears to be invalid.' self.assertTrue(expected_part in actual) @@ -120,7 +120,7 @@ class SubscribeTest(unittest.TestCase): def test_not_ok(self, mock_get_utility): self.response.ok = False self.response.raise_for_status.side_effect = requests.exceptions.HTTPError - self._call() # pylint: disable=no-value-for-parameter + self._call() actual = self._get_reported_message(mock_get_utility) unexpected_part = 'because' self.assertFalse(unexpected_part in actual) @@ -128,7 +128,7 @@ class SubscribeTest(unittest.TestCase): @test_util.patch_get_utility() def test_response_not_json(self, mock_get_utility): self.response.json.side_effect = ValueError() - self._call() # pylint: disable=no-value-for-parameter + self._call() actual = self._get_reported_message(mock_get_utility) expected_part = 'problem' self.assertTrue(expected_part in actual) @@ -136,7 +136,7 @@ class SubscribeTest(unittest.TestCase): @test_util.patch_get_utility() def test_response_json_missing_status_element(self, mock_get_utility): self.json.clear() - self._call() # pylint: disable=no-value-for-parameter + self._call() actual = self._get_reported_message(mock_get_utility) expected_part = 'problem' self.assertTrue(expected_part in actual) @@ -147,7 +147,7 @@ class SubscribeTest(unittest.TestCase): @test_util.patch_get_utility() def test_subscribe(self, mock_get_utility): - self._call() # pylint: disable=no-value-for-parameter + self._call() self.assertFalse(mock_get_utility.called) diff --git a/certbot/tests/error_handler_test.py b/certbot/tests/error_handler_test.py index 8508a3df5..45fec7f39 100644 --- a/certbot/tests/error_handler_test.py +++ b/certbot/tests/error_handler_test.py @@ -1,16 +1,16 @@ -"""Tests for certbot.error_handler.""" +"""Tests for certbot._internal.error_handler.""" import contextlib -import os import signal import sys import unittest import mock -# pylint: disable=unused-import, no-name-in-module -from acme.magic_typing import Callable, Dict, Union -# pylint: enable=unused-import, no-name-in-module -import certbot.tests.util as test_util +from acme.magic_typing import Callable # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Union # pylint: disable=unused-import, no-name-in-module +from certbot.compat import os + def get_signals(signums): """Get the handlers for an iterable of signums.""" @@ -39,10 +39,10 @@ def send_signal(signum): class ErrorHandlerTest(unittest.TestCase): - """Tests for certbot.error_handler.ErrorHandler.""" + """Tests for certbot._internal.error_handler.ErrorHandler.""" def setUp(self): - from certbot import error_handler + from certbot._internal import error_handler self.init_func = mock.MagicMock() self.init_args = set((42,)) @@ -66,9 +66,9 @@ class ErrorHandlerTest(unittest.TestCase): self.init_func.assert_called_once_with(*self.init_args, **self.init_kwargs) - # On Windows, this test kills pytest itself ! - @test_util.broken_on_windows def test_context_manager_with_signal(self): + if not self.signals: + self.skipTest(reason='Signals cannot be handled on Windows.') init_signals = get_signals(self.signals) with signal_receiver(self.signals) as signals_received: with self.handler: @@ -98,9 +98,9 @@ class ErrorHandlerTest(unittest.TestCase): **self.init_kwargs) bad_func.assert_called_once_with() - # On Windows, this test kills pytest itself ! - @test_util.broken_on_windows def test_bad_recovery_with_signal(self): + if not self.signals: + self.skipTest(reason='Signals cannot be handled on Windows.') sig1 = self.signals[0] sig2 = self.signals[-1] bad_func = mock.MagicMock(side_effect=lambda: send_signal(sig1)) @@ -131,10 +131,10 @@ class ErrorHandlerTest(unittest.TestCase): class ExitHandlerTest(ErrorHandlerTest): - """Tests for certbot.error_handler.ExitHandler.""" + """Tests for certbot._internal.error_handler.ExitHandler.""" def setUp(self): - from certbot import error_handler + from certbot._internal import error_handler super(ExitHandlerTest, self).setUp() self.handler = error_handler.ExitHandler(self.init_func, *self.init_args, @@ -149,10 +149,6 @@ class ExitHandlerTest(ErrorHandlerTest): **self.init_kwargs) func.assert_called_once_with() - # On Windows, this test kills pytest itself ! - @test_util.broken_on_windows - def test_bad_recovery_with_signal(self): - super(ExitHandlerTest, self).test_bad_recovery_with_signal() if __name__ == "__main__": unittest.main() # pragma: no cover diff --git a/certbot/tests/errors_test.py b/certbot/tests/errors_test.py index c8a6c4ac5..d6c829322 100644 --- a/certbot/tests/errors_test.py +++ b/certbot/tests/errors_test.py @@ -4,7 +4,6 @@ import unittest import mock from acme import messages - from certbot import achallenges from certbot.tests import acme_util @@ -14,25 +13,27 @@ class FailedChallengesTest(unittest.TestCase): def setUp(self): from certbot.errors import FailedChallenges - self.error = FailedChallenges(set([achallenges.DNS( + self.error = FailedChallenges({achallenges.DNS( domain="example.com", challb=messages.ChallengeBody( chall=acme_util.DNS01, uri=None, - error=messages.Error(typ="tls", detail="detail")))])) + error=messages.Error.with_code("tls", detail="detail")))}) def test_str(self): self.assertTrue(str(self.error).startswith( - "Failed authorization procedure. example.com (dns-01): tls")) + "Failed authorization procedure. example.com (dns-01): " + "urn:ietf:params:acme:error:tls")) def test_unicode(self): from certbot.errors import FailedChallenges arabic_detail = u'\u0639\u062f\u0627\u0644\u0629' - arabic_error = FailedChallenges(set([achallenges.DNS( + arabic_error = FailedChallenges({achallenges.DNS( domain="example.com", challb=messages.ChallengeBody( chall=acme_util.DNS01, uri=None, - error=messages.Error(typ="tls", detail=arabic_detail)))])) + error=messages.Error.with_code("tls", detail=arabic_detail)))}) self.assertTrue(str(arabic_error).startswith( - "Failed authorization procedure. example.com (dns-01): tls")) + "Failed authorization procedure. example.com (dns-01): " + "urn:ietf:params:acme:error:tls")) class StandaloneBindErrorTest(unittest.TestCase): diff --git a/certbot/tests/hook_test.py b/certbot/tests/hook_test.py index f5bb0c8b5..a3bba57d2 100644 --- a/certbot/tests/hook_test.py +++ b/certbot/tests/hook_test.py @@ -1,24 +1,25 @@ -"""Tests for certbot.hooks.""" -import os -import stat +"""Tests for certbot._internal.hooks.""" import unittest import mock from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module from certbot import errors -from certbot.tests import util +from certbot import util +from certbot.compat import filesystem +from certbot.compat import os +from certbot.tests import util as test_util class ValidateHooksTest(unittest.TestCase): - """Tests for certbot.hooks.validate_hooks.""" + """Tests for certbot._internal.hooks.validate_hooks.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.hooks import validate_hooks + from certbot._internal.hooks import validate_hooks return validate_hooks(*args, **kwargs) - @mock.patch("certbot.hooks.validate_hook") + @mock.patch("certbot._internal.hooks.validate_hook") def test_it(self, mock_validate_hook): config = mock.MagicMock() self._call(config) @@ -29,73 +30,71 @@ class ValidateHooksTest(unittest.TestCase): self.assertEqual("renew", types[-1]) -class ValidateHookTest(util.TempDirTestCase): - """Tests for certbot.hooks.validate_hook.""" +class ValidateHookTest(test_util.TempDirTestCase): + """Tests for certbot._internal.hooks.validate_hook.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.hooks import validate_hook + from certbot._internal.hooks import validate_hook return validate_hook(*args, **kwargs) - @util.broken_on_windows - def test_not_executable(self): - file_path = os.path.join(self.tempdir, "foo") - # create a non-executable file - os.close(os.open(file_path, os.O_CREAT | os.O_WRONLY, 0o666)) + def test_hook_not_executable(self): # prevent unnecessary modifications to PATH - with mock.patch("certbot.hooks.plug_util.path_surgery"): - self.assertRaises(errors.HookCommandNotFound, - self._call, file_path, "foo") + with mock.patch("certbot._internal.hooks.plug_util.path_surgery"): + # We just mock out filesystem.is_executable since on Windows, it is difficult + # to get a fully working test around executable permissions. See + # certbot.tests.compat.filesystem::NotExecutableTest for more in-depth tests. + with mock.patch("certbot._internal.hooks.filesystem.is_executable", return_value=False): + self.assertRaises(errors.HookCommandNotFound, self._call, 'dummy', "foo") - @mock.patch("certbot.hooks.util.exe_exists") + @mock.patch("certbot._internal.hooks.util.exe_exists") def test_not_found(self, mock_exe_exists): mock_exe_exists.return_value = False - with mock.patch("certbot.hooks.plug_util.path_surgery") as mock_ps: - self.assertRaises(errors.HookCommandNotFound, - self._call, "foo", "bar") + with mock.patch("certbot._internal.hooks.plug_util.path_surgery") as mock_ps: + self.assertRaises(errors.HookCommandNotFound, self._call, "foo", "bar") self.assertTrue(mock_ps.called) - @mock.patch("certbot.hooks._prog") + @mock.patch("certbot._internal.hooks._prog") def test_unset(self, mock_prog): self._call(None, "foo") self.assertFalse(mock_prog.called) -class HookTest(util.ConfigTestCase): +class HookTest(test_util.ConfigTestCase): """Common base class for hook tests.""" @classmethod - def _call(cls, *args, **kwargs): + def _call(cls, *args, **kwargs): # pragma: no cover """Calls the method being tested with the given arguments.""" raise NotImplementedError @classmethod def _call_with_mock_execute(cls, *args, **kwargs): - """Calls self._call after mocking out certbot.hooks.execute. + """Calls self._call after mocking out certbot._internal.hooks.execute. The mock execute object is returned rather than the return value of self._call. """ - with mock.patch("certbot.hooks.execute") as mock_execute: + with mock.patch("certbot._internal.hooks.execute") as mock_execute: mock_execute.return_value = ("", "") cls._call(*args, **kwargs) return mock_execute class PreHookTest(HookTest): - """Tests for certbot.hooks.pre_hook.""" + """Tests for certbot._internal.hooks.pre_hook.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.hooks import pre_hook + from certbot._internal.hooks import pre_hook return pre_hook(*args, **kwargs) def setUp(self): super(PreHookTest, self).setUp() self.config.pre_hook = "foo" - os.makedirs(self.config.renewal_pre_hooks_dir) + filesystem.makedirs(self.config.renewal_pre_hooks_dir) self.dir_hook = os.path.join(self.config.renewal_pre_hooks_dir, "bar") create_hook(self.dir_hook) @@ -108,7 +107,7 @@ class PreHookTest(HookTest): super(PreHookTest, self).tearDown() def _reset_pre_hook_already(self): - from certbot.hooks import executed_pre_hooks + from certbot._internal.hooks import executed_pre_hooks executed_pre_hooks.clear() def test_certonly(self): @@ -121,7 +120,7 @@ class PreHookTest(HookTest): def _test_nonrenew_common(self): mock_execute = self._call_with_mock_execute(self.config) - mock_execute.assert_called_once_with(self.config.pre_hook) + mock_execute.assert_called_once_with("pre-hook", self.config.pre_hook) self._test_no_executions_common() def test_no_hooks(self): @@ -129,7 +128,7 @@ class PreHookTest(HookTest): self.config.verb = "renew" os.remove(self.dir_hook) - with mock.patch("certbot.hooks.logger") as mock_logger: + with mock.patch("certbot._internal.hooks.logger") as mock_logger: mock_execute = self._call_with_mock_execute(self.config) self.assertFalse(mock_execute.called) self.assertFalse(mock_logger.info.called) @@ -137,43 +136,43 @@ class PreHookTest(HookTest): def test_renew_disabled_dir_hooks(self): self.config.directory_hooks = False mock_execute = self._call_with_mock_execute(self.config) - mock_execute.assert_called_once_with(self.config.pre_hook) + mock_execute.assert_called_once_with("pre-hook", self.config.pre_hook) self._test_no_executions_common() def test_renew_no_overlap(self): self.config.verb = "renew" mock_execute = self._call_with_mock_execute(self.config) - mock_execute.assert_any_call(self.dir_hook) - mock_execute.assert_called_with(self.config.pre_hook) + mock_execute.assert_any_call("pre-hook", self.dir_hook) + mock_execute.assert_called_with("pre-hook", self.config.pre_hook) self._test_no_executions_common() def test_renew_with_overlap(self): self.config.pre_hook = self.dir_hook self.config.verb = "renew" mock_execute = self._call_with_mock_execute(self.config) - mock_execute.assert_called_once_with(self.dir_hook) + mock_execute.assert_called_once_with("pre-hook", self.dir_hook) self._test_no_executions_common() def _test_no_executions_common(self): - with mock.patch("certbot.hooks.logger") as mock_logger: + with mock.patch("certbot._internal.hooks.logger") as mock_logger: mock_execute = self._call_with_mock_execute(self.config) self.assertFalse(mock_execute.called) self.assertTrue(mock_logger.info.called) class PostHookTest(HookTest): - """Tests for certbot.hooks.post_hook.""" + """Tests for certbot._internal.hooks.post_hook.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.hooks import post_hook + from certbot._internal.hooks import post_hook return post_hook(*args, **kwargs) def setUp(self): super(PostHookTest, self).setUp() self.config.post_hook = "bar" - os.makedirs(self.config.renewal_post_hooks_dir) + filesystem.makedirs(self.config.renewal_post_hooks_dir) self.dir_hook = os.path.join(self.config.renewal_post_hooks_dir, "foo") create_hook(self.dir_hook) @@ -186,14 +185,14 @@ class PostHookTest(HookTest): super(PostHookTest, self).tearDown() def _reset_post_hook_eventually(self): - from certbot.hooks import post_hooks + from certbot._internal.hooks import post_hooks del post_hooks[:] def test_certonly_and_run_with_hook(self): for verb in ("certonly", "run",): self.config.verb = verb mock_execute = self._call_with_mock_execute(self.config) - mock_execute.assert_called_once_with(self.config.post_hook) + mock_execute.assert_called_once_with("post-hook", self.config.post_hook) self.assertFalse(self._get_eventually()) def test_cert_only_and_run_without_hook(self): @@ -240,27 +239,27 @@ class PostHookTest(HookTest): self.assertEqual(self._get_eventually(), expected) def _get_eventually(self): - from certbot.hooks import post_hooks + from certbot._internal.hooks import post_hooks return post_hooks class RunSavedPostHooksTest(HookTest): - """Tests for certbot.hooks.run_saved_post_hooks.""" + """Tests for certbot._internal.hooks.run_saved_post_hooks.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.hooks import run_saved_post_hooks + from certbot._internal.hooks import run_saved_post_hooks return run_saved_post_hooks() def _call_with_mock_execute_and_eventually(self, *args, **kwargs): """Call run_saved_post_hooks but mock out execute and eventually - certbot.hooks.post_hooks is replaced with + certbot._internal.hooks.post_hooks is replaced with self.eventually. The mock execute object is returned rather than the return value of run_saved_post_hooks. """ - eventually_path = "certbot.hooks.post_hooks" + eventually_path = "certbot._internal.hooks.post_hooks" with mock.patch(eventually_path, new=self.eventually): return self._call_with_mock_execute(*args, **kwargs) @@ -277,12 +276,12 @@ class RunSavedPostHooksTest(HookTest): calls = mock_execute.call_args_list for actual_call, expected_arg in zip(calls, self.eventually): - self.assertEqual(actual_call[0][0], expected_arg) + self.assertEqual(actual_call[0][1], expected_arg) def test_single(self): self.eventually = ["foo"] mock_execute = self._call_with_mock_execute_and_eventually() - mock_execute.assert_called_once_with(self.eventually[0]) + mock_execute.assert_called_once_with("post-hook", self.eventually[0]) class RenewalHookTest(HookTest): @@ -291,7 +290,7 @@ class RenewalHookTest(HookTest): # pylint: disable=abstract-method def _call_with_mock_execute(self, *args, **kwargs): - """Calls self._call after mocking out certbot.hooks.execute. + """Calls self._call after mocking out certbot._internal.hooks.execute. The mock execute object is returned rather than the return value of self._call. The mock execute object asserts that environment @@ -312,7 +311,7 @@ class RenewalHookTest(HookTest): self.assertEqual(os.environ["RENEWED_LINEAGE"], lineage) return ("", "") - with mock.patch("certbot.hooks.execute") as mock_execute: + with mock.patch("certbot._internal.hooks.execute") as mock_execute: mock_execute.side_effect = execute_side_effect self._call(*args, **kwargs) return mock_execute @@ -330,14 +329,14 @@ class RenewalHookTest(HookTest): class DeployHookTest(RenewalHookTest): - """Tests for certbot.hooks.deploy_hook.""" + """Tests for certbot._internal.hooks.deploy_hook.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.hooks import deploy_hook + from certbot._internal.hooks import deploy_hook return deploy_hook(*args, **kwargs) - @mock.patch("certbot.hooks.logger") + @mock.patch("certbot._internal.hooks.logger") def test_dry_run(self, mock_logger): self.config.deploy_hook = "foo" self.config.dry_run = True @@ -346,7 +345,7 @@ class DeployHookTest(RenewalHookTest): self.assertFalse(mock_execute.called) self.assertTrue(mock_logger.warning.called) - @mock.patch("certbot.hooks.logger") + @mock.patch("certbot._internal.hooks.logger") def test_no_hook(self, mock_logger): self.config.deploy_hook = None mock_execute = self._call_with_mock_execute( @@ -360,22 +359,22 @@ class DeployHookTest(RenewalHookTest): self.config.deploy_hook = "foo" mock_execute = self._call_with_mock_execute( self.config, domains, lineage) - mock_execute.assert_called_once_with(self.config.deploy_hook) + mock_execute.assert_called_once_with("deploy-hook", self.config.deploy_hook) class RenewHookTest(RenewalHookTest): - """Tests for certbot.hooks.renew_hook""" + """Tests for certbot._internal.hooks.renew_hook""" @classmethod def _call(cls, *args, **kwargs): - from certbot.hooks import renew_hook + from certbot._internal.hooks import renew_hook return renew_hook(*args, **kwargs) def setUp(self): super(RenewHookTest, self).setUp() self.config.renew_hook = "foo" - os.makedirs(self.config.renewal_deploy_hooks_dir) + filesystem.makedirs(self.config.renewal_deploy_hooks_dir) self.dir_hook = os.path.join(self.config.renewal_deploy_hooks_dir, "bar") create_hook(self.dir_hook) @@ -384,9 +383,9 @@ class RenewHookTest(RenewalHookTest): self.config.directory_hooks = False mock_execute = self._call_with_mock_execute( self.config, ["example.org"], "/foo/bar") - mock_execute.assert_called_once_with(self.config.renew_hook) + mock_execute.assert_called_once_with("deploy-hook", self.config.renew_hook) - @mock.patch("certbot.hooks.logger") + @mock.patch("certbot._internal.hooks.logger") def test_dry_run(self, mock_logger): self.config.dry_run = True mock_execute = self._call_with_mock_execute( @@ -398,7 +397,7 @@ class RenewHookTest(RenewalHookTest): self.config.renew_hook = None os.remove(self.dir_hook) - with mock.patch("certbot.hooks.logger") as mock_logger: + with mock.patch("certbot._internal.hooks.logger") as mock_logger: mock_execute = self._call_with_mock_execute( self.config, ["example.org"], "/foo/bar") self.assertFalse(mock_execute.called) @@ -408,21 +407,21 @@ class RenewHookTest(RenewalHookTest): self.config.renew_hook = self.dir_hook mock_execute = self._call_with_mock_execute( self.config, ["example.net", "example.org"], "/foo/bar") - mock_execute.assert_called_once_with(self.dir_hook) + mock_execute.assert_called_once_with("deploy-hook", self.dir_hook) def test_no_overlap(self): mock_execute = self._call_with_mock_execute( self.config, ["example.org"], "/foo/bar") - mock_execute.assert_any_call(self.dir_hook) - mock_execute.assert_called_with(self.config.renew_hook) + mock_execute.assert_any_call("deploy-hook", self.dir_hook) + mock_execute.assert_called_with("deploy-hook", self.config.renew_hook) class ExecuteTest(unittest.TestCase): - """Tests for certbot.hooks.execute.""" + """Tests for certbot._internal.hooks.execute.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.hooks import execute + from certbot._internal.hooks import execute return execute(*args, **kwargs) def test_it(self): @@ -433,28 +432,32 @@ class ExecuteTest(unittest.TestCase): def _test_common(self, returncode, stdout, stderr): given_command = "foo" - with mock.patch("certbot.hooks.Popen") as mock_popen: + given_name = "foo-hook" + with mock.patch("certbot._internal.hooks.Popen") as mock_popen: mock_popen.return_value.communicate.return_value = (stdout, stderr) mock_popen.return_value.returncode = returncode - with mock.patch("certbot.hooks.logger") as mock_logger: - self.assertEqual(self._call(given_command), (stderr, stdout)) + with mock.patch("certbot._internal.hooks.logger") as mock_logger: + self.assertEqual(self._call(given_name, given_command), (stderr, stdout)) executed_command = mock_popen.call_args[1].get( "args", mock_popen.call_args[0][0]) self.assertEqual(executed_command, given_command) + mock_logger.info.assert_any_call("Running %s command: %s", + given_name, given_command) if stdout: - self.assertTrue(mock_logger.info.called) + mock_logger.info.assert_any_call(mock.ANY, mock.ANY, + mock.ANY, stdout) if stderr or returncode: self.assertTrue(mock_logger.error.called) -class ListHooksTest(util.TempDirTestCase): - """Tests for certbot.hooks.list_hooks.""" +class ListHooksTest(test_util.TempDirTestCase): + """Tests for certbot._internal.hooks.list_hooks.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.hooks import list_hooks + from certbot._internal.hooks import list_hooks return list_hooks(*args, **kwargs) def test_empty(self): @@ -476,6 +479,12 @@ class ListHooksTest(util.TempDirTestCase): self.assertEqual(self._call(self.tempdir), [name]) + def test_ignore_tilde(self): + name = os.path.join(self.tempdir, "foo~") + create_hook(name) + + self.assertEqual(self._call(self.tempdir), []) + def create_hook(file_path): """Creates an executable file at the specified path. @@ -483,8 +492,7 @@ def create_hook(file_path): :param str file_path: path to create the file at """ - open(file_path, "w").close() - os.chmod(file_path, os.stat(file_path).st_mode | stat.S_IXUSR) + util.safe_open(file_path, mode="w", chmod=0o744).close() if __name__ == '__main__': diff --git a/certbot/tests/lock_test.py b/certbot/tests/lock_test.py index 2ade47827..5a48009fd 100644 --- a/certbot/tests/lock_test.py +++ b/certbot/tests/lock_test.py @@ -1,20 +1,29 @@ -"""Tests for certbot.lock.""" +"""Tests for certbot._internal.lock.""" import functools import multiprocessing -import os import unittest import mock from certbot import errors +from certbot.compat import os from certbot.tests import util as test_util +try: + import fcntl # pylint: disable=import-error,unused-import +except ImportError: + POSIX_MODE = False +else: + POSIX_MODE = True + + + class LockDirTest(test_util.TempDirTestCase): - """Tests for certbot.lock.lock_dir.""" + """Tests for certbot._internal.lock.lock_dir.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.lock import lock_dir + from certbot._internal.lock import lock_dir return lock_dir(*args, **kwargs) def test_it(self): @@ -25,17 +34,16 @@ class LockDirTest(test_util.TempDirTestCase): class LockFileTest(test_util.TempDirTestCase): - """Tests for certbot.lock.LockFile.""" + """Tests for certbot._internal.lock.LockFile.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.lock import LockFile + from certbot._internal.lock import LockFile return LockFile(*args, **kwargs) def setUp(self): super(LockFileTest, self).setUp() self.lock_path = os.path.join(self.tempdir, 'test.lock') - @test_util.broken_on_windows def test_acquire_without_deletion(self): # acquire the lock in another process but don't delete the file child = multiprocessing.Process(target=self._call, @@ -53,12 +61,14 @@ class LockFileTest(test_util.TempDirTestCase): self.assertRaises, errors.LockError, self._call, self.lock_path) test_util.lock_and_call(assert_raises, self.lock_path) - @test_util.broken_on_windows def test_locked_repr(self): lock_file = self._call(self.lock_path) - locked_repr = repr(lock_file) - self._test_repr_common(lock_file, locked_repr) - self.assertTrue('acquired' in locked_repr) + try: + locked_repr = repr(lock_file) + self._test_repr_common(lock_file, locked_repr) + self.assertTrue('acquired' in locked_repr) + finally: + lock_file.release() def test_released_repr(self): lock_file = self._call(self.lock_path) @@ -71,10 +81,14 @@ class LockFileTest(test_util.TempDirTestCase): self.assertTrue(lock_file.__class__.__name__ in lock_repr) self.assertTrue(self.lock_path in lock_repr) - @test_util.broken_on_windows + @test_util.skip_on_windows( + 'Race conditions on lock are specific to the non-blocking file access approach on Linux.') def test_race(self): should_delete = [True, False] - stat = os.stat + # Normally os module should not be imported in certbot codebase except in certbot.compat + # for the sake of compatibility over Windows and Linux. + # We make an exception here, since test_race is a test function called only on Linux. + from os import stat # pylint: disable=os-module-forbidden def delete_and_stat(path): """Wrap os.stat and maybe delete the file first.""" @@ -82,39 +96,46 @@ class LockFileTest(test_util.TempDirTestCase): os.remove(path) return stat(path) - with mock.patch('certbot.lock.os.stat') as mock_stat: + with mock.patch('certbot._internal.lock.filesystem.os.stat') as mock_stat: mock_stat.side_effect = delete_and_stat self._call(self.lock_path) self.assertFalse(should_delete) - @test_util.broken_on_windows def test_removed(self): lock_file = self._call(self.lock_path) lock_file.release() self.assertFalse(os.path.exists(self.lock_path)) - @test_util.broken_on_windows - @mock.patch('certbot.compat.fcntl.lockf') - def test_unexpected_lockf_err(self, mock_lockf): + def test_unexpected_lockf_or_locking_err(self): + if POSIX_MODE: + mocked_function = 'certbot._internal.lock.fcntl.lockf' + else: + mocked_function = 'certbot._internal.lock.msvcrt.locking' msg = 'hi there' - mock_lockf.side_effect = IOError(msg) - try: - self._call(self.lock_path) - except IOError as err: - self.assertTrue(msg in str(err)) - else: # pragma: no cover - self.fail('IOError not raised') + with mock.patch(mocked_function) as mock_lock: + mock_lock.side_effect = IOError(msg) + try: + self._call(self.lock_path) + except IOError as err: + self.assertTrue(msg in str(err)) + else: # pragma: no cover + self.fail('IOError not raised') - @mock.patch('certbot.lock.os.stat') - def test_unexpected_stat_err(self, mock_stat): + def test_unexpected_os_err(self): + if POSIX_MODE: + mock_function = 'certbot._internal.lock.filesystem.os.stat' + else: + mock_function = 'certbot._internal.lock.msvcrt.locking' + # The only expected errno are ENOENT and EACCES in lock module. msg = 'hi there' - mock_stat.side_effect = OSError(msg) - try: - self._call(self.lock_path) - except OSError as err: - self.assertTrue(msg in str(err)) - else: # pragma: no cover - self.fail('OSError not raised') + with mock.patch(mock_function) as mock_os: + mock_os.side_effect = OSError(msg) + try: + self._call(self.lock_path) + except OSError as err: + self.assertTrue(msg in str(err)) + else: # pragma: no cover + self.fail('OSError not raised') if __name__ == "__main__": diff --git a/certbot/tests/log_test.py b/certbot/tests/log_test.py index b82cc6ca1..3b9adbbf2 100644 --- a/certbot/tests/log_test.py +++ b/certbot/tests/log_test.py @@ -1,7 +1,6 @@ -"""Tests for certbot.log.""" +"""Tests for certbot._internal.log.""" import logging import logging.handlers -import os import sys import time import unittest @@ -11,26 +10,26 @@ import six from acme import messages from acme.magic_typing import Optional # pylint: disable=unused-import, no-name-in-module - -from certbot import compat -from certbot import constants from certbot import errors from certbot import util +from certbot._internal import constants +from certbot.compat import filesystem +from certbot.compat import os from certbot.tests import util as test_util class PreArgParseSetupTest(unittest.TestCase): - """Tests for certbot.log.pre_arg_parse_setup.""" + """Tests for certbot._internal.log.pre_arg_parse_setup.""" @classmethod def _call(cls, *args, **kwargs): # pylint: disable=unused-argument - from certbot.log import pre_arg_parse_setup + from certbot._internal.log import pre_arg_parse_setup return pre_arg_parse_setup() - @mock.patch('certbot.log.sys') - @mock.patch('certbot.log.pre_arg_parse_except_hook') - @mock.patch('certbot.log.logging.getLogger') - @mock.patch('certbot.log.util.atexit_register') + @mock.patch('certbot._internal.log.sys') + @mock.patch('certbot._internal.log.pre_arg_parse_except_hook') + @mock.patch('certbot._internal.log.logging.getLogger') + @mock.patch('certbot._internal.log.util.atexit_register') def test_it(self, mock_register, mock_get, mock_except_hook, mock_sys): mock_sys.argv = ['--debug'] mock_sys.version_info = sys.version_info @@ -58,11 +57,11 @@ class PreArgParseSetupTest(unittest.TestCase): class PostArgParseSetupTest(test_util.ConfigTestCase): - """Tests for certbot.log.post_arg_parse_setup.""" + """Tests for certbot._internal.log.post_arg_parse_setup.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.log import post_arg_parse_setup + from certbot._internal.log import post_arg_parse_setup return post_arg_parse_setup(*args, **kwargs) def setUp(self): @@ -73,9 +72,9 @@ class PostArgParseSetupTest(test_util.ConfigTestCase): self.config.verbose_count = constants.CLI_DEFAULTS['verbose_count'] self.devnull = open(os.devnull, 'w') - from certbot.log import ColoredStreamHandler + from certbot._internal.log import ColoredStreamHandler self.stream_handler = ColoredStreamHandler(six.StringIO()) - from certbot.log import MemoryHandler, TempHandler + from certbot._internal.log import MemoryHandler, TempHandler self.temp_handler = TempHandler() self.temp_path = self.temp_handler.path self.memory_handler = MemoryHandler(self.temp_handler) @@ -90,11 +89,11 @@ class PostArgParseSetupTest(test_util.ConfigTestCase): super(PostArgParseSetupTest, self).tearDown() def test_common(self): - with mock.patch('certbot.log.logging.getLogger') as mock_get_logger: + with mock.patch('certbot._internal.log.logging.getLogger') as mock_get_logger: mock_get_logger.return_value = self.root_logger - except_hook_path = 'certbot.log.post_arg_parse_except_hook' + except_hook_path = 'certbot._internal.log.post_arg_parse_except_hook' with mock.patch(except_hook_path) as mock_except_hook: - with mock.patch('certbot.log.sys') as mock_sys: + with mock.patch('certbot._internal.log.sys') as mock_sys: mock_sys.version_info = sys.version_info self._call(self.config) @@ -124,18 +123,18 @@ class PostArgParseSetupTest(test_util.ConfigTestCase): class SetupLogFileHandlerTest(test_util.ConfigTestCase): - """Tests for certbot.log.setup_log_file_handler.""" + """Tests for certbot._internal.log.setup_log_file_handler.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.log import setup_log_file_handler + from certbot._internal.log import setup_log_file_handler return setup_log_file_handler(*args, **kwargs) def setUp(self): super(SetupLogFileHandlerTest, self).setUp() self.config.max_log_backups = 42 - @mock.patch('certbot.main.logging.handlers.RotatingFileHandler') + @mock.patch('certbot._internal.main.logging.handlers.RotatingFileHandler') def test_failure(self, mock_handler): mock_handler.side_effect = IOError @@ -167,7 +166,7 @@ class SetupLogFileHandlerTest(test_util.ConfigTestCase): backup_path = os.path.join(self.config.logs_dir, log_file + '.1') self.assertEqual(os.path.exists(backup_path), should_rollover) - @mock.patch('certbot.log.logging.handlers.RotatingFileHandler') + @mock.patch('certbot._internal.log.logging.handlers.RotatingFileHandler') def test_max_log_backups_used(self, mock_handler): self._call(self.config, 'test.log', '%(message)s') backup_count = mock_handler.call_args[1]['backupCount'] @@ -175,7 +174,7 @@ class SetupLogFileHandlerTest(test_util.ConfigTestCase): class ColoredStreamHandlerTest(unittest.TestCase): - """Tests for certbot.log.ColoredStreamHandler""" + """Tests for certbot._internal.log.ColoredStreamHandler""" def setUp(self): self.stream = six.StringIO() @@ -183,7 +182,7 @@ class ColoredStreamHandlerTest(unittest.TestCase): self.logger = logging.getLogger() self.logger.setLevel(logging.DEBUG) - from certbot.log import ColoredStreamHandler + from certbot._internal.log import ColoredStreamHandler self.handler = ColoredStreamHandler(self.stream) self.logger.addHandler(self.handler) @@ -207,7 +206,7 @@ class ColoredStreamHandlerTest(unittest.TestCase): class MemoryHandlerTest(unittest.TestCase): - """Tests for certbot.log.MemoryHandler""" + """Tests for certbot._internal.log.MemoryHandler""" def setUp(self): self.logger = logging.getLogger(__name__) self.logger.setLevel(logging.DEBUG) @@ -215,7 +214,7 @@ class MemoryHandlerTest(unittest.TestCase): self.stream = six.StringIO() self.stream_handler = logging.StreamHandler(self.stream) - from certbot.log import MemoryHandler + from certbot._internal.log import MemoryHandler self.handler = MemoryHandler(self.stream_handler) self.logger.addHandler(self.handler) @@ -250,18 +249,17 @@ class MemoryHandlerTest(unittest.TestCase): class TempHandlerTest(unittest.TestCase): - """Tests for certbot.log.TempHandler.""" + """Tests for certbot._internal.log.TempHandler.""" def setUp(self): self.closed = False - from certbot.log import TempHandler + from certbot._internal.log import TempHandler self.handler = TempHandler() def tearDown(self): self.handler.close() def test_permissions(self): - self.assertTrue( - util.check_permissions(self.handler.path, 0o600, compat.os_geteuid())) + self.assertTrue(filesystem.check_permissions(self.handler.path, 0o600)) def test_delete(self): self.handler.close() @@ -275,15 +273,14 @@ class TempHandlerTest(unittest.TestCase): class PreArgParseExceptHookTest(unittest.TestCase): - """Tests for certbot.log.pre_arg_parse_except_hook.""" + """Tests for certbot._internal.log.pre_arg_parse_except_hook.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.log import pre_arg_parse_except_hook + from certbot._internal.log import pre_arg_parse_except_hook return pre_arg_parse_except_hook(*args, **kwargs) - @mock.patch('certbot.log.post_arg_parse_except_hook') + @mock.patch('certbot._internal.log.post_arg_parse_except_hook') def test_it(self, mock_post_arg_parse_except_hook): - # pylint: disable=star-args memory_handler = mock.MagicMock() args = ('some', 'args',) kwargs = {'some': 'kwargs'} @@ -296,10 +293,10 @@ class PreArgParseExceptHookTest(unittest.TestCase): class PostArgParseExceptHookTest(unittest.TestCase): - """Tests for certbot.log.post_arg_parse_except_hook.""" + """Tests for certbot._internal.log.post_arg_parse_except_hook.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.log import post_arg_parse_except_hook + from certbot._internal.log import post_arg_parse_except_hook return post_arg_parse_except_hook(*args, **kwargs) def setUp(self): @@ -355,11 +352,10 @@ class PostArgParseExceptHookTest(unittest.TestCase): raise error_type(self.error_msg) except BaseException: exc_info = sys.exc_info() - with mock.patch('certbot.log.logger') as mock_logger: + with mock.patch('certbot._internal.log.logger') as mock_logger: mock_logger.error.side_effect = write_err - with mock.patch('certbot.log.sys.stderr', mock_err): + with mock.patch('certbot._internal.log.sys.stderr', mock_err): try: - # pylint: disable=star-args self._call( *exc_info, debug=debug, log_path=self.log_path) except SystemExit as exit_err: @@ -390,10 +386,10 @@ class PostArgParseExceptHookTest(unittest.TestCase): class ExitWithLogPathTest(test_util.TempDirTestCase): - """Tests for certbot.log.exit_with_log_path.""" + """Tests for certbot._internal.log.exit_with_log_path.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.log import exit_with_log_path + from certbot._internal.log import exit_with_log_path return exit_with_log_path(*args, **kwargs) def test_log_file(self): @@ -409,13 +405,13 @@ class ExitWithLogPathTest(test_util.TempDirTestCase): self.assertTrue('logfiles' in err_str) self.assertTrue(self.tempdir in err_str) + # pylint: disable=inconsistent-return-statements def _test_common(self, *args, **kwargs): try: self._call(*args, **kwargs) except SystemExit as err: return str(err) - else: # pragma: no cover - self.fail('SystemExit was not raised.') + self.fail('SystemExit was not raised.') # pragma: no cover if __name__ == "__main__": diff --git a/certbot/tests/main_test.py b/certbot/tests/main_test.py index 786b91a94..7b22c81d6 100644 --- a/certbot/tests/main_test.py +++ b/certbot/tests/main_test.py @@ -1,42 +1,40 @@ # coding=utf-8 -"""Tests for certbot.main.""" +"""Tests for certbot._internal.main.""" # pylint: disable=too-many-lines from __future__ import print_function +import datetime import itertools import json -import mock -import os import shutil +import sys +import tempfile import traceback import unittest -import datetime -import pytz -import tempfile -import sys import josepy as jose +import mock +import pytz import six from six.moves import reload_module # pylint: disable=import-error from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module -from certbot import account -from certbot import cli -from certbot import compat -from certbot import constants -from certbot import configuration from certbot import crypto_util from certbot import errors from certbot import interfaces # pylint: disable=unused-import -from certbot import main -from certbot import updater from certbot import util - -from certbot.plugins import disco +from certbot._internal import account +from certbot._internal import cli +from certbot._internal import configuration +from certbot._internal import constants +from certbot._internal import main +from certbot._internal import updater +from certbot._internal.plugins import disco +from certbot._internal.plugins import manual +from certbot._internal.plugins import null +from certbot.compat import filesystem +from certbot.compat import os from certbot.plugins import enhancements -from certbot.plugins import manual -from certbot.plugins import null - import certbot.tests.util as test_util CERT_PATH = test_util.vector_path('cert_512.pem') @@ -49,7 +47,7 @@ SS_CERT_PATH = test_util.vector_path('cert_2048.pem') class TestHandleIdenticalCerts(unittest.TestCase): - """Test for certbot.main._handle_identical_cert_request""" + """Test for certbot._internal.main._handle_identical_cert_request""" def test_handle_identical_cert_request_pending(self): mock_lineage = mock.Mock() mock_lineage.ensure_deployed.return_value = False @@ -59,19 +57,19 @@ class TestHandleIdenticalCerts(unittest.TestCase): class RunTest(test_util.ConfigTestCase): - """Tests for certbot.main.run.""" + """Tests for certbot._internal.main.run.""" def setUp(self): super(RunTest, self).setUp() self.domain = 'example.org' self.patches = [ - mock.patch('certbot.main._get_and_save_cert'), - mock.patch('certbot.main.display_ops.success_installation'), - mock.patch('certbot.main.display_ops.success_renewal'), - mock.patch('certbot.main._init_le_client'), - mock.patch('certbot.main._suggest_donation_if_appropriate'), - mock.patch('certbot.main._report_new_cert'), - mock.patch('certbot.main._find_cert')] + mock.patch('certbot._internal.main._get_and_save_cert'), + mock.patch('certbot._internal.main.display_ops.success_installation'), + mock.patch('certbot._internal.main.display_ops.success_renewal'), + mock.patch('certbot._internal.main._init_le_client'), + mock.patch('certbot._internal.main._suggest_donation_if_appropriate'), + mock.patch('certbot._internal.main._report_new_cert'), + mock.patch('certbot._internal.main._find_cert')] self.mock_auth = self.patches[0].start() self.mock_success_installation = self.patches[1].start() @@ -91,7 +89,7 @@ class RunTest(test_util.ConfigTestCase): config = configuration.NamespaceConfig( cli.prepare_and_parse_args(plugins, args)) - from certbot.main import run + from certbot._internal.main import run run(config, plugins) def test_newcert_success(self): @@ -112,7 +110,7 @@ class RunTest(test_util.ConfigTestCase): self._call() self.mock_success_renewal.assert_called_once_with([self.domain]) - @mock.patch('certbot.main.plug_sel.choose_configurator_plugins') + @mock.patch('certbot._internal.main.plug_sel.choose_configurator_plugins') def test_run_enhancement_not_supported(self, mock_choose): mock_choose.return_value = (null.Installer(self.config, "null"), None) plugins = disco.PluginsRegistry.find_all() @@ -123,7 +121,7 @@ class RunTest(test_util.ConfigTestCase): class CertonlyTest(unittest.TestCase): - """Tests for certbot.main.certonly.""" + """Tests for certbot._internal.main.certonly.""" def setUp(self): self.get_utility_patch = test_util.patch_get_utility() @@ -137,15 +135,15 @@ class CertonlyTest(unittest.TestCase): config = configuration.NamespaceConfig( cli.prepare_and_parse_args(plugins, args)) - with mock.patch('certbot.main._init_le_client') as mock_init: - with mock.patch('certbot.main._suggest_donation_if_appropriate'): + with mock.patch('certbot._internal.main._init_le_client') as mock_init: + with mock.patch('certbot._internal.main._suggest_donation_if_appropriate'): main.certonly(config, plugins) return mock_init() # returns the client - @mock.patch('certbot.main._find_cert') - @mock.patch('certbot.main._get_and_save_cert') - @mock.patch('certbot.main._report_new_cert') + @mock.patch('certbot._internal.main._find_cert') + @mock.patch('certbot._internal.main._get_and_save_cert') + @mock.patch('certbot._internal.main._report_new_cert') def test_no_reinstall_text_pause(self, unused_report, mock_auth, mock_find_cert): mock_notification = self.mock_get_utility().notification @@ -154,14 +152,13 @@ class CertonlyTest(unittest.TestCase): mock_find_cert.return_value = False, None self._call('certonly --webroot -d example.com'.split()) - def _assert_no_pause(self, message, pause=True): - # pylint: disable=unused-argument + def _assert_no_pause(self, message, pause=True): # pylint: disable=unused-argument self.assertFalse(pause) - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.cert_manager.domains_for_certname') - @mock.patch('certbot.renewal.renew_cert') - @mock.patch('certbot.main._report_new_cert') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.cert_manager.domains_for_certname') + @mock.patch('certbot._internal.renewal.renew_cert') + @mock.patch('certbot._internal.main._report_new_cert') def test_find_lineage_for_domains_and_certname(self, mock_report_cert, mock_renew_cert, mock_domains, mock_lineage): domains = ['example.com', 'test.org'] @@ -187,10 +184,10 @@ class CertonlyTest(unittest.TestCase): self.assertRaises(errors.ConfigurationError, self._call, ('certonly --webroot -d example.com -d test.com --cert-name example.com').split()) - @mock.patch('certbot.cert_manager.domains_for_certname') + @mock.patch('certbot._internal.cert_manager.domains_for_certname') @mock.patch('certbot.display.ops.choose_names') - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.main._report_new_cert') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.main._report_new_cert') def test_find_lineage_for_domains_new_certname(self, mock_report_cert, mock_lineage, mock_choose_names, mock_domains_for_certname): mock_lineage.return_value = None @@ -208,7 +205,7 @@ class CertonlyTest(unittest.TestCase): self.assertTrue(mock_choose_names.called) class FindDomainsOrCertnameTest(unittest.TestCase): - """Tests for certbot.main._find_domains_or_certname.""" + """Tests for certbot._internal.main._find_domains_or_certname.""" @mock.patch('certbot.display.ops.choose_names') def test_display_ops(self, mock_choose_names): @@ -225,7 +222,7 @@ class FindDomainsOrCertnameTest(unittest.TestCase): # pylint: disable=protected-access self.assertRaises(errors.Error, main._find_domains_or_certname, mock_config, None) - @mock.patch('certbot.cert_manager.domains_for_certname') + @mock.patch('certbot._internal.cert_manager.domains_for_certname') def test_grab_domains(self, mock_domains): mock_config = mock.Mock(domains=None, certname="one.com") mock_domains.return_value = ["one.com", "two.com"] @@ -235,7 +232,7 @@ class FindDomainsOrCertnameTest(unittest.TestCase): class RevokeTest(test_util.TempDirTestCase): - """Tests for certbot.main.revoke.""" + """Tests for certbot._internal.main.revoke.""" def setUp(self): super(RevokeTest, self).setUp() @@ -248,16 +245,16 @@ class RevokeTest(test_util.TempDirTestCase): self.patches = [ mock.patch('acme.client.BackwardsCompatibleClientV2'), - mock.patch('certbot.client.Client'), - mock.patch('certbot.main._determine_account'), - mock.patch('certbot.main.display_ops.success_revocation') + mock.patch('certbot._internal.client.Client'), + mock.patch('certbot._internal.main._determine_account'), + mock.patch('certbot._internal.main.display_ops.success_revocation') ] self.mock_acme_client = self.patches[0].start() self.patches[1].start() self.mock_determine_account = self.patches[2].start() self.mock_success_revoke = self.patches[3].start() - from certbot.account import Account + from certbot._internal.account import Account self.regr = mock.MagicMock() self.meta = Account.Meta( @@ -282,11 +279,11 @@ class RevokeTest(test_util.TempDirTestCase): config = configuration.NamespaceConfig( cli.prepare_and_parse_args(plugins, args)) - from certbot.main import revoke + from certbot._internal.main import revoke revoke(config, plugins) - @mock.patch('certbot.main._delete_if_appropriate') - @mock.patch('certbot.main.client.acme_client') + @mock.patch('certbot._internal.main._delete_if_appropriate') + @mock.patch('certbot._internal.main.client.acme_client') def test_revoke_with_reason(self, mock_acme_client, mock_delete_if_appropriate): mock_delete_if_appropriate.return_value = False @@ -302,8 +299,8 @@ class RevokeTest(test_util.TempDirTestCase): expected.append(mock.call(mock.ANY, code)) self.assertEqual(expected, mock_revoke.call_args_list) - @mock.patch('certbot.main._delete_if_appropriate') - @mock.patch('certbot.storage.cert_path_for_cert_name') + @mock.patch('certbot._internal.main._delete_if_appropriate') + @mock.patch('certbot._internal.storage.cert_path_for_cert_name') def test_revoke_by_certname(self, mock_cert_path_for_cert_name, mock_delete_if_appropriate): args = 'revoke --cert-name=example.com'.split() @@ -312,7 +309,7 @@ class RevokeTest(test_util.TempDirTestCase): self._call(args) self.mock_success_revoke.assert_called_once_with(self.tmp_cert_path) - @mock.patch('certbot.main._delete_if_appropriate') + @mock.patch('certbot._internal.main._delete_if_appropriate') def test_revocation_success(self, mock_delete_if_appropriate): self._call() mock_delete_if_appropriate.return_value = False @@ -324,8 +321,8 @@ class RevokeTest(test_util.TempDirTestCase): self.assertRaises(acme_errors.ClientError, self._call) self.mock_success_revoke.assert_not_called() - @mock.patch('certbot.main._delete_if_appropriate') - @mock.patch('certbot.cert_manager.delete') + @mock.patch('certbot._internal.main._delete_if_appropriate') + @mock.patch('certbot._internal.cert_manager.delete') @test_util.patch_get_utility() def test_revocation_with_prompt(self, mock_get_utility, mock_delete, mock_delete_if_appropriate): @@ -335,14 +332,14 @@ class RevokeTest(test_util.TempDirTestCase): self.assertFalse(mock_delete.called) class DeleteIfAppropriateTest(test_util.ConfigTestCase): - """Tests for certbot.main._delete_if_appropriate """ + """Tests for certbot._internal.main._delete_if_appropriate """ def _call(self, mock_config): - from certbot.main import _delete_if_appropriate + from certbot._internal.main import _delete_if_appropriate _delete_if_appropriate(mock_config) def _test_delete_opt_out_common(self, mock_get_utility): - with mock.patch('certbot.cert_manager.delete') as mock_delete: + with mock.patch('certbot._internal.cert_manager.delete') as mock_delete: self._call(self.config) mock_delete.assert_not_called() self.assertTrue(mock_get_utility().add_message.called) @@ -358,12 +355,11 @@ class DeleteIfAppropriateTest(test_util.ConfigTestCase): util_mock.yesno.return_value = False self._test_delete_opt_out_common(mock_get_utility) - # pylint: disable=too-many-arguments - @mock.patch('certbot.storage.renewal_file_for_certname') - @mock.patch('certbot.cert_manager.delete') - @mock.patch('certbot.cert_manager.match_and_check_overlaps') - @mock.patch('certbot.storage.full_archive_path') - @mock.patch('certbot.cert_manager.cert_path_to_lineage') + @mock.patch('certbot._internal.storage.renewal_file_for_certname') + @mock.patch('certbot._internal.cert_manager.delete') + @mock.patch('certbot._internal.cert_manager.match_and_check_overlaps') + @mock.patch('certbot._internal.storage.full_archive_path') + @mock.patch('certbot._internal.cert_manager.cert_path_to_lineage') @test_util.patch_get_utility() def test_overlapping_archive_dirs(self, mock_get_utility, mock_cert_path_to_lineage, mock_archive, @@ -378,12 +374,11 @@ class DeleteIfAppropriateTest(test_util.ConfigTestCase): self._call(config) mock_delete.assert_not_called() - # pylint: disable=too-many-arguments - @mock.patch('certbot.storage.renewal_file_for_certname') - @mock.patch('certbot.cert_manager.match_and_check_overlaps') - @mock.patch('certbot.storage.full_archive_path') - @mock.patch('certbot.cert_manager.delete') - @mock.patch('certbot.cert_manager.cert_path_to_lineage') + @mock.patch('certbot._internal.storage.renewal_file_for_certname') + @mock.patch('certbot._internal.cert_manager.match_and_check_overlaps') + @mock.patch('certbot._internal.storage.full_archive_path') + @mock.patch('certbot._internal.cert_manager.delete') + @mock.patch('certbot._internal.cert_manager.cert_path_to_lineage') @test_util.patch_get_utility() def test_cert_path_only(self, mock_get_utility, mock_cert_path_to_lineage, mock_delete, mock_archive, @@ -397,12 +392,11 @@ class DeleteIfAppropriateTest(test_util.ConfigTestCase): self._call(config) self.assertEqual(mock_delete.call_count, 1) - # pylint: disable=too-many-arguments - @mock.patch('certbot.storage.renewal_file_for_certname') - @mock.patch('certbot.cert_manager.match_and_check_overlaps') - @mock.patch('certbot.storage.full_archive_path') - @mock.patch('certbot.cert_manager.cert_path_to_lineage') - @mock.patch('certbot.cert_manager.delete') + @mock.patch('certbot._internal.storage.renewal_file_for_certname') + @mock.patch('certbot._internal.cert_manager.match_and_check_overlaps') + @mock.patch('certbot._internal.storage.full_archive_path') + @mock.patch('certbot._internal.cert_manager.cert_path_to_lineage') + @mock.patch('certbot._internal.cert_manager.delete') @test_util.patch_get_utility() def test_noninteractive_deletion(self, mock_get_utility, mock_delete, mock_cert_path_to_lineage, mock_full_archive_dir, @@ -418,12 +412,11 @@ class DeleteIfAppropriateTest(test_util.ConfigTestCase): self._call(config) self.assertEqual(mock_delete.call_count, 1) - # pylint: disable=too-many-arguments - @mock.patch('certbot.storage.renewal_file_for_certname') - @mock.patch('certbot.cert_manager.match_and_check_overlaps') - @mock.patch('certbot.storage.full_archive_path') - @mock.patch('certbot.cert_manager.cert_path_to_lineage') - @mock.patch('certbot.cert_manager.delete') + @mock.patch('certbot._internal.storage.renewal_file_for_certname') + @mock.patch('certbot._internal.cert_manager.match_and_check_overlaps') + @mock.patch('certbot._internal.storage.full_archive_path') + @mock.patch('certbot._internal.cert_manager.cert_path_to_lineage') + @mock.patch('certbot._internal.cert_manager.delete') @test_util.patch_get_utility() def test_opt_in_deletion(self, mock_get_utility, mock_delete, mock_cert_path_to_lineage, mock_full_archive_dir, @@ -442,7 +435,7 @@ class DeleteIfAppropriateTest(test_util.ConfigTestCase): class DetermineAccountTest(test_util.ConfigTestCase): - """Tests for certbot.main._determine_account.""" + """Tests for certbot._internal.main._determine_account.""" def setUp(self): super(DetermineAccountTest, self).setUp() @@ -457,8 +450,8 @@ class DetermineAccountTest(test_util.ConfigTestCase): def _call(self): # pylint: disable=protected-access - from certbot.main import _determine_account - with mock.patch('certbot.main.account.AccountFileStorage') as mock_storage: + from certbot._internal.main import _determine_account + with mock.patch('certbot._internal.main.account.AccountFileStorage') as mock_storage: mock_storage.return_value = self.account_storage return _determine_account(self.config) @@ -475,7 +468,7 @@ class DetermineAccountTest(test_util.ConfigTestCase): self.assertEqual(self.accs[0].id, self.config.account) self.assertTrue(self.config.email is None) - @mock.patch('certbot.client.display_ops.choose_account') + @mock.patch('certbot._internal.client.display_ops.choose_account') def test_multiple_accounts(self, mock_choose_accounts): for acc in self.accs: self.account_storage.save(acc, self.mock_client) @@ -486,11 +479,11 @@ class DetermineAccountTest(test_util.ConfigTestCase): self.assertEqual(self.accs[1].id, self.config.account) self.assertTrue(self.config.email is None) - @mock.patch('certbot.client.display_ops.get_email') + @mock.patch('certbot._internal.client.display_ops.get_email') def test_no_accounts_no_email(self, mock_get_email): mock_get_email.return_value = 'foo@bar.baz' - with mock.patch('certbot.main.client') as client: + with mock.patch('certbot._internal.main.client') as client: client.register.return_value = ( self.accs[0], mock.sentinel.acme) self.assertEqual((self.accs[0], mock.sentinel.acme), self._call()) @@ -502,20 +495,20 @@ class DetermineAccountTest(test_util.ConfigTestCase): def test_no_accounts_email(self): self.config.email = 'other email' - with mock.patch('certbot.main.client') as client: + with mock.patch('certbot._internal.main.client') as client: client.register.return_value = (self.accs[1], mock.sentinel.acme) self._call() self.assertEqual(self.accs[1].id, self.config.account) self.assertEqual('other email', self.config.email) -class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-methods +class MainTest(test_util.ConfigTestCase): """Tests for different commands.""" def setUp(self): super(MainTest, self).setUp() - os.mkdir(self.config.logs_dir) + filesystem.mkdir(self.config.logs_dir) self.standard_args = ['--config-dir', self.config.config_dir, '--work-dir', self.config.work_dir, '--logs-dir', self.config.logs_dir, '--text'] @@ -534,22 +527,22 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met if mockisfile: orig_open = os.path.isfile + def mock_isfile(fn, *args, **kwargs): # pylint: disable=unused-argument """Mock os.path.isfile()""" if (fn.endswith("cert") or - fn.endswith("chain") or - fn.endswith("privkey")): + fn.endswith("chain") or + fn.endswith("privkey")): return True - else: - return orig_open(fn) + return orig_open(fn) - with mock.patch("os.path.isfile") as mock_if: + with mock.patch("certbot.compat.os.path.isfile") as mock_if: mock_if.side_effect = mock_isfile - with mock.patch('certbot.main.client') as client: + with mock.patch('certbot._internal.main.client') as client: ret, stdout, stderr = self._call_no_clientmock(args, stdout) return ret, stdout, stderr, client else: - with mock.patch('certbot.main.client') as client: + with mock.patch('certbot._internal.main.client') as client: ret, stdout, stderr = self._call_no_clientmock(args, stdout) return ret, stdout, stderr, client @@ -558,22 +551,22 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met args = self.standard_args + args toy_stdout = stdout if stdout else six.StringIO() - with mock.patch('certbot.main.sys.stdout', new=toy_stdout): - with mock.patch('certbot.main.sys.stderr') as stderr: + with mock.patch('certbot._internal.main.sys.stdout', new=toy_stdout): + with mock.patch('certbot._internal.main.sys.stderr') as stderr: with mock.patch("certbot.util.atexit"): ret = main.main(args[:]) # NOTE: parser can alter its args! return ret, toy_stdout, stderr def test_no_flags(self): - with mock.patch('certbot.main.run') as mock_run: + with mock.patch('certbot._internal.main.run') as mock_run: self._call([]) self.assertEqual(1, mock_run.call_count) def test_version_string_program_name(self): toy_out = six.StringIO() toy_err = six.StringIO() - with mock.patch('certbot.main.sys.stdout', new=toy_out): - with mock.patch('certbot.main.sys.stderr', new=toy_err): + with mock.patch('certbot._internal.main.sys.stdout', new=toy_out): + with mock.patch('certbot._internal.main.sys.stderr', new=toy_err): try: main.main(["--version"]) except SystemExit: @@ -586,27 +579,27 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met "Ensure that a particular error raises a missing cli flag error containing message" exc = None try: - with mock.patch('certbot.main.sys.stderr'): + with mock.patch('certbot._internal.main.sys.stderr'): main.main(self.standard_args + args[:]) # NOTE: parser can alter its args! except errors.MissingCommandlineFlag as exc_: exc = exc_ self.assertTrue(message in str(exc)) self.assertTrue(exc is not None) - @test_util.broken_on_windows - def test_noninteractive(self): + @mock.patch('certbot._internal.log.post_arg_parse_setup') + def test_noninteractive(self, _): args = ['-n', 'certonly'] self._cli_missing_flag(args, "specify a plugin") args.extend(['--standalone', '-d', 'eg.is']) self._cli_missing_flag(args, "register before running") - @test_util.broken_on_windows - @mock.patch('certbot.main._report_new_cert') - @mock.patch('certbot.main.client.acme_client.Client') - @mock.patch('certbot.main._determine_account') - @mock.patch('certbot.main.client.Client.obtain_and_enroll_certificate') - @mock.patch('certbot.main._get_and_save_cert') - def test_user_agent(self, gsc, _obt, det, _client, unused_report): + @mock.patch('certbot._internal.log.post_arg_parse_setup') + @mock.patch('certbot._internal.main._report_new_cert') + @mock.patch('certbot._internal.main.client.acme_client.Client') + @mock.patch('certbot._internal.main._determine_account') + @mock.patch('certbot._internal.main.client.Client.obtain_and_enroll_certificate') + @mock.patch('certbot._internal.main._get_and_save_cert') + def test_user_agent(self, gsc, _obt, det, _client, _, __): # Normally the client is totally mocked out, but here we need more # arguments to automate it... args = ["--standalone", "certonly", "-m", "none@none.com", @@ -614,7 +607,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met det.return_value = mock.MagicMock(), None gsc.return_value = mock.MagicMock() - with mock.patch('certbot.main.client.acme_client.ClientNetwork') as acme_net: + with mock.patch('certbot._internal.main.client.acme_client.ClientNetwork') as acme_net: self._call_no_clientmock(args) os_ver = util.get_os_info_ua() ua = acme_net.call_args[1]["user_agent"] @@ -624,30 +617,30 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met if "linux" in plat.lower(): self.assertTrue(util.get_os_info_ua() in ua) - with mock.patch('certbot.main.client.acme_client.ClientNetwork') as acme_net: + with mock.patch('certbot._internal.main.client.acme_client.ClientNetwork') as acme_net: ua = "bandersnatch" args += ["--user-agent", ua] self._call_no_clientmock(args) acme_net.assert_called_once_with(mock.ANY, account=mock.ANY, verify_ssl=True, user_agent=ua) - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') - @mock.patch('certbot.main.plug_sel.pick_installer') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.main.plug_sel.pick_installer') def test_installer_selection(self, mock_pick_installer, _rec): self._call(['install', '--domains', 'foo.bar', '--cert-path', 'cert', '--key-path', 'privkey', '--chain-path', 'chain'], mockisfile=True) self.assertEqual(mock_pick_installer.call_count, 1) - @mock.patch('certbot.main._install_cert') - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') - @mock.patch('certbot.main.plug_sel.pick_installer') + @mock.patch('certbot._internal.main._install_cert') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.main.plug_sel.pick_installer') def test_installer_certname(self, _inst, _rec, mock_install): mock_lineage = mock.MagicMock(cert_path=test_util.temp_join('cert'), chain_path=test_util.temp_join('chain'), fullchain_path=test_util.temp_join('chain'), key_path=test_util.temp_join('privkey')) - with mock.patch("certbot.cert_manager.lineage_for_certname") as mock_getlin: + with mock.patch("certbot._internal.cert_manager.lineage_for_certname") as mock_getlin: mock_getlin.return_value = mock_lineage self._call(['install', '--cert-name', 'whatever'], mockisfile=True) call_config = mock_install.call_args[0][0] @@ -655,16 +648,16 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self.assertEqual(call_config.fullchain_path, test_util.temp_join('chain')) self.assertEqual(call_config.key_path, test_util.temp_join('privkey')) - @test_util.broken_on_windows - @mock.patch('certbot.main._install_cert') - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') - @mock.patch('certbot.main.plug_sel.pick_installer') - def test_installer_param_override(self, _inst, _rec, mock_install): + @mock.patch('certbot._internal.log.post_arg_parse_setup') + @mock.patch('certbot._internal.main._install_cert') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.main.plug_sel.pick_installer') + def test_installer_param_override(self, _inst, _rec, mock_install, _): mock_lineage = mock.MagicMock(cert_path=test_util.temp_join('cert'), chain_path=test_util.temp_join('chain'), fullchain_path=test_util.temp_join('chain'), key_path=test_util.temp_join('privkey')) - with mock.patch("certbot.cert_manager.lineage_for_certname") as mock_getlin: + with mock.patch("certbot._internal.cert_manager.lineage_for_certname") as mock_getlin: mock_getlin.return_value = mock_lineage self._call(['install', '--cert-name', 'whatever', '--key-path', test_util.temp_join('overriding_privkey')], mockisfile=True) @@ -683,33 +676,33 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self.assertEqual(call_config.fullchain_path, test_util.temp_join('chain')) self.assertEqual(call_config.key_path, test_util.temp_join('privkey')) - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') - @mock.patch('certbot.main.plug_sel.pick_installer') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.main.plug_sel.pick_installer') def test_installer_param_error(self, _inst, _rec): self.assertRaises(errors.ConfigurationError, self._call, ['install', '--cert-name', 'notfound', '--key-path', 'invalid']) - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') - @mock.patch('certbot.main.plug_sel.pick_installer') - @mock.patch('certbot.cert_manager.get_certnames') - @mock.patch('certbot.main._install_cert') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.main.plug_sel.pick_installer') + @mock.patch('certbot._internal.cert_manager.get_certnames') + @mock.patch('certbot._internal.main._install_cert') def test_installer_select_cert(self, mock_inst, mock_getcert, _inst, _rec): mock_lineage = mock.MagicMock(cert_path=test_util.temp_join('cert'), chain_path=test_util.temp_join('chain'), fullchain_path=test_util.temp_join('chain'), key_path=test_util.temp_join('privkey')) - with mock.patch("certbot.cert_manager.lineage_for_certname") as mock_getlin: + with mock.patch("certbot._internal.cert_manager.lineage_for_certname") as mock_getlin: mock_getlin.return_value = mock_lineage self._call(['install'], mockisfile=True) self.assertTrue(mock_getcert.called) self.assertTrue(mock_inst.called) - @test_util.broken_on_windows - @mock.patch('certbot.main._report_new_cert') + @mock.patch('certbot._internal.log.post_arg_parse_setup') + @mock.patch('certbot._internal.main._report_new_cert') @mock.patch('certbot.util.exe_exists') - def test_configurator_selection(self, mock_exe_exists, unused_report): + def test_configurator_selection(self, mock_exe_exists, _, __): mock_exe_exists.return_value = True real_plugins = disco.PluginsRegistry.find_all() args = ['--apache', '--authenticator', 'standalone'] @@ -717,7 +710,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met # This needed two calls to find_all(), which we're avoiding for now # because of possible side effects: # https://github.com/letsencrypt/letsencrypt/commit/51ed2b681f87b1eb29088dd48718a54f401e4855 - #with mock.patch('certbot.cli.plugins_testable') as plugins: + #with mock.patch('certbot._internal.cli.plugins_testable') as plugins: # plugins.return_value = {"apache": True, "nginx": True} # ret, _, _, _ = self._call(args) # self.assertTrue("Too many flags setting" in ret) @@ -735,19 +728,19 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self._cli_missing_flag(["--standalone"], "With the standalone plugin, you probably") - with mock.patch("certbot.main._init_le_client") as mock_init: - with mock.patch("certbot.main._get_and_save_cert") as mock_gsc: + with mock.patch("certbot._internal.main._init_le_client") as mock_init: + with mock.patch("certbot._internal.main._get_and_save_cert") as mock_gsc: mock_gsc.return_value = mock.MagicMock() self._call(["certonly", "--manual", "-d", "foo.bar"]) unused_config, auth, unused_installer = mock_init.call_args[0] self.assertTrue(isinstance(auth, manual.Authenticator)) - with mock.patch('certbot.main.certonly') as mock_certonly: + with mock.patch('certbot._internal.main.certonly') as mock_certonly: self._call(["auth", "--standalone"]) self.assertEqual(1, mock_certonly.call_count) - @test_util.broken_on_windows - def test_rollback(self): + @mock.patch('certbot._internal.log.post_arg_parse_setup') + def test_rollback(self, _): _, _, _, client = self._call(['rollback']) self.assertEqual(1, client.rollback.call_count) @@ -755,35 +748,31 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met client.rollback.assert_called_once_with( mock.ANY, 123, mock.ANY, mock.ANY) - def test_config_changes(self): - _, _, _, client = self._call(['config_changes']) - self.assertEqual(1, client.view_config_changes.call_count) - - @mock.patch('certbot.cert_manager.update_live_symlinks') + @mock.patch('certbot._internal.cert_manager.update_live_symlinks') def test_update_symlinks(self, mock_cert_manager): self._call_no_clientmock(['update_symlinks']) self.assertEqual(1, mock_cert_manager.call_count) - @mock.patch('certbot.cert_manager.certificates') + @mock.patch('certbot._internal.cert_manager.certificates') def test_certificates(self, mock_cert_manager): self._call_no_clientmock(['certificates']) self.assertEqual(1, mock_cert_manager.call_count) - @mock.patch('certbot.cert_manager.delete') + @mock.patch('certbot._internal.cert_manager.delete') def test_delete(self, mock_cert_manager): self._call_no_clientmock(['delete']) self.assertEqual(1, mock_cert_manager.call_count) - @test_util.broken_on_windows - def test_plugins(self): + @mock.patch('certbot._internal.log.post_arg_parse_setup') + def test_plugins(self, _): flags = ['--init', '--prepare', '--authenticators', '--installers'] for args in itertools.chain( *(itertools.combinations(flags, r) for r in six.moves.range(len(flags)))): self._call(['plugins'] + list(args)) - @mock.patch('certbot.main.plugins_disco') - @mock.patch('certbot.main.cli.HelpfulArgumentParser.determine_help_topics') + @mock.patch('certbot._internal.main.plugins_disco') + @mock.patch('certbot._internal.main.cli.HelpfulArgumentParser.determine_help_topics') def test_plugins_no_args(self, _det, mock_disco): ifaces = [] # type: List[interfaces.IPlugin] plugins = mock_disco.PluginsRegistry.find_all() @@ -797,15 +786,15 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met filtered = plugins.visible().ifaces() self.assertEqual(stdout.getvalue().strip(), str(filtered)) - @mock.patch('certbot.main.plugins_disco') - @mock.patch('certbot.main.cli.HelpfulArgumentParser.determine_help_topics') + @mock.patch('certbot._internal.main.plugins_disco') + @mock.patch('certbot._internal.main.cli.HelpfulArgumentParser.determine_help_topics') def test_plugins_no_args_unprivileged(self, _det, mock_disco): ifaces = [] # type: List[interfaces.IPlugin] plugins = mock_disco.PluginsRegistry.find_all() - def throw_error(directory, mode, uid, strict): + def throw_error(directory, mode, strict): """Raises error.Error.""" - _, _, _, _ = directory, mode, uid, strict + _, _, _ = directory, mode, strict raise errors.Error() stdout = six.StringIO() @@ -819,8 +808,8 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met filtered = plugins.visible().ifaces() self.assertEqual(stdout.getvalue().strip(), str(filtered)) - @mock.patch('certbot.main.plugins_disco') - @mock.patch('certbot.main.cli.HelpfulArgumentParser.determine_help_topics') + @mock.patch('certbot._internal.main.plugins_disco') + @mock.patch('certbot._internal.main.cli.HelpfulArgumentParser.determine_help_topics') def test_plugins_init(self, _det, mock_disco): ifaces = [] # type: List[interfaces.IPlugin] plugins = mock_disco.PluginsRegistry.find_all() @@ -837,8 +826,8 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met verified = filtered.verify() self.assertEqual(stdout.getvalue().strip(), str(verified)) - @mock.patch('certbot.main.plugins_disco') - @mock.patch('certbot.main.cli.HelpfulArgumentParser.determine_help_topics') + @mock.patch('certbot._internal.main.plugins_disco') + @mock.patch('certbot._internal.main.cli.HelpfulArgumentParser.determine_help_topics') def test_plugins_prepare(self, _det, mock_disco): ifaces = [] # type: List[interfaces.IPlugin] plugins = mock_disco.PluginsRegistry.find_all() @@ -864,7 +853,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met chain = 'chain' fullchain = 'fullchain' - with mock.patch('certbot.main.certonly') as mock_certonly: + with mock.patch('certbot._internal.main.certonly') as mock_certonly: self._call(['certonly', '--cert-path', cert, '--key-path', 'key', '--chain-path', 'chain', '--fullchain-path', 'fullchain']) @@ -922,9 +911,10 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met 'certonly -d example.org --csr {0}'.format(CSR).split()) def _certonly_new_request_common(self, mock_client, args=None): - with mock.patch('certbot.main._find_lineage_for_domains_and_certname') as mock_renewal: + with mock.patch('certbot._internal.main._find_lineage_for_domains_and_certname') \ + as mock_renewal: mock_renewal.return_value = ("newcert", None) - with mock.patch('certbot.main._init_le_client') as mock_init: + with mock.patch('certbot._internal.main._init_le_client') as mock_init: mock_init.return_value = mock_client if args is None: args = [] @@ -975,7 +965,6 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met args=None, should_renew=True, error_expected=False, quiet_mode=False, expiry_date=datetime.datetime.now(), reuse_key=False): - # pylint: disable=too-many-locals,too-many-arguments,too-many-branches cert_path = test_util.vector_path('cert_512.pem') chain_path = os.path.normpath(os.path.join(self.config.config_dir, 'live/foo.bar/fullchain.pem')) @@ -996,18 +985,19 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met stdout.write(message) try: - with mock.patch('certbot.cert_manager.find_duplicative_certs') as mock_fdc: + with mock.patch('certbot._internal.cert_manager.find_duplicative_certs') as mock_fdc: mock_fdc.return_value = (mock_lineage, None) - with mock.patch('certbot.main._init_le_client') as mock_init: + with mock.patch('certbot._internal.main._init_le_client') as mock_init: mock_init.return_value = mock_client with test_util.patch_get_utility() as mock_get_utility: if not quiet_mode: mock_get_utility().notification.side_effect = write_msg - with mock.patch('certbot.main.renewal.OpenSSL') as mock_ssl: + with mock.patch('certbot._internal.main.renewal.OpenSSL') as mock_ssl: mock_latest = mock.MagicMock() mock_latest.get_issuer.return_value = "Fake fake" mock_ssl.crypto.load_certificate.return_value = mock_latest - with mock.patch('certbot.main.renewal.crypto_util') as mock_crypto_util: + with mock.patch('certbot._internal.main.renewal.crypto_util') \ + as mock_crypto_util: mock_crypto_util.notAfter.return_value = expiry_date if not args: args = ['-d', 'isnot.org', '-a', 'standalone', 'certonly'] @@ -1047,7 +1037,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met return mock_lineage, mock_get_utility, stdout @mock.patch('certbot.crypto_util.notAfter') - def test_certonly_renewal(self, unused_notafter): + def test_certonly_renewal(self, _): lineage, get_utility, _ = self._test_renewal_common(True, []) self.assertEqual(lineage.save_successor.call_count, 1) lineage.update_all_links_to.assert_called_once_with( @@ -1056,9 +1046,9 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self.assertTrue('fullchain.pem' in cert_msg) self.assertTrue('donate' in get_utility().add_message.call_args[0][0]) - @test_util.broken_on_windows + @mock.patch('certbot._internal.log.logging.handlers.RotatingFileHandler.doRollover') @mock.patch('certbot.crypto_util.notAfter') - def test_certonly_renewal_triggers(self, unused_notafter): + def test_certonly_renewal_triggers(self, _, __): # --dry-run should force renewal _, get_utility, _ = self._test_renewal_common(False, ['--dry-run', '--keep'], log_out="simulating renewal") @@ -1089,7 +1079,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met args = ["renew", "--dry-run", "--reuse-key"] self._test_renewal_common(True, [], args=args, should_renew=True, reuse_key=True) - @mock.patch('certbot.storage.RenewableCert.save_successor') + @mock.patch('certbot._internal.storage.RenewableCert.save_successor') def test_reuse_key_no_dry_run(self, unused_save_successor): test_util.make_lineage(self.config.config_dir, 'sample-renewal.conf') args = ["renew", "--reuse-key"] @@ -1115,7 +1105,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self._test_renewal_common(True, [], args=args, should_renew=True) self.assertEqual(self.mock_sleep.call_count, 0) - @mock.patch('certbot.renewal.should_renew') + @mock.patch('certbot._internal.renewal.should_renew') def test_renew_skips_recent_certs(self, should_renew): should_renew.return_value = False test_util.make_lineage(self.config.config_dir, 'sample-renewal.conf') @@ -1125,8 +1115,8 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self.assertTrue('No renewals were attempted.' in stdout.getvalue()) self.assertTrue('The following certs are not due for renewal yet:' in stdout.getvalue()) - @test_util.broken_on_windows - def test_quiet_renew(self): + @mock.patch('certbot._internal.log.post_arg_parse_setup') + def test_quiet_renew(self, _): test_util.make_lineage(self.config.config_dir, 'sample-renewal.conf') args = ["renew", "--dry-run"] _, _, stdout = self._test_renewal_common(True, [], args=args, should_renew=True) @@ -1149,14 +1139,14 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met test_util.make_lineage(self.config.config_dir, 'sample-renewal.conf') args = ["renew", "--dry-run", "--post-hook=no-such-command", "--disable-hook-validation"] - with mock.patch("certbot.hooks.post_hook"): + with mock.patch("certbot._internal.hooks.post_hook"): self._test_renewal_common(True, [], args=args, should_renew=True, error_expected=False) def test_renew_verb_empty_config(self): rd = os.path.join(self.config.config_dir, 'renewal') if not os.path.exists(rd): - os.makedirs(rd) + filesystem.makedirs(rd) with open(os.path.join(rd, 'empty.conf'), 'w'): pass # leave the file empty args = ["renew", "--dry-run", "-tvv"] @@ -1174,14 +1164,14 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met def _make_dummy_renewal_config(self): renewer_configs_dir = os.path.join(self.config.config_dir, 'renewal') - os.makedirs(renewer_configs_dir) + filesystem.makedirs(renewer_configs_dir) with open(os.path.join(renewer_configs_dir, 'test.conf'), 'w') as f: f.write("My contents don't matter") def _test_renew_common(self, renewalparams=None, names=None, assert_oc_called=None, **kwargs): self._make_dummy_renewal_config() - with mock.patch('certbot.storage.RenewableCert') as mock_rc: + with mock.patch('certbot._internal.storage.RenewableCert') as mock_rc: mock_lineage = mock.MagicMock() mock_lineage.fullchain = "somepath/fullchain.pem" if renewalparams is not None: @@ -1189,7 +1179,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met if names is not None: mock_lineage.names.return_value = names mock_rc.return_value = mock_lineage - with mock.patch('certbot.main.renew_cert') as mock_renew_cert: + with mock.patch('certbot._internal.main.renew_cert') as mock_renew_cert: kwargs.setdefault('args', ['renew']) self._test_renewal_common(True, None, should_renew=False, **kwargs) @@ -1224,7 +1214,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self._test_renew_common(renewalparams=renewalparams, error_expected=True, names=names, assert_oc_called=False) - @mock.patch('certbot.plugins.selection.choose_configurator_plugins') + @mock.patch('certbot._internal.plugins.selection.choose_configurator_plugins') def test_renew_with_configurator(self, mock_sel): mock_sel.return_value = (mock.MagicMock(), mock.MagicMock()) renewalparams = {'authenticator': 'webroot'} @@ -1247,19 +1237,19 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met def test_renew_reconstitute_error(self): # pylint: disable=protected-access - with mock.patch('certbot.main.renewal._reconstitute') as mock_reconstitute: + with mock.patch('certbot._internal.main.renewal._reconstitute') as mock_reconstitute: mock_reconstitute.side_effect = Exception self._test_renew_common(assert_oc_called=False, error_expected=True) def test_renew_obtain_cert_error(self): self._make_dummy_renewal_config() - with mock.patch('certbot.storage.RenewableCert') as mock_rc: + with mock.patch('certbot._internal.storage.RenewableCert') as mock_rc: mock_lineage = mock.MagicMock() mock_lineage.fullchain = "somewhere/fullchain.pem" mock_rc.return_value = mock_lineage mock_lineage.configuration = { 'renewalparams': {'authenticator': 'webroot'}} - with mock.patch('certbot.main.renew_cert') as mock_renew_cert: + with mock.patch('certbot._internal.main.renew_cert') as mock_renew_cert: mock_renew_cert.side_effect = Exception self._test_renewal_common(True, None, error_expected=True, args=['renew'], should_renew=False) @@ -1279,8 +1269,8 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self.assertTrue('No hooks were run.' in stdout.getvalue()) @test_util.patch_get_utility() - @mock.patch('certbot.main._find_lineage_for_domains_and_certname') - @mock.patch('certbot.main._init_le_client') + @mock.patch('certbot._internal.main._find_lineage_for_domains_and_certname') + @mock.patch('certbot._internal.main._init_le_client') def test_certonly_reinstall(self, mock_init, mock_renewal, mock_get_utility): mock_renewal.return_value = ('reinstall', mock.MagicMock()) mock_init.return_value = mock_client = mock.MagicMock() @@ -1302,7 +1292,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self.config.config_dir, 'live/example.com/fullchain.pem')) mock_client.save_certificate.return_value = cert_path, None, full_path - with mock.patch('certbot.main._init_le_client') as mock_init: + with mock.patch('certbot._internal.main._init_le_client') as mock_init: mock_init.return_value = mock_client with test_util.patch_get_utility() as mock_get_utility: chain_path = os.path.normpath(os.path.join( @@ -1313,7 +1303,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met CSR, cert_path, chain_path, full_path).split() if extra_args: args += extra_args - with mock.patch('certbot.main.crypto_util'): + with mock.patch('certbot._internal.main.crypto_util'): self._call(args) if '--dry-run' in args: @@ -1338,8 +1328,8 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self.assertTrue( 'dry run' in mock_get_utility().add_message.call_args[0][0]) - @mock.patch('certbot.main._delete_if_appropriate') - @mock.patch('certbot.main.client.acme_client') + @mock.patch('certbot._internal.main._delete_if_appropriate') + @mock.patch('certbot._internal.main.client.acme_client') def test_revoke_with_key(self, mock_acme_client, mock_delete_if_appropriate): mock_delete_if_appropriate.return_value = False @@ -1362,8 +1352,8 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met ['--cert-path', CERT, '--key-path', KEY, '--server', server, 'revoke']) - @mock.patch('certbot.main._delete_if_appropriate') - @mock.patch('certbot.main._determine_account') + @mock.patch('certbot._internal.main._delete_if_appropriate') + @mock.patch('certbot._internal.main._determine_account') def test_revoke_without_key(self, mock_determine_account, mock_delete_if_appropriate): mock_delete_if_appropriate.return_value = False @@ -1376,14 +1366,9 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met jose.ComparableX509(cert), mock.ANY) - def test_agree_dev_preview_config(self): - with mock.patch('certbot.main.run') as mocked_run: - self._call(['-c', test_util.vector_path('cli.ini')]) - self.assertTrue(mocked_run.called) - - @test_util.broken_on_windows - def test_register(self): - with mock.patch('certbot.main.client') as mocked_client: + @mock.patch('certbot._internal.log.post_arg_parse_setup') + def test_register(self, _): + with mock.patch('certbot._internal.main.client') as mocked_client: acc = mock.MagicMock() acc.id = "imaginary_account" mocked_client.register.return_value = (acc, "worked") @@ -1391,7 +1376,7 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met # TODO: It would be more correct to explicitly check that # _determine_account() gets called in the above case, # but coverage statistics should also show that it did. - with mock.patch('certbot.main.account') as mocked_account: + with mock.patch('certbot._internal.main.account') as mocked_account: mocked_storage = mock.MagicMock() mocked_account.AccountFileStorage.return_value = mocked_storage mocked_storage.find_all.return_value = ["an account"] @@ -1399,8 +1384,8 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met self.assertTrue("There is an existing account" in x[0]) def test_update_account_no_existing_accounts(self): - # with mock.patch('certbot.main.client') as mocked_client: - with mock.patch('certbot.main.account') as mocked_account: + # with mock.patch('certbot._internal.main.client') as mocked_client: + with mock.patch('certbot._internal.main.account') as mocked_account: mocked_storage = mock.MagicMock() mocked_account.AccountFileStorage.return_value = mocked_storage mocked_storage.find_all.return_value = [] @@ -1409,42 +1394,15 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met "user@example.org"]) self.assertTrue("Could not find an existing account" in x[0]) - # TODO: When `certbot register --update-registration` is fully deprecated, - # delete the following test - def test_update_registration_no_existing_accounts_deprecated(self): - # with mock.patch('certbot.main.client') as mocked_client: - with mock.patch('certbot.main.account') as mocked_account: - mocked_storage = mock.MagicMock() - mocked_account.AccountFileStorage.return_value = mocked_storage - mocked_storage.find_all.return_value = [] - x = self._call_no_clientmock( - ["register", "--update-registration", "--email", - "user@example.org"]) - self.assertTrue("Could not find an existing account" in x[0]) - - # TODO: When `certbot register --update-registration` is fully deprecated, - # delete the following test - def test_update_registration_unsafely_deprecated(self): - # This test will become obsolete when register --update-registration - # supports removing an e-mail address from the account - with mock.patch('certbot.main.account') as mocked_account: - mocked_storage = mock.MagicMock() - mocked_account.AccountFileStorage.return_value = mocked_storage - mocked_storage.find_all.return_value = ["an account"] - x = self._call_no_clientmock( - "register --update-registration " - "--register-unsafely-without-email".split()) - self.assertTrue("--register-unsafely-without-email" in x[0]) - - @mock.patch('certbot.main.display_ops.get_email') + @mock.patch('certbot._internal.main.display_ops.get_email') @test_util.patch_get_utility() def test_update_account_with_email(self, mock_utility, mock_email): email = "user@example.com" mock_email.return_value = email - with mock.patch('certbot.eff.handle_subscription') as mock_handle: - with mock.patch('certbot.main._determine_account') as mocked_det: - with mock.patch('certbot.main.account') as mocked_account: - with mock.patch('certbot.main.client') as mocked_client: + with mock.patch('certbot._internal.eff.handle_subscription') as mock_handle: + with mock.patch('certbot._internal.main._determine_account') as mocked_det: + with mock.patch('certbot._internal.main.account') as mocked_account: + with mock.patch('certbot._internal.main.client') as mocked_client: mocked_storage = mock.MagicMock() mocked_account.AccountFileStorage.return_value = mocked_storage mocked_storage.find_all.return_value = ["an account"] @@ -1466,44 +1424,8 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met email in mock_utility().add_message.call_args[0][0]) self.assertTrue(mock_handle.called) - # TODO: When `certbot register --update-registration` is fully deprecated, - # delete the following test - @mock.patch('certbot.main.display_ops.get_email') - @test_util.patch_get_utility() - def test_update_registration_with_email_deprecated(self, mock_utility, mock_email): - email = "user@example.com" - mock_email.return_value = email - with mock.patch('certbot.eff.handle_subscription') as mock_handle: - with mock.patch('certbot.main._determine_account') as mocked_det: - with mock.patch('certbot.main.account') as mocked_account: - with mock.patch('certbot.main.client') as mocked_client: - mocked_storage = mock.MagicMock() - mocked_account.AccountFileStorage.return_value = mocked_storage - mocked_storage.find_all.return_value = ["an account"] - mock_acc = mock.MagicMock() - mock_regr = mock_acc.regr - mocked_det.return_value = (mock_acc, "foo") - cb_client = mock.MagicMock() - mocked_client.Client.return_value = cb_client - x = self._call_no_clientmock( - ["register", "--update-registration"]) - # When registration change succeeds, the return value - # of register() is None - self.assertTrue(x[0] is None) - # and we got supposedly did update the registration from - # the server - reg_arg = cb_client.acme.update_registration.call_args[0][0] - # Test the return value of .update() was used because - # the regr is immutable. - self.assertEqual(reg_arg, mock_regr.update()) - # and we saved the updated registration on disk - self.assertTrue(mocked_storage.save_regr.called) - self.assertTrue( - email in mock_utility().add_message.call_args[0][0]) - self.assertTrue(mock_handle.called) - - @mock.patch('certbot.plugins.selection.choose_configurator_plugins') - @mock.patch('certbot.updater._run_updaters') + @mock.patch('certbot._internal.plugins.selection.choose_configurator_plugins') + @mock.patch('certbot._internal.updater._run_updaters') def test_plugin_selection_error(self, mock_run, mock_choose): mock_choose.side_effect = errors.PluginSelectionError self.assertRaises(errors.PluginSelectionError, main.renew_cert, @@ -1519,9 +1441,9 @@ class MainTest(test_util.ConfigTestCase): # pylint: disable=too-many-public-met class UnregisterTest(unittest.TestCase): def setUp(self): self.patchers = { - '_determine_account': mock.patch('certbot.main._determine_account'), - 'account': mock.patch('certbot.main.account'), - 'client': mock.patch('certbot.main.client'), + '_determine_account': mock.patch('certbot._internal.main._determine_account'), + 'account': mock.patch('certbot._internal.main.account'), + 'client': mock.patch('certbot._internal.main.client'), 'get_utility': test_util.patch_get_utility()} self.mocks = dict((k, v.start()) for k, v in self.patchers.items()) @@ -1579,15 +1501,15 @@ class UnregisterTest(unittest.TestCase): class MakeOrVerifyNeededDirs(test_util.ConfigTestCase): - """Tests for certbot.main.make_or_verify_needed_dirs.""" + """Tests for certbot._internal.main.make_or_verify_needed_dirs.""" - @mock.patch("certbot.main.util") + @mock.patch("certbot._internal.main.util") def test_it(self, mock_util): main.make_or_verify_needed_dirs(self.config) for core_dir in (self.config.config_dir, self.config.work_dir,): mock_util.set_up_core_dir.assert_any_call( core_dir, constants.CONFIG_DIRS_MODE, - compat.os_geteuid(), self.config.strict_permissions + self.config.strict_permissions ) hook_dirs = (self.config.renewal_pre_hooks_dir, @@ -1596,12 +1518,11 @@ class MakeOrVerifyNeededDirs(test_util.ConfigTestCase): for hook_dir in hook_dirs: # default mode of 755 is used mock_util.make_or_verify_dir.assert_any_call( - hook_dir, uid=compat.os_geteuid(), - strict=self.config.strict_permissions) + hook_dir, strict=self.config.strict_permissions) class EnhanceTest(test_util.ConfigTestCase): - """Tests for certbot.main.enhance.""" + """Tests for certbot._internal.main.enhance.""" def setUp(self): super(EnhanceTest, self).setUp() @@ -1617,53 +1538,53 @@ class EnhanceTest(test_util.ConfigTestCase): config = configuration.NamespaceConfig( cli.prepare_and_parse_args(plugins, args)) - with mock.patch('certbot.cert_manager.get_certnames') as mock_certs: + with mock.patch('certbot._internal.cert_manager.get_certnames') as mock_certs: mock_certs.return_value = ['example.com'] - with mock.patch('certbot.cert_manager.domains_for_certname') as mock_dom: + with mock.patch('certbot._internal.cert_manager.domains_for_certname') as mock_dom: mock_dom.return_value = ['example.com'] - with mock.patch('certbot.main._init_le_client') as mock_init: + with mock.patch('certbot._internal.main._init_le_client') as mock_init: mock_client = mock.MagicMock() mock_client.config = config mock_init.return_value = mock_client main.enhance(config, plugins) return mock_client # returns the client - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.main.display_ops.choose_values') - @mock.patch('certbot.main._find_domains_or_certname') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.main.display_ops.choose_values') + @mock.patch('certbot._internal.main._find_domains_or_certname') def test_selection_question(self, mock_find, mock_choose, mock_lineage, _rec): mock_lineage.return_value = mock.MagicMock(chain_path="/tmp/nonexistent") mock_choose.return_value = ['example.com'] mock_find.return_value = (None, None) - with mock.patch('certbot.main.plug_sel.pick_installer') as mock_pick: + with mock.patch('certbot._internal.main.plug_sel.pick_installer') as mock_pick: self._call(['enhance', '--redirect']) self.assertTrue(mock_pick.called) # Check that the message includes "enhancements" self.assertTrue("enhancements" in mock_pick.call_args[0][3]) - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.main.display_ops.choose_values') - @mock.patch('certbot.main._find_domains_or_certname') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.main.display_ops.choose_values') + @mock.patch('certbot._internal.main._find_domains_or_certname') def test_selection_auth_warning(self, mock_find, mock_choose, mock_lineage, _rec): mock_lineage.return_value = mock.MagicMock(chain_path="/tmp/nonexistent") mock_choose.return_value = ["example.com"] mock_find.return_value = (None, None) - with mock.patch('certbot.main.plug_sel.pick_installer'): - with mock.patch('certbot.main.plug_sel.logger.warning') as mock_log: + with mock.patch('certbot._internal.main.plug_sel.pick_installer'): + with mock.patch('certbot._internal.main.plug_sel.logger.warning') as mock_log: mock_client = self._call(['enhance', '-a', 'webroot', '--redirect']) self.assertTrue(mock_log.called) self.assertTrue("make sense" in mock_log.call_args[0][0]) self.assertTrue(mock_client.enhance_config.called) - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.main.display_ops.choose_values') - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.main.display_ops.choose_values') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') def test_enhance_config_call(self, _rec, mock_choose, mock_lineage): mock_lineage.return_value = mock.MagicMock(chain_path="/tmp/nonexistent") mock_choose.return_value = ["example.com"] - with mock.patch('certbot.main.plug_sel.pick_installer'): + with mock.patch('certbot._internal.main.plug_sel.pick_installer'): mock_client = self._call(['enhance', '--redirect', '--hsts']) req_enh = ["redirect", "hsts"] not_req_enh = ["uir"] @@ -1675,24 +1596,24 @@ class EnhanceTest(test_util.ConfigTestCase): self.assertTrue( "example.com" in mock_client.enhance_config.call_args[0][0]) - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.main.display_ops.choose_values') - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.main.display_ops.choose_values') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') def test_enhance_noninteractive(self, _rec, mock_choose, mock_lineage): mock_lineage.return_value = mock.MagicMock( chain_path="/tmp/nonexistent") mock_choose.return_value = ["example.com"] - with mock.patch('certbot.main.plug_sel.pick_installer'): + with mock.patch('certbot._internal.main.plug_sel.pick_installer'): mock_client = self._call(['enhance', '--redirect', '--hsts', '--non-interactive']) self.assertTrue(mock_client.enhance_config.called) self.assertFalse(mock_choose.called) - @mock.patch('certbot.main.display_ops.choose_values') - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.main.display_ops.choose_values') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') def test_user_abort_domains(self, _rec, mock_choose): mock_choose.return_value = [] - with mock.patch('certbot.main.plug_sel.pick_installer'): + with mock.patch('certbot._internal.main.plug_sel.pick_installer'): self.assertRaises(errors.Error, self._call, ['enhance', '--redirect', '--hsts']) @@ -1701,9 +1622,9 @@ class EnhanceTest(test_util.ConfigTestCase): self.assertRaises(errors.MisconfigurationError, self._call, ['enhance', '-a', 'null']) - @mock.patch('certbot.main.plug_sel.choose_configurator_plugins') - @mock.patch('certbot.main.display_ops.choose_values') - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.main.plug_sel.choose_configurator_plugins') + @mock.patch('certbot._internal.main.display_ops.choose_values') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') def test_plugin_selection_error(self, _rec, mock_choose, mock_pick): mock_choose.return_value = ["example.com"] mock_pick.return_value = (None, None) @@ -1711,10 +1632,10 @@ class EnhanceTest(test_util.ConfigTestCase): mock_client = self._call(['enhance', '--hsts']) self.assertFalse(mock_client.enhance_config.called) - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.main.display_ops.choose_values') - @mock.patch('certbot.main.plug_sel.pick_installer') - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.main.display_ops.choose_values') + @mock.patch('certbot._internal.main.plug_sel.pick_installer') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') @test_util.patch_get_utility() def test_enhancement_enable(self, _, _rec, mock_inst, mock_choose, mock_lineage): mock_inst.return_value = self.mockinstaller @@ -1725,10 +1646,10 @@ class EnhanceTest(test_util.ConfigTestCase): self.assertEqual(self.mockinstaller.enable_autohsts.call_args[0][1], ["example.com", "another.tld"]) - @mock.patch('certbot.cert_manager.lineage_for_certname') - @mock.patch('certbot.main.display_ops.choose_values') - @mock.patch('certbot.main.plug_sel.pick_installer') - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.cert_manager.lineage_for_certname') + @mock.patch('certbot._internal.main.display_ops.choose_values') + @mock.patch('certbot._internal.main.plug_sel.pick_installer') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') @test_util.patch_get_utility() def test_enhancement_enable_not_supported(self, _, _rec, mock_inst, mock_choose, mock_lineage): mock_inst.return_value = null.Installer(self.config, "null") @@ -1745,14 +1666,14 @@ class EnhanceTest(test_util.ConfigTestCase): class InstallTest(test_util.ConfigTestCase): - """Tests for certbot.main.install.""" + """Tests for certbot._internal.main.install.""" def setUp(self): super(InstallTest, self).setUp() self.mockinstaller = mock.MagicMock(spec=enhancements.AutoHSTSEnhancement) - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') - @mock.patch('certbot.main.plug_sel.pick_installer') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.main.plug_sel.pick_installer') def test_install_enhancement_not_supported(self, mock_inst, _rec): mock_inst.return_value = null.Installer(self.config, "null") plugins = disco.PluginsRegistry.find_all() @@ -1762,8 +1683,8 @@ class InstallTest(test_util.ConfigTestCase): main.install, self.config, plugins) - @mock.patch('certbot.main.plug_sel.record_chosen_plugins') - @mock.patch('certbot.main.plug_sel.pick_installer') + @mock.patch('certbot._internal.main.plug_sel.record_chosen_plugins') + @mock.patch('certbot._internal.main.plug_sel.pick_installer') def test_install_enhancement_no_certname(self, mock_inst, _rec): mock_inst.return_value = self.mockinstaller plugins = disco.PluginsRegistry.find_all() diff --git a/certbot/tests/notify_test.py b/certbot/tests/notify_test.py index d2af5b001..d6f7d2239 100644 --- a/certbot/tests/notify_test.py +++ b/certbot/tests/notify_test.py @@ -1,4 +1,4 @@ -"""Tests for certbot.notify.""" +"""Tests for certbot._internal.notify.""" import socket import unittest @@ -8,9 +8,9 @@ import mock class NotifyTests(unittest.TestCase): """Tests for the notifier.""" - @mock.patch("certbot.notify.smtplib.LMTP") + @mock.patch("certbot._internal.notify.smtplib.LMTP") def test_smtp_success(self, mock_lmtp): - from certbot.notify import notify + from certbot._internal.notify import notify lmtp_obj = mock.MagicMock() mock_lmtp.return_value = lmtp_obj self.assertTrue(notify("Goose", "auntrhody@example.com", @@ -18,10 +18,10 @@ class NotifyTests(unittest.TestCase): self.assertEqual(lmtp_obj.connect.call_count, 1) self.assertEqual(lmtp_obj.sendmail.call_count, 1) - @mock.patch("certbot.notify.smtplib.LMTP") - @mock.patch("certbot.notify.subprocess.Popen") + @mock.patch("certbot._internal.notify.smtplib.LMTP") + @mock.patch("certbot._internal.notify.subprocess.Popen") def test_smtp_failure(self, mock_popen, mock_lmtp): - from certbot.notify import notify + from certbot._internal.notify import notify lmtp_obj = mock.MagicMock() mock_lmtp.return_value = lmtp_obj lmtp_obj.sendmail.side_effect = socket.error(17) @@ -32,10 +32,10 @@ class NotifyTests(unittest.TestCase): self.assertEqual(lmtp_obj.sendmail.call_count, 1) self.assertEqual(proc.communicate.call_count, 1) - @mock.patch("certbot.notify.smtplib.LMTP") - @mock.patch("certbot.notify.subprocess.Popen") + @mock.patch("certbot._internal.notify.smtplib.LMTP") + @mock.patch("certbot._internal.notify.subprocess.Popen") def test_everything_fails(self, mock_popen, mock_lmtp): - from certbot.notify import notify + from certbot._internal.notify import notify lmtp_obj = mock.MagicMock() mock_lmtp.return_value = lmtp_obj lmtp_obj.sendmail.side_effect = socket.error(17) diff --git a/certbot/tests/ocsp_test.py b/certbot/tests/ocsp_test.py index 55cd24adb..6e4ab52b8 100644 --- a/certbot/tests/ocsp_test.py +++ b/certbot/tests/ocsp_test.py @@ -1,34 +1,55 @@ """Tests for ocsp.py""" # pylint: disable=protected-access - +import contextlib +from datetime import datetime +from datetime import timedelta import unittest +from cryptography import x509 +from cryptography.exceptions import InvalidSignature +from cryptography.exceptions import UnsupportedAlgorithm +from cryptography.hazmat.backends import default_backend +from cryptography.hazmat.primitives import hashes # type: ignore import mock +import pytz from certbot import errors +from certbot.tests import util as test_util + +try: + # Only cryptography>=2.5 has ocsp module + # and signature_hash_algorithm attribute in OCSPResponse class + from cryptography.x509 import ocsp as ocsp_lib # pylint: disable=import-error + getattr(ocsp_lib.OCSPResponse, 'signature_hash_algorithm') +except (ImportError, AttributeError): # pragma: no cover + ocsp_lib = None # type: ignore + out = """Missing = in header key=value ocsp: Use -help for summary. """ -class OCSPTest(unittest.TestCase): +class OCSPTestOpenSSL(unittest.TestCase): + """ + OCSP revokation tests using OpenSSL binary. + """ def setUp(self): - from certbot import ocsp - with mock.patch('certbot.ocsp.Popen') as mock_popen: + from certbot._internal import ocsp + with mock.patch('certbot._internal.ocsp.Popen') as mock_popen: with mock.patch('certbot.util.exe_exists') as mock_exists: mock_communicate = mock.MagicMock() mock_communicate.communicate.return_value = (None, out) mock_popen.return_value = mock_communicate mock_exists.return_value = True - self.checker = ocsp.RevocationChecker() + self.checker = ocsp.RevocationChecker(enforce_openssl_binary_usage=True) def tearDown(self): pass - @mock.patch('certbot.ocsp.logger.info') - @mock.patch('certbot.ocsp.Popen') + @mock.patch('certbot._internal.ocsp.logger.info') + @mock.patch('certbot._internal.ocsp.Popen') @mock.patch('certbot.util.exe_exists') def test_init(self, mock_exists, mock_popen, mock_log): mock_communicate = mock.MagicMock() @@ -36,63 +57,67 @@ class OCSPTest(unittest.TestCase): mock_popen.return_value = mock_communicate mock_exists.return_value = True - from certbot import ocsp - checker = ocsp.RevocationChecker() + from certbot._internal import ocsp + checker = ocsp.RevocationChecker(enforce_openssl_binary_usage=True) self.assertEqual(mock_popen.call_count, 1) self.assertEqual(checker.host_args("x"), ["Host=x"]) mock_communicate.communicate.return_value = (None, out.partition("\n")[2]) - checker = ocsp.RevocationChecker() + checker = ocsp.RevocationChecker(enforce_openssl_binary_usage=True) self.assertEqual(checker.host_args("x"), ["Host", "x"]) self.assertEqual(checker.broken, False) mock_exists.return_value = False mock_popen.call_count = 0 - checker = ocsp.RevocationChecker() + checker = ocsp.RevocationChecker(enforce_openssl_binary_usage=True) self.assertEqual(mock_popen.call_count, 0) self.assertEqual(mock_log.call_count, 1) self.assertEqual(checker.broken, True) - @mock.patch('certbot.ocsp.RevocationChecker.determine_ocsp_server') + @mock.patch('certbot._internal.ocsp._determine_ocsp_server') @mock.patch('certbot.util.run_script') def test_ocsp_revoked(self, mock_run, mock_determine): + now = pytz.UTC.fromutc(datetime.utcnow()) + cert_obj = mock.MagicMock() + cert_obj.cert = "x" + cert_obj.chain = "y" + cert_obj.target_expiry = now + timedelta(hours=2) + self.checker.broken = True mock_determine.return_value = ("", "") - self.assertEqual(self.checker.ocsp_revoked("x", "y"), False) + self.assertEqual(self.checker.ocsp_revoked(cert_obj), False) self.checker.broken = False mock_run.return_value = tuple(openssl_happy[1:]) - self.assertEqual(self.checker.ocsp_revoked("x", "y"), False) + self.assertEqual(self.checker.ocsp_revoked(cert_obj), False) self.assertEqual(mock_run.call_count, 0) mock_determine.return_value = ("http://x.co", "x.co") - self.assertEqual(self.checker.ocsp_revoked("blah.pem", "chain.pem"), False) + self.assertEqual(self.checker.ocsp_revoked(cert_obj), False) mock_run.side_effect = errors.SubprocessError("Unable to load certificate launcher") - self.assertEqual(self.checker.ocsp_revoked("x", "y"), False) + self.assertEqual(self.checker.ocsp_revoked(cert_obj), False) self.assertEqual(mock_run.call_count, 2) + # cert expired + cert_obj.target_expiry = now + mock_determine.return_value = ("", "") + count_before = mock_determine.call_count + self.assertEqual(self.checker.ocsp_revoked(cert_obj), False) + self.assertEqual(mock_determine.call_count, count_before) - @mock.patch('certbot.ocsp.logger.info') - @mock.patch('certbot.util.run_script') - def test_determine_ocsp_server(self, mock_run, mock_info): - uri = "http://ocsp.stg-int-x1.letsencrypt.org/" - host = "ocsp.stg-int-x1.letsencrypt.org" - mock_run.return_value = uri, "" - self.assertEqual(self.checker.determine_ocsp_server("beep"), (uri, host)) - mock_run.return_value = "ftp:/" + host + "/", "" - self.assertEqual(self.checker.determine_ocsp_server("beep"), (None, None)) - self.assertEqual(mock_info.call_count, 1) + def test_determine_ocsp_server(self): + cert_path = test_util.vector_path('ocsp_certificate.pem') - c = "confusion" - mock_run.side_effect = errors.SubprocessError(c) - self.assertEqual(self.checker.determine_ocsp_server("beep"), (None, None)) + from certbot._internal import ocsp + result = ocsp._determine_ocsp_server(cert_path) + self.assertEqual(('http://ocsp.test4.buypass.com', 'ocsp.test4.buypass.com'), result) - @mock.patch('certbot.ocsp.logger') + @mock.patch('certbot._internal.ocsp.logger') @mock.patch('certbot.util.run_script') def test_translate_ocsp(self, mock_run, mock_log): - # pylint: disable=protected-access,star-args + # pylint: disable=protected-access mock_run.return_value = openssl_confused - from certbot import ocsp + from certbot._internal import ocsp self.assertEqual(ocsp._translate_ocsp_query(*openssl_happy), False) self.assertEqual(ocsp._translate_ocsp_query(*openssl_confused), False) self.assertEqual(mock_log.debug.call_count, 1) @@ -112,6 +137,186 @@ class OCSPTest(unittest.TestCase): self.assertEqual(mock_log.info.call_count, 1) +@unittest.skipIf(not ocsp_lib, + reason='This class tests functionalities available only on cryptography>=2.5.0') +class OSCPTestCryptography(unittest.TestCase): + """ + OCSP revokation tests using Cryptography >= 2.4.0 + """ + + def setUp(self): + from certbot._internal import ocsp + self.checker = ocsp.RevocationChecker() + self.cert_path = test_util.vector_path('ocsp_certificate.pem') + self.chain_path = test_util.vector_path('ocsp_issuer_certificate.pem') + self.cert_obj = mock.MagicMock() + self.cert_obj.cert = self.cert_path + self.cert_obj.chain = self.chain_path + now = pytz.UTC.fromutc(datetime.utcnow()) + self.cert_obj.target_expiry = now + timedelta(hours=2) + + @mock.patch('certbot._internal.ocsp._determine_ocsp_server') + @mock.patch('certbot._internal.ocsp._check_ocsp_cryptography') + def test_ensure_cryptography_toggled(self, mock_revoke, mock_determine): + mock_determine.return_value = ('http://example.com', 'example.com') + self.checker.ocsp_revoked(self.cert_obj) + + mock_revoke.assert_called_once_with(self.cert_path, self.chain_path, 'http://example.com') + + def test_revoke(self): + with _ocsp_mock(ocsp_lib.OCSPCertStatus.REVOKED, ocsp_lib.OCSPResponseStatus.SUCCESSFUL): + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertTrue(revoked) + + def test_responder_is_issuer(self): + issuer = x509.load_pem_x509_certificate( + test_util.load_vector('ocsp_issuer_certificate.pem'), default_backend()) + + with _ocsp_mock(ocsp_lib.OCSPCertStatus.REVOKED, + ocsp_lib.OCSPResponseStatus.SUCCESSFUL) as mocks: + mocks['mock_response'].return_value.responder_name = issuer.subject + self.checker.ocsp_revoked(self.cert_obj) + # Here responder and issuer are the same. So only the signature of the OCSP + # response is checked (using the issuer/responder public key). + self.assertEqual(mocks['mock_check'].call_count, 1) + self.assertEqual(mocks['mock_check'].call_args[0][0].public_numbers(), + issuer.public_key().public_numbers()) + + def test_responder_is_authorized_delegate(self): + issuer = x509.load_pem_x509_certificate( + test_util.load_vector('ocsp_issuer_certificate.pem'), default_backend()) + responder = x509.load_pem_x509_certificate( + test_util.load_vector('ocsp_responder_certificate.pem'), default_backend()) + + with _ocsp_mock(ocsp_lib.OCSPCertStatus.REVOKED, + ocsp_lib.OCSPResponseStatus.SUCCESSFUL) as mocks: + self.checker.ocsp_revoked(self.cert_obj) + # Here responder and issuer are not the same. Two signatures will be checked then, + # first to verify the responder cert (using the issuer public key), second to + # to verify the OCSP response itself (using the responder public key). + self.assertEqual(mocks['mock_check'].call_count, 2) + self.assertEqual(mocks['mock_check'].call_args_list[0][0][0].public_numbers(), + issuer.public_key().public_numbers()) + self.assertEqual(mocks['mock_check'].call_args_list[1][0][0].public_numbers(), + responder.public_key().public_numbers()) + + def test_revoke_resiliency(self): + # Server return an invalid HTTP response + with _ocsp_mock(ocsp_lib.OCSPCertStatus.UNKNOWN, ocsp_lib.OCSPResponseStatus.SUCCESSFUL, + http_status_code=400): + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertFalse(revoked) + + # OCSP response in invalid + with _ocsp_mock(ocsp_lib.OCSPCertStatus.UNKNOWN, ocsp_lib.OCSPResponseStatus.UNAUTHORIZED): + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertFalse(revoked) + + # OCSP response is valid, but certificate status is unknown + with _ocsp_mock(ocsp_lib.OCSPCertStatus.UNKNOWN, ocsp_lib.OCSPResponseStatus.SUCCESSFUL): + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertFalse(revoked) + + # The OCSP response says that the certificate is revoked, but certificate + # does not contain the OCSP extension. + with _ocsp_mock(ocsp_lib.OCSPCertStatus.REVOKED, ocsp_lib.OCSPResponseStatus.SUCCESSFUL): + with mock.patch('cryptography.x509.Extensions.get_extension_for_class', + side_effect=x509.ExtensionNotFound( + 'Not found', x509.AuthorityInformationAccessOID.OCSP)): + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertFalse(revoked) + + # OCSP response uses an unsupported signature. + with _ocsp_mock(ocsp_lib.OCSPCertStatus.REVOKED, ocsp_lib.OCSPResponseStatus.SUCCESSFUL, + check_signature_side_effect=UnsupportedAlgorithm('foo')): + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertFalse(revoked) + + # OSCP signature response is invalid. + with _ocsp_mock(ocsp_lib.OCSPCertStatus.REVOKED, ocsp_lib.OCSPResponseStatus.SUCCESSFUL, + check_signature_side_effect=InvalidSignature('foo')): + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertFalse(revoked) + + # Assertion error on OCSP response validity + with _ocsp_mock(ocsp_lib.OCSPCertStatus.REVOKED, ocsp_lib.OCSPResponseStatus.SUCCESSFUL, + check_signature_side_effect=AssertionError('foo')): + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertFalse(revoked) + + # No responder cert in OCSP response + with _ocsp_mock(ocsp_lib.OCSPCertStatus.REVOKED, + ocsp_lib.OCSPResponseStatus.SUCCESSFUL) as mocks: + mocks['mock_response'].return_value.certificates = [] + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertFalse(revoked) + + # Responder cert is not signed by certificate issuer + with _ocsp_mock(ocsp_lib.OCSPCertStatus.REVOKED, + ocsp_lib.OCSPResponseStatus.SUCCESSFUL) as mocks: + cert = mocks['mock_response'].return_value.certificates[0] + mocks['mock_response'].return_value.certificates[0] = mock.Mock( + issuer='fake', subject=cert.subject) + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertFalse(revoked) + + with _ocsp_mock(ocsp_lib.OCSPCertStatus.REVOKED, ocsp_lib.OCSPResponseStatus.SUCCESSFUL): + # This mock is necessary to avoid the first call contained in _determine_ocsp_server + # of the method cryptography.x509.Extensions.get_extension_for_class. + with mock.patch('certbot._internal.ocsp._determine_ocsp_server') as mock_server: + mock_server.return_value = ('https://example.com', 'example.com') + with mock.patch('cryptography.x509.Extensions.get_extension_for_class', + side_effect=x509.ExtensionNotFound( + 'Not found', x509.AuthorityInformationAccessOID.OCSP)): + revoked = self.checker.ocsp_revoked(self.cert_obj) + self.assertFalse(revoked) + + +@contextlib.contextmanager +def _ocsp_mock(certificate_status, response_status, + http_status_code=200, check_signature_side_effect=None): + with mock.patch('certbot._internal.ocsp.ocsp.load_der_ocsp_response') as mock_response: + mock_response.return_value = _construct_mock_ocsp_response( + certificate_status, response_status) + with mock.patch('certbot._internal.ocsp.requests.post') as mock_post: + mock_post.return_value = mock.Mock(status_code=http_status_code) + with mock.patch('certbot._internal.ocsp.crypto_util.verify_signed_payload') \ + as mock_check: + if check_signature_side_effect: + mock_check.side_effect = check_signature_side_effect + yield { + 'mock_response': mock_response, + 'mock_post': mock_post, + 'mock_check': mock_check, + } + + +def _construct_mock_ocsp_response(certificate_status, response_status): + cert = x509.load_pem_x509_certificate( + test_util.load_vector('ocsp_certificate.pem'), default_backend()) + issuer = x509.load_pem_x509_certificate( + test_util.load_vector('ocsp_issuer_certificate.pem'), default_backend()) + responder = x509.load_pem_x509_certificate( + test_util.load_vector('ocsp_responder_certificate.pem'), default_backend()) + builder = ocsp_lib.OCSPRequestBuilder() + builder = builder.add_certificate(cert, issuer, hashes.SHA1()) + request = builder.build() + + return mock.Mock( + response_status=response_status, + certificate_status=certificate_status, + serial_number=request.serial_number, + issuer_key_hash=request.issuer_key_hash, + issuer_name_hash=request.issuer_name_hash, + responder_name=responder.subject, + certificates=[responder], + hash_algorithm=hashes.SHA1(), + next_update=datetime.now() + timedelta(days=1), + this_update=datetime.now() - timedelta(days=1), + signature_algorithm_oid=x509.oid.SignatureAlgorithmOID.RSA_WITH_SHA1, + ) + + # pylint: disable=line-too-long openssl_confused = ("", """ /etc/letsencrypt/live/example.org/cert.pem: good @@ -165,5 +370,6 @@ revoked """, """Response verify OK""") + if __name__ == '__main__': unittest.main() # pragma: no cover diff --git a/certbot/tests/plugins/__init__.py b/certbot/tests/plugins/__init__.py new file mode 100644 index 000000000..3cfcb5008 --- /dev/null +++ b/certbot/tests/plugins/__init__.py @@ -0,0 +1 @@ +"""Certbot Plugins Tests""" diff --git a/certbot/plugins/common_test.py b/certbot/tests/plugins/common_test.py similarity index 75% rename from certbot/plugins/common_test.py rename to certbot/tests/plugins/common_test.py index 103a12499..915a3ae6c 100644 --- a/certbot/plugins/common_test.py +++ b/certbot/tests/plugins/common_test.py @@ -1,34 +1,25 @@ """Tests for certbot.plugins.common.""" import functools -import os import shutil -import tempfile import unittest import josepy as jose import mock -import OpenSSL from acme import challenges - from certbot import achallenges from certbot import crypto_util from certbot import errors - +from certbot.compat import filesystem +from certbot.compat import os from certbot.tests import acme_util from certbot.tests import util as test_util AUTH_KEY = jose.JWKRSA.load(test_util.load_vector("rsa512_key.pem")) -ACHALLS = [ - achallenges.KeyAuthorizationAnnotatedChallenge( - challb=acme_util.chall_to_challb( - challenges.TLSSNI01(token=b'token1'), "pending"), - domain="encryption-example.demo", account_key=AUTH_KEY), - achallenges.KeyAuthorizationAnnotatedChallenge( - challb=acme_util.chall_to_challb( - challenges.TLSSNI01(token=b'token2'), "pending"), - domain="certbot.demo", account_key=AUTH_KEY), -] +ACHALL = achallenges.KeyAuthorizationAnnotatedChallenge( + challb=acme_util.chall_to_challb(challenges.HTTP01(token=b'token1'), + "pending"), + domain="encryption-example.demo", account_key=AUTH_KEY) class NamespaceFunctionsTest(unittest.TestCase): """Tests for certbot.plugins.common.*_namespace functions.""" @@ -95,12 +86,11 @@ class InstallerTest(test_util.ConfigTestCase): def setUp(self): super(InstallerTest, self).setUp() - os.mkdir(self.config.config_dir) + filesystem.mkdir(self.config.config_dir) from certbot.plugins.common import Installer - with mock.patch("certbot.plugins.common.reverter.Reverter"): - self.installer = Installer(config=self.config, - name="Installer") + self.installer = Installer(config=self.config, + name="Installer") self.reverter = self.installer.reverter def test_add_to_real_checkpoint(self): @@ -122,12 +112,11 @@ class InstallerTest(test_util.ConfigTestCase): temporary=temporary) if temporary: - reverter_func = self.reverter.add_to_temp_checkpoint + reverter_func_name = "add_to_temp_checkpoint" else: - reverter_func = self.reverter.add_to_checkpoint + reverter_func_name = "add_to_checkpoint" - self._test_adapted_method( - installer_func, reverter_func, files, save_notes) + self._test_adapted_method(installer_func, reverter_func_name, files, save_notes) def test_finalize_checkpoint(self): self._test_wrapped_method("finalize_checkpoint", "foo") @@ -141,9 +130,6 @@ class InstallerTest(test_util.ConfigTestCase): def test_rollback_checkpoints(self): self._test_wrapped_method("rollback_checkpoints", 42) - def test_view_config_changes(self): - self._test_wrapped_method("view_config_changes") - def _test_wrapped_method(self, name, *args, **kwargs): """Test a wrapped reverter method. @@ -153,46 +139,45 @@ class InstallerTest(test_util.ConfigTestCase): """ installer_func = getattr(self.installer, name) - reverter_func = getattr(self.reverter, name) - self._test_adapted_method( - installer_func, reverter_func, *args, **kwargs) + self._test_adapted_method(installer_func, name, *args, **kwargs) def _test_adapted_method(self, installer_func, - reverter_func, *passed_args, **passed_kwargs): + reverter_func_name, *passed_args, **passed_kwargs): """Test an adapted reverter method :param callable installer_func: installer method to test - :param mock.MagicMock reverter_func: mocked adapated - reverter method + :param str reverter_func_name: name of the method on the + reverter that should be called :param tuple passed_args: positional arguments passed from installer method to the reverter method :param dict passed_kargs: keyword arguments passed from installer method to the reverter method """ - installer_func(*passed_args, **passed_kwargs) - reverter_func.assert_called_once_with(*passed_args, **passed_kwargs) - reverter_func.side_effect = errors.ReverterError - self.assertRaises( - errors.PluginError, installer_func, *passed_args, **passed_kwargs) + with mock.patch.object(self.reverter, reverter_func_name) as reverter_func: + installer_func(*passed_args, **passed_kwargs) + reverter_func.assert_called_once_with(*passed_args, **passed_kwargs) + reverter_func.side_effect = errors.ReverterError + self.assertRaises( + errors.PluginError, installer_func, *passed_args, **passed_kwargs) def test_install_ssl_dhparams(self): self.installer.install_ssl_dhparams() self.assertTrue(os.path.isfile(self.installer.ssl_dhparams)) def _current_ssl_dhparams_hash(self): - from certbot.constants import SSL_DHPARAMS_SRC + from certbot._internal.constants import SSL_DHPARAMS_SRC return crypto_util.sha256sum(SSL_DHPARAMS_SRC) def test_current_file_hash_in_all_hashes(self): - from certbot.constants import ALL_SSL_DHPARAMS_HASHES + from certbot._internal.constants import ALL_SSL_DHPARAMS_HASHES self.assertTrue(self._current_ssl_dhparams_hash() in ALL_SSL_DHPARAMS_HASHES, "Constants.ALL_SSL_DHPARAMS_HASHES must be appended" " with the sha256 hash of self.config.ssl_dhparams when it is updated.") class AddrTest(unittest.TestCase): - """Tests for certbot.client.plugins.common.Addr.""" + """Tests for certbot._internal.client.plugins.common.Addr.""" def setUp(self): from certbot.plugins.common import Addr @@ -282,7 +267,7 @@ class ChallengePerformerTest(unittest.TestCase): self.performer = ChallengePerformer(configurator) def test_add_chall(self): - self.performer.add_chall(ACHALLS[0], 0) + self.performer.add_chall(ACHALL, 0) self.assertEqual(1, len(self.performer.achalls)) self.assertEqual([0], self.performer.indices) @@ -290,58 +275,6 @@ class ChallengePerformerTest(unittest.TestCase): self.assertRaises(NotImplementedError, self.performer.perform) -class TLSSNI01Test(unittest.TestCase): - """Tests for certbot.plugins.common.TLSSNI01.""" - - def setUp(self): - self.tempdir = tempfile.mkdtemp() - configurator = mock.MagicMock() - configurator.config.config_dir = os.path.join(self.tempdir, "config") - configurator.config.work_dir = os.path.join(self.tempdir, "work") - - from certbot.plugins.common import TLSSNI01 - self.sni = TLSSNI01(configurator=configurator) - - def tearDown(self): - shutil.rmtree(self.tempdir) - - def test_setup_challenge_cert(self): - # This is a helper function that can be used for handling - # open context managers more elegantly. It avoids dealing with - # __enter__ and __exit__ calls. - # http://www.voidspace.org.uk/python/mock/helpers.html#mock.mock_open - mock_open, mock_safe_open = mock.mock_open(), mock.mock_open() - - response = challenges.TLSSNI01Response() - achall = mock.MagicMock() - achall.chall.encode.return_value = "token" - key = test_util.load_pyopenssl_private_key("rsa512_key.pem") - achall.response_and_validation.return_value = ( - response, (test_util.load_cert("cert_512.pem"), key)) - - with mock.patch("certbot.plugins.common.open", - mock_open, create=True): - with mock.patch("certbot.plugins.common.util.safe_open", - mock_safe_open): - # pylint: disable=protected-access - self.assertEqual(response, self.sni._setup_challenge_cert( - achall, "randomS1")) - - # pylint: disable=no-member - mock_open.assert_called_once_with(self.sni.get_cert_path(achall), "wb") - mock_open.return_value.write.assert_called_once_with( - test_util.load_vector("cert_512.pem")) - mock_safe_open.assert_called_once_with( - self.sni.get_key_path(achall), "wb", chmod=0o400) - mock_safe_open.return_value.write.assert_called_once_with( - OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, key)) - - def test_get_z_domain(self): - achall = ACHALLS[0] - self.assertEqual(self.sni.get_z_domain(achall), - achall.response(achall.account_key).z_domain.decode("utf-8")) - - class InstallVersionControlledFileTest(test_util.TempDirTestCase): """Tests for certbot.plugins.common.install_version_controlled_file.""" diff --git a/certbot/plugins/disco_test.py b/certbot/tests/plugins/disco_test.py similarity index 93% rename from certbot/plugins/disco_test.py rename to certbot/tests/plugins/disco_test.py index 720b90b16..6d3c7d97e 100644 --- a/certbot/plugins/disco_test.py +++ b/certbot/tests/plugins/disco_test.py @@ -1,4 +1,4 @@ -"""Tests for certbot.plugins.disco.""" +"""Tests for certbot._internal.plugins.disco.""" import functools import string import unittest @@ -11,22 +11,21 @@ import zope.interface from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module from certbot import errors from certbot import interfaces - -from certbot.plugins import standalone -from certbot.plugins import webroot +from certbot._internal.plugins import standalone +from certbot._internal.plugins import webroot EP_SA = pkg_resources.EntryPoint( - "sa", "certbot.plugins.standalone", + "sa", "certbot._internal.plugins.standalone", attrs=("Authenticator",), dist=mock.MagicMock(key="certbot")) EP_WR = pkg_resources.EntryPoint( - "wr", "certbot.plugins.webroot", + "wr", "certbot._internal.plugins.webroot", attrs=("Authenticator",), dist=mock.MagicMock(key="certbot")) class PluginEntryPointTest(unittest.TestCase): - """Tests for certbot.plugins.disco.PluginEntryPoint.""" + """Tests for certbot._internal.plugins.disco.PluginEntryPoint.""" def setUp(self): self.ep1 = pkg_resources.EntryPoint( @@ -40,11 +39,11 @@ class PluginEntryPointTest(unittest.TestCase): self.ep3 = pkg_resources.EntryPoint( "ep3", "a.ep3", dist=mock.MagicMock(key="p3")) - from certbot.plugins.disco import PluginEntryPoint + from certbot._internal.plugins.disco import PluginEntryPoint self.plugin_ep = PluginEntryPoint(EP_SA) def test_entry_point_to_plugin_name(self): - from certbot.plugins.disco import PluginEntryPoint + from certbot._internal.plugins.disco import PluginEntryPoint names = { self.ep1: "p1:ep1", @@ -119,7 +118,7 @@ class PluginEntryPointTest(unittest.TestCase): self.plugin_ep._initialized = plugin = mock.MagicMock() exceptions = zope.interface.exceptions - with mock.patch("certbot.plugins." + with mock.patch("certbot._internal.plugins." "disco.zope.interface") as mock_zope: mock_zope.exceptions = exceptions @@ -183,11 +182,11 @@ class PluginEntryPointTest(unittest.TestCase): class PluginsRegistryTest(unittest.TestCase): - """Tests for certbot.plugins.disco.PluginsRegistry.""" + """Tests for certbot._internal.plugins.disco.PluginsRegistry.""" @classmethod def _create_new_registry(cls, plugins): - from certbot.plugins.disco import PluginsRegistry + from certbot._internal.plugins.disco import PluginsRegistry return PluginsRegistry(plugins) def setUp(self): @@ -198,8 +197,8 @@ class PluginsRegistryTest(unittest.TestCase): self.reg = self._create_new_registry(self.plugins) def test_find_all(self): - from certbot.plugins.disco import PluginsRegistry - with mock.patch("certbot.plugins.disco.pkg_resources") as mock_pkg: + from certbot._internal.plugins.disco import PluginsRegistry + with mock.patch("certbot._internal.plugins.disco.pkg_resources") as mock_pkg: mock_pkg.iter_entry_points.side_effect = [iter([EP_SA]), iter([EP_WR])] plugins = PluginsRegistry.find_all() diff --git a/certbot/plugins/dns_common_lexicon_test.py b/certbot/tests/plugins/dns_common_lexicon_test.py similarity index 100% rename from certbot/plugins/dns_common_lexicon_test.py rename to certbot/tests/plugins/dns_common_lexicon_test.py diff --git a/certbot/plugins/dns_common_test.py b/certbot/tests/plugins/dns_common_test.py similarity index 92% rename from certbot/plugins/dns_common_test.py rename to certbot/tests/plugins/dns_common_test.py index 9b0f0c875..eba3c89d6 100644 --- a/certbot/plugins/dns_common_test.py +++ b/certbot/tests/plugins/dns_common_test.py @@ -2,19 +2,20 @@ import collections import logging -import os import unittest import mock from certbot import errors +from certbot import util +from certbot.compat import os from certbot.display import util as display_util from certbot.plugins import dns_common from certbot.plugins import dns_test_common -from certbot.tests import util +from certbot.tests import util as test_util -class DNSAuthenticatorTest(util.TempDirTestCase, dns_test_common.BaseAuthenticatorTest): +class DNSAuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthenticatorTest): # pylint: disable=protected-access class _FakeDNSAuthenticator(dns_common.DNSAuthenticator): @@ -22,10 +23,6 @@ class DNSAuthenticatorTest(util.TempDirTestCase, dns_test_common.BaseAuthenticat _perform = mock.MagicMock() _cleanup = mock.MagicMock() - def __init__(self, *args, **kwargs): - # pylint: disable=protected-access - super(DNSAuthenticatorTest._FakeDNSAuthenticator, self).__init__(*args, **kwargs) - def more_info(self): # pylint: disable=missing-docstring,no-self-use return 'A fake authenticator for testing.' @@ -54,7 +51,7 @@ class DNSAuthenticatorTest(util.TempDirTestCase, dns_test_common.BaseAuthenticat self.auth._cleanup.assert_called_once_with(dns_test_common.DOMAIN, mock.ANY, mock.ANY) - @util.patch_get_utility() + @test_util.patch_get_utility() def test_prompt(self, mock_get_utility): mock_display = mock_get_utility() mock_display.input.side_effect = ((display_util.OK, "",), @@ -63,14 +60,14 @@ class DNSAuthenticatorTest(util.TempDirTestCase, dns_test_common.BaseAuthenticat self.auth._configure("other_key", "") self.assertEqual(self.auth.config.fake_other_key, "value") - @util.patch_get_utility() + @test_util.patch_get_utility() def test_prompt_canceled(self, mock_get_utility): mock_display = mock_get_utility() mock_display.input.side_effect = ((display_util.CANCEL, "c",),) self.assertRaises(errors.PluginError, self.auth._configure, "other_key", "") - @util.patch_get_utility() + @test_util.patch_get_utility() def test_prompt_file(self, mock_get_utility): path = os.path.join(self.tempdir, 'file.ini') open(path, "wb").close() @@ -84,7 +81,7 @@ class DNSAuthenticatorTest(util.TempDirTestCase, dns_test_common.BaseAuthenticat self.auth._configure_file("file_path", "") self.assertEqual(self.auth.config.fake_file_path, path) - @util.patch_get_utility() + @test_util.patch_get_utility() def test_prompt_file_canceled(self, mock_get_utility): mock_display = mock_get_utility() mock_display.directory_select.side_effect = ((display_util.CANCEL, "c",),) @@ -100,7 +97,7 @@ class DNSAuthenticatorTest(util.TempDirTestCase, dns_test_common.BaseAuthenticat self.assertEqual(credentials.conf("test"), "value") - @util.patch_get_utility() + @test_util.patch_get_utility() def test_prompt_credentials(self, mock_get_utility): bad_path = os.path.join(self.tempdir, 'bad-file.ini') dns_test_common.write({"fake_other": "other_value"}, bad_path) @@ -120,7 +117,7 @@ class DNSAuthenticatorTest(util.TempDirTestCase, dns_test_common.BaseAuthenticat self.assertEqual(credentials.conf("test"), "value") -class CredentialsConfigurationTest(util.TempDirTestCase): +class CredentialsConfigurationTest(test_util.TempDirTestCase): class _MockLoggingHandler(logging.Handler): messages = None @@ -154,14 +151,14 @@ class CredentialsConfigurationTest(util.TempDirTestCase): dns_common.logger.addHandler(log) path = os.path.join(self.tempdir, 'too-permissive-file.ini') - open(path, "wb").close() + util.safe_open(path, "wb", 0o744).close() dns_common.CredentialsConfiguration(path) self.assertEqual(1, len([_ for _ in log.messages['warning'] if _.startswith("Unsafe")])) -class CredentialsConfigurationRequireTest(util.TempDirTestCase): +class CredentialsConfigurationRequireTest(test_util.TempDirTestCase): def setUp(self): super(CredentialsConfigurationRequireTest, self).setUp() diff --git a/certbot/plugins/enhancements_test.py b/certbot/tests/plugins/enhancements_test.py similarity index 89% rename from certbot/plugins/enhancements_test.py rename to certbot/tests/plugins/enhancements_test.py index 22f6f54e9..05fbc5028 100644 --- a/certbot/plugins/enhancements_test.py +++ b/certbot/tests/plugins/enhancements_test.py @@ -1,10 +1,10 @@ """Tests for new style enhancements""" import unittest + import mock +from certbot._internal.plugins import null from certbot.plugins import enhancements -from certbot.plugins import null - import certbot.tests.util as test_util @@ -37,12 +37,10 @@ class EnhancementTest(test_util.ConfigTestCase): self.assertTrue([i for i in enabled if i["name"] == "somethingelse"]) def test_are_requested(self): - self.assertEqual( - len([i for i in enhancements.enabled_enhancements(self.config)]), 0) + self.assertEqual(len(list(enhancements.enabled_enhancements(self.config))), 0) self.assertFalse(enhancements.are_requested(self.config)) self.config.auto_hsts = True - self.assertEqual( - len([i for i in enhancements.enabled_enhancements(self.config)]), 1) + self.assertEqual(len(list(enhancements.enabled_enhancements(self.config))), 1) self.assertTrue(enhancements.are_requested(self.config)) def test_are_supported(self): diff --git a/certbot/plugins/manual_test.py b/certbot/tests/plugins/manual_test.py similarity index 64% rename from certbot/plugins/manual_test.py rename to certbot/tests/plugins/manual_test.py index 0938e8a7d..bd11a9538 100644 --- a/certbot/plugins/manual_test.py +++ b/certbot/tests/plugins/manual_test.py @@ -1,31 +1,29 @@ -"""Tests for certbot.plugins.manual""" -import os +"""Tests for certbot._internal.plugins.manual""" +import sys import unittest -import six import mock -import sys +import six from acme import challenges - from certbot import errors - +from certbot.compat import filesystem +from certbot.compat import os from certbot.tests import acme_util from certbot.tests import util as test_util class AuthenticatorTest(test_util.TempDirTestCase): - """Tests for certbot.plugins.manual.Authenticator.""" + """Tests for certbot._internal.plugins.manual.Authenticator.""" def setUp(self): super(AuthenticatorTest, self).setUp() self.http_achall = acme_util.HTTP01_A self.dns_achall = acme_util.DNS01_A self.dns_achall_2 = acme_util.DNS01_A_2 - self.tls_sni_achall = acme_util.TLSSNI01_A - self.achalls = [self.http_achall, self.dns_achall, self.tls_sni_achall, self.dns_achall_2] + self.achalls = [self.http_achall, self.dns_achall, self.dns_achall_2] for d in ["config_dir", "work_dir", "in_progress"]: - os.mkdir(os.path.join(self.tempdir, d)) + filesystem.mkdir(os.path.join(self.tempdir, d)) # "backup_dir" and "temp_checkpoint_dir" get created in # certbot.util.make_or_verify_dir() during the Reverter # initialization. @@ -38,10 +36,9 @@ class AuthenticatorTest(test_util.TempDirTestCase): backup_dir=os.path.join(self.tempdir, "backup_dir"), temp_checkpoint_dir=os.path.join( self.tempdir, "temp_checkpoint_dir"), - in_progress_dir=os.path.join(self.tempdir, "in_progess"), - tls_sni_01_port=5001) + in_progress_dir=os.path.join(self.tempdir, "in_progess")) - from certbot.plugins.manual import Authenticator + from certbot._internal.plugins.manual import Authenticator self.auth = Authenticator(self.config, name='manual') def test_prepare_no_hook_noninteractive(self): @@ -58,9 +55,7 @@ class AuthenticatorTest(test_util.TempDirTestCase): def test_get_chall_pref(self): self.assertEqual(self.auth.get_chall_pref('example.org'), - [challenges.HTTP01, - challenges.DNS01, - challenges.TLSSNI01]) + [challenges.HTTP01, challenges.DNS01]) @test_util.patch_get_utility() def test_ip_logging_not_ok(self, mock_get_utility): @@ -77,20 +72,15 @@ class AuthenticatorTest(test_util.TempDirTestCase): self.config.manual_public_ip_logging_ok = True self.config.manual_auth_hook = ( '{0} -c "from __future__ import print_function;' - 'import os; print(os.environ.get(\'CERTBOT_DOMAIN\'));' + 'from certbot.compat import os; print(os.environ.get(\'CERTBOT_DOMAIN\'));' 'print(os.environ.get(\'CERTBOT_TOKEN\', \'notoken\'));' - 'print(os.environ.get(\'CERTBOT_CERT_PATH\', \'nocert\'));' - 'print(os.environ.get(\'CERTBOT_KEY_PATH\', \'nokey\'));' - 'print(os.environ.get(\'CERTBOT_SNI_DOMAIN\', \'nosnidomain\'));' 'print(os.environ.get(\'CERTBOT_VALIDATION\', \'novalidation\'));"' .format(sys.executable)) - dns_expected = '{0}\n{1}\n{2}\n{3}\n{4}\n{5}'.format( + dns_expected = '{0}\n{1}\n{2}'.format( self.dns_achall.domain, 'notoken', - 'nocert', 'nokey', 'nosnidomain', self.dns_achall.validation(self.dns_achall.account_key)) - http_expected = '{0}\n{1}\n{2}\n{3}\n{4}\n{5}'.format( + http_expected = '{0}\n{1}\n{2}'.format( self.http_achall.domain, self.http_achall.chall.encode('token'), - 'nocert', 'nokey', 'nosnidomain', self.http_achall.validation(self.http_achall.account_key)) self.assertEqual( @@ -102,17 +92,6 @@ class AuthenticatorTest(test_util.TempDirTestCase): self.assertEqual( self.auth.env[self.http_achall]['CERTBOT_AUTH_OUTPUT'], http_expected) - # tls_sni_01 challenge must be perform()ed above before we can - # get the cert_path and key_path. - tls_sni_expected = '{0}\n{1}\n{2}\n{3}\n{4}\n{5}'.format( - self.tls_sni_achall.domain, 'notoken', - self.auth.tls_sni_01.get_cert_path(self.tls_sni_achall), - self.auth.tls_sni_01.get_key_path(self.tls_sni_achall), - self.auth.tls_sni_01.get_z_domain(self.tls_sni_achall), - 'novalidation') - self.assertEqual( - self.auth.env[self.tls_sni_achall]['CERTBOT_AUTH_OUTPUT'], - tls_sni_expected) @test_util.patch_get_utility() def test_manual_perform(self, mock_get_utility): @@ -122,19 +101,14 @@ class AuthenticatorTest(test_util.TempDirTestCase): [achall.response(achall.account_key) for achall in self.achalls]) for i, (args, kwargs) in enumerate(mock_get_utility().notification.call_args_list): achall = self.achalls[i] - if isinstance(achall.chall, challenges.TLSSNI01): - self.assertTrue( - self.auth.tls_sni_01.get_cert_path( - self.tls_sni_achall) in args[0]) - else: - self.assertTrue( - achall.validation(achall.account_key) in args[0]) + self.assertTrue( + achall.validation(achall.account_key) in args[0]) self.assertFalse(kwargs['wrap']) - @test_util.broken_on_windows def test_cleanup(self): self.config.manual_public_ip_logging_ok = True - self.config.manual_auth_hook = 'echo foo;' + self.config.manual_auth_hook = ('{0} -c "import sys; sys.stdout.write(\'foo\')"' + .format(sys.executable)) self.config.manual_cleanup_hook = '# cleanup' self.auth.perform(self.achalls) @@ -142,8 +116,7 @@ class AuthenticatorTest(test_util.TempDirTestCase): self.auth.cleanup([achall]) self.assertEqual(os.environ['CERTBOT_AUTH_OUTPUT'], 'foo') self.assertEqual(os.environ['CERTBOT_DOMAIN'], achall.domain) - if (isinstance(achall.chall, challenges.HTTP01) or - isinstance(achall.chall, challenges.DNS01)): + if isinstance(achall.chall, (challenges.HTTP01, challenges.DNS01)): self.assertEqual( os.environ['CERTBOT_VALIDATION'], achall.validation(achall.account_key)) @@ -153,18 +126,6 @@ class AuthenticatorTest(test_util.TempDirTestCase): achall.chall.encode('token')) else: self.assertFalse('CERTBOT_TOKEN' in os.environ) - if isinstance(achall.chall, challenges.TLSSNI01): - self.assertEqual( - os.environ['CERTBOT_CERT_PATH'], - self.auth.tls_sni_01.get_cert_path(achall)) - self.assertEqual( - os.environ['CERTBOT_KEY_PATH'], - self.auth.tls_sni_01.get_key_path(achall)) - self.assertFalse( - os.path.exists(os.environ['CERTBOT_CERT_PATH'])) - self.assertFalse( - os.path.exists(os.environ['CERTBOT_KEY_PATH'])) - if __name__ == '__main__': diff --git a/certbot/plugins/null_test.py b/certbot/tests/plugins/null_test.py similarity index 73% rename from certbot/plugins/null_test.py rename to certbot/tests/plugins/null_test.py index d5de33fb3..db0213813 100644 --- a/certbot/plugins/null_test.py +++ b/certbot/tests/plugins/null_test.py @@ -1,15 +1,15 @@ -"""Tests for certbot.plugins.null.""" +"""Tests for certbot._internal.plugins.null.""" import unittest -import six import mock +import six class InstallerTest(unittest.TestCase): - """Tests for certbot.plugins.null.Installer.""" + """Tests for certbot._internal.plugins.null.Installer.""" def setUp(self): - from certbot.plugins.null import Installer + from certbot._internal.plugins.null import Installer self.installer = Installer(config=mock.MagicMock(), name="null") def test_it(self): diff --git a/certbot/plugins/selection_test.py b/certbot/tests/plugins/selection_test.py similarity index 82% rename from certbot/plugins/selection_test.py rename to certbot/tests/plugins/selection_test.py index 5f8e42516..ac846af7b 100644 --- a/certbot/plugins/selection_test.py +++ b/certbot/tests/plugins/selection_test.py @@ -1,50 +1,49 @@ """Tests for letsencrypt.plugins.selection""" -import os import sys import unittest import mock import zope.component +from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module from certbot import errors from certbot import interfaces - -from acme.magic_typing import List # pylint: disable=unused-import, no-name-in-module +from certbot._internal.plugins.disco import PluginsRegistry +from certbot.compat import os from certbot.display import util as display_util -from certbot.plugins.disco import PluginsRegistry from certbot.tests import util as test_util class ConveniencePickPluginTest(unittest.TestCase): - """Tests for certbot.plugins.selection.pick_*.""" + """Tests for certbot._internal.plugins.selection.pick_*.""" def _test(self, fun, ifaces): config = mock.Mock() default = mock.Mock() plugins = mock.Mock() - with mock.patch("certbot.plugins.selection.pick_plugin") as mock_p: + with mock.patch("certbot._internal.plugins.selection.pick_plugin") as mock_p: mock_p.return_value = "foo" self.assertEqual("foo", fun(config, default, plugins, "Question?")) mock_p.assert_called_once_with( config, default, plugins, "Question?", ifaces) def test_authenticator(self): - from certbot.plugins.selection import pick_authenticator + from certbot._internal.plugins.selection import pick_authenticator self._test(pick_authenticator, (interfaces.IAuthenticator,)) def test_installer(self): - from certbot.plugins.selection import pick_installer + from certbot._internal.plugins.selection import pick_installer self._test(pick_installer, (interfaces.IInstaller,)) def test_configurator(self): - from certbot.plugins.selection import pick_configurator + from certbot._internal.plugins.selection import pick_configurator self._test(pick_configurator, (interfaces.IAuthenticator, interfaces.IInstaller)) class PickPluginTest(unittest.TestCase): - """Tests for certbot.plugins.selection.pick_plugin.""" + """Tests for certbot._internal.plugins.selection.pick_plugin.""" def setUp(self): self.config = mock.Mock(noninteractive_mode=False) @@ -54,7 +53,7 @@ class PickPluginTest(unittest.TestCase): self.ifaces = [] # type: List[interfaces.IPlugin] def _call(self): - from certbot.plugins.selection import pick_plugin + from certbot._internal.plugins.selection import pick_plugin return pick_plugin(self.config, self.default, self.reg, self.question, self.ifaces) @@ -95,7 +94,7 @@ class PickPluginTest(unittest.TestCase): "bar": plugin_ep, "baz": plugin_ep, } - with mock.patch("certbot.plugins.selection.choose_plugin") as mock_choose: + with mock.patch("certbot._internal.plugins.selection.choose_plugin") as mock_choose: mock_choose.return_value = plugin_ep self.assertEqual("foo", self._call()) mock_choose.assert_called_once_with( @@ -107,13 +106,13 @@ class PickPluginTest(unittest.TestCase): "baz": None, } - with mock.patch("certbot.plugins.selection.choose_plugin") as mock_choose: + with mock.patch("certbot._internal.plugins.selection.choose_plugin") as mock_choose: mock_choose.return_value = None self.assertTrue(self._call() is None) class ChoosePluginTest(unittest.TestCase): - """Tests for certbot.plugins.selection.choose_plugin.""" + """Tests for certbot._internal.plugins.selection.choose_plugin.""" def setUp(self): zope.component.provideUtility(display_util.FileDisplay(sys.stdout, @@ -130,17 +129,17 @@ class ChoosePluginTest(unittest.TestCase): ] def _call(self): - from certbot.plugins.selection import choose_plugin + from certbot._internal.plugins.selection import choose_plugin return choose_plugin(self.plugins, "Question?") - @test_util.patch_get_utility("certbot.plugins.selection.z_util") + @test_util.patch_get_utility("certbot._internal.plugins.selection.z_util") def test_selection(self, mock_util): mock_util().menu.side_effect = [(display_util.OK, 0), (display_util.OK, 1)] self.assertEqual(self.mock_stand, self._call()) self.assertEqual(mock_util().notification.call_count, 1) - @test_util.patch_get_utility("certbot.plugins.selection.z_util") + @test_util.patch_get_utility("certbot._internal.plugins.selection.z_util") def test_more_info(self, mock_util): mock_util().menu.side_effect = [ (display_util.OK, 1), @@ -148,12 +147,12 @@ class ChoosePluginTest(unittest.TestCase): self.assertEqual(self.mock_stand, self._call()) - @test_util.patch_get_utility("certbot.plugins.selection.z_util") + @test_util.patch_get_utility("certbot._internal.plugins.selection.z_util") def test_no_choice(self, mock_util): mock_util().menu.return_value = (display_util.CANCEL, 0) self.assertTrue(self._call() is None) - @test_util.patch_get_utility("certbot.plugins.selection.z_util") + @test_util.patch_get_utility("certbot._internal.plugins.selection.z_util") def test_new_interaction_avoidance(self, mock_util): mock_nginx = mock.Mock( description_with_name="n", misconfigured=False) @@ -174,7 +173,7 @@ class ChoosePluginTest(unittest.TestCase): self.assertTrue("default" in mock_util().menu.call_args[1]) class GetUnpreparedInstallerTest(test_util.ConfigTestCase): - """Tests for certbot.plugins.selection.get_unprepared_installer.""" + """Tests for certbot._internal.plugins.selection.get_unprepared_installer.""" def setUp(self): super(GetUnpreparedInstallerTest, self).setUp() @@ -192,7 +191,7 @@ class GetUnpreparedInstallerTest(test_util.ConfigTestCase): }) def _call(self): - from certbot.plugins.selection import get_unprepared_installer + from certbot._internal.plugins.selection import get_unprepared_installer return get_unprepared_installer(self.config, self.plugins) def test_no_installer_defined(self): diff --git a/certbot/plugins/standalone_test.py b/certbot/tests/plugins/standalone_test.py similarity index 64% rename from certbot/plugins/standalone_test.py rename to certbot/tests/plugins/standalone_test.py index 9b741fc6f..5d9ff5244 100644 --- a/certbot/plugins/standalone_test.py +++ b/certbot/tests/plugins/standalone_test.py @@ -1,32 +1,30 @@ -"""Tests for certbot.plugins.standalone.""" -import argparse -import socket -import unittest +"""Tests for certbot._internal.plugins.standalone.""" # https://github.com/python/typeshed/blob/master/stdlib/2and3/socket.pyi +import socket from socket import errno as socket_errors # type: ignore +import unittest import josepy as jose import mock -import six - import OpenSSL.crypto # pylint: disable=unused-import +import six from acme import challenges from acme import standalone as acme_standalone # pylint: disable=unused-import -from acme.magic_typing import Dict, Tuple, Set # pylint: disable=unused-import, no-name-in-module - +from acme.magic_typing import Dict # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Set # pylint: disable=unused-import, no-name-in-module +from acme.magic_typing import Tuple # pylint: disable=unused-import, no-name-in-module from certbot import achallenges from certbot import errors - from certbot.tests import acme_util from certbot.tests import util as test_util class ServerManagerTest(unittest.TestCase): - """Tests for certbot.plugins.standalone.ServerManager.""" + """Tests for certbot._internal.plugins.standalone.ServerManager.""" def setUp(self): - from certbot.plugins.standalone import ServerManager + from certbot._internal.plugins.standalone import ServerManager self.certs = {} # type: Dict[bytes, Tuple[OpenSSL.crypto.PKey, OpenSSL.crypto.X509]] self.http_01_resources = {} \ # type: Set[acme_standalone.HTTP01RequestHandler.HTTP01Resource] @@ -39,20 +37,17 @@ class ServerManagerTest(unittest.TestCase): def _test_run_stop(self, challenge_type): server = self.mgr.run(port=0, challenge_type=challenge_type) - port = server.getsocknames()[0][1] # pylint: disable=no-member + port = server.getsocknames()[0][1] self.assertEqual(self.mgr.running(), {port: server}) self.mgr.stop(port=port) self.assertEqual(self.mgr.running(), {}) - def test_run_stop_tls_sni_01(self): - self._test_run_stop(challenges.TLSSNI01) - def test_run_stop_http_01(self): self._test_run_stop(challenges.HTTP01) def test_run_idempotent(self): server = self.mgr.run(port=0, challenge_type=challenges.HTTP01) - port = server.getsocknames()[0][1] # pylint: disable=no-member + port = server.getsocknames()[0][1] server2 = self.mgr.run(port=port, challenge_type=challenges.HTTP01) self.assertEqual(self.mgr.running(), {port: server}) self.assertTrue(server is server2) @@ -76,46 +71,6 @@ class ServerManagerTest(unittest.TestCase): maybe_another_server.close() -class SupportedChallengesActionTest(unittest.TestCase): - """Tests for plugins.standalone.SupportedChallengesAction.""" - - def _call(self, value): - with mock.patch("certbot.plugins.standalone.logger") as mock_logger: - # stderr is mocked to prevent potential argparse error - # output from cluttering test output - with mock.patch("sys.stderr"): - config = self.parser.parse_args([self.flag, value]) - - self.assertTrue(mock_logger.warning.called) - return getattr(config, self.dest) - - def setUp(self): - self.flag = "--standalone-supported-challenges" - self.dest = self.flag[2:].replace("-", "_") - self.parser = argparse.ArgumentParser() - - from certbot.plugins.standalone import SupportedChallengesAction - self.parser.add_argument(self.flag, action=SupportedChallengesAction) - - def test_correct(self): - self.assertEqual("tls-sni-01", self._call("tls-sni-01")) - self.assertEqual("http-01", self._call("http-01")) - self.assertEqual("tls-sni-01,http-01", self._call("tls-sni-01,http-01")) - self.assertEqual("http-01,tls-sni-01", self._call("http-01,tls-sni-01")) - - def test_unrecognized(self): - assert "foo" not in challenges.Challenge.TYPES - self.assertRaises(SystemExit, self._call, "foo") - - def test_not_subset(self): - self.assertRaises(SystemExit, self._call, "dns") - - def test_dvsni(self): - self.assertEqual("tls-sni-01", self._call("dvsni")) - self.assertEqual("http-01,tls-sni-01", self._call("http-01,dvsni")) - self.assertEqual("tls-sni-01,http-01", self._call("dvsni,http-01")) - - def get_open_port(): """Gets an open port number from the OS.""" open_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) @@ -126,37 +81,21 @@ def get_open_port(): class AuthenticatorTest(unittest.TestCase): - """Tests for certbot.plugins.standalone.Authenticator.""" + """Tests for certbot._internal.plugins.standalone.Authenticator.""" def setUp(self): - from certbot.plugins.standalone import Authenticator + from certbot._internal.plugins.standalone import Authenticator - self.config = mock.MagicMock( - tls_sni_01_port=get_open_port(), http01_port=get_open_port(), - standalone_supported_challenges="tls-sni-01,http-01") + self.config = mock.MagicMock(http01_port=get_open_port()) self.auth = Authenticator(self.config, name="standalone") self.auth.servers = mock.MagicMock() - def test_supported_challenges(self): - self.assertEqual(self.auth.supported_challenges, - [challenges.TLSSNI01, challenges.HTTP01]) - - def test_supported_challenges_configured(self): - self.config.standalone_supported_challenges = "tls-sni-01" - self.assertEqual(self.auth.supported_challenges, - [challenges.TLSSNI01]) - def test_more_info(self): self.assertTrue(isinstance(self.auth.more_info(), six.string_types)) def test_get_chall_pref(self): self.assertEqual(self.auth.get_chall_pref(domain=None), - [challenges.TLSSNI01, challenges.HTTP01]) - - def test_get_chall_pref_configured(self): - self.config.standalone_supported_challenges = "tls-sni-01" - self.assertEqual(self.auth.get_chall_pref(domain=None), - [challenges.TLSSNI01]) + [challenges.HTTP01]) def test_perform(self): achalls = self._get_achalls() @@ -212,10 +151,8 @@ class AuthenticatorTest(unittest.TestCase): key = jose.JWK.load(test_util.load_vector('rsa512_key.pem')) http_01 = achallenges.KeyAuthorizationAnnotatedChallenge( challb=acme_util.HTTP01_P, domain=domain, account_key=key) - tls_sni_01 = achallenges.KeyAuthorizationAnnotatedChallenge( - challb=acme_util.TLSSNI01_P, domain=domain, account_key=key) - return [http_01, tls_sni_01] + return [http_01] def test_cleanup(self): self.auth.servers.running.return_value = { @@ -243,5 +180,6 @@ class AuthenticatorTest(unittest.TestCase): "server1": set(), "server2": set([])}) self.auth.servers.stop.assert_called_with(2) + if __name__ == "__main__": unittest.main() # pragma: no cover diff --git a/certbot/plugins/storage_test.py b/certbot/tests/plugins/storage_test.py similarity index 95% rename from certbot/plugins/storage_test.py rename to certbot/tests/plugins/storage_test.py index 8d96f400c..e9ca2007f 100644 --- a/certbot/plugins/storage_test.py +++ b/certbot/tests/plugins/storage_test.py @@ -1,21 +1,23 @@ """Tests for certbot.plugins.storage.PluginStorage""" import json -import mock -import os import unittest -from certbot import errors +import mock +from certbot import errors +from certbot.compat import filesystem +from certbot.compat import os from certbot.plugins import common from certbot.tests import util as test_util + class PluginStorageTest(test_util.ConfigTestCase): """Test for certbot.plugins.storage.PluginStorage""" def setUp(self): super(PluginStorageTest, self).setUp() self.plugin_cls = common.Installer - os.mkdir(self.config.config_dir) + filesystem.mkdir(self.config.config_dir) with mock.patch("certbot.reverter.util"): self.plugin = self.plugin_cls(config=self.config, name="mockplugin") @@ -29,7 +31,7 @@ class PluginStorageTest(test_util.ConfigTestCase): self.plugin.storage.storagepath = os.path.join(self.config.config_dir, ".pluginstorage.json") with mock.patch("six.moves.builtins.open", mock_open): - with mock.patch('os.path.isfile', return_value=True): + with mock.patch('certbot.compat.os.path.isfile', return_value=True): with mock.patch("certbot.reverter.util"): self.assertRaises(errors.PluginStorageError, self.plugin.storage._load) # pylint: disable=protected-access @@ -71,7 +73,7 @@ class PluginStorageTest(test_util.ConfigTestCase): def test_save_errors_unable_to_write_file(self): mock_open = mock.mock_open() mock_open.side_effect = IOError - with mock.patch("os.open", mock_open): + with mock.patch("certbot.compat.filesystem.open", mock_open): with mock.patch("certbot.plugins.storage.logger.error") as mock_log: self.plugin.storage._data = {"valid": "data"} # pylint: disable=protected-access self.plugin.storage._initialized = True # pylint: disable=protected-access diff --git a/certbot/plugins/util_test.py b/certbot/tests/plugins/util_test.py similarity index 68% rename from certbot/plugins/util_test.py rename to certbot/tests/plugins/util_test.py index 8ecd380b8..c41e55222 100644 --- a/certbot/plugins/util_test.py +++ b/certbot/tests/plugins/util_test.py @@ -1,9 +1,11 @@ """Tests for certbot.plugins.util.""" -import os import unittest import mock +from certbot.compat import os + + class GetPrefixTest(unittest.TestCase): """Tests for certbot.plugins.get_prefixes.""" def test_get_prefix(self): @@ -14,6 +16,7 @@ class GetPrefixTest(unittest.TestCase): self.assertEqual(get_prefixes('/'), [os.path.normpath('/')]) self.assertEqual(get_prefixes('a'), ['a']) + class PathSurgeryTest(unittest.TestCase): """Tests for certbot.plugins.path_surgery.""" @@ -27,13 +30,15 @@ class PathSurgeryTest(unittest.TestCase): self.assertEqual(path_surgery("eg"), True) self.assertEqual(mock_debug.call_count, 0) self.assertEqual(os.environ["PATH"], all_path["PATH"]) - no_path = {"PATH": "/tmp/"} - with mock.patch.dict('os.environ', no_path): - path_surgery("thingy") - self.assertEqual(mock_debug.call_count, 2) - self.assertTrue("Failed to find" in mock_debug.call_args[0][0]) - self.assertTrue("/usr/local/bin" in os.environ["PATH"]) - self.assertTrue("/tmp" in os.environ["PATH"]) + if os.name != 'nt': + # This part is specific to Linux since on Windows no PATH surgery is ever done. + no_path = {"PATH": "/tmp/"} + with mock.patch.dict('os.environ', no_path): + path_surgery("thingy") + self.assertEqual(mock_debug.call_count, 2 if os.name != 'nt' else 1) + self.assertTrue("Failed to find" in mock_debug.call_args[0][0]) + self.assertTrue("/usr/local/bin" in os.environ["PATH"]) + self.assertTrue("/tmp" in os.environ["PATH"]) if __name__ == "__main__": diff --git a/certbot/plugins/webroot_test.py b/certbot/tests/plugins/webroot_test.py similarity index 81% rename from certbot/plugins/webroot_test.py rename to certbot/tests/plugins/webroot_test.py index 5303fe4da..fade12bb1 100644 --- a/certbot/plugins/webroot_test.py +++ b/certbot/tests/plugins/webroot_test.py @@ -1,11 +1,10 @@ -"""Tests for certbot.plugins.webroot.""" +"""Tests for certbot._internal.plugins.webroot.""" from __future__ import print_function import argparse import errno import json -import os import shutil import tempfile import unittest @@ -15,28 +14,32 @@ import mock import six from acme import challenges - from certbot import achallenges -from certbot import compat from certbot import errors +from certbot.compat import filesystem +from certbot.compat import os from certbot.display import util as display_util - from certbot.tests import acme_util from certbot.tests import util as test_util - KEY = jose.JWKRSA.load(test_util.load_vector("rsa512_key.pem")) class AuthenticatorTest(unittest.TestCase): - """Tests for certbot.plugins.webroot.Authenticator.""" + """Tests for certbot._internal.plugins.webroot.Authenticator.""" achall = achallenges.KeyAuthorizationAnnotatedChallenge( challb=acme_util.HTTP01_P, domain="thing.com", account_key=KEY) def setUp(self): - from certbot.plugins.webroot import Authenticator - self.path = tempfile.mkdtemp() + from certbot._internal.plugins.webroot import Authenticator + # On Linux directories created by tempfile.mkdtemp inherit their permissions from their + # parent directory. So the actual permissions are inconsistent over various tests env. + # To circumvent this, a dedicated sub-workspace is created under the workspace, using + # filesystem.mkdir to get consistent permissions. + self.workspace = tempfile.mkdtemp() + self.path = os.path.join(self.workspace, 'webroot') + filesystem.mkdir(self.path) self.partial_root_challenge_path = os.path.join( self.path, ".well-known") self.root_challenge_path = os.path.join( @@ -134,22 +137,20 @@ class AuthenticatorTest(unittest.TestCase): permission_canary = os.path.join(self.path, "rnd") with open(permission_canary, "w") as f: f.write("thingimy") - os.chmod(self.path, 0o000) + filesystem.chmod(self.path, 0o000) try: open(permission_canary, "r") print("Warning, running tests as root skips permissions tests...") except IOError: # ok, permissions work, test away... self.assertRaises(errors.PluginError, self.auth.perform, []) - os.chmod(self.path, 0o700) + filesystem.chmod(self.path, 0o700) - @test_util.skip_on_windows('On Windows, there is no chown.') - @mock.patch("certbot.plugins.webroot.os.chown") - def test_failed_chown(self, mock_chown): - mock_chown.side_effect = OSError(errno.EACCES, "msg") + @mock.patch("certbot._internal.plugins.webroot.filesystem.copy_ownership_and_apply_mode") + def test_failed_chown(self, mock_ownership): + mock_ownership.side_effect = OSError(errno.EACCES, "msg") self.auth.perform([self.achall]) # exception caught and logged - @test_util.patch_get_utility() def test_perform_new_webroot_not_in_map(self, mock_get_utility): new_webroot = tempfile.mkdtemp() @@ -171,20 +172,15 @@ class AuthenticatorTest(unittest.TestCase): # Remove exec bit from permission check, so that it # matches the file self.auth.perform([self.achall]) - self.assertTrue(compat.compare_file_modes(os.stat(self.validation_path).st_mode, 0o644)) + self.assertTrue(filesystem.check_mode(self.validation_path, 0o644)) # Check permissions of the directories - for dirpath, dirnames, _ in os.walk(self.path): for directory in dirnames: full_path = os.path.join(dirpath, directory) - self.assertTrue(compat.compare_file_modes(os.stat(full_path).st_mode, 0o755)) + self.assertTrue(filesystem.check_mode(full_path, 0o755)) - parent_gid = os.stat(self.path).st_gid - parent_uid = os.stat(self.path).st_uid - - self.assertEqual(os.stat(self.validation_path).st_gid, parent_gid) - self.assertEqual(os.stat(self.validation_path).st_uid, parent_uid) + self.assertTrue(filesystem.has_same_ownership(self.validation_path, self.path)) def test_perform_cleanup(self): self.auth.prepare() @@ -204,7 +200,7 @@ class AuthenticatorTest(unittest.TestCase): self.assertFalse(os.path.exists(self.partial_root_challenge_path)) def test_perform_cleanup_existing_dirs(self): - os.mkdir(self.partial_root_challenge_path) + filesystem.mkdir(self.partial_root_challenge_path) self.auth.prepare() self.auth.perform([self.achall]) self.auth.cleanup([self.achall]) @@ -220,7 +216,7 @@ class AuthenticatorTest(unittest.TestCase): domain="thing.com", account_key=KEY) bingo_validation_path = "YmluZ28" - os.mkdir(self.partial_root_challenge_path) + filesystem.mkdir(self.partial_root_challenge_path) self.auth.prepare() self.auth.perform([bingo_achall, self.achall]) @@ -236,7 +232,7 @@ class AuthenticatorTest(unittest.TestCase): self.auth.perform([self.achall]) leftover_path = os.path.join(self.root_challenge_path, 'leftover') - os.mkdir(leftover_path) + filesystem.mkdir(leftover_path) self.auth.cleanup([self.achall]) self.assertFalse(os.path.exists(self.validation_path)) @@ -244,7 +240,7 @@ class AuthenticatorTest(unittest.TestCase): os.rmdir(leftover_path) - @mock.patch('os.rmdir') + @mock.patch('certbot.compat.os.rmdir') def test_cleanup_failure(self, mock_rmdir): self.auth.prepare() self.auth.perform([self.achall]) @@ -265,7 +261,7 @@ class WebrootActionTest(unittest.TestCase): challb=acme_util.HTTP01_P, domain="thing.com", account_key=KEY) def setUp(self): - from certbot.plugins.webroot import Authenticator + from certbot._internal.plugins.webroot import Authenticator self.path = tempfile.mkdtemp() self.parser = argparse.ArgumentParser() self.parser.add_argument("-d", "--domains", @@ -297,8 +293,21 @@ class WebrootActionTest(unittest.TestCase): self.assertEqual( config.webroot_map[self.achall.domain], self.path) + def test_webroot_map_partial_without_perform(self): + # This test acknowledges the fact that webroot_map content will be partial if webroot + # plugin perform method is not invoked (corner case when all auths are already valid). + # To not be a problem, the webroot_path must always been conserved during renew. + # This condition is challenged by: + # certbot.tests.renewal_tests::RenewalTest::test_webroot_params_conservation + # See https://github.com/certbot/certbot/pull/7095 for details. + other_webroot_path = tempfile.mkdtemp() + args = self.parser.parse_args("-w {0} -d {1} -w {2} -d bar".format( + self.path, self.achall.domain, other_webroot_path).split()) + self.assertEqual(args.webroot_map, {self.achall.domain: self.path}) + self.assertEqual(args.webroot_path, [self.path, other_webroot_path]) + def _get_config_after_perform(self, config): - from certbot.plugins.webroot import Authenticator + from certbot._internal.plugins.webroot import Authenticator auth = Authenticator(config, "webroot") auth.perform([self.achall]) return auth.config diff --git a/certbot/tests/renewal_test.py b/certbot/tests/renewal_test.py index d292d909a..e92211ea2 100644 --- a/certbot/tests/renewal_test.py +++ b/certbot/tests/renewal_test.py @@ -1,21 +1,17 @@ -"""Tests for certbot.renewal""" -import mock +"""Tests for certbot._internal.renewal""" import unittest +import mock + from acme import challenges - -from certbot import configuration from certbot import errors -from certbot import storage - +from certbot._internal import configuration +from certbot._internal import storage import certbot.tests.util as test_util class RenewalTest(test_util.ConfigTestCase): - def setUp(self): - super(RenewalTest, self).setUp() - - @mock.patch('certbot.cli.set_by_cli') + @mock.patch('certbot._internal.cli.set_by_cli') def test_ancient_webroot_renewal_conf(self, mock_set_by_cli): mock_set_by_cli.return_value = False rc_path = test_util.make_lineage( @@ -27,44 +23,64 @@ class RenewalTest(test_util.ConfigTestCase): lineage = storage.RenewableCert(rc_path, config) renewalparams = lineage.configuration['renewalparams'] # pylint: disable=protected-access - from certbot import renewal + from certbot._internal import renewal renewal._restore_webroot_config(config, renewalparams) self.assertEqual(config.webroot_path, ['/var/www/']) + @mock.patch('certbot._internal.renewal.cli.set_by_cli') + def test_webroot_params_conservation(self, mock_set_by_cli): + # For more details about why this test is important, see: + # certbot._internal.plugins.webroot_test:: + # WebrootActionTest::test_webroot_map_partial_without_perform + from certbot._internal import renewal + mock_set_by_cli.return_value = False + + renewalparams = { + 'webroot_map': {'test.example.com': '/var/www/test'}, + 'webroot_path': ['/var/www/test', '/var/www/other'], + } + renewal._restore_webroot_config(self.config, renewalparams) # pylint: disable=protected-access + self.assertEqual(self.config.webroot_map, {'test.example.com': '/var/www/test'}) + self.assertEqual(self.config.webroot_path, ['/var/www/test', '/var/www/other']) + + renewalparams = { + 'webroot_map': {}, + 'webroot_path': '/var/www/test', + } + renewal._restore_webroot_config(self.config, renewalparams) # pylint: disable=protected-access + self.assertEqual(self.config.webroot_map, {}) + self.assertEqual(self.config.webroot_path, ['/var/www/test']) + class RestoreRequiredConfigElementsTest(test_util.ConfigTestCase): - """Tests for certbot.renewal.restore_required_config_elements.""" - def setUp(self): - super(RestoreRequiredConfigElementsTest, self).setUp() - + """Tests for certbot._internal.renewal.restore_required_config_elements.""" @classmethod def _call(cls, *args, **kwargs): - from certbot.renewal import restore_required_config_elements + from certbot._internal.renewal import restore_required_config_elements return restore_required_config_elements(*args, **kwargs) - @mock.patch('certbot.renewal.cli.set_by_cli') + @mock.patch('certbot._internal.renewal.cli.set_by_cli') def test_allow_subset_of_names_success(self, mock_set_by_cli): mock_set_by_cli.return_value = False self._call(self.config, {'allow_subset_of_names': 'True'}) self.assertTrue(self.config.allow_subset_of_names is True) - @mock.patch('certbot.renewal.cli.set_by_cli') + @mock.patch('certbot._internal.renewal.cli.set_by_cli') def test_allow_subset_of_names_failure(self, mock_set_by_cli): mock_set_by_cli.return_value = False renewalparams = {'allow_subset_of_names': 'maybe'} self.assertRaises( errors.Error, self._call, self.config, renewalparams) - @mock.patch('certbot.renewal.cli.set_by_cli') + @mock.patch('certbot._internal.renewal.cli.set_by_cli') def test_pref_challs_list(self, mock_set_by_cli): mock_set_by_cli.return_value = False - renewalparams = {'pref_challs': 'tls-sni, http-01, dns'.split(',')} + renewalparams = {'pref_challs': 'http-01, dns'.split(',')} self._call(self.config, renewalparams) - expected = [challenges.TLSSNI01.typ, - challenges.HTTP01.typ, challenges.DNS01.typ] + expected = [challenges.HTTP01.typ, challenges.DNS01.typ] self.assertEqual(self.config.pref_challs, expected) - @mock.patch('certbot.renewal.cli.set_by_cli') + @mock.patch('certbot._internal.renewal.cli.set_by_cli') def test_pref_challs_str(self, mock_set_by_cli): mock_set_by_cli.return_value = False renewalparams = {'pref_challs': 'dns'} @@ -72,24 +88,25 @@ class RestoreRequiredConfigElementsTest(test_util.ConfigTestCase): expected = [challenges.DNS01.typ] self.assertEqual(self.config.pref_challs, expected) - @mock.patch('certbot.renewal.cli.set_by_cli') + @mock.patch('certbot._internal.renewal.cli.set_by_cli') def test_pref_challs_failure(self, mock_set_by_cli): mock_set_by_cli.return_value = False renewalparams = {'pref_challs': 'finding-a-shrubbery'} self.assertRaises(errors.Error, self._call, self.config, renewalparams) - @mock.patch('certbot.renewal.cli.set_by_cli') + @mock.patch('certbot._internal.renewal.cli.set_by_cli') def test_must_staple_success(self, mock_set_by_cli): mock_set_by_cli.return_value = False self._call(self.config, {'must_staple': 'True'}) self.assertTrue(self.config.must_staple is True) - @mock.patch('certbot.renewal.cli.set_by_cli') + @mock.patch('certbot._internal.renewal.cli.set_by_cli') def test_must_staple_failure(self, mock_set_by_cli): mock_set_by_cli.return_value = False renewalparams = {'must_staple': 'maybe'} self.assertRaises( errors.Error, self._call, self.config, renewalparams) + if __name__ == "__main__": unittest.main() # pragma: no cover diff --git a/certbot/tests/renewupdater_test.py b/certbot/tests/renewupdater_test.py index 5fe188c42..c6f8f3713 100644 --- a/certbot/tests/renewupdater_test.py +++ b/certbot/tests/renewupdater_test.py @@ -1,13 +1,12 @@ """Tests for renewal updater interfaces""" import unittest + import mock from certbot import interfaces -from certbot import main -from certbot import updater - +from certbot._internal import main +from certbot._internal import updater from certbot.plugins import enhancements - import certbot.tests.util as test_util @@ -21,9 +20,9 @@ class RenewUpdaterTest(test_util.ConfigTestCase): self.renew_deployer = mock.MagicMock(spec=interfaces.RenewDeployer) self.mockinstaller = mock.MagicMock(spec=enhancements.AutoHSTSEnhancement) - @mock.patch('certbot.main._get_and_save_cert') - @mock.patch('certbot.plugins.selection.choose_configurator_plugins') - @mock.patch('certbot.plugins.selection.get_unprepared_installer') + @mock.patch('certbot._internal.main._get_and_save_cert') + @mock.patch('certbot._internal.plugins.selection.choose_configurator_plugins') + @mock.patch('certbot._internal.plugins.selection.get_unprepared_installer') @test_util.patch_get_utility() def test_server_updates(self, _, mock_geti, mock_select, mock_getsave): mock_getsave.return_value = mock.MagicMock() @@ -32,7 +31,7 @@ class RenewUpdaterTest(test_util.ConfigTestCase): # Generic Updater mock_select.return_value = (mock_generic_updater, None) mock_geti.return_value = mock_generic_updater - with mock.patch('certbot.main._init_le_client'): + with mock.patch('certbot._internal.main._init_le_client'): main.renew_cert(self.config, None, mock.MagicMock()) self.assertTrue(mock_generic_updater.restart.called) @@ -48,7 +47,7 @@ class RenewUpdaterTest(test_util.ConfigTestCase): updater.run_renewal_deployer(self.config, lineage, mock_deployer) self.assertTrue(mock_deployer.renew_deploy.called_with(lineage)) - @mock.patch("certbot.updater.logger.debug") + @mock.patch("certbot._internal.updater.logger.debug") def test_updater_skip_dry_run(self, mock_log): self.config.dry_run = True updater.run_generic_updaters(self.config, None, None) @@ -56,7 +55,7 @@ class RenewUpdaterTest(test_util.ConfigTestCase): self.assertEqual(mock_log.call_args[0][0], "Skipping updaters in dry-run mode.") - @mock.patch("certbot.updater.logger.debug") + @mock.patch("certbot._internal.updater.logger.debug") def test_deployer_skip_dry_run(self, mock_log): self.config.dry_run = True updater.run_renewal_deployer(self.config, None, None) @@ -64,7 +63,7 @@ class RenewUpdaterTest(test_util.ConfigTestCase): self.assertEqual(mock_log.call_args[0][0], "Skipping renewal deployer in dry-run mode.") - @mock.patch('certbot.plugins.selection.get_unprepared_installer') + @mock.patch('certbot._internal.plugins.selection.get_unprepared_installer') def test_enhancement_updates(self, mock_geti): mock_geti.return_value = self.mockinstaller updater.run_generic_updaters(self.config, mock.MagicMock(), None) @@ -76,7 +75,7 @@ class RenewUpdaterTest(test_util.ConfigTestCase): self.mockinstaller) self.assertTrue(self.mockinstaller.deploy_autohsts.called) - @mock.patch('certbot.plugins.selection.get_unprepared_installer') + @mock.patch('certbot._internal.plugins.selection.get_unprepared_installer') def test_enhancement_updates_not_called(self, mock_geti): self.config.disable_renew_updates = True mock_geti.return_value = self.mockinstaller @@ -89,7 +88,7 @@ class RenewUpdaterTest(test_util.ConfigTestCase): self.mockinstaller) self.assertFalse(self.mockinstaller.deploy_autohsts.called) - @mock.patch('certbot.plugins.selection.get_unprepared_installer') + @mock.patch('certbot._internal.plugins.selection.get_unprepared_installer') def test_enhancement_no_updater(self, mock_geti): FAKEINDEX = [ { diff --git a/certbot/tests/reporter_test.py b/certbot/tests/reporter_test.py index 22e11e672..3d7c80172 100644 --- a/certbot/tests/reporter_test.py +++ b/certbot/tests/reporter_test.py @@ -1,15 +1,15 @@ -"""Tests for certbot.reporter.""" -import mock +"""Tests for certbot._internal.reporter.""" import sys import unittest +import mock import six class ReporterTest(unittest.TestCase): - """Tests for certbot.reporter.Reporter.""" + """Tests for certbot._internal.reporter.Reporter.""" def setUp(self): - from certbot import reporter + from certbot._internal import reporter self.reporter = reporter.Reporter(mock.MagicMock(quiet=False)) self.old_stdout = sys.stdout # type: ignore diff --git a/certbot/tests/reverter_test.py b/certbot/tests/reverter_test.py index d04e3c641..fc9133c5f 100644 --- a/certbot/tests/reverter_test.py +++ b/certbot/tests/reverter_test.py @@ -1,7 +1,6 @@ """Test certbot.reverter.""" import csv import logging -import os import shutil import tempfile import unittest @@ -10,12 +9,11 @@ import mock import six from certbot import errors - +from certbot.compat import os from certbot.tests import util as test_util class ReverterCheckpointLocalTest(test_util.ConfigTestCase): - # pylint: disable=too-many-instance-attributes, too-many-public-methods """Test the Reverter Class.""" def setUp(self): super(ReverterCheckpointLocalTest, self).setUp() @@ -50,7 +48,6 @@ class ReverterCheckpointLocalTest(test_util.ConfigTestCase): x = f.read() self.assertTrue("No changes" in x) - @test_util.broken_on_windows def test_basic_add_to_temp_checkpoint(self): # These shouldn't conflict even though they are both named config.txt self.reverter.add_to_temp_checkpoint(self.sets[0], "save1") @@ -92,7 +89,6 @@ class ReverterCheckpointLocalTest(test_util.ConfigTestCase): self.assertRaises(errors.ReverterError, self.reverter.add_to_checkpoint, set([config3]), "invalid save") - @test_util.broken_on_windows def test_multiple_saves_and_temp_revert(self): self.reverter.add_to_temp_checkpoint(self.sets[0], "save1") update_file(self.config1, "updated-directive") @@ -122,7 +118,6 @@ class ReverterCheckpointLocalTest(test_util.ConfigTestCase): self.assertFalse(os.path.isfile(config3)) self.assertFalse(os.path.isfile(config4)) - @test_util.broken_on_windows def test_multiple_registration_same_file(self): self.reverter.register_file_creation(True, self.config1) self.reverter.register_file_creation(True, self.config1) @@ -147,7 +142,6 @@ class ReverterCheckpointLocalTest(test_util.ConfigTestCase): errors.ReverterError, self.reverter.register_file_creation, "filepath") - @test_util.broken_on_windows def test_register_undo_command(self): coms = [ ["a2dismod", "ssl"], @@ -170,7 +164,6 @@ class ReverterCheckpointLocalTest(test_util.ConfigTestCase): errors.ReverterError, self.reverter.register_undo_command, True, ["command"]) - @test_util.broken_on_windows @mock.patch("certbot.util.run_script") def test_run_undo_commands(self, mock_run): mock_run.side_effect = ["", errors.SubprocessError] @@ -234,7 +227,6 @@ class ReverterCheckpointLocalTest(test_util.ConfigTestCase): self.assertRaises( errors.ReverterError, self.reverter.revert_temporary_config) - @test_util.broken_on_windows @mock.patch("certbot.reverter.logger.warning") def test_recover_checkpoint_missing_new_files(self, mock_warn): self.reverter.register_file_creation( @@ -249,7 +241,6 @@ class ReverterCheckpointLocalTest(test_util.ConfigTestCase): self.assertRaises( errors.ReverterError, self.reverter.revert_temporary_config) - @test_util.broken_on_windows def test_recovery_routine_temp_and_perm(self): # Register a new perm checkpoint file config3 = os.path.join(self.dir1, "config3.txt") @@ -285,7 +276,6 @@ class ReverterCheckpointLocalTest(test_util.ConfigTestCase): class TestFullCheckpointsReverter(test_util.ConfigTestCase): - # pylint: disable=too-many-instance-attributes """Tests functions having to deal with full checkpoints.""" def setUp(self): super(TestFullCheckpointsReverter, self).setUp() @@ -313,7 +303,6 @@ class TestFullCheckpointsReverter(test_util.ConfigTestCase): self.assertRaises( errors.ReverterError, self.reverter.rollback_checkpoints, "one") - @test_util.broken_on_windows def test_rollback_finalize_checkpoint_valid_inputs(self): config3 = self._setup_three_checkpoints() @@ -356,16 +345,15 @@ class TestFullCheckpointsReverter(test_util.ConfigTestCase): self.assertRaises( errors.ReverterError, self.reverter.finalize_checkpoint, "Title") - @mock.patch("certbot.reverter.compat.os_rename") - def test_finalize_checkpoint_no_rename_directory(self, mock_rename): + @mock.patch("certbot.reverter.filesystem.replace") + def test_finalize_checkpoint_no_rename_directory(self, mock_replace): self.reverter.add_to_checkpoint(self.sets[0], "perm save") - mock_rename.side_effect = OSError + mock_replace.side_effect = OSError self.assertRaises( errors.ReverterError, self.reverter.finalize_checkpoint, "Title") - @test_util.broken_on_windows @mock.patch("certbot.reverter.logger") def test_rollback_too_many(self, mock_logger): # Test no exist warning... @@ -378,7 +366,6 @@ class TestFullCheckpointsReverter(test_util.ConfigTestCase): self.reverter.rollback_checkpoints(4) self.assertEqual(mock_logger.warning.call_count, 1) - @test_util.broken_on_windows def test_multi_rollback(self): config3 = self._setup_three_checkpoints() self.reverter.rollback_checkpoints(3) @@ -387,39 +374,6 @@ class TestFullCheckpointsReverter(test_util.ConfigTestCase): self.assertEqual(read_in(self.config2), "directive-dir2") self.assertFalse(os.path.isfile(config3)) - @test_util.patch_get_utility() - def test_view_config_changes(self, mock_output): - """This is not strict as this is subject to change.""" - self._setup_three_checkpoints() - - # Make sure it doesn't throw any errors - self.reverter.view_config_changes() - - # Make sure notification is output - self.assertEqual(mock_output().notification.call_count, 1) - - @mock.patch("certbot.reverter.logger") - def test_view_config_changes_no_backups(self, mock_logger): - self.reverter.view_config_changes() - self.assertTrue(mock_logger.info.call_count > 0) - - def test_view_config_changes_bad_backups_dir(self): - # There shouldn't be any "in progress directories when this is called - # It must just be clean checkpoints - os.makedirs(os.path.join(self.config.backup_dir, "in_progress")) - - self.assertRaises( - errors.ReverterError, self.reverter.view_config_changes) - - def test_view_config_changes_for_logging(self): - self._setup_three_checkpoints() - - config_changes = self.reverter.view_config_changes(for_logging=True) - - self.assertTrue("First Checkpoint" in config_changes) - self.assertTrue("Second Checkpoint" in config_changes) - self.assertTrue("Third Checkpoint" in config_changes) - def _setup_three_checkpoints(self): """Generate some finalized checkpoints.""" # Checkpoint1 - config1 diff --git a/certbot/tests/storage_test.py b/certbot/tests/storage_test.py index e61ed2aca..6208974ec 100644 --- a/certbot/tests/storage_test.py +++ b/certbot/tests/storage_test.py @@ -1,7 +1,6 @@ -"""Tests for certbot.storage.""" +"""Tests for certbot._internal.storage.""" # pylint disable=protected-access import datetime -import os import shutil import stat import unittest @@ -12,18 +11,15 @@ import pytz import six import certbot -from certbot import cli -from certbot import compat from certbot import errors -from certbot.storage import ALL_FOUR - +from certbot._internal.storage import ALL_FOUR +from certbot.compat import filesystem +from certbot.compat import os import certbot.tests.util as test_util - CERT = test_util.load_cert('cert_512.pem') - def unlink_all(rc_object): """Unlink all four items associated with this RenewableCert.""" for kind in ALL_FOUR: @@ -37,6 +33,48 @@ def fill_with_sample_data(rc_object): f.write(kind) +class RelevantValuesTest(unittest.TestCase): + """Tests for certbot._internal.storage.relevant_values.""" + + def setUp(self): + self.values = {"server": "example.org"} + + def _call(self, *args, **kwargs): + from certbot._internal.storage import relevant_values + return relevant_values(*args, **kwargs) + + @mock.patch("certbot._internal.cli.option_was_set") + @mock.patch("certbot._internal.plugins.disco.PluginsRegistry.find_all") + def test_namespace(self, mock_find_all, mock_option_was_set): + mock_find_all.return_value = ["certbot-foo:bar"] + mock_option_was_set.return_value = True + + self.values["certbot_foo:bar_baz"] = 42 + self.assertEqual( + self._call(self.values.copy()), self.values) + + @mock.patch("certbot._internal.cli.option_was_set") + def test_option_set(self, mock_option_was_set): + mock_option_was_set.return_value = True + + self.values["allow_subset_of_names"] = True + self.values["authenticator"] = "apache" + self.values["rsa_key_size"] = 1337 + expected_relevant_values = self.values.copy() + self.values["hello"] = "there" + + self.assertEqual(self._call(self.values), expected_relevant_values) + + @mock.patch("certbot._internal.cli.option_was_set") + def test_option_unset(self, mock_option_was_set): + mock_option_was_set.return_value = False + + expected_relevant_values = self.values.copy() + self.values["rsa_key_size"] = 2048 + + self.assertEqual(self._call(self.values), expected_relevant_values) + + class BaseRenewableCertTest(test_util.ConfigTestCase): """Base class for setting up Renewable Cert tests. @@ -46,16 +84,16 @@ class BaseRenewableCertTest(test_util.ConfigTestCase): """ def setUp(self): - from certbot import storage + from certbot._internal import storage super(BaseRenewableCertTest, self).setUp() # TODO: maybe provide NamespaceConfig.make_dirs? # TODO: main() should create those dirs, c.f. #902 - os.makedirs(os.path.join(self.config.config_dir, "live", "example.org")) + filesystem.makedirs(os.path.join(self.config.config_dir, "live", "example.org")) archive_path = os.path.join(self.config.config_dir, "archive", "example.org") - os.makedirs(archive_path) - os.makedirs(os.path.join(self.config.config_dir, "renewal")) + filesystem.makedirs(archive_path) + filesystem.makedirs(os.path.join(self.config.config_dir, "renewal")) config_file = configobj.ConfigObj() for kind in ALL_FOUR: @@ -79,7 +117,7 @@ class BaseRenewableCertTest(test_util.ConfigTestCase): self.defaults = configobj.ConfigObj() - with mock.patch("certbot.storage.RenewableCert._check_symlinks") as check: + with mock.patch("certbot._internal.storage.RenewableCert._check_symlinks") as check: check.return_value = True self.test_rc = storage.RenewableCert(config_file.filename, self.config) @@ -93,7 +131,7 @@ class BaseRenewableCertTest(test_util.ConfigTestCase): with open(link, "wb") as f: f.write(kind.encode('ascii') if value is None else value) if kind == "privkey": - os.chmod(link, 0o600) + filesystem.chmod(link, 0o600) def _write_out_ex_kinds(self): for kind in ALL_FOUR: @@ -102,8 +140,7 @@ class BaseRenewableCertTest(test_util.ConfigTestCase): class RenewableCertTests(BaseRenewableCertTest): - # pylint: disable=too-many-public-methods - """Tests for certbot.storage.""" + """Tests for certbot._internal.storage.""" def test_initialization(self): self.assertEqual(self.test_rc.lineagename, "example.org") @@ -117,7 +154,7 @@ class RenewableCertTests(BaseRenewableCertTest): the renewal configuration file doesn't end in ".conf" """ - from certbot import storage + from certbot._internal import storage broken = os.path.join(self.config.config_dir, "broken.conf") with open(broken, "w") as f: f.write("[No closing bracket for you!") @@ -130,7 +167,7 @@ class RenewableCertTests(BaseRenewableCertTest): def test_renewal_incomplete_config(self): """Test that the RenewableCert constructor will complain if the renewal configuration file is missing a required file element.""" - from certbot import storage + from certbot._internal import storage config = configobj.ConfigObj() config["cert"] = "imaginary_cert.pem" # Here the required privkey is missing. @@ -142,29 +179,29 @@ class RenewableCertTests(BaseRenewableCertTest): config.filename, self.config) def test_no_renewal_version(self): - from certbot import storage + from certbot._internal import storage self._write_out_ex_kinds() self.assertTrue("version" not in self.config_file) - with mock.patch("certbot.storage.logger") as mock_logger: + with mock.patch("certbot._internal.storage.logger") as mock_logger: storage.RenewableCert(self.config_file.filename, self.config) self.assertFalse(mock_logger.warning.called) def test_renewal_newer_version(self): - from certbot import storage + from certbot._internal import storage self._write_out_ex_kinds() self.config_file["version"] = "99.99.99" self.config_file.write() - with mock.patch("certbot.storage.logger") as mock_logger: + with mock.patch("certbot._internal.storage.logger") as mock_logger: storage.RenewableCert(self.config_file.filename, self.config) self.assertTrue(mock_logger.info.called) self.assertTrue("version" in mock_logger.info.call_args[0][0]) def test_consistent(self): - # pylint: disable=too-many-statements,protected-access + # pylint: disable=protected-access oldcert = self.test_rc.cert self.test_rc.cert = "relative/path" # Absolute path for item requirement @@ -257,7 +294,7 @@ class RenewableCertTests(BaseRenewableCertTest): self.assertEqual(self.test_rc.latest_common_version(), 17) self.assertEqual(self.test_rc.next_free_version(), 18) - @mock.patch("certbot.storage.logger") + @mock.patch("certbot._internal.storage.logger") def test_ensure_deployed(self, mock_logger): mock_update = self.test_rc.update_all_links_to = mock.Mock() mock_has_pending = self.test_rc.has_pending_deployment = mock.Mock() @@ -319,11 +356,10 @@ class RenewableCertTests(BaseRenewableCertTest): basename = os.path.basename(path) if "fullchain" in basename and basename.startswith("prev"): raise ValueError - else: - real_unlink(path) + real_unlink(path) self._write_out_ex_kinds() - with mock.patch("certbot.storage.os.unlink") as mock_unlink: + with mock.patch("certbot._internal.storage.os.unlink") as mock_unlink: mock_unlink.side_effect = unlink_or_raise self.assertRaises(ValueError, self.test_rc.update_all_links_to, 12) @@ -335,11 +371,10 @@ class RenewableCertTests(BaseRenewableCertTest): # pylint: disable=missing-docstring if "fullchain" in os.path.basename(path): raise ValueError - else: - real_unlink(path) + real_unlink(path) self._write_out_ex_kinds() - with mock.patch("certbot.storage.os.unlink") as mock_unlink: + with mock.patch("certbot._internal.storage.os.unlink") as mock_unlink: mock_unlink.side_effect = unlink_or_raise self.assertRaises(ValueError, self.test_rc.update_all_links_to, 12) @@ -367,26 +402,12 @@ class RenewableCertTests(BaseRenewableCertTest): self.assertEqual(self.test_rc.names(), ["example.com", "www.example.com"]) - # Trying a non-current version - self._write_out_kind("cert", 15, test_util.load_vector("cert_512.pem")) - - self.assertEqual(self.test_rc.names(12), - ["example.com", "www.example.com"]) - - # Testing common name is listed first - self._write_out_kind( - "cert", 12, test_util.load_vector("cert-5sans_512.pem")) - - self.assertEqual( - self.test_rc.names(12), - ["example.com"] + ["{0}.example.com".format(c) for c in "abcd"]) - # Trying missing cert os.unlink(self.test_rc.cert) self.assertRaises(errors.CertStorageError, self.test_rc.names) - @mock.patch("certbot.storage.cli") - @mock.patch("certbot.storage.datetime") + @mock.patch("certbot._internal.storage.cli") + @mock.patch("certbot._internal.storage.datetime") def test_time_interval_judgments(self, mock_datetime, mock_cli): """Test should_autorenew() on the basis of expiry time windows.""" test_cert = test_util.load_vector("cert_512.pem") @@ -440,12 +461,11 @@ class RenewableCertTests(BaseRenewableCertTest): self.test_rc.configuration["renewalparams"]["autorenew"] = "False" self.assertFalse(self.test_rc.autorenewal_is_enabled()) - @mock.patch("certbot.storage.cli") - @mock.patch("certbot.storage.RenewableCert.ocsp_revoked") + @mock.patch("certbot._internal.storage.cli") + @mock.patch("certbot._internal.storage.RenewableCert.ocsp_revoked") def test_should_autorenew(self, mock_ocsp, mock_cli): """Test should_autorenew on the basis of reasons other than expiry time window.""" - # pylint: disable=too-many-statements mock_cli.set_by_cli.return_value = False # Autorenewal turned off self.test_rc.configuration["renewalparams"] = {"autorenew": "False"} @@ -458,8 +478,7 @@ class RenewableCertTests(BaseRenewableCertTest): self.assertTrue(self.test_rc.should_autorenew()) mock_ocsp.return_value = False - @test_util.broken_on_windows - @mock.patch("certbot.storage.relevant_values") + @mock.patch("certbot._internal.storage.relevant_values") def test_save_successor(self, mock_rv): # Mock relevant_values() to claim that all values are relevant here # (to avoid instantiating parser) @@ -522,8 +541,8 @@ class RenewableCertTests(BaseRenewableCertTest): self.assertFalse(os.path.islink(self.test_rc.version("privkey", 10))) self.assertFalse(os.path.exists(temp_config_file)) - @test_util.broken_on_windows - @mock.patch("certbot.storage.relevant_values") + @test_util.skip_on_windows('Group/everybody permissions are not maintained on Windows.') + @mock.patch("certbot._internal.storage.relevant_values") def test_save_successor_maintains_group_mode(self, mock_rv): # Mock relevant_values() to claim that all values are relevant here # (to avoid instantiating parser) @@ -531,27 +550,22 @@ class RenewableCertTests(BaseRenewableCertTest): for kind in ALL_FOUR: self._write_out_kind(kind, 1) self.test_rc.update_all_links_to(1) - self.assertTrue(compat.compare_file_modes( - os.stat(self.test_rc.version("privkey", 1)).st_mode, 0o600)) - os.chmod(self.test_rc.version("privkey", 1), 0o444) + self.assertTrue(filesystem.check_mode(self.test_rc.version("privkey", 1), 0o600)) + filesystem.chmod(self.test_rc.version("privkey", 1), 0o444) # If no new key, permissions should be the same (we didn't write any keys) self.test_rc.save_successor(1, b"newcert", None, b"new chain", self.config) - self.assertTrue(compat.compare_file_modes( - os.stat(self.test_rc.version("privkey", 2)).st_mode, 0o444)) + self.assertTrue(filesystem.check_mode(self.test_rc.version("privkey", 2), 0o444)) # If new key, permissions should be kept as 644 self.test_rc.save_successor(2, b"newcert", b"new_privkey", b"new chain", self.config) - self.assertTrue(compat.compare_file_modes( - os.stat(self.test_rc.version("privkey", 3)).st_mode, 0o644)) + self.assertTrue(filesystem.check_mode(self.test_rc.version("privkey", 3), 0o644)) # If permissions reverted, next renewal will also revert permissions of new key - os.chmod(self.test_rc.version("privkey", 3), 0o400) + filesystem.chmod(self.test_rc.version("privkey", 3), 0o400) self.test_rc.save_successor(3, b"newcert", b"new_privkey", b"new chain", self.config) - self.assertTrue(compat.compare_file_modes( - os.stat(self.test_rc.version("privkey", 4)).st_mode, 0o600)) + self.assertTrue(filesystem.check_mode(self.test_rc.version("privkey", 4), 0o600)) - @test_util.broken_on_windows - @mock.patch("certbot.storage.relevant_values") - @mock.patch("certbot.storage.os.chown") - def test_save_successor_maintains_gid(self, mock_chown, mock_rv): + @mock.patch("certbot._internal.storage.relevant_values") + @mock.patch("certbot._internal.storage.filesystem.copy_ownership_and_apply_mode") + def test_save_successor_maintains_gid(self, mock_ownership, mock_rv): # Mock relevant_values() to claim that all values are relevant here # (to avoid instantiating parser) mock_rv.side_effect = lambda x: x @@ -559,84 +573,18 @@ class RenewableCertTests(BaseRenewableCertTest): self._write_out_kind(kind, 1) self.test_rc.update_all_links_to(1) self.test_rc.save_successor(1, b"newcert", None, b"new chain", self.config) - self.assertFalse(mock_chown.called) + self.assertFalse(mock_ownership.called) self.test_rc.save_successor(2, b"newcert", b"new_privkey", b"new chain", self.config) - self.assertTrue(mock_chown.called) + self.assertTrue(mock_ownership.called) - def _test_relevant_values_common(self, values): - defaults = dict((option, cli.flag_default(option)) - for option in ("authenticator", "installer", - "rsa_key_size", "server",)) - mock_parser = mock.Mock(args=[], verb="plugins", - defaults=defaults) - - # make a copy to ensure values isn't modified - values = values.copy() - values.setdefault("server", defaults["server"]) - expected_server = values["server"] - - from certbot.storage import relevant_values - with mock.patch("certbot.cli.helpful_parser", mock_parser): - rv = relevant_values(values) - self.assertIn("server", rv) - self.assertEqual(rv.pop("server"), expected_server) - return rv - - def test_relevant_values(self): - """Test that relevant_values() can reject an irrelevant value.""" - self.assertEqual( - self._test_relevant_values_common({"hello": "there"}), {}) - - def test_relevant_values_default(self): - """Test that relevant_values() can reject a default value.""" - option = "rsa_key_size" - values = {option: cli.flag_default(option)} - self.assertEqual(self._test_relevant_values_common(values), {}) - - def test_relevant_values_nondefault(self): - """Test that relevant_values() can retain a non-default value.""" - values = {"rsa_key_size": 12} - self.assertEqual( - self._test_relevant_values_common(values), values) - - def test_relevant_values_bool(self): - values = {"allow_subset_of_names": True} - self.assertEqual( - self._test_relevant_values_common(values), values) - - def test_relevant_values_str(self): - values = {"authenticator": "apache"} - self.assertEqual( - self._test_relevant_values_common(values), values) - - def test_relevant_values_plugins_none(self): - self.assertEqual( - self._test_relevant_values_common( - {"authenticator": None, "installer": None}), {}) - - @mock.patch("certbot.cli.set_by_cli") - @mock.patch("certbot.plugins.disco.PluginsRegistry.find_all") - def test_relevant_values_namespace(self, mock_find_all, mock_set_by_cli): - mock_set_by_cli.return_value = True - mock_find_all.return_value = ["certbot-foo:bar"] - values = {"certbot_foo:bar_baz": 42} - self.assertEqual( - self._test_relevant_values_common(values), values) - - def test_relevant_values_server(self): - self.assertEqual( - # _test_relevant_values_common handles testing the server - # value and removes it - self._test_relevant_values_common({"server": "example.org"}), {}) - - @mock.patch("certbot.storage.relevant_values") + @mock.patch("certbot._internal.storage.relevant_values") def test_new_lineage(self, mock_rv): """Test for new_lineage() class method.""" # Mock relevant_values to say everything is relevant here (so we # don't have to mock the parser to help it decide!) mock_rv.side_effect = lambda x: x - from certbot import storage + from certbot._internal import storage result = storage.RenewableCert.new_lineage( "the-lineage.com", b"cert", b"privkey", b"chain", self.config) # This consistency check tests most relevant properties about the @@ -649,7 +597,7 @@ class RenewableCertTests(BaseRenewableCertTest): self.config.live_dir, "README"))) self.assertTrue(os.path.exists(os.path.join( self.config.live_dir, "the-lineage.com", "README"))) - self.assertTrue(compat.compare_file_modes(os.stat(result.key_path).st_mode, 0o600)) + self.assertTrue(filesystem.check_mode(result.key_path, 0o600)) with open(result.fullchain, "rb") as f: self.assertEqual(f.read(), b"cert" + b"chain") # Let's do it again and make sure it makes a different lineage @@ -660,12 +608,12 @@ class RenewableCertTests(BaseRenewableCertTest): self.assertTrue(os.path.exists(os.path.join( self.config.live_dir, "the-lineage.com-0001", "README"))) # Now trigger the detection of already existing files - os.mkdir(os.path.join( + filesystem.mkdir(os.path.join( self.config.live_dir, "the-lineage.com-0002")) self.assertRaises(errors.CertStorageError, storage.RenewableCert.new_lineage, "the-lineage.com", b"cert3", b"privkey3", b"chain3", self.config) - os.mkdir(os.path.join(self.config.default_archive_dir, "other-example.com")) + filesystem.mkdir(os.path.join(self.config.default_archive_dir, "other-example.com")) self.assertRaises(errors.CertStorageError, storage.RenewableCert.new_lineage, "other-example.com", b"cert4", @@ -676,14 +624,14 @@ class RenewableCertTests(BaseRenewableCertTest): # TODO: Conceivably we could test that the renewal parameters actually # got saved - @mock.patch("certbot.storage.relevant_values") + @mock.patch("certbot._internal.storage.relevant_values") def test_new_lineage_nonexistent_dirs(self, mock_rv): """Test that directories can be created if they don't exist.""" # Mock relevant_values to say everything is relevant here (so we # don't have to mock the parser to help it decide!) mock_rv.side_effect = lambda x: x - from certbot import storage + from certbot._internal import storage shutil.rmtree(self.config.renewal_configs_dir) shutil.rmtree(self.config.default_archive_dir) shutil.rmtree(self.config.live_dir) @@ -698,9 +646,9 @@ class RenewableCertTests(BaseRenewableCertTest): self.assertTrue(os.path.exists(os.path.join( self.config.default_archive_dir, "the-lineage.com", "privkey1.pem"))) - @mock.patch("certbot.storage.util.unique_lineage_name") + @mock.patch("certbot._internal.storage.util.unique_lineage_name") def test_invalid_config_filename(self, mock_uln): - from certbot import storage + from certbot._internal import storage mock_uln.return_value = "this_does_not_end_with_dot_conf", "yikes" self.assertRaises(errors.CertStorageError, storage.RenewableCert.new_lineage, "example.com", @@ -730,7 +678,7 @@ class RenewableCertTests(BaseRenewableCertTest): self.assertFalse(self.test_rc.ocsp_revoked()) def test_add_time_interval(self): - from certbot import storage + from certbot._internal import storage # this month has 30 days, and the next year is a leap year time_1 = pytz.UTC.fromutc(datetime.datetime(2003, 11, 20, 11, 59, 21)) @@ -785,7 +733,7 @@ class RenewableCertTests(BaseRenewableCertTest): self.assertEqual(self.test_rc.is_test_cert, False) def test_missing_cert(self): - from certbot import storage + from certbot._internal import storage self.assertRaises(errors.CertStorageError, storage.RenewableCert, self.config_file.filename, self.config) @@ -803,14 +751,14 @@ class RenewableCertTests(BaseRenewableCertTest): with open(temp, "w") as f: f.write("[renewalparams]\nuseful = value # A useful value\n" "useless = value # Not needed\n") - os.chmod(temp, 0o640) + filesystem.chmod(temp, 0o640) target = {} for x in ALL_FOUR: target[x] = "somewhere" archive_dir = "the_archive" relevant_data = {"useful": "new_value"} - from certbot import storage + from certbot._internal import storage storage.write_renewal_config(temp, temp2, archive_dir, target, relevant_data) with open(temp2, "r") as f: @@ -828,7 +776,7 @@ class RenewableCertTests(BaseRenewableCertTest): stat.S_IMODE(os.lstat(temp2).st_mode)) def test_update_symlinks(self): - from certbot import storage + from certbot._internal import storage archive_dir_path = os.path.join(self.config.config_dir, "archive", "example.org") for kind in ALL_FOUR: live_path = self.config_file[kind] @@ -843,7 +791,7 @@ class RenewableCertTests(BaseRenewableCertTest): update_symlinks=True) class DeleteFilesTest(BaseRenewableCertTest): - """Tests for certbot.storage.delete_files""" + """Tests for certbot._internal.storage.delete_files""" def setUp(self): super(DeleteFilesTest, self).setUp() @@ -861,8 +809,8 @@ class DeleteFilesTest(BaseRenewableCertTest): self.config.config_dir, "archive", "example.org"))) def _call(self): - from certbot import storage - with mock.patch("certbot.storage.logger"): + from certbot._internal import storage + with mock.patch("certbot._internal.storage.logger"): storage.delete_files(self.config, "example.org") def test_delete_all_files(self): @@ -933,7 +881,7 @@ class DeleteFilesTest(BaseRenewableCertTest): self.assertFalse(os.path.exists(archive_dir)) class CertPathForCertNameTest(BaseRenewableCertTest): - """Test for certbot.storage.cert_path_for_cert_name""" + """Test for certbot._internal.storage.cert_path_for_cert_name""" def setUp(self): super(CertPathForCertNameTest, self).setUp() self.config_file.write() @@ -943,7 +891,7 @@ class CertPathForCertNameTest(BaseRenewableCertTest): self.config.cert_path = (self.fullchain, '') def _call(self, cli_config, certname): - from certbot.storage import cert_path_for_cert_name + from certbot._internal.storage import cert_path_for_cert_name return cert_path_for_cert_name(cli_config, certname) def test_simple_cert_name(self): diff --git a/certbot/tests/util_test.py b/certbot/tests/util_test.py index 81d0629c8..3ff09a83f 100644 --- a/certbot/tests/util_test.py +++ b/certbot/tests/util_test.py @@ -1,16 +1,16 @@ """Tests for certbot.util.""" import argparse import errno -import os -import shutil +import sys import unittest import mock import six from six.moves import reload_module # pylint: disable=import-error -from certbot import compat from certbot import errors +from certbot.compat import filesystem +from certbot.compat import os import certbot.tests.util as test_util @@ -53,26 +53,13 @@ class ExeExistsTest(unittest.TestCase): from certbot.util import exe_exists return exe_exists(exe) - @mock.patch("certbot.util.os.path.isfile") - @mock.patch("certbot.util.os.access") - def test_full_path(self, mock_access, mock_isfile): - mock_access.return_value = True - mock_isfile.return_value = True - self.assertTrue(self._call("/path/to/exe")) + def test_exe_exists(self): + with mock.patch("certbot.util.filesystem.is_executable", return_value=True): + self.assertTrue(self._call("/path/to/exe")) - @mock.patch("certbot.util.os.path.isfile") - @mock.patch("certbot.util.os.access") - def test_on_path(self, mock_access, mock_isfile): - mock_access.return_value = True - mock_isfile.return_value = True - self.assertTrue(self._call("exe")) - - @mock.patch("certbot.util.os.path.isfile") - @mock.patch("certbot.util.os.access") - def test_not_found(self, mock_access, mock_isfile): - mock_access.return_value = False - mock_isfile.return_value = True - self.assertFalse(self._call("exe")) + def test_exe_not_exists(self): + with mock.patch("certbot.util.filesystem.is_executable", return_value=False): + self.assertFalse(self._call("/path/to/exe")) class LockDirUntilExit(test_util.TempDirTestCase): @@ -88,23 +75,26 @@ class LockDirUntilExit(test_util.TempDirTestCase): import certbot.util reload_module(certbot.util) - @test_util.broken_on_windows @mock.patch('certbot.util.logger') @mock.patch('certbot.util.atexit_register') def test_it(self, mock_register, mock_logger): subdir = os.path.join(self.tempdir, 'subdir') - os.mkdir(subdir) + filesystem.mkdir(subdir) self._call(self.tempdir) self._call(subdir) self._call(subdir) self.assertEqual(mock_register.call_count, 1) registered_func = mock_register.call_args[0][0] - shutil.rmtree(subdir) - registered_func() # exception not raised - # logger.debug is only called once because the second call - # to lock subdir was ignored because it was already locked - self.assertEqual(mock_logger.debug.call_count, 1) + + from certbot import util + # Despite lock_dir_until_exit has been called twice to subdir, its lock should have been + # added only once. So we expect to have two lock references: for self.tempdir and subdir + self.assertTrue(len(util._LOCKS) == 2) # pylint: disable=protected-access + registered_func() # Exception should not be raised + # Logically, logger.debug, that would be invoked in case of unlock failure, + # should never been called. + self.assertEqual(mock_logger.debug.call_count, 0) class SetUpCoreDirTest(test_util.TempDirTestCase): @@ -117,15 +107,14 @@ class SetUpCoreDirTest(test_util.TempDirTestCase): @mock.patch('certbot.util.lock_dir_until_exit') def test_success(self, mock_lock): new_dir = os.path.join(self.tempdir, 'new') - self._call(new_dir, 0o700, compat.os_geteuid(), False) + self._call(new_dir, 0o700, False) self.assertTrue(os.path.exists(new_dir)) self.assertEqual(mock_lock.call_count, 1) @mock.patch('certbot.util.make_or_verify_dir') def test_failure(self, mock_make_or_verify): mock_make_or_verify.side_effect = OSError - self.assertRaises(errors.Error, self._call, - self.tempdir, 0o700, compat.os_geteuid(), False) + self.assertRaises(errors.Error, self._call, self.tempdir, 0o700, False) class MakeOrVerifyDirTest(test_util.TempDirTestCase): @@ -140,65 +129,31 @@ class MakeOrVerifyDirTest(test_util.TempDirTestCase): super(MakeOrVerifyDirTest, self).setUp() self.path = os.path.join(self.tempdir, "foo") - os.mkdir(self.path, 0o600) - - self.uid = compat.os_geteuid() + filesystem.mkdir(self.path, 0o600) def _call(self, directory, mode): from certbot.util import make_or_verify_dir - return make_or_verify_dir(directory, mode, self.uid, strict=True) + return make_or_verify_dir(directory, mode, strict=True) def test_creates_dir_when_missing(self): path = os.path.join(self.tempdir, "bar") self._call(path, 0o650) self.assertTrue(os.path.isdir(path)) - self.assertTrue(compat.compare_file_modes(os.stat(path).st_mode, 0o650)) + self.assertTrue(filesystem.check_mode(path, 0o650)) def test_existing_correct_mode_does_not_fail(self): self._call(self.path, 0o600) - self.assertTrue(compat.compare_file_modes(os.stat(self.path).st_mode, 0o600)) + self.assertTrue(filesystem.check_mode(self.path, 0o600)) - @test_util.skip_on_windows('Umask modes are mostly ignored on Windows.') def test_existing_wrong_mode_fails(self): self.assertRaises(errors.Error, self._call, self.path, 0o400) def test_reraises_os_error(self): - with mock.patch.object(os, "makedirs") as makedirs: + with mock.patch.object(filesystem, "makedirs") as makedirs: makedirs.side_effect = OSError() self.assertRaises(OSError, self._call, "bar", 12312312) -class CheckPermissionsTest(test_util.TempDirTestCase): - """Tests for certbot.util.check_permissions. - - Note that it is not possible to test for a wrong file owner, - as this testing script would have to be run as root. - - """ - - def setUp(self): - super(CheckPermissionsTest, self).setUp() - - self.uid = compat.os_geteuid() - - def _call(self, mode): - from certbot.util import check_permissions - return check_permissions(self.tempdir, mode, self.uid) - - def test_ok_mode(self): - os.chmod(self.tempdir, 0o600) - self.assertTrue(self._call(0o600)) - - def test_wrong_mode(self): - os.chmod(self.tempdir, 0o400) - try: - self.assertFalse(self._call(0o600)) - finally: - # Without proper write permissions, Windows is unable to delete a folder, - # even with admin permissions. Write access must be explicitly set first. - os.chmod(self.tempdir, 0o700) - - class UniqueFileTest(test_util.TempDirTestCase): """Tests for certbot.util.unique_file.""" @@ -221,8 +176,8 @@ class UniqueFileTest(test_util.TempDirTestCase): def test_right_mode(self): fd1, name1 = self._call(0o700) fd2, name2 = self._call(0o600) - self.assertTrue(compat.compare_file_modes(0o700, os.stat(name1).st_mode)) - self.assertTrue(compat.compare_file_modes(0o600, os.stat(name2).st_mode)) + self.assertTrue(filesystem.check_mode(name1, 0o700)) + self.assertTrue(filesystem.check_mode(name2, 0o600)) fd1.close() fd2.close() @@ -283,7 +238,7 @@ class UniqueLineageNameTest(test_util.TempDirTestCase): f.close() def test_failure(self): - with mock.patch("certbot.util.os.open", side_effect=OSError(errno.EIO)): + with mock.patch("certbot.compat.filesystem.open", side_effect=OSError(errno.EIO)): self.assertRaises(OSError, self._call, "wow") @@ -310,10 +265,10 @@ class SafelyRemoveTest(test_util.TempDirTestCase): # no error, yay! self.assertFalse(os.path.exists(self.path)) - @mock.patch("certbot.util.os.remove") - def test_other_error_passthrough(self, mock_remove): - mock_remove.side_effect = OSError - self.assertRaises(OSError, self._call) + def test_other_error_passthrough(self): + with mock.patch("certbot.util.os.remove") as mock_remove: + mock_remove.side_effect = OSError + self.assertRaises(OSError, self._call) class SafeEmailTest(unittest.TestCase): @@ -349,29 +304,28 @@ class AddDeprecatedArgumentTest(unittest.TestCase): def _call(self, argument_name, nargs): from certbot.util import add_deprecated_argument - add_deprecated_argument(self.parser.add_argument, argument_name, nargs) def test_warning_no_arg(self): self._call("--old-option", 0) - stderr = self._get_argparse_warnings(["--old-option"]) - self.assertTrue("--old-option is deprecated" in stderr) + with mock.patch("certbot.util.logger.warning") as mock_warn: + self.parser.parse_args(["--old-option"]) + self.assertEqual(mock_warn.call_count, 1) + self.assertTrue("is deprecated" in mock_warn.call_args[0][0]) + self.assertEqual("--old-option", mock_warn.call_args[0][1]) def test_warning_with_arg(self): self._call("--old-option", 1) - stderr = self._get_argparse_warnings(["--old-option", "42"]) - self.assertTrue("--old-option is deprecated" in stderr) - - def _get_argparse_warnings(self, args): - stderr = six.StringIO() - with mock.patch("certbot.util.sys.stderr", new=stderr): - self.parser.parse_args(args) - return stderr.getvalue() + with mock.patch("certbot.util.logger.warning") as mock_warn: + self.parser.parse_args(["--old-option", "42"]) + self.assertEqual(mock_warn.call_count, 1) + self.assertTrue("is deprecated" in mock_warn.call_args[0][0]) + self.assertEqual("--old-option", mock_warn.call_args[0][1]) def test_help(self): self._call("--old-option", 2) stdout = six.StringIO() - with mock.patch("certbot.util.sys.stdout", new=stdout): + with mock.patch("sys.stdout", new=stdout): try: self.parser.parse_args(["-h"]) except SystemExit: @@ -520,71 +474,84 @@ class IsWildcardDomainTest(unittest.TestCase): class OsInfoTest(unittest.TestCase): """Test OS / distribution detection""" - def test_systemd_os_release(self): - from certbot.util import (get_os_info, get_systemd_os_info, - get_os_info_ua) + @mock.patch("certbot.util.distro") + @unittest.skipUnless(sys.platform.startswith("linux"), "requires Linux") + def test_systemd_os_release_like(self, m_distro): + import certbot.util as cbutil + m_distro.like.return_value = "first debian third" + id_likes = cbutil.get_systemd_os_like() + self.assertEqual(len(id_likes), 3) + self.assertTrue("debian" in id_likes) - with mock.patch('os.path.isfile', return_value=True): - self.assertEqual(get_os_info( - test_util.vector_path("os-release"))[0], 'systemdos') - self.assertEqual(get_os_info( - test_util.vector_path("os-release"))[1], '42') - self.assertEqual(get_systemd_os_info(os.devnull), ("", "")) - self.assertEqual(get_os_info_ua( - test_util.vector_path("os-release")), "SystemdOS") - with mock.patch('os.path.isfile', return_value=False): - self.assertEqual(get_systemd_os_info(), ("", "")) + @mock.patch("certbot.util.distro") + @unittest.skipUnless(sys.platform.startswith("linux"), "requires Linux") + def test_get_os_info_ua(self, m_distro): + import certbot.util as cbutil + with mock.patch('platform.system_alias', + return_value=('linux', '42', '42')): + m_distro.name.return_value = "" + m_distro.linux_distribution.return_value = ("something", "1.0", "codename") + cbutil.get_python_os_info(pretty=True) + self.assertEqual(cbutil.get_os_info_ua(), + " ".join(cbutil.get_python_os_info(pretty=True))) - def test_systemd_os_release_like(self): - from certbot.util import get_systemd_os_like + m_distro.name.return_value = "whatever" + self.assertEqual(cbutil.get_os_info_ua(), "whatever") - with mock.patch('os.path.isfile', return_value=True): - id_likes = get_systemd_os_like(test_util.vector_path( - "os-release")) - self.assertEqual(len(id_likes), 3) - self.assertTrue("debian" in id_likes) + @mock.patch("certbot.util.distro") + @unittest.skipUnless(sys.platform.startswith("linux"), "requires Linux") + def test_get_os_info(self, m_distro): + import certbot.util as cbutil + with mock.patch("platform.system") as mock_platform: + m_distro.linux_distribution.return_value = ("name", "version", 'x') + mock_platform.return_value = "linux" + self.assertEqual(cbutil.get_os_info(), ("name", "version")) + + m_distro.linux_distribution.return_value = ("something", "else") + self.assertEqual(cbutil.get_os_info(), ("something", "else")) @mock.patch("certbot.util.subprocess.Popen") def test_non_systemd_os_info(self, popen_mock): - from certbot.util import (get_os_info, get_python_os_info, - get_os_info_ua) - with mock.patch('os.path.isfile', return_value=False): + import certbot.util as cbutil + with mock.patch('certbot.util._USE_DISTRO', False): with mock.patch('platform.system_alias', return_value=('NonSystemD', '42', '42')): - self.assertEqual(get_os_info()[0], 'nonsystemd') - self.assertEqual(get_os_info_ua(), - " ".join(get_python_os_info())) + self.assertEqual(cbutil.get_python_os_info()[0], 'nonsystemd') with mock.patch('platform.system_alias', return_value=('darwin', '', '')): comm_mock = mock.Mock() comm_attrs = {'communicate.return_value': - ('42.42.42', 'error')} - comm_mock.configure_mock(**comm_attrs) # pylint: disable=star-args + ('42.42.42', 'error')} + comm_mock.configure_mock(**comm_attrs) popen_mock.return_value = comm_mock - self.assertEqual(get_os_info()[0], 'darwin') - self.assertEqual(get_os_info()[1], '42.42.42') - - with mock.patch('platform.system_alias', - return_value=('linux', '', '')): - with mock.patch('platform.linux_distribution', - return_value=('', '', '')): - self.assertEqual(get_python_os_info(), ("linux", "")) - - with mock.patch('platform.linux_distribution', - return_value=('testdist', '42', '')): - self.assertEqual(get_python_os_info(), ("testdist", "42")) + self.assertEqual(cbutil.get_python_os_info()[0], 'darwin') + self.assertEqual(cbutil.get_python_os_info()[1], '42.42.42') with mock.patch('platform.system_alias', return_value=('freebsd', '9.3-RC3-p1', '')): - self.assertEqual(get_python_os_info(), ("freebsd", "9")) + self.assertEqual(cbutil.get_python_os_info(), ("freebsd", "9")) with mock.patch('platform.system_alias', return_value=('windows', '', '')): with mock.patch('platform.win32_ver', return_value=('4242', '95', '2', '')): - self.assertEqual(get_python_os_info(), - ("windows", "95")) + self.assertEqual(cbutil.get_python_os_info(), + ("windows", "95")) + + @mock.patch("certbot.util.distro") + @unittest.skipUnless(sys.platform.startswith("linux"), "requires Linux") + def test_python_os_info_notfound(self, m_distro): + import certbot.util as cbutil + m_distro.linux_distribution.return_value = ('', '', '') + self.assertEqual(cbutil.get_python_os_info()[0], "linux") + + @mock.patch("certbot.util.distro") + @unittest.skipUnless(sys.platform.startswith("linux"), "requires Linux") + def test_python_os_info_custom(self, m_distro): + import certbot.util as cbutil + m_distro.linux_distribution.return_value = ('testdist', '42', '') + self.assertEqual(cbutil.get_python_os_info(), ("testdist", "42")) class AtexitRegisterTest(unittest.TestCase): @@ -612,11 +579,11 @@ class AtexitRegisterTest(unittest.TestCase): with mock.patch('certbot.util.atexit') as mock_atexit: self._call(self.func, *self.args, **self.kwargs) - # _INITAL_PID must be mocked when calling atexit_func + # _INITIAL_PID must be mocked when calling atexit_func self.assertTrue(mock_atexit.register.called) args, kwargs = mock_atexit.register.call_args atexit_func = args[0] - atexit_func(*args[1:], **kwargs) # pylint: disable=star-args + atexit_func(*args[1:], **kwargs) if __name__ == "__main__": diff --git a/docs/.gitignore b/docs/.gitignore deleted file mode 100644 index ba65b13af..000000000 --- a/docs/.gitignore +++ /dev/null @@ -1 +0,0 @@ -/_build/ diff --git a/docs/_static/.gitignore b/docs/_static/.gitignore deleted file mode 100644 index e69de29bb..000000000 diff --git a/docs/api.rst b/docs/api.rst deleted file mode 100644 index 8668ec5d8..000000000 --- a/docs/api.rst +++ /dev/null @@ -1,8 +0,0 @@ -================= -API Documentation -================= - -.. toctree:: - :glob: - - api/** diff --git a/docs/api/account.rst b/docs/api/account.rst deleted file mode 100644 index fd90230ea..000000000 --- a/docs/api/account.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.account` --------------------------- - -.. automodule:: certbot.account - :members: diff --git a/docs/api/achallenges.rst b/docs/api/achallenges.rst deleted file mode 100644 index 90dda3f06..000000000 --- a/docs/api/achallenges.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.achallenges` ------------------------------- - -.. automodule:: certbot.achallenges - :members: diff --git a/docs/api/auth_handler.rst b/docs/api/auth_handler.rst deleted file mode 100644 index 8819bb1bd..000000000 --- a/docs/api/auth_handler.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.auth_handler` -------------------------------- - -.. automodule:: certbot.auth_handler - :members: diff --git a/docs/api/cert_manager.rst b/docs/api/cert_manager.rst deleted file mode 100644 index c8c86f8b9..000000000 --- a/docs/api/cert_manager.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.cert_manager` -------------------------------- - -.. automodule:: certbot.cert_manager - :members: diff --git a/docs/api/cli.rst b/docs/api/cli.rst deleted file mode 100644 index 91be86758..000000000 --- a/docs/api/cli.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.cli` ----------------------- - -.. automodule:: certbot.cli - :members: diff --git a/docs/api/client.rst b/docs/api/client.rst deleted file mode 100644 index 00a443cd9..000000000 --- a/docs/api/client.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.client` -------------------------- - -.. automodule:: certbot.client - :members: diff --git a/docs/api/configuration.rst b/docs/api/configuration.rst deleted file mode 100644 index 4e99c73d2..000000000 --- a/docs/api/configuration.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.configuration` --------------------------------- - -.. automodule:: certbot.configuration - :members: diff --git a/docs/api/constants.rst b/docs/api/constants.rst deleted file mode 100644 index 99ecc240a..000000000 --- a/docs/api/constants.rst +++ /dev/null @@ -1,9 +0,0 @@ -:mod:`certbot.constants` ------------------------------------ - -.. automodule:: certbot.constants - :members: - :exclude-members: SSL_DHPARAMS_SRC - -.. autodata:: SSL_DHPARAMS_SRC - :annotation: = '/path/to/certbot/ssl-dhparams.pem' diff --git a/docs/api/crypto_util.rst b/docs/api/crypto_util.rst deleted file mode 100644 index 2f473944c..000000000 --- a/docs/api/crypto_util.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.crypto_util` ------------------------------- - -.. automodule:: certbot.crypto_util - :members: diff --git a/docs/api/display.rst b/docs/api/display.rst deleted file mode 100644 index 1a18e6534..000000000 --- a/docs/api/display.rst +++ /dev/null @@ -1,23 +0,0 @@ -:mod:`certbot.display` --------------------------- - -.. automodule:: certbot.display - :members: - -:mod:`certbot.display.util` -=============================== - -.. automodule:: certbot.display.util - :members: - -:mod:`certbot.display.ops` -============================== - -.. automodule:: certbot.display.ops - :members: - -:mod:`certbot.display.enhancements` -======================================= - -.. automodule:: certbot.display.enhancements - :members: diff --git a/docs/api/eff.rst b/docs/api/eff.rst deleted file mode 100644 index 2924b256d..000000000 --- a/docs/api/eff.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.eff` ----------------------- - -.. automodule:: certbot.eff - :members: diff --git a/docs/api/error_handler.rst b/docs/api/error_handler.rst deleted file mode 100644 index f1306177d..000000000 --- a/docs/api/error_handler.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.error_handler` --------------------------------- - -.. automodule:: certbot.error_handler - :members: diff --git a/docs/api/errors.rst b/docs/api/errors.rst deleted file mode 100644 index a9324765b..000000000 --- a/docs/api/errors.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.errors` -------------------------- - -.. automodule:: certbot.errors - :members: diff --git a/docs/api/hooks.rst b/docs/api/hooks.rst deleted file mode 100644 index 140fbf284..000000000 --- a/docs/api/hooks.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.hooks` ------------------------- - -.. automodule:: certbot.hooks - :members: diff --git a/docs/api/index.rst b/docs/api/index.rst deleted file mode 100644 index be94214c9..000000000 --- a/docs/api/index.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot` ------------------- - -.. automodule:: certbot - :members: diff --git a/docs/api/interfaces.rst b/docs/api/interfaces.rst deleted file mode 100644 index 2988b3b87..000000000 --- a/docs/api/interfaces.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.interfaces` ------------------------------ - -.. automodule:: certbot.interfaces - :members: diff --git a/docs/api/lock.rst b/docs/api/lock.rst deleted file mode 100644 index 6dcbf9589..000000000 --- a/docs/api/lock.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.lock` ------------------------ - -.. automodule:: certbot.lock - :members: diff --git a/docs/api/log.rst b/docs/api/log.rst deleted file mode 100644 index 41311de90..000000000 --- a/docs/api/log.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.log` ----------------------- - -.. automodule:: certbot.log - :members: diff --git a/docs/api/main.rst b/docs/api/main.rst deleted file mode 100644 index a555bab01..000000000 --- a/docs/api/main.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.main` ------------------------ - -.. automodule:: certbot.main - :members: diff --git a/docs/api/notify.rst b/docs/api/notify.rst deleted file mode 100644 index fa042b2d2..000000000 --- a/docs/api/notify.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.notify` -------------------------- - -.. automodule:: certbot.notify - :members: diff --git a/docs/api/ocsp.rst b/docs/api/ocsp.rst deleted file mode 100644 index 7044f4052..000000000 --- a/docs/api/ocsp.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.ocsp` ------------------------ - -.. automodule:: certbot.ocsp - :members: diff --git a/docs/api/plugins/common.rst b/docs/api/plugins/common.rst deleted file mode 100644 index 7cfaf8d70..000000000 --- a/docs/api/plugins/common.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.plugins.common` ---------------------------------- - -.. automodule:: certbot.plugins.common - :members: diff --git a/docs/api/plugins/disco.rst b/docs/api/plugins/disco.rst deleted file mode 100644 index 1a27f0f69..000000000 --- a/docs/api/plugins/disco.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.plugins.disco` --------------------------------- - -.. automodule:: certbot.plugins.disco - :members: diff --git a/docs/api/plugins/dns_common.rst b/docs/api/plugins/dns_common.rst deleted file mode 100644 index ee3945e74..000000000 --- a/docs/api/plugins/dns_common.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.plugins.dns_common` ---------------------------------- - -.. automodule:: certbot.plugins.dns_common - :members: diff --git a/docs/api/plugins/dns_common_lexicon.rst b/docs/api/plugins/dns_common_lexicon.rst deleted file mode 100644 index a48166828..000000000 --- a/docs/api/plugins/dns_common_lexicon.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.plugins.dns_common_lexicon` ------------------------------------------ - -.. automodule:: certbot.plugins.dns_common_lexicon - :members: diff --git a/docs/api/plugins/manual.rst b/docs/api/plugins/manual.rst deleted file mode 100644 index eea443499..000000000 --- a/docs/api/plugins/manual.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.plugins.manual` ---------------------------------- - -.. automodule:: certbot.plugins.manual - :members: diff --git a/docs/api/plugins/selection.rst b/docs/api/plugins/selection.rst deleted file mode 100644 index 6211bf9c0..000000000 --- a/docs/api/plugins/selection.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.plugins.selection` ------------------------------------- - -.. automodule:: certbot.plugins.selection - :members: diff --git a/docs/api/plugins/standalone.rst b/docs/api/plugins/standalone.rst deleted file mode 100644 index 60aa48b4f..000000000 --- a/docs/api/plugins/standalone.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.plugins.standalone` -------------------------------------- - -.. automodule:: certbot.plugins.standalone - :members: diff --git a/docs/api/plugins/util.rst b/docs/api/plugins/util.rst deleted file mode 100644 index 30ab3d49f..000000000 --- a/docs/api/plugins/util.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.plugins.util` -------------------------------- - -.. automodule:: certbot.plugins.util - :members: diff --git a/docs/api/plugins/webroot.rst b/docs/api/plugins/webroot.rst deleted file mode 100644 index e1f4523f7..000000000 --- a/docs/api/plugins/webroot.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.plugins.webroot` ----------------------------------- - -.. automodule:: certbot.plugins.webroot - :members: diff --git a/docs/api/renewal.rst b/docs/api/renewal.rst deleted file mode 100644 index 58557351f..000000000 --- a/docs/api/renewal.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.renewal` --------------------------- - -.. automodule:: certbot.renewal - :members: diff --git a/docs/api/reporter.rst b/docs/api/reporter.rst deleted file mode 100644 index ad71dbb69..000000000 --- a/docs/api/reporter.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.reporter` ---------------------------- - -.. automodule:: certbot.reporter - :members: diff --git a/docs/api/reverter.rst b/docs/api/reverter.rst deleted file mode 100644 index 3e0ac750b..000000000 --- a/docs/api/reverter.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.reverter` ---------------------------- - -.. automodule:: certbot.reverter - :members: diff --git a/docs/api/storage.rst b/docs/api/storage.rst deleted file mode 100644 index 34e3a45c0..000000000 --- a/docs/api/storage.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.storage` --------------------------- - -.. automodule:: certbot.storage - :members: diff --git a/docs/api/util.rst b/docs/api/util.rst deleted file mode 100644 index 7d0e33501..000000000 --- a/docs/api/util.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`certbot.util` --------------------------- - -.. automodule:: certbot.util - :members: diff --git a/docs/packaging.rst b/docs/packaging.rst deleted file mode 100644 index c13a14af3..000000000 --- a/docs/packaging.rst +++ /dev/null @@ -1,126 +0,0 @@ -=============== -Packaging Guide -=============== - -Releases -======== - -We release packages and upload them to PyPI (wheels and source tarballs). - -- https://pypi.python.org/pypi/acme -- https://pypi.python.org/pypi/certbot -- https://pypi.python.org/pypi/certbot-apache -- https://pypi.python.org/pypi/certbot-nginx -- https://pypi.python.org/pypi/certbot-dns-cloudflare -- https://pypi.python.org/pypi/certbot-dns-cloudxns -- https://pypi.python.org/pypi/certbot-dns-digitalocean -- https://pypi.python.org/pypi/certbot-dns-dnsimple -- https://pypi.python.org/pypi/certbot-dns-dnsmadeeasy -- https://pypi.python.org/pypi/certbot-dns-google -- https://pypi.python.org/pypi/certbot-dns-linode -- https://pypi.python.org/pypi/certbot-dns-luadns -- https://pypi.python.org/pypi/certbot-dns-nsone -- https://pypi.python.org/pypi/certbot-dns-ovh -- https://pypi.python.org/pypi/certbot-dns-rfc2136 -- https://pypi.python.org/pypi/certbot-dns-route53 - -The following scripts are used in the process: - -- https://github.com/letsencrypt/letsencrypt/blob/master/tools/release.sh - -We use git tags to identify releases, using `Semantic Versioning`_. For -example: `v0.11.1`. - -.. _`Semantic Versioning`: http://semver.org/ - -Notes for package maintainers -============================= - -0. Please use our tagged releases, not ``master``! - -1. Do not package ``certbot-compatibility-test`` or ``letshelp-certbot`` - it's only used internally. - -2. If you'd like to include automated renewal in your package ``certbot renew -q`` should be added to crontab or systemd timer. Additionally you should include a random per-machine time offset to avoid having a large number of your clients hit Let's Encrypt's servers simultaneously. - -3. ``jws`` is an internal script for ``acme`` module and it doesn't have to be packaged - it's mostly for debugging: you can use it as ``echo foo | jws sign | jws verify``. - -4. Do get in touch with us. We are happy to make any changes that will make packaging easier. If you need to apply some patches don't do it downstream - make a PR here. - -Already ongoing efforts -======================= - - -Arch ----- - -From our official releases: - -- https://www.archlinux.org/packages/community/any/python-acme -- https://www.archlinux.org/packages/community/any/certbot -- https://www.archlinux.org/packages/community/any/certbot-apache -- https://www.archlinux.org/packages/community/any/certbot-nginx -- https://www.archlinux.org/packages/community/any/certbot-dns-cloudflare -- https://www.archlinux.org/packages/community/any/certbot-dns-cloudxns -- https://www.archlinux.org/packages/community/any/certbot-dns-digitalocean -- https://www.archlinux.org/packages/community/any/certbot-dns-dnsimple -- https://www.archlinux.org/packages/community/any/certbot-dns-dnsmadeeasy -- https://www.archlinux.org/packages/community/any/certbot-dns-google -- https://www.archlinux.org/packages/community/any/certbot-dns-luadns -- https://www.archlinux.org/packages/community/any/certbot-dns-nsone -- https://www.archlinux.org/packages/community/any/certbot-dns-rfc2136 -- https://www.archlinux.org/packages/community/any/certbot-dns-route53 - -From ``master``: https://aur.archlinux.org/packages/certbot-git - -Debian (and its derivatives, including Ubuntu) ----------------------------------------------- - -- https://packages.debian.org/sid/certbot -- https://packages.debian.org/sid/python-certbot -- https://packages.debian.org/sid/python-certbot-apache - -Fedora ------- - -In Fedora 23+. - -- https://apps.fedoraproject.org/packages/python-acme -- https://apps.fedoraproject.org/packages/certbot -- https://apps.fedoraproject.org/packages/python-certbot-apache -- https://apps.fedoraproject.org/packages/python-certbot-dns-cloudflare -- https://apps.fedoraproject.org/packages/python-certbot-dns-cloudxns -- https://apps.fedoraproject.org/packages/python-certbot-dns-digitalocean -- https://apps.fedoraproject.org/packages/python-certbot-dns-dnsimple -- https://apps.fedoraproject.org/packages/python-certbot-dns-dnsmadeeasy -- https://apps.fedoraproject.org/packages/python-certbot-dns-google -- https://apps.fedoraproject.org/packages/python-certbot-dns-luadns -- https://apps.fedoraproject.org/packages/python-certbot-dns-nsone -- https://apps.fedoraproject.org/packages/python-certbot-dns-rfc2136 -- https://apps.fedoraproject.org/packages/python-certbot-dns-route53 -- https://apps.fedoraproject.org/packages/python-certbot-nginx - -FreeBSD -------- - -- https://www.freshports.org/security/py-acme/ -- https://www.freshports.org/security/py-certbot/ - -Gentoo ------- - -Currently, all ``certbot`` related packages are in the testing branch: - -- https://packages.gentoo.org/packages/app-crypt/certbot -- https://packages.gentoo.org/packages/app-crypt/certbot-apache -- https://packages.gentoo.org/packages/app-crypt/certbot-nginx -- https://packages.gentoo.org/packages/app-crypt/acme - -GNU Guix --------- - -- https://www.gnu.org/software/guix/package-list.html#certbot - -OpenBSD -------- - -- http://cvsweb.openbsd.org/cgi-bin/cvsweb/ports/security/letsencrypt/client/ diff --git a/letsencrypt-auto b/letsencrypt-auto index a79a9c5ae..cea58e2cb 100755 --- a/letsencrypt-auto +++ b/letsencrypt-auto @@ -31,7 +31,7 @@ if [ -z "$VENV_PATH" ]; then fi VENV_BIN="$VENV_PATH/bin" BOOTSTRAP_VERSION_PATH="$VENV_PATH/certbot-auto-bootstrap-version.txt" -LE_AUTO_VERSION="0.30.2" +LE_AUTO_VERSION="1.2.0" BASENAME=$(basename $0) USAGE="Usage: $BASENAME [OPTIONS] A self-updating wrapper script for the Certbot ACME client. When run, updates @@ -45,6 +45,7 @@ Help for certbot itself cannot be provided until it is installed. -h, --help print this help -n, --non-interactive, --noninteractive run without asking for user input --no-bootstrap do not install OS dependencies + --no-permissions-check do not warn about file system permissions --no-self-upgrade do not download updates --os-packages-only install OS dependencies and exit --install-only install certbot, upgrade if needed, and exit @@ -67,6 +68,8 @@ for arg in "$@" ; do # Do not upgrade this script (also prevents client upgrades, because each # copy of the script pins a hash of the python client) NO_SELF_UPGRADE=1;; + --no-permissions-check) + NO_PERMISSIONS_CHECK=1;; --no-bootstrap) NO_BOOTSTRAP=1;; --help) @@ -172,7 +175,11 @@ SetRootAuthMechanism() { sudo) SUDO="sudo -E" ;; - '') ;; # Nothing to do for plain root method. + '') + # If we're not running with root, don't check that this script can only + # be modified by system users and groups. + NO_PERMISSIONS_CHECK=1 + ;; *) error "Error: unknown root authorization mechanism '$LE_AUTO_SUDO'." exit 1 @@ -249,20 +256,28 @@ DeprecationBootstrap() { fi } -MIN_PYTHON_VERSION="2.7" -MIN_PYVER=$(echo "$MIN_PYTHON_VERSION" | sed 's/\.//') +MIN_PYTHON_2_VERSION="2.7" +MIN_PYVER2=$(echo "$MIN_PYTHON_2_VERSION" | sed 's/\.//') +MIN_PYTHON_3_VERSION="3.5" +MIN_PYVER3=$(echo "$MIN_PYTHON_3_VERSION" | sed 's/\.//') # Sets LE_PYTHON to Python version string and PYVER to the first two -# digits of the python version +# digits of the python version. +# MIN_PYVER and MIN_PYTHON_VERSION are also set by this function, and their +# values depend on if we try to use Python 3 or Python 2. DeterminePythonVersion() { # Arguments: "NOCRASH" if we shouldn't crash if we don't find a good python # # If no Python is found, PYVER is set to 0. if [ "$USE_PYTHON_3" = 1 ]; then + MIN_PYVER=$MIN_PYVER3 + MIN_PYTHON_VERSION=$MIN_PYTHON_3_VERSION for LE_PYTHON in "$LE_PYTHON" python3; do # Break (while keeping the LE_PYTHON value) if found. $EXISTS "$LE_PYTHON" > /dev/null && break done else + MIN_PYVER=$MIN_PYVER2 + MIN_PYTHON_VERSION=$MIN_PYTHON_2_VERSION for LE_PYTHON in "$LE_PYTHON" python2.7 python27 python2 python; do # Break (while keeping the LE_PYTHON value) if found. $EXISTS "$LE_PYTHON" > /dev/null && break @@ -278,7 +293,7 @@ DeterminePythonVersion() { fi fi - PYVER=`"$LE_PYTHON" -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//'` + PYVER=$("$LE_PYTHON" -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') if [ "$PYVER" -lt "$MIN_PYVER" ]; then if [ "$1" != "NOCRASH" ]; then error "You have an ancient version of Python entombed in your operating system..." @@ -333,63 +348,11 @@ BootstrapDebCommon() { fi augeas_pkg="libaugeas0 augeas-lenses" - AUGVERSION=`LC_ALL=C apt-cache show --no-all-versions libaugeas0 | grep ^Version: | cut -d" " -f2` if [ "$ASSUME_YES" = 1 ]; then YES_FLAG="-y" fi - AddBackportRepo() { - # ARGS: - BACKPORT_NAME="$1" - BACKPORT_SOURCELINE="$2" - say "To use the Apache Certbot plugin, augeas needs to be installed from $BACKPORT_NAME." - if ! grep -v -e ' *#' /etc/apt/sources.list | grep -q "$BACKPORT_NAME" ; then - # This can theoretically error if sources.list.d is empty, but in that case we don't care. - if ! grep -v -e ' *#' /etc/apt/sources.list.d/* 2>/dev/null | grep -q "$BACKPORT_NAME"; then - if [ "$ASSUME_YES" = 1 ]; then - /bin/echo -n "Installing augeas from $BACKPORT_NAME in 3 seconds..." - sleep 1s - /bin/echo -ne "\e[0K\rInstalling augeas from $BACKPORT_NAME in 2 seconds..." - sleep 1s - /bin/echo -e "\e[0K\rInstalling augeas from $BACKPORT_NAME in 1 second ..." - sleep 1s - add_backports=1 - else - read -p "Would you like to enable the $BACKPORT_NAME repository [Y/n]? " response - case $response in - [yY][eE][sS]|[yY]|"") - add_backports=1;; - *) - add_backports=0;; - esac - fi - if [ "$add_backports" = 1 ]; then - sh -c "echo $BACKPORT_SOURCELINE >> /etc/apt/sources.list.d/$BACKPORT_NAME.list" - apt-get $QUIET_FLAG update - fi - fi - fi - if [ "$add_backports" != 0 ]; then - apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends -t "$BACKPORT_NAME" $augeas_pkg - augeas_pkg= - fi - } - - - if dpkg --compare-versions 1.0 gt "$AUGVERSION" ; then - if lsb_release -a | grep -q wheezy ; then - AddBackportRepo wheezy-backports "deb http://http.debian.net/debian wheezy-backports main" - elif lsb_release -a | grep -q precise ; then - # XXX add ARM case - AddBackportRepo precise-backports "deb http://archive.ubuntu.com/ubuntu precise-backports main restricted universe multiverse" - else - echo "No libaugeas0 version is available that's new enough to run the" - echo "Certbot apache plugin..." - fi - # XXX add a case for ubuntu PPAs - fi - apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends \ python \ python-dev \ @@ -413,7 +376,9 @@ BootstrapDebCommon() { # Sets TOOL to the name of the package manager # Sets appropriate values for YES_FLAG and QUIET_FLAG based on $ASSUME_YES and $QUIET_FLAG. -# Enables EPEL if applicable and possible. +# Note: this function is called both while selecting the bootstrap scripts and +# during the actual bootstrap. Some things like prompting to user can be done in the latter +# case, but not in the former one. InitializeRPMCommonBase() { if type dnf 2>/dev/null then @@ -433,26 +398,6 @@ InitializeRPMCommonBase() { if [ "$QUIET" = 1 ]; then QUIET_FLAG='--quiet' fi - - if ! $TOOL list *virtualenv >/dev/null 2>&1; then - echo "To use Certbot, packages from the EPEL repository need to be installed." - if ! $TOOL list epel-release >/dev/null 2>&1; then - error "Enable the EPEL repository and try running Certbot again." - exit 1 - fi - if [ "$ASSUME_YES" = 1 ]; then - /bin/echo -n "Enabling the EPEL repository in 3 seconds..." - sleep 1s - /bin/echo -ne "\e[0K\rEnabling the EPEL repository in 2 seconds..." - sleep 1s - /bin/echo -e "\e[0K\rEnabling the EPEL repository in 1 second..." - sleep 1s - fi - if ! $TOOL install $YES_FLAG $QUIET_FLAG epel-release; then - error "Could not enable EPEL. Aborting bootstrap!" - exit 1 - fi - fi } BootstrapRpmCommonBase() { @@ -535,19 +480,98 @@ BootstrapRpmCommon() { # If new packages are installed by BootstrapRpmPython3 below, this version # number must be increased. -BOOTSTRAP_RPM_PYTHON3_VERSION=1 +BOOTSTRAP_RPM_PYTHON3_LEGACY_VERSION=1 -BootstrapRpmPython3() { +# Checks if rh-python36 can be installed. +Python36SclIsAvailable() { + InitializeRPMCommonBase >/dev/null 2>&1; + + if "${TOOL}" list rh-python36 >/dev/null 2>&1; then + return 0 + fi + if "${TOOL}" list centos-release-scl >/dev/null 2>&1; then + return 0 + fi + return 1 +} + +# Try to enable rh-python36 from SCL if it is necessary and possible. +EnablePython36SCL() { + if "$EXISTS" python3.6 > /dev/null 2> /dev/null; then + return 0 + fi + if [ ! -f /opt/rh/rh-python36/enable ]; then + return 0 + fi + set +e + if ! . /opt/rh/rh-python36/enable; then + error 'Unable to enable rh-python36!' + exit 1 + fi + set -e +} + +# This bootstrap concerns old RedHat-based distributions that do not ship by default +# with Python 2.7, but only Python 2.6. We bootstrap them by enabling SCL and installing +# Python 3.6. Some of these distributions are: CentOS/RHEL/OL/SL 6. +BootstrapRpmPython3Legacy() { # Tested with: # - CentOS 6 InitializeRPMCommonBase - # EPEL uses python34 - if $TOOL list python34 >/dev/null 2>&1; then - python_pkgs="python34 - python34-devel - python34-tools + if ! "${TOOL}" list rh-python36 >/dev/null 2>&1; then + echo "To use Certbot on this operating system, packages from the SCL repository need to be installed." + if ! "${TOOL}" list centos-release-scl >/dev/null 2>&1; then + error "Enable the SCL repository and try running Certbot again." + exit 1 + fi + if [ "${ASSUME_YES}" = 1 ]; then + /bin/echo -n "Enabling the SCL repository in 3 seconds... (Press Ctrl-C to cancel)" + sleep 1s + /bin/echo -ne "\e[0K\rEnabling the SCL repository in 2 seconds... (Press Ctrl-C to cancel)" + sleep 1s + /bin/echo -e "\e[0K\rEnabling the SCL repository in 1 second... (Press Ctrl-C to cancel)" + sleep 1s + fi + if ! "${TOOL}" install "${YES_FLAG}" "${QUIET_FLAG}" centos-release-scl; then + error "Could not enable SCL. Aborting bootstrap!" + exit 1 + fi + fi + + # CentOS 6 must use rh-python36 from SCL + if "${TOOL}" list rh-python36 >/dev/null 2>&1; then + python_pkgs="rh-python36-python + rh-python36-python-virtualenv + rh-python36-python-devel + " + else + error "No supported Python package available to install. Aborting bootstrap!" + exit 1 + fi + + BootstrapRpmCommonBase "${python_pkgs}" + + # Enable SCL rh-python36 after bootstrapping. + EnablePython36SCL +} + +# If new packages are installed by BootstrapRpmPython3 below, this version +# number must be increased. +BOOTSTRAP_RPM_PYTHON3_VERSION=1 + +BootstrapRpmPython3() { + # Tested with: + # - Fedora 29 + + InitializeRPMCommonBase + + # Fedora 29 must use python3-virtualenv + if $TOOL list python3-virtualenv >/dev/null 2>&1; then + python_pkgs="python3 + python3-virtualenv + python3-devel " else error "No supported Python package available to install. Aborting bootstrap!" @@ -573,10 +597,20 @@ BootstrapSuseCommon() { QUIET_FLAG='-qq' fi + if zypper search -x python-virtualenv >/dev/null 2>&1; then + OPENSUSE_VIRTUALENV_PACKAGES="python-virtualenv" + else + # Since Leap 15.0 (and associated Tumbleweed version), python-virtualenv + # is a source package, and python2-virtualenv must be used instead. + # Also currently python2-setuptools is not a dependency of python2-virtualenv, + # while it should be. Installing it explicitly until upstream fix. + OPENSUSE_VIRTUALENV_PACKAGES="python2-virtualenv python2-setuptools" + fi + zypper $QUIET_FLAG $zypper_flags in $install_flags \ python \ python-devel \ - python-virtualenv \ + $OPENSUSE_VIRTUALENV_PACKAGES \ gcc \ augeas-lenses \ libopenssl-devel \ @@ -783,20 +817,71 @@ elif [ -f /etc/redhat-release ]; then prev_le_python="$LE_PYTHON" unset LE_PYTHON DeterminePythonVersion "NOCRASH" + + RPM_DIST_NAME=`(. /etc/os-release 2> /dev/null && echo $ID) || echo "unknown"` + + if [ "$PYVER" -eq 26 -a $(uname -m) != 'x86_64' ]; then + # 32 bits CentOS 6 and affiliates are not supported anymore by certbot-auto. + DEPRECATED_OS=1 + fi + + # Set RPM_DIST_VERSION to VERSION_ID from /etc/os-release after splitting on + # '.' characters (e.g. "8.0" becomes "8"). If the command exits with an + # error, RPM_DIST_VERSION is set to "unknown". + RPM_DIST_VERSION=$( (. /etc/os-release 2> /dev/null && echo "$VERSION_ID") | cut -d '.' -f1 || echo "unknown") + + # If RPM_DIST_VERSION is an empty string or it contains any nonnumeric + # characters, the value is unexpected so we set RPM_DIST_VERSION to 0. + if [ -z "$RPM_DIST_VERSION" ] || [ -n "$(echo "$RPM_DIST_VERSION" | tr -d '[0-9]')" ]; then + RPM_DIST_VERSION=0 + fi + + # Handle legacy RPM distributions if [ "$PYVER" -eq 26 ]; then + # Check if an automated bootstrap can be achieved on this system. + if ! Python36SclIsAvailable; then + INTERACTIVE_BOOTSTRAP=1 + fi + Bootstrap() { - BootstrapMessage "RedHat-based OSes that will use Python3" - BootstrapRpmPython3 + BootstrapMessage "Legacy RedHat-based OSes that will use Python3" + BootstrapRpmPython3Legacy } USE_PYTHON_3=1 - BOOTSTRAP_VERSION="BootstrapRpmPython3 $BOOTSTRAP_RPM_PYTHON3_VERSION" + BOOTSTRAP_VERSION="BootstrapRpmPython3Legacy $BOOTSTRAP_RPM_PYTHON3_LEGACY_VERSION" + + # Try now to enable SCL rh-python36 for systems already bootstrapped + # NB: EnablePython36SCL has been defined along with BootstrapRpmPython3Legacy in certbot-auto + EnablePython36SCL else - Bootstrap() { - BootstrapMessage "RedHat-based OSes" - BootstrapRpmCommon - } - BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION" + # Starting to Fedora 29, python2 is on a deprecation path. Let's move to python3 then. + # RHEL 8 also uses python3 by default. + if [ "$RPM_DIST_NAME" = "fedora" -a "$RPM_DIST_VERSION" -ge 29 ]; then + RPM_USE_PYTHON_3=1 + elif [ "$RPM_DIST_NAME" = "rhel" -a "$RPM_DIST_VERSION" -ge 8 ]; then + RPM_USE_PYTHON_3=1 + elif [ "$RPM_DIST_NAME" = "centos" -a "$RPM_DIST_VERSION" -ge 8 ]; then + RPM_USE_PYTHON_3=1 + else + RPM_USE_PYTHON_3=0 + fi + + if [ "$RPM_USE_PYTHON_3" = 1 ]; then + Bootstrap() { + BootstrapMessage "RedHat-based OSes that will use Python3" + BootstrapRpmPython3 + } + USE_PYTHON_3=1 + BOOTSTRAP_VERSION="BootstrapRpmPython3 $BOOTSTRAP_RPM_PYTHON3_VERSION" + else + Bootstrap() { + BootstrapMessage "RedHat-based OSes" + BootstrapRpmCommon + } + BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION" + fi fi + LE_PYTHON="$prev_le_python" elif [ -f /etc/os-release ] && `grep -q openSUSE /etc/os-release` ; then Bootstrap() { @@ -871,6 +956,13 @@ if [ "$NO_BOOTSTRAP" = 1 ]; then unset BOOTSTRAP_VERSION fi +if [ "$DEPRECATED_OS" = 1 ]; then + Bootstrap() { + error "Skipping bootstrap because certbot-auto is deprecated on this system." + } + unset BOOTSTRAP_VERSION +fi + # Sets PREV_BOOTSTRAP_VERSION to the identifier for the bootstrap script used # to install OS dependencies on this system. PREV_BOOTSTRAP_VERSION isn't set # if it is unknown how OS dependencies were installed on this system. @@ -940,10 +1032,156 @@ else: UNLIKELY_EOF } +# Create a new virtual environment for Certbot. It will overwrite any existing one. +# Parameters: LE_PYTHON, VENV_PATH, PYVER, VERBOSE +CreateVenv() { + "$1" - "$2" "$3" "$4" << "UNLIKELY_EOF" +#!/usr/bin/env python +import os +import shutil +import subprocess +import sys + + +def create_venv(venv_path, pyver, verbose): + if os.path.exists(venv_path): + shutil.rmtree(venv_path) + + stdout = sys.stdout if verbose == '1' else open(os.devnull, 'w') + + if int(pyver) <= 27: + # Use virtualenv binary + environ = os.environ.copy() + environ['VIRTUALENV_NO_DOWNLOAD'] = '1' + command = ['virtualenv', '--no-site-packages', '--python', sys.executable, venv_path] + subprocess.check_call(command, stdout=stdout, env=environ) + else: + # Use embedded venv module in Python 3 + command = [sys.executable, '-m', 'venv', venv_path] + subprocess.check_call(command, stdout=stdout) + + +if __name__ == '__main__': + create_venv(*sys.argv[1:]) + +UNLIKELY_EOF +} + +# Check that the given PATH_TO_CHECK has secured permissions. +# Parameters: LE_PYTHON, PATH_TO_CHECK +CheckPathPermissions() { + "$1" - "$2" << "UNLIKELY_EOF" +"""Verifies certbot-auto cannot be modified by unprivileged users. + +This script takes the path to certbot-auto as its only command line +argument. It then checks that the file can only be modified by uid/gid +< 1000 and if other users can modify the file, it prints a warning with +a suggestion on how to solve the problem. + +Permissions on symlinks in the absolute path of certbot-auto are ignored +and only the canonical path to certbot-auto is checked. There could be +permissions problems due to the symlinks that are unreported by this +script, however, issues like this were not caused by our documentation +and are ignored for the sake of simplicity. + +All warnings are printed to stdout rather than stderr so all stderr +output from this script can be suppressed to avoid printing messages if +this script fails for some reason. + +""" +from __future__ import print_function + +import os +import stat +import sys + + +FORUM_POST_URL = 'https://community.letsencrypt.org/t/certbot-auto-deployment-best-practices/91979/' + + +def has_safe_permissions(path): + """Returns True if the given path has secure permissions. + + The permissions are considered safe if the file is only writable by + uid/gid < 1000. + + The reason we allow more IDs than 0 is because on some systems such + as Debian, system users/groups other than uid/gid 0 are used for the + path we recommend in our instructions which is /usr/local/bin. 1000 + was chosen because on Debian 0-999 is reserved for system IDs[1] and + on RHEL either 0-499 or 0-999 is reserved depending on the + version[2][3]. Due to these differences across different OSes, this + detection isn't perfect so we only determine permissions are + insecure when we can be reasonably confident there is a problem + regardless of the underlying OS. + + [1] https://www.debian.org/doc/debian-policy/ch-opersys.html#uid-and-gid-classes + [2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/ch-managing_users_and_groups + [3] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-managing_users_and_groups + + :param str path: filesystem path to check + :returns: True if the path has secure permissions, otherwise, False + :rtype: bool + + """ + # os.stat follows symlinks before obtaining information about a file. + stat_result = os.stat(path) + if stat_result.st_mode & stat.S_IWOTH: + return False + if stat_result.st_mode & stat.S_IWGRP and stat_result.st_gid >= 1000: + return False + if stat_result.st_mode & stat.S_IWUSR and stat_result.st_uid >= 1000: + return False + return True + + +def main(certbot_auto_path): + current_path = os.path.realpath(certbot_auto_path) + last_path = None + permissions_ok = True + # This loop makes use of the fact that os.path.dirname('/') == '/'. + while current_path != last_path and permissions_ok: + permissions_ok = has_safe_permissions(current_path) + last_path = current_path + current_path = os.path.dirname(current_path) + + if not permissions_ok: + print('{0} has insecure permissions!'.format(certbot_auto_path)) + print('To learn how to fix them, visit {0}'.format(FORUM_POST_URL)) + + +if __name__ == '__main__': + main(sys.argv[1]) + +UNLIKELY_EOF +} + if [ "$1" = "--le-auto-phase2" ]; then # Phase 2: Create venv, install LE, and run. shift 1 # the --le-auto-phase2 arg + + if [ "$DEPRECATED_OS" = 1 ]; then + # Phase 2 damage control mode for deprecated OSes. + # In this situation, we bypass any bootstrap or certbot venv setup. + error "Your system is not supported by certbot-auto anymore." + + if [ ! -d "$VENV_PATH" ] && OldVenvExists; then + VENV_BIN="$OLD_VENV_PATH/bin" + fi + + if [ -f "$VENV_BIN/letsencrypt" -a "$INSTALL_ONLY" != 1 ]; then + error "Certbot will no longer receive updates." + error "Please visit https://certbot.eff.org/ to check for other alternatives." + "$VENV_BIN/letsencrypt" "$@" + exit 0 + else + error "Certbot cannot be installed." + error "Please visit https://certbot.eff.org/ to check for other alternatives." + exit 1 + fi + fi + SetPrevBootstrapVersion if [ -z "$PHASE_1_VERSION" -a "$USE_PYTHON_3" = 1 ]; then @@ -955,8 +1193,15 @@ if [ "$1" = "--le-auto-phase2" ]; then # If the selected Bootstrap function isn't a noop and it differs from the # previously used version if [ -n "$BOOTSTRAP_VERSION" -a "$BOOTSTRAP_VERSION" != "$PREV_BOOTSTRAP_VERSION" ]; then - # if non-interactive mode or stdin and stdout are connected to a terminal - if [ \( "$NONINTERACTIVE" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then + # Check if we can rebootstrap without manual user intervention: this requires that + # certbot-auto is in non-interactive mode AND selected bootstrap does not claim to + # require a manual user intervention. + if [ "$NONINTERACTIVE" = 1 -a "$INTERACTIVE_BOOTSTRAP" != 1 ]; then + CAN_REBOOTSTRAP=1 + fi + # Check if rebootstrap can be done non-interactively and current shell is non-interactive + # (true if stdin and stdout are not attached to a terminal). + if [ \( "$CAN_REBOOTSTRAP" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then if [ -d "$VENV_PATH" ]; then rm -rf "$VENV_PATH" fi @@ -967,12 +1212,21 @@ if [ "$1" = "--le-auto-phase2" ]; then ln -s "$VENV_PATH" "$OLD_VENV_PATH" fi RerunWithArgs "$@" + # Otherwise bootstrap needs to be done manually by the user. else - error "Skipping upgrade because new OS dependencies may need to be installed." - error - error "To upgrade to a newer version, please run this script again manually so you can" - error "approve changes or with --non-interactive on the command line to automatically" - error "install any required packages." + # If it is because bootstrapping is interactive, --non-interactive will be of no use. + if [ "$INTERACTIVE_BOOTSTRAP" = 1 ]; then + error "Skipping upgrade because new OS dependencies may need to be installed." + error "This requires manual user intervention: please run this script again manually." + # If this is because of the environment (eg. non interactive shell without + # --non-interactive flag set), help the user in that direction. + else + error "Skipping upgrade because new OS dependencies may need to be installed." + error + error "To upgrade to a newer version, please run this script again manually so you can" + error "approve changes or with --non-interactive on the command line to automatically" + error "install any required packages." + fi # Set INSTALLED_VERSION to be the same so we don't update the venv INSTALLED_VERSION="$LE_AUTO_VERSION" # Continue to use OLD_VENV_PATH if the new venv doesn't exist @@ -995,22 +1249,7 @@ if [ "$1" = "--le-auto-phase2" ]; then if [ "$LE_AUTO_VERSION" != "$INSTALLED_VERSION" ]; then say "Creating virtual environment..." DeterminePythonVersion - rm -rf "$VENV_PATH" - if [ "$PYVER" -le 27 ]; then - # Use an environment variable instead of a flag for compatibility with old versions - if [ "$VERBOSE" = 1 ]; then - VIRTUALENV_NO_DOWNLOAD=1 virtualenv --no-site-packages --python "$LE_PYTHON" "$VENV_PATH" - else - VIRTUALENV_NO_DOWNLOAD=1 virtualenv --no-site-packages --python "$LE_PYTHON" "$VENV_PATH" \ - > /dev/null - fi - else - if [ "$VERBOSE" = 1 ]; then - "$LE_PYTHON" -m venv "$VENV_PATH" - else - "$LE_PYTHON" -m venv "$VENV_PATH" > /dev/null - fi - fi + CreateVenv "$LE_PYTHON" "$VENV_PATH" "$PYVER" "$VERBOSE" if [ -n "$BOOTSTRAP_VERSION" ]; then echo "$BOOTSTRAP_VERSION" > "$BOOTSTRAP_VERSION_PATH" @@ -1024,202 +1263,271 @@ if [ "$1" = "--le-auto-phase2" ]; then # There is no $ interpolation due to quotes on starting heredoc delimiter. # ------------------------------------------------------------------------- cat << "UNLIKELY_EOF" > "$TEMP_DIR/letsencrypt-auto-requirements.txt" -# This is the flattened list of packages certbot-auto installs. To generate -# this, do -# `pip install --no-cache-dir -e acme -e . -e certbot-apache -e certbot-nginx`, -# and then use `hashin` or a more secure method to gather the hashes. - -# Hashin example: +# This is the flattened list of packages certbot-auto installs. +# To generate this, do (with docker and package hashin installed): +# ``` +# letsencrypt-auto-source/rebuild_dependencies.py \ +# letsencrypt-auto-source/pieces/dependency-requirements.txt +# ``` +# If you want to update a single dependency, run commands similar to these: +# ``` # pip install hashin # hashin -r dependency-requirements.txt cryptography==1.5.2 -# sets the new certbot-auto pinned version of cryptography to 1.5.2 - -argparse==1.4.0 \ - --hash=sha256:c31647edb69fd3d465a847ea3157d37bed1f95f19760b11a47aa91c04b666314 \ - --hash=sha256:62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4 - -# This comes before cffi because cffi will otherwise install an unchecked -# version via setup_requires. -pycparser==2.14 \ - --hash=sha256:7959b4a74abdc27b312fed1c21e6caf9309ce0b29ea86b591fd2e99ecdf27f73 \ - --no-binary pycparser - -asn1crypto==0.22.0 \ - --hash=sha256:d232509fefcfcdb9a331f37e9c9dc20441019ad927c7d2176cf18ed5da0ba097 \ - --hash=sha256:cbbadd640d3165ab24b06ef25d1dca09a3441611ac15f6a6b452474fdf0aed1a -cffi==1.11.5 \ - --hash=sha256:1b0493c091a1898f1136e3f4f991a784437fac3673780ff9de3bcf46c80b6b50 \ - --hash=sha256:87f37fe5130574ff76c17cab61e7d2538a16f843bb7bca8ebbc4b12de3078596 \ - --hash=sha256:1553d1e99f035ace1c0544050622b7bc963374a00c467edafac50ad7bd276aef \ - --hash=sha256:151b7eefd035c56b2b2e1eb9963c90c6302dc15fbd8c1c0a83a163ff2c7d7743 \ - --hash=sha256:edabd457cd23a02965166026fd9bfd196f4324fe6032e866d0f3bd0301cd486f \ - --hash=sha256:ba5e697569f84b13640c9e193170e89c13c6244c24400fc57e88724ef610cd31 \ - --hash=sha256:79f9b6f7c46ae1f8ded75f68cf8ad50e5729ed4d590c74840471fc2823457d04 \ - --hash=sha256:b0f7d4a3df8f06cf49f9f121bead236e328074de6449866515cea4907bbc63d6 \ - --hash=sha256:4c91af6e967c2015729d3e69c2e51d92f9898c330d6a851bf8f121236f3defd3 \ - --hash=sha256:7a33145e04d44ce95bcd71e522b478d282ad0eafaf34fe1ec5bbd73e662f22b6 \ - --hash=sha256:95d5251e4b5ca00061f9d9f3d6fe537247e145a8524ae9fd30a2f8fbce993b5b \ - --hash=sha256:b75110fb114fa366b29a027d0c9be3709579602ae111ff61674d28c93606acca \ - --hash=sha256:ae5e35a2c189d397b91034642cb0eab0e346f776ec2eb44a49a459e6615d6e2e \ - --hash=sha256:fdf1c1dc5bafc32bc5d08b054f94d659422b05aba244d6be4ddc1c72d9aa70fb \ - --hash=sha256:9d1d3e63a4afdc29bd76ce6aa9d58c771cd1599fbba8cf5057e7860b203710dd \ - --hash=sha256:be2a9b390f77fd7676d80bc3cdc4f8edb940d8c198ed2d8c0be1319018c778e1 \ - --hash=sha256:ed01918d545a38998bfa5902c7c00e0fee90e957ce036a4000a88e3fe2264917 \ - --hash=sha256:857959354ae3a6fa3da6651b966d13b0a8bed6bbc87a0de7b38a549db1d2a359 \ - --hash=sha256:2ba8a45822b7aee805ab49abfe7eec16b90587f7f26df20c71dd89e45a97076f \ - --hash=sha256:a36c5c154f9d42ec176e6e620cb0dd275744aa1d804786a71ac37dc3661a5e95 \ - --hash=sha256:e55e22ac0a30023426564b1059b035973ec82186ddddbac867078435801c7801 \ - --hash=sha256:3eb6434197633b7748cea30bf0ba9f66727cdce45117a712b29a443943733257 \ - --hash=sha256:ecbb7b01409e9b782df5ded849c178a0aa7c906cf8c5a67368047daab282b184 \ - --hash=sha256:770f3782b31f50b68627e22f91cb182c48c47c02eb405fd689472aa7b7aa16dc \ - --hash=sha256:d5d8555d9bfc3f02385c1c37e9f998e2011f0db4f90e250e5bc0c0a85a813085 \ - --hash=sha256:3c85641778460581c42924384f5e68076d724ceac0f267d66c757f7535069c93 \ - --hash=sha256:ca1bd81f40adc59011f58159e4aa6445fc585a32bb8ac9badf7a2c1aa23822f2 \ - --hash=sha256:3bb6bd7266598f318063e584378b8e27c67de998a43362e8fce664c54ee52d30 \ - --hash=sha256:a6a5cb8809091ec9ac03edde9304b3ad82ad4466333432b16d78ef40e0cce0d5 \ - --hash=sha256:57b2533356cb2d8fac1555815929f7f5f14d68ac77b085d2326b571310f34f6e \ - --hash=sha256:495c5c2d43bf6cebe0178eb3e88f9c4aa48d8934aa6e3cddb865c058da76756b \ - --hash=sha256:e90f17980e6ab0f3c2f3730e56d1fe9bcba1891eeea58966e89d352492cc74f4 -ConfigArgParse==0.12.0 \ - --hash=sha256:28cd7d67669651f2a4518367838c49539457504584a139709b2b8f6c208ef339 \ - --no-binary ConfigArgParse +# ``` +ConfigArgParse==1.0 \ + --hash=sha256:bf378245bc9cdc403a527e5b7406b991680c2a530e7e81af747880b54eb57133 +certifi==2019.11.28 \ + --hash=sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3 \ + --hash=sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f +cffi==1.13.2 \ + --hash=sha256:0b49274afc941c626b605fb59b59c3485c17dc776dc3cc7cc14aca74cc19cc42 \ + --hash=sha256:0e3ea92942cb1168e38c05c1d56b0527ce31f1a370f6117f1d490b8dcd6b3a04 \ + --hash=sha256:135f69aecbf4517d5b3d6429207b2dff49c876be724ac0c8bf8e1ea99df3d7e5 \ + --hash=sha256:19db0cdd6e516f13329cba4903368bff9bb5a9331d3410b1b448daaadc495e54 \ + --hash=sha256:2781e9ad0e9d47173c0093321bb5435a9dfae0ed6a762aabafa13108f5f7b2ba \ + --hash=sha256:291f7c42e21d72144bb1c1b2e825ec60f46d0a7468f5346841860454c7aa8f57 \ + --hash=sha256:2c5e309ec482556397cb21ede0350c5e82f0eb2621de04b2633588d118da4396 \ + --hash=sha256:2e9c80a8c3344a92cb04661115898a9129c074f7ab82011ef4b612f645939f12 \ + --hash=sha256:32a262e2b90ffcfdd97c7a5e24a6012a43c61f1f5a57789ad80af1d26c6acd97 \ + --hash=sha256:3c9fff570f13480b201e9ab69453108f6d98244a7f495e91b6c654a47486ba43 \ + --hash=sha256:415bdc7ca8c1c634a6d7163d43fb0ea885a07e9618a64bda407e04b04333b7db \ + --hash=sha256:42194f54c11abc8583417a7cf4eaff544ce0de8187abaf5d29029c91b1725ad3 \ + --hash=sha256:4424e42199e86b21fc4db83bd76909a6fc2a2aefb352cb5414833c030f6ed71b \ + --hash=sha256:4a43c91840bda5f55249413037b7a9b79c90b1184ed504883b72c4df70778579 \ + --hash=sha256:599a1e8ff057ac530c9ad1778293c665cb81a791421f46922d80a86473c13346 \ + --hash=sha256:5c4fae4e9cdd18c82ba3a134be256e98dc0596af1e7285a3d2602c97dcfa5159 \ + --hash=sha256:5ecfa867dea6fabe2a58f03ac9186ea64da1386af2159196da51c4904e11d652 \ + --hash=sha256:62f2578358d3a92e4ab2d830cd1c2049c9c0d0e6d3c58322993cc341bdeac22e \ + --hash=sha256:6471a82d5abea994e38d2c2abc77164b4f7fbaaf80261cb98394d5793f11b12a \ + --hash=sha256:6d4f18483d040e18546108eb13b1dfa1000a089bcf8529e30346116ea6240506 \ + --hash=sha256:71a608532ab3bd26223c8d841dde43f3516aa5d2bf37b50ac410bb5e99053e8f \ + --hash=sha256:74a1d8c85fb6ff0b30fbfa8ad0ac23cd601a138f7509dc617ebc65ef305bb98d \ + --hash=sha256:7b93a885bb13073afb0aa73ad82059a4c41f4b7d8eb8368980448b52d4c7dc2c \ + --hash=sha256:7d4751da932caaec419d514eaa4215eaf14b612cff66398dd51129ac22680b20 \ + --hash=sha256:7f627141a26b551bdebbc4855c1157feeef18241b4b8366ed22a5c7d672ef858 \ + --hash=sha256:8169cf44dd8f9071b2b9248c35fc35e8677451c52f795daa2bb4643f32a540bc \ + --hash=sha256:aa00d66c0fab27373ae44ae26a66a9e43ff2a678bf63a9c7c1a9a4d61172827a \ + --hash=sha256:ccb032fda0873254380aa2bfad2582aedc2959186cce61e3a17abc1a55ff89c3 \ + --hash=sha256:d754f39e0d1603b5b24a7f8484b22d2904fa551fe865fd0d4c3332f078d20d4e \ + --hash=sha256:d75c461e20e29afc0aee7172a0950157c704ff0dd51613506bd7d82b718e7410 \ + --hash=sha256:dcd65317dd15bc0451f3e01c80da2216a31916bdcffd6221ca1202d96584aa25 \ + --hash=sha256:e570d3ab32e2c2861c4ebe6ffcad6a8abf9347432a37608fe1fbd157b3f0036b \ + --hash=sha256:fd43a88e045cf992ed09fa724b5315b790525f2676883a6ea64e3263bae6549d +chardet==3.0.4 \ + --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \ + --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 configobj==5.0.6 \ - --hash=sha256:a2f5650770e1c87fb335af19a9b7eb73fc05ccf22144eb68db7d00cd2bcb0902 \ - --no-binary configobj -cryptography==2.2.2 \ - --hash=sha256:3f3b65d5a16e6b52fba63dc860b62ca9832f51f1a2ae5083c78b6840275f12dd \ - --hash=sha256:5251e7de0de66810833606439ca65c9b9e45da62196b0c88bfadf27740aac09f \ - --hash=sha256:551a3abfe0c8c6833df4192a63371aa2ff43afd8f570ed345d31f251d78e7e04 \ - --hash=sha256:5cb990056b7cadcca26813311187ad751ea644712022a3976443691168781b6f \ - --hash=sha256:60bda7f12ecb828358be53095fc9c6edda7de8f1ef571f96c00b2363643fa3cd \ - --hash=sha256:64b5c67acc9a7c83fbb4b69166f3105a0ab722d27934fac2cb26456718eec2ba \ - --hash=sha256:6fef51ec447fe9f8351894024e94736862900d3a9aa2961528e602eb65c92bdb \ - --hash=sha256:77d0ad229d47a6e0272d00f6bf8ac06ce14715a9fd02c9a97f5a2869aab3ccb2 \ - --hash=sha256:808fe471b1a6b777f026f7dc7bd9a4959da4bfab64972f2bbe91e22527c1c037 \ - --hash=sha256:9b62fb4d18529c84b961efd9187fecbb48e89aa1a0f9f4161c61b7fc42a101bd \ - --hash=sha256:9e5bed45ec6b4f828866ac6a6bedf08388ffcfa68abe9e94b34bb40977aba531 \ - --hash=sha256:9fc295bf69130a342e7a19a39d7bbeb15c0bcaabc7382ec33ef3b2b7d18d2f63 \ - --hash=sha256:abd070b5849ed64e6d349199bef955ee0ad99aefbad792f0c587f8effa681a5e \ - --hash=sha256:ba6a774749b6e510cffc2fb98535f717e0e5fd91c7c99a61d223293df79ab351 \ - --hash=sha256:c332118647f084c983c6a3e1dba0f3bcb051f69d12baccac68db8d62d177eb8a \ - --hash=sha256:d6f46e862ee36df81e6342c2177ba84e70f722d9dc9c6c394f9f1f434c4a5563 \ - --hash=sha256:db6013746f73bf8edd9c3d1d3f94db635b9422f503db3fc5ef105233d4c011ab \ - --hash=sha256:f57008eaff597c69cf692c3518f6d4800f0309253bb138b526a37fe9ef0c7471 \ - --hash=sha256:f6c821ac253c19f2ad4c8691633ae1d1a17f120d5b01ea1d256d7b602bc59887 -enum34==1.1.2 ; python_version < '3.4' \ - --hash=sha256:2475d7fcddf5951e92ff546972758802de5260bf409319a9f1934e6bbc8b1dc7 \ - --hash=sha256:35907defb0f992b75ab7788f65fedc1cf20ffa22688e0e6f6f12afc06b3ea501 + --hash=sha256:a2f5650770e1c87fb335af19a9b7eb73fc05ccf22144eb68db7d00cd2bcb0902 +cryptography==2.8 \ + --hash=sha256:02079a6addc7b5140ba0825f542c0869ff4df9a69c360e339ecead5baefa843c \ + --hash=sha256:1df22371fbf2004c6f64e927668734070a8953362cd8370ddd336774d6743595 \ + --hash=sha256:369d2346db5934345787451504853ad9d342d7f721ae82d098083e1f49a582ad \ + --hash=sha256:3cda1f0ed8747339bbdf71b9f38ca74c7b592f24f65cdb3ab3765e4b02871651 \ + --hash=sha256:44ff04138935882fef7c686878e1c8fd80a723161ad6a98da31e14b7553170c2 \ + --hash=sha256:4b1030728872c59687badcca1e225a9103440e467c17d6d1730ab3d2d64bfeff \ + --hash=sha256:58363dbd966afb4f89b3b11dfb8ff200058fbc3b947507675c19ceb46104b48d \ + --hash=sha256:6ec280fb24d27e3d97aa731e16207d58bd8ae94ef6eab97249a2afe4ba643d42 \ + --hash=sha256:7270a6c29199adc1297776937a05b59720e8a782531f1f122f2eb8467f9aab4d \ + --hash=sha256:73fd30c57fa2d0a1d7a49c561c40c2f79c7d6c374cc7750e9ac7c99176f6428e \ + --hash=sha256:7f09806ed4fbea8f51585231ba742b58cbcfbfe823ea197d8c89a5e433c7e912 \ + --hash=sha256:90df0cc93e1f8d2fba8365fb59a858f51a11a394d64dbf3ef844f783844cc793 \ + --hash=sha256:971221ed40f058f5662a604bd1ae6e4521d84e6cad0b7b170564cc34169c8f13 \ + --hash=sha256:a518c153a2b5ed6b8cc03f7ae79d5ffad7315ad4569b2d5333a13c38d64bd8d7 \ + --hash=sha256:b0de590a8b0979649ebeef8bb9f54394d3a41f66c5584fff4220901739b6b2f0 \ + --hash=sha256:b43f53f29816ba1db8525f006fa6f49292e9b029554b3eb56a189a70f2a40879 \ + --hash=sha256:d31402aad60ed889c7e57934a03477b572a03af7794fa8fb1780f21ea8f6551f \ + --hash=sha256:de96157ec73458a7f14e3d26f17f8128c959084931e8997b9e655a39c8fde9f9 \ + --hash=sha256:df6b4dca2e11865e6cfbfb708e800efb18370f5a46fd601d3755bc7f85b3a8a2 \ + --hash=sha256:ecadccc7ba52193963c0475ac9f6fa28ac01e01349a2ca48509667ef41ffd2cf \ + --hash=sha256:fb81c17e0ebe3358486cd8cc3ad78adbae58af12fc2bf2bc0bb84e8090fa5ce8 +distro==1.4.0 \ + --hash=sha256:362dde65d846d23baee4b5c058c8586f219b5a54be1cf5fc6ff55c4578392f57 \ + --hash=sha256:eedf82a470ebe7d010f1872c17237c79ab04097948800029994fa458e52fb4b4 +enum34==1.1.6 \ + --hash=sha256:2d81cbbe0e73112bdfe6ef8576f2238f2ba27dd0d55752a776c41d38b7da2850 \ + --hash=sha256:644837f692e5f550741432dd3f223bbb9852018674981b1664e5dc339387588a \ + --hash=sha256:6bd0f6ad48ec2aa117d3d141940d484deccda84d4fcd884f5c3d93c23ecd8c79 \ + --hash=sha256:8ad8c4783bf61ded74527bffb48ed9b54166685e4230386a9ed9b1279e2df5b1 funcsigs==1.0.2 \ --hash=sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca \ --hash=sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50 -idna==2.5 \ - --hash=sha256:cc19709fd6d0cbfed39ea875d29ba6d4e22c0cebc510a76d6302a28385e8bb70 \ - --hash=sha256:3cb5ce08046c4e3a560fc02f138d0ac63e00f8ce5901a56b32ec8b7994082aab -ipaddress==1.0.16 \ - --hash=sha256:935712800ce4760701d89ad677666cd52691fd2f6f0b340c8b4239a3c17988a5 \ - --hash=sha256:5a3182b322a706525c46282ca6f064d27a02cffbd449f9f47416f1dc96aa71b0 -josepy==1.1.0 \ - --hash=sha256:1309a25aac3caeff5239729c58ff9b583f7d022ffdb1553406ddfc8e5b52b76e \ - --hash=sha256:fb5c62c77d26e04df29cb5ecd01b9ce69b6fcc9e521eb1ca193b7faa2afa7086 -linecache2==1.0.0 \ - --hash=sha256:e78be9c0a0dfcbac712fe04fbf92b96cddae80b1b842f24248214c8496f006ef \ - --hash=sha256:4b26ff4e7110db76eeb6f5a7b64a82623839d595c2038eeda662f2a2db78e97c -# Using an older version of mock here prevents regressions of #5276. +idna==2.8 \ + --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 \ + --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c +ipaddress==1.0.23 \ + --hash=sha256:6e0f4a39e66cb5bb9a137b00276a2eff74f93b71dcbdad6f10ff7df9d3557fcc \ + --hash=sha256:b7f8e0369580bb4a24d5ba1d7cc29660a4a6987763faf1d8a8046830e020e7e2 +josepy==1.2.0 \ + --hash=sha256:8ea15573203f28653c00f4ac0142520777b1c59d9eddd8da3f256c6ba3cac916 \ + --hash=sha256:9cec9a839fe9520f0420e4f38e7219525daccce4813296627436fe444cd002d3 mock==1.3.0 \ - --hash=sha256:3f573a18be94de886d1191f27c168427ef693e8dcfcecf95b170577b2eb69cbb \ - --hash=sha256:1e247dbecc6ce057299eb7ee019ad68314bb93152e81d9a6110d35f4d5eca0f6 -ordereddict==1.1 \ - --hash=sha256:1c35b4ac206cef2d24816c89f89cf289dd3d38cf7c449bb3fab7bf6d43f01b1f \ - --no-binary ordereddict -packaging==16.8 \ - --hash=sha256:99276dc6e3a7851f32027a68f1095cd3f77c148091b092ea867a351811cfe388 \ - --hash=sha256:5d50835fdf0a7edf0b55e311b7c887786504efea1177abd7e69329a8e5ea619e -parsedatetime==2.1 \ - --hash=sha256:ce9d422165cf6e963905cd5f74f274ebf7cc98c941916169178ef93f0e557838 \ - --hash=sha256:17c578775520c99131634e09cfca5a05ea9e1bd2a05cd06967ebece10df7af2d -pbr==1.8.1 \ - --hash=sha256:46c8db75ae75a056bd1cc07fa21734fe2e603d11a07833ecc1eeb74c35c72e0c \ - --hash=sha256:e2127626a91e6c885db89668976db31020f0af2da728924b56480fc7ccf09649 -pyOpenSSL==16.2.0 \ - --hash=sha256:26ca380ddf272f7556e48064bbcd5bd71f83dfc144f3583501c7ddbd9434ee17 \ - --hash=sha256:7779a3bbb74e79db234af6a08775568c6769b5821faecf6e2f4143edb227516e -pyparsing==2.1.8 \ - --hash=sha256:2f0f5ceb14eccd5aef809d6382e87df22ca1da583c79f6db01675ce7d7f49c18 \ - --hash=sha256:03a4869b9f3493807ee1f1cb405e6d576a1a2ca4d81a982677c0c1ad6177c56b \ - --hash=sha256:ab09aee814c0241ff0c503cff30018219fe1fc14501d89f406f4664a0ec9fbcd \ - --hash=sha256:6e9a7f052f8e26bcf749e4033e3115b6dc7e3c85aafcb794b9a88c9d9ef13c97 \ - --hash=sha256:9f463a6bcc4eeb6c08f1ed84439b17818e2085937c0dee0d7674ac127c67c12b \ - --hash=sha256:3626b4d81cfb300dad57f52f2f791caaf7b06c09b368c0aa7b868e53a5775424 \ - --hash=sha256:367b90cc877b46af56d4580cd0ae278062903f02b8204ab631f5a2c0f50adfd0 \ - --hash=sha256:9f1ea360086cd68681e7f4ca8f1f38df47bf81942a0d76a9673c2d23eff35b13 -pyRFC3339==1.0 \ - --hash=sha256:eea31835c56e2096af4363a5745a784878a61d043e247d3a6d6a0a32a9741f56 \ - --hash=sha256:8dfbc6c458b8daba1c0f3620a8c78008b323a268b27b7359e92a4ae41325f535 + --hash=sha256:1e247dbecc6ce057299eb7ee019ad68314bb93152e81d9a6110d35f4d5eca0f6 \ + --hash=sha256:3f573a18be94de886d1191f27c168427ef693e8dcfcecf95b170577b2eb69cbb +parsedatetime==2.5 \ + --hash=sha256:3b835fc54e472c17ef447be37458b400e3fefdf14bb1ffdedb5d2c853acf4ba1 \ + --hash=sha256:d2e9ddb1e463de871d32088a3f3cea3dc8282b1b2800e081bd0ef86900451667 +pbr==5.4.4 \ + --hash=sha256:139d2625547dbfa5fb0b81daebb39601c478c21956dc57e2e07b74450a8c506b \ + --hash=sha256:61aa52a0f18b71c5cc58232d2cf8f8d09cd67fcad60b742a60124cb8d6951488 +pyOpenSSL==19.1.0 \ + --hash=sha256:621880965a720b8ece2f1b2f54ea2071966ab00e2970ad2ce11d596102063504 \ + --hash=sha256:9a24494b2602aaf402be5c9e30a0b82d4a5c67528fe8fb475e3f3bc00dd69507 +pyRFC3339==1.1 \ + --hash=sha256:67196cb83b470709c580bb4738b83165e67c6cc60e1f2e4f286cfcb402a926f4 \ + --hash=sha256:81b8cbe1519cdb79bed04910dd6fa4e181faf8c88dff1e1b987b5f7ab23a5b1a +pycparser==2.19 \ + --hash=sha256:a988718abfad80b6b157acce7bf130a30876d27603738ac39f140993246b25b3 +pyparsing==2.4.6 \ + --hash=sha256:4c830582a84fb022400b85429791bc551f1f4871c33f23e44f353119e92f969f \ + --hash=sha256:c342dccb5250c08d45fd6f8b4a559613ca603b57498511740e65cd11a2e7dcec python-augeas==0.5.0 \ - --hash=sha256:67d59d66cdba8d624e0389b87b2a83a176f21f16a87553b50f5703b23f29bac2 \ - --no-binary python-augeas -pytz==2015.7 \ - --hash=sha256:3abe6a6d3fc2fbbe4c60144211f45da2edbe3182a6f6511af6bbba0598b1f992 \ - --hash=sha256:939ef9c1e1224d980405689a97ffcf7828c56d1517b31d73464356c1f2b7769e \ - --hash=sha256:ead4aefa7007249e05e51b01095719d5a8dd95760089f5730aac5698b1932918 \ - --hash=sha256:3cca0df08bd0ed98432390494ce3ded003f5e661aa460be7a734bffe35983605 \ - --hash=sha256:3ede470d3d17ba3c07638dfa0d10452bc1b6e5ad326127a65ba77e6aaeb11bec \ - --hash=sha256:68c47964f7186eec306b13629627722b9079cd4447ed9e5ecaecd4eac84ca734 \ - --hash=sha256:dd5d3991950aae40a6c81de1578942e73d629808cefc51d12cd157980e6cfc18 \ - --hash=sha256:a77c52062c07eb7c7b30545dbc73e32995b7e117eea750317b5cb5c7a4618f14 \ - --hash=sha256:81af9aec4bc960a9a0127c488f18772dae4634689233f06f65443e7b11ebeb51 \ - --hash=sha256:e079b1dadc5c06246cc1bb6fe1b23a50b1d1173f2edd5104efd40bb73a28f406 \ - --hash=sha256:fbd26746772c24cb93c8b97cbdad5cb9e46c86bbdb1b9d8a743ee00e2fb1fc5d \ - --hash=sha256:99266ef30a37e43932deec2b7ca73e83c8dbc3b9ff703ec73eca6b1dae6befea \ - --hash=sha256:8b6ce1c993909783bc96e0b4f34ea223bff7a4df2c90bdb9c4e0f1ac928689e3 -requests==2.20.0 \ - --hash=sha256:99dcfdaaeb17caf6e526f32b6a7b780461512ab3f1d992187801694cba42770c \ - --hash=sha256:a84b8c9ab6239b578f22d1c21d51b696dcfe004032bb80ea832398d6909d7279 -six==1.10.0 \ - --hash=sha256:0ff78c403d9bccf5a425a6d31a12aa6b47f1c21ca4dc2573a7e2f32a97335eb1 \ - --hash=sha256:105f8d68616f8248e24bf0e9372ef04d3cc10104f1980f54d57b2ce73a5ad56a -traceback2==1.4.0 \ - --hash=sha256:8253cebec4b19094d67cc5ed5af99bf1dba1285292226e98a31929f87a5d6b23 \ - --hash=sha256:05acc67a09980c2ecfedd3423f7ae0104839eccb55fc645773e1caa0951c3030 -unittest2==1.1.0 \ - --hash=sha256:13f77d0875db6d9b435e1d4f41e74ad4cc2eb6e1d5c824996092b3430f088bb8 \ - --hash=sha256:22882a0e418c284e1f718a822b3b022944d53d2d908e1690b319a9d3eb2c0579 -zope.component==4.2.2 \ - --hash=sha256:282c112b55dd8e3c869a3571f86767c150ab1284a9ace2bdec226c592acaf81a \ - --no-binary zope.component -zope.event==4.1.0 \ - --hash=sha256:dc7a59a2fd91730d3793131a5d261b29e93ec4e2a97f1bc487ce8defee2fe786 \ - --no-binary zope.event -zope.interface==4.1.3 \ - --hash=sha256:f07b631f7a601cd8cbd3332d54f43142c7088a83299f859356f08d1d4d4259b3 \ - --hash=sha256:de5cca083b9439d8002fb76bbe6b4998c5a5a721fab25b84298967f002df4c94 \ - --hash=sha256:6788416f7ea7f5b8a97be94825377aa25e8bdc73463e07baaf9858b29e737077 \ - --hash=sha256:6f3230f7254518201e5a3708cbb2de98c848304f06e3ded8bfb39e5825cba2e1 \ - --hash=sha256:5fa575a5240f04200c3088427d0d4b7b737f6e9018818a51d8d0f927a6a2517a \ - --hash=sha256:522194ad6a545735edd75c8a83f48d65d1af064e432a7d320d64f56bafc12e99 \ - --hash=sha256:e8c7b2d40943f71c99148c97f66caa7f5134147f57423f8db5b4825099ce9a09 \ - --hash=sha256:279024f0208601c3caa907c53876e37ad88625f7eaf1cb3842dbe360b2287017 \ - --hash=sha256:2e221a9eec7ccc58889a278ea13dcfed5ef939d80b07819a9a8b3cb1c681484f \ - --hash=sha256:69118965410ec86d44dc6b9017ee3ddbd582e0c0abeef62b3a19dbf6c8ad132b \ - --hash=sha256:d04df8686ec864d0cade8cf199f7f83aecd416109a20834d568f8310ded12dea \ - --hash=sha256:e75a947e15ee97e7e71e02ea302feb2fc62d3a2bb4668bf9dfbed43a506ac7e7 \ - --hash=sha256:4e45d22fb883222a5ab9f282a116fec5ee2e8d1a568ccff6a2d75bbd0eb6bcfc \ - --hash=sha256:bce9339bb3c7a55e0803b63d21c5839e8e479bc85c4adf42ae415b72f94facb2 \ - --hash=sha256:928138365245a0e8869a5999fbcc2a45475a0a6ed52a494d60dbdc540335fedd \ - --hash=sha256:0d841ba1bb840eea0e6489dc5ecafa6125554971f53b5acb87764441e61bceba \ - --hash=sha256:b09c8c1d47b3531c400e0195697f1414a63221de6ef478598a4f1460f7d9a392 -requests-toolbelt==0.8.0 \ - --hash=sha256:42c9c170abc2cacb78b8ab23ac957945c7716249206f90874651971a4acff237 \ - --hash=sha256:f6a531936c6fa4c6cfce1b9c10d5c4f498d16528d2a54a22ca00011205a187b5 -chardet==3.0.2 \ - --hash=sha256:4f7832e7c583348a9eddd927ee8514b3bf717c061f57b21dbe7697211454d9bb \ - --hash=sha256:6ebf56457934fdce01fb5ada5582762a84eed94cad43ed877964aebbdd8174c0 -urllib3==1.24.1 \ - --hash=sha256:61bf29cada3fc2fbefad4fdf059ea4bd1b4a86d2b6d15e1c7c0b582b9752fe39 \ - --hash=sha256:de9529817c93f27c8ccbfead6985011db27bd0ddfcdb2d86f3f663385c6a9c22 -certifi==2017.4.17 \ - --hash=sha256:f4318671072f030a33c7ca6acaef720ddd50ff124d1388e50c1bda4cbd6d7010 \ - --hash=sha256:f7527ebf7461582ce95f7a9e03dd141ce810d40590834f4ec20cddd54234c10a + --hash=sha256:67d59d66cdba8d624e0389b87b2a83a176f21f16a87553b50f5703b23f29bac2 +pytz==2019.3 \ + --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \ + --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be +requests==2.22.0 \ + --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 \ + --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31 +requests-toolbelt==0.9.1 \ + --hash=sha256:380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f \ + --hash=sha256:968089d4584ad4ad7c171454f0a5c6dac23971e9472521ea3b6d49d610aa6fc0 +six==1.14.0 \ + --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \ + --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c +urllib3==1.25.8 \ + --hash=sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc \ + --hash=sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc +zope.component==4.6 \ + --hash=sha256:ec2afc5bbe611dcace98bb39822c122d44743d635dafc7315b9aef25097db9e6 +zope.deferredimport==4.3.1 \ + --hash=sha256:57b2345e7b5eef47efcd4f634ff16c93e4265de3dcf325afc7315ade48d909e1 \ + --hash=sha256:9a0c211df44aa95f1c4e6d2626f90b400f56989180d3ef96032d708da3d23e0a +zope.deprecation==4.4.0 \ + --hash=sha256:0d453338f04bacf91bbfba545d8bcdf529aa829e67b705eac8c1a7fdce66e2df \ + --hash=sha256:f1480b74995958b24ce37b0ef04d3663d2683e5d6debc96726eff18acf4ea113 +zope.event==4.4 \ + --hash=sha256:69c27debad9bdacd9ce9b735dad382142281ac770c4a432b533d6d65c4614bcf \ + --hash=sha256:d8e97d165fd5a0997b45f5303ae11ea3338becfe68c401dd88ffd2113fe5cae7 +zope.hookable==5.0.0 \ + --hash=sha256:0992a0dd692003c09fb958e1480cebd1a28f2ef32faa4857d864f3ca8e9d6952 \ + --hash=sha256:0f325838dbac827a1e2ed5d482c1f2656b6844dc96aa098f7727e76395fcd694 \ + --hash=sha256:22a317ba00f61bac99eac1a5e330be7cb8c316275a21269ec58aa396b602af0c \ + --hash=sha256:25531cb5e7b35e8a6d1d6eddef624b9a22ce5dcf8f4448ef0f165acfa8c3fc21 \ + --hash=sha256:30890892652766fc80d11f078aca9a5b8150bef6b88aba23799581a53515c404 \ + --hash=sha256:342d682d93937e5b8c232baffb32a87d5eee605d44f74566657c64a239b7f342 \ + --hash=sha256:46b2fddf1f5aeb526e02b91f7e62afbb9fff4ffd7aafc97cdb00a0d717641567 \ + --hash=sha256:523318ff96df9b8d378d997c00c5d4cbfbff68dc48ff5ee5addabdb697d27528 \ + --hash=sha256:53aa02eb8921d4e667c69d76adeed8fe426e43870c101cb08dcd2f3468aff742 \ + --hash=sha256:62e79e8fdde087cb20822d7874758f5acbedbffaf3c0fbe06309eb8a41ee4e06 \ + --hash=sha256:74bf2f757f7385b56dc3548adae508d8b3ef952d600b4b12b88f7d1706b05dcc \ + --hash=sha256:751ee9d89eb96e00c1d7048da9725ce392a708ed43406416dc5ed61e4d199764 \ + --hash=sha256:7b83bc341e682771fe810b360cd5d9c886a948976aea4b979ff214e10b8b523b \ + --hash=sha256:81eeeb27dbb0ddaed8070daee529f0d1bfe4f74c7351cce2aaca3ea287c4cc32 \ + --hash=sha256:856509191e16930335af4d773c0fc31a17bae8991eb6f167a09d5eddf25b56cc \ + --hash=sha256:8853e81fd07b18fa9193b19e070dc0557848d9945b1d2dac3b7782543458c87d \ + --hash=sha256:94506a732da2832029aecdfe6ea07eb1b70ee06d802fff34e1b3618fe7cdf026 \ + --hash=sha256:95ad874a8cc94e786969215d660143817f745225579bfe318c4676e218d3147c \ + --hash=sha256:9758ec9174966ffe5c499b6c3d149f80aa0a9238020006a2b87c6af5963fcf48 \ + --hash=sha256:a169823e331da939aa7178fc152e65699aeb78957e46c6f80ccb50ee4c3616c2 \ + --hash=sha256:a67878a798f6ca292729a28c2226592b3d000dc6ee7825d31887b553686c7ac7 \ + --hash=sha256:a9a6d9eb2319a09905670810e2de971d6c49013843700b4975e2fc0afe96c8db \ + --hash=sha256:b3e118b58a3d2301960e6f5f25736d92f6b9f861728d3b8c26d69f54d8a157d2 \ + --hash=sha256:ca6705c2a1fb5059a4efbe9f5426be4cdf71b3c9564816916fc7aa7902f19ede \ + --hash=sha256:cf711527c9d4ae72085f137caffb4be74fc007ffb17cd103628c7d5ba17e205f \ + --hash=sha256:d087602a6845ebe9d5a1c5a949fedde2c45f372d77fbce4f7fe44b68b28a1d03 \ + --hash=sha256:d1080e1074ddf75ad6662a9b34626650759c19a9093e1a32a503d37e48da135b \ + --hash=sha256:db9c60368aff2b7e6c47115f3ad9bd6e96aa298b12ed5f8cb13f5673b30be565 \ + --hash=sha256:dbeb127a04473f5a989169eb400b67beb921c749599b77650941c21fe39cb8d9 \ + --hash=sha256:dca336ca3682d869d291d7cd18284f6ff6876e4244eb1821430323056b000e2c \ + --hash=sha256:dd69a9be95346d10c853b6233fcafe3c0315b89424b378f2ad45170d8e161568 \ + --hash=sha256:dd79f8fae5894f1ee0a0042214685f2d039341250c994b825c10a4cd075d80f6 \ + --hash=sha256:e647d850aa1286d98910133cee12bd87c354f7b7bb3f3cd816a62ba7fa2f7007 \ + --hash=sha256:f37a210b5c04b2d4e4bac494ab15b70196f219a1e1649ddca78560757d4278fb \ + --hash=sha256:f67820b6d33a705dc3c1c457156e51686f7b350ff57f2112e1a9a4dad38ec268 \ + --hash=sha256:f68969978ccf0e6123902f7365aae5b7a9e99169d4b9105c47cf28e788116894 \ + --hash=sha256:f717a0b34460ae1ac0064e91b267c0588ac2c098ffd695992e72cd5462d97a67 \ + --hash=sha256:f9d58ccec8684ca276d5a4e7b0dfacca028336300a8f715d616d9f0ce9ae8096 \ + --hash=sha256:fcc3513a54e656067cbf7b98bab0d6b9534b9eabc666d1f78aad6acdf0962736 +zope.interface==4.7.1 \ + --hash=sha256:048b16ac882a05bc7ef534e8b9f15c9d7a6c190e24e8938a19b7617af4ed854a \ + --hash=sha256:05816cf8e7407cf62f2ec95c0a5d69ec4fa5741d9ccd10db9f21691916a9a098 \ + --hash=sha256:065d6a1ac89d35445168813bed45048ed4e67a4cdfc5a68fdb626a770378869f \ + --hash=sha256:14157421f4121a57625002cc4f48ac7521ea238d697c4a4459a884b62132b977 \ + --hash=sha256:18dc895945694f397a0be86be760ff664b790f95d8e7752d5bab80284ff9105d \ + --hash=sha256:1962c9f838bd6ae4075d0014f72697510daefc7e1c7e48b2607df0b6e157989c \ + --hash=sha256:1a67408cacd198c7e6274a19920bb4568d56459e659e23c4915528686ac1763a \ + --hash=sha256:21bf781076dd616bd07cf0223f79d61ab4f45176076f90bc2890e18c48195da4 \ + --hash=sha256:21c0a5d98650aebb84efa16ce2c8df1a46bdc4fe8a9e33237d0ca0b23f416ead \ + --hash=sha256:23cfeea25d1e42ff3bf4f9a0c31e9d5950aa9e7c4b12f0c4bd086f378f7b7a71 \ + --hash=sha256:24b6fce1fb71abf9f4093e3259084efcc0ef479f89356757780685bd2b06ef37 \ + --hash=sha256:24f84ce24eb6b5fcdcb38ad9761524f1ae96f7126abb5e597f8a3973d9921409 \ + --hash=sha256:25e0ef4a824017809d6d8b0ce4ab3288594ba283e4d4f94d8cfb81d73ed65114 \ + --hash=sha256:2e8fdd625e9aba31228e7ddbc36bad5c38dc3ee99a86aa420f89a290bd987ce9 \ + --hash=sha256:2f3bc2f49b67b1bea82b942d25bc958d4f4ea6709b411cb2b6b9718adf7914ce \ + --hash=sha256:35d24be9d04d50da3a6f4d61de028c1dd087045385a0ff374d93ef85af61b584 \ + --hash=sha256:35dbe4e8c73003dff40dfaeb15902910a4360699375e7b47d3c909a83ff27cd0 \ + --hash=sha256:3dfce831b824ab5cf446ed0c350b793ac6fa5fe33b984305cb4c966a86a8fb79 \ + --hash=sha256:3f7866365df5a36a7b8de8056cd1c605648f56f9a226d918ed84c85d25e8d55f \ + --hash=sha256:455cc8c01de3bac6f9c223967cea41f4449f58b4c2e724ec8177382ddd183ab4 \ + --hash=sha256:4bb937e998be9d5e345f486693e477ba79e4344674484001a0b646be1d530487 \ + --hash=sha256:52303a20902ca0888dfb83230ca3ee6fbe63c0ad1dd60aa0bba7958ccff454d8 \ + --hash=sha256:6e0a897d4e09859cc80c6a16a29697406ead752292ace17f1805126a4f63c838 \ + --hash=sha256:6e1816e7c10966330d77af45f77501f9a68818c065dec0ad11d22b50a0e212e7 \ + --hash=sha256:73b5921c5c6ce3358c836461b5470bf675601c96d5e5d8f2a446951470614f67 \ + --hash=sha256:8093cd45cdb5f6c8591cfd1af03d32b32965b0f79b94684cd0c9afdf841982bb \ + --hash=sha256:864b4a94b60db301899cf373579fd9ef92edddbf0fb2cd5ae99f53ef423ccc56 \ + --hash=sha256:8a27b4d3ea9c6d086ce8e7cdb3e8d319b6752e2a03238a388ccc83ccbe165f50 \ + --hash=sha256:91b847969d4784abd855165a2d163f72ac1e58e6dce09a5e46c20e58f19cc96d \ + --hash=sha256:b47b1028be4758c3167e474884ccc079b94835f058984b15c145966c4df64d27 \ + --hash=sha256:b68814a322835d8ad671b7acc23a3b2acecba527bb14f4b53fc925f8a27e44d8 \ + --hash=sha256:bcb50a032c3b6ec7fb281b3a83d2b31ab5246c5b119588725b1350d3a1d9f6a3 \ + --hash=sha256:c56db7d10b25ce8918b6aec6b08ac401842b47e6c136773bfb3b590753f7fb67 \ + --hash=sha256:c94b77a13d4f47883e4f97f9fa00f5feadd38af3e6b3c7be45cfdb0a14c7149b \ + --hash=sha256:db381f6fdaef483ad435f778086ccc4890120aff8df2ba5cfeeac24d280b3145 \ + --hash=sha256:e6487d01c8b7ed86af30ea141fcc4f93f8a7dde26f94177c1ad637c353bd5c07 \ + --hash=sha256:e86923fa728dfba39c5bb6046a450bd4eec8ad949ac404eca728cfce320d1732 \ + --hash=sha256:f6ca36dc1e9eeb46d779869c60001b3065fb670b5775c51421c099ea2a77c3c9 \ + --hash=sha256:fb62f2cbe790a50d95593fb40e8cca261c31a2f5637455ea39440d6457c2ba25 +zope.proxy==4.3.3 \ + --hash=sha256:04646ac04ffa9c8e32fb2b5c3cd42995b2548ea14251f3c21ca704afae88e42c \ + --hash=sha256:07b6bceea232559d24358832f1cd2ed344bbf05ca83855a5b9698b5f23c5ed60 \ + --hash=sha256:1ef452cc02e0e2f8e3c917b1a5b936ef3280f2c2ca854ee70ac2164d1655f7e6 \ + --hash=sha256:22bf61857c5977f34d4e391476d40f9a3b8c6ab24fb0cac448d42d8f8b9bf7b2 \ + --hash=sha256:299870e3428cbff1cd9f9b34144e76ecdc1d9e3192a8cf5f1b0258f47a239f58 \ + --hash=sha256:2bfc36bfccbe047671170ea5677efd3d5ab730a55d7e45611d76d495e5b96766 \ + --hash=sha256:32e82d5a640febc688c0789e15ea875bf696a10cf358f049e1ed841f01710a9b \ + --hash=sha256:3b2051bdc4bc3f02fa52483f6381cf40d4d48167645241993f9d7ebbd142ed9b \ + --hash=sha256:3f734bd8a08f5185a64fb6abb8f14dc97ec27a689ca808fb7a83cdd38d745e4f \ + --hash=sha256:3f78dd8de3112df8bbd970f0916ac876dc3fbe63810bd1cf7cc5eec4cbac4f04 \ + --hash=sha256:4eabeb48508953ba1f3590ad0773b8daea9e104eec66d661917e9bbcd7125a67 \ + --hash=sha256:4f05ecc33808187f430f249cb1ccab35c38f570b181f2d380fbe253da94b18d8 \ + --hash=sha256:4f4f4cbf23d3afc1526294a31e7b3eaa0f682cc28ac5366065dc1d6bb18bd7be \ + --hash=sha256:5483d5e70aacd06f0aa3effec9fed597c0b50f45060956eeeb1203c44d4338c3 \ + --hash=sha256:56a5f9b46892b115a75d0a1f2292431ad5988461175826600acc69a24cb3edee \ + --hash=sha256:64bb63af8a06f736927d260efdd4dfc5253d42244f281a8063e4b9eea2ddcbc5 \ + --hash=sha256:653f8cbefcf7c6ac4cece2cdef367c4faa2b7c19795d52bd7cbec11a8739a7c1 \ + --hash=sha256:664211d63306e4bd4eec35bf2b4bd9db61c394037911cf2d1804c43b511a49f1 \ + --hash=sha256:6651e6caed66a8fff0fef1a3e81c0ed2253bf361c0fdc834500488732c5d16e9 \ + --hash=sha256:6c1fba6cdfdf105739d3069cf7b07664f2944d82a8098218ab2300a82d8f40fc \ + --hash=sha256:6e64246e6e9044a4534a69dca1283c6ddab6e757be5e6874f69024329b3aa61f \ + --hash=sha256:838390245c7ec137af4993c0c8052f49d5ec79e422b4451bfa37fee9b9ccaa01 \ + --hash=sha256:856b410a14793069d8ba35f33fff667213ea66f2df25a0024cc72a7493c56d4c \ + --hash=sha256:8b932c364c1d1605a91907a41128ed0ee8a2d326fc0fafb2c55cd46f545f4599 \ + --hash=sha256:9086cf6d20f08dae7f296a78f6c77d1f8d24079d448f023ee0eb329078dd35e1 \ + --hash=sha256:9698533c14afa0548188de4968a7932d1f3f965f3f5ba1474de673596bb875af \ + --hash=sha256:9b12b05dd7c28f5068387c1afee8cb94f9d02501e7ef495a7c5c7e27139b96ad \ + --hash=sha256:a884c7426a5bc6fb7fc71a55ad14e66818e13f05b78b20a6f37175f324b7acb8 \ + --hash=sha256:abe9e7f1a3e76286c5f5baf2bf5162d41dc0310da493b34a2c36555f38d928f7 \ + --hash=sha256:bd6fde63b015a27262be06bd6bbdd895273cc2bdf2d4c7e1c83711d26a8fbace \ + --hash=sha256:bda7c62c954f47b87ed9a89f525eee1b318ec7c2162dfdba76c2ccfa334e0caa \ + --hash=sha256:be8a4908dd3f6e965993c0068b006bdbd0474fbcbd1da4893b49356e73fc1557 \ + --hash=sha256:ced65fc3c7d7205267506d854bb1815bb445899cca9d21d1d4b949070a635546 \ + --hash=sha256:dac4279aa05055d3897ab5e5ee5a7b39db121f91df65a530f8b1ac7f9bd93119 \ + --hash=sha256:e4f1863056e3e4f399c285b67fa816f411a7bfa1c81ef50e186126164e396e59 \ + --hash=sha256:ecd85f68b8cd9ab78a0141e87ea9a53b2f31fd9b1350a1c44da1f7481b5363ef \ + --hash=sha256:ed269b83750413e8fc5c96276372f49ee3fcb7ed61c49fe8e5a67f54459a5a4a \ + --hash=sha256:f19b0b80cba73b204dee68501870b11067711d21d243fb6774256d3ca2e5391f \ + --hash=sha256:ffdafb98db7574f9da84c489a10a5d582079a888cb43c64e9e6b0e3fe1034685 # Contains the requirements for the letsencrypt package. # @@ -1232,18 +1540,18 @@ letsencrypt==0.7.0 \ --hash=sha256:105a5fb107e45bcd0722eb89696986dcf5f08a86a321d6aef25a0c7c63375ade \ --hash=sha256:c36e532c486a7e92155ee09da54b436a3c420813ec1c590b98f635d924720de9 -certbot==0.30.2 \ - --hash=sha256:e411b72fa86eec1018e6de28e649e8c9c71191a7431dcc77f207b57ca9484c11 \ - --hash=sha256:534487cb552ced8e47948ba3d2e7ca12c3a439133fc609485012b1a02fc7776e -acme==0.30.2 \ - --hash=sha256:68982576492dfa99c7e2be0fce4371adc9344740b05420ce0ab53238d2bb9b3b \ - --hash=sha256:295a5b7fce9f908e6e5cff8c40be1a3daf3e1ebabd2e139a4c87274e68eeb8f2 -certbot-apache==0.30.2 \ - --hash=sha256:3b7fa4e59772da7c9975ef2a49ceff157c9d7cb31eb9475928b5986d89701a3a \ - --hash=sha256:32fa915a8a51810fdfe828ac1361da4425c231d7384891e49e6338e4741464b2 -certbot-nginx==0.30.2 \ - --hash=sha256:7dc785f6f0c0c57b19cea8d98f9ea8feef53945613967b52c9348c81327010e2 \ - --hash=sha256:6ba4dd772d0c7cdfb3383ca325b35639e01ac9e142e4baa6445cd85c7fb59552 +certbot==1.2.0 \ + --hash=sha256:e25c17125c00b3398c8e9b9d54ef473c0e8f5aff53389f313a51b06cf472d335 \ + --hash=sha256:95dcbae085f8e4eb18442fe7b12994b08964a9a6e8e352e556cdb4a8a625373c +acme==1.2.0 \ + --hash=sha256:284d22fde75687a8ea72d737cac6bcbdc91f3c796221aa25378b8732ba6f6875 \ + --hash=sha256:0630c740d49bda945e97bd35fc8d6f02d082c8cb9e18f8fec0dbb3d395ac26ab +certbot-apache==1.2.0 \ + --hash=sha256:3f7493918353d3bd6067d446a2cf263e03831c4c10ec685b83d644b47767090d \ + --hash=sha256:b46e9def272103a68108e48bf7e410ea46801529b1ea6954f6506b14dd9df9b3 +certbot-nginx==1.2.0 \ + --hash=sha256:efd32a2b32f2439279da446b6bf67684f591f289323c5f494ebfd86a566a28fd \ + --hash=sha256:6fd7cf4f2545ad66e57000343227df9ccccaf04420e835e05cb3250fac1fa6db UNLIKELY_EOF # ------------------------------------------------------------------------- @@ -1273,7 +1581,6 @@ from distutils.version import StrictVersion from hashlib import sha256 from os import environ from os.path import join -from pipes import quote from shutil import rmtree try: from subprocess import check_output @@ -1293,7 +1600,7 @@ except ImportError: cmd = popenargs[0] raise CalledProcessError(retcode, cmd) return output -from sys import exit, version_info +import sys from tempfile import mkdtemp try: from urllib2 import build_opener, HTTPHandler, HTTPSHandler @@ -1315,7 +1622,7 @@ maybe_argparse = ( [('18/dd/e617cfc3f6210ae183374cd9f6a26b20514bbb5a792af97949c5aacddf0f/' 'argparse-1.4.0.tar.gz', '62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4')] - if version_info < (2, 7, 0) else []) + if sys.version_info < (2, 7, 0) else []) PACKAGES = maybe_argparse + [ @@ -1396,7 +1703,8 @@ def get_index_base(): def main(): - pip_version = StrictVersion(check_output(['pip', '--version']) + python = sys.executable or 'python' + pip_version = StrictVersion(check_output([python, '-m', 'pip', '--version']) .decode('utf-8').split()[1]) has_pip_cache = pip_version >= StrictVersion('6.0') index_base = get_index_base() @@ -1406,12 +1714,12 @@ def main(): temp, digest) for path, digest in PACKAGES] - check_output('pip install --no-index --no-deps -U ' + - # Disable cache since we're not using it and it otherwise - # sometimes throws permission warnings: - ('--no-cache-dir ' if has_pip_cache else '') + - ' '.join(quote(d) for d in downloads), - shell=True) + # Calling pip as a module is the preferred way to avoid problems about pip self-upgrade. + command = [python, '-m', 'pip', 'install', '--no-index', '--no-deps', '-U'] + # Disable cache since it is not used and it otherwise sometimes throws permission warnings: + command.extend(['--no-cache-dir'] if has_pip_cache else []) + command.extend(downloads) + check_output(command) except HashError as exc: print(exc) except Exception: @@ -1424,7 +1732,7 @@ def main(): if __name__ == '__main__': - exit(main()) + sys.exit(main()) UNLIKELY_EOF # ------------------------------------------------------------------------- @@ -1477,6 +1785,9 @@ UNLIKELY_EOF say "Installation succeeded." fi + # If you're modifying any of the code after this point in this current `if` block, you + # may need to update the "$DEPRECATED_OS" = 1 case at the beginning of phase 2 as well. + if [ "$INSTALL_ONLY" = 1 ]; then say "Certbot is installed." exit 0 @@ -1508,6 +1819,24 @@ else exit 0 fi + DeterminePythonVersion "NOCRASH" + # Don't warn about file permissions if the user disabled the check or we + # can't find an up-to-date Python. + if [ "$PYVER" -ge "$MIN_PYVER" -a "$NO_PERMISSIONS_CHECK" != 1 ]; then + # If the script fails for some reason, don't break certbot-auto. + set +e + # Suppress unexpected error output. + CHECK_PERM_OUT=$(CheckPathPermissions "$LE_PYTHON" "$0" 2>/dev/null) + CHECK_PERM_STATUS="$?" + set -e + # Only print output if the script ran successfully and it actually produced + # output. The latter check resolves + # https://github.com/certbot/certbot/issues/7012. + if [ "$CHECK_PERM_STATUS" = 0 -a -n "$CHECK_PERM_OUT" ]; then + error "$CHECK_PERM_OUT" + fi + fi + if [ "$NO_SELF_UPGRADE" != 1 ]; then TEMP_DIR=$(TempDir) trap 'rm -rf "$TEMP_DIR"' EXIT @@ -1664,37 +1993,41 @@ if __name__ == '__main__': UNLIKELY_EOF # --------------------------------------------------------------------------- - DeterminePythonVersion "NOCRASH" if [ "$PYVER" -lt "$MIN_PYVER" ]; then error "WARNING: couldn't find Python $MIN_PYTHON_VERSION+ to check for updates." elif ! REMOTE_VERSION=`"$LE_PYTHON" "$TEMP_DIR/fetch.py" --latest-version` ; then error "WARNING: unable to check for updates." fi - LE_VERSION_STATE=`CompareVersions "$LE_PYTHON" "$LE_AUTO_VERSION" "$REMOTE_VERSION"` - if [ "$LE_VERSION_STATE" = "UNOFFICIAL" ]; then - say "Unofficial certbot-auto version detected, self-upgrade is disabled: $LE_AUTO_VERSION" - elif [ "$LE_VERSION_STATE" = "OUTDATED" ]; then - say "Upgrading certbot-auto $LE_AUTO_VERSION to $REMOTE_VERSION..." + # If for any reason REMOTE_VERSION is not set, let's assume certbot-auto is up-to-date, + # and do not go into the self-upgrading process. + if [ -n "$REMOTE_VERSION" ]; then + LE_VERSION_STATE=`CompareVersions "$LE_PYTHON" "$LE_AUTO_VERSION" "$REMOTE_VERSION"` - # Now we drop into Python so we don't have to install even more - # dependencies (curl, etc.), for better flow control, and for the option of - # future Windows compatibility. - "$LE_PYTHON" "$TEMP_DIR/fetch.py" --le-auto-script "v$REMOTE_VERSION" + if [ "$LE_VERSION_STATE" = "UNOFFICIAL" ]; then + say "Unofficial certbot-auto version detected, self-upgrade is disabled: $LE_AUTO_VERSION" + elif [ "$LE_VERSION_STATE" = "OUTDATED" ]; then + say "Upgrading certbot-auto $LE_AUTO_VERSION to $REMOTE_VERSION..." - # Install new copy of certbot-auto. - # TODO: Deal with quotes in pathnames. - say "Replacing certbot-auto..." - # Clone permissions with cp. chmod and chown don't have a --reference - # option on macOS or BSD, and stat -c on Linux is stat -f on macOS and BSD: - cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone" - cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone" - # Using mv rather than cp leaves the old file descriptor pointing to the - # original copy so the shell can continue to read it unmolested. mv across - # filesystems is non-atomic, doing `rm dest, cp src dest, rm src`, but the - # cp is unlikely to fail if the rm doesn't. - mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0" - fi # A newer version is available. + # Now we drop into Python so we don't have to install even more + # dependencies (curl, etc.), for better flow control, and for the option of + # future Windows compatibility. + "$LE_PYTHON" "$TEMP_DIR/fetch.py" --le-auto-script "v$REMOTE_VERSION" + + # Install new copy of certbot-auto. + # TODO: Deal with quotes in pathnames. + say "Replacing certbot-auto..." + # Clone permissions with cp. chmod and chown don't have a --reference + # option on macOS or BSD, and stat -c on Linux is stat -f on macOS and BSD: + cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone" + cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone" + # Using mv rather than cp leaves the old file descriptor pointing to the + # original copy so the shell can continue to read it unmolested. mv across + # filesystems is non-atomic, doing `rm dest, cp src dest, rm src`, but the + # cp is unlikely to fail if the rm doesn't. + mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0" + fi # A newer version is available. + fi fi # Self-upgrading is allowed. RerunWithArgs --le-auto-phase2 "$@" diff --git a/letsencrypt-auto-source/Dockerfile.centos6 b/letsencrypt-auto-source/Dockerfile.centos6 deleted file mode 100644 index 09aa52dcd..000000000 --- a/letsencrypt-auto-source/Dockerfile.centos6 +++ /dev/null @@ -1,37 +0,0 @@ -# For running tests, build a docker image with a passwordless sudo and a trust -# store we can manipulate. - -FROM centos:6 - -RUN yum install -y epel-release - -# Install pip and sudo: -RUN yum install -y python-pip sudo -# Update to a stable and tested version of pip. -# We do not use pipstrap here because it no longer supports Python 2.6. -RUN pip install pip==9.0.1 setuptools==29.0.1 wheel==0.29.0 -# Pin pytest version for increased stability -RUN pip install pytest==3.2.5 six==1.10.0 - -# Add an unprivileged user: -RUN useradd --create-home --home-dir /home/lea --shell /bin/bash --groups wheel --uid 1000 lea - -# Let that user sudo: -RUN sed -i.bkp -e \ - 's/# %wheel\(NOPASSWD: ALL\)\?/%wheel/g' \ - /etc/sudoers - -RUN mkdir -p /home/lea/certbot - -# Install fake testing CA: -COPY ./tests/certs/ca/my-root-ca.crt.pem /usr/local/share/ca-certificates/ -RUN update-ca-trust - -# Copy code: -COPY . /home/lea/certbot/letsencrypt-auto-source - -USER lea -WORKDIR /home/lea - -RUN sudo chmod +x certbot/letsencrypt-auto-source/tests/centos6_tests.sh -CMD sudo certbot/letsencrypt-auto-source/tests/centos6_tests.sh diff --git a/letsencrypt-auto-source/Dockerfile.redhat6 b/letsencrypt-auto-source/Dockerfile.redhat6 new file mode 100644 index 000000000..66f21bc14 --- /dev/null +++ b/letsencrypt-auto-source/Dockerfile.redhat6 @@ -0,0 +1,54 @@ +# For running tests, build a docker image with a passwordless sudo and a trust +# store we can manipulate. + +ARG REDHAT_DIST_FLAVOR +FROM ${REDHAT_DIST_FLAVOR}:6 + +ARG REDHAT_DIST_FLAVOR + +RUN curl -O https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm \ + && rpm -ivh epel-release-latest-6.noarch.rpm + +# Install pip and sudo: +RUN yum install -y python-pip sudo +# Update to a stable and tested version of pip. +# We do not use pipstrap here because it no longer supports Python 2.6. +RUN pip install pip==9.0.1 setuptools==29.0.1 wheel==0.29.0 +# Pin pytest version for increased stability +RUN pip install pytest==3.2.5 six==1.10.0 + +# Add an unprivileged user: +RUN useradd --create-home --home-dir /home/lea --shell /bin/bash --groups wheel --uid 1000 lea + +# Let that user sudo: +RUN sed -i.bkp -e \ + 's/# %wheel\(NOPASSWD: ALL\)\?/%wheel/g' \ + /etc/sudoers + +RUN mkdir -p /home/lea/certbot + +# Install fake testing CA: +COPY ./tests/certs/ca/my-root-ca.crt.pem /usr/local/share/ca-certificates/ +RUN update-ca-trust + +# Copy current letsencrypt-auto: +COPY . /home/lea/certbot/letsencrypt-auto-source + +# Tweak uname binary for tests on fake 32bits +COPY tests/uname_wrapper.sh /bin +RUN mv /bin/uname /bin/uname_orig \ + && mv /bin/uname_wrapper.sh /bin/uname \ + && chmod +x /bin/uname + +# Fetch previous letsencrypt-auto that was installing python 3.4 +RUN curl https://raw.githubusercontent.com/certbot/certbot/v0.38.0/letsencrypt-auto-source/letsencrypt-auto \ + -o /home/lea/certbot/letsencrypt-auto-source/letsencrypt-auto_py_34 \ + && chmod +x /home/lea/certbot/letsencrypt-auto-source/letsencrypt-auto_py_34 + +RUN cp /home/lea/certbot/letsencrypt-auto-source/tests/${REDHAT_DIST_FLAVOR}6_tests.sh /home/lea/certbot/letsencrypt-auto-source/tests/redhat6_tests.sh \ + && chmod +x /home/lea/certbot/letsencrypt-auto-source/tests/redhat6_tests.sh + +USER lea +WORKDIR /home/lea + +CMD ["sudo", "certbot/letsencrypt-auto-source/tests/redhat6_tests.sh"] diff --git a/letsencrypt-auto-source/Dockerfile.trusty b/letsencrypt-auto-source/Dockerfile.trusty deleted file mode 100644 index 3de88f9af..000000000 --- a/letsencrypt-auto-source/Dockerfile.trusty +++ /dev/null @@ -1,36 +0,0 @@ -# For running tests, build a docker image with a passwordless sudo and a trust -# store we can manipulate. - -FROM ubuntu:trusty - -# Add an unprivileged user: -RUN useradd --create-home --home-dir /home/lea --shell /bin/bash --groups sudo --uid 1000 lea - -# Let that user sudo: -RUN sed -i.bkp -e \ - 's/%sudo\s\+ALL=(ALL\(:ALL\)\?)\s\+ALL/%sudo ALL=NOPASSWD:ALL/g' \ - /etc/sudoers - -# Install pip: -RUN apt-get update && \ - apt-get -q -y install python-pip && \ - apt-get clean -# Use pipstrap to update to a stable and tested version of pip -COPY ./pieces/pipstrap.py /opt -RUN /opt/pipstrap.py -# Pin pytest version for increased stability -RUN pip install pytest==3.2.5 six==1.10.0 - -RUN mkdir -p /home/lea/certbot - -# Install fake testing CA: -COPY ./tests/certs/ca/my-root-ca.crt.pem /usr/local/share/ca-certificates/ -RUN update-ca-certificates - -# Copy code: -COPY . /home/lea/certbot/letsencrypt-auto-source - -USER lea -WORKDIR /home/lea - -CMD ["pytest", "-v", "-s", "certbot/letsencrypt-auto-source/tests"] diff --git a/letsencrypt-auto-source/certbot-auto.asc b/letsencrypt-auto-source/certbot-auto.asc index a60ccd8bb..488d0bf2e 100644 --- a/letsencrypt-auto-source/certbot-auto.asc +++ b/letsencrypt-auto-source/certbot-auto.asc @@ -1,11 +1,11 @@ -----BEGIN PGP SIGNATURE----- -iQEzBAABCAAdFiEEos+1H6J1pyhiNOeyTRfJlc2XdfIFAlxLcw8ACgkQTRfJlc2X -dfK35gf+PoxtrJJIjvybNqd3lb8HOg2ntIVmXcYJGuuUo6m09fzai+XI6cOm5Dpu -l2D5OrbLqmez8tYkCkEWHV0OfwyVWw+m8T3sXlcrv14eA1RfgMnZ+cmmlpDskzHU -EOtaXo1/IkLDwBRrsl8IUbwD2XxbjuLsA2Sevoa59NlfTXJUApfAzohl3epRiJjB -gugdqcsfjRRAqQqOz+iJCKBCWSTIrr/g6Y9aZu9V93t/WDSLRFjehxO1GQrLnCnX -17JGlr0/AXd67jOKS1OWmORPPAFfLIXezUMtgrz5hE7T5UviaUu9ySV8UCxq1N79 -cfSBb/HIUxZ0wf1CkTUMRFQpA7cGtw== -=cNcT +iQEzBAABCAAdFiEEos+1H6J1pyhiNOeyTRfJlc2XdfIFAl456ZoACgkQTRfJlc2X +dfJx8wf/addMw4kUlwu6poHqLvsifZzHAESgvq+qybgFvl5yTh2U+99PGBgxRYx+ +bENIWBi6+XB+CiVuLzIXWw/VkXh+za99orRkkVK9PI33Xr7jBMZo5Oa3JviYjl3X +PcfjioRQCD+a9Tf9RO25LXQmxn87Ql9x3nxJuk//YeSpuImFmYjIBPE4n/LPEf7z +8WHU4oxxa/bgqGCPgv6O7ZBw7ipd3g+VHcDZcNQMP4tWYb6m7x/nN61yirid7q3M +uqQ1lbitN48ISyru6xPyE6WGTvfl1SIQd21FNRETpcoesx+MTv3ApWT4dqXjZvaX +FeM55IS65e7ci6yLV9qdAbqGKzhX0Q== +=uLcV -----END PGP SIGNATURE----- diff --git a/letsencrypt-auto-source/letsencrypt-auto b/letsencrypt-auto-source/letsencrypt-auto index 48ddfb570..e2813853b 100755 --- a/letsencrypt-auto-source/letsencrypt-auto +++ b/letsencrypt-auto-source/letsencrypt-auto @@ -31,7 +31,7 @@ if [ -z "$VENV_PATH" ]; then fi VENV_BIN="$VENV_PATH/bin" BOOTSTRAP_VERSION_PATH="$VENV_PATH/certbot-auto-bootstrap-version.txt" -LE_AUTO_VERSION="0.31.0.dev0" +LE_AUTO_VERSION="1.3.0.dev0" BASENAME=$(basename $0) USAGE="Usage: $BASENAME [OPTIONS] A self-updating wrapper script for the Certbot ACME client. When run, updates @@ -45,6 +45,7 @@ Help for certbot itself cannot be provided until it is installed. -h, --help print this help -n, --non-interactive, --noninteractive run without asking for user input --no-bootstrap do not install OS dependencies + --no-permissions-check do not warn about file system permissions --no-self-upgrade do not download updates --os-packages-only install OS dependencies and exit --install-only install certbot, upgrade if needed, and exit @@ -67,6 +68,8 @@ for arg in "$@" ; do # Do not upgrade this script (also prevents client upgrades, because each # copy of the script pins a hash of the python client) NO_SELF_UPGRADE=1;; + --no-permissions-check) + NO_PERMISSIONS_CHECK=1;; --no-bootstrap) NO_BOOTSTRAP=1;; --help) @@ -172,7 +175,11 @@ SetRootAuthMechanism() { sudo) SUDO="sudo -E" ;; - '') ;; # Nothing to do for plain root method. + '') + # If we're not running with root, don't check that this script can only + # be modified by system users and groups. + NO_PERMISSIONS_CHECK=1 + ;; *) error "Error: unknown root authorization mechanism '$LE_AUTO_SUDO'." exit 1 @@ -249,20 +256,28 @@ DeprecationBootstrap() { fi } -MIN_PYTHON_VERSION="2.7" -MIN_PYVER=$(echo "$MIN_PYTHON_VERSION" | sed 's/\.//') +MIN_PYTHON_2_VERSION="2.7" +MIN_PYVER2=$(echo "$MIN_PYTHON_2_VERSION" | sed 's/\.//') +MIN_PYTHON_3_VERSION="3.5" +MIN_PYVER3=$(echo "$MIN_PYTHON_3_VERSION" | sed 's/\.//') # Sets LE_PYTHON to Python version string and PYVER to the first two -# digits of the python version +# digits of the python version. +# MIN_PYVER and MIN_PYTHON_VERSION are also set by this function, and their +# values depend on if we try to use Python 3 or Python 2. DeterminePythonVersion() { # Arguments: "NOCRASH" if we shouldn't crash if we don't find a good python # # If no Python is found, PYVER is set to 0. if [ "$USE_PYTHON_3" = 1 ]; then + MIN_PYVER=$MIN_PYVER3 + MIN_PYTHON_VERSION=$MIN_PYTHON_3_VERSION for LE_PYTHON in "$LE_PYTHON" python3; do # Break (while keeping the LE_PYTHON value) if found. $EXISTS "$LE_PYTHON" > /dev/null && break done else + MIN_PYVER=$MIN_PYVER2 + MIN_PYTHON_VERSION=$MIN_PYTHON_2_VERSION for LE_PYTHON in "$LE_PYTHON" python2.7 python27 python2 python; do # Break (while keeping the LE_PYTHON value) if found. $EXISTS "$LE_PYTHON" > /dev/null && break @@ -278,7 +293,7 @@ DeterminePythonVersion() { fi fi - PYVER=`"$LE_PYTHON" -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//'` + PYVER=$("$LE_PYTHON" -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') if [ "$PYVER" -lt "$MIN_PYVER" ]; then if [ "$1" != "NOCRASH" ]; then error "You have an ancient version of Python entombed in your operating system..." @@ -361,7 +376,9 @@ BootstrapDebCommon() { # Sets TOOL to the name of the package manager # Sets appropriate values for YES_FLAG and QUIET_FLAG based on $ASSUME_YES and $QUIET_FLAG. -# Enables EPEL if applicable and possible. +# Note: this function is called both while selecting the bootstrap scripts and +# during the actual bootstrap. Some things like prompting to user can be done in the latter +# case, but not in the former one. InitializeRPMCommonBase() { if type dnf 2>/dev/null then @@ -381,26 +398,6 @@ InitializeRPMCommonBase() { if [ "$QUIET" = 1 ]; then QUIET_FLAG='--quiet' fi - - if ! $TOOL list *virtualenv >/dev/null 2>&1; then - echo "To use Certbot, packages from the EPEL repository need to be installed." - if ! $TOOL list epel-release >/dev/null 2>&1; then - error "Enable the EPEL repository and try running Certbot again." - exit 1 - fi - if [ "$ASSUME_YES" = 1 ]; then - /bin/echo -n "Enabling the EPEL repository in 3 seconds..." - sleep 1s - /bin/echo -ne "\e[0K\rEnabling the EPEL repository in 2 seconds..." - sleep 1s - /bin/echo -e "\e[0K\rEnabling the EPEL repository in 1 second..." - sleep 1s - fi - if ! $TOOL install $YES_FLAG $QUIET_FLAG epel-release; then - error "Could not enable EPEL. Aborting bootstrap!" - exit 1 - fi - fi } BootstrapRpmCommonBase() { @@ -483,19 +480,98 @@ BootstrapRpmCommon() { # If new packages are installed by BootstrapRpmPython3 below, this version # number must be increased. -BOOTSTRAP_RPM_PYTHON3_VERSION=1 +BOOTSTRAP_RPM_PYTHON3_LEGACY_VERSION=1 -BootstrapRpmPython3() { +# Checks if rh-python36 can be installed. +Python36SclIsAvailable() { + InitializeRPMCommonBase >/dev/null 2>&1; + + if "${TOOL}" list rh-python36 >/dev/null 2>&1; then + return 0 + fi + if "${TOOL}" list centos-release-scl >/dev/null 2>&1; then + return 0 + fi + return 1 +} + +# Try to enable rh-python36 from SCL if it is necessary and possible. +EnablePython36SCL() { + if "$EXISTS" python3.6 > /dev/null 2> /dev/null; then + return 0 + fi + if [ ! -f /opt/rh/rh-python36/enable ]; then + return 0 + fi + set +e + if ! . /opt/rh/rh-python36/enable; then + error 'Unable to enable rh-python36!' + exit 1 + fi + set -e +} + +# This bootstrap concerns old RedHat-based distributions that do not ship by default +# with Python 2.7, but only Python 2.6. We bootstrap them by enabling SCL and installing +# Python 3.6. Some of these distributions are: CentOS/RHEL/OL/SL 6. +BootstrapRpmPython3Legacy() { # Tested with: # - CentOS 6 InitializeRPMCommonBase - # EPEL uses python34 - if $TOOL list python34 >/dev/null 2>&1; then - python_pkgs="python34 - python34-devel - python34-tools + if ! "${TOOL}" list rh-python36 >/dev/null 2>&1; then + echo "To use Certbot on this operating system, packages from the SCL repository need to be installed." + if ! "${TOOL}" list centos-release-scl >/dev/null 2>&1; then + error "Enable the SCL repository and try running Certbot again." + exit 1 + fi + if [ "${ASSUME_YES}" = 1 ]; then + /bin/echo -n "Enabling the SCL repository in 3 seconds... (Press Ctrl-C to cancel)" + sleep 1s + /bin/echo -ne "\e[0K\rEnabling the SCL repository in 2 seconds... (Press Ctrl-C to cancel)" + sleep 1s + /bin/echo -e "\e[0K\rEnabling the SCL repository in 1 second... (Press Ctrl-C to cancel)" + sleep 1s + fi + if ! "${TOOL}" install "${YES_FLAG}" "${QUIET_FLAG}" centos-release-scl; then + error "Could not enable SCL. Aborting bootstrap!" + exit 1 + fi + fi + + # CentOS 6 must use rh-python36 from SCL + if "${TOOL}" list rh-python36 >/dev/null 2>&1; then + python_pkgs="rh-python36-python + rh-python36-python-virtualenv + rh-python36-python-devel + " + else + error "No supported Python package available to install. Aborting bootstrap!" + exit 1 + fi + + BootstrapRpmCommonBase "${python_pkgs}" + + # Enable SCL rh-python36 after bootstrapping. + EnablePython36SCL +} + +# If new packages are installed by BootstrapRpmPython3 below, this version +# number must be increased. +BOOTSTRAP_RPM_PYTHON3_VERSION=1 + +BootstrapRpmPython3() { + # Tested with: + # - Fedora 29 + + InitializeRPMCommonBase + + # Fedora 29 must use python3-virtualenv + if $TOOL list python3-virtualenv >/dev/null 2>&1; then + python_pkgs="python3 + python3-virtualenv + python3-devel " else error "No supported Python package available to install. Aborting bootstrap!" @@ -521,10 +597,20 @@ BootstrapSuseCommon() { QUIET_FLAG='-qq' fi + if zypper search -x python-virtualenv >/dev/null 2>&1; then + OPENSUSE_VIRTUALENV_PACKAGES="python-virtualenv" + else + # Since Leap 15.0 (and associated Tumbleweed version), python-virtualenv + # is a source package, and python2-virtualenv must be used instead. + # Also currently python2-setuptools is not a dependency of python2-virtualenv, + # while it should be. Installing it explicitly until upstream fix. + OPENSUSE_VIRTUALENV_PACKAGES="python2-virtualenv python2-setuptools" + fi + zypper $QUIET_FLAG $zypper_flags in $install_flags \ python \ python-devel \ - python-virtualenv \ + $OPENSUSE_VIRTUALENV_PACKAGES \ gcc \ augeas-lenses \ libopenssl-devel \ @@ -731,20 +817,71 @@ elif [ -f /etc/redhat-release ]; then prev_le_python="$LE_PYTHON" unset LE_PYTHON DeterminePythonVersion "NOCRASH" + + RPM_DIST_NAME=`(. /etc/os-release 2> /dev/null && echo $ID) || echo "unknown"` + + if [ "$PYVER" -eq 26 -a $(uname -m) != 'x86_64' ]; then + # 32 bits CentOS 6 and affiliates are not supported anymore by certbot-auto. + DEPRECATED_OS=1 + fi + + # Set RPM_DIST_VERSION to VERSION_ID from /etc/os-release after splitting on + # '.' characters (e.g. "8.0" becomes "8"). If the command exits with an + # error, RPM_DIST_VERSION is set to "unknown". + RPM_DIST_VERSION=$( (. /etc/os-release 2> /dev/null && echo "$VERSION_ID") | cut -d '.' -f1 || echo "unknown") + + # If RPM_DIST_VERSION is an empty string or it contains any nonnumeric + # characters, the value is unexpected so we set RPM_DIST_VERSION to 0. + if [ -z "$RPM_DIST_VERSION" ] || [ -n "$(echo "$RPM_DIST_VERSION" | tr -d '[0-9]')" ]; then + RPM_DIST_VERSION=0 + fi + + # Handle legacy RPM distributions if [ "$PYVER" -eq 26 ]; then + # Check if an automated bootstrap can be achieved on this system. + if ! Python36SclIsAvailable; then + INTERACTIVE_BOOTSTRAP=1 + fi + Bootstrap() { - BootstrapMessage "RedHat-based OSes that will use Python3" - BootstrapRpmPython3 + BootstrapMessage "Legacy RedHat-based OSes that will use Python3" + BootstrapRpmPython3Legacy } USE_PYTHON_3=1 - BOOTSTRAP_VERSION="BootstrapRpmPython3 $BOOTSTRAP_RPM_PYTHON3_VERSION" + BOOTSTRAP_VERSION="BootstrapRpmPython3Legacy $BOOTSTRAP_RPM_PYTHON3_LEGACY_VERSION" + + # Try now to enable SCL rh-python36 for systems already bootstrapped + # NB: EnablePython36SCL has been defined along with BootstrapRpmPython3Legacy in certbot-auto + EnablePython36SCL else - Bootstrap() { - BootstrapMessage "RedHat-based OSes" - BootstrapRpmCommon - } - BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION" + # Starting to Fedora 29, python2 is on a deprecation path. Let's move to python3 then. + # RHEL 8 also uses python3 by default. + if [ "$RPM_DIST_NAME" = "fedora" -a "$RPM_DIST_VERSION" -ge 29 ]; then + RPM_USE_PYTHON_3=1 + elif [ "$RPM_DIST_NAME" = "rhel" -a "$RPM_DIST_VERSION" -ge 8 ]; then + RPM_USE_PYTHON_3=1 + elif [ "$RPM_DIST_NAME" = "centos" -a "$RPM_DIST_VERSION" -ge 8 ]; then + RPM_USE_PYTHON_3=1 + else + RPM_USE_PYTHON_3=0 + fi + + if [ "$RPM_USE_PYTHON_3" = 1 ]; then + Bootstrap() { + BootstrapMessage "RedHat-based OSes that will use Python3" + BootstrapRpmPython3 + } + USE_PYTHON_3=1 + BOOTSTRAP_VERSION="BootstrapRpmPython3 $BOOTSTRAP_RPM_PYTHON3_VERSION" + else + Bootstrap() { + BootstrapMessage "RedHat-based OSes" + BootstrapRpmCommon + } + BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION" + fi fi + LE_PYTHON="$prev_le_python" elif [ -f /etc/os-release ] && `grep -q openSUSE /etc/os-release` ; then Bootstrap() { @@ -819,6 +956,13 @@ if [ "$NO_BOOTSTRAP" = 1 ]; then unset BOOTSTRAP_VERSION fi +if [ "$DEPRECATED_OS" = 1 ]; then + Bootstrap() { + error "Skipping bootstrap because certbot-auto is deprecated on this system." + } + unset BOOTSTRAP_VERSION +fi + # Sets PREV_BOOTSTRAP_VERSION to the identifier for the bootstrap script used # to install OS dependencies on this system. PREV_BOOTSTRAP_VERSION isn't set # if it is unknown how OS dependencies were installed on this system. @@ -888,10 +1032,156 @@ else: UNLIKELY_EOF } +# Create a new virtual environment for Certbot. It will overwrite any existing one. +# Parameters: LE_PYTHON, VENV_PATH, PYVER, VERBOSE +CreateVenv() { + "$1" - "$2" "$3" "$4" << "UNLIKELY_EOF" +#!/usr/bin/env python +import os +import shutil +import subprocess +import sys + + +def create_venv(venv_path, pyver, verbose): + if os.path.exists(venv_path): + shutil.rmtree(venv_path) + + stdout = sys.stdout if verbose == '1' else open(os.devnull, 'w') + + if int(pyver) <= 27: + # Use virtualenv binary + environ = os.environ.copy() + environ['VIRTUALENV_NO_DOWNLOAD'] = '1' + command = ['virtualenv', '--no-site-packages', '--python', sys.executable, venv_path] + subprocess.check_call(command, stdout=stdout, env=environ) + else: + # Use embedded venv module in Python 3 + command = [sys.executable, '-m', 'venv', venv_path] + subprocess.check_call(command, stdout=stdout) + + +if __name__ == '__main__': + create_venv(*sys.argv[1:]) + +UNLIKELY_EOF +} + +# Check that the given PATH_TO_CHECK has secured permissions. +# Parameters: LE_PYTHON, PATH_TO_CHECK +CheckPathPermissions() { + "$1" - "$2" << "UNLIKELY_EOF" +"""Verifies certbot-auto cannot be modified by unprivileged users. + +This script takes the path to certbot-auto as its only command line +argument. It then checks that the file can only be modified by uid/gid +< 1000 and if other users can modify the file, it prints a warning with +a suggestion on how to solve the problem. + +Permissions on symlinks in the absolute path of certbot-auto are ignored +and only the canonical path to certbot-auto is checked. There could be +permissions problems due to the symlinks that are unreported by this +script, however, issues like this were not caused by our documentation +and are ignored for the sake of simplicity. + +All warnings are printed to stdout rather than stderr so all stderr +output from this script can be suppressed to avoid printing messages if +this script fails for some reason. + +""" +from __future__ import print_function + +import os +import stat +import sys + + +FORUM_POST_URL = 'https://community.letsencrypt.org/t/certbot-auto-deployment-best-practices/91979/' + + +def has_safe_permissions(path): + """Returns True if the given path has secure permissions. + + The permissions are considered safe if the file is only writable by + uid/gid < 1000. + + The reason we allow more IDs than 0 is because on some systems such + as Debian, system users/groups other than uid/gid 0 are used for the + path we recommend in our instructions which is /usr/local/bin. 1000 + was chosen because on Debian 0-999 is reserved for system IDs[1] and + on RHEL either 0-499 or 0-999 is reserved depending on the + version[2][3]. Due to these differences across different OSes, this + detection isn't perfect so we only determine permissions are + insecure when we can be reasonably confident there is a problem + regardless of the underlying OS. + + [1] https://www.debian.org/doc/debian-policy/ch-opersys.html#uid-and-gid-classes + [2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/ch-managing_users_and_groups + [3] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-managing_users_and_groups + + :param str path: filesystem path to check + :returns: True if the path has secure permissions, otherwise, False + :rtype: bool + + """ + # os.stat follows symlinks before obtaining information about a file. + stat_result = os.stat(path) + if stat_result.st_mode & stat.S_IWOTH: + return False + if stat_result.st_mode & stat.S_IWGRP and stat_result.st_gid >= 1000: + return False + if stat_result.st_mode & stat.S_IWUSR and stat_result.st_uid >= 1000: + return False + return True + + +def main(certbot_auto_path): + current_path = os.path.realpath(certbot_auto_path) + last_path = None + permissions_ok = True + # This loop makes use of the fact that os.path.dirname('/') == '/'. + while current_path != last_path and permissions_ok: + permissions_ok = has_safe_permissions(current_path) + last_path = current_path + current_path = os.path.dirname(current_path) + + if not permissions_ok: + print('{0} has insecure permissions!'.format(certbot_auto_path)) + print('To learn how to fix them, visit {0}'.format(FORUM_POST_URL)) + + +if __name__ == '__main__': + main(sys.argv[1]) + +UNLIKELY_EOF +} + if [ "$1" = "--le-auto-phase2" ]; then # Phase 2: Create venv, install LE, and run. shift 1 # the --le-auto-phase2 arg + + if [ "$DEPRECATED_OS" = 1 ]; then + # Phase 2 damage control mode for deprecated OSes. + # In this situation, we bypass any bootstrap or certbot venv setup. + error "Your system is not supported by certbot-auto anymore." + + if [ ! -d "$VENV_PATH" ] && OldVenvExists; then + VENV_BIN="$OLD_VENV_PATH/bin" + fi + + if [ -f "$VENV_BIN/letsencrypt" -a "$INSTALL_ONLY" != 1 ]; then + error "Certbot will no longer receive updates." + error "Please visit https://certbot.eff.org/ to check for other alternatives." + "$VENV_BIN/letsencrypt" "$@" + exit 0 + else + error "Certbot cannot be installed." + error "Please visit https://certbot.eff.org/ to check for other alternatives." + exit 1 + fi + fi + SetPrevBootstrapVersion if [ -z "$PHASE_1_VERSION" -a "$USE_PYTHON_3" = 1 ]; then @@ -903,8 +1193,15 @@ if [ "$1" = "--le-auto-phase2" ]; then # If the selected Bootstrap function isn't a noop and it differs from the # previously used version if [ -n "$BOOTSTRAP_VERSION" -a "$BOOTSTRAP_VERSION" != "$PREV_BOOTSTRAP_VERSION" ]; then - # if non-interactive mode or stdin and stdout are connected to a terminal - if [ \( "$NONINTERACTIVE" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then + # Check if we can rebootstrap without manual user intervention: this requires that + # certbot-auto is in non-interactive mode AND selected bootstrap does not claim to + # require a manual user intervention. + if [ "$NONINTERACTIVE" = 1 -a "$INTERACTIVE_BOOTSTRAP" != 1 ]; then + CAN_REBOOTSTRAP=1 + fi + # Check if rebootstrap can be done non-interactively and current shell is non-interactive + # (true if stdin and stdout are not attached to a terminal). + if [ \( "$CAN_REBOOTSTRAP" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then if [ -d "$VENV_PATH" ]; then rm -rf "$VENV_PATH" fi @@ -915,12 +1212,21 @@ if [ "$1" = "--le-auto-phase2" ]; then ln -s "$VENV_PATH" "$OLD_VENV_PATH" fi RerunWithArgs "$@" + # Otherwise bootstrap needs to be done manually by the user. else - error "Skipping upgrade because new OS dependencies may need to be installed." - error - error "To upgrade to a newer version, please run this script again manually so you can" - error "approve changes or with --non-interactive on the command line to automatically" - error "install any required packages." + # If it is because bootstrapping is interactive, --non-interactive will be of no use. + if [ "$INTERACTIVE_BOOTSTRAP" = 1 ]; then + error "Skipping upgrade because new OS dependencies may need to be installed." + error "This requires manual user intervention: please run this script again manually." + # If this is because of the environment (eg. non interactive shell without + # --non-interactive flag set), help the user in that direction. + else + error "Skipping upgrade because new OS dependencies may need to be installed." + error + error "To upgrade to a newer version, please run this script again manually so you can" + error "approve changes or with --non-interactive on the command line to automatically" + error "install any required packages." + fi # Set INSTALLED_VERSION to be the same so we don't update the venv INSTALLED_VERSION="$LE_AUTO_VERSION" # Continue to use OLD_VENV_PATH if the new venv doesn't exist @@ -943,22 +1249,7 @@ if [ "$1" = "--le-auto-phase2" ]; then if [ "$LE_AUTO_VERSION" != "$INSTALLED_VERSION" ]; then say "Creating virtual environment..." DeterminePythonVersion - rm -rf "$VENV_PATH" - if [ "$PYVER" -le 27 ]; then - # Use an environment variable instead of a flag for compatibility with old versions - if [ "$VERBOSE" = 1 ]; then - VIRTUALENV_NO_DOWNLOAD=1 virtualenv --no-site-packages --python "$LE_PYTHON" "$VENV_PATH" - else - VIRTUALENV_NO_DOWNLOAD=1 virtualenv --no-site-packages --python "$LE_PYTHON" "$VENV_PATH" \ - > /dev/null - fi - else - if [ "$VERBOSE" = 1 ]; then - "$LE_PYTHON" -m venv "$VENV_PATH" - else - "$LE_PYTHON" -m venv "$VENV_PATH" > /dev/null - fi - fi + CreateVenv "$LE_PYTHON" "$VENV_PATH" "$PYVER" "$VERBOSE" if [ -n "$BOOTSTRAP_VERSION" ]; then echo "$BOOTSTRAP_VERSION" > "$BOOTSTRAP_VERSION_PATH" @@ -972,202 +1263,271 @@ if [ "$1" = "--le-auto-phase2" ]; then # There is no $ interpolation due to quotes on starting heredoc delimiter. # ------------------------------------------------------------------------- cat << "UNLIKELY_EOF" > "$TEMP_DIR/letsencrypt-auto-requirements.txt" -# This is the flattened list of packages certbot-auto installs. To generate -# this, do -# `pip install --no-cache-dir -e acme -e . -e certbot-apache -e certbot-nginx`, -# and then use `hashin` or a more secure method to gather the hashes. - -# Hashin example: +# This is the flattened list of packages certbot-auto installs. +# To generate this, do (with docker and package hashin installed): +# ``` +# letsencrypt-auto-source/rebuild_dependencies.py \ +# letsencrypt-auto-source/pieces/dependency-requirements.txt +# ``` +# If you want to update a single dependency, run commands similar to these: +# ``` # pip install hashin # hashin -r dependency-requirements.txt cryptography==1.5.2 -# sets the new certbot-auto pinned version of cryptography to 1.5.2 - -argparse==1.4.0 \ - --hash=sha256:c31647edb69fd3d465a847ea3157d37bed1f95f19760b11a47aa91c04b666314 \ - --hash=sha256:62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4 - -# This comes before cffi because cffi will otherwise install an unchecked -# version via setup_requires. -pycparser==2.14 \ - --hash=sha256:7959b4a74abdc27b312fed1c21e6caf9309ce0b29ea86b591fd2e99ecdf27f73 \ - --no-binary pycparser - -asn1crypto==0.22.0 \ - --hash=sha256:d232509fefcfcdb9a331f37e9c9dc20441019ad927c7d2176cf18ed5da0ba097 \ - --hash=sha256:cbbadd640d3165ab24b06ef25d1dca09a3441611ac15f6a6b452474fdf0aed1a -cffi==1.11.5 \ - --hash=sha256:1b0493c091a1898f1136e3f4f991a784437fac3673780ff9de3bcf46c80b6b50 \ - --hash=sha256:87f37fe5130574ff76c17cab61e7d2538a16f843bb7bca8ebbc4b12de3078596 \ - --hash=sha256:1553d1e99f035ace1c0544050622b7bc963374a00c467edafac50ad7bd276aef \ - --hash=sha256:151b7eefd035c56b2b2e1eb9963c90c6302dc15fbd8c1c0a83a163ff2c7d7743 \ - --hash=sha256:edabd457cd23a02965166026fd9bfd196f4324fe6032e866d0f3bd0301cd486f \ - --hash=sha256:ba5e697569f84b13640c9e193170e89c13c6244c24400fc57e88724ef610cd31 \ - --hash=sha256:79f9b6f7c46ae1f8ded75f68cf8ad50e5729ed4d590c74840471fc2823457d04 \ - --hash=sha256:b0f7d4a3df8f06cf49f9f121bead236e328074de6449866515cea4907bbc63d6 \ - --hash=sha256:4c91af6e967c2015729d3e69c2e51d92f9898c330d6a851bf8f121236f3defd3 \ - --hash=sha256:7a33145e04d44ce95bcd71e522b478d282ad0eafaf34fe1ec5bbd73e662f22b6 \ - --hash=sha256:95d5251e4b5ca00061f9d9f3d6fe537247e145a8524ae9fd30a2f8fbce993b5b \ - --hash=sha256:b75110fb114fa366b29a027d0c9be3709579602ae111ff61674d28c93606acca \ - --hash=sha256:ae5e35a2c189d397b91034642cb0eab0e346f776ec2eb44a49a459e6615d6e2e \ - --hash=sha256:fdf1c1dc5bafc32bc5d08b054f94d659422b05aba244d6be4ddc1c72d9aa70fb \ - --hash=sha256:9d1d3e63a4afdc29bd76ce6aa9d58c771cd1599fbba8cf5057e7860b203710dd \ - --hash=sha256:be2a9b390f77fd7676d80bc3cdc4f8edb940d8c198ed2d8c0be1319018c778e1 \ - --hash=sha256:ed01918d545a38998bfa5902c7c00e0fee90e957ce036a4000a88e3fe2264917 \ - --hash=sha256:857959354ae3a6fa3da6651b966d13b0a8bed6bbc87a0de7b38a549db1d2a359 \ - --hash=sha256:2ba8a45822b7aee805ab49abfe7eec16b90587f7f26df20c71dd89e45a97076f \ - --hash=sha256:a36c5c154f9d42ec176e6e620cb0dd275744aa1d804786a71ac37dc3661a5e95 \ - --hash=sha256:e55e22ac0a30023426564b1059b035973ec82186ddddbac867078435801c7801 \ - --hash=sha256:3eb6434197633b7748cea30bf0ba9f66727cdce45117a712b29a443943733257 \ - --hash=sha256:ecbb7b01409e9b782df5ded849c178a0aa7c906cf8c5a67368047daab282b184 \ - --hash=sha256:770f3782b31f50b68627e22f91cb182c48c47c02eb405fd689472aa7b7aa16dc \ - --hash=sha256:d5d8555d9bfc3f02385c1c37e9f998e2011f0db4f90e250e5bc0c0a85a813085 \ - --hash=sha256:3c85641778460581c42924384f5e68076d724ceac0f267d66c757f7535069c93 \ - --hash=sha256:ca1bd81f40adc59011f58159e4aa6445fc585a32bb8ac9badf7a2c1aa23822f2 \ - --hash=sha256:3bb6bd7266598f318063e584378b8e27c67de998a43362e8fce664c54ee52d30 \ - --hash=sha256:a6a5cb8809091ec9ac03edde9304b3ad82ad4466333432b16d78ef40e0cce0d5 \ - --hash=sha256:57b2533356cb2d8fac1555815929f7f5f14d68ac77b085d2326b571310f34f6e \ - --hash=sha256:495c5c2d43bf6cebe0178eb3e88f9c4aa48d8934aa6e3cddb865c058da76756b \ - --hash=sha256:e90f17980e6ab0f3c2f3730e56d1fe9bcba1891eeea58966e89d352492cc74f4 -ConfigArgParse==0.12.0 \ - --hash=sha256:28cd7d67669651f2a4518367838c49539457504584a139709b2b8f6c208ef339 \ - --no-binary ConfigArgParse +# ``` +ConfigArgParse==1.0 \ + --hash=sha256:bf378245bc9cdc403a527e5b7406b991680c2a530e7e81af747880b54eb57133 +certifi==2019.11.28 \ + --hash=sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3 \ + --hash=sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f +cffi==1.13.2 \ + --hash=sha256:0b49274afc941c626b605fb59b59c3485c17dc776dc3cc7cc14aca74cc19cc42 \ + --hash=sha256:0e3ea92942cb1168e38c05c1d56b0527ce31f1a370f6117f1d490b8dcd6b3a04 \ + --hash=sha256:135f69aecbf4517d5b3d6429207b2dff49c876be724ac0c8bf8e1ea99df3d7e5 \ + --hash=sha256:19db0cdd6e516f13329cba4903368bff9bb5a9331d3410b1b448daaadc495e54 \ + --hash=sha256:2781e9ad0e9d47173c0093321bb5435a9dfae0ed6a762aabafa13108f5f7b2ba \ + --hash=sha256:291f7c42e21d72144bb1c1b2e825ec60f46d0a7468f5346841860454c7aa8f57 \ + --hash=sha256:2c5e309ec482556397cb21ede0350c5e82f0eb2621de04b2633588d118da4396 \ + --hash=sha256:2e9c80a8c3344a92cb04661115898a9129c074f7ab82011ef4b612f645939f12 \ + --hash=sha256:32a262e2b90ffcfdd97c7a5e24a6012a43c61f1f5a57789ad80af1d26c6acd97 \ + --hash=sha256:3c9fff570f13480b201e9ab69453108f6d98244a7f495e91b6c654a47486ba43 \ + --hash=sha256:415bdc7ca8c1c634a6d7163d43fb0ea885a07e9618a64bda407e04b04333b7db \ + --hash=sha256:42194f54c11abc8583417a7cf4eaff544ce0de8187abaf5d29029c91b1725ad3 \ + --hash=sha256:4424e42199e86b21fc4db83bd76909a6fc2a2aefb352cb5414833c030f6ed71b \ + --hash=sha256:4a43c91840bda5f55249413037b7a9b79c90b1184ed504883b72c4df70778579 \ + --hash=sha256:599a1e8ff057ac530c9ad1778293c665cb81a791421f46922d80a86473c13346 \ + --hash=sha256:5c4fae4e9cdd18c82ba3a134be256e98dc0596af1e7285a3d2602c97dcfa5159 \ + --hash=sha256:5ecfa867dea6fabe2a58f03ac9186ea64da1386af2159196da51c4904e11d652 \ + --hash=sha256:62f2578358d3a92e4ab2d830cd1c2049c9c0d0e6d3c58322993cc341bdeac22e \ + --hash=sha256:6471a82d5abea994e38d2c2abc77164b4f7fbaaf80261cb98394d5793f11b12a \ + --hash=sha256:6d4f18483d040e18546108eb13b1dfa1000a089bcf8529e30346116ea6240506 \ + --hash=sha256:71a608532ab3bd26223c8d841dde43f3516aa5d2bf37b50ac410bb5e99053e8f \ + --hash=sha256:74a1d8c85fb6ff0b30fbfa8ad0ac23cd601a138f7509dc617ebc65ef305bb98d \ + --hash=sha256:7b93a885bb13073afb0aa73ad82059a4c41f4b7d8eb8368980448b52d4c7dc2c \ + --hash=sha256:7d4751da932caaec419d514eaa4215eaf14b612cff66398dd51129ac22680b20 \ + --hash=sha256:7f627141a26b551bdebbc4855c1157feeef18241b4b8366ed22a5c7d672ef858 \ + --hash=sha256:8169cf44dd8f9071b2b9248c35fc35e8677451c52f795daa2bb4643f32a540bc \ + --hash=sha256:aa00d66c0fab27373ae44ae26a66a9e43ff2a678bf63a9c7c1a9a4d61172827a \ + --hash=sha256:ccb032fda0873254380aa2bfad2582aedc2959186cce61e3a17abc1a55ff89c3 \ + --hash=sha256:d754f39e0d1603b5b24a7f8484b22d2904fa551fe865fd0d4c3332f078d20d4e \ + --hash=sha256:d75c461e20e29afc0aee7172a0950157c704ff0dd51613506bd7d82b718e7410 \ + --hash=sha256:dcd65317dd15bc0451f3e01c80da2216a31916bdcffd6221ca1202d96584aa25 \ + --hash=sha256:e570d3ab32e2c2861c4ebe6ffcad6a8abf9347432a37608fe1fbd157b3f0036b \ + --hash=sha256:fd43a88e045cf992ed09fa724b5315b790525f2676883a6ea64e3263bae6549d +chardet==3.0.4 \ + --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \ + --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 configobj==5.0.6 \ - --hash=sha256:a2f5650770e1c87fb335af19a9b7eb73fc05ccf22144eb68db7d00cd2bcb0902 \ - --no-binary configobj -cryptography==2.2.2 \ - --hash=sha256:3f3b65d5a16e6b52fba63dc860b62ca9832f51f1a2ae5083c78b6840275f12dd \ - --hash=sha256:5251e7de0de66810833606439ca65c9b9e45da62196b0c88bfadf27740aac09f \ - --hash=sha256:551a3abfe0c8c6833df4192a63371aa2ff43afd8f570ed345d31f251d78e7e04 \ - --hash=sha256:5cb990056b7cadcca26813311187ad751ea644712022a3976443691168781b6f \ - --hash=sha256:60bda7f12ecb828358be53095fc9c6edda7de8f1ef571f96c00b2363643fa3cd \ - --hash=sha256:64b5c67acc9a7c83fbb4b69166f3105a0ab722d27934fac2cb26456718eec2ba \ - --hash=sha256:6fef51ec447fe9f8351894024e94736862900d3a9aa2961528e602eb65c92bdb \ - --hash=sha256:77d0ad229d47a6e0272d00f6bf8ac06ce14715a9fd02c9a97f5a2869aab3ccb2 \ - --hash=sha256:808fe471b1a6b777f026f7dc7bd9a4959da4bfab64972f2bbe91e22527c1c037 \ - --hash=sha256:9b62fb4d18529c84b961efd9187fecbb48e89aa1a0f9f4161c61b7fc42a101bd \ - --hash=sha256:9e5bed45ec6b4f828866ac6a6bedf08388ffcfa68abe9e94b34bb40977aba531 \ - --hash=sha256:9fc295bf69130a342e7a19a39d7bbeb15c0bcaabc7382ec33ef3b2b7d18d2f63 \ - --hash=sha256:abd070b5849ed64e6d349199bef955ee0ad99aefbad792f0c587f8effa681a5e \ - --hash=sha256:ba6a774749b6e510cffc2fb98535f717e0e5fd91c7c99a61d223293df79ab351 \ - --hash=sha256:c332118647f084c983c6a3e1dba0f3bcb051f69d12baccac68db8d62d177eb8a \ - --hash=sha256:d6f46e862ee36df81e6342c2177ba84e70f722d9dc9c6c394f9f1f434c4a5563 \ - --hash=sha256:db6013746f73bf8edd9c3d1d3f94db635b9422f503db3fc5ef105233d4c011ab \ - --hash=sha256:f57008eaff597c69cf692c3518f6d4800f0309253bb138b526a37fe9ef0c7471 \ - --hash=sha256:f6c821ac253c19f2ad4c8691633ae1d1a17f120d5b01ea1d256d7b602bc59887 -enum34==1.1.2 ; python_version < '3.4' \ - --hash=sha256:2475d7fcddf5951e92ff546972758802de5260bf409319a9f1934e6bbc8b1dc7 \ - --hash=sha256:35907defb0f992b75ab7788f65fedc1cf20ffa22688e0e6f6f12afc06b3ea501 + --hash=sha256:a2f5650770e1c87fb335af19a9b7eb73fc05ccf22144eb68db7d00cd2bcb0902 +cryptography==2.8 \ + --hash=sha256:02079a6addc7b5140ba0825f542c0869ff4df9a69c360e339ecead5baefa843c \ + --hash=sha256:1df22371fbf2004c6f64e927668734070a8953362cd8370ddd336774d6743595 \ + --hash=sha256:369d2346db5934345787451504853ad9d342d7f721ae82d098083e1f49a582ad \ + --hash=sha256:3cda1f0ed8747339bbdf71b9f38ca74c7b592f24f65cdb3ab3765e4b02871651 \ + --hash=sha256:44ff04138935882fef7c686878e1c8fd80a723161ad6a98da31e14b7553170c2 \ + --hash=sha256:4b1030728872c59687badcca1e225a9103440e467c17d6d1730ab3d2d64bfeff \ + --hash=sha256:58363dbd966afb4f89b3b11dfb8ff200058fbc3b947507675c19ceb46104b48d \ + --hash=sha256:6ec280fb24d27e3d97aa731e16207d58bd8ae94ef6eab97249a2afe4ba643d42 \ + --hash=sha256:7270a6c29199adc1297776937a05b59720e8a782531f1f122f2eb8467f9aab4d \ + --hash=sha256:73fd30c57fa2d0a1d7a49c561c40c2f79c7d6c374cc7750e9ac7c99176f6428e \ + --hash=sha256:7f09806ed4fbea8f51585231ba742b58cbcfbfe823ea197d8c89a5e433c7e912 \ + --hash=sha256:90df0cc93e1f8d2fba8365fb59a858f51a11a394d64dbf3ef844f783844cc793 \ + --hash=sha256:971221ed40f058f5662a604bd1ae6e4521d84e6cad0b7b170564cc34169c8f13 \ + --hash=sha256:a518c153a2b5ed6b8cc03f7ae79d5ffad7315ad4569b2d5333a13c38d64bd8d7 \ + --hash=sha256:b0de590a8b0979649ebeef8bb9f54394d3a41f66c5584fff4220901739b6b2f0 \ + --hash=sha256:b43f53f29816ba1db8525f006fa6f49292e9b029554b3eb56a189a70f2a40879 \ + --hash=sha256:d31402aad60ed889c7e57934a03477b572a03af7794fa8fb1780f21ea8f6551f \ + --hash=sha256:de96157ec73458a7f14e3d26f17f8128c959084931e8997b9e655a39c8fde9f9 \ + --hash=sha256:df6b4dca2e11865e6cfbfb708e800efb18370f5a46fd601d3755bc7f85b3a8a2 \ + --hash=sha256:ecadccc7ba52193963c0475ac9f6fa28ac01e01349a2ca48509667ef41ffd2cf \ + --hash=sha256:fb81c17e0ebe3358486cd8cc3ad78adbae58af12fc2bf2bc0bb84e8090fa5ce8 +distro==1.4.0 \ + --hash=sha256:362dde65d846d23baee4b5c058c8586f219b5a54be1cf5fc6ff55c4578392f57 \ + --hash=sha256:eedf82a470ebe7d010f1872c17237c79ab04097948800029994fa458e52fb4b4 +enum34==1.1.6 \ + --hash=sha256:2d81cbbe0e73112bdfe6ef8576f2238f2ba27dd0d55752a776c41d38b7da2850 \ + --hash=sha256:644837f692e5f550741432dd3f223bbb9852018674981b1664e5dc339387588a \ + --hash=sha256:6bd0f6ad48ec2aa117d3d141940d484deccda84d4fcd884f5c3d93c23ecd8c79 \ + --hash=sha256:8ad8c4783bf61ded74527bffb48ed9b54166685e4230386a9ed9b1279e2df5b1 funcsigs==1.0.2 \ --hash=sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca \ --hash=sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50 -idna==2.5 \ - --hash=sha256:cc19709fd6d0cbfed39ea875d29ba6d4e22c0cebc510a76d6302a28385e8bb70 \ - --hash=sha256:3cb5ce08046c4e3a560fc02f138d0ac63e00f8ce5901a56b32ec8b7994082aab -ipaddress==1.0.16 \ - --hash=sha256:935712800ce4760701d89ad677666cd52691fd2f6f0b340c8b4239a3c17988a5 \ - --hash=sha256:5a3182b322a706525c46282ca6f064d27a02cffbd449f9f47416f1dc96aa71b0 -josepy==1.1.0 \ - --hash=sha256:1309a25aac3caeff5239729c58ff9b583f7d022ffdb1553406ddfc8e5b52b76e \ - --hash=sha256:fb5c62c77d26e04df29cb5ecd01b9ce69b6fcc9e521eb1ca193b7faa2afa7086 -linecache2==1.0.0 \ - --hash=sha256:e78be9c0a0dfcbac712fe04fbf92b96cddae80b1b842f24248214c8496f006ef \ - --hash=sha256:4b26ff4e7110db76eeb6f5a7b64a82623839d595c2038eeda662f2a2db78e97c -# Using an older version of mock here prevents regressions of #5276. +idna==2.8 \ + --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 \ + --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c +ipaddress==1.0.23 \ + --hash=sha256:6e0f4a39e66cb5bb9a137b00276a2eff74f93b71dcbdad6f10ff7df9d3557fcc \ + --hash=sha256:b7f8e0369580bb4a24d5ba1d7cc29660a4a6987763faf1d8a8046830e020e7e2 +josepy==1.2.0 \ + --hash=sha256:8ea15573203f28653c00f4ac0142520777b1c59d9eddd8da3f256c6ba3cac916 \ + --hash=sha256:9cec9a839fe9520f0420e4f38e7219525daccce4813296627436fe444cd002d3 mock==1.3.0 \ - --hash=sha256:3f573a18be94de886d1191f27c168427ef693e8dcfcecf95b170577b2eb69cbb \ - --hash=sha256:1e247dbecc6ce057299eb7ee019ad68314bb93152e81d9a6110d35f4d5eca0f6 -ordereddict==1.1 \ - --hash=sha256:1c35b4ac206cef2d24816c89f89cf289dd3d38cf7c449bb3fab7bf6d43f01b1f \ - --no-binary ordereddict -packaging==16.8 \ - --hash=sha256:99276dc6e3a7851f32027a68f1095cd3f77c148091b092ea867a351811cfe388 \ - --hash=sha256:5d50835fdf0a7edf0b55e311b7c887786504efea1177abd7e69329a8e5ea619e -parsedatetime==2.1 \ - --hash=sha256:ce9d422165cf6e963905cd5f74f274ebf7cc98c941916169178ef93f0e557838 \ - --hash=sha256:17c578775520c99131634e09cfca5a05ea9e1bd2a05cd06967ebece10df7af2d -pbr==1.8.1 \ - --hash=sha256:46c8db75ae75a056bd1cc07fa21734fe2e603d11a07833ecc1eeb74c35c72e0c \ - --hash=sha256:e2127626a91e6c885db89668976db31020f0af2da728924b56480fc7ccf09649 -pyOpenSSL==18.0.0 \ - --hash=sha256:26ff56a6b5ecaf3a2a59f132681e2a80afcc76b4f902f612f518f92c2a1bf854 \ - --hash=sha256:6488f1423b00f73b7ad5167885312bb0ce410d3312eb212393795b53c8caa580 -pyparsing==2.1.8 \ - --hash=sha256:2f0f5ceb14eccd5aef809d6382e87df22ca1da583c79f6db01675ce7d7f49c18 \ - --hash=sha256:03a4869b9f3493807ee1f1cb405e6d576a1a2ca4d81a982677c0c1ad6177c56b \ - --hash=sha256:ab09aee814c0241ff0c503cff30018219fe1fc14501d89f406f4664a0ec9fbcd \ - --hash=sha256:6e9a7f052f8e26bcf749e4033e3115b6dc7e3c85aafcb794b9a88c9d9ef13c97 \ - --hash=sha256:9f463a6bcc4eeb6c08f1ed84439b17818e2085937c0dee0d7674ac127c67c12b \ - --hash=sha256:3626b4d81cfb300dad57f52f2f791caaf7b06c09b368c0aa7b868e53a5775424 \ - --hash=sha256:367b90cc877b46af56d4580cd0ae278062903f02b8204ab631f5a2c0f50adfd0 \ - --hash=sha256:9f1ea360086cd68681e7f4ca8f1f38df47bf81942a0d76a9673c2d23eff35b13 -pyRFC3339==1.0 \ - --hash=sha256:eea31835c56e2096af4363a5745a784878a61d043e247d3a6d6a0a32a9741f56 \ - --hash=sha256:8dfbc6c458b8daba1c0f3620a8c78008b323a268b27b7359e92a4ae41325f535 + --hash=sha256:1e247dbecc6ce057299eb7ee019ad68314bb93152e81d9a6110d35f4d5eca0f6 \ + --hash=sha256:3f573a18be94de886d1191f27c168427ef693e8dcfcecf95b170577b2eb69cbb +parsedatetime==2.5 \ + --hash=sha256:3b835fc54e472c17ef447be37458b400e3fefdf14bb1ffdedb5d2c853acf4ba1 \ + --hash=sha256:d2e9ddb1e463de871d32088a3f3cea3dc8282b1b2800e081bd0ef86900451667 +pbr==5.4.4 \ + --hash=sha256:139d2625547dbfa5fb0b81daebb39601c478c21956dc57e2e07b74450a8c506b \ + --hash=sha256:61aa52a0f18b71c5cc58232d2cf8f8d09cd67fcad60b742a60124cb8d6951488 +pyOpenSSL==19.1.0 \ + --hash=sha256:621880965a720b8ece2f1b2f54ea2071966ab00e2970ad2ce11d596102063504 \ + --hash=sha256:9a24494b2602aaf402be5c9e30a0b82d4a5c67528fe8fb475e3f3bc00dd69507 +pyRFC3339==1.1 \ + --hash=sha256:67196cb83b470709c580bb4738b83165e67c6cc60e1f2e4f286cfcb402a926f4 \ + --hash=sha256:81b8cbe1519cdb79bed04910dd6fa4e181faf8c88dff1e1b987b5f7ab23a5b1a +pycparser==2.19 \ + --hash=sha256:a988718abfad80b6b157acce7bf130a30876d27603738ac39f140993246b25b3 +pyparsing==2.4.6 \ + --hash=sha256:4c830582a84fb022400b85429791bc551f1f4871c33f23e44f353119e92f969f \ + --hash=sha256:c342dccb5250c08d45fd6f8b4a559613ca603b57498511740e65cd11a2e7dcec python-augeas==0.5.0 \ - --hash=sha256:67d59d66cdba8d624e0389b87b2a83a176f21f16a87553b50f5703b23f29bac2 \ - --no-binary python-augeas -pytz==2015.7 \ - --hash=sha256:3abe6a6d3fc2fbbe4c60144211f45da2edbe3182a6f6511af6bbba0598b1f992 \ - --hash=sha256:939ef9c1e1224d980405689a97ffcf7828c56d1517b31d73464356c1f2b7769e \ - --hash=sha256:ead4aefa7007249e05e51b01095719d5a8dd95760089f5730aac5698b1932918 \ - --hash=sha256:3cca0df08bd0ed98432390494ce3ded003f5e661aa460be7a734bffe35983605 \ - --hash=sha256:3ede470d3d17ba3c07638dfa0d10452bc1b6e5ad326127a65ba77e6aaeb11bec \ - --hash=sha256:68c47964f7186eec306b13629627722b9079cd4447ed9e5ecaecd4eac84ca734 \ - --hash=sha256:dd5d3991950aae40a6c81de1578942e73d629808cefc51d12cd157980e6cfc18 \ - --hash=sha256:a77c52062c07eb7c7b30545dbc73e32995b7e117eea750317b5cb5c7a4618f14 \ - --hash=sha256:81af9aec4bc960a9a0127c488f18772dae4634689233f06f65443e7b11ebeb51 \ - --hash=sha256:e079b1dadc5c06246cc1bb6fe1b23a50b1d1173f2edd5104efd40bb73a28f406 \ - --hash=sha256:fbd26746772c24cb93c8b97cbdad5cb9e46c86bbdb1b9d8a743ee00e2fb1fc5d \ - --hash=sha256:99266ef30a37e43932deec2b7ca73e83c8dbc3b9ff703ec73eca6b1dae6befea \ - --hash=sha256:8b6ce1c993909783bc96e0b4f34ea223bff7a4df2c90bdb9c4e0f1ac928689e3 -requests==2.20.0 \ - --hash=sha256:99dcfdaaeb17caf6e526f32b6a7b780461512ab3f1d992187801694cba42770c \ - --hash=sha256:a84b8c9ab6239b578f22d1c21d51b696dcfe004032bb80ea832398d6909d7279 -six==1.10.0 \ - --hash=sha256:0ff78c403d9bccf5a425a6d31a12aa6b47f1c21ca4dc2573a7e2f32a97335eb1 \ - --hash=sha256:105f8d68616f8248e24bf0e9372ef04d3cc10104f1980f54d57b2ce73a5ad56a -traceback2==1.4.0 \ - --hash=sha256:8253cebec4b19094d67cc5ed5af99bf1dba1285292226e98a31929f87a5d6b23 \ - --hash=sha256:05acc67a09980c2ecfedd3423f7ae0104839eccb55fc645773e1caa0951c3030 -unittest2==1.1.0 \ - --hash=sha256:13f77d0875db6d9b435e1d4f41e74ad4cc2eb6e1d5c824996092b3430f088bb8 \ - --hash=sha256:22882a0e418c284e1f718a822b3b022944d53d2d908e1690b319a9d3eb2c0579 -zope.component==4.2.2 \ - --hash=sha256:282c112b55dd8e3c869a3571f86767c150ab1284a9ace2bdec226c592acaf81a \ - --no-binary zope.component -zope.event==4.1.0 \ - --hash=sha256:dc7a59a2fd91730d3793131a5d261b29e93ec4e2a97f1bc487ce8defee2fe786 \ - --no-binary zope.event -zope.interface==4.1.3 \ - --hash=sha256:f07b631f7a601cd8cbd3332d54f43142c7088a83299f859356f08d1d4d4259b3 \ - --hash=sha256:de5cca083b9439d8002fb76bbe6b4998c5a5a721fab25b84298967f002df4c94 \ - --hash=sha256:6788416f7ea7f5b8a97be94825377aa25e8bdc73463e07baaf9858b29e737077 \ - --hash=sha256:6f3230f7254518201e5a3708cbb2de98c848304f06e3ded8bfb39e5825cba2e1 \ - --hash=sha256:5fa575a5240f04200c3088427d0d4b7b737f6e9018818a51d8d0f927a6a2517a \ - --hash=sha256:522194ad6a545735edd75c8a83f48d65d1af064e432a7d320d64f56bafc12e99 \ - --hash=sha256:e8c7b2d40943f71c99148c97f66caa7f5134147f57423f8db5b4825099ce9a09 \ - --hash=sha256:279024f0208601c3caa907c53876e37ad88625f7eaf1cb3842dbe360b2287017 \ - --hash=sha256:2e221a9eec7ccc58889a278ea13dcfed5ef939d80b07819a9a8b3cb1c681484f \ - --hash=sha256:69118965410ec86d44dc6b9017ee3ddbd582e0c0abeef62b3a19dbf6c8ad132b \ - --hash=sha256:d04df8686ec864d0cade8cf199f7f83aecd416109a20834d568f8310ded12dea \ - --hash=sha256:e75a947e15ee97e7e71e02ea302feb2fc62d3a2bb4668bf9dfbed43a506ac7e7 \ - --hash=sha256:4e45d22fb883222a5ab9f282a116fec5ee2e8d1a568ccff6a2d75bbd0eb6bcfc \ - --hash=sha256:bce9339bb3c7a55e0803b63d21c5839e8e479bc85c4adf42ae415b72f94facb2 \ - --hash=sha256:928138365245a0e8869a5999fbcc2a45475a0a6ed52a494d60dbdc540335fedd \ - --hash=sha256:0d841ba1bb840eea0e6489dc5ecafa6125554971f53b5acb87764441e61bceba \ - --hash=sha256:b09c8c1d47b3531c400e0195697f1414a63221de6ef478598a4f1460f7d9a392 -requests-toolbelt==0.8.0 \ - --hash=sha256:42c9c170abc2cacb78b8ab23ac957945c7716249206f90874651971a4acff237 \ - --hash=sha256:f6a531936c6fa4c6cfce1b9c10d5c4f498d16528d2a54a22ca00011205a187b5 -chardet==3.0.2 \ - --hash=sha256:4f7832e7c583348a9eddd927ee8514b3bf717c061f57b21dbe7697211454d9bb \ - --hash=sha256:6ebf56457934fdce01fb5ada5582762a84eed94cad43ed877964aebbdd8174c0 -urllib3==1.24.1 \ - --hash=sha256:61bf29cada3fc2fbefad4fdf059ea4bd1b4a86d2b6d15e1c7c0b582b9752fe39 \ - --hash=sha256:de9529817c93f27c8ccbfead6985011db27bd0ddfcdb2d86f3f663385c6a9c22 -certifi==2017.4.17 \ - --hash=sha256:f4318671072f030a33c7ca6acaef720ddd50ff124d1388e50c1bda4cbd6d7010 \ - --hash=sha256:f7527ebf7461582ce95f7a9e03dd141ce810d40590834f4ec20cddd54234c10a + --hash=sha256:67d59d66cdba8d624e0389b87b2a83a176f21f16a87553b50f5703b23f29bac2 +pytz==2019.3 \ + --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \ + --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be +requests==2.22.0 \ + --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 \ + --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31 +requests-toolbelt==0.9.1 \ + --hash=sha256:380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f \ + --hash=sha256:968089d4584ad4ad7c171454f0a5c6dac23971e9472521ea3b6d49d610aa6fc0 +six==1.14.0 \ + --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \ + --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c +urllib3==1.25.8 \ + --hash=sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc \ + --hash=sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc +zope.component==4.6 \ + --hash=sha256:ec2afc5bbe611dcace98bb39822c122d44743d635dafc7315b9aef25097db9e6 +zope.deferredimport==4.3.1 \ + --hash=sha256:57b2345e7b5eef47efcd4f634ff16c93e4265de3dcf325afc7315ade48d909e1 \ + --hash=sha256:9a0c211df44aa95f1c4e6d2626f90b400f56989180d3ef96032d708da3d23e0a +zope.deprecation==4.4.0 \ + --hash=sha256:0d453338f04bacf91bbfba545d8bcdf529aa829e67b705eac8c1a7fdce66e2df \ + --hash=sha256:f1480b74995958b24ce37b0ef04d3663d2683e5d6debc96726eff18acf4ea113 +zope.event==4.4 \ + --hash=sha256:69c27debad9bdacd9ce9b735dad382142281ac770c4a432b533d6d65c4614bcf \ + --hash=sha256:d8e97d165fd5a0997b45f5303ae11ea3338becfe68c401dd88ffd2113fe5cae7 +zope.hookable==5.0.0 \ + --hash=sha256:0992a0dd692003c09fb958e1480cebd1a28f2ef32faa4857d864f3ca8e9d6952 \ + --hash=sha256:0f325838dbac827a1e2ed5d482c1f2656b6844dc96aa098f7727e76395fcd694 \ + --hash=sha256:22a317ba00f61bac99eac1a5e330be7cb8c316275a21269ec58aa396b602af0c \ + --hash=sha256:25531cb5e7b35e8a6d1d6eddef624b9a22ce5dcf8f4448ef0f165acfa8c3fc21 \ + --hash=sha256:30890892652766fc80d11f078aca9a5b8150bef6b88aba23799581a53515c404 \ + --hash=sha256:342d682d93937e5b8c232baffb32a87d5eee605d44f74566657c64a239b7f342 \ + --hash=sha256:46b2fddf1f5aeb526e02b91f7e62afbb9fff4ffd7aafc97cdb00a0d717641567 \ + --hash=sha256:523318ff96df9b8d378d997c00c5d4cbfbff68dc48ff5ee5addabdb697d27528 \ + --hash=sha256:53aa02eb8921d4e667c69d76adeed8fe426e43870c101cb08dcd2f3468aff742 \ + --hash=sha256:62e79e8fdde087cb20822d7874758f5acbedbffaf3c0fbe06309eb8a41ee4e06 \ + --hash=sha256:74bf2f757f7385b56dc3548adae508d8b3ef952d600b4b12b88f7d1706b05dcc \ + --hash=sha256:751ee9d89eb96e00c1d7048da9725ce392a708ed43406416dc5ed61e4d199764 \ + --hash=sha256:7b83bc341e682771fe810b360cd5d9c886a948976aea4b979ff214e10b8b523b \ + --hash=sha256:81eeeb27dbb0ddaed8070daee529f0d1bfe4f74c7351cce2aaca3ea287c4cc32 \ + --hash=sha256:856509191e16930335af4d773c0fc31a17bae8991eb6f167a09d5eddf25b56cc \ + --hash=sha256:8853e81fd07b18fa9193b19e070dc0557848d9945b1d2dac3b7782543458c87d \ + --hash=sha256:94506a732da2832029aecdfe6ea07eb1b70ee06d802fff34e1b3618fe7cdf026 \ + --hash=sha256:95ad874a8cc94e786969215d660143817f745225579bfe318c4676e218d3147c \ + --hash=sha256:9758ec9174966ffe5c499b6c3d149f80aa0a9238020006a2b87c6af5963fcf48 \ + --hash=sha256:a169823e331da939aa7178fc152e65699aeb78957e46c6f80ccb50ee4c3616c2 \ + --hash=sha256:a67878a798f6ca292729a28c2226592b3d000dc6ee7825d31887b553686c7ac7 \ + --hash=sha256:a9a6d9eb2319a09905670810e2de971d6c49013843700b4975e2fc0afe96c8db \ + --hash=sha256:b3e118b58a3d2301960e6f5f25736d92f6b9f861728d3b8c26d69f54d8a157d2 \ + --hash=sha256:ca6705c2a1fb5059a4efbe9f5426be4cdf71b3c9564816916fc7aa7902f19ede \ + --hash=sha256:cf711527c9d4ae72085f137caffb4be74fc007ffb17cd103628c7d5ba17e205f \ + --hash=sha256:d087602a6845ebe9d5a1c5a949fedde2c45f372d77fbce4f7fe44b68b28a1d03 \ + --hash=sha256:d1080e1074ddf75ad6662a9b34626650759c19a9093e1a32a503d37e48da135b \ + --hash=sha256:db9c60368aff2b7e6c47115f3ad9bd6e96aa298b12ed5f8cb13f5673b30be565 \ + --hash=sha256:dbeb127a04473f5a989169eb400b67beb921c749599b77650941c21fe39cb8d9 \ + --hash=sha256:dca336ca3682d869d291d7cd18284f6ff6876e4244eb1821430323056b000e2c \ + --hash=sha256:dd69a9be95346d10c853b6233fcafe3c0315b89424b378f2ad45170d8e161568 \ + --hash=sha256:dd79f8fae5894f1ee0a0042214685f2d039341250c994b825c10a4cd075d80f6 \ + --hash=sha256:e647d850aa1286d98910133cee12bd87c354f7b7bb3f3cd816a62ba7fa2f7007 \ + --hash=sha256:f37a210b5c04b2d4e4bac494ab15b70196f219a1e1649ddca78560757d4278fb \ + --hash=sha256:f67820b6d33a705dc3c1c457156e51686f7b350ff57f2112e1a9a4dad38ec268 \ + --hash=sha256:f68969978ccf0e6123902f7365aae5b7a9e99169d4b9105c47cf28e788116894 \ + --hash=sha256:f717a0b34460ae1ac0064e91b267c0588ac2c098ffd695992e72cd5462d97a67 \ + --hash=sha256:f9d58ccec8684ca276d5a4e7b0dfacca028336300a8f715d616d9f0ce9ae8096 \ + --hash=sha256:fcc3513a54e656067cbf7b98bab0d6b9534b9eabc666d1f78aad6acdf0962736 +zope.interface==4.7.1 \ + --hash=sha256:048b16ac882a05bc7ef534e8b9f15c9d7a6c190e24e8938a19b7617af4ed854a \ + --hash=sha256:05816cf8e7407cf62f2ec95c0a5d69ec4fa5741d9ccd10db9f21691916a9a098 \ + --hash=sha256:065d6a1ac89d35445168813bed45048ed4e67a4cdfc5a68fdb626a770378869f \ + --hash=sha256:14157421f4121a57625002cc4f48ac7521ea238d697c4a4459a884b62132b977 \ + --hash=sha256:18dc895945694f397a0be86be760ff664b790f95d8e7752d5bab80284ff9105d \ + --hash=sha256:1962c9f838bd6ae4075d0014f72697510daefc7e1c7e48b2607df0b6e157989c \ + --hash=sha256:1a67408cacd198c7e6274a19920bb4568d56459e659e23c4915528686ac1763a \ + --hash=sha256:21bf781076dd616bd07cf0223f79d61ab4f45176076f90bc2890e18c48195da4 \ + --hash=sha256:21c0a5d98650aebb84efa16ce2c8df1a46bdc4fe8a9e33237d0ca0b23f416ead \ + --hash=sha256:23cfeea25d1e42ff3bf4f9a0c31e9d5950aa9e7c4b12f0c4bd086f378f7b7a71 \ + --hash=sha256:24b6fce1fb71abf9f4093e3259084efcc0ef479f89356757780685bd2b06ef37 \ + --hash=sha256:24f84ce24eb6b5fcdcb38ad9761524f1ae96f7126abb5e597f8a3973d9921409 \ + --hash=sha256:25e0ef4a824017809d6d8b0ce4ab3288594ba283e4d4f94d8cfb81d73ed65114 \ + --hash=sha256:2e8fdd625e9aba31228e7ddbc36bad5c38dc3ee99a86aa420f89a290bd987ce9 \ + --hash=sha256:2f3bc2f49b67b1bea82b942d25bc958d4f4ea6709b411cb2b6b9718adf7914ce \ + --hash=sha256:35d24be9d04d50da3a6f4d61de028c1dd087045385a0ff374d93ef85af61b584 \ + --hash=sha256:35dbe4e8c73003dff40dfaeb15902910a4360699375e7b47d3c909a83ff27cd0 \ + --hash=sha256:3dfce831b824ab5cf446ed0c350b793ac6fa5fe33b984305cb4c966a86a8fb79 \ + --hash=sha256:3f7866365df5a36a7b8de8056cd1c605648f56f9a226d918ed84c85d25e8d55f \ + --hash=sha256:455cc8c01de3bac6f9c223967cea41f4449f58b4c2e724ec8177382ddd183ab4 \ + --hash=sha256:4bb937e998be9d5e345f486693e477ba79e4344674484001a0b646be1d530487 \ + --hash=sha256:52303a20902ca0888dfb83230ca3ee6fbe63c0ad1dd60aa0bba7958ccff454d8 \ + --hash=sha256:6e0a897d4e09859cc80c6a16a29697406ead752292ace17f1805126a4f63c838 \ + --hash=sha256:6e1816e7c10966330d77af45f77501f9a68818c065dec0ad11d22b50a0e212e7 \ + --hash=sha256:73b5921c5c6ce3358c836461b5470bf675601c96d5e5d8f2a446951470614f67 \ + --hash=sha256:8093cd45cdb5f6c8591cfd1af03d32b32965b0f79b94684cd0c9afdf841982bb \ + --hash=sha256:864b4a94b60db301899cf373579fd9ef92edddbf0fb2cd5ae99f53ef423ccc56 \ + --hash=sha256:8a27b4d3ea9c6d086ce8e7cdb3e8d319b6752e2a03238a388ccc83ccbe165f50 \ + --hash=sha256:91b847969d4784abd855165a2d163f72ac1e58e6dce09a5e46c20e58f19cc96d \ + --hash=sha256:b47b1028be4758c3167e474884ccc079b94835f058984b15c145966c4df64d27 \ + --hash=sha256:b68814a322835d8ad671b7acc23a3b2acecba527bb14f4b53fc925f8a27e44d8 \ + --hash=sha256:bcb50a032c3b6ec7fb281b3a83d2b31ab5246c5b119588725b1350d3a1d9f6a3 \ + --hash=sha256:c56db7d10b25ce8918b6aec6b08ac401842b47e6c136773bfb3b590753f7fb67 \ + --hash=sha256:c94b77a13d4f47883e4f97f9fa00f5feadd38af3e6b3c7be45cfdb0a14c7149b \ + --hash=sha256:db381f6fdaef483ad435f778086ccc4890120aff8df2ba5cfeeac24d280b3145 \ + --hash=sha256:e6487d01c8b7ed86af30ea141fcc4f93f8a7dde26f94177c1ad637c353bd5c07 \ + --hash=sha256:e86923fa728dfba39c5bb6046a450bd4eec8ad949ac404eca728cfce320d1732 \ + --hash=sha256:f6ca36dc1e9eeb46d779869c60001b3065fb670b5775c51421c099ea2a77c3c9 \ + --hash=sha256:fb62f2cbe790a50d95593fb40e8cca261c31a2f5637455ea39440d6457c2ba25 +zope.proxy==4.3.3 \ + --hash=sha256:04646ac04ffa9c8e32fb2b5c3cd42995b2548ea14251f3c21ca704afae88e42c \ + --hash=sha256:07b6bceea232559d24358832f1cd2ed344bbf05ca83855a5b9698b5f23c5ed60 \ + --hash=sha256:1ef452cc02e0e2f8e3c917b1a5b936ef3280f2c2ca854ee70ac2164d1655f7e6 \ + --hash=sha256:22bf61857c5977f34d4e391476d40f9a3b8c6ab24fb0cac448d42d8f8b9bf7b2 \ + --hash=sha256:299870e3428cbff1cd9f9b34144e76ecdc1d9e3192a8cf5f1b0258f47a239f58 \ + --hash=sha256:2bfc36bfccbe047671170ea5677efd3d5ab730a55d7e45611d76d495e5b96766 \ + --hash=sha256:32e82d5a640febc688c0789e15ea875bf696a10cf358f049e1ed841f01710a9b \ + --hash=sha256:3b2051bdc4bc3f02fa52483f6381cf40d4d48167645241993f9d7ebbd142ed9b \ + --hash=sha256:3f734bd8a08f5185a64fb6abb8f14dc97ec27a689ca808fb7a83cdd38d745e4f \ + --hash=sha256:3f78dd8de3112df8bbd970f0916ac876dc3fbe63810bd1cf7cc5eec4cbac4f04 \ + --hash=sha256:4eabeb48508953ba1f3590ad0773b8daea9e104eec66d661917e9bbcd7125a67 \ + --hash=sha256:4f05ecc33808187f430f249cb1ccab35c38f570b181f2d380fbe253da94b18d8 \ + --hash=sha256:4f4f4cbf23d3afc1526294a31e7b3eaa0f682cc28ac5366065dc1d6bb18bd7be \ + --hash=sha256:5483d5e70aacd06f0aa3effec9fed597c0b50f45060956eeeb1203c44d4338c3 \ + --hash=sha256:56a5f9b46892b115a75d0a1f2292431ad5988461175826600acc69a24cb3edee \ + --hash=sha256:64bb63af8a06f736927d260efdd4dfc5253d42244f281a8063e4b9eea2ddcbc5 \ + --hash=sha256:653f8cbefcf7c6ac4cece2cdef367c4faa2b7c19795d52bd7cbec11a8739a7c1 \ + --hash=sha256:664211d63306e4bd4eec35bf2b4bd9db61c394037911cf2d1804c43b511a49f1 \ + --hash=sha256:6651e6caed66a8fff0fef1a3e81c0ed2253bf361c0fdc834500488732c5d16e9 \ + --hash=sha256:6c1fba6cdfdf105739d3069cf7b07664f2944d82a8098218ab2300a82d8f40fc \ + --hash=sha256:6e64246e6e9044a4534a69dca1283c6ddab6e757be5e6874f69024329b3aa61f \ + --hash=sha256:838390245c7ec137af4993c0c8052f49d5ec79e422b4451bfa37fee9b9ccaa01 \ + --hash=sha256:856b410a14793069d8ba35f33fff667213ea66f2df25a0024cc72a7493c56d4c \ + --hash=sha256:8b932c364c1d1605a91907a41128ed0ee8a2d326fc0fafb2c55cd46f545f4599 \ + --hash=sha256:9086cf6d20f08dae7f296a78f6c77d1f8d24079d448f023ee0eb329078dd35e1 \ + --hash=sha256:9698533c14afa0548188de4968a7932d1f3f965f3f5ba1474de673596bb875af \ + --hash=sha256:9b12b05dd7c28f5068387c1afee8cb94f9d02501e7ef495a7c5c7e27139b96ad \ + --hash=sha256:a884c7426a5bc6fb7fc71a55ad14e66818e13f05b78b20a6f37175f324b7acb8 \ + --hash=sha256:abe9e7f1a3e76286c5f5baf2bf5162d41dc0310da493b34a2c36555f38d928f7 \ + --hash=sha256:bd6fde63b015a27262be06bd6bbdd895273cc2bdf2d4c7e1c83711d26a8fbace \ + --hash=sha256:bda7c62c954f47b87ed9a89f525eee1b318ec7c2162dfdba76c2ccfa334e0caa \ + --hash=sha256:be8a4908dd3f6e965993c0068b006bdbd0474fbcbd1da4893b49356e73fc1557 \ + --hash=sha256:ced65fc3c7d7205267506d854bb1815bb445899cca9d21d1d4b949070a635546 \ + --hash=sha256:dac4279aa05055d3897ab5e5ee5a7b39db121f91df65a530f8b1ac7f9bd93119 \ + --hash=sha256:e4f1863056e3e4f399c285b67fa816f411a7bfa1c81ef50e186126164e396e59 \ + --hash=sha256:ecd85f68b8cd9ab78a0141e87ea9a53b2f31fd9b1350a1c44da1f7481b5363ef \ + --hash=sha256:ed269b83750413e8fc5c96276372f49ee3fcb7ed61c49fe8e5a67f54459a5a4a \ + --hash=sha256:f19b0b80cba73b204dee68501870b11067711d21d243fb6774256d3ca2e5391f \ + --hash=sha256:ffdafb98db7574f9da84c489a10a5d582079a888cb43c64e9e6b0e3fe1034685 # Contains the requirements for the letsencrypt package. # @@ -1180,18 +1540,18 @@ letsencrypt==0.7.0 \ --hash=sha256:105a5fb107e45bcd0722eb89696986dcf5f08a86a321d6aef25a0c7c63375ade \ --hash=sha256:c36e532c486a7e92155ee09da54b436a3c420813ec1c590b98f635d924720de9 -certbot==0.30.2 \ - --hash=sha256:e411b72fa86eec1018e6de28e649e8c9c71191a7431dcc77f207b57ca9484c11 \ - --hash=sha256:534487cb552ced8e47948ba3d2e7ca12c3a439133fc609485012b1a02fc7776e -acme==0.30.2 \ - --hash=sha256:68982576492dfa99c7e2be0fce4371adc9344740b05420ce0ab53238d2bb9b3b \ - --hash=sha256:295a5b7fce9f908e6e5cff8c40be1a3daf3e1ebabd2e139a4c87274e68eeb8f2 -certbot-apache==0.30.2 \ - --hash=sha256:3b7fa4e59772da7c9975ef2a49ceff157c9d7cb31eb9475928b5986d89701a3a \ - --hash=sha256:32fa915a8a51810fdfe828ac1361da4425c231d7384891e49e6338e4741464b2 -certbot-nginx==0.30.2 \ - --hash=sha256:7dc785f6f0c0c57b19cea8d98f9ea8feef53945613967b52c9348c81327010e2 \ - --hash=sha256:6ba4dd772d0c7cdfb3383ca325b35639e01ac9e142e4baa6445cd85c7fb59552 +certbot==1.2.0 \ + --hash=sha256:e25c17125c00b3398c8e9b9d54ef473c0e8f5aff53389f313a51b06cf472d335 \ + --hash=sha256:95dcbae085f8e4eb18442fe7b12994b08964a9a6e8e352e556cdb4a8a625373c +acme==1.2.0 \ + --hash=sha256:284d22fde75687a8ea72d737cac6bcbdc91f3c796221aa25378b8732ba6f6875 \ + --hash=sha256:0630c740d49bda945e97bd35fc8d6f02d082c8cb9e18f8fec0dbb3d395ac26ab +certbot-apache==1.2.0 \ + --hash=sha256:3f7493918353d3bd6067d446a2cf263e03831c4c10ec685b83d644b47767090d \ + --hash=sha256:b46e9def272103a68108e48bf7e410ea46801529b1ea6954f6506b14dd9df9b3 +certbot-nginx==1.2.0 \ + --hash=sha256:efd32a2b32f2439279da446b6bf67684f591f289323c5f494ebfd86a566a28fd \ + --hash=sha256:6fd7cf4f2545ad66e57000343227df9ccccaf04420e835e05cb3250fac1fa6db UNLIKELY_EOF # ------------------------------------------------------------------------- @@ -1221,7 +1581,6 @@ from distutils.version import StrictVersion from hashlib import sha256 from os import environ from os.path import join -from pipes import quote from shutil import rmtree try: from subprocess import check_output @@ -1241,7 +1600,7 @@ except ImportError: cmd = popenargs[0] raise CalledProcessError(retcode, cmd) return output -from sys import exit, version_info +import sys from tempfile import mkdtemp try: from urllib2 import build_opener, HTTPHandler, HTTPSHandler @@ -1263,7 +1622,7 @@ maybe_argparse = ( [('18/dd/e617cfc3f6210ae183374cd9f6a26b20514bbb5a792af97949c5aacddf0f/' 'argparse-1.4.0.tar.gz', '62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4')] - if version_info < (2, 7, 0) else []) + if sys.version_info < (2, 7, 0) else []) PACKAGES = maybe_argparse + [ @@ -1344,7 +1703,8 @@ def get_index_base(): def main(): - pip_version = StrictVersion(check_output(['pip', '--version']) + python = sys.executable or 'python' + pip_version = StrictVersion(check_output([python, '-m', 'pip', '--version']) .decode('utf-8').split()[1]) has_pip_cache = pip_version >= StrictVersion('6.0') index_base = get_index_base() @@ -1354,12 +1714,12 @@ def main(): temp, digest) for path, digest in PACKAGES] - check_output('pip install --no-index --no-deps -U ' + - # Disable cache since we're not using it and it otherwise - # sometimes throws permission warnings: - ('--no-cache-dir ' if has_pip_cache else '') + - ' '.join(quote(d) for d in downloads), - shell=True) + # Calling pip as a module is the preferred way to avoid problems about pip self-upgrade. + command = [python, '-m', 'pip', 'install', '--no-index', '--no-deps', '-U'] + # Disable cache since it is not used and it otherwise sometimes throws permission warnings: + command.extend(['--no-cache-dir'] if has_pip_cache else []) + command.extend(downloads) + check_output(command) except HashError as exc: print(exc) except Exception: @@ -1372,7 +1732,7 @@ def main(): if __name__ == '__main__': - exit(main()) + sys.exit(main()) UNLIKELY_EOF # ------------------------------------------------------------------------- @@ -1425,6 +1785,9 @@ UNLIKELY_EOF say "Installation succeeded." fi + # If you're modifying any of the code after this point in this current `if` block, you + # may need to update the "$DEPRECATED_OS" = 1 case at the beginning of phase 2 as well. + if [ "$INSTALL_ONLY" = 1 ]; then say "Certbot is installed." exit 0 @@ -1456,6 +1819,24 @@ else exit 0 fi + DeterminePythonVersion "NOCRASH" + # Don't warn about file permissions if the user disabled the check or we + # can't find an up-to-date Python. + if [ "$PYVER" -ge "$MIN_PYVER" -a "$NO_PERMISSIONS_CHECK" != 1 ]; then + # If the script fails for some reason, don't break certbot-auto. + set +e + # Suppress unexpected error output. + CHECK_PERM_OUT=$(CheckPathPermissions "$LE_PYTHON" "$0" 2>/dev/null) + CHECK_PERM_STATUS="$?" + set -e + # Only print output if the script ran successfully and it actually produced + # output. The latter check resolves + # https://github.com/certbot/certbot/issues/7012. + if [ "$CHECK_PERM_STATUS" = 0 -a -n "$CHECK_PERM_OUT" ]; then + error "$CHECK_PERM_OUT" + fi + fi + if [ "$NO_SELF_UPGRADE" != 1 ]; then TEMP_DIR=$(TempDir) trap 'rm -rf "$TEMP_DIR"' EXIT @@ -1612,37 +1993,41 @@ if __name__ == '__main__': UNLIKELY_EOF # --------------------------------------------------------------------------- - DeterminePythonVersion "NOCRASH" if [ "$PYVER" -lt "$MIN_PYVER" ]; then error "WARNING: couldn't find Python $MIN_PYTHON_VERSION+ to check for updates." elif ! REMOTE_VERSION=`"$LE_PYTHON" "$TEMP_DIR/fetch.py" --latest-version` ; then error "WARNING: unable to check for updates." fi - LE_VERSION_STATE=`CompareVersions "$LE_PYTHON" "$LE_AUTO_VERSION" "$REMOTE_VERSION"` - if [ "$LE_VERSION_STATE" = "UNOFFICIAL" ]; then - say "Unofficial certbot-auto version detected, self-upgrade is disabled: $LE_AUTO_VERSION" - elif [ "$LE_VERSION_STATE" = "OUTDATED" ]; then - say "Upgrading certbot-auto $LE_AUTO_VERSION to $REMOTE_VERSION..." + # If for any reason REMOTE_VERSION is not set, let's assume certbot-auto is up-to-date, + # and do not go into the self-upgrading process. + if [ -n "$REMOTE_VERSION" ]; then + LE_VERSION_STATE=`CompareVersions "$LE_PYTHON" "$LE_AUTO_VERSION" "$REMOTE_VERSION"` - # Now we drop into Python so we don't have to install even more - # dependencies (curl, etc.), for better flow control, and for the option of - # future Windows compatibility. - "$LE_PYTHON" "$TEMP_DIR/fetch.py" --le-auto-script "v$REMOTE_VERSION" + if [ "$LE_VERSION_STATE" = "UNOFFICIAL" ]; then + say "Unofficial certbot-auto version detected, self-upgrade is disabled: $LE_AUTO_VERSION" + elif [ "$LE_VERSION_STATE" = "OUTDATED" ]; then + say "Upgrading certbot-auto $LE_AUTO_VERSION to $REMOTE_VERSION..." - # Install new copy of certbot-auto. - # TODO: Deal with quotes in pathnames. - say "Replacing certbot-auto..." - # Clone permissions with cp. chmod and chown don't have a --reference - # option on macOS or BSD, and stat -c on Linux is stat -f on macOS and BSD: - cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone" - cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone" - # Using mv rather than cp leaves the old file descriptor pointing to the - # original copy so the shell can continue to read it unmolested. mv across - # filesystems is non-atomic, doing `rm dest, cp src dest, rm src`, but the - # cp is unlikely to fail if the rm doesn't. - mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0" - fi # A newer version is available. + # Now we drop into Python so we don't have to install even more + # dependencies (curl, etc.), for better flow control, and for the option of + # future Windows compatibility. + "$LE_PYTHON" "$TEMP_DIR/fetch.py" --le-auto-script "v$REMOTE_VERSION" + + # Install new copy of certbot-auto. + # TODO: Deal with quotes in pathnames. + say "Replacing certbot-auto..." + # Clone permissions with cp. chmod and chown don't have a --reference + # option on macOS or BSD, and stat -c on Linux is stat -f on macOS and BSD: + cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone" + cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone" + # Using mv rather than cp leaves the old file descriptor pointing to the + # original copy so the shell can continue to read it unmolested. mv across + # filesystems is non-atomic, doing `rm dest, cp src dest, rm src`, but the + # cp is unlikely to fail if the rm doesn't. + mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0" + fi # A newer version is available. + fi fi # Self-upgrading is allowed. RerunWithArgs --le-auto-phase2 "$@" diff --git a/letsencrypt-auto-source/letsencrypt-auto.sig b/letsencrypt-auto-source/letsencrypt-auto.sig index f5175187a..fefc81b37 100644 --- a/letsencrypt-auto-source/letsencrypt-auto.sig +++ b/letsencrypt-auto-source/letsencrypt-auto.sig @@ -1 +1,3 @@ -^)kAT{`$91`tp ^T%nl/cnjdr=V+~\ ޡQ[ tYni5[N ZA&K(ڴd]L|?;UfRĨxK+nt;ekni,6ROŤPV|;l,W]zպiLgCZQD3 ԟ*ULf_O?_!ȵ6y1o+p \ No newline at end of file +< @!K27 + 3[&/,^eX^(mxTlݲeDd߹D!K{]4abZvZ NMdxy,Ә\E,|l*{ʓ>(L=fk23 >|A'd Q6BN tm +. ԅ`t#+'KApo>3Ql3'->~ /dev/null && break done else + MIN_PYVER=$MIN_PYVER2 + MIN_PYTHON_VERSION=$MIN_PYTHON_2_VERSION for LE_PYTHON in "$LE_PYTHON" python2.7 python27 python2 python; do # Break (while keeping the LE_PYTHON value) if found. $EXISTS "$LE_PYTHON" > /dev/null && break @@ -278,7 +293,7 @@ DeterminePythonVersion() { fi fi - PYVER=`"$LE_PYTHON" -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//'` + PYVER=$("$LE_PYTHON" -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') if [ "$PYVER" -lt "$MIN_PYVER" ]; then if [ "$1" != "NOCRASH" ]; then error "You have an ancient version of Python entombed in your operating system..." @@ -291,6 +306,7 @@ DeterminePythonVersion() { {{ bootstrappers/deb_common.sh }} {{ bootstrappers/rpm_common_base.sh }} {{ bootstrappers/rpm_common.sh }} +{{ bootstrappers/rpm_python3_legacy.sh }} {{ bootstrappers/rpm_python3.sh }} {{ bootstrappers/suse_common.sh }} {{ bootstrappers/arch_common.sh }} @@ -323,20 +339,71 @@ elif [ -f /etc/redhat-release ]; then prev_le_python="$LE_PYTHON" unset LE_PYTHON DeterminePythonVersion "NOCRASH" + + RPM_DIST_NAME=`(. /etc/os-release 2> /dev/null && echo $ID) || echo "unknown"` + + if [ "$PYVER" -eq 26 -a $(uname -m) != 'x86_64' ]; then + # 32 bits CentOS 6 and affiliates are not supported anymore by certbot-auto. + DEPRECATED_OS=1 + fi + + # Set RPM_DIST_VERSION to VERSION_ID from /etc/os-release after splitting on + # '.' characters (e.g. "8.0" becomes "8"). If the command exits with an + # error, RPM_DIST_VERSION is set to "unknown". + RPM_DIST_VERSION=$( (. /etc/os-release 2> /dev/null && echo "$VERSION_ID") | cut -d '.' -f1 || echo "unknown") + + # If RPM_DIST_VERSION is an empty string or it contains any nonnumeric + # characters, the value is unexpected so we set RPM_DIST_VERSION to 0. + if [ -z "$RPM_DIST_VERSION" ] || [ -n "$(echo "$RPM_DIST_VERSION" | tr -d '[0-9]')" ]; then + RPM_DIST_VERSION=0 + fi + + # Handle legacy RPM distributions if [ "$PYVER" -eq 26 ]; then + # Check if an automated bootstrap can be achieved on this system. + if ! Python36SclIsAvailable; then + INTERACTIVE_BOOTSTRAP=1 + fi + Bootstrap() { - BootstrapMessage "RedHat-based OSes that will use Python3" - BootstrapRpmPython3 + BootstrapMessage "Legacy RedHat-based OSes that will use Python3" + BootstrapRpmPython3Legacy } USE_PYTHON_3=1 - BOOTSTRAP_VERSION="BootstrapRpmPython3 $BOOTSTRAP_RPM_PYTHON3_VERSION" + BOOTSTRAP_VERSION="BootstrapRpmPython3Legacy $BOOTSTRAP_RPM_PYTHON3_LEGACY_VERSION" + + # Try now to enable SCL rh-python36 for systems already bootstrapped + # NB: EnablePython36SCL has been defined along with BootstrapRpmPython3Legacy in certbot-auto + EnablePython36SCL else - Bootstrap() { - BootstrapMessage "RedHat-based OSes" - BootstrapRpmCommon - } - BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION" + # Starting to Fedora 29, python2 is on a deprecation path. Let's move to python3 then. + # RHEL 8 also uses python3 by default. + if [ "$RPM_DIST_NAME" = "fedora" -a "$RPM_DIST_VERSION" -ge 29 ]; then + RPM_USE_PYTHON_3=1 + elif [ "$RPM_DIST_NAME" = "rhel" -a "$RPM_DIST_VERSION" -ge 8 ]; then + RPM_USE_PYTHON_3=1 + elif [ "$RPM_DIST_NAME" = "centos" -a "$RPM_DIST_VERSION" -ge 8 ]; then + RPM_USE_PYTHON_3=1 + else + RPM_USE_PYTHON_3=0 + fi + + if [ "$RPM_USE_PYTHON_3" = 1 ]; then + Bootstrap() { + BootstrapMessage "RedHat-based OSes that will use Python3" + BootstrapRpmPython3 + } + USE_PYTHON_3=1 + BOOTSTRAP_VERSION="BootstrapRpmPython3 $BOOTSTRAP_RPM_PYTHON3_VERSION" + else + Bootstrap() { + BootstrapMessage "RedHat-based OSes" + BootstrapRpmCommon + } + BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION" + fi fi + LE_PYTHON="$prev_le_python" elif [ -f /etc/os-release ] && `grep -q openSUSE /etc/os-release` ; then Bootstrap() { @@ -411,6 +478,13 @@ if [ "$NO_BOOTSTRAP" = 1 ]; then unset BOOTSTRAP_VERSION fi +if [ "$DEPRECATED_OS" = 1 ]; then + Bootstrap() { + error "Skipping bootstrap because certbot-auto is deprecated on this system." + } + unset BOOTSTRAP_VERSION +fi + # Sets PREV_BOOTSTRAP_VERSION to the identifier for the bootstrap script used # to install OS dependencies on this system. PREV_BOOTSTRAP_VERSION isn't set # if it is unknown how OS dependencies were installed on this system. @@ -480,10 +554,48 @@ else: UNLIKELY_EOF } +# Create a new virtual environment for Certbot. It will overwrite any existing one. +# Parameters: LE_PYTHON, VENV_PATH, PYVER, VERBOSE +CreateVenv() { + "$1" - "$2" "$3" "$4" << "UNLIKELY_EOF" +{{ create_venv.py }} +UNLIKELY_EOF +} + +# Check that the given PATH_TO_CHECK has secured permissions. +# Parameters: LE_PYTHON, PATH_TO_CHECK +CheckPathPermissions() { + "$1" - "$2" << "UNLIKELY_EOF" +{{ check_permissions.py }} +UNLIKELY_EOF +} + if [ "$1" = "--le-auto-phase2" ]; then # Phase 2: Create venv, install LE, and run. shift 1 # the --le-auto-phase2 arg + + if [ "$DEPRECATED_OS" = 1 ]; then + # Phase 2 damage control mode for deprecated OSes. + # In this situation, we bypass any bootstrap or certbot venv setup. + error "Your system is not supported by certbot-auto anymore." + + if [ ! -d "$VENV_PATH" ] && OldVenvExists; then + VENV_BIN="$OLD_VENV_PATH/bin" + fi + + if [ -f "$VENV_BIN/letsencrypt" -a "$INSTALL_ONLY" != 1 ]; then + error "Certbot will no longer receive updates." + error "Please visit https://certbot.eff.org/ to check for other alternatives." + "$VENV_BIN/letsencrypt" "$@" + exit 0 + else + error "Certbot cannot be installed." + error "Please visit https://certbot.eff.org/ to check for other alternatives." + exit 1 + fi + fi + SetPrevBootstrapVersion if [ -z "$PHASE_1_VERSION" -a "$USE_PYTHON_3" = 1 ]; then @@ -495,8 +607,15 @@ if [ "$1" = "--le-auto-phase2" ]; then # If the selected Bootstrap function isn't a noop and it differs from the # previously used version if [ -n "$BOOTSTRAP_VERSION" -a "$BOOTSTRAP_VERSION" != "$PREV_BOOTSTRAP_VERSION" ]; then - # if non-interactive mode or stdin and stdout are connected to a terminal - if [ \( "$NONINTERACTIVE" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then + # Check if we can rebootstrap without manual user intervention: this requires that + # certbot-auto is in non-interactive mode AND selected bootstrap does not claim to + # require a manual user intervention. + if [ "$NONINTERACTIVE" = 1 -a "$INTERACTIVE_BOOTSTRAP" != 1 ]; then + CAN_REBOOTSTRAP=1 + fi + # Check if rebootstrap can be done non-interactively and current shell is non-interactive + # (true if stdin and stdout are not attached to a terminal). + if [ \( "$CAN_REBOOTSTRAP" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then if [ -d "$VENV_PATH" ]; then rm -rf "$VENV_PATH" fi @@ -507,12 +626,21 @@ if [ "$1" = "--le-auto-phase2" ]; then ln -s "$VENV_PATH" "$OLD_VENV_PATH" fi RerunWithArgs "$@" + # Otherwise bootstrap needs to be done manually by the user. else - error "Skipping upgrade because new OS dependencies may need to be installed." - error - error "To upgrade to a newer version, please run this script again manually so you can" - error "approve changes or with --non-interactive on the command line to automatically" - error "install any required packages." + # If it is because bootstrapping is interactive, --non-interactive will be of no use. + if [ "$INTERACTIVE_BOOTSTRAP" = 1 ]; then + error "Skipping upgrade because new OS dependencies may need to be installed." + error "This requires manual user intervention: please run this script again manually." + # If this is because of the environment (eg. non interactive shell without + # --non-interactive flag set), help the user in that direction. + else + error "Skipping upgrade because new OS dependencies may need to be installed." + error + error "To upgrade to a newer version, please run this script again manually so you can" + error "approve changes or with --non-interactive on the command line to automatically" + error "install any required packages." + fi # Set INSTALLED_VERSION to be the same so we don't update the venv INSTALLED_VERSION="$LE_AUTO_VERSION" # Continue to use OLD_VENV_PATH if the new venv doesn't exist @@ -535,22 +663,7 @@ if [ "$1" = "--le-auto-phase2" ]; then if [ "$LE_AUTO_VERSION" != "$INSTALLED_VERSION" ]; then say "Creating virtual environment..." DeterminePythonVersion - rm -rf "$VENV_PATH" - if [ "$PYVER" -le 27 ]; then - # Use an environment variable instead of a flag for compatibility with old versions - if [ "$VERBOSE" = 1 ]; then - VIRTUALENV_NO_DOWNLOAD=1 virtualenv --no-site-packages --python "$LE_PYTHON" "$VENV_PATH" - else - VIRTUALENV_NO_DOWNLOAD=1 virtualenv --no-site-packages --python "$LE_PYTHON" "$VENV_PATH" \ - > /dev/null - fi - else - if [ "$VERBOSE" = 1 ]; then - "$LE_PYTHON" -m venv "$VENV_PATH" - else - "$LE_PYTHON" -m venv "$VENV_PATH" > /dev/null - fi - fi + CreateVenv "$LE_PYTHON" "$VENV_PATH" "$PYVER" "$VERBOSE" if [ -n "$BOOTSTRAP_VERSION" ]; then echo "$BOOTSTRAP_VERSION" > "$BOOTSTRAP_VERSION_PATH" @@ -622,6 +735,9 @@ UNLIKELY_EOF say "Installation succeeded." fi + # If you're modifying any of the code after this point in this current `if` block, you + # may need to update the "$DEPRECATED_OS" = 1 case at the beginning of phase 2 as well. + if [ "$INSTALL_ONLY" = 1 ]; then say "Certbot is installed." exit 0 @@ -653,6 +769,24 @@ else exit 0 fi + DeterminePythonVersion "NOCRASH" + # Don't warn about file permissions if the user disabled the check or we + # can't find an up-to-date Python. + if [ "$PYVER" -ge "$MIN_PYVER" -a "$NO_PERMISSIONS_CHECK" != 1 ]; then + # If the script fails for some reason, don't break certbot-auto. + set +e + # Suppress unexpected error output. + CHECK_PERM_OUT=$(CheckPathPermissions "$LE_PYTHON" "$0" 2>/dev/null) + CHECK_PERM_STATUS="$?" + set -e + # Only print output if the script ran successfully and it actually produced + # output. The latter check resolves + # https://github.com/certbot/certbot/issues/7012. + if [ "$CHECK_PERM_STATUS" = 0 -a -n "$CHECK_PERM_OUT" ]; then + error "$CHECK_PERM_OUT" + fi + fi + if [ "$NO_SELF_UPGRADE" != 1 ]; then TEMP_DIR=$(TempDir) trap 'rm -rf "$TEMP_DIR"' EXIT @@ -661,37 +795,41 @@ else {{ fetch.py }} UNLIKELY_EOF # --------------------------------------------------------------------------- - DeterminePythonVersion "NOCRASH" if [ "$PYVER" -lt "$MIN_PYVER" ]; then error "WARNING: couldn't find Python $MIN_PYTHON_VERSION+ to check for updates." elif ! REMOTE_VERSION=`"$LE_PYTHON" "$TEMP_DIR/fetch.py" --latest-version` ; then error "WARNING: unable to check for updates." fi - LE_VERSION_STATE=`CompareVersions "$LE_PYTHON" "$LE_AUTO_VERSION" "$REMOTE_VERSION"` - if [ "$LE_VERSION_STATE" = "UNOFFICIAL" ]; then - say "Unofficial certbot-auto version detected, self-upgrade is disabled: $LE_AUTO_VERSION" - elif [ "$LE_VERSION_STATE" = "OUTDATED" ]; then - say "Upgrading certbot-auto $LE_AUTO_VERSION to $REMOTE_VERSION..." + # If for any reason REMOTE_VERSION is not set, let's assume certbot-auto is up-to-date, + # and do not go into the self-upgrading process. + if [ -n "$REMOTE_VERSION" ]; then + LE_VERSION_STATE=`CompareVersions "$LE_PYTHON" "$LE_AUTO_VERSION" "$REMOTE_VERSION"` - # Now we drop into Python so we don't have to install even more - # dependencies (curl, etc.), for better flow control, and for the option of - # future Windows compatibility. - "$LE_PYTHON" "$TEMP_DIR/fetch.py" --le-auto-script "v$REMOTE_VERSION" + if [ "$LE_VERSION_STATE" = "UNOFFICIAL" ]; then + say "Unofficial certbot-auto version detected, self-upgrade is disabled: $LE_AUTO_VERSION" + elif [ "$LE_VERSION_STATE" = "OUTDATED" ]; then + say "Upgrading certbot-auto $LE_AUTO_VERSION to $REMOTE_VERSION..." - # Install new copy of certbot-auto. - # TODO: Deal with quotes in pathnames. - say "Replacing certbot-auto..." - # Clone permissions with cp. chmod and chown don't have a --reference - # option on macOS or BSD, and stat -c on Linux is stat -f on macOS and BSD: - cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone" - cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone" - # Using mv rather than cp leaves the old file descriptor pointing to the - # original copy so the shell can continue to read it unmolested. mv across - # filesystems is non-atomic, doing `rm dest, cp src dest, rm src`, but the - # cp is unlikely to fail if the rm doesn't. - mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0" - fi # A newer version is available. + # Now we drop into Python so we don't have to install even more + # dependencies (curl, etc.), for better flow control, and for the option of + # future Windows compatibility. + "$LE_PYTHON" "$TEMP_DIR/fetch.py" --le-auto-script "v$REMOTE_VERSION" + + # Install new copy of certbot-auto. + # TODO: Deal with quotes in pathnames. + say "Replacing certbot-auto..." + # Clone permissions with cp. chmod and chown don't have a --reference + # option on macOS or BSD, and stat -c on Linux is stat -f on macOS and BSD: + cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone" + cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone" + # Using mv rather than cp leaves the old file descriptor pointing to the + # original copy so the shell can continue to read it unmolested. mv across + # filesystems is non-atomic, doing `rm dest, cp src dest, rm src`, but the + # cp is unlikely to fail if the rm doesn't. + mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0" + fi # A newer version is available. + fi fi # Self-upgrading is allowed. RerunWithArgs --le-auto-phase2 "$@" diff --git a/letsencrypt-auto-source/pieces/bootstrappers/rpm_common_base.sh b/letsencrypt-auto-source/pieces/bootstrappers/rpm_common_base.sh index 326ad8b3f..2b00b199b 100644 --- a/letsencrypt-auto-source/pieces/bootstrappers/rpm_common_base.sh +++ b/letsencrypt-auto-source/pieces/bootstrappers/rpm_common_base.sh @@ -3,7 +3,9 @@ # Sets TOOL to the name of the package manager # Sets appropriate values for YES_FLAG and QUIET_FLAG based on $ASSUME_YES and $QUIET_FLAG. -# Enables EPEL if applicable and possible. +# Note: this function is called both while selecting the bootstrap scripts and +# during the actual bootstrap. Some things like prompting to user can be done in the latter +# case, but not in the former one. InitializeRPMCommonBase() { if type dnf 2>/dev/null then @@ -23,26 +25,6 @@ InitializeRPMCommonBase() { if [ "$QUIET" = 1 ]; then QUIET_FLAG='--quiet' fi - - if ! $TOOL list *virtualenv >/dev/null 2>&1; then - echo "To use Certbot, packages from the EPEL repository need to be installed." - if ! $TOOL list epel-release >/dev/null 2>&1; then - error "Enable the EPEL repository and try running Certbot again." - exit 1 - fi - if [ "$ASSUME_YES" = 1 ]; then - /bin/echo -n "Enabling the EPEL repository in 3 seconds..." - sleep 1s - /bin/echo -ne "\e[0K\rEnabling the EPEL repository in 2 seconds..." - sleep 1s - /bin/echo -e "\e[0K\rEnabling the EPEL repository in 1 second..." - sleep 1s - fi - if ! $TOOL install $YES_FLAG $QUIET_FLAG epel-release; then - error "Could not enable EPEL. Aborting bootstrap!" - exit 1 - fi - fi } BootstrapRpmCommonBase() { diff --git a/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3.sh b/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3.sh index b011a7235..ac0553db5 100644 --- a/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3.sh +++ b/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3.sh @@ -4,15 +4,15 @@ BOOTSTRAP_RPM_PYTHON3_VERSION=1 BootstrapRpmPython3() { # Tested with: - # - CentOS 6 + # - Fedora 29 InitializeRPMCommonBase - # EPEL uses python34 - if $TOOL list python34 >/dev/null 2>&1; then - python_pkgs="python34 - python34-devel - python34-tools + # Fedora 29 must use python3-virtualenv + if $TOOL list python3-virtualenv >/dev/null 2>&1; then + python_pkgs="python3 + python3-virtualenv + python3-devel " else error "No supported Python package available to install. Aborting bootstrap!" diff --git a/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3_legacy.sh b/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3_legacy.sh new file mode 100644 index 000000000..febfc7a83 --- /dev/null +++ b/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3_legacy.sh @@ -0,0 +1,78 @@ +# If new packages are installed by BootstrapRpmPython3 below, this version +# number must be increased. +BOOTSTRAP_RPM_PYTHON3_LEGACY_VERSION=1 + +# Checks if rh-python36 can be installed. +Python36SclIsAvailable() { + InitializeRPMCommonBase >/dev/null 2>&1; + + if "${TOOL}" list rh-python36 >/dev/null 2>&1; then + return 0 + fi + if "${TOOL}" list centos-release-scl >/dev/null 2>&1; then + return 0 + fi + return 1 +} + +# Try to enable rh-python36 from SCL if it is necessary and possible. +EnablePython36SCL() { + if "$EXISTS" python3.6 > /dev/null 2> /dev/null; then + return 0 + fi + if [ ! -f /opt/rh/rh-python36/enable ]; then + return 0 + fi + set +e + if ! . /opt/rh/rh-python36/enable; then + error 'Unable to enable rh-python36!' + exit 1 + fi + set -e +} + +# This bootstrap concerns old RedHat-based distributions that do not ship by default +# with Python 2.7, but only Python 2.6. We bootstrap them by enabling SCL and installing +# Python 3.6. Some of these distributions are: CentOS/RHEL/OL/SL 6. +BootstrapRpmPython3Legacy() { + # Tested with: + # - CentOS 6 + + InitializeRPMCommonBase + + if ! "${TOOL}" list rh-python36 >/dev/null 2>&1; then + echo "To use Certbot on this operating system, packages from the SCL repository need to be installed." + if ! "${TOOL}" list centos-release-scl >/dev/null 2>&1; then + error "Enable the SCL repository and try running Certbot again." + exit 1 + fi + if [ "${ASSUME_YES}" = 1 ]; then + /bin/echo -n "Enabling the SCL repository in 3 seconds... (Press Ctrl-C to cancel)" + sleep 1s + /bin/echo -ne "\e[0K\rEnabling the SCL repository in 2 seconds... (Press Ctrl-C to cancel)" + sleep 1s + /bin/echo -e "\e[0K\rEnabling the SCL repository in 1 second... (Press Ctrl-C to cancel)" + sleep 1s + fi + if ! "${TOOL}" install "${YES_FLAG}" "${QUIET_FLAG}" centos-release-scl; then + error "Could not enable SCL. Aborting bootstrap!" + exit 1 + fi + fi + + # CentOS 6 must use rh-python36 from SCL + if "${TOOL}" list rh-python36 >/dev/null 2>&1; then + python_pkgs="rh-python36-python + rh-python36-python-virtualenv + rh-python36-python-devel + " + else + error "No supported Python package available to install. Aborting bootstrap!" + exit 1 + fi + + BootstrapRpmCommonBase "${python_pkgs}" + + # Enable SCL rh-python36 after bootstrapping. + EnablePython36SCL +} diff --git a/letsencrypt-auto-source/pieces/bootstrappers/suse_common.sh b/letsencrypt-auto-source/pieces/bootstrappers/suse_common.sh index c531cbe99..7fa28ce50 100755 --- a/letsencrypt-auto-source/pieces/bootstrappers/suse_common.sh +++ b/letsencrypt-auto-source/pieces/bootstrappers/suse_common.sh @@ -14,10 +14,20 @@ BootstrapSuseCommon() { QUIET_FLAG='-qq' fi + if zypper search -x python-virtualenv >/dev/null 2>&1; then + OPENSUSE_VIRTUALENV_PACKAGES="python-virtualenv" + else + # Since Leap 15.0 (and associated Tumbleweed version), python-virtualenv + # is a source package, and python2-virtualenv must be used instead. + # Also currently python2-setuptools is not a dependency of python2-virtualenv, + # while it should be. Installing it explicitly until upstream fix. + OPENSUSE_VIRTUALENV_PACKAGES="python2-virtualenv python2-setuptools" + fi + zypper $QUIET_FLAG $zypper_flags in $install_flags \ python \ python-devel \ - python-virtualenv \ + $OPENSUSE_VIRTUALENV_PACKAGES \ gcc \ augeas-lenses \ libopenssl-devel \ diff --git a/letsencrypt-auto-source/pieces/certbot-requirements.txt b/letsencrypt-auto-source/pieces/certbot-requirements.txt index 80249cd9a..eb9027edb 100644 --- a/letsencrypt-auto-source/pieces/certbot-requirements.txt +++ b/letsencrypt-auto-source/pieces/certbot-requirements.txt @@ -1,12 +1,12 @@ -certbot==0.30.2 \ - --hash=sha256:e411b72fa86eec1018e6de28e649e8c9c71191a7431dcc77f207b57ca9484c11 \ - --hash=sha256:534487cb552ced8e47948ba3d2e7ca12c3a439133fc609485012b1a02fc7776e -acme==0.30.2 \ - --hash=sha256:68982576492dfa99c7e2be0fce4371adc9344740b05420ce0ab53238d2bb9b3b \ - --hash=sha256:295a5b7fce9f908e6e5cff8c40be1a3daf3e1ebabd2e139a4c87274e68eeb8f2 -certbot-apache==0.30.2 \ - --hash=sha256:3b7fa4e59772da7c9975ef2a49ceff157c9d7cb31eb9475928b5986d89701a3a \ - --hash=sha256:32fa915a8a51810fdfe828ac1361da4425c231d7384891e49e6338e4741464b2 -certbot-nginx==0.30.2 \ - --hash=sha256:7dc785f6f0c0c57b19cea8d98f9ea8feef53945613967b52c9348c81327010e2 \ - --hash=sha256:6ba4dd772d0c7cdfb3383ca325b35639e01ac9e142e4baa6445cd85c7fb59552 +certbot==1.2.0 \ + --hash=sha256:e25c17125c00b3398c8e9b9d54ef473c0e8f5aff53389f313a51b06cf472d335 \ + --hash=sha256:95dcbae085f8e4eb18442fe7b12994b08964a9a6e8e352e556cdb4a8a625373c +acme==1.2.0 \ + --hash=sha256:284d22fde75687a8ea72d737cac6bcbdc91f3c796221aa25378b8732ba6f6875 \ + --hash=sha256:0630c740d49bda945e97bd35fc8d6f02d082c8cb9e18f8fec0dbb3d395ac26ab +certbot-apache==1.2.0 \ + --hash=sha256:3f7493918353d3bd6067d446a2cf263e03831c4c10ec685b83d644b47767090d \ + --hash=sha256:b46e9def272103a68108e48bf7e410ea46801529b1ea6954f6506b14dd9df9b3 +certbot-nginx==1.2.0 \ + --hash=sha256:efd32a2b32f2439279da446b6bf67684f591f289323c5f494ebfd86a566a28fd \ + --hash=sha256:6fd7cf4f2545ad66e57000343227df9ccccaf04420e835e05cb3250fac1fa6db diff --git a/letsencrypt-auto-source/pieces/check_permissions.py b/letsencrypt-auto-source/pieces/check_permissions.py new file mode 100644 index 000000000..ba55e6d97 --- /dev/null +++ b/letsencrypt-auto-source/pieces/check_permissions.py @@ -0,0 +1,81 @@ +"""Verifies certbot-auto cannot be modified by unprivileged users. + +This script takes the path to certbot-auto as its only command line +argument. It then checks that the file can only be modified by uid/gid +< 1000 and if other users can modify the file, it prints a warning with +a suggestion on how to solve the problem. + +Permissions on symlinks in the absolute path of certbot-auto are ignored +and only the canonical path to certbot-auto is checked. There could be +permissions problems due to the symlinks that are unreported by this +script, however, issues like this were not caused by our documentation +and are ignored for the sake of simplicity. + +All warnings are printed to stdout rather than stderr so all stderr +output from this script can be suppressed to avoid printing messages if +this script fails for some reason. + +""" +from __future__ import print_function + +import os +import stat +import sys + + +FORUM_POST_URL = 'https://community.letsencrypt.org/t/certbot-auto-deployment-best-practices/91979/' + + +def has_safe_permissions(path): + """Returns True if the given path has secure permissions. + + The permissions are considered safe if the file is only writable by + uid/gid < 1000. + + The reason we allow more IDs than 0 is because on some systems such + as Debian, system users/groups other than uid/gid 0 are used for the + path we recommend in our instructions which is /usr/local/bin. 1000 + was chosen because on Debian 0-999 is reserved for system IDs[1] and + on RHEL either 0-499 or 0-999 is reserved depending on the + version[2][3]. Due to these differences across different OSes, this + detection isn't perfect so we only determine permissions are + insecure when we can be reasonably confident there is a problem + regardless of the underlying OS. + + [1] https://www.debian.org/doc/debian-policy/ch-opersys.html#uid-and-gid-classes + [2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/ch-managing_users_and_groups + [3] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-managing_users_and_groups + + :param str path: filesystem path to check + :returns: True if the path has secure permissions, otherwise, False + :rtype: bool + + """ + # os.stat follows symlinks before obtaining information about a file. + stat_result = os.stat(path) + if stat_result.st_mode & stat.S_IWOTH: + return False + if stat_result.st_mode & stat.S_IWGRP and stat_result.st_gid >= 1000: + return False + if stat_result.st_mode & stat.S_IWUSR and stat_result.st_uid >= 1000: + return False + return True + + +def main(certbot_auto_path): + current_path = os.path.realpath(certbot_auto_path) + last_path = None + permissions_ok = True + # This loop makes use of the fact that os.path.dirname('/') == '/'. + while current_path != last_path and permissions_ok: + permissions_ok = has_safe_permissions(current_path) + last_path = current_path + current_path = os.path.dirname(current_path) + + if not permissions_ok: + print('{0} has insecure permissions!'.format(certbot_auto_path)) + print('To learn how to fix them, visit {0}'.format(FORUM_POST_URL)) + + +if __name__ == '__main__': + main(sys.argv[1]) diff --git a/letsencrypt-auto-source/pieces/create_venv.py b/letsencrypt-auto-source/pieces/create_venv.py new file mode 100755 index 000000000..a618e228a --- /dev/null +++ b/letsencrypt-auto-source/pieces/create_venv.py @@ -0,0 +1,27 @@ +#!/usr/bin/env python +import os +import shutil +import subprocess +import sys + + +def create_venv(venv_path, pyver, verbose): + if os.path.exists(venv_path): + shutil.rmtree(venv_path) + + stdout = sys.stdout if verbose == '1' else open(os.devnull, 'w') + + if int(pyver) <= 27: + # Use virtualenv binary + environ = os.environ.copy() + environ['VIRTUALENV_NO_DOWNLOAD'] = '1' + command = ['virtualenv', '--no-site-packages', '--python', sys.executable, venv_path] + subprocess.check_call(command, stdout=stdout, env=environ) + else: + # Use embedded venv module in Python 3 + command = [sys.executable, '-m', 'venv', venv_path] + subprocess.check_call(command, stdout=stdout) + + +if __name__ == '__main__': + create_venv(*sys.argv[1:]) diff --git a/letsencrypt-auto-source/pieces/dependency-requirements.txt b/letsencrypt-auto-source/pieces/dependency-requirements.txt index 1fac78836..eec5a9946 100644 --- a/letsencrypt-auto-source/pieces/dependency-requirements.txt +++ b/letsencrypt-auto-source/pieces/dependency-requirements.txt @@ -1,196 +1,265 @@ -# This is the flattened list of packages certbot-auto installs. To generate -# this, do -# `pip install --no-cache-dir -e acme -e . -e certbot-apache -e certbot-nginx`, -# and then use `hashin` or a more secure method to gather the hashes. - -# Hashin example: +# This is the flattened list of packages certbot-auto installs. +# To generate this, do (with docker and package hashin installed): +# ``` +# letsencrypt-auto-source/rebuild_dependencies.py \ +# letsencrypt-auto-source/pieces/dependency-requirements.txt +# ``` +# If you want to update a single dependency, run commands similar to these: +# ``` # pip install hashin # hashin -r dependency-requirements.txt cryptography==1.5.2 -# sets the new certbot-auto pinned version of cryptography to 1.5.2 - -argparse==1.4.0 \ - --hash=sha256:c31647edb69fd3d465a847ea3157d37bed1f95f19760b11a47aa91c04b666314 \ - --hash=sha256:62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4 - -# This comes before cffi because cffi will otherwise install an unchecked -# version via setup_requires. -pycparser==2.14 \ - --hash=sha256:7959b4a74abdc27b312fed1c21e6caf9309ce0b29ea86b591fd2e99ecdf27f73 \ - --no-binary pycparser - -asn1crypto==0.22.0 \ - --hash=sha256:d232509fefcfcdb9a331f37e9c9dc20441019ad927c7d2176cf18ed5da0ba097 \ - --hash=sha256:cbbadd640d3165ab24b06ef25d1dca09a3441611ac15f6a6b452474fdf0aed1a -cffi==1.11.5 \ - --hash=sha256:1b0493c091a1898f1136e3f4f991a784437fac3673780ff9de3bcf46c80b6b50 \ - --hash=sha256:87f37fe5130574ff76c17cab61e7d2538a16f843bb7bca8ebbc4b12de3078596 \ - --hash=sha256:1553d1e99f035ace1c0544050622b7bc963374a00c467edafac50ad7bd276aef \ - --hash=sha256:151b7eefd035c56b2b2e1eb9963c90c6302dc15fbd8c1c0a83a163ff2c7d7743 \ - --hash=sha256:edabd457cd23a02965166026fd9bfd196f4324fe6032e866d0f3bd0301cd486f \ - --hash=sha256:ba5e697569f84b13640c9e193170e89c13c6244c24400fc57e88724ef610cd31 \ - --hash=sha256:79f9b6f7c46ae1f8ded75f68cf8ad50e5729ed4d590c74840471fc2823457d04 \ - --hash=sha256:b0f7d4a3df8f06cf49f9f121bead236e328074de6449866515cea4907bbc63d6 \ - --hash=sha256:4c91af6e967c2015729d3e69c2e51d92f9898c330d6a851bf8f121236f3defd3 \ - --hash=sha256:7a33145e04d44ce95bcd71e522b478d282ad0eafaf34fe1ec5bbd73e662f22b6 \ - --hash=sha256:95d5251e4b5ca00061f9d9f3d6fe537247e145a8524ae9fd30a2f8fbce993b5b \ - --hash=sha256:b75110fb114fa366b29a027d0c9be3709579602ae111ff61674d28c93606acca \ - --hash=sha256:ae5e35a2c189d397b91034642cb0eab0e346f776ec2eb44a49a459e6615d6e2e \ - --hash=sha256:fdf1c1dc5bafc32bc5d08b054f94d659422b05aba244d6be4ddc1c72d9aa70fb \ - --hash=sha256:9d1d3e63a4afdc29bd76ce6aa9d58c771cd1599fbba8cf5057e7860b203710dd \ - --hash=sha256:be2a9b390f77fd7676d80bc3cdc4f8edb940d8c198ed2d8c0be1319018c778e1 \ - --hash=sha256:ed01918d545a38998bfa5902c7c00e0fee90e957ce036a4000a88e3fe2264917 \ - --hash=sha256:857959354ae3a6fa3da6651b966d13b0a8bed6bbc87a0de7b38a549db1d2a359 \ - --hash=sha256:2ba8a45822b7aee805ab49abfe7eec16b90587f7f26df20c71dd89e45a97076f \ - --hash=sha256:a36c5c154f9d42ec176e6e620cb0dd275744aa1d804786a71ac37dc3661a5e95 \ - --hash=sha256:e55e22ac0a30023426564b1059b035973ec82186ddddbac867078435801c7801 \ - --hash=sha256:3eb6434197633b7748cea30bf0ba9f66727cdce45117a712b29a443943733257 \ - --hash=sha256:ecbb7b01409e9b782df5ded849c178a0aa7c906cf8c5a67368047daab282b184 \ - --hash=sha256:770f3782b31f50b68627e22f91cb182c48c47c02eb405fd689472aa7b7aa16dc \ - --hash=sha256:d5d8555d9bfc3f02385c1c37e9f998e2011f0db4f90e250e5bc0c0a85a813085 \ - --hash=sha256:3c85641778460581c42924384f5e68076d724ceac0f267d66c757f7535069c93 \ - --hash=sha256:ca1bd81f40adc59011f58159e4aa6445fc585a32bb8ac9badf7a2c1aa23822f2 \ - --hash=sha256:3bb6bd7266598f318063e584378b8e27c67de998a43362e8fce664c54ee52d30 \ - --hash=sha256:a6a5cb8809091ec9ac03edde9304b3ad82ad4466333432b16d78ef40e0cce0d5 \ - --hash=sha256:57b2533356cb2d8fac1555815929f7f5f14d68ac77b085d2326b571310f34f6e \ - --hash=sha256:495c5c2d43bf6cebe0178eb3e88f9c4aa48d8934aa6e3cddb865c058da76756b \ - --hash=sha256:e90f17980e6ab0f3c2f3730e56d1fe9bcba1891eeea58966e89d352492cc74f4 -ConfigArgParse==0.12.0 \ - --hash=sha256:28cd7d67669651f2a4518367838c49539457504584a139709b2b8f6c208ef339 \ - --no-binary ConfigArgParse +# ``` +ConfigArgParse==1.0 \ + --hash=sha256:bf378245bc9cdc403a527e5b7406b991680c2a530e7e81af747880b54eb57133 +certifi==2019.11.28 \ + --hash=sha256:017c25db2a153ce562900032d5bc68e9f191e44e9a0f762f373977de9df1fbb3 \ + --hash=sha256:25b64c7da4cd7479594d035c08c2d809eb4aab3a26e5a990ea98cc450c320f1f +cffi==1.13.2 \ + --hash=sha256:0b49274afc941c626b605fb59b59c3485c17dc776dc3cc7cc14aca74cc19cc42 \ + --hash=sha256:0e3ea92942cb1168e38c05c1d56b0527ce31f1a370f6117f1d490b8dcd6b3a04 \ + --hash=sha256:135f69aecbf4517d5b3d6429207b2dff49c876be724ac0c8bf8e1ea99df3d7e5 \ + --hash=sha256:19db0cdd6e516f13329cba4903368bff9bb5a9331d3410b1b448daaadc495e54 \ + --hash=sha256:2781e9ad0e9d47173c0093321bb5435a9dfae0ed6a762aabafa13108f5f7b2ba \ + --hash=sha256:291f7c42e21d72144bb1c1b2e825ec60f46d0a7468f5346841860454c7aa8f57 \ + --hash=sha256:2c5e309ec482556397cb21ede0350c5e82f0eb2621de04b2633588d118da4396 \ + --hash=sha256:2e9c80a8c3344a92cb04661115898a9129c074f7ab82011ef4b612f645939f12 \ + --hash=sha256:32a262e2b90ffcfdd97c7a5e24a6012a43c61f1f5a57789ad80af1d26c6acd97 \ + --hash=sha256:3c9fff570f13480b201e9ab69453108f6d98244a7f495e91b6c654a47486ba43 \ + --hash=sha256:415bdc7ca8c1c634a6d7163d43fb0ea885a07e9618a64bda407e04b04333b7db \ + --hash=sha256:42194f54c11abc8583417a7cf4eaff544ce0de8187abaf5d29029c91b1725ad3 \ + --hash=sha256:4424e42199e86b21fc4db83bd76909a6fc2a2aefb352cb5414833c030f6ed71b \ + --hash=sha256:4a43c91840bda5f55249413037b7a9b79c90b1184ed504883b72c4df70778579 \ + --hash=sha256:599a1e8ff057ac530c9ad1778293c665cb81a791421f46922d80a86473c13346 \ + --hash=sha256:5c4fae4e9cdd18c82ba3a134be256e98dc0596af1e7285a3d2602c97dcfa5159 \ + --hash=sha256:5ecfa867dea6fabe2a58f03ac9186ea64da1386af2159196da51c4904e11d652 \ + --hash=sha256:62f2578358d3a92e4ab2d830cd1c2049c9c0d0e6d3c58322993cc341bdeac22e \ + --hash=sha256:6471a82d5abea994e38d2c2abc77164b4f7fbaaf80261cb98394d5793f11b12a \ + --hash=sha256:6d4f18483d040e18546108eb13b1dfa1000a089bcf8529e30346116ea6240506 \ + --hash=sha256:71a608532ab3bd26223c8d841dde43f3516aa5d2bf37b50ac410bb5e99053e8f \ + --hash=sha256:74a1d8c85fb6ff0b30fbfa8ad0ac23cd601a138f7509dc617ebc65ef305bb98d \ + --hash=sha256:7b93a885bb13073afb0aa73ad82059a4c41f4b7d8eb8368980448b52d4c7dc2c \ + --hash=sha256:7d4751da932caaec419d514eaa4215eaf14b612cff66398dd51129ac22680b20 \ + --hash=sha256:7f627141a26b551bdebbc4855c1157feeef18241b4b8366ed22a5c7d672ef858 \ + --hash=sha256:8169cf44dd8f9071b2b9248c35fc35e8677451c52f795daa2bb4643f32a540bc \ + --hash=sha256:aa00d66c0fab27373ae44ae26a66a9e43ff2a678bf63a9c7c1a9a4d61172827a \ + --hash=sha256:ccb032fda0873254380aa2bfad2582aedc2959186cce61e3a17abc1a55ff89c3 \ + --hash=sha256:d754f39e0d1603b5b24a7f8484b22d2904fa551fe865fd0d4c3332f078d20d4e \ + --hash=sha256:d75c461e20e29afc0aee7172a0950157c704ff0dd51613506bd7d82b718e7410 \ + --hash=sha256:dcd65317dd15bc0451f3e01c80da2216a31916bdcffd6221ca1202d96584aa25 \ + --hash=sha256:e570d3ab32e2c2861c4ebe6ffcad6a8abf9347432a37608fe1fbd157b3f0036b \ + --hash=sha256:fd43a88e045cf992ed09fa724b5315b790525f2676883a6ea64e3263bae6549d +chardet==3.0.4 \ + --hash=sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae \ + --hash=sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 configobj==5.0.6 \ - --hash=sha256:a2f5650770e1c87fb335af19a9b7eb73fc05ccf22144eb68db7d00cd2bcb0902 \ - --no-binary configobj -cryptography==2.2.2 \ - --hash=sha256:3f3b65d5a16e6b52fba63dc860b62ca9832f51f1a2ae5083c78b6840275f12dd \ - --hash=sha256:5251e7de0de66810833606439ca65c9b9e45da62196b0c88bfadf27740aac09f \ - --hash=sha256:551a3abfe0c8c6833df4192a63371aa2ff43afd8f570ed345d31f251d78e7e04 \ - --hash=sha256:5cb990056b7cadcca26813311187ad751ea644712022a3976443691168781b6f \ - --hash=sha256:60bda7f12ecb828358be53095fc9c6edda7de8f1ef571f96c00b2363643fa3cd \ - --hash=sha256:64b5c67acc9a7c83fbb4b69166f3105a0ab722d27934fac2cb26456718eec2ba \ - --hash=sha256:6fef51ec447fe9f8351894024e94736862900d3a9aa2961528e602eb65c92bdb \ - --hash=sha256:77d0ad229d47a6e0272d00f6bf8ac06ce14715a9fd02c9a97f5a2869aab3ccb2 \ - --hash=sha256:808fe471b1a6b777f026f7dc7bd9a4959da4bfab64972f2bbe91e22527c1c037 \ - --hash=sha256:9b62fb4d18529c84b961efd9187fecbb48e89aa1a0f9f4161c61b7fc42a101bd \ - --hash=sha256:9e5bed45ec6b4f828866ac6a6bedf08388ffcfa68abe9e94b34bb40977aba531 \ - --hash=sha256:9fc295bf69130a342e7a19a39d7bbeb15c0bcaabc7382ec33ef3b2b7d18d2f63 \ - --hash=sha256:abd070b5849ed64e6d349199bef955ee0ad99aefbad792f0c587f8effa681a5e \ - --hash=sha256:ba6a774749b6e510cffc2fb98535f717e0e5fd91c7c99a61d223293df79ab351 \ - --hash=sha256:c332118647f084c983c6a3e1dba0f3bcb051f69d12baccac68db8d62d177eb8a \ - --hash=sha256:d6f46e862ee36df81e6342c2177ba84e70f722d9dc9c6c394f9f1f434c4a5563 \ - --hash=sha256:db6013746f73bf8edd9c3d1d3f94db635b9422f503db3fc5ef105233d4c011ab \ - --hash=sha256:f57008eaff597c69cf692c3518f6d4800f0309253bb138b526a37fe9ef0c7471 \ - --hash=sha256:f6c821ac253c19f2ad4c8691633ae1d1a17f120d5b01ea1d256d7b602bc59887 -enum34==1.1.2 ; python_version < '3.4' \ - --hash=sha256:2475d7fcddf5951e92ff546972758802de5260bf409319a9f1934e6bbc8b1dc7 \ - --hash=sha256:35907defb0f992b75ab7788f65fedc1cf20ffa22688e0e6f6f12afc06b3ea501 + --hash=sha256:a2f5650770e1c87fb335af19a9b7eb73fc05ccf22144eb68db7d00cd2bcb0902 +cryptography==2.8 \ + --hash=sha256:02079a6addc7b5140ba0825f542c0869ff4df9a69c360e339ecead5baefa843c \ + --hash=sha256:1df22371fbf2004c6f64e927668734070a8953362cd8370ddd336774d6743595 \ + --hash=sha256:369d2346db5934345787451504853ad9d342d7f721ae82d098083e1f49a582ad \ + --hash=sha256:3cda1f0ed8747339bbdf71b9f38ca74c7b592f24f65cdb3ab3765e4b02871651 \ + --hash=sha256:44ff04138935882fef7c686878e1c8fd80a723161ad6a98da31e14b7553170c2 \ + --hash=sha256:4b1030728872c59687badcca1e225a9103440e467c17d6d1730ab3d2d64bfeff \ + --hash=sha256:58363dbd966afb4f89b3b11dfb8ff200058fbc3b947507675c19ceb46104b48d \ + --hash=sha256:6ec280fb24d27e3d97aa731e16207d58bd8ae94ef6eab97249a2afe4ba643d42 \ + --hash=sha256:7270a6c29199adc1297776937a05b59720e8a782531f1f122f2eb8467f9aab4d \ + --hash=sha256:73fd30c57fa2d0a1d7a49c561c40c2f79c7d6c374cc7750e9ac7c99176f6428e \ + --hash=sha256:7f09806ed4fbea8f51585231ba742b58cbcfbfe823ea197d8c89a5e433c7e912 \ + --hash=sha256:90df0cc93e1f8d2fba8365fb59a858f51a11a394d64dbf3ef844f783844cc793 \ + --hash=sha256:971221ed40f058f5662a604bd1ae6e4521d84e6cad0b7b170564cc34169c8f13 \ + --hash=sha256:a518c153a2b5ed6b8cc03f7ae79d5ffad7315ad4569b2d5333a13c38d64bd8d7 \ + --hash=sha256:b0de590a8b0979649ebeef8bb9f54394d3a41f66c5584fff4220901739b6b2f0 \ + --hash=sha256:b43f53f29816ba1db8525f006fa6f49292e9b029554b3eb56a189a70f2a40879 \ + --hash=sha256:d31402aad60ed889c7e57934a03477b572a03af7794fa8fb1780f21ea8f6551f \ + --hash=sha256:de96157ec73458a7f14e3d26f17f8128c959084931e8997b9e655a39c8fde9f9 \ + --hash=sha256:df6b4dca2e11865e6cfbfb708e800efb18370f5a46fd601d3755bc7f85b3a8a2 \ + --hash=sha256:ecadccc7ba52193963c0475ac9f6fa28ac01e01349a2ca48509667ef41ffd2cf \ + --hash=sha256:fb81c17e0ebe3358486cd8cc3ad78adbae58af12fc2bf2bc0bb84e8090fa5ce8 +distro==1.4.0 \ + --hash=sha256:362dde65d846d23baee4b5c058c8586f219b5a54be1cf5fc6ff55c4578392f57 \ + --hash=sha256:eedf82a470ebe7d010f1872c17237c79ab04097948800029994fa458e52fb4b4 +enum34==1.1.6 \ + --hash=sha256:2d81cbbe0e73112bdfe6ef8576f2238f2ba27dd0d55752a776c41d38b7da2850 \ + --hash=sha256:644837f692e5f550741432dd3f223bbb9852018674981b1664e5dc339387588a \ + --hash=sha256:6bd0f6ad48ec2aa117d3d141940d484deccda84d4fcd884f5c3d93c23ecd8c79 \ + --hash=sha256:8ad8c4783bf61ded74527bffb48ed9b54166685e4230386a9ed9b1279e2df5b1 funcsigs==1.0.2 \ --hash=sha256:330cc27ccbf7f1e992e69fef78261dc7c6569012cf397db8d3de0234e6c937ca \ --hash=sha256:a7bb0f2cf3a3fd1ab2732cb49eba4252c2af4240442415b4abce3b87022a8f50 -idna==2.5 \ - --hash=sha256:cc19709fd6d0cbfed39ea875d29ba6d4e22c0cebc510a76d6302a28385e8bb70 \ - --hash=sha256:3cb5ce08046c4e3a560fc02f138d0ac63e00f8ce5901a56b32ec8b7994082aab -ipaddress==1.0.16 \ - --hash=sha256:935712800ce4760701d89ad677666cd52691fd2f6f0b340c8b4239a3c17988a5 \ - --hash=sha256:5a3182b322a706525c46282ca6f064d27a02cffbd449f9f47416f1dc96aa71b0 -josepy==1.1.0 \ - --hash=sha256:1309a25aac3caeff5239729c58ff9b583f7d022ffdb1553406ddfc8e5b52b76e \ - --hash=sha256:fb5c62c77d26e04df29cb5ecd01b9ce69b6fcc9e521eb1ca193b7faa2afa7086 -linecache2==1.0.0 \ - --hash=sha256:e78be9c0a0dfcbac712fe04fbf92b96cddae80b1b842f24248214c8496f006ef \ - --hash=sha256:4b26ff4e7110db76eeb6f5a7b64a82623839d595c2038eeda662f2a2db78e97c -# Using an older version of mock here prevents regressions of #5276. +idna==2.8 \ + --hash=sha256:c357b3f628cf53ae2c4c05627ecc484553142ca23264e593d327bcde5e9c3407 \ + --hash=sha256:ea8b7f6188e6fa117537c3df7da9fc686d485087abf6ac197f9c46432f7e4a3c +ipaddress==1.0.23 \ + --hash=sha256:6e0f4a39e66cb5bb9a137b00276a2eff74f93b71dcbdad6f10ff7df9d3557fcc \ + --hash=sha256:b7f8e0369580bb4a24d5ba1d7cc29660a4a6987763faf1d8a8046830e020e7e2 +josepy==1.2.0 \ + --hash=sha256:8ea15573203f28653c00f4ac0142520777b1c59d9eddd8da3f256c6ba3cac916 \ + --hash=sha256:9cec9a839fe9520f0420e4f38e7219525daccce4813296627436fe444cd002d3 mock==1.3.0 \ - --hash=sha256:3f573a18be94de886d1191f27c168427ef693e8dcfcecf95b170577b2eb69cbb \ - --hash=sha256:1e247dbecc6ce057299eb7ee019ad68314bb93152e81d9a6110d35f4d5eca0f6 -ordereddict==1.1 \ - --hash=sha256:1c35b4ac206cef2d24816c89f89cf289dd3d38cf7c449bb3fab7bf6d43f01b1f \ - --no-binary ordereddict -packaging==16.8 \ - --hash=sha256:99276dc6e3a7851f32027a68f1095cd3f77c148091b092ea867a351811cfe388 \ - --hash=sha256:5d50835fdf0a7edf0b55e311b7c887786504efea1177abd7e69329a8e5ea619e -parsedatetime==2.1 \ - --hash=sha256:ce9d422165cf6e963905cd5f74f274ebf7cc98c941916169178ef93f0e557838 \ - --hash=sha256:17c578775520c99131634e09cfca5a05ea9e1bd2a05cd06967ebece10df7af2d -pbr==1.8.1 \ - --hash=sha256:46c8db75ae75a056bd1cc07fa21734fe2e603d11a07833ecc1eeb74c35c72e0c \ - --hash=sha256:e2127626a91e6c885db89668976db31020f0af2da728924b56480fc7ccf09649 -pyOpenSSL==18.0.0 \ - --hash=sha256:26ff56a6b5ecaf3a2a59f132681e2a80afcc76b4f902f612f518f92c2a1bf854 \ - --hash=sha256:6488f1423b00f73b7ad5167885312bb0ce410d3312eb212393795b53c8caa580 -pyparsing==2.1.8 \ - --hash=sha256:2f0f5ceb14eccd5aef809d6382e87df22ca1da583c79f6db01675ce7d7f49c18 \ - --hash=sha256:03a4869b9f3493807ee1f1cb405e6d576a1a2ca4d81a982677c0c1ad6177c56b \ - --hash=sha256:ab09aee814c0241ff0c503cff30018219fe1fc14501d89f406f4664a0ec9fbcd \ - --hash=sha256:6e9a7f052f8e26bcf749e4033e3115b6dc7e3c85aafcb794b9a88c9d9ef13c97 \ - --hash=sha256:9f463a6bcc4eeb6c08f1ed84439b17818e2085937c0dee0d7674ac127c67c12b \ - --hash=sha256:3626b4d81cfb300dad57f52f2f791caaf7b06c09b368c0aa7b868e53a5775424 \ - --hash=sha256:367b90cc877b46af56d4580cd0ae278062903f02b8204ab631f5a2c0f50adfd0 \ - --hash=sha256:9f1ea360086cd68681e7f4ca8f1f38df47bf81942a0d76a9673c2d23eff35b13 -pyRFC3339==1.0 \ - --hash=sha256:eea31835c56e2096af4363a5745a784878a61d043e247d3a6d6a0a32a9741f56 \ - --hash=sha256:8dfbc6c458b8daba1c0f3620a8c78008b323a268b27b7359e92a4ae41325f535 + --hash=sha256:1e247dbecc6ce057299eb7ee019ad68314bb93152e81d9a6110d35f4d5eca0f6 \ + --hash=sha256:3f573a18be94de886d1191f27c168427ef693e8dcfcecf95b170577b2eb69cbb +parsedatetime==2.5 \ + --hash=sha256:3b835fc54e472c17ef447be37458b400e3fefdf14bb1ffdedb5d2c853acf4ba1 \ + --hash=sha256:d2e9ddb1e463de871d32088a3f3cea3dc8282b1b2800e081bd0ef86900451667 +pbr==5.4.4 \ + --hash=sha256:139d2625547dbfa5fb0b81daebb39601c478c21956dc57e2e07b74450a8c506b \ + --hash=sha256:61aa52a0f18b71c5cc58232d2cf8f8d09cd67fcad60b742a60124cb8d6951488 +pyOpenSSL==19.1.0 \ + --hash=sha256:621880965a720b8ece2f1b2f54ea2071966ab00e2970ad2ce11d596102063504 \ + --hash=sha256:9a24494b2602aaf402be5c9e30a0b82d4a5c67528fe8fb475e3f3bc00dd69507 +pyRFC3339==1.1 \ + --hash=sha256:67196cb83b470709c580bb4738b83165e67c6cc60e1f2e4f286cfcb402a926f4 \ + --hash=sha256:81b8cbe1519cdb79bed04910dd6fa4e181faf8c88dff1e1b987b5f7ab23a5b1a +pycparser==2.19 \ + --hash=sha256:a988718abfad80b6b157acce7bf130a30876d27603738ac39f140993246b25b3 +pyparsing==2.4.6 \ + --hash=sha256:4c830582a84fb022400b85429791bc551f1f4871c33f23e44f353119e92f969f \ + --hash=sha256:c342dccb5250c08d45fd6f8b4a559613ca603b57498511740e65cd11a2e7dcec python-augeas==0.5.0 \ - --hash=sha256:67d59d66cdba8d624e0389b87b2a83a176f21f16a87553b50f5703b23f29bac2 \ - --no-binary python-augeas -pytz==2015.7 \ - --hash=sha256:3abe6a6d3fc2fbbe4c60144211f45da2edbe3182a6f6511af6bbba0598b1f992 \ - --hash=sha256:939ef9c1e1224d980405689a97ffcf7828c56d1517b31d73464356c1f2b7769e \ - --hash=sha256:ead4aefa7007249e05e51b01095719d5a8dd95760089f5730aac5698b1932918 \ - --hash=sha256:3cca0df08bd0ed98432390494ce3ded003f5e661aa460be7a734bffe35983605 \ - --hash=sha256:3ede470d3d17ba3c07638dfa0d10452bc1b6e5ad326127a65ba77e6aaeb11bec \ - --hash=sha256:68c47964f7186eec306b13629627722b9079cd4447ed9e5ecaecd4eac84ca734 \ - --hash=sha256:dd5d3991950aae40a6c81de1578942e73d629808cefc51d12cd157980e6cfc18 \ - --hash=sha256:a77c52062c07eb7c7b30545dbc73e32995b7e117eea750317b5cb5c7a4618f14 \ - --hash=sha256:81af9aec4bc960a9a0127c488f18772dae4634689233f06f65443e7b11ebeb51 \ - --hash=sha256:e079b1dadc5c06246cc1bb6fe1b23a50b1d1173f2edd5104efd40bb73a28f406 \ - --hash=sha256:fbd26746772c24cb93c8b97cbdad5cb9e46c86bbdb1b9d8a743ee00e2fb1fc5d \ - --hash=sha256:99266ef30a37e43932deec2b7ca73e83c8dbc3b9ff703ec73eca6b1dae6befea \ - --hash=sha256:8b6ce1c993909783bc96e0b4f34ea223bff7a4df2c90bdb9c4e0f1ac928689e3 -requests==2.20.0 \ - --hash=sha256:99dcfdaaeb17caf6e526f32b6a7b780461512ab3f1d992187801694cba42770c \ - --hash=sha256:a84b8c9ab6239b578f22d1c21d51b696dcfe004032bb80ea832398d6909d7279 -six==1.10.0 \ - --hash=sha256:0ff78c403d9bccf5a425a6d31a12aa6b47f1c21ca4dc2573a7e2f32a97335eb1 \ - --hash=sha256:105f8d68616f8248e24bf0e9372ef04d3cc10104f1980f54d57b2ce73a5ad56a -traceback2==1.4.0 \ - --hash=sha256:8253cebec4b19094d67cc5ed5af99bf1dba1285292226e98a31929f87a5d6b23 \ - --hash=sha256:05acc67a09980c2ecfedd3423f7ae0104839eccb55fc645773e1caa0951c3030 -unittest2==1.1.0 \ - --hash=sha256:13f77d0875db6d9b435e1d4f41e74ad4cc2eb6e1d5c824996092b3430f088bb8 \ - --hash=sha256:22882a0e418c284e1f718a822b3b022944d53d2d908e1690b319a9d3eb2c0579 -zope.component==4.2.2 \ - --hash=sha256:282c112b55dd8e3c869a3571f86767c150ab1284a9ace2bdec226c592acaf81a \ - --no-binary zope.component -zope.event==4.1.0 \ - --hash=sha256:dc7a59a2fd91730d3793131a5d261b29e93ec4e2a97f1bc487ce8defee2fe786 \ - --no-binary zope.event -zope.interface==4.1.3 \ - --hash=sha256:f07b631f7a601cd8cbd3332d54f43142c7088a83299f859356f08d1d4d4259b3 \ - --hash=sha256:de5cca083b9439d8002fb76bbe6b4998c5a5a721fab25b84298967f002df4c94 \ - --hash=sha256:6788416f7ea7f5b8a97be94825377aa25e8bdc73463e07baaf9858b29e737077 \ - --hash=sha256:6f3230f7254518201e5a3708cbb2de98c848304f06e3ded8bfb39e5825cba2e1 \ - --hash=sha256:5fa575a5240f04200c3088427d0d4b7b737f6e9018818a51d8d0f927a6a2517a \ - --hash=sha256:522194ad6a545735edd75c8a83f48d65d1af064e432a7d320d64f56bafc12e99 \ - --hash=sha256:e8c7b2d40943f71c99148c97f66caa7f5134147f57423f8db5b4825099ce9a09 \ - --hash=sha256:279024f0208601c3caa907c53876e37ad88625f7eaf1cb3842dbe360b2287017 \ - --hash=sha256:2e221a9eec7ccc58889a278ea13dcfed5ef939d80b07819a9a8b3cb1c681484f \ - --hash=sha256:69118965410ec86d44dc6b9017ee3ddbd582e0c0abeef62b3a19dbf6c8ad132b \ - --hash=sha256:d04df8686ec864d0cade8cf199f7f83aecd416109a20834d568f8310ded12dea \ - --hash=sha256:e75a947e15ee97e7e71e02ea302feb2fc62d3a2bb4668bf9dfbed43a506ac7e7 \ - --hash=sha256:4e45d22fb883222a5ab9f282a116fec5ee2e8d1a568ccff6a2d75bbd0eb6bcfc \ - --hash=sha256:bce9339bb3c7a55e0803b63d21c5839e8e479bc85c4adf42ae415b72f94facb2 \ - --hash=sha256:928138365245a0e8869a5999fbcc2a45475a0a6ed52a494d60dbdc540335fedd \ - --hash=sha256:0d841ba1bb840eea0e6489dc5ecafa6125554971f53b5acb87764441e61bceba \ - --hash=sha256:b09c8c1d47b3531c400e0195697f1414a63221de6ef478598a4f1460f7d9a392 -requests-toolbelt==0.8.0 \ - --hash=sha256:42c9c170abc2cacb78b8ab23ac957945c7716249206f90874651971a4acff237 \ - --hash=sha256:f6a531936c6fa4c6cfce1b9c10d5c4f498d16528d2a54a22ca00011205a187b5 -chardet==3.0.2 \ - --hash=sha256:4f7832e7c583348a9eddd927ee8514b3bf717c061f57b21dbe7697211454d9bb \ - --hash=sha256:6ebf56457934fdce01fb5ada5582762a84eed94cad43ed877964aebbdd8174c0 -urllib3==1.24.1 \ - --hash=sha256:61bf29cada3fc2fbefad4fdf059ea4bd1b4a86d2b6d15e1c7c0b582b9752fe39 \ - --hash=sha256:de9529817c93f27c8ccbfead6985011db27bd0ddfcdb2d86f3f663385c6a9c22 -certifi==2017.4.17 \ - --hash=sha256:f4318671072f030a33c7ca6acaef720ddd50ff124d1388e50c1bda4cbd6d7010 \ - --hash=sha256:f7527ebf7461582ce95f7a9e03dd141ce810d40590834f4ec20cddd54234c10a + --hash=sha256:67d59d66cdba8d624e0389b87b2a83a176f21f16a87553b50f5703b23f29bac2 +pytz==2019.3 \ + --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \ + --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be +requests==2.22.0 \ + --hash=sha256:11e007a8a2aa0323f5a921e9e6a2d7e4e67d9877e85773fba9ba6419025cbeb4 \ + --hash=sha256:9cf5292fcd0f598c671cfc1e0d7d1a7f13bb8085e9a590f48c010551dc6c4b31 +requests-toolbelt==0.9.1 \ + --hash=sha256:380606e1d10dc85c3bd47bf5a6095f815ec007be7a8b69c878507068df059e6f \ + --hash=sha256:968089d4584ad4ad7c171454f0a5c6dac23971e9472521ea3b6d49d610aa6fc0 +six==1.14.0 \ + --hash=sha256:236bdbdce46e6e6a3d61a337c0f8b763ca1e8717c03b369e87a7ec7ce1319c0a \ + --hash=sha256:8f3cd2e254d8f793e7f3d6d9df77b92252b52637291d0f0da013c76ea2724b6c +urllib3==1.25.8 \ + --hash=sha256:2f3db8b19923a873b3e5256dc9c2dedfa883e33d87c690d9c7913e1f40673cdc \ + --hash=sha256:87716c2d2a7121198ebcb7ce7cccf6ce5e9ba539041cfbaeecfb641dc0bf6acc +zope.component==4.6 \ + --hash=sha256:ec2afc5bbe611dcace98bb39822c122d44743d635dafc7315b9aef25097db9e6 +zope.deferredimport==4.3.1 \ + --hash=sha256:57b2345e7b5eef47efcd4f634ff16c93e4265de3dcf325afc7315ade48d909e1 \ + --hash=sha256:9a0c211df44aa95f1c4e6d2626f90b400f56989180d3ef96032d708da3d23e0a +zope.deprecation==4.4.0 \ + --hash=sha256:0d453338f04bacf91bbfba545d8bcdf529aa829e67b705eac8c1a7fdce66e2df \ + --hash=sha256:f1480b74995958b24ce37b0ef04d3663d2683e5d6debc96726eff18acf4ea113 +zope.event==4.4 \ + --hash=sha256:69c27debad9bdacd9ce9b735dad382142281ac770c4a432b533d6d65c4614bcf \ + --hash=sha256:d8e97d165fd5a0997b45f5303ae11ea3338becfe68c401dd88ffd2113fe5cae7 +zope.hookable==5.0.0 \ + --hash=sha256:0992a0dd692003c09fb958e1480cebd1a28f2ef32faa4857d864f3ca8e9d6952 \ + --hash=sha256:0f325838dbac827a1e2ed5d482c1f2656b6844dc96aa098f7727e76395fcd694 \ + --hash=sha256:22a317ba00f61bac99eac1a5e330be7cb8c316275a21269ec58aa396b602af0c \ + --hash=sha256:25531cb5e7b35e8a6d1d6eddef624b9a22ce5dcf8f4448ef0f165acfa8c3fc21 \ + --hash=sha256:30890892652766fc80d11f078aca9a5b8150bef6b88aba23799581a53515c404 \ + --hash=sha256:342d682d93937e5b8c232baffb32a87d5eee605d44f74566657c64a239b7f342 \ + --hash=sha256:46b2fddf1f5aeb526e02b91f7e62afbb9fff4ffd7aafc97cdb00a0d717641567 \ + --hash=sha256:523318ff96df9b8d378d997c00c5d4cbfbff68dc48ff5ee5addabdb697d27528 \ + --hash=sha256:53aa02eb8921d4e667c69d76adeed8fe426e43870c101cb08dcd2f3468aff742 \ + --hash=sha256:62e79e8fdde087cb20822d7874758f5acbedbffaf3c0fbe06309eb8a41ee4e06 \ + --hash=sha256:74bf2f757f7385b56dc3548adae508d8b3ef952d600b4b12b88f7d1706b05dcc \ + --hash=sha256:751ee9d89eb96e00c1d7048da9725ce392a708ed43406416dc5ed61e4d199764 \ + --hash=sha256:7b83bc341e682771fe810b360cd5d9c886a948976aea4b979ff214e10b8b523b \ + --hash=sha256:81eeeb27dbb0ddaed8070daee529f0d1bfe4f74c7351cce2aaca3ea287c4cc32 \ + --hash=sha256:856509191e16930335af4d773c0fc31a17bae8991eb6f167a09d5eddf25b56cc \ + --hash=sha256:8853e81fd07b18fa9193b19e070dc0557848d9945b1d2dac3b7782543458c87d \ + --hash=sha256:94506a732da2832029aecdfe6ea07eb1b70ee06d802fff34e1b3618fe7cdf026 \ + --hash=sha256:95ad874a8cc94e786969215d660143817f745225579bfe318c4676e218d3147c \ + --hash=sha256:9758ec9174966ffe5c499b6c3d149f80aa0a9238020006a2b87c6af5963fcf48 \ + --hash=sha256:a169823e331da939aa7178fc152e65699aeb78957e46c6f80ccb50ee4c3616c2 \ + --hash=sha256:a67878a798f6ca292729a28c2226592b3d000dc6ee7825d31887b553686c7ac7 \ + --hash=sha256:a9a6d9eb2319a09905670810e2de971d6c49013843700b4975e2fc0afe96c8db \ + --hash=sha256:b3e118b58a3d2301960e6f5f25736d92f6b9f861728d3b8c26d69f54d8a157d2 \ + --hash=sha256:ca6705c2a1fb5059a4efbe9f5426be4cdf71b3c9564816916fc7aa7902f19ede \ + --hash=sha256:cf711527c9d4ae72085f137caffb4be74fc007ffb17cd103628c7d5ba17e205f \ + --hash=sha256:d087602a6845ebe9d5a1c5a949fedde2c45f372d77fbce4f7fe44b68b28a1d03 \ + --hash=sha256:d1080e1074ddf75ad6662a9b34626650759c19a9093e1a32a503d37e48da135b \ + --hash=sha256:db9c60368aff2b7e6c47115f3ad9bd6e96aa298b12ed5f8cb13f5673b30be565 \ + --hash=sha256:dbeb127a04473f5a989169eb400b67beb921c749599b77650941c21fe39cb8d9 \ + --hash=sha256:dca336ca3682d869d291d7cd18284f6ff6876e4244eb1821430323056b000e2c \ + --hash=sha256:dd69a9be95346d10c853b6233fcafe3c0315b89424b378f2ad45170d8e161568 \ + --hash=sha256:dd79f8fae5894f1ee0a0042214685f2d039341250c994b825c10a4cd075d80f6 \ + --hash=sha256:e647d850aa1286d98910133cee12bd87c354f7b7bb3f3cd816a62ba7fa2f7007 \ + --hash=sha256:f37a210b5c04b2d4e4bac494ab15b70196f219a1e1649ddca78560757d4278fb \ + --hash=sha256:f67820b6d33a705dc3c1c457156e51686f7b350ff57f2112e1a9a4dad38ec268 \ + --hash=sha256:f68969978ccf0e6123902f7365aae5b7a9e99169d4b9105c47cf28e788116894 \ + --hash=sha256:f717a0b34460ae1ac0064e91b267c0588ac2c098ffd695992e72cd5462d97a67 \ + --hash=sha256:f9d58ccec8684ca276d5a4e7b0dfacca028336300a8f715d616d9f0ce9ae8096 \ + --hash=sha256:fcc3513a54e656067cbf7b98bab0d6b9534b9eabc666d1f78aad6acdf0962736 +zope.interface==4.7.1 \ + --hash=sha256:048b16ac882a05bc7ef534e8b9f15c9d7a6c190e24e8938a19b7617af4ed854a \ + --hash=sha256:05816cf8e7407cf62f2ec95c0a5d69ec4fa5741d9ccd10db9f21691916a9a098 \ + --hash=sha256:065d6a1ac89d35445168813bed45048ed4e67a4cdfc5a68fdb626a770378869f \ + --hash=sha256:14157421f4121a57625002cc4f48ac7521ea238d697c4a4459a884b62132b977 \ + --hash=sha256:18dc895945694f397a0be86be760ff664b790f95d8e7752d5bab80284ff9105d \ + --hash=sha256:1962c9f838bd6ae4075d0014f72697510daefc7e1c7e48b2607df0b6e157989c \ + --hash=sha256:1a67408cacd198c7e6274a19920bb4568d56459e659e23c4915528686ac1763a \ + --hash=sha256:21bf781076dd616bd07cf0223f79d61ab4f45176076f90bc2890e18c48195da4 \ + --hash=sha256:21c0a5d98650aebb84efa16ce2c8df1a46bdc4fe8a9e33237d0ca0b23f416ead \ + --hash=sha256:23cfeea25d1e42ff3bf4f9a0c31e9d5950aa9e7c4b12f0c4bd086f378f7b7a71 \ + --hash=sha256:24b6fce1fb71abf9f4093e3259084efcc0ef479f89356757780685bd2b06ef37 \ + --hash=sha256:24f84ce24eb6b5fcdcb38ad9761524f1ae96f7126abb5e597f8a3973d9921409 \ + --hash=sha256:25e0ef4a824017809d6d8b0ce4ab3288594ba283e4d4f94d8cfb81d73ed65114 \ + --hash=sha256:2e8fdd625e9aba31228e7ddbc36bad5c38dc3ee99a86aa420f89a290bd987ce9 \ + --hash=sha256:2f3bc2f49b67b1bea82b942d25bc958d4f4ea6709b411cb2b6b9718adf7914ce \ + --hash=sha256:35d24be9d04d50da3a6f4d61de028c1dd087045385a0ff374d93ef85af61b584 \ + --hash=sha256:35dbe4e8c73003dff40dfaeb15902910a4360699375e7b47d3c909a83ff27cd0 \ + --hash=sha256:3dfce831b824ab5cf446ed0c350b793ac6fa5fe33b984305cb4c966a86a8fb79 \ + --hash=sha256:3f7866365df5a36a7b8de8056cd1c605648f56f9a226d918ed84c85d25e8d55f \ + --hash=sha256:455cc8c01de3bac6f9c223967cea41f4449f58b4c2e724ec8177382ddd183ab4 \ + --hash=sha256:4bb937e998be9d5e345f486693e477ba79e4344674484001a0b646be1d530487 \ + --hash=sha256:52303a20902ca0888dfb83230ca3ee6fbe63c0ad1dd60aa0bba7958ccff454d8 \ + --hash=sha256:6e0a897d4e09859cc80c6a16a29697406ead752292ace17f1805126a4f63c838 \ + --hash=sha256:6e1816e7c10966330d77af45f77501f9a68818c065dec0ad11d22b50a0e212e7 \ + --hash=sha256:73b5921c5c6ce3358c836461b5470bf675601c96d5e5d8f2a446951470614f67 \ + --hash=sha256:8093cd45cdb5f6c8591cfd1af03d32b32965b0f79b94684cd0c9afdf841982bb \ + --hash=sha256:864b4a94b60db301899cf373579fd9ef92edddbf0fb2cd5ae99f53ef423ccc56 \ + --hash=sha256:8a27b4d3ea9c6d086ce8e7cdb3e8d319b6752e2a03238a388ccc83ccbe165f50 \ + --hash=sha256:91b847969d4784abd855165a2d163f72ac1e58e6dce09a5e46c20e58f19cc96d \ + --hash=sha256:b47b1028be4758c3167e474884ccc079b94835f058984b15c145966c4df64d27 \ + --hash=sha256:b68814a322835d8ad671b7acc23a3b2acecba527bb14f4b53fc925f8a27e44d8 \ + --hash=sha256:bcb50a032c3b6ec7fb281b3a83d2b31ab5246c5b119588725b1350d3a1d9f6a3 \ + --hash=sha256:c56db7d10b25ce8918b6aec6b08ac401842b47e6c136773bfb3b590753f7fb67 \ + --hash=sha256:c94b77a13d4f47883e4f97f9fa00f5feadd38af3e6b3c7be45cfdb0a14c7149b \ + --hash=sha256:db381f6fdaef483ad435f778086ccc4890120aff8df2ba5cfeeac24d280b3145 \ + --hash=sha256:e6487d01c8b7ed86af30ea141fcc4f93f8a7dde26f94177c1ad637c353bd5c07 \ + --hash=sha256:e86923fa728dfba39c5bb6046a450bd4eec8ad949ac404eca728cfce320d1732 \ + --hash=sha256:f6ca36dc1e9eeb46d779869c60001b3065fb670b5775c51421c099ea2a77c3c9 \ + --hash=sha256:fb62f2cbe790a50d95593fb40e8cca261c31a2f5637455ea39440d6457c2ba25 +zope.proxy==4.3.3 \ + --hash=sha256:04646ac04ffa9c8e32fb2b5c3cd42995b2548ea14251f3c21ca704afae88e42c \ + --hash=sha256:07b6bceea232559d24358832f1cd2ed344bbf05ca83855a5b9698b5f23c5ed60 \ + --hash=sha256:1ef452cc02e0e2f8e3c917b1a5b936ef3280f2c2ca854ee70ac2164d1655f7e6 \ + --hash=sha256:22bf61857c5977f34d4e391476d40f9a3b8c6ab24fb0cac448d42d8f8b9bf7b2 \ + --hash=sha256:299870e3428cbff1cd9f9b34144e76ecdc1d9e3192a8cf5f1b0258f47a239f58 \ + --hash=sha256:2bfc36bfccbe047671170ea5677efd3d5ab730a55d7e45611d76d495e5b96766 \ + --hash=sha256:32e82d5a640febc688c0789e15ea875bf696a10cf358f049e1ed841f01710a9b \ + --hash=sha256:3b2051bdc4bc3f02fa52483f6381cf40d4d48167645241993f9d7ebbd142ed9b \ + --hash=sha256:3f734bd8a08f5185a64fb6abb8f14dc97ec27a689ca808fb7a83cdd38d745e4f \ + --hash=sha256:3f78dd8de3112df8bbd970f0916ac876dc3fbe63810bd1cf7cc5eec4cbac4f04 \ + --hash=sha256:4eabeb48508953ba1f3590ad0773b8daea9e104eec66d661917e9bbcd7125a67 \ + --hash=sha256:4f05ecc33808187f430f249cb1ccab35c38f570b181f2d380fbe253da94b18d8 \ + --hash=sha256:4f4f4cbf23d3afc1526294a31e7b3eaa0f682cc28ac5366065dc1d6bb18bd7be \ + --hash=sha256:5483d5e70aacd06f0aa3effec9fed597c0b50f45060956eeeb1203c44d4338c3 \ + --hash=sha256:56a5f9b46892b115a75d0a1f2292431ad5988461175826600acc69a24cb3edee \ + --hash=sha256:64bb63af8a06f736927d260efdd4dfc5253d42244f281a8063e4b9eea2ddcbc5 \ + --hash=sha256:653f8cbefcf7c6ac4cece2cdef367c4faa2b7c19795d52bd7cbec11a8739a7c1 \ + --hash=sha256:664211d63306e4bd4eec35bf2b4bd9db61c394037911cf2d1804c43b511a49f1 \ + --hash=sha256:6651e6caed66a8fff0fef1a3e81c0ed2253bf361c0fdc834500488732c5d16e9 \ + --hash=sha256:6c1fba6cdfdf105739d3069cf7b07664f2944d82a8098218ab2300a82d8f40fc \ + --hash=sha256:6e64246e6e9044a4534a69dca1283c6ddab6e757be5e6874f69024329b3aa61f \ + --hash=sha256:838390245c7ec137af4993c0c8052f49d5ec79e422b4451bfa37fee9b9ccaa01 \ + --hash=sha256:856b410a14793069d8ba35f33fff667213ea66f2df25a0024cc72a7493c56d4c \ + --hash=sha256:8b932c364c1d1605a91907a41128ed0ee8a2d326fc0fafb2c55cd46f545f4599 \ + --hash=sha256:9086cf6d20f08dae7f296a78f6c77d1f8d24079d448f023ee0eb329078dd35e1 \ + --hash=sha256:9698533c14afa0548188de4968a7932d1f3f965f3f5ba1474de673596bb875af \ + --hash=sha256:9b12b05dd7c28f5068387c1afee8cb94f9d02501e7ef495a7c5c7e27139b96ad \ + --hash=sha256:a884c7426a5bc6fb7fc71a55ad14e66818e13f05b78b20a6f37175f324b7acb8 \ + --hash=sha256:abe9e7f1a3e76286c5f5baf2bf5162d41dc0310da493b34a2c36555f38d928f7 \ + --hash=sha256:bd6fde63b015a27262be06bd6bbdd895273cc2bdf2d4c7e1c83711d26a8fbace \ + --hash=sha256:bda7c62c954f47b87ed9a89f525eee1b318ec7c2162dfdba76c2ccfa334e0caa \ + --hash=sha256:be8a4908dd3f6e965993c0068b006bdbd0474fbcbd1da4893b49356e73fc1557 \ + --hash=sha256:ced65fc3c7d7205267506d854bb1815bb445899cca9d21d1d4b949070a635546 \ + --hash=sha256:dac4279aa05055d3897ab5e5ee5a7b39db121f91df65a530f8b1ac7f9bd93119 \ + --hash=sha256:e4f1863056e3e4f399c285b67fa816f411a7bfa1c81ef50e186126164e396e59 \ + --hash=sha256:ecd85f68b8cd9ab78a0141e87ea9a53b2f31fd9b1350a1c44da1f7481b5363ef \ + --hash=sha256:ed269b83750413e8fc5c96276372f49ee3fcb7ed61c49fe8e5a67f54459a5a4a \ + --hash=sha256:f19b0b80cba73b204dee68501870b11067711d21d243fb6774256d3ca2e5391f \ + --hash=sha256:ffdafb98db7574f9da84c489a10a5d582079a888cb43c64e9e6b0e3fe1034685 diff --git a/letsencrypt-auto-source/pieces/pipstrap.py b/letsencrypt-auto-source/pieces/pipstrap.py index 6a00dd9cb..346e23938 100755 --- a/letsencrypt-auto-source/pieces/pipstrap.py +++ b/letsencrypt-auto-source/pieces/pipstrap.py @@ -23,7 +23,6 @@ from distutils.version import StrictVersion from hashlib import sha256 from os import environ from os.path import join -from pipes import quote from shutil import rmtree try: from subprocess import check_output @@ -43,7 +42,7 @@ except ImportError: cmd = popenargs[0] raise CalledProcessError(retcode, cmd) return output -from sys import exit, version_info +import sys from tempfile import mkdtemp try: from urllib2 import build_opener, HTTPHandler, HTTPSHandler @@ -65,7 +64,7 @@ maybe_argparse = ( [('18/dd/e617cfc3f6210ae183374cd9f6a26b20514bbb5a792af97949c5aacddf0f/' 'argparse-1.4.0.tar.gz', '62b089a55be1d8949cd2bc7e0df0bddb9e028faefc8c32038cc84862aefdd6e4')] - if version_info < (2, 7, 0) else []) + if sys.version_info < (2, 7, 0) else []) PACKAGES = maybe_argparse + [ @@ -146,7 +145,8 @@ def get_index_base(): def main(): - pip_version = StrictVersion(check_output(['pip', '--version']) + python = sys.executable or 'python' + pip_version = StrictVersion(check_output([python, '-m', 'pip', '--version']) .decode('utf-8').split()[1]) has_pip_cache = pip_version >= StrictVersion('6.0') index_base = get_index_base() @@ -156,12 +156,12 @@ def main(): temp, digest) for path, digest in PACKAGES] - check_output('pip install --no-index --no-deps -U ' + - # Disable cache since we're not using it and it otherwise - # sometimes throws permission warnings: - ('--no-cache-dir ' if has_pip_cache else '') + - ' '.join(quote(d) for d in downloads), - shell=True) + # Calling pip as a module is the preferred way to avoid problems about pip self-upgrade. + command = [python, '-m', 'pip', 'install', '--no-index', '--no-deps', '-U'] + # Disable cache since it is not used and it otherwise sometimes throws permission warnings: + command.extend(['--no-cache-dir'] if has_pip_cache else []) + command.extend(downloads) + check_output(command) except HashError as exc: print(exc) except Exception: @@ -174,4 +174,4 @@ def main(): if __name__ == '__main__': - exit(main()) + sys.exit(main()) diff --git a/letsencrypt-auto-source/rebuild_dependencies.py b/letsencrypt-auto-source/rebuild_dependencies.py new file mode 100755 index 000000000..6d1ec15ff --- /dev/null +++ b/letsencrypt-auto-source/rebuild_dependencies.py @@ -0,0 +1,283 @@ +#!/usr/bin/env python +""" +Gather and consolidate the up-to-date dependencies available and required to install certbot +on various Linux distributions. It generates a requirements file contained the pinned and hashed +versions, ready to be used by pip to install the certbot dependencies. + +This script is typically used to update the certbot-requirements.txt file of certbot-auto. + +To achieve its purpose, this script will start a certbot installation with unpinned dependencies, +then gather them, on various distributions started as Docker containers. + +Usage: letsencrypt-auto-source/rebuild_dependencies new_requirements.txt + +NB1: Docker must be installed on the machine running this script. +NB2: Python library 'hashin' must be installed on the machine running this script. +""" +from __future__ import print_function +import re +import shutil +import subprocess +import tempfile +import os +from os.path import dirname, abspath, join +import sys +import argparse + +# The list of docker distributions to test dependencies against with. +DISTRIBUTION_LIST = [ + 'ubuntu:18.04', 'ubuntu:16.04', + 'debian:stretch', 'debian:jessie', + 'centos:7', 'centos:6', + 'opensuse/leap:15', + 'fedora:29', +] + +# These constraints will be added while gathering dependencies on each distribution. +# It can be used because a particular version for a package is required for any reason, +# or to solve a version conflict between two distributions requirements. +AUTHORITATIVE_CONSTRAINTS = { + # Using an older version of mock here prevents regressions of #5276. + 'mock': '1.3.0', + # Too touchy to move to a new version. And will be removed soon + # in favor of pure python parser for Apache. + 'python-augeas': '0.5.0', + # Package enum34 needs to be explicitly limited to Python2.x, in order to avoid + # certbot-auto failures on Python 3.6+ which enum34 doesn't support. See #5456. + # TODO: hashin seems to overwrite environment markers in dependencies. This needs to be fixed. + 'enum34': '1.1.6 ; python_version < \'3.4\'', +} + + +# ./certbot/letsencrypt-auto-source/rebuild_dependencies.py (2 levels from certbot root path) +CERTBOT_REPO_PATH = dirname(dirname(abspath(__file__))) + +# The script will be used to gather dependencies for a given distribution. +# - certbot-auto is used to install relevant OS packages, and set up an initial venv +# - then this venv is used to consistently construct an empty new venv +# - once pipstraped, this new venv pip-installs certbot runtime (including apache/nginx), +# without pinned dependencies, and respecting input authoritative requirements +# - `certbot plugins` is called to check we have a healthy environment +# - finally current set of dependencies is extracted out of the docker using pip freeze +SCRIPT = r"""#!/bin/sh +set -e + +cd /tmp/certbot +letsencrypt-auto-source/letsencrypt-auto --install-only -n +PYVER=`/opt/eff.org/certbot/venv/bin/python --version 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//'` + +/opt/eff.org/certbot/venv/bin/python letsencrypt-auto-source/pieces/create_venv.py /tmp/venv "$PYVER" 1 + +/tmp/venv/bin/python letsencrypt-auto-source/pieces/pipstrap.py +/tmp/venv/bin/pip install -e acme -e certbot -e certbot-apache -e certbot-nginx -c /tmp/constraints.txt +/tmp/venv/bin/certbot plugins +/tmp/venv/bin/pip freeze >> /tmp/workspace/requirements.txt +""" + + +def _read_from(file): + """Read all content of the file, and return it as a string.""" + with open(file, 'r') as file_h: + return file_h.read() + + +def _write_to(file, content): + """Write given string content to the file, overwriting its initial content.""" + with open(file, 'w') as file_h: + file_h.write(content) + + +def _requirements_from_one_distribution(distribution, verbose): + """ + Calculate the Certbot dependencies expressed for the given distribution, using the official + Docker for this distribution, and return the lines of the generated requirements file. + """ + print('===> Gathering dependencies for {0}.'.format(distribution)) + workspace = tempfile.mkdtemp() + script = join(workspace, 'script.sh') + authoritative_constraints = join(workspace, 'constraints.txt') + cid_file = join(workspace, 'cid') + + try: + _write_to(script, SCRIPT) + os.chmod(script, 0o755) + + _write_to(authoritative_constraints, '\n'.join( + ['{0}=={1}'.format(package, version) for package, version in AUTHORITATIVE_CONSTRAINTS.items()])) + + command = ['docker', 'run', '--rm', '--cidfile', cid_file, + '-v', '{0}:/tmp/certbot'.format(CERTBOT_REPO_PATH), + '-v', '{0}:/tmp/workspace'.format(workspace), + '-v', '{0}:/tmp/constraints.txt'.format(authoritative_constraints), + distribution, '/tmp/workspace/script.sh'] + sub_stdout = sys.stdout if verbose else subprocess.PIPE + sub_stderr = sys.stderr if verbose else subprocess.STDOUT + process = subprocess.Popen(command, stdout=sub_stdout, stderr=sub_stderr, universal_newlines=True) + stdoutdata, _ = process.communicate() + + if process.returncode: + if stdoutdata: + sys.stderr.write('Output was:\n{0}'.format(stdoutdata)) + raise RuntimeError('Error while gathering dependencies for {0}.'.format(distribution)) + + with open(join(workspace, 'requirements.txt'), 'r') as file_h: + return file_h.readlines() + finally: + if os.path.isfile(cid_file): + cid = _read_from(cid_file) + try: + subprocess.check_output(['docker', 'kill', cid], stderr=subprocess.PIPE) + except subprocess.CalledProcessError: + pass + shutil.rmtree(workspace) + + +def _parse_and_merge_requirements(dependencies_map, requirements_file_lines, distribution): + """ + Extract every requirement from the given requirements file, and merge it in the dependency map. + Merging here means that the map contain every encountered dependency, and the version used in + each distribution. + + Example: + # dependencies_map = { + # } + _parse_and_merge_requirements(['cryptography=='1.2','requests=='2.1.0'], dependencies_map, 'debian:stretch') + # dependencies_map = { + # 'cryptography': [('1.2', 'debian:stretch)], + # 'requests': [('2.1.0', 'debian:stretch')] + # } + _parse_and_merge_requirements(['requests=='2.4.0', 'mock==1.3'], dependencies_map, 'centos:7') + # dependencies_map = { + # 'cryptography': [('1.2', 'debian:stretch)], + # 'requests': [('2.1.0', 'debian:stretch'), ('2.4.0', 'centos:7')], + # 'mock': [('2.4.0', 'centos:7')] + # } + """ + for line in requirements_file_lines: + match = re.match(r'([^=]+)==([^=]+)', line.strip()) + if not line.startswith('-e') and match: + package, version = match.groups() + if package not in ['acme', 'certbot', 'certbot-apache', 'certbot-nginx', 'pkg-resources']: + dependencies_map.setdefault(package, []).append((version, distribution)) + + +def _consolidate_and_validate_dependencies(dependency_map): + """ + Given the dependency map of all requirements found in all distributions for Certbot, + construct an array containing the unit requirements for Certbot to be used by pip, + and the version conflicts, if any, between several distributions for a package. + Return requirements and conflicts as a tuple. + """ + print('===> Consolidate and validate the dependency map.') + requirements = [] + conflicts = [] + for package, versions in dependency_map.items(): + reduced_versions = _reduce_versions(versions) + + if len(reduced_versions) > 1: + version_list = ['{0} ({1})'.format(version, ','.join(distributions)) + for version, distributions in reduced_versions.items()] + conflict = ('package {0} is declared with several versions: {1}' + .format(package, ', '.join(version_list))) + conflicts.append(conflict) + sys.stderr.write('ERROR: {0}\n'.format(conflict)) + else: + requirements.append((package, list(reduced_versions)[0])) + + requirements.sort(key=lambda x: x[0]) + return requirements, conflicts + + +def _reduce_versions(version_dist_tuples): + """ + Get an array of version/distribution tuples, + and reduce it to a map based on the version values. + + Example: [('1.2.0', 'debian:stretch'), ('1.4.0', 'ubuntu:18.04'), ('1.2.0', 'centos:6')] + => {'1.2.0': ['debiqn:stretch', 'centos:6'], '1.4.0': ['ubuntu:18.04']} + """ + version_dist_map = {} + for version, distribution in version_dist_tuples: + version_dist_map.setdefault(version, []).append(distribution) + + return version_dist_map + + +def _write_requirements(dest_file, requirements, conflicts): + """ + Given the list of requirements and conflicts, write a well-formatted requirements file, + whose requirements are hashed signed using hashin library. Conflicts are written at the end + of the generated file. + """ + print('===> Calculating hashes for the requirement file.') + + _write_to(dest_file, '''\ +# This is the flattened list of packages certbot-auto installs. +# To generate this, do (with docker and package hashin installed): +# ``` +# letsencrypt-auto-source/rebuild_dependencies.py \\ +# letsencrypt-auto-source/pieces/dependency-requirements.txt +# ``` +# If you want to update a single dependency, run commands similar to these: +# ``` +# pip install hashin +# hashin -r dependency-requirements.txt cryptography==1.5.2 +# ``` +''') + + for req in requirements: + subprocess.check_call(['hashin', '{0}=={1}'.format(req[0], req[1]), + '--requirements-file', dest_file]) + + if conflicts: + with open(dest_file, 'a') as file_h: + file_h.write('\n## ! SOME ERRORS OCCURRED ! ##\n') + file_h.write('\n'.join('# {0}'.format(conflict) for conflict in conflicts)) + file_h.write('\n') + + return _read_from(dest_file) + + +def _gather_dependencies(dest_file, verbose): + """ + Main method of this script. Given a destination file path, will write the file + containing the consolidated and hashed requirements for Certbot, validated + against several Linux distributions. + """ + dependencies_map = {} + + for distribution in DISTRIBUTION_LIST: + requirements_file_lines = _requirements_from_one_distribution(distribution, verbose) + _parse_and_merge_requirements(dependencies_map, requirements_file_lines, distribution) + + requirements, conflicts = _consolidate_and_validate_dependencies(dependencies_map) + + return _write_requirements(dest_file, requirements, conflicts) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + description=('Build a sanitized, pinned and hashed requirements file for certbot-auto, ' + 'validated against several OS distributions using Docker.')) + parser.add_argument('requirements_path', + help='path for the generated requirements file') + parser.add_argument('--verbose', '-v', action='store_true', + help='verbose will display all output during docker execution') + + namespace = parser.parse_args() + + try: + subprocess.check_output(['hashin', '--version']) + except subprocess.CalledProcessError: + raise RuntimeError('Python library hashin is not installed in the current environment.') + + try: + subprocess.check_output(['docker', '--version'], stderr=subprocess.STDOUT) + except subprocess.CalledProcessError: + raise RuntimeError('Docker is not installed or accessible to current user.') + + file_content = _gather_dependencies(namespace.requirements_path, namespace.verbose) + + print(file_content) + print('===> Rebuilt requirement file is available on path {0}' + .format(abspath(namespace.requirements_path))) diff --git a/letsencrypt-auto-source/tests/auto_test.py b/letsencrypt-auto-source/tests/auto_test.py index 16c478f20..9c823fb55 100644 --- a/letsencrypt-auto-source/tests/auto_test.py +++ b/letsencrypt-auto-source/tests/auto_test.py @@ -4,13 +4,13 @@ from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler from contextlib import contextmanager from functools import partial from json import dumps -from os import chmod, environ, makedirs +from os import chmod, environ, makedirs, stat from os.path import abspath, dirname, exists, join import re from shutil import copy, rmtree import socket import ssl -from stat import S_IRUSR, S_IXUSR +from stat import S_IMODE, S_IRUSR, S_IWUSR, S_IXUSR, S_IWGRP, S_IWOTH from subprocess import CalledProcessError, Popen, PIPE import sys from tempfile import mkdtemp @@ -192,7 +192,7 @@ def install_le_auto(contents, install_path): chmod(install_path, S_IRUSR | S_IXUSR) -def run_le_auto(le_auto_path, venv_dir, base_url, **kwargs): +def run_le_auto(le_auto_path, venv_dir, base_url=None, le_auto_args_str='--version', **kwargs): """Run the prebuilt version of letsencrypt-auto, returning stdout and stderr strings. @@ -201,13 +201,17 @@ def run_le_auto(le_auto_path, venv_dir, base_url, **kwargs): """ env = environ.copy() d = dict(VENV_PATH=venv_dir, - # URL to PyPI-style JSON that tell us the latest released version - # of LE: - LE_AUTO_JSON_URL=base_url + 'certbot/json', - # URL to dir containing letsencrypt-auto and letsencrypt-auto.sig: - LE_AUTO_DIR_TEMPLATE=base_url + '%s/', - # The public key corresponding to signing.key: - LE_AUTO_PUBLIC_KEY="""-----BEGIN PUBLIC KEY----- + NO_CERT_VERIFY='1', + **kwargs) + + if base_url is not None: + # URL to PyPI-style JSON that tell us the latest released version + # of LE: + d['LE_AUTO_JSON_URL'] = base_url + 'certbot/json' + # URL to dir containing letsencrypt-auto and letsencrypt-auto.sig: + d['LE_AUTO_DIR_TEMPLATE'] = base_url + '%s/' + # The public key corresponding to signing.key: + d['LE_AUTO_PUBLIC_KEY'] = """-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsMoSzLYQ7E1sdSOkwelg tzKIh2qi3bpXuYtcfFC0XrvWig071NwIj+dZiT0OLZ2hPispEH0B7ISuuWg1ll7G hFW0VdbxL6JdGzS2ShNWkX9hE9z+j8VqwDPOBn3ZHm03qwpYkBDwQib3KqOdYbTT @@ -215,12 +219,12 @@ uUtJmmGcuk3a9Aq/sCT6DdfmTSdP5asdQYwIcaQreDrOosaS84DTWI3IU+UYJVgl LsIVPBuy9IcgHidUQ96hJnoPsDCWsHwX62495QKEarauyKQrJzFes0EY95orDM47 Z5o/NDiQB11m91yNB0MmPYY9QSbnOA9j7IaaC97AwRLuwXY+/R2ablTcxurWou68 iQIDAQAB ------END PUBLIC KEY-----""", - NO_CERT_VERIFY='1', - **kwargs) +-----END PUBLIC KEY-----""" + env.update(d) + return out_and_err( - le_auto_path + ' --version', + le_auto_path + ' ' + le_auto_args_str, shell=True, env=env) @@ -240,6 +244,12 @@ def set_le_script_version(venv_dir, version): chmod(letsencrypt_path, S_IRUSR | S_IXUSR) +def sudo_chmod(path, mode): + """Runs `sudo chmod mode path`.""" + mode = oct(mode).replace('o', '') + out_and_err(['sudo', 'chmod', mode, path]) + + class AutoTests(TestCase): """Test the major branch points of letsencrypt-auto: @@ -395,3 +405,95 @@ class AutoTests(TestCase): else: self.fail("Pip didn't detect a bad hash and stop the " "installation.") + + def test_permissions_warnings(self): + """Make sure letsencrypt-auto properly warns about permissions problems.""" + # This test assumes that only the parent of the directory containing + # letsencrypt-auto (usually /tmp) may have permissions letsencrypt-auto + # considers insecure. + with temp_paths() as (le_auto_path, venv_dir): + le_auto_path = abspath(le_auto_path) + le_auto_dir = dirname(le_auto_path) + le_auto_dir_parent = dirname(le_auto_dir) + install_le_auto(self.NEW_LE_AUTO, le_auto_path) + + run_letsencrypt_auto = partial( + run_le_auto, le_auto_path, venv_dir, + le_auto_args_str='--install-only --no-self-upgrade', + PIP_FIND_LINKS=join(tests_dir(), 'fake-letsencrypt', 'dist')) + # Run letsencrypt-auto once with current permissions to avoid + # potential problems when the script tries to write to temporary + # directories. + run_letsencrypt_auto() + + le_auto_dir_mode = stat(le_auto_dir).st_mode + le_auto_dir_parent_mode = S_IMODE(stat(le_auto_dir_parent).st_mode) + try: + # Make letsencrypt-auto happy with the current permissions + chmod(le_auto_dir, S_IRUSR | S_IXUSR) + sudo_chmod(le_auto_dir_parent, 0o755) + + self._test_permissions_warnings_about_path(le_auto_path, run_letsencrypt_auto) + self._test_permissions_warnings_about_path(le_auto_dir, run_letsencrypt_auto) + finally: + chmod(le_auto_dir, le_auto_dir_mode) + sudo_chmod(le_auto_dir_parent, le_auto_dir_parent_mode) + + def _test_permissions_warnings_about_path(self, path, run_le_auto_func): + # Test that there are no problems with the current permissions + out, _ = run_le_auto_func() + self.assertFalse('insecure permissions' in out) + + stat_result = stat(path) + original_mode = stat_result.st_mode + + # Test world permissions + chmod(path, original_mode | S_IWOTH) + out, _ = run_le_auto_func() + self.assertTrue('insecure permissions' in out) + + # Test group permissions + if stat_result.st_gid >= 1000: + chmod(path, original_mode | S_IWGRP) + out, _ = run_le_auto_func() + self.assertTrue('insecure permissions' in out) + + # Test owner permissions + if stat_result.st_uid >= 1000: + chmod(path, original_mode | S_IWUSR) + out, _ = run_le_auto_func() + self.assertTrue('insecure permissions' in out) + + # Test that permissions were properly restored + chmod(path, original_mode) + out, _ = run_le_auto_func() + self.assertFalse('insecure permissions' in out) + + def test_disabled_permissions_warnings(self): + """Make sure that letsencrypt-auto permissions warnings can be disabled.""" + with temp_paths() as (le_auto_path, venv_dir): + le_auto_path = abspath(le_auto_path) + install_le_auto(self.NEW_LE_AUTO, le_auto_path) + + le_auto_args_str='--install-only --no-self-upgrade' + pip_links=join(tests_dir(), 'fake-letsencrypt', 'dist') + out, _ = run_le_auto(le_auto_path, venv_dir, + le_auto_args_str=le_auto_args_str, + PIP_FIND_LINKS=pip_links) + self.assertTrue('insecure permissions' in out) + + # Test that warnings are disabled when the script isn't run as + # root. + out, _ = run_le_auto(le_auto_path, venv_dir, + le_auto_args_str=le_auto_args_str, + LE_AUTO_SUDO='', + PIP_FIND_LINKS=pip_links) + self.assertFalse('insecure permissions' in out) + + # Test that --no-permissions-check disables warnings. + le_auto_args_str += ' --no-permissions-check' + out, _ = run_le_auto( + le_auto_path, venv_dir, + le_auto_args_str=le_auto_args_str, + PIP_FIND_LINKS=pip_links) + self.assertFalse('insecure permissions' in out) diff --git a/letsencrypt-auto-source/tests/centos6_tests.sh b/letsencrypt-auto-source/tests/centos6_tests.sh index 2c6dcf734..8bdffec87 100644 --- a/letsencrypt-auto-source/tests/centos6_tests.sh +++ b/letsencrypt-auto-source/tests/centos6_tests.sh @@ -1,81 +1,173 @@ #!/bin/bash +set -e # Start by making sure your system is up-to-date: -yum update -y > /dev/null -yum install -y centos-release-scl > /dev/null -yum install -y python27 > /dev/null 2> /dev/null +yum update -y >/dev/null +yum install -y centos-release-scl >/dev/null +yum install -y python27 >/dev/null 2>/dev/null +LE_AUTO_PY_34="certbot/letsencrypt-auto-source/letsencrypt-auto_py_34" LE_AUTO="certbot/letsencrypt-auto-source/letsencrypt-auto" +# Last version of certbot-auto that was bootstraping Python 3.4 for CentOS 6 users +INITIAL_CERTBOT_VERSION_PY34="certbot 0.38.0" + # we're going to modify env variables, so do this in a subshell ( -source /opt/rh/python27/enable - -# ensure python 3 isn't installed -python3 --version 2> /dev/null -RESULT=$? -if [ $RESULT -eq 0 ]; then - error "Python3 is already installed." +# ensure CentOS6 32bits is not supported anymore, and so certbot is not installed +export UNAME_FAKE_32BITS=true +if ! "$LE_AUTO" 2>&1 | grep -q "Certbot cannot be installed."; then + echo "ERROR: certbot-auto installed certbot on 32-bit CentOS." exit 1 fi +) -# ensure python2.7 is available -python2.7 --version 2> /dev/null -RESULT=$? -if [ $RESULT -ne 0 ]; then - error "Python3 is not available." - exit 1 -fi +echo "PASSED: On CentOS 6 32 bits, certbot-auto refused to install certbot." -# bootstrap, but don't install python 3. -"$LE_AUTO" --no-self-upgrade -n > /dev/null 2> /dev/null +# we're going to modify env variables, so do this in a subshell +( + . /opt/rh/python27/enable -# ensure python 3 isn't installed -python3 --version 2> /dev/null -RESULT=$? -if [ $RESULT -eq 0 ]; then - error "letsencrypt-auto installed Python3 even though Python2.7 is present." - exit 1 -fi + # ensure python 3 isn't installed + if python3 --version 2> /dev/null; then + echo "ERROR: Python3 is already installed." + exit 1 + fi -echo "" -echo "PASSED: Did not upgrade to Python3 when Python2.7 is present." + # ensure python2.7 is available + if ! python2.7 --version 2> /dev/null; then + echo "ERROR: Python2.7 is not available." + exit 1 + fi + + # bootstrap, but don't install python 3. + "$LE_AUTO" --no-self-upgrade -n --version > /dev/null 2> /dev/null + + # ensure python 3 isn't installed + if python3 --version 2> /dev/null; then + echo "ERROR: letsencrypt-auto installed Python3 even though Python2.7 is present." + exit 1 + fi + + echo "PASSED: Did not upgrade to Python3 when Python2.7 is present." ) # ensure python2.7 isn't available -python2.7 --version 2> /dev/null -RESULT=$? -if [ $RESULT -eq 0 ]; then - error "Python2.7 is still available." +if python2.7 --version 2> /dev/null; then + echo "ERROR: Python2.7 is still available." exit 1 fi # Skip self upgrade due to Python 3 not being available. if ! "$LE_AUTO" 2>&1 | grep -q "WARNING: couldn't find Python"; then - echo "Python upgrade failure warning not printed!" + echo "ERROR: Python upgrade failure warning not printed!" exit 1 fi -# bootstrap, this time installing python3 -"$LE_AUTO" --no-self-upgrade -n > /dev/null 2> /dev/null +# bootstrap from the old letsencrypt-auto, this time installing python3.4 +"$LE_AUTO_PY_34" --no-self-upgrade -n --version >/dev/null 2>/dev/null -# ensure python 3 is installed -python3 --version > /dev/null -RESULT=$? -if [ $RESULT -ne 0 ]; then - error "letsencrypt-auto failed to install Python3 when only Python2.6 is present." +# ensure python 3.4 is installed +if ! python3.4 --version >/dev/null 2>/dev/null; then + echo "ERROR: letsencrypt-auto failed to install Python3.4 using letsencrypt-auto < 0.37.0 when only Python2.6 is present." exit 1 fi -echo "PASSED: Successfully upgraded to Python3 when only Python2.6 is present." -echo "" +echo "PASSED: Successfully upgraded to Python3.4 using letsencrypt-auto < 0.37.0 when only Python2.6 is present." -export VENV_PATH=$(mktemp -d) -"$LE_AUTO" -n --no-bootstrap --no-self-upgrade --version >/dev/null 2>&1 -if [ "$($VENV_PATH/bin/python -V 2>&1 | cut -d" " -f2 | cut -d. -f1)" != 3 ]; then - echo "Python 3 wasn't used with --no-bootstrap!" +# As "certbot-auto" (so without implicit --non-interactive flag set), check that the script +# refuses to install SCL Python 3.6 when run in a non interactive shell (simulated here +# using | tee /dev/null) if --non-interactive flag is not provided. +cp "$LE_AUTO" /tmp/certbot-auto +# NB: Readline has an issue on all Python versions for CentOS 6, making `certbot --version` +# output an unprintable ASCII character on a new line at the end. +# So we take the second last line of the output. +version=$(/tmp/certbot-auto --version 2>/dev/null | tee /dev/null | tail -2 | head -1) + +if [ "$version" != "$INITIAL_CERTBOT_VERSION_PY34" ]; then + echo "ERROR: certbot-auto upgraded certbot in a non-interactive shell with --non-interactive flag not set." exit 1 fi -unset VENV_PATH + +echo "PASSED: certbot-auto did not upgrade certbot in a non-interactive shell with --non-interactive flag not set." + +if [ -f /opt/rh/rh-python36/enable ]; then + echo "ERROR: certbot-auto installed Python3.6 in a non-interactive shell with --non-interactive flag not set." + exit 1 +fi + +echo "PASSED: certbot-auto did not install Python3.6 in a non-interactive shell with --non-interactive flag not set." + +# now bootstrap from current letsencrypt-auto, that will install python3.6 from SCL +"$LE_AUTO" --no-self-upgrade -n --version >/dev/null 2>/dev/null + +# Following test is executed in a subshell, to not leak any environment variable +( + # enable SCL rh-python36 + . /opt/rh/rh-python36/enable + + # ensure python 3.6 is installed + if ! python3.6 --version >/dev/null 2>/dev/null; then + echo "ERROR: letsencrypt-auto failed to install Python3.6 using current letsencrypt-auto when only Python2.6/Python3.4 are present." + exit 1 + fi + + echo "PASSED: Successfully upgraded to Python3.6 using current letsencrypt-auto when only Python2.6/Python3.4 are present." +) + +# Following test is executed in a subshell, to not leak any environment variable +( + export VENV_PATH=$(mktemp -d) + "$LE_AUTO" -n --no-bootstrap --no-self-upgrade --version >/dev/null 2>&1 + if [ "$($VENV_PATH/bin/python -V 2>&1 | cut -d" " -f2 | cut -d. -f1-2)" != "3.6" ]; then + echo "ERROR: Python 3.6 wasn't used with --no-bootstrap!" + exit 1 + fi +) + +# Following test is executed in a subshell, to not leak any environment variable +( + # enable SCL rh-python36 + . /opt/rh/rh-python36/enable + + # ensure everything works fine with certbot-auto bootstrap when python 3.6 is already enabled + export VENV_PATH=$(mktemp -d) + if ! "$LE_AUTO" --no-self-upgrade -n --version >/dev/null 2>/dev/null; then + echo "ERROR: Certbot-auto broke when Python 3.6 SCL is already enabled." + exit 1 + fi +) + +# we're going to modify env variables, so do this in a subshell +( + # ensure CentOS6 32bits is not supported anymore, and so certbot + # is not upgraded nor reinstalled. + export UNAME_FAKE_32BITS=true + OUTPUT=$("$LE_AUTO" --version 2>&1) + if ! echo "$OUTPUT" | grep -q "Certbot will no longer receive updates."; then + echo "ERROR: certbot-auto failed to run or upgraded pre-existing Certbot instance on 32-bit CentOS 6." + exit 1 + fi + if ! "$LE_AUTO" --install-only 2>&1 | grep -q "Certbot cannot be installed."; then + echo "ERROR: certbot-auto reinstalled Certbot on 32-bit CentOS 6." + exit 1 + fi +) + +# we're going to modify env variables, so do this in a subshell +( + # Prepare a certbot installation in the old venv path + rm -rf /opt/eff.org + VENV_PATH=~/.local/share/letsencrypt "$LE_AUTO" --install-only > /dev/null 2> /dev/null + # fake 32 bits mode + export UNAME_FAKE_32BITS=true + OUTPUT=$("$LE_AUTO" --version 2>&1) + if ! echo "$OUTPUT" | grep -q "Certbot will no longer receive updates."; then + echo "ERROR: certbot-auto failed to run or upgraded pre-existing Certbot instance in the old venv path on 32-bit CentOS 6." + exit 1 + fi +) + +echo "PASSED: certbot-auto refused to install/upgrade certbot on 32-bit CentOS 6." # test using python3 pytest -v -s certbot/letsencrypt-auto-source/tests diff --git a/letsencrypt-auto-source/tests/oraclelinux6_tests.sh b/letsencrypt-auto-source/tests/oraclelinux6_tests.sh new file mode 100644 index 000000000..f3fd952f3 --- /dev/null +++ b/letsencrypt-auto-source/tests/oraclelinux6_tests.sh @@ -0,0 +1,85 @@ +#!/bin/bash +set -eo pipefail +# Start by making sure your system is up-to-date: +yum update -y >/dev/null + +LE_AUTO_PY_34="certbot/letsencrypt-auto-source/letsencrypt-auto_py_34" +LE_AUTO="certbot/letsencrypt-auto-source/letsencrypt-auto" + +# Apply installation instructions from official documentation: +# https://certbot.eff.org/lets-encrypt/centosrhel6-other +cp "$LE_AUTO" /usr/local/bin/certbot-auto +chown root /usr/local/bin/certbot-auto +chmod 0755 /usr/local/bin/certbot-auto +LE_AUTO=/usr/local/bin/certbot-auto + +# Last version of certbot-auto that was bootstraping Python 3.4 for CentOS 6 users +INITIAL_CERTBOT_VERSION_PY34="certbot 0.38.0" + +# Check bootstrap from current certbot-auto will fail, because SCL is not enabled. +set +o pipefail +if ! "$LE_AUTO" -n 2>&1 | grep -q "Enable the SCL repository and try running Certbot again."; then + echo "ERROR: Bootstrap was not aborted although SCL was not installed!" + exit 1 +fi +set -o pipefail + +echo "PASSED: Bootstrap was aborted since SCL was not installed." + +# Bootstrap from the old letsencrypt-auto, Python 3.4 will be installed from EPEL. +"$LE_AUTO_PY_34" --no-self-upgrade -n --install-only >/dev/null 2>/dev/null + +# Ensure Python 3.4 is installed +if ! command -v python3.4 &>/dev/null; then + echo "ERROR: old letsencrypt-auto failed to install Python3.4 using letsencrypt-auto < 0.37.0 when only Python2.6 is present." + exit 1 +fi + +echo "PASSED: Bootstrap from old letsencrypt-auto succeeded and installed Python 3.4" + +# Expect certbot-auto to skip rebootstrapping with a warning since SCL is not installed. +if ! "$LE_AUTO" --non-interactive --version 2>&1 | grep -q "This requires manual user intervention"; then + echo "FAILED: Script certbot-auto did not print a warning about needing manual intervention!" + exit 1 +fi + +echo "PASSED: Script certbot-auto did not rebootstrap." + +# NB: Readline has an issue on all Python versions for OL 6, making `certbot --version` +# output an unprintable ASCII character on a new line at the end. +# So we take the second last line of the output. +version=$($LE_AUTO --version 2>/dev/null | tail -2 | head -1) + +if [ "$version" != "$INITIAL_CERTBOT_VERSION_PY34" ]; then + echo "ERROR: Script certbot-auto upgraded certbot in a non-interactive shell while SCL was not enabled." + exit 1 +fi + +echo "PASSED: Script certbot-auto did not upgrade certbot but started it successfully while SCL was not enabled." + +# Enable SCL +yum install -y oracle-softwarecollection-release-el6 >/dev/null + +# Expect certbot-auto to bootstrap successfully since SCL is available. +"$LE_AUTO" -n --version &>/dev/null + +if [ "$(/opt/eff.org/certbot/venv/bin/python -V 2>&1 | cut -d" " -f2 | cut -d. -f1-2)" != "3.6" ]; then + echo "ERROR: Script certbot-auto failed to bootstrap and install Python 3.6 while SCL is available." + exit 1 +fi + +if ! /opt/eff.org/certbot/venv/bin/certbot --version > /dev/null 2> /dev/null; then + echo "ERROR: Script certbot-auto did not install certbot correctly while SCL is enabled." + exit 1 +fi + +echo "PASSED: Script certbot-auto correctly bootstraped Certbot using rh-python36 when SCL is available." + +# Expect certbot-auto will be totally silent now that everything has been correctly boostraped. +OUTPUT_LEN=$("$LE_AUTO" --install-only --no-self-upgrade --quiet 2>&1 | wc -c) +if [ "$OUTPUT_LEN" != 0 ]; then + echo certbot-auto produced unexpected output! + exit 1 +fi + +echo "PASSED: Script certbot-auto did not print anything in quiet mode." diff --git a/letsencrypt-auto-source/tests/uname_wrapper.sh b/letsencrypt-auto-source/tests/uname_wrapper.sh new file mode 100644 index 000000000..df1f568c6 --- /dev/null +++ b/letsencrypt-auto-source/tests/uname_wrapper.sh @@ -0,0 +1,10 @@ +#!/bin/bash +set -e + +uname_output=$(/bin/uname_orig "$@") + +if [ "$UNAME_FAKE_32BITS" = true ]; then + uname_output="${uname_output//x86_64/i686}" +fi + +echo "$uname_output" diff --git a/letsencrypt-auto-source/version.py b/letsencrypt-auto-source/version.py index c49d96654..d70ffefac 100755 --- a/letsencrypt-auto-source/version.py +++ b/letsencrypt-auto-source/version.py @@ -14,6 +14,7 @@ def certbot_version(build_script_dir): """Return the version number stamped in certbot/__init__.py.""" return re.search('''^__version__ = ['"](.+)['"].*''', file_contents(join(dirname(build_script_dir), + 'certbot', 'certbot', '__init__.py')), re.M).group(1) diff --git a/letshelp-certbot/docs/conf.py b/letshelp-certbot/docs/conf.py index 17d8b3ea9..fc482a348 100644 --- a/letshelp-certbot/docs/conf.py +++ b/letshelp-certbot/docs/conf.py @@ -12,10 +12,9 @@ # All configuration values have a default; values that are commented out # serve to show the default. -import sys import os import shlex - +import sys here = os.path.abspath(os.path.dirname(__file__)) @@ -41,7 +40,7 @@ extensions = [ ] autodoc_member_order = 'bysource' -autodoc_default_flags = ['show-inheritance', 'private-members'] +autodoc_default_flags = ['show-inheritance'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] diff --git a/letshelp-certbot/letshelp_certbot/apache.py b/letshelp-certbot/letshelp_certbot/apache.py index 50f3c5ef6..ebe4e3671 100755 --- a/letshelp-certbot/letshelp_certbot/apache.py +++ b/letshelp-certbot/letshelp_certbot/apache.py @@ -74,7 +74,7 @@ def make_and_verify_selection(server_root, temp_dir): ans = six.moves.input("(Y)es/(N)o: ").lower() if ans.startswith("y"): return - elif ans.startswith("n"): + if ans.startswith("n"): sys.exit("Your files were not submitted") @@ -159,7 +159,7 @@ def safe_config_file(config_file): empty_or_all_comments = False if line.startswith("-----BEGIN"): return False - elif ":" not in line: + if ":" not in line: possible_password_file = False # If file isn't empty or commented out and could be a password file, # don't include it in selection. It is safe to include the file if diff --git a/letshelp-certbot/letshelp_certbot/apache_test.py b/letshelp-certbot/letshelp_certbot/apache_test.py index a1115bc06..0853046b4 100644 --- a/letshelp-certbot/letshelp_certbot/apache_test.py +++ b/letshelp-certbot/letshelp_certbot/apache_test.py @@ -2,19 +2,18 @@ import argparse import functools import os -import pkg_resources import subprocess import tarfile import tempfile import unittest -import mock # six is used in mock.patch() +import mock +import pkg_resources import six # pylint: disable=unused-import import letshelp_certbot.apache as letshelp_le_apache - _PARTIAL_CONF_PATH = os.path.join("mods-available", "ssl.load") _PARTIAL_LINK_PATH = os.path.join("mods-enabled", "ssl.load") _CONFIG_FILE = pkg_resources.resource_filename( diff --git a/letshelp-certbot/letshelp_certbot/magic_typing.py b/letshelp-certbot/letshelp_certbot/magic_typing.py index 471b8dfa9..5a6358c69 100644 --- a/letshelp-certbot/letshelp_certbot/magic_typing.py +++ b/letshelp-certbot/letshelp_certbot/magic_typing.py @@ -1,6 +1,7 @@ """Shim class to not have to depend on typing module in prod.""" import sys + class TypingClass(object): """Ignore import errors by getting anything""" def __getattr__(self, name): diff --git a/letshelp-certbot/readthedocs.org.requirements.txt b/letshelp-certbot/readthedocs.org.requirements.txt index 7858b312f..b24681caa 100644 --- a/letshelp-certbot/readthedocs.org.requirements.txt +++ b/letshelp-certbot/readthedocs.org.requirements.txt @@ -1,10 +1,10 @@ # readthedocs.org gives no way to change the install command to "pip -# install -e .[docs]" (that would in turn install documentation +# install -e certbot[docs]" (that would in turn install documentation # dependencies), but it allows to specify a requirements.txt file at # https://readthedocs.org/dashboard/letsencrypt/advanced/ (c.f. #259) # Although ReadTheDocs certainly doesn't need to install the project # in --editable mode (-e), just "pip install .[docs]" does not work as -# expected and "pip install -e .[docs]" must be used instead +# expected and "pip install -e certbot[docs]" must be used instead -e letshelp-certbot[docs] diff --git a/letshelp-certbot/setup.py b/letshelp-certbot/setup.py index 3e9e31725..448c145ce 100644 --- a/letshelp-certbot/setup.py +++ b/letshelp-certbot/setup.py @@ -1,6 +1,5 @@ -from setuptools import setup from setuptools import find_packages - +from setuptools import setup version = '0.7.0.dev0' @@ -22,7 +21,7 @@ setup( author="Certbot Project", author_email='client-dev@letsencrypt.org', license='Apache License 2.0', - python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', + python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', classifiers=[ 'Development Status :: 3 - Alpha', 'Intended Audience :: System Administrators', @@ -32,10 +31,10 @@ setup( 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', - 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Topic :: Internet :: WWW/HTTP', 'Topic :: Security', 'Topic :: System :: Installation/Setup', diff --git a/linter_plugin.py b/linter_plugin.py index 4938755cf..1754b1a2a 100644 --- a/linter_plugin.py +++ b/linter_plugin.py @@ -1,29 +1,52 @@ -"""Certbot ACME PyLint plugin. - -http://docs.pylint.org/plugins.html - """ -from astroid import MANAGER -from astroid import nodes +Certbot PyLint plugin. + +The built-in ImportChecker of Pylint does a similar job to ForbidStandardOsModule to detect +deprecated modules. You can check its behavior as a reference to what is coded here. +See https://github.com/PyCQA/pylint/blob/b20a2984c94e2946669d727dbda78735882bf50a/pylint/checkers/imports.py#L287 +See http://docs.pylint.org/plugins.html +""" +from pylint.checkers import BaseChecker +from pylint.interfaces import IAstroidChecker + +# Modules in theses packages can import the os module. +WHITELIST_PACKAGES = ['acme', 'certbot_compatibility_test', 'letshelp_certbot', 'lock_test'] -def register(unused_linter): - """Register this module as PyLint plugin.""" +class ForbidStandardOsModule(BaseChecker): + """ + This checker ensures that standard os module (and submodules) is not imported by certbot + modules. Otherwise an 'os-module-forbidden' error will be registered for the faulty lines. + """ + __implements__ = IAstroidChecker -def _transform(cls): - # fix the "no-member" error on instances of - # letsencrypt.acme.util.ImmutableMap subclasses (instance - # attributes are initialized dynamically based on __slots__) + name = 'forbid-os-module' + msgs = { + 'E5001': ( + 'Forbidden use of os module, certbot.compat.os must be used instead', + 'os-module-forbidden', + 'Some methods from the standard os module cannot be used for security reasons on Windows: ' + 'the safe wrapper certbot.compat.os must be used instead in Certbot.' + ) + } + priority = -1 - # TODO: this is too broad and applies to any tested class... + def visit_import(self, node): + os_used = any(name for name in node.names if name[0] == 'os' or name[0].startswith('os.')) + if os_used and not _check_disabled(node): + self.add_message('os-module-forbidden', node=node) - if cls.slots() is not None: - for slot in cls.slots(): - cls.locals[slot.value] = [nodes.EmptyNode()] - - if cls.name == 'JSONObjectWithFields': - # _fields is magically introduced by JSONObjectWithFieldsMeta - cls.locals['_fields'] = [nodes.EmptyNode()] + def visit_importfrom(self, node): + if node.modname == 'os' or node.modname.startswith('os.') and not _check_disabled(node): + self.add_message('os-module-forbidden', node=node) -MANAGER.register_transform(nodes.Class, _transform) +def register(linter): + """Pylint hook to auto-register this linter""" + linter.register_checker(ForbidStandardOsModule(linter)) + + +def _check_disabled(node): + module = node.root() + return any(package for package in WHITELIST_PACKAGES + if module.name.startswith(package + '.') or module.name == package) diff --git a/local-oldest-requirements.txt b/local-oldest-requirements.txt deleted file mode 100644 index d582d5c65..000000000 --- a/local-oldest-requirements.txt +++ /dev/null @@ -1 +0,0 @@ -acme[dev]==0.29.0 diff --git a/pull_request_template.md b/pull_request_template.md index 60fd6da7e..c806d33e8 100644 --- a/pull_request_template.md +++ b/pull_request_template.md @@ -1,3 +1,4 @@ -Be sure to edit the `master` section of `CHANGELOG.md`. This includes a -description of the change and ensuring the modified package(s) are listed as -having been changed. +## Pull Request Checklist + +- [ ] If the change being made is to a [distributed component](https://certbot.eff.org/docs/contributing.html#code-components-and-layout), edit the `master` section of `certbot/CHANGELOG.md` to include a description of the change being made. +- [ ] Include your name in `AUTHORS.md` if you like. diff --git a/pytest.ini b/pytest.ini index 49db7da09..e09813e52 100644 --- a/pytest.ini +++ b/pytest.ini @@ -4,15 +4,6 @@ [pytest] # In general, all warnings are treated as errors. Here are the exceptions: # 1- decodestring: https://github.com/rthalley/dnspython/issues/338 -# 2- ignore our own TLS-SNI-01 warning -# 3- ignore warn for importing abstract classes from collections instead of collections.abc, -# too much third party dependencies are still relying on this behavior, -# but it should be corrected to allow Certbot compatiblity with Python >= 3.8 -# 4- ipdb uses deprecated functionality of IPython. See -# https://github.com/gotcha/ipdb/issues/144. filterwarnings = error ignore:decodestring:DeprecationWarning - ignore:TLS-SNI-01:DeprecationWarning - ignore:.*collections\.abc:DeprecationWarning - ignore:The `color_scheme` argument is deprecated:DeprecationWarning:IPython.* diff --git a/tests/boulder-fetch.sh b/tests/boulder-fetch.sh deleted file mode 100755 index a06d37325..000000000 --- a/tests/boulder-fetch.sh +++ /dev/null @@ -1,38 +0,0 @@ -#!/bin/bash -# Download and run Boulder instance for integration testing -set -xe - -# Clone Boulder into a GOPATH-style directory structure even if Go isn't -# installed, because Boulder's docker-compose.yml file wll look for it there. -export GOPATH=${GOPATH:-$HOME/gopath} -BOULDERPATH=${BOULDERPATH:-$GOPATH/src/github.com/letsencrypt/boulder} -if [ ! -d ${BOULDERPATH} ]; then - git clone --depth=1 https://github.com/letsencrypt/boulder ${BOULDERPATH} -fi - -cd ${BOULDERPATH} - -# Since https://github.com/letsencrypt/boulder/commit/92e8e1708a725e9d08a5da2f4a7132320ed2158b, -# Boulder support for tls-sni-01 challenges is disabled. We still need to support it until this -# challenge is officially removed from ACME CA server on production, and also removed from Certbot. -# This sed command reactivate tls-sni-01 challenges inplace temporarily. -sed -i 's/tls-alpn-01/tls-sni-01/g' test/config/ra.json - -docker-compose up -d boulder - -set +x # reduce verbosity while waiting for boulder -for n in `seq 1 150` ; do - if curl http://localhost:4000/directory 2>/dev/null; then - break - else - sleep 1 - fi -done - -if ! curl http://localhost:4000/directory 2>/dev/null; then - echo "timed out waiting for boulder to start" - exit 1 -fi - -# Setup the DNS resolution used by boulder instance to docker host -curl -X POST -d '{"ip":"10.77.77.1"}' http://localhost:8055/set-default-ipv4 diff --git a/tests/boulder-integration.sh b/tests/boulder-integration.sh deleted file mode 100755 index 3e16fcbbc..000000000 --- a/tests/boulder-integration.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash - -set -e - -if [ "$INTEGRATION_TEST" = "certbot" ]; then - tests/certbot-boulder-integration.sh -elif [ "$INTEGRATION_TEST" = "nginx" ]; then - certbot-nginx/tests/boulder-integration.sh -else - tests/certbot-boulder-integration.sh - # Most CI systems set this variable to true. - # If the tests are running as part of CI, Nginx should be available. - if ${CI:-false} || type nginx; then - certbot-nginx/tests/boulder-integration.sh - fi -fi diff --git a/tests/certbot-boulder-integration.sh b/tests/certbot-boulder-integration.sh deleted file mode 100755 index 630571148..000000000 --- a/tests/certbot-boulder-integration.sh +++ /dev/null @@ -1,529 +0,0 @@ -#!/bin/bash -# Simple integration test. Make sure to activate virtualenv beforehand -# (source venv/bin/activate) and that you are running Boulder test -# instance (see ./boulder-fetch.sh). -# -# Environment variables: -# SERVER: Passed as "certbot --server" argument. -# -# Note: this script is called by Boulder integration test suite! - -set -eux - -# Check that python executable is available in the PATH. Fail immediatly if not. -command -v python > /dev/null || (echo "Error, python executable is not in the PATH" && exit 1) - -. ./tests/integration/_common.sh -export PATH="$PATH:/usr/sbin" # /usr/sbin/nginx - -cleanup_and_exit() { - EXIT_STATUS=$? - if SERVER_STILL_RUNNING=`ps -p $python_server_pid -o pid=` - then - echo Kill server subprocess, left running by abnormal exit - kill $SERVER_STILL_RUNNING - fi - if [ -f "$HOOK_DIRS_TEST" ]; then - rm -f "$HOOK_DIRS_TEST" - fi - exit $EXIT_STATUS -} - -trap cleanup_and_exit EXIT - -export HOOK_DIRS_TEST="$(mktemp)" -renewal_hooks_root="$config_dir/renewal-hooks" -renewal_hooks_dirs=$(echo "$renewal_hooks_root/"{pre,deploy,post}) -renewal_dir_pre_hook="$(echo $renewal_hooks_dirs | cut -f 1 -d " ")/hook.sh" -renewal_dir_deploy_hook="$(echo $renewal_hooks_dirs | cut -f 2 -d " ")/hook.sh" -renewal_dir_post_hook="$(echo $renewal_hooks_dirs | cut -f 3 -d " ")/hook.sh" - -# Creates hooks in Certbot's renewal hook directory that write to a file -CreateDirHooks() { - for hook_dir in $renewal_hooks_dirs; do - mkdir -p $hook_dir - hook_path="$hook_dir/hook.sh" - cat << EOF > "$hook_path" -#!/bin/bash -xe -if [ "\$0" = "$renewal_dir_deploy_hook" ]; then - if [ -z "\$RENEWED_DOMAINS" -o -z "\$RENEWED_LINEAGE" ]; then - echo "Environment variables not properly set!" >&2 - exit 1 - fi -fi -echo \$(basename \$(dirname "\$0")) >> "\$HOOK_DIRS_TEST" -EOF - chmod +x "$hook_path" - done -} - -# Asserts that the hooks created by CreateDirHooks have been run once and -# resets the file. -# -# Arguments: -# The number of times the deploy hook should have been run. (It should run -# once for each certificate that was issued in that run of Certbot.) -CheckDirHooks() { - expected="pre\n" - for ((i=0; i<$1; i++)); do - expected=$expected"deploy\n" - done - expected=$expected"post" - - if ! diff "$HOOK_DIRS_TEST" <(echo -e "$expected"); then - echo "Unexpected directory hook output!" >&2 - echo "Expected:" >&2 - echo -e "$expected" >&2 - echo "Got:" >&2 - cat "$HOOK_DIRS_TEST" >&2 - exit 1 - fi - - rm -f "$HOOK_DIRS_TEST" - export HOOK_DIRS_TEST="$(mktemp)" -} - -common_no_force_renew() { - certbot_test_no_force_renew \ - --authenticator standalone \ - --installer null \ - "$@" -} - -common() { - common_no_force_renew \ - --renew-by-default \ - "$@" -} - -export HOOK_TEST="/tmp/hook$$" -CheckHooks() { - if [ $(head -n1 "$HOOK_TEST") = "wtf.pre" ]; then - expected="wtf.pre\ndeploy\n" - if [ $(sed '3q;d' "$HOOK_TEST") = "deploy" ]; then - expected=$expected"deploy\nwtf2.pre\n" - else - expected=$expected"wtf2.pre\ndeploy\n" - fi - expected=$expected"deploy\ndeploy\nwtf.post\nwtf2.post" - else - expected="wtf2.pre\ndeploy\n" - if [ $(sed '3q;d' "$HOOK_TEST") = "deploy" ]; then - expected=$expected"deploy\nwtf.pre\n" - else - expected=$expected"wtf.pre\ndeploy\n" - fi - expected=$expected"deploy\ndeploy\nwtf2.post\nwtf.post" - fi - - if ! cmp --quiet <(echo -e "$expected") "$HOOK_TEST" ; then - echo Hooks did not run as expected\; got >&2 - cat "$HOOK_TEST" >&2 - echo -e "Expected\n$expected" >&2 - rm "$HOOK_TEST" - exit 1 - fi - rm "$HOOK_TEST" -} - -# Checks if deploy is in the hook output and deletes the file -DeployInHookOutput() { - CONTENTS=$(cat "$HOOK_TEST") - rm "$HOOK_TEST" - grep deploy <(echo "$CONTENTS") -} - -# Asserts that there is a saved renew_hook for a lineage. -# -# Arguments: -# Name of lineage to check -CheckSavedRenewHook() { - if ! grep renew_hook "$config_dir/renewal/$1.conf"; then - echo "Hook wasn't saved as renew_hook" >&2 - exit 1 - fi -} - -# Asserts the deploy hook was properly run and saved and deletes the hook file -# -# Arguments: -# Lineage name of the issued cert -CheckDeployHook() { - if ! DeployInHookOutput; then - echo "The deploy hook wasn't run" >&2 - exit 1 - fi - CheckSavedRenewHook $1 -} - -# Asserts the renew hook wasn't run but was saved and deletes the hook file -# -# Arguments: -# Lineage name of the issued cert -# Asserts the deploy hook wasn't run and deletes the hook file -CheckRenewHook() { - if DeployInHookOutput; then - echo "The renew hook was incorrectly run" >&2 - exit 1 - fi - CheckSavedRenewHook $1 -} - -# Return success only if input contains exactly $1 lines of text, of -# which $2 different values occur in the first field. -TotalAndDistinctLines() { - total=$1 - distinct=$2 - awk '{a[$1] = 1}; END {n = 0; for (i in a) { n++ }; exit(NR !='$total' || n !='$distinct')}' -} - -# Cleanup coverage data -coverage erase - -# test for regressions of #4719 -get_num_tmp_files() { - ls -1 /tmp | wc -l -} -num_tmp_files=$(get_num_tmp_files) -common --csr / > /dev/null && echo expected error && exit 1 || true -common --help > /dev/null -common --help all > /dev/null -common --version > /dev/null -if [ $(get_num_tmp_files) -ne $num_tmp_files ]; then - echo "New files or directories created in /tmp!" - exit 1 -fi -CreateDirHooks - -common register -for dir in $renewal_hooks_dirs; do - if [ ! -d "$dir" ]; then - echo "Hook directory not created by Certbot!" >&2 - exit 1 - fi -done - -common unregister - -common register --email ex1@domain.org,ex2@domain.org - -# TODO: When `certbot register --update-registration` is fully deprecated, delete the two following deprecated uses - -common register --update-registration --email ex1@domain.org - -common register --update-registration --email ex1@domain.org,ex2@domain.org - -common update_account --email example@domain.org - -common update_account --email ex1@domain.org,ex2@domain.org - -common plugins --init --prepare | grep webroot - -# We start a server listening on the port for the -# unrequested challenge to prevent regressions in #3601. -python ./tests/run_http_server.py $http_01_port & -python_server_pid=$! - -certname="le1.wtf" -common --domains le1.wtf --preferred-challenges tls-sni-01 auth \ - --cert-name $certname \ - --pre-hook 'echo wtf.pre >> "$HOOK_TEST"' \ - --post-hook 'echo wtf.post >> "$HOOK_TEST"'\ - --deploy-hook 'echo deploy >> "$HOOK_TEST"' -kill $python_server_pid -CheckDeployHook $certname - -python ./tests/run_http_server.py $tls_sni_01_port & -python_server_pid=$! -certname="le2.wtf" -common --domains le2.wtf --preferred-challenges http-01 run \ - --cert-name $certname \ - --pre-hook 'echo wtf.pre >> "$HOOK_TEST"' \ - --post-hook 'echo wtf.post >> "$HOOK_TEST"'\ - --deploy-hook 'echo deploy >> "$HOOK_TEST"' -kill $python_server_pid -CheckDeployHook $certname - -certname="le.wtf" -common certonly -a manual -d le.wtf --rsa-key-size 4096 --cert-name $certname \ - --manual-auth-hook ./tests/manual-http-auth.sh \ - --manual-cleanup-hook ./tests/manual-http-cleanup.sh \ - --pre-hook 'echo wtf2.pre >> "$HOOK_TEST"' \ - --post-hook 'echo wtf2.post >> "$HOOK_TEST"' \ - --renew-hook 'echo deploy >> "$HOOK_TEST"' -CheckRenewHook $certname - -certname="dns.le.wtf" -common -a manual -d dns.le.wtf --preferred-challenges dns,tls-sni run \ - --cert-name $certname \ - --manual-auth-hook ./tests/manual-dns-auth.sh \ - --manual-cleanup-hook ./tests/manual-dns-cleanup.sh \ - --pre-hook 'echo wtf2.pre >> "$HOOK_TEST"' \ - --post-hook 'echo wtf2.post >> "$HOOK_TEST"' \ - --renew-hook 'echo deploy >> "$HOOK_TEST"' -CheckRenewHook $certname - -common certonly --cert-name newname -d newname.le.wtf - -export CSR_PATH="${root}/csr.der" KEY_PATH="${root}/key.pem" \ - OPENSSL_CNF=examples/openssl.cnf -./examples/generate-csr.sh le3.wtf -common auth --csr "$CSR_PATH" \ - --cert-path "${root}/csr/cert.pem" \ - --chain-path "${root}/csr/chain.pem" -openssl x509 -in "${root}/csr/cert.pem" -text -openssl x509 -in "${root}/csr/chain.pem" -text - -common --domains le3.wtf install \ - --cert-path "${root}/csr/cert.pem" \ - --key-path "${root}/key.pem" - -CheckCertCount() { - CERTCOUNT=`ls "${root}/conf/archive/$1/cert"* | wc -l` - if [ "$CERTCOUNT" -ne "$2" ] ; then - echo Wrong cert count, not "$2" `ls "${root}/conf/archive/$1/"*` - exit 1 - fi -} - -CheckPermissions() { -# Args: -# Checks mode of two files match under - masked_mode() { echo $((0`stat -c %a $1` & 0$2)); } - if [ `masked_mode $1 $3` -ne `masked_mode $2 $3` ] ; then - echo "With $3 mask, expected mode `masked_mode $1 $3`, got `masked_mode $2 $3` on file $2" - exit 1 - fi -} - -CheckGID() { -# Args: -# Checks group owner of two files match - group_owner() { echo `stat -c %G $1`; } - if [ `group_owner $1` != `group_owner $2` ] ; then - echo "Expected group owner `group_owner $1`, got `group_owner $2` on file $2" - exit 1 - fi -} - -CheckOthersPermission() { -# Args: -# Tests file's other/world permission against expected mode - other_permission=$((0`stat -c %a $1` & 07)) - if [ $other_permission -ne $2 ] ; then - echo "Expected file $1 to have others mode $2, got $other_permission instead" - exit 1 - fi -} - -CheckCertCount "le.wtf" 1 - -# This won't renew (because it's not time yet) -common_no_force_renew renew -CheckCertCount "le.wtf" 1 -if [ -s "$HOOK_DIRS_TEST" ]; then - echo "Directory hooks were executed for non-renewal!" >&2; - exit 1 -fi - -rm -rf "$renewal_hooks_root" -# renew using HTTP manual auth hooks -common renew --cert-name le.wtf --authenticator manual -CheckCertCount "le.wtf" 2 - -CheckOthersPermission "${root}/conf/archive/le.wtf/privkey1.pem" 0 -CheckOthersPermission "${root}/conf/archive/le.wtf/privkey2.pem" 0 -CheckPermissions "${root}/conf/archive/le.wtf/privkey1.pem" "${root}/conf/archive/le.wtf/privkey2.pem" 074 -CheckGID "${root}/conf/archive/le.wtf/privkey1.pem" "${root}/conf/archive/le.wtf/privkey2.pem" -chmod 0444 "${root}/conf/archive/le.wtf/privkey2.pem" - -# test renewal with no executables in hook directories -for hook_dir in $renewal_hooks_dirs; do - touch "$hook_dir/file" - mkdir "$hook_dir/dir" -done -# renew using DNS manual auth hooks -common renew --cert-name dns.le.wtf --authenticator manual -CheckCertCount "dns.le.wtf" 2 - -# test with disabled directory hooks -rm -rf "$renewal_hooks_root" -CreateDirHooks -# This will renew because the expiry is less than 10 years from now -sed -i "4arenew_before_expiry = 4 years" "$root/conf/renewal/le.wtf.conf" -common_no_force_renew renew --rsa-key-size 2048 --no-directory-hooks -CheckCertCount "le.wtf" 3 -CheckGID "${root}/conf/archive/le.wtf/privkey2.pem" "${root}/conf/archive/le.wtf/privkey3.pem" -CheckPermissions "${root}/conf/archive/le.wtf/privkey2.pem" "${root}/conf/archive/le.wtf/privkey3.pem" 074 -CheckOthersPermission "${root}/conf/archive/le.wtf/privkey3.pem" 04 - -if [ -s "$HOOK_DIRS_TEST" ]; then - echo "Directory hooks were executed with --no-directory-hooks!" >&2 - exit 1 -fi - -# The 4096 bit setting should persist to the first renewal, but be overridden in the second - -size1=`wc -c ${root}/conf/archive/le.wtf/privkey1.pem | cut -d" " -f1` -size2=`wc -c ${root}/conf/archive/le.wtf/privkey2.pem | cut -d" " -f1` -size3=`wc -c ${root}/conf/archive/le.wtf/privkey3.pem | cut -d" " -f1` -# 4096 bit PEM keys are about ~3270 bytes, 2048 ones are about 1700 bytes -if [ "$size1" -lt 3000 ] || [ "$size2" -lt 3000 ] || [ "$size3" -gt 1800 ] ; then - echo key sizes violate assumptions: - ls -l "${root}/conf/archive/le.wtf/privkey"* - exit 1 -fi - -# --renew-by-default is used, so renewal should occur -[ -f "$HOOK_TEST" ] && rm -f "$HOOK_TEST" -common renew -CheckCertCount "le.wtf" 4 -CheckHooks -CheckDirHooks 5 - -# test with overlapping directory hooks on the command line -common renew --cert-name le2.wtf \ - --pre-hook "$renewal_dir_pre_hook" \ - --deploy-hook "$renewal_dir_deploy_hook" \ - --post-hook "$renewal_dir_post_hook" -CheckDirHooks 1 - -# test with overlapping directory hooks in the renewal conf files -common renew --cert-name le2.wtf -CheckDirHooks 1 - -# manual-dns-auth.sh will skip completing the challenge for domains that begin -# with fail. -common -a manual -d dns1.le.wtf,fail.dns1.le.wtf \ - --allow-subset-of-names \ - --preferred-challenges dns,tls-sni \ - --manual-auth-hook ./tests/manual-dns-auth.sh \ - --manual-cleanup-hook ./tests/manual-dns-cleanup.sh - -if common certificates | grep "fail\.dns1\.le\.wtf"; then - echo "certificate should not have been issued for domain!" >&2 - exit 1 -fi - -# reuse-key -common --domains reusekey.le.wtf --reuse-key -common renew --cert-name reusekey.le.wtf -CheckCertCount "reusekey.le.wtf" 2 -ls -l "${root}/conf/archive/reusekey.le.wtf/privkey"* -# The final awk command here exits successfully if its input consists of -# exactly two lines with identical first fields, and unsuccessfully otherwise. -sha256sum "${root}/conf/archive/reusekey.le.wtf/privkey"* | TotalAndDistinctLines 2 1 - -# don't reuse key (just by forcing reissuance without --reuse-key) -common --cert-name reusekey.le.wtf --domains reusekey.le.wtf --force-renewal -CheckCertCount "reusekey.le.wtf" 3 -ls -l "${root}/conf/archive/reusekey.le.wtf/privkey"* -# Exactly three lines, of which exactly two identical first fields. -sha256sum "${root}/conf/archive/reusekey.le.wtf/privkey"* | TotalAndDistinctLines 3 2 - -# Nonetheless, all three certificates are different even though two of them -# share the same subject key. -sha256sum "${root}/conf/archive/reusekey.le.wtf/cert"* | TotalAndDistinctLines 3 3 - -# ECDSA -openssl ecparam -genkey -name secp384r1 -out "${root}/privkey-p384.pem" -SAN="DNS:ecdsa.le.wtf" openssl req -new -sha256 \ - -config "${OPENSSL_CNF:-openssl.cnf}" \ - -key "${root}/privkey-p384.pem" \ - -subj "/" \ - -reqexts san \ - -outform der \ - -out "${root}/csr-p384.der" -common auth --csr "${root}/csr-p384.der" \ - --cert-path "${root}/csr/cert-p384.pem" \ - --chain-path "${root}/csr/chain-p384.pem" -openssl x509 -in "${root}/csr/cert-p384.pem" -text | grep 'ASN1 OID: secp384r1' - -# OCSP Must Staple -common auth --must-staple --domains "must-staple.le.wtf" -openssl x509 -in "${root}/conf/live/must-staple.le.wtf/cert.pem" -text | grep -E 'status_request|1\.3\.6\.1\.5\.5\.7\.1\.24' - -# revoke by account key -common revoke --cert-path "$root/conf/live/le.wtf/cert.pem" --delete-after-revoke -# revoke renewed -common revoke --cert-path "$root/conf/live/le1.wtf/cert.pem" --no-delete-after-revoke -if [ ! -d "$root/conf/live/le1.wtf" ]; then - echo "cert deleted when --no-delete-after-revoke was used!" - exit 1 -fi -common delete --cert-name le1.wtf -# revoke by cert key -common revoke --cert-path "$root/conf/live/le2.wtf/cert.pem" \ - --key-path "$root/conf/live/le2.wtf/privkey.pem" - -# Get new certs to test revoke with a reason, by account and by cert key -common --domains le1.wtf -common revoke --cert-path "$root/conf/live/le1.wtf/cert.pem" \ - --reason cessationOfOperation -common --domains le2.wtf -common revoke --cert-path "$root/conf/live/le2.wtf/cert.pem" \ - --key-path "$root/conf/live/le2.wtf/privkey.pem" \ - --reason keyCompromise - -common unregister - -out=$(common certificates) -subdomains="le dns.le newname.le must-staple.le" -for subdomain in $subdomains; do - domain="$subdomain.wtf" - if ! echo $out | grep "$domain"; then - echo "$domain not in certificates output!" - exit 1; - fi -done - -# Testing that revocation also deletes by default -subdomains="le1 le2" -for subdomain in $subdomains; do - domain="$subdomain.wtf" - if echo $out | grep "$domain"; then - echo "Revoked $domain in certificates output! Should not be!" - exit 1; - fi -done - -# Test that revocation raises correct error when both --cert-name and --cert-path specified -common --domains le1.wtf -out=$(common revoke --cert-path "$root/conf/live/le1.wtf/fullchain.pem" --cert-name "le1.wtf" 2>&1) || true -if ! echo $out | grep "Exactly one of --cert-path or --cert-name must be specified"; then - echo "Non-interactive revoking with both --cert-name and --cert-path " - echo "did not raise the correct error!" - exit 1 -fi - -# Test that revocation doesn't delete if multiple lineages share an archive dir -common --domains le1.wtf -common --domains le2.wtf -sed -i "s|^archive_dir = .*$|archive_dir = $root/conf/archive/le1.wtf|" "$root/conf/renewal/le2.wtf.conf" -#common update_symlinks # not needed, but a bit more context for what this test is about -out=$(common revoke --cert-path "$root/conf/live/le1.wtf/cert.pem") -if ! echo $out | grep "Not deleting revoked certs due to overlapping archive dirs"; then - echo "Deleted a cert that had an overlapping archive dir with another lineage!" - exit 1 -fi - -cert_name="must-staple.le.wtf" -common delete --cert-name $cert_name -archive="$root/conf/archive/$cert_name" -conf="$root/conf/renewal/$cert_name.conf" -live="$root/conf/live/$cert_name" -for path in $archive $conf $live; do - if [ -e $path ]; then - echo "Lineage not properly deleted!" - exit 1 - fi -done - -# Test ACMEv2-only features -if [ "${BOULDER_INTEGRATION:-v1}" = "v2" ]; then - common -a manual -d '*.le4.wtf,le4.wtf' --preferred-challenges dns \ - --manual-auth-hook ./tests/manual-dns-auth.sh \ - --manual-cleanup-hook ./tests/manual-dns-cleanup.sh -fi - -coverage report --fail-under 64 --include 'certbot/*' --show-missing diff --git a/tests/certbot-pebble-integration.sh b/tests/certbot-pebble-integration.sh deleted file mode 100755 index 8711f72c1..000000000 --- a/tests/certbot-pebble-integration.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash -# Simple integration test. Make sure to activate virtualenv beforehand -# (source venv/bin/activate) and that you are running Pebble test -# instance (see ./pebble-fetch.sh). - -cleanup_and_exit() { - EXIT_STATUS=$? - unset SERVER - exit $EXIT_STATUS -} - -trap cleanup_and_exit EXIT - -export SERVER=https://localhost:14000/dir - -./tests/certbot-boulder-integration.sh diff --git a/tests/display.py b/tests/display.py deleted file mode 100644 index 1f548e33d..000000000 --- a/tests/display.py +++ /dev/null @@ -1,22 +0,0 @@ -"""Manual test of display functions.""" -import sys - -from certbot.display import util -from certbot.tests.display import util_test - - -def test_visual(displayer, choices): - """Visually test all of the display functions.""" - displayer.notification("Random notification!") - displayer.menu("Question?", choices, - ok_label="O", cancel_label="Can", help_label="??") - displayer.menu("Question?", [choice[1] for choice in choices], - ok_label="O", cancel_label="Can", help_label="??") - displayer.input("Input Message") - displayer.yesno("YesNo Message", yes_label="Yessir", no_label="Nosir") - displayer.checklist("Checklist Message", [choice[0] for choice in choices]) - - -if __name__ == "__main__": - displayer = util.FileDisplay(sys.stdout, False) - test_visual(displayer, util_test.CHOICES) diff --git a/tests/integration/_common.sh b/tests/integration/_common.sh deleted file mode 100755 index 83aa91a9e..000000000 --- a/tests/integration/_common.sh +++ /dev/null @@ -1,74 +0,0 @@ -# The -t is required on macOS. It provides a template file path for -# the kernel to use. -root=${root:-$(mktemp -d -t leitXXXX)} -echo "Root integration tests directory: $root" -config_dir="$root/conf" -tls_sni_01_port=5001 -http_01_port=5002 -sources="acme/,$(ls -dm certbot*/ | tr -d ' \n')" -export root config_dir tls_sni_01_port http_01_port sources -certbot_path="$(command -v certbot)" -# Flags that are added here will be added to Certbot calls within -# certbot_test_no_force_renew. -other_flags="--config-dir $config_dir --work-dir $root/work" -other_flags="$other_flags --logs-dir $root/logs" - -certbot_test () { - certbot_test_no_force_renew \ - --renew-by-default \ - "$@" -} - -# Succeeds if Certbot version is at least the given version number and fails -# otherwise. This is useful for making sure Certbot has certain features -# available. The patch version is currently ignored. -# -# Arguments: -# First argument is the minimum major version -# Second argument is the minimum minor version -version_at_least () { - # Certbot major and minor version (e.g. 0.30) - major_minor=$("$certbot_path" --version 2>&1 | cut -d' ' -f2 | cut -d. -f1,2) - major=$(echo "$major_minor" | cut -d. -f1) - minor=$(echo "$major_minor" | cut -d. -f2) - # Test that either the major version is greater or major version is equal - # and minor version is greater than or equal to. - [ \( "$major" -gt "$1" \) -o \( "$major" -eq "$1" -a "$minor" -ge "$2" \) ] -} - -# Use local ACMEv2 endpoint if requested and SERVER isn't already set. -if [ "${BOULDER_INTEGRATION:-v1}" = "v2" -a -z "${SERVER:+x}" ]; then - SERVER="http://localhost:4001/directory" -fi - -# --no-random-sleep-on-renew was added in -# https://github.com/certbot/certbot/pull/6599 and first released in Certbot -# 0.30.0. -if version_at_least 0 30; then - other_flags="$other_flags --no-random-sleep-on-renew" -fi - -certbot_test_no_force_renew () { - omit_patterns="*/*.egg-info/*,*/dns_common*,*/setup.py,*/test_*,*/tests/*" - omit_patterns="$omit_patterns,*_test.py,*_test_*,certbot-apache/*" - omit_patterns="$omit_patterns,certbot-compatibility-test/*,certbot-dns*/" - omit_patterns="$omit_patterns,certbot-nginx/certbot_nginx/parser_obj.py" - coverage run \ - --append \ - --source $sources \ - --omit $omit_patterns \ - "$certbot_path" \ - --server "${SERVER:-http://localhost:4000/directory}" \ - --no-verify-ssl \ - --tls-sni-01-port $tls_sni_01_port \ - --http-01-port $http_01_port \ - --manual-public-ip-logging-ok \ - $other_flags \ - --non-interactive \ - --no-redirect \ - --agree-tos \ - --register-unsafely-without-email \ - --debug \ - -vv \ - "$@" -} diff --git a/tests/letstest/README.md b/tests/letstest/README.md index 0155065b0..f8a15208e 100644 --- a/tests/letstest/README.md +++ b/tests/letstest/README.md @@ -14,15 +14,17 @@ Simple AWS testfarm scripts for certbot client testing - AWS EC2 has a default limit of 20 t2/t1 instances, if more are needed, they need to be requested via online webform. -## Usage - - To install the necessary dependencies on Ubuntu 16.04, run: +## Installation and configuration +These tests require Python 2.7, awscli, boto3, PyYAML, and fabric<2.0. If you +have Python 2.7 and virtualenv installed, you can use requirements.txt to +create a virtual environment with a known set of dependencies by running: ``` -sudo apt install awscli python-yaml python-boto3 fabric +virtualenv --python $(command -v python2.7 || command -v python2 || command -v python) venv +. ./venv/bin/activate +pip install --requirement requirements.txt ``` - - Requires AWS IAM secrets to be set up with aws cli - - Requires an AWS associated keyfile .pem - +You can then configure AWS credentials and create a key by running: ``` >aws configure --profile [interactive: enter secrets for IAM role] @@ -30,9 +32,10 @@ sudo apt install awscli python-yaml python-boto3 fabric ``` Note: whatever you pick for `` will be shown to other users with AWS access. -When prompted for a default region name, enter: `us-east-1` +When prompted for a default region name, enter: `us-east-1`. -then: +## Usage +To run tests, activate the virtual environment you created above and run: ``` >python multitester.py targets.yaml /path/to/your/key.pem scripts/ ``` diff --git a/tests/letstest/apache2_targets.yaml b/tests/letstest/apache2_targets.yaml index e707b8636..1450a8578 100644 --- a/tests/letstest/apache2_targets.yaml +++ b/tests/letstest/apache2_targets.yaml @@ -1,57 +1,55 @@ targets: #----------------------------------------------------------------------------- - # Apache 2.4 - - ami: ami-26d5af4c - name: ubuntu15.10 + #Ubuntu + - ami: ami-08ab45c4343f5f5c6 + name: ubuntu19.04 type: ubuntu virt: hvm user: ubuntu - - ami: ami-d92e6bb3 - name: ubuntu15.04LTS + - ami: ami-095192256fe1477ad + name: ubuntu18.04LTS type: ubuntu virt: hvm user: ubuntu - - ami: ami-7b89cc11 - name: ubuntu14.04LTS + - ami: ami-09677e0a6b14905b0 + name: ubuntu16.04LTS type: ubuntu virt: hvm user: ubuntu - - ami: ami-9295d0f8 - name: ubuntu14.04LTS_32bit + #----------------------------------------------------------------------------- + # Debian + - ami: ami-01db78123b2b99496 + name: debian10 + type: ubuntu + virt: hvm + user: admin + - ami: ami-003f19e0e687de1cd + name: debian9 + type: ubuntu + virt: hvm + user: admin + - ami: ami-0ed54dd1b25657636 + name: debian9_arm64 + type: ubuntu + virt: hvm + user: admin + machine_type: a1.medium + - ami: ami-077bf3962f29d3fa4 + name: debian8.1 type: ubuntu - virt: pv - user: ubuntu - - ami: ami-116d857a - name: debian8.1 - type: debian virt: hvm user: admin - userdata: | - #cloud-init - runcmd: - - [ apt-get, install, -y, curl ] #----------------------------------------------------------------------------- - # Apache 2.2 - # - ami: ami-0611546c - # name: ubuntu12.04LTS - # type: ubuntu - # virt: hvm - # user: ubuntu - # - ami: ami-e0efab88 - # name: debian7.8.aws.1 - # type: debian - # virt: hvm - # user: admin - # userdata: | - # #cloud-init - # runcmd: - # - [ apt-get, install, -y, curl ] - # - ami: ami-e6eeaa8e - # name: debian7.8.aws.1_32bit - # type: debian - # virt: pv - # user: admin - # userdata: | - # #cloud-init - # runcmd: - # - [ apt-get, install, -y, curl ] \ No newline at end of file + # Fedora + - ami: ami-00bbc6858140f19ed + name: fedora30 + type: centos + virt: hvm + user: fedora + #----------------------------------------------------------------------------- + # CentOS + - ami: ami-9887c6e7 + name: centos7 + type: centos + virt: hvm + user: centos diff --git a/tests/letstest/multitester.py b/tests/letstest/multitester.py index 8babc67b3..9ea9fe76b 100644 --- a/tests/letstest/multitester.py +++ b/tests/letstest/multitester.py @@ -23,7 +23,7 @@ Usage: >aws ec2 create-key-pair --profile HappyHacker --key-name MyKeyPair \ --query 'KeyMaterial' --output text > MyKeyPair.pem then: ->python multitester.py targets.yaml MyKeyPair.pem HappyHacker scripts/test_letsencrypt_auto_venv_only.sh +>python multitester.py targets.yaml MyKeyPair.pem HappyHacker scripts/test_leauto_upgrades.sh see: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html https://docs.aws.amazon.com/cli/latest/userguide/cli-ec2-keypairs.html @@ -32,17 +32,31 @@ see: from __future__ import print_function from __future__ import with_statement -import sys, os, time, argparse, socket, traceback +import argparse import multiprocessing as mp from multiprocessing import Manager +import os +import socket +import sys +import time +import traceback import urllib2 -import yaml + import boto3 from botocore.exceptions import ClientError +import yaml + import fabric -from fabric.api import run, execute, local, env, sudo, cd, lcd -from fabric.operations import get, put +from fabric.api import cd +from fabric.api import env +from fabric.api import execute +from fabric.api import lcd +from fabric.api import local +from fabric.api import run +from fabric.api import sudo from fabric.context_managers import shell_env +from fabric.operations import get +from fabric.operations import put # Command line parser #------------------------------------------------------------------------------- @@ -84,9 +98,6 @@ parser.add_argument('--killboulder', parser.add_argument('--boulderonly', action='store_true', help="only make a boulder server") -parser.add_argument('--fast', - action='store_true', - help="use larger instance types to run faster (saves about a minute, probably not worth it)") cl_args = parser.parse_args() # Credential Variables @@ -94,18 +105,21 @@ cl_args = parser.parse_args() # assumes naming: = .pem KEYFILE = cl_args.key_file KEYNAME = os.path.split(cl_args.key_file)[1].split('.pem')[0] -PROFILE = cl_args.aws_profile +PROFILE = None if cl_args.aws_profile == 'SET_BY_ENV' else cl_args.aws_profile # Globals #------------------------------------------------------------------------------- -BOULDER_AMI = 'ami-5f490b35' # premade shared boulder AMI 14.04LTS us-east-1 -LOGDIR = "" #points to logging / working directory -# boto3/AWS api globals -AWS_SESSION = None -EC2 = None +BOULDER_AMI = 'ami-072a9534772bec854' # premade shared boulder AMI 18.04LTS us-east-1 +LOGDIR = "letest-%d"%int(time.time()) #points to logging / working directory SECURITY_GROUP_NAME = 'certbot-security-group' +SENTINEL = None #queue kill signal SUBNET_NAME = 'certbot-subnet' +class Status(object): + """Possible statuses of client tests.""" + PASS = 'pass' + FAIL = 'fail' + # Boto3/AWS automation functions #------------------------------------------------------------------------------- def should_use_subnet(subnet): @@ -139,16 +153,19 @@ def make_security_group(vpc): mysg.authorize_ingress(IpProtocol="udp", CidrIp="0.0.0.0/0", FromPort=60000, ToPort=61000) return mysg -def make_instance(instance_name, +def make_instance(ec2_client, + instance_name, ami_id, keyname, security_group_id, subnet_id, machine_type='t2.micro', userdata=""): #userdata contains bash or cloud-init script - - new_instance = EC2.create_instances( - BlockDeviceMappings=_get_block_device_mappings(ami_id), + block_device_mappings = _get_block_device_mappings(ec2_client, ami_id) + tags = [{'Key': 'Name', 'Value': instance_name}] + tag_spec = [{'ResourceType': 'instance', 'Tags': tags}] + return ec2_client.create_instances( + BlockDeviceMappings=block_device_mappings, ImageId=ami_id, SecurityGroupIds=[security_group_id], SubnetId=subnet_id, @@ -156,24 +173,10 @@ def make_instance(instance_name, MinCount=1, MaxCount=1, UserData=userdata, - InstanceType=machine_type)[0] + InstanceType=machine_type, + TagSpecifications=tag_spec)[0] - # brief pause to prevent rare error on EC2 delay, should block until ready instead - time.sleep(1.0) - - # give instance a name - try: - new_instance.create_tags(Tags=[{'Key': 'Name', 'Value': instance_name}]) - except ClientError as e: - if "InvalidInstanceID.NotFound" in str(e): - # This seems to be ephemeral... retry - time.sleep(1) - new_instance.create_tags(Tags=[{'Key': 'Name', 'Value': instance_name}]) - else: - raise - return new_instance - -def _get_block_device_mappings(ami_id): +def _get_block_device_mappings(ec2_client, ami_id): """Returns the list of block device mappings to ensure cleanup. This list sets connected EBS volumes to be deleted when the EC2 @@ -186,7 +189,7 @@ def _get_block_device_mappings(ami_id): # * https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-blockdev-template.html return [{'DeviceName': mapping['DeviceName'], 'Ebs': {'DeleteOnTermination': True}} - for mapping in EC2.Image(ami_id).block_device_mappings + for mapping in ec2_client.Image(ami_id).block_device_mappings if not mapping.get('Ebs', {}).get('DeleteOnTermination', True)] @@ -225,20 +228,18 @@ def block_until_ssh_open(ipstring, wait_time=10, timeout=120): def block_until_instance_ready(booting_instance, wait_time=5, extra_wait_time=20): "Blocks booting_instance until AWS EC2 instance is ready to accept SSH connections" - # the reinstantiation from id is necessary to force boto3 - # to correctly update the 'state' variable during init - _id = booting_instance.id - _instance = EC2.Instance(id=_id) - _state = _instance.state['Name'] - _ip = _instance.public_ip_address - while _state != 'running' or _ip is None: + state = booting_instance.state['Name'] + ip = booting_instance.public_ip_address + while state != 'running' or ip is None: time.sleep(wait_time) - _instance = EC2.Instance(id=_id) - _state = _instance.state['Name'] - _ip = _instance.public_ip_address - block_until_ssh_open(_ip) + # The instance needs to be reloaded to update its local attributes. See + # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Instance.reload. + booting_instance.reload() + state = booting_instance.state['Name'] + ip = booting_instance.public_ip_address + block_until_ssh_open(ip) time.sleep(extra_wait_time) - return _instance + return booting_instance # Fabric Routines @@ -290,8 +291,7 @@ def deploy_script(scriptpath, *args): def run_boulder(): with cd('$GOPATH/src/github.com/letsencrypt/boulder'): - run('go run cmd/rabbitmq-setup/main.go -server amqp://localhost') - run('nohup ./start.py >& /dev/null < /dev/null &') + run('sudo docker-compose up -d') def config_and_launch_boulder(instance): execute(deploy_script, 'scripts/boulder_config.sh') @@ -315,53 +315,58 @@ def grab_certbot_log(): sudo('if [ -f ./certbot.log ]; then \ cat ./certbot.log; else echo "[nolocallog]"; fi') -def create_client_instances(targetlist, security_group_id, subnet_id): - "Create a fleet of client instances" - instances = [] - print("Creating instances: ", end="") - for target in targetlist: - if target['virt'] == 'hvm': - machine_type = 't2.medium' if cl_args.fast else 't2.micro' - else: - # 32 bit systems - machine_type = 'c1.medium' if cl_args.fast else 't1.micro' - if 'userdata' in target.keys(): - userdata = target['userdata'] - else: - userdata = '' - name = 'le-%s'%target['name'] - print(name, end=" ") - instances.append(make_instance(name, - target['ami'], - KEYNAME, - machine_type=machine_type, - security_group_id=security_group_id, - subnet_id=subnet_id, - userdata=userdata)) - print() - return instances + +def create_client_instance(ec2_client, target, security_group_id, subnet_id): + """Create a single client instance for running tests.""" + if 'machine_type' in target: + machine_type = target['machine_type'] + elif target['virt'] == 'hvm': + machine_type = 't2.medium' + else: + # 32 bit systems + machine_type = 'c1.medium' + if 'userdata' in target.keys(): + userdata = target['userdata'] + else: + userdata = '' + name = 'le-%s'%target['name'] + print(name, end=" ") + return make_instance(ec2_client, + name, + target['ami'], + KEYNAME, + machine_type=machine_type, + security_group_id=security_group_id, + subnet_id=subnet_id, + userdata=userdata) -def test_client_process(inqueue, outqueue): +def test_client_process(inqueue, outqueue, boulder_url): cur_proc = mp.current_process() for inreq in iter(inqueue.get, SENTINEL): - ii, target = inreq + ii, instance_id, target = inreq + + # Each client process is given its own session due to the suggestion at + # https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html?highlight=multithreading#multithreading-multiprocessing. + aws_session = boto3.session.Session(profile_name=PROFILE) + ec2_client = aws_session.resource('ec2') + instance = ec2_client.Instance(id=instance_id) #save all stdout to log file sys.stdout = open(LOGDIR+'/'+'%d_%s.log'%(ii,target['name']), 'w') print("[%s : client %d %s %s]" % (cur_proc.name, ii, target['ami'], target['name'])) - instances[ii] = block_until_instance_ready(instances[ii]) - print("server %s at %s"%(instances[ii], instances[ii].public_ip_address)) - env.host_string = "%s@%s"%(target['user'], instances[ii].public_ip_address) + instance = block_until_instance_ready(instance) + print("server %s at %s"%(instance, instance.public_ip_address)) + env.host_string = "%s@%s"%(target['user'], instance.public_ip_address) print(env.host_string) try: - install_and_launch_certbot(instances[ii], boulder_url, target) - outqueue.put((ii, target, 'pass')) + install_and_launch_certbot(instance, boulder_url, target) + outqueue.put((ii, target, Status.PASS)) print("%s - %s SUCCESS"%(target['ami'], target['name'])) except: - outqueue.put((ii, target, 'fail')) + outqueue.put((ii, target, Status.FAIL)) print("%s - %s FAIL"%(target['ami'], target['name'])) traceback.print_exc(file=sys.stdout) pass @@ -378,7 +383,10 @@ def test_client_process(inqueue, outqueue): def cleanup(cl_args, instances, targetlist): print('Logs in ', LOGDIR) - if not cl_args.saveinstances: + # If lengths of instances and targetlist aren't equal, instances failed to + # start before running tests so leaving instances running for debugging + # isn't very useful. Let's cleanup after ourselves instead. + if len(instances) != len(targetlist) or not cl_args.saveinstances: print('Terminating EC2 Instances') if cl_args.killboulder: boulder_server.terminate() @@ -392,182 +400,205 @@ def cleanup(cl_args, instances, targetlist): "%s@%s"%(target['user'], instances[ii].public_ip_address)) +def main(): + # Fabric library controlled through global env parameters + env.key_filename = KEYFILE + env.shell = '/bin/bash -l -i -c' + env.connection_attempts = 5 + env.timeout = 10 + # replace default SystemExit thrown by fabric during trouble + class FabricException(Exception): + pass + env['abort_exception'] = FabricException -#------------------------------------------------------------------------------- -# SCRIPT BEGINS -#------------------------------------------------------------------------------- + # Set up local copy of git repo + #------------------------------------------------------------------------------- + print("Making local dir for test repo and logs: %s"%LOGDIR) + local('mkdir %s'%LOGDIR) -# Fabric library controlled through global env parameters -env.key_filename = KEYFILE -env.shell = '/bin/bash -l -i -c' -env.connection_attempts = 5 -env.timeout = 10 -# replace default SystemExit thrown by fabric during trouble -class FabricException(Exception): - pass -env['abort_exception'] = FabricException + # figure out what git object to test and locally create it in LOGDIR + print("Making local git repo") + try: + if cl_args.pull_request != '~': + print('Testing PR %s '%cl_args.pull_request, + "MERGING into master" if cl_args.merge_master else "") + execute(local_git_PR, cl_args.repo, cl_args.pull_request, cl_args.merge_master) + elif cl_args.branch != '~': + print('Testing branch %s of %s'%(cl_args.branch, cl_args.repo)) + execute(local_git_branch, cl_args.repo, cl_args.branch) + else: + print('Testing master of %s'%cl_args.repo) + execute(local_git_clone, cl_args.repo) + except FabricException: + print("FAIL: trouble with git repo") + traceback.print_exc() + exit() -# Set up local copy of git repo -#------------------------------------------------------------------------------- -LOGDIR = "letest-%d"%int(time.time()) -print("Making local dir for test repo and logs: %s"%LOGDIR) -local('mkdir %s'%LOGDIR) -# figure out what git object to test and locally create it in LOGDIR -print("Making local git repo") -try: - if cl_args.pull_request != '~': - print('Testing PR %s '%cl_args.pull_request, - "MERGING into master" if cl_args.merge_master else "") - execute(local_git_PR, cl_args.repo, cl_args.pull_request, cl_args.merge_master) - elif cl_args.branch != '~': - print('Testing branch %s of %s'%(cl_args.branch, cl_args.repo)) - execute(local_git_branch, cl_args.repo, cl_args.branch) + # Set up EC2 instances + #------------------------------------------------------------------------------- + configdata = yaml.load(open(cl_args.config_file, 'r')) + targetlist = configdata['targets'] + print('Testing against these images: [%d total]'%len(targetlist)) + for target in targetlist: + print(target['ami'], target['name']) + + print("Connecting to EC2 using\n profile %s\n keyname %s\n keyfile %s"%(PROFILE, KEYNAME, KEYFILE)) + aws_session = boto3.session.Session(profile_name=PROFILE) + ec2_client = aws_session.resource('ec2') + + print("Determining Subnet") + for subnet in ec2_client.subnets.all(): + if should_use_subnet(subnet): + subnet_id = subnet.id + vpc_id = subnet.vpc.id + break else: - print('Testing master of %s'%cl_args.repo) - execute(local_git_clone, cl_args.repo) -except FabricException: - print("FAIL: trouble with git repo") - traceback.print_exc() - exit() + print("No usable subnet exists!") + print("Please create a VPC with a subnet named {0}".format(SUBNET_NAME)) + print("that maps public IPv4 addresses to instances launched in the subnet.") + sys.exit(1) + + print("Making Security Group") + vpc = ec2_client.Vpc(vpc_id) + sg_exists = False + for sg in vpc.security_groups.all(): + if sg.group_name == SECURITY_GROUP_NAME: + security_group_id = sg.id + sg_exists = True + print(" %s already exists"%SECURITY_GROUP_NAME) + if not sg_exists: + security_group_id = make_security_group(vpc).id + time.sleep(30) + + boulder_preexists = False + boulder_servers = ec2_client.instances.filter(Filters=[ + {'Name': 'tag:Name', 'Values': ['le-boulderserver']}, + {'Name': 'instance-state-name', 'Values': ['running']}]) + + boulder_server = next(iter(boulder_servers), None) + + print("Requesting Instances...") + if boulder_server: + print("Found existing boulder server:", boulder_server) + boulder_preexists = True + else: + print("Can't find a boulder server, starting one...") + boulder_server = make_instance(ec2_client, + 'le-boulderserver', + BOULDER_AMI, + KEYNAME, + machine_type='t2.micro', + #machine_type='t2.medium', + security_group_id=security_group_id, + subnet_id=subnet_id) + + instances = [] + try: + if not cl_args.boulderonly: + print("Creating instances: ", end="") + for target in targetlist: + instances.append( + create_client_instance(ec2_client, target, + security_group_id, subnet_id) + ) + print() + + # Configure and launch boulder server + #------------------------------------------------------------------------------- + print("Waiting on Boulder Server") + boulder_server = block_until_instance_ready(boulder_server) + print(" server %s"%boulder_server) -# Set up EC2 instances -#------------------------------------------------------------------------------- -configdata = yaml.load(open(cl_args.config_file, 'r')) -targetlist = configdata['targets'] -print('Testing against these images: [%d total]'%len(targetlist)) -for target in targetlist: - print(target['ami'], target['name']) + # env.host_string defines the ssh user and host for connection + env.host_string = "ubuntu@%s"%boulder_server.public_ip_address + print("Boulder Server at (SSH):", env.host_string) + if not boulder_preexists: + print("Configuring and Launching Boulder") + config_and_launch_boulder(boulder_server) + # blocking often unnecessary, but cheap EC2 VMs can get very slow + block_until_http_ready('http://%s:4000'%boulder_server.public_ip_address, + wait_time=10, timeout=500) -print("Connecting to EC2 using\n profile %s\n keyname %s\n keyfile %s"%(PROFILE, KEYNAME, KEYFILE)) -AWS_SESSION = boto3.session.Session(profile_name=PROFILE) -EC2 = AWS_SESSION.resource('ec2') + boulder_url = "http://%s:4000/directory"%boulder_server.private_ip_address + print("Boulder Server at (public ip): http://%s:4000/directory"%boulder_server.public_ip_address) + print("Boulder Server at (EC2 private ip): %s"%boulder_url) -print("Determining Subnet") -for subnet in EC2.subnets.all(): - if should_use_subnet(subnet): - subnet_id = subnet.id - vpc_id = subnet.vpc.id - break -else: - print("No usable subnet exists!") - print("Please create a VPC with a subnet named {0}".format(SUBNET_NAME)) - print("that maps public IPv4 addresses to instances launched in the subnet.") - sys.exit(1) + if cl_args.boulderonly: + sys.exit(0) -print("Making Security Group") -vpc = EC2.Vpc(vpc_id) -sg_exists = False -for sg in vpc.security_groups.all(): - if sg.group_name == SECURITY_GROUP_NAME: - security_group_id = sg.id - sg_exists = True - print(" %s already exists"%SECURITY_GROUP_NAME) -if not sg_exists: - security_group_id = make_security_group(vpc).id - time.sleep(30) + # Install and launch client scripts in parallel + #------------------------------------------------------------------------------- + print("Uploading and running test script in parallel: %s"%cl_args.test_script) + print("Output routed to log files in %s"%LOGDIR) + # (Advice: always use Manager.Queue, never regular multiprocessing.Queue + # the latter has implementation flaws that deadlock it in some circumstances) + manager = Manager() + outqueue = manager.Queue() + inqueue = manager.Queue() -boulder_preexists = False -boulder_servers = EC2.instances.filter(Filters=[ - {'Name': 'tag:Name', 'Values': ['le-boulderserver']}, - {'Name': 'instance-state-name', 'Values': ['running']}]) - -boulder_server = next(iter(boulder_servers), None) - -print("Requesting Instances...") -if boulder_server: - print("Found existing boulder server:", boulder_server) - boulder_preexists = True -else: - print("Can't find a boulder server, starting one...") - boulder_server = make_instance('le-boulderserver', - BOULDER_AMI, - KEYNAME, - machine_type='t2.micro', - #machine_type='t2.medium', - security_group_id=security_group_id, - subnet_id=subnet_id) - -try: - if not cl_args.boulderonly: - instances = create_client_instances(targetlist, security_group_id, subnet_id) - - # Configure and launch boulder server - #------------------------------------------------------------------------------- - print("Waiting on Boulder Server") - boulder_server = block_until_instance_ready(boulder_server) - print(" server %s"%boulder_server) + # launch as many processes as clients to test + num_processes = len(targetlist) + jobs = [] #keep a reference to current procs - # env.host_string defines the ssh user and host for connection - env.host_string = "ubuntu@%s"%boulder_server.public_ip_address - print("Boulder Server at (SSH):", env.host_string) - if not boulder_preexists: - print("Configuring and Launching Boulder") - config_and_launch_boulder(boulder_server) - # blocking often unnecessary, but cheap EC2 VMs can get very slow - block_until_http_ready('http://%s:4000'%boulder_server.public_ip_address, - wait_time=10, timeout=500) + # initiate process execution + for i in range(num_processes): + p = mp.Process(target=test_client_process, args=(inqueue, outqueue, boulder_url)) + jobs.append(p) + p.daemon = True # kills subprocesses if parent is killed + p.start() - boulder_url = "http://%s:4000/directory"%boulder_server.private_ip_address - print("Boulder Server at (public ip): http://%s:4000/directory"%boulder_server.public_ip_address) - print("Boulder Server at (EC2 private ip): %s"%boulder_url) + # fill up work queue + for ii, target in enumerate(targetlist): + inqueue.put((ii, instances[ii].id, target)) - if cl_args.boulderonly: - sys.exit(0) + # add SENTINELs to end client processes + for i in range(num_processes): + inqueue.put(SENTINEL) + print('Waiting on client processes', end='') + for p in jobs: + while p.is_alive(): + p.join(5 * 60) + # Regularly print output to keep Travis happy + print('.', end='') + sys.stdout.flush() + print() + # add SENTINEL to output queue + outqueue.put(SENTINEL) - # Install and launch client scripts in parallel - #------------------------------------------------------------------------------- - print("Uploading and running test script in parallel: %s"%cl_args.test_script) - print("Output routed to log files in %s"%LOGDIR) - # (Advice: always use Manager.Queue, never regular multiprocessing.Queue - # the latter has implementation flaws that deadlock it in some circumstances) - manager = Manager() - outqueue = manager.Queue() - inqueue = manager.Queue() - SENTINEL = None #queue kill signal + # clean up + execute(local_repo_clean) - # launch as many processes as clients to test - num_processes = len(targetlist) - jobs = [] #keep a reference to current procs + # print and save summary results + results_file = open(LOGDIR+'/results', 'w') + outputs = [outq for outq in iter(outqueue.get, SENTINEL)] + outputs.sort(key=lambda x: x[0]) + failed = False + for outq in outputs: + ii, target, status = outq + if status == Status.FAIL: + failed = True + print('%d %s %s'%(ii, target['name'], status)) + results_file.write('%d %s %s\n'%(ii, target['name'], status)) + if len(outputs) != num_processes: + failed = True + failure_message = 'FAILURE: Some target machines failed to run and were not tested. ' +\ + 'Tests should be rerun.' + print(failure_message) + results_file.write(failure_message + '\n') + results_file.close() + + if failed: + sys.exit(1) + + finally: + cleanup(cl_args, instances, targetlist) + + # kill any connections + fabric.network.disconnect_all() - # initiate process execution - for i in range(num_processes): - p = mp.Process(target=test_client_process, args=(inqueue, outqueue)) - jobs.append(p) - p.daemon = True # kills subprocesses if parent is killed - p.start() - - # fill up work queue - for ii, target in enumerate(targetlist): - inqueue.put((ii, target)) - - # add SENTINELs to end client processes - for i in range(num_processes): - inqueue.put(SENTINEL) - # wait on termination of client processes - for p in jobs: - p.join() - # add SENTINEL to output queue - outqueue.put(SENTINEL) - - # clean up - execute(local_repo_clean) - - # print and save summary results - results_file = open(LOGDIR+'/results', 'w') - outputs = [outq for outq in iter(outqueue.get, SENTINEL)] - outputs.sort(key=lambda x: x[0]) - for outq in outputs: - ii, target, status = outq - print('%d %s %s'%(ii, target['name'], status)) - results_file.write('%d %s %s\n'%(ii, target['name'], status)) - results_file.close() - -finally: - cleanup(cl_args, instances, targetlist) - - # kill any connections - fabric.network.disconnect_all() +if __name__ == '__main__': + main() diff --git a/tests/letstest/requirements.txt b/tests/letstest/requirements.txt new file mode 100644 index 000000000..64e1f6a0c --- /dev/null +++ b/tests/letstest/requirements.txt @@ -0,0 +1,25 @@ +asn1crypto==0.24.0 +awscli==1.16.157 +bcrypt==3.1.6 +boto3==1.9.146 +botocore==1.12.147 +cffi==1.12.3 +colorama==0.3.9 +cryptography==2.4.2 +docutils==0.14 +enum34==1.1.6 +Fabric==1.14.1 +futures==3.2.0 +idna==2.8 +ipaddress==1.0.22 +jmespath==0.9.4 +paramiko==2.4.2 +pyasn1==0.4.5 +pycparser==2.19 +PyNaCl==1.3.0 +python-dateutil==2.8.0 +PyYAML==3.10 +rsa==3.4.2 +s3transfer==0.2.0 +six==1.12.0 +urllib3==1.24.3 diff --git a/tests/letstest/scripts/boulder_config.sh b/tests/letstest/scripts/boulder_config.sh index 1ef63ca10..b99bbabbe 100755 --- a/tests/letstest/scripts/boulder_config.sh +++ b/tests/letstest/scripts/boulder_config.sh @@ -1,32 +1,24 @@ #!/bin/bash -x # Configures and Launches Boulder Server installed on -# us-east-1 ami-5f490b35 bouldertestserver (boulder commit 8b433f54dab) +# us-east-1 ami-072a9534772bec854 bouldertestserver3 (boulder commit b24fe7c3ea4) # fetch instance data from EC2 metadata service public_host=$(curl -s http://169.254.169.254/2014-11-05/meta-data/public-hostname) public_ip=$(curl -s http://169.254.169.254/2014-11-05/meta-data/public-ipv4) private_ip=$(curl -s http://169.254.169.254/2014-11-05/meta-data/local-ipv4) -# get local DNS resolver for VPC -resolver_ip=$(grep nameserver /etc/resolv.conf |cut -d" " -f2 |head -1) +# set to public DNS resolver +resolver_ip=8.8.8.8 resolver=$resolver_ip':53' # modifies integration testing boulder setup for local AWS VPC network # connections instead of localhost cd $GOPATH/src/github.com/letsencrypt/boulder -# configure boulder to receive outside connection on 4000 -sed -i '/listenAddress/ s/127.0.0.1:4000/'$private_ip':4000/' ./test/boulder-config.json -sed -i '/baseURL/ s/127.0.0.1:4000/'$private_ip':4000/' ./test/boulder-config.json # change test ports to real -sed -i '/httpPort/ s/5002/80/' ./test/boulder-config.json -sed -i '/httpsPort/ s/5001/443/' ./test/boulder-config.json -sed -i '/tlsPort/ s/5001/443/' ./test/boulder-config.json -# set local dns resolver -sed -i '/dnsResolver/ s/127.0.0.1:8053/'$resolver'/' ./test/boulder-config.json - -# start rabbitMQ -#go run cmd/rabbitmq-setup/main.go -server amqp://localhost -# start acme services -#nohup ./start.py >& /dev/null < /dev/null & -#./start.py +sed -i '/httpPort/ s/5002/80/' ./test/config/va.json +sed -i '/httpsPort/ s/5001/443/' ./test/config/va.json +sed -i '/tlsPort/ s/5001/443/' ./test/config/va.json +# set dns resolver +sed -i 's/"127.0.0.1:8053",/"'$resolver'"/' ./test/config/va.json +sed -i 's/"127.0.0.1:8054"//' ./test/config/va.json diff --git a/tests/letstest/scripts/boulder_install.sh b/tests/letstest/scripts/boulder_install.sh index f997268bd..5161de374 100755 --- a/tests/letstest/scripts/boulder_install.sh +++ b/tests/letstest/scripts/boulder_install.sh @@ -1,7 +1,5 @@ #!/bin/bash -x -# >>>> only tested on Ubuntu 14.04LTS <<<< - # Check out special branch until latest docker changes land in Boulder master. git clone -b docker-integration https://github.com/letsencrypt/boulder $BOULDERPATH cd $BOULDERPATH diff --git a/tests/letstest/scripts/set_python_envvars.sh b/tests/letstest/scripts/set_python_envvars.sh new file mode 100755 index 000000000..668444209 --- /dev/null +++ b/tests/letstest/scripts/set_python_envvars.sh @@ -0,0 +1,17 @@ +#!/bin/sh +# This is a simple script that can be sourced to set Python environment +# variables for use in Certbot's letstest test farm tests. + +# Some distros like Fedora may only have an executable named python3 installed. +if command -v python; then + PYTHON_NAME="python" + VENV_SCRIPT="tools/venv.py" + VENV_PATH="venv" +else + # We could check for "python2" here, however, the addition of "python3" + # only systems is what necessitated this change so checking for "python2" + # isn't necessary. + PYTHON_NAME="python3" + VENV_PATH="venv3" + VENV_SCRIPT="tools/venv3.py" +fi diff --git a/tests/letstest/scripts/test_apache2.sh b/tests/letstest/scripts/test_apache2.sh index d24de2458..9af39e8bb 100755 --- a/tests/letstest/scripts/test_apache2.sh +++ b/tests/letstest/scripts/test_apache2.sh @@ -45,8 +45,13 @@ if [ $? -ne 0 ] ; then exit 1 fi -python tools/_venv_common.py -e acme[dev] -e .[dev,docs] -e certbot-apache -sudo venv/bin/certbot -v --debug --text --agree-dev-preview --agree-tos \ +# This script sets the environment variables PYTHON_NAME, VENV_PATH, and +# VENV_SCRIPT based on the version of Python available on the system. For +# instance, Fedora uses Python 3 and Python 2 is not installed. +. tests/letstest/scripts/set_python_envvars.sh + +"$VENV_SCRIPT" -e acme[dev] -e certbot[dev,docs] -e certbot-apache +sudo "$VENV_PATH/bin/certbot" -v --debug --text --agree-tos \ --renew-by-default --redirect --register-unsafely-without-email \ --domain $PUBLIC_HOSTNAME --server $BOULDER_URL if [ $? -ne 0 ] ; then @@ -55,7 +60,7 @@ fi if [ "$OS_TYPE" = "ubuntu" ] ; then export SERVER="$BOULDER_URL" - venv/bin/tox -e apacheconftest + "$VENV_PATH/bin/tox" -e apacheconftest else echo Not running hackish apache tests on $OS_TYPE fi diff --git a/tests/letstest/scripts/test_leauto_upgrades.sh b/tests/letstest/scripts/test_leauto_upgrades.sh index 0c2b374f2..fc7632793 100755 --- a/tests/letstest/scripts/test_leauto_upgrades.sh +++ b/tests/letstest/scripts/test_leauto_upgrades.sh @@ -15,27 +15,37 @@ if ! command -v git ; then exit 1 fi fi -# 0.17.0 is the oldest version of letsencrypt-auto that has precompiled -# cryptography and the tagged commit is in master. 0.16.0 was the first version -# to use precompiled cryptography, but the release PR was squashed losing the -# commit. We want to use a precompiled version of cryptography for stability. -# Previous versions that have to compile against OpenSSL on installation -# started failing on newer distros with newer versions of OpenSSL. -INITIAL_VERSION="0.17.0" +# If we're on a RHEL 6 based system, we can be confident Python is already +# installed because the package manager is written in Python. +if command -v python && [ $(python -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') -eq 26 ]; then + # 0.20.0 is the latest version of letsencrypt-auto that doesn't install + # Python 3 on RHEL 6. + INITIAL_VERSION="0.20.0" + RUN_RHEL6_TESTS=1 +else + # 0.37.x is the oldest version of letsencrypt-auto that works on RHEL 8. + INITIAL_VERSION="0.37.1" +fi + git checkout -f "v$INITIAL_VERSION" letsencrypt-auto -if ! ./letsencrypt-auto -v --debug --version --no-self-upgrade 2>&1 | grep "$INITIAL_VERSION" ; then +if ! ./letsencrypt-auto -v --debug --version --no-self-upgrade 2>&1 | tail -n1 | grep "^certbot $INITIAL_VERSION$" ; then echo initial installation appeared to fail exit 1 fi +# This script sets the environment variables PYTHON_NAME, VENV_PATH, and +# VENV_SCRIPT based on the version of Python available on the system. For +# instance, Fedora uses Python 3 and Python 2 is not installed. +. tests/letstest/scripts/set_python_envvars.sh + # Now that python and openssl have been installed, we can set up a fake server # to provide a new version of letsencrypt-auto. First, we start the server and # directory to be served. MY_TEMP_DIR=$(mktemp -d) PORT_FILE="$MY_TEMP_DIR/port" -SERVER_PATH=$(tools/readlink.py tools/simple_http_server.py) +SERVER_PATH=$("$PYTHON_NAME" tools/readlink.py tools/simple_http_server.py) cd "$MY_TEMP_DIR" -"$SERVER_PATH" 0 > $PORT_FILE & +"$PYTHON_NAME" "$SERVER_PATH" 0 > $PORT_FILE & SERVER_PID=$! trap 'kill "$SERVER_PID" && rm -rf "$MY_TEMP_DIR"' EXIT cd ~- @@ -68,8 +78,7 @@ iQIDAQAB -----END PUBLIC KEY----- " -if [ $(python -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') -eq 26 ]; then - RUN_PYTHON3_TESTS=1 +if [ "$RUN_RHEL6_TESTS" = 1 ]; then if command -v python3; then echo "Didn't expect Python 3 to be installed!" exit 1 @@ -79,13 +88,12 @@ if [ $(python -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') -eq 26 ]; echo "Certbot shouldn't have updated to a new version!" exit 1 fi - if [ -d "/opt/eff.org" ]; then - echo "New directory shouldn't have been created!" - exit 1 - fi - # Create a 2nd venv at the new path to ensure we properly handle this case - export VENV_PATH="/opt/eff.org/certbot/venv" - if ! sudo -E ./letsencrypt-auto -v --debug --version --no-self-upgrade 2>&1 | grep "$INITIAL_VERSION" ; then + # Create a 2nd venv at the old path to ensure we properly handle the (unlikely) case of two separate virtual environments below. + HOME=${HOME:-~root} + XDG_DATA_HOME=${XDG_DATA_HOME:-~/.local/share} + OLD_VENV_PATH="$XDG_DATA_HOME/letsencrypt" + export VENV_PATH="$OLD_VENV_PATH" + if ! sudo -E ./letsencrypt-auto -v --debug --version --no-self-upgrade 2>&1 | tail -n1 | grep "^certbot $INITIAL_VERSION$" ; then echo second installation appeared to fail exit 1 fi @@ -98,7 +106,7 @@ if ./letsencrypt-auto -v --debug --version | grep "WARNING: couldn't find Python fi EXPECTED_VERSION=$(grep -m1 LE_AUTO_VERSION certbot-auto | cut -d\" -f2) -if ! /opt/eff.org/certbot/venv/bin/letsencrypt --version 2>&1 | grep "$EXPECTED_VERSION" ; then +if ! /opt/eff.org/certbot/venv/bin/letsencrypt --version 2>&1 | tail -n1 | grep "^certbot $EXPECTED_VERSION$" ; then echo upgrade appeared to fail exit 1 fi @@ -108,7 +116,9 @@ if ! diff letsencrypt-auto letsencrypt-auto-source/letsencrypt-auto ; then exit 1 fi -if [ "$RUN_PYTHON3_TESTS" = 1 ]; then +if [ "$RUN_RHEL6_TESTS" = 1 ]; then + # Add the SCL python release to PATH in order to resolve python3 command + PATH="/opt/rh/rh-python36/root/usr/bin:$PATH" if ! command -v python3; then echo "Python3 wasn't properly installed" exit 1 @@ -117,11 +127,10 @@ if [ "$RUN_PYTHON3_TESTS" = 1 ]; then echo "Python3 wasn't used in venv!" exit 1 fi + + if [ "$("$PYTHON_NAME" tools/readlink.py $OLD_VENV_PATH)" != "/opt/eff.org/certbot/venv" ]; then + echo symlink from old venv path not properly created! + exit 1 + fi fi echo upgrade appeared to be successful - -if [ "$(tools/readlink.py ${XDG_DATA_HOME:-~/.local/share}/letsencrypt)" != "/opt/eff.org/certbot/venv" ]; then - echo symlink from old venv path not properly created! - exit 1 -fi -echo symlink properly created diff --git a/tests/letstest/scripts/test_letsencrypt_auto_certonly_standalone.sh b/tests/letstest/scripts/test_letsencrypt_auto_certonly_standalone.sh index 2cbe66a83..c028031c7 100755 --- a/tests/letstest/scripts/test_letsencrypt_auto_certonly_standalone.sh +++ b/tests/letstest/scripts/test_letsencrypt_auto_certonly_standalone.sh @@ -9,32 +9,37 @@ set -eo pipefail #private_ip=$(curl -s http://169.254.169.254/2014-11-05/meta-data/local-ipv4) cd letsencrypt -export PATH="$PWD/letsencrypt-auto-source:$PATH" +LE_AUTO_DIR="/usr/local/bin" +LE_AUTO_PATH="$LE_AUTO_DIR/letsencrypt-auto" +sudo cp letsencrypt-auto-source/letsencrypt-auto "$LE_AUTO_PATH" +sudo chown root "$LE_AUTO_PATH" +sudo chmod 0755 "$LE_AUTO_PATH" +export PATH="$LE_AUTO_DIR:$PATH" + letsencrypt-auto --os-packages-only --debug --version + +# This script sets the environment variables PYTHON_NAME, VENV_PATH, and +# VENV_SCRIPT based on the version of Python available on the system. For +# instance, Fedora uses Python 3 and Python 2 is not installed. +. tests/letstest/scripts/set_python_envvars.sh + +# Create a venv-like layout at the old virtual environment path to test that a +# symlink is properly created when letsencrypt-auto runs. +HOME=${HOME:-~root} +XDG_DATA_HOME=${XDG_DATA_HOME:-~/.local/share} +OLD_VENV_BIN="$XDG_DATA_HOME/letsencrypt/bin" +mkdir -p "$OLD_VENV_BIN" +touch "$OLD_VENV_BIN/letsencrypt" + letsencrypt-auto certonly --no-self-upgrade -v --standalone --debug \ - --text --agree-dev-preview --agree-tos \ + --text --agree-tos \ --renew-by-default --redirect \ --register-unsafely-without-email \ --domain $PUBLIC_HOSTNAME --server $BOULDER_URL -# we have to jump through some hoops to cope with relative paths in renewal -# conf files ... -# 1. be in the right directory -cd tests/letstest/testdata/ - -# 2. refer to the config with the same level of relativity that it itself -# contains :/ -OUT=`letsencrypt-auto certificates --config-dir sample-config -v --no-self-upgrade` -TEST_CERTS=`echo "$OUT" | grep TEST_CERT | wc -l` -REVOKED=`echo "$OUT" | grep REVOKED | wc -l` - -if [ "$TEST_CERTS" != 2 ] ; then - echo "Did not find two test certs as expected ($TEST_CERTS)" - exit 1 -fi - -if [ "$REVOKED" != 1 ] ; then - echo "Did not find one revoked cert as expected ($REVOKED)" +LINK_PATH=$("$PYTHON_NAME" tools/readlink.py ${XDG_DATA_HOME:-~/.local/share}/letsencrypt) +if [ "$LINK_PATH" != "/opt/eff.org/certbot/venv" ]; then + echo symlink from old venv path not properly created! exit 1 fi @@ -42,3 +47,9 @@ if ! letsencrypt-auto --help --no-self-upgrade | grep -F "letsencrypt-auto [SUBC echo "letsencrypt-auto not included in help output!" exit 1 fi + +OUTPUT_LEN=$(letsencrypt-auto --install-only --no-self-upgrade --quiet 2>&1 | wc -c) +if [ "$OUTPUT_LEN" != 0 ]; then + echo letsencrypt-auto produced unexpected output! + exit 1 +fi diff --git a/tests/letstest/scripts/test_letsencrypt_auto_venv_only.sh b/tests/letstest/scripts/test_letsencrypt_auto_venv_only.sh deleted file mode 100755 index c55e12e8b..000000000 --- a/tests/letstest/scripts/test_letsencrypt_auto_venv_only.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash -x - -# $PUBLIC_IP $PRIVATE_IP $PUBLIC_HOSTNAME $BOULDER_URL are dynamically set at execution - -cd letsencrypt -# help installs virtualenv and does nothing else -./letsencrypt-auto-source/letsencrypt-auto -v --debug --help all diff --git a/tests/letstest/scripts/test_renew_standalone.sh b/tests/letstest/scripts/test_renew_standalone.sh deleted file mode 100755 index 31c38ea46..000000000 --- a/tests/letstest/scripts/test_renew_standalone.sh +++ /dev/null @@ -1,55 +0,0 @@ -#!/bin/bash -x - -# $OS_TYPE $PUBLIC_IP $PRIVATE_IP $PUBLIC_HOSTNAME $BOULDER_URL -# are dynamically set at execution - -# run certbot-apache2 via letsencrypt-auto -cd letsencrypt - -export SUDO=sudo -if [ -f /etc/debian_version ] ; then - echo "Bootstrapping dependencies for Debian-based OSes..." - $SUDO bootstrap/_deb_common.sh -elif [ -f /etc/redhat-release ] ; then - echo "Bootstrapping dependencies for RedHat-based OSes..." - $SUDO bootstrap/_rpm_common.sh -else - echo "Don't have bootstrapping for this OS!" - exit 1 -fi - -bootstrap/dev/venv.sh -sudo venv/bin/certbot certonly --debug --standalone -t --agree-dev-preview --agree-tos \ - --renew-by-default --redirect --register-unsafely-without-email \ - --domain $PUBLIC_HOSTNAME --server $BOULDER_URL -v -if [ $? -ne 0 ] ; then - FAIL=1 -fi - -if [ "$OS_TYPE" = "ubuntu" ] ; then - venv/bin/tox -e apacheconftest -else - echo Not running hackish apache tests on $OS_TYPE -fi - -if [ $? -ne 0 ] ; then - FAIL=1 -fi - -sudo venv/bin/certbot renew --renew-by-default - -if [ $? -ne 0 ] ; then - FAIL=1 -fi - - -ls /etc/letsencrypt/archive/$PUBLIC_HOSTNAME | grep -q 2.pem - -if [ $? -ne 0 ] ; then - FAIL=1 -fi - -# return error if any of the subtests failed -if [ "$FAIL" = 1 ] ; then - exit 1 -fi diff --git a/tests/letstest/scripts/test_sdists.sh b/tests/letstest/scripts/test_sdists.sh index 0b9a91ffd..204f55d55 100755 --- a/tests/letstest/scripts/test_sdists.sh +++ b/tests/letstest/scripts/test_sdists.sh @@ -1,20 +1,46 @@ #!/bin/sh -xe cd letsencrypt -./certbot-auto --os-packages-only -n --debug + +# If we're on a RHEL 6 based system, we can be confident Python is already +# installed because the package manager is written in Python. +if command -v python && [ $(python -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') -eq 26 ]; then + # RHEL/CentOS 6 will need a special treatment, so we need to detect that environment + RUN_RHEL6_TESTS=1 +fi + +letsencrypt-auto-source/letsencrypt-auto --install-only -n --debug + +if [ "$RUN_RHEL6_TESTS" = 1 ]; then + # Enable the SCL Python 3.6 installed by letsencrypt-auto bootstrap + PATH="/opt/rh/rh-python36/root/usr/bin:$PATH" +fi PLUGINS="certbot-apache certbot-nginx" -PYTHON=$(command -v python2.7 || command -v python27 || command -v python2 || command -v python) +PYTHON_MAJOR_VERSION=$(/opt/eff.org/certbot/venv/bin/python --version 2>&1 | cut -d" " -f 2 | cut -d. -f1) TEMP_DIR=$(mktemp -d) -VERSION=$(letsencrypt-auto-source/version.py) -export VENV_ARGS="-p $PYTHON" + +if [ "$PYTHON_MAJOR_VERSION" = "3" ]; then + # Some distros like Fedora may only have an executable named python3 installed. + PYTHON_NAME="python3" + VENV_PATH="venv3" + VENV_SCRIPT="tools/venv3.py" +else + PYTHON_NAME="python" + VENV_SCRIPT="tools/venv.py" + VENV_PATH="venv" +fi + +VERSION=$("$PYTHON_NAME" letsencrypt-auto-source/version.py) # setup venv -tools/_venv_common.py --requirement letsencrypt-auto-source/pieces/dependency-requirements.txt -. ./venv/bin/activate +"$VENV_SCRIPT" --requirement letsencrypt-auto-source/pieces/dependency-requirements.txt +. "$VENV_PATH/bin/activate" +# pytest is needed to run tests on some of our packages so we install a pinned version here. +tools/pip_install.py pytest # build sdists -for pkg_dir in acme . $PLUGINS; do +for pkg_dir in acme certbot $PLUGINS; do cd $pkg_dir python setup.py clean rm -rf build dist diff --git a/tests/letstest/scripts/test_tests.sh b/tests/letstest/scripts/test_tests.sh index e6ab836b8..fb86ce4cd 100755 --- a/tests/letstest/scripts/test_tests.sh +++ b/tests/letstest/scripts/test_tests.sh @@ -1,20 +1,29 @@ #!/bin/sh -xe +# +# This script is useful for testing that the packages we've built for a release +# work on a variety of systems. For an example of the kinds of problems that +# can occur, see https://github.com/certbot/certbot/issues/3455. -LE_AUTO="letsencrypt/letsencrypt-auto-source/letsencrypt-auto" +REPO_ROOT="letsencrypt" +LE_AUTO="$REPO_ROOT/letsencrypt-auto-source/letsencrypt-auto" LE_AUTO="$LE_AUTO --debug --no-self-upgrade --non-interactive" -MODULES="acme certbot certbot_apache certbot_nginx" +MODULES="acme certbot certbot-apache certbot-nginx" +PIP_INSTALL="$REPO_ROOT/tools/pip_install.py" VENV_NAME=venv # *-auto respects VENV_PATH $LE_AUTO --os-packages-only LE_AUTO_SUDO="" VENV_PATH="$VENV_NAME" $LE_AUTO --no-bootstrap --version . $VENV_NAME/bin/activate +"$PIP_INSTALL" pytest -# change to an empty directory to ensure CWD doesn't affect tests -cd $(mktemp -d) -pip install pytest==3.2.5 +# To run tests that aren't packaged in modules, run pytest +# from the repo root. The directory structure should still +# cause the installed packages to be tested while using +# the tests available in the subdirectories. +cd $REPO_ROOT for module in $MODULES ; do echo testing $module - pytest -v --pyargs $module + pytest -v $module done diff --git a/tests/letstest/scripts/test_tox.sh b/tests/letstest/scripts/test_tox.sh deleted file mode 100755 index bb9126673..000000000 --- a/tests/letstest/scripts/test_tox.sh +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash -x -XDG_DATA_HOME=${XDG_DATA_HOME:-~/.local/share} -VENV_NAME="venv" -# The path to the letsencrypt-auto script. Everything that uses these might -# at some point be inlined... -LEA_PATH=./letsencrypt/ -VENV_PATH=${LEA_PATH/$VENV_NAME} -VENV_BIN=${VENV_PATH}/bin - - -# virtualenv call is not idempotent: it overwrites pip upgraded in -# later steps, causing "ImportError: cannot import name unpack_url" - -"$LEA_PATH/letsencrypt-auto" --os-packages-only - -cd letsencrypt -python tools/venv.py -venv/bin/tox -e py27 diff --git a/tests/letstest/targets.yaml b/tests/letstest/targets.yaml index c1a28af98..188be8e24 100644 --- a/tests/letstest/targets.yaml +++ b/tests/letstest/targets.yaml @@ -1,12 +1,12 @@ targets: #----------------------------------------------------------------------------- #Ubuntu - - ami: ami-064bd2d44a1d6c097 - name: ubuntu18.10 + - ami: ami-08ab45c4343f5f5c6 + name: ubuntu19.04 type: ubuntu virt: hvm user: ubuntu - - ami: ami-012fd5eb46f56731f + - ami: ami-095192256fe1477ad name: ubuntu18.04LTS type: ubuntu virt: hvm @@ -16,24 +16,25 @@ targets: type: ubuntu virt: hvm user: ubuntu - - ami: ami-7b89cc11 - name: ubuntu14.04LTS - type: ubuntu - virt: hvm - user: ubuntu - - ami: ami-9295d0f8 - name: ubuntu14.04LTS_32bit - type: ubuntu - virt: pv - user: ubuntu #----------------------------------------------------------------------------- # Debian + - ami: ami-01db78123b2b99496 + name: debian10 + type: ubuntu + virt: hvm + user: admin - ami: ami-003f19e0e687de1cd name: debian9 type: ubuntu virt: hvm user: admin - - ami: ami-116d857a + - ami: ami-0ed54dd1b25657636 + name: debian9_arm64 + type: ubuntu + virt: hvm + user: admin + machine_type: a1.medium + - ami: ami-077bf3962f29d3fa4 name: debian8.1 type: ubuntu virt: hvm @@ -44,23 +45,18 @@ targets: # - [ apt-get, install, -y, curl ] #----------------------------------------------------------------------------- # Other Redhat Distros - - ami: ami-60b6c60a - name: amazonlinux-2015.09.1 - type: centos - virt: hvm - user: ec2-user - - ami: ami-0d4cfd66 - name: amazonlinux-2015.03.1 - type: centos - virt: hvm - user: ec2-user - - ami: ami-a8d369c0 + - ami: ami-0916c408cb02e310b name: RHEL7 type: centos virt: hvm user: ec2-user - - ami: ami-518bfb3b - name: fedora23 + - ami: ami-0c322300a1dd5dc79 + name: RHEL8 + type: centos + virt: hvm + user: ec2-user + - ami: ami-00bbc6858140f19ed + name: fedora30 type: centos virt: hvm user: fedora diff --git a/tests/letstest/travis-setup.sh b/tests/letstest/travis-setup.sh new file mode 100755 index 000000000..261a1504f --- /dev/null +++ b/tests/letstest/travis-setup.sh @@ -0,0 +1,10 @@ +#!/bin/bash -ex +# +# Preps the test farm tests to be run in Travis. + +if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then + echo This script must be run in Travis on a non-pull request build + exit 1 +fi + +openssl aes-256-cbc -K "${encrypted_9a387195a62e_key}" -iv "${encrypted_9a387195a62e_iv}" -in travis-test-farm.pem.enc -out travis-test-farm.pem -d diff --git a/tests/letstest/travis-test-farm.pem.enc b/tests/letstest/travis-test-farm.pem.enc new file mode 100644 index 000000000..f8b1d576c Binary files /dev/null and b/tests/letstest/travis-test-farm.pem.enc differ diff --git a/tests/lock_test.py b/tests/lock_test.py index 0266cf029..29a77ae17 100644 --- a/tests/lock_test.py +++ b/tests/lock_test.py @@ -2,6 +2,7 @@ from __future__ import print_function import atexit +import datetime import functools import logging import os @@ -11,12 +12,19 @@ import subprocess import sys import tempfile -from certbot import lock +from cryptography import x509 +from cryptography.hazmat.backends import default_backend +# TODO: once mypy has cryptography types bundled, type: ignore can be removed. +# See https://github.com/pyca/cryptography/issues/4275 +from cryptography.hazmat.primitives import hashes # type: ignore +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric import rsa + from certbot import util - +from certbot._internal import lock +from certbot.compat import filesystem from certbot.tests import util as test_util - logger = logging.getLogger(__name__) @@ -84,7 +92,7 @@ def set_up_dirs(): nginx_dir = os.path.join(temp_dir, 'nginx') for directory in (config_dir, logs_dir, work_dir, nginx_dir,): - os.mkdir(directory) + filesystem.mkdir(directory) test_util.make_lineage(config_dir, 'sample-renewal.conf') set_up_nginx_dir(nginx_dir) @@ -102,12 +110,11 @@ def set_up_nginx_dir(root_path): repo_root = check_call('git rev-parse --show-toplevel'.split()).strip() conf_script = os.path.join( repo_root, 'certbot-nginx', 'tests', 'boulder-integration.conf.sh') - # boulder-integration.conf.sh uses the root environment variable as - # the Nginx server root when writing paths - os.environ['root'] = root_path + # Prepare self-signed certificates for Nginx + key_path, cert_path = setup_certificate(root_path) + # Generate Nginx configuration with open(os.path.join(root_path, 'nginx.conf'), 'w') as f: - f.write(check_call(['/bin/sh', conf_script])) - del os.environ['root'] + f.write(check_call(['/bin/sh', conf_script, root_path, key_path, cert_path])) def set_up_command(config_dir, logs_dir, work_dir, nginx_dir): @@ -134,6 +141,51 @@ def set_up_command(config_dir, logs_dir, work_dir, nginx_dir): config_dir, logs_dir, work_dir, nginx_dir).split()) +def setup_certificate(workspace): + """Generate a self-signed certificate for nginx. + :param workspace: path of folder where to put the certificate + :return: tuple containing the key path and certificate path + :rtype: `tuple` + """ + # Generate key + # See comment on cryptography import about type: ignore + private_key = rsa.generate_private_key( # type: ignore + public_exponent=65537, + key_size=2048, + backend=default_backend() + ) + subject = issuer = x509.Name([ + x509.NameAttribute(x509.NameOID.COMMON_NAME, u'nginx.wtf') + ]) + certificate = x509.CertificateBuilder().subject_name( + subject + ).issuer_name( + issuer + ).public_key( + private_key.public_key() + ).serial_number( + 1 + ).not_valid_before( + datetime.datetime.utcnow() + ).not_valid_after( + datetime.datetime.utcnow() + datetime.timedelta(days=1) + ).sign(private_key, hashes.SHA256(), default_backend()) + + key_path = os.path.join(workspace, 'cert.key') + with open(key_path, 'wb') as file_handle: + file_handle.write(private_key.private_bytes( # type: ignore + encoding=serialization.Encoding.PEM, + format=serialization.PrivateFormat.TraditionalOpenSSL, + encryption_algorithm=serialization.NoEncryption() + )) + + cert_path = os.path.join(workspace, 'cert.pem') + with open(cert_path, 'wb') as file_handle: + file_handle.write(certificate.public_bytes(serialization.Encoding.PEM)) + + return key_path, cert_path + + def test_command(command, directories): """Assert Certbot acquires locks in a specific order. diff --git a/tests/manual-dns-auth.sh b/tests/manual-dns-auth.sh deleted file mode 100755 index febecf455..000000000 --- a/tests/manual-dns-auth.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/bin/bash - -# If domain begins with fail, fail the challenge by not completing it. -if [[ "$CERTBOT_DOMAIN" != fail* ]]; then - curl -X POST 'http://localhost:8055/set-txt' -d \ - "{\"host\": \"_acme-challenge.$CERTBOT_DOMAIN.\", \ - \"value\": \"$CERTBOT_VALIDATION\"}" -fi diff --git a/tests/manual-dns-cleanup.sh b/tests/manual-dns-cleanup.sh deleted file mode 100755 index 1c09e892c..000000000 --- a/tests/manual-dns-cleanup.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/bin/bash - -# If domain begins with fail, we didn't complete the challenge so there is -# nothing to clean up. -if [[ "$CERTBOT_DOMAIN" != fail* ]]; then - curl -X POST 'http://localhost:8055/clear-txt' -d \ - "{\"host\": \"_acme-challenge.$CERTBOT_DOMAIN.\"}" -fi diff --git a/tests/manual-http-auth.sh b/tests/manual-http-auth.sh deleted file mode 100755 index 48c33f04b..000000000 --- a/tests/manual-http-auth.sh +++ /dev/null @@ -1,14 +0,0 @@ -#!/bin/sh -uri_path=".well-known/acme-challenge/$CERTBOT_TOKEN" - -# This script should be run from the top level. e.g. ./tests/manual-http-auth.sh -source_dir="$(pwd)" -cd $(mktemp -d) -mkdir -p $(dirname $uri_path) -echo $CERTBOT_VALIDATION > $uri_path -python "$source_dir/tests/run_http_server.py" $http_01_port >/dev/null 2>&1 & -server_pid=$! -while ! curl "http://localhost:$http_01_port/$uri_path" >/dev/null 2>&1; do - sleep 1s -done -echo $server_pid diff --git a/tests/manual-http-cleanup.sh b/tests/manual-http-cleanup.sh deleted file mode 100755 index 5e437bf08..000000000 --- a/tests/manual-http-cleanup.sh +++ /dev/null @@ -1,2 +0,0 @@ -#!/bin/sh -kill $CERTBOT_AUTH_OUTPUT diff --git a/tests/modification-check.py b/tests/modification-check.py index 8abc0fbfe..811f369d4 100755 --- a/tests/modification-check.py +++ b/tests/modification-check.py @@ -3,10 +3,11 @@ from __future__ import print_function import os +import shutil import subprocess import sys import tempfile -import shutil + try: from urllib.request import urlretrieve except ImportError: diff --git a/tests/pebble-fetch.sh b/tests/pebble-fetch.sh deleted file mode 100755 index b0ba08961..000000000 --- a/tests/pebble-fetch.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/bash -# Download and run Pebble instance for integration testing -set -xe - -PEBBLE_VERSION=2018-11-02 - -# We reuse the same GOPATH-style directory than for Boulder. -# Pebble does not need it, but it will make the installation consistent with Boulder's one. -export GOPATH=${GOPATH:-$HOME/gopath} -PEBBLEPATH=${PEBBLEPATH:-$GOPATH/src/github.com/letsencrypt/pebble} - -mkdir -p ${PEBBLEPATH} - -cat << UNLIKELY_EOF > "$PEBBLEPATH/docker-compose.yml" -version: '3' - -services: - pebble: - image: letsencrypt/pebble:${PEBBLE_VERSION} - command: pebble -strict ${PEBBLE_STRICT:-false} -dnsserver 10.77.77.1 - ports: - - 14000:14000 - environment: - - PEBBLE_VA_NOSLEEP=1 -UNLIKELY_EOF - -docker-compose -f "$PEBBLEPATH/docker-compose.yml" up -d pebble - -set +x # reduce verbosity while waiting for boulder -for n in `seq 1 150` ; do - if curl -k https://localhost:14000/dir 2>/dev/null; then - break - else - sleep 1 - fi -done - -if ! curl -k https://localhost:14000/dir 2>/dev/null; then - echo "timed out waiting for pebble to start" - exit 1 -fi diff --git a/tests/run_http_server.py b/tests/run_http_server.py deleted file mode 100644 index 0e4f8ac79..000000000 --- a/tests/run_http_server.py +++ /dev/null @@ -1,11 +0,0 @@ -import runpy -import sys - -# Run Python's built-in HTTP server -# Usage: python ./tests/run_http_server.py port_num -# NOTE: This script should be compatible with 2.7, 3.4+ - -# sys.argv (port number) is passed as-is to the HTTP server module -runpy.run_module( - 'http.server' if sys.version_info[0] == 3 else 'SimpleHTTPServer', - run_name='__main__') diff --git a/tests/tox-boulder-integration.sh b/tests/tox-boulder-integration.sh deleted file mode 100755 index 8c8a967fd..000000000 --- a/tests/tox-boulder-integration.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/bash -e -# A simple wrapper around tests/boulder-integration.sh that activates the tox -# virtual environment defined by the environment variable TOXENV before running -# integration tests. - -if [ -z "${TOXENV+x}" ]; then - echo "The environment variable TOXENV must be set to use this script!" >&2 - exit 1 -fi - -source .tox/$TOXENV/bin/activate -tests/boulder-integration.sh diff --git a/tools/_changelog_top.txt b/tools/_changelog_top.txt index 6983b3a43..a0f9b6553 100644 --- a/tools/_changelog_top.txt +++ b/tools/_changelog_top.txt @@ -12,10 +12,4 @@ * -Despite us having broken lockstep, we are continuing to release new versions of -all Certbot components during releases for the time being, however, the only -package with changes other than its version number was: - -* - More details about these changes can be found on our GitHub repo. diff --git a/tools/_release.sh b/tools/_release.sh index d75a0f487..1819adad2 100755 --- a/tools/_release.sh +++ b/tools/_release.sh @@ -7,6 +7,24 @@ if [ "$RELEASE_DIR" = "" ]; then exit 1 fi +ExitWarning() { + exit_status="$?" + if [ "$exit_status" != 0 ]; then + # Don't print each command before executing it because it will disrupt + # the desired output. + set +x + echo '******************************' + echo '* *' + echo '* THE RELEASE SCRIPT FAILED! *' + echo '* *' + echo '******************************' + set -x + fi + exit "$exit_status" +} + +trap ExitWarning EXIT + version="$1" echo Releasing production version "$version"... nextversion="$2" @@ -30,7 +48,6 @@ SUBPKGS_NOT_IN_AUTO="certbot-dns-cloudflare certbot-dns-cloudxns certbot-dns-dig SUBPKGS_IN_AUTO="certbot $SUBPKGS_IN_AUTO_NO_CERTBOT" SUBPKGS_NO_CERTBOT="$SUBPKGS_IN_AUTO_NO_CERTBOT $SUBPKGS_NOT_IN_AUTO" SUBPKGS="$SUBPKGS_IN_AUTO $SUBPKGS_NOT_IN_AUTO" -subpkgs_modules="$(echo $SUBPKGS | sed s/-/_/g)" # certbot_compatibility_test is not packaged because: # - it is not meant to be used by anyone else than Certbot devs # - it causes problems when running pytest - the latter tries to @@ -66,17 +83,23 @@ fi git checkout "$RELEASE_BRANCH" # Update changelog -sed -i "s/master/$(date +'%Y-%m-%d')/" CHANGELOG.md -git add CHANGELOG.md -git diff --cached +sed -i "s/master/$(date +'%Y-%m-%d')/" certbot/CHANGELOG.md +git add certbot/CHANGELOG.md git commit -m "Update changelog for $version release" -for pkg_dir in $SUBPKGS_NO_CERTBOT certbot-compatibility-test . +for pkg_dir in $SUBPKGS certbot-compatibility-test do sed -i 's/\.dev0//' "$pkg_dir/setup.py" git add "$pkg_dir/setup.py" -done + if [ -f "$pkg_dir/local-oldest-requirements.txt" ]; then + sed -i "s/-e acme\[dev\]/acme[dev]==$version/" "$pkg_dir/local-oldest-requirements.txt" + sed -i "s/-e acme/acme[dev]==$version/" "$pkg_dir/local-oldest-requirements.txt" + sed -i "s/-e certbot\[dev\]/certbot[dev]==$version/" "$pkg_dir/local-oldest-requirements.txt" + sed -i "s/-e certbot/certbot[dev]==$version/" "$pkg_dir/local-oldest-requirements.txt" + git add "$pkg_dir/local-oldest-requirements.txt" + fi +done SetVersion() { ver="$1" @@ -90,7 +113,7 @@ SetVersion() { fi sed -i "s/^version.*/version = '$ver'/" $pkg_dir/setup.py done - init_file="certbot/__init__.py" + init_file="certbot/certbot/__init__.py" if [ $(grep -c '^__version' "$init_file") != 1 ]; then echo "Unexpected count of __version variables in $init_file" exit 1 @@ -102,8 +125,11 @@ SetVersion() { SetVersion "$version" +# Unset CERTBOT_OLDEST to prevent wheels from being built improperly due to +# conditionals like the one found in certbot-dns-dnsimple's setup.py file. +unset CERTBOT_OLDEST echo "Preparing sdists and wheels" -for pkg_dir in . $SUBPKGS_NO_CERTBOT +for pkg_dir in $SUBPKGS do cd $pkg_dir @@ -123,8 +149,7 @@ done mkdir "dist.$version" -mv dist "dist.$version/certbot" -for pkg_dir in $SUBPKGS_NO_CERTBOT +for pkg_dir in $SUBPKGS do mv $pkg_dir/dist "dist.$version/$pkg_dir/" done @@ -153,7 +178,7 @@ cd ~- # get a snapshot of the CLI help for the docs # We set CERTBOT_DOCS to use dummy values in example user-agent string. -CERTBOT_DOCS=1 certbot --help all > docs/cli-help.txt +CERTBOT_DOCS=1 certbot --help all > certbot/docs/cli-help.txt jws --help > acme/docs/jws-help.txt cd .. @@ -167,12 +192,12 @@ mkdir kgs kgs="kgs/$version" pip freeze | tee $kgs python ../tools/pip_install.py pytest -for module in $subpkgs_modules ; do +cd ~- +for module in $SUBPKGS ; do echo testing $module # use an empty configuration file rather than the one in the repo root - pytest -c <(echo '') --pyargs $module + pytest -c <(echo '') $module done -cd ~- # pin pip hashes of the things we just built for pkg in $SUBPKGS_IN_AUTO ; do @@ -221,8 +246,7 @@ mv letsencrypt-auto-source/letsencrypt-auto.asc letsencrypt-auto-source/certbot- cp -p letsencrypt-auto-source/letsencrypt-auto certbot-auto cp -p letsencrypt-auto-source/letsencrypt-auto letsencrypt-auto -git add certbot-auto letsencrypt-auto letsencrypt-auto-source docs/cli-help.txt -git diff --cached +git add certbot-auto letsencrypt-auto letsencrypt-auto-source certbot/docs/cli-help.txt while ! git commit --gpg-sign="$RELEASE_GPG_KEY" -m "Release $version"; do echo "Unable to sign the release commit using git." echo "You may have to configure git to use gpg2 by running:" @@ -241,17 +265,16 @@ echo gpg2 -U $RELEASE_GPG_KEY --detach-sign --armor $name.$rev.tar.xz cd ~- # Add master section to CHANGELOG.md -header=$(head -n 4 CHANGELOG.md) +header=$(head -n 4 certbot/CHANGELOG.md) body=$(sed s/nextversion/$nextversion/ tools/_changelog_top.txt) -footer=$(tail -n +5 CHANGELOG.md) +footer=$(tail -n +5 certbot/CHANGELOG.md) echo "$header $body -$footer" > CHANGELOG.md -git add CHANGELOG.md -git diff --cached -git commit -m "Add contents to CHANGELOG.md for next version" +$footer" > certbot/CHANGELOG.md +git add certbot/CHANGELOG.md +git commit -m "Add contents to certbot/CHANGELOG.md for next version" echo "New root: $root" echo "Test commands (in the letstest repo):" @@ -265,16 +288,5 @@ if [ "$RELEASE_BRANCH" = candidate-"$version" ] ; then SetVersion "$nextversion".dev0 letsencrypt-auto-source/build.py git add letsencrypt-auto-source/letsencrypt-auto - for pkg_dir in $SUBPKGS_NO_CERTBOT . - do - if [ -f "$pkg_dir/local-oldest-requirements.txt" ]; then - sed -i "s/-e acme\[dev\]/acme[dev]==$version/" "$pkg_dir/local-oldest-requirements.txt" - sed -i "s/-e acme/acme[dev]==$version/" "$pkg_dir/local-oldest-requirements.txt" - sed -i "s/-e \.\[dev\]/certbot[dev]==$version/" "$pkg_dir/local-oldest-requirements.txt" - sed -i "s/-e \./certbot[dev]==$version/" "$pkg_dir/local-oldest-requirements.txt" - git add "$pkg_dir/local-oldest-requirements.txt" - fi - done - git diff git commit -m "Bump version to $nextversion" fi diff --git a/tools/_venv_common.py b/tools/_venv_common.py old mode 100755 new mode 100644 index 540842773..c61385054 --- a/tools/_venv_common.py +++ b/tools/_venv_common.py @@ -12,14 +12,37 @@ VENV_NAME. from __future__ import print_function -import os -import shutil import glob -import time +import os +import re +import shutil import subprocess import sys -import re -import shlex +import time + +REQUIREMENTS = [ + '-e acme[dev]', + '-e certbot[dev,docs]', + '-e certbot-apache', + '-e certbot-dns-cloudflare', + '-e certbot-dns-cloudxns', + '-e certbot-dns-digitalocean', + '-e certbot-dns-dnsimple', + '-e certbot-dns-dnsmadeeasy', + '-e certbot-dns-gehirn', + '-e certbot-dns-google', + '-e certbot-dns-linode', + '-e certbot-dns-luadns', + '-e certbot-dns-nsone', + '-e certbot-dns-ovh', + '-e certbot-dns-rfc2136', + '-e certbot-dns-route53', + '-e certbot-dns-sakuracloud', + '-e certbot-nginx', + '-e letshelp-certbot', + '-e certbot-compatibility-test', + '-e certbot-ci', +] VERSION_PATTERN = re.compile(r'^(\d+)\.(\d+).*$') @@ -107,29 +130,35 @@ def subprocess_with_print(cmd, env=os.environ, shell=False): subprocess.check_call(cmd, env=env, shell=shell) -def get_venv_bin_path(venv_path): +def get_venv_python_path(venv_path): python_linux = os.path.join(venv_path, 'bin/python') if os.path.isfile(python_linux): - return os.path.abspath(os.path.dirname(python_linux)) + return os.path.abspath(python_linux) python_windows = os.path.join(venv_path, 'Scripts\\python.exe') if os.path.isfile(python_windows): - return os.path.abspath(os.path.dirname(python_windows)) + return os.path.abspath(python_windows) raise ValueError(( 'Error, could not find python executable in venv path {0}: is it a valid venv ?' .format(venv_path))) -def main(venv_name, venv_args, args): - """Creates a virtual environment and installs packages. +def prepare_venv_path(venv_name): + """Determines the venv path and prepares it for use. + + This function cleans up any Python eggs in the current working directory + and ensures the venv path is available for use. The path used is the + VENV_NAME environment variable if it is set and venv_name otherwise. If + there is already a directory at the desired path, the existing directory is + renamed by appending a timestamp to the directory name. :param str venv_name: The name or path at where the virtual - environment should be created. - :param str venv_args: Command line arguments for virtualenv - :param str args: Command line arguments that should be given to pip - to install packages - """ + environment should be created if VENV_NAME isn't set. + :returns: path where the virtual environment should be created + :rtype: str + + """ for path in glob.glob('*.egg-info'): if os.path.isdir(path): shutil.rmtree(path) @@ -145,21 +174,25 @@ def main(venv_name, venv_args, args): if os.path.isdir(venv_name): os.rename(venv_name, '{0}.{1}.bak'.format(venv_name, int(time.time()))) - command = [sys.executable, '-m', 'virtualenv', '--no-site-packages', '--setuptools', venv_name] - command.extend(shlex.split(venv_args)) - subprocess_with_print(command) + return venv_name - # We execute the following commands in the context of the virtual environment, to install - # the packages in it. To do so, we append the venv binary to the PATH that will be used for - # these commands. With this trick, correct python executable will be selected. - new_environ = os.environ.copy() - new_environ['PATH'] = os.pathsep.join([get_venv_bin_path(venv_name), new_environ['PATH']]) - subprocess_with_print('python {0}'.format('./letsencrypt-auto-source/pieces/pipstrap.py'), - env=new_environ, shell=True) - subprocess_with_print('python -m pip install --upgrade "setuptools>=30.3"', - env=new_environ, shell=True) - subprocess_with_print('python {0} {1}'.format('./tools/pip_install.py', ' '.join(args)), - env=new_environ, shell=True) + +def install_packages(venv_name, pip_args): + """Installs packages in the given venv. + + :param str venv_name: The name or path at where the virtual + environment should be created. + :param pip_args: Command line arguments that should be given to + pip to install packages + :type pip_args: `list` of `str` + + """ + # Using the python executable from venv, we ensure to execute following commands in this venv. + py_venv = get_venv_python_path(venv_name) + subprocess_with_print([py_venv, os.path.abspath('letsencrypt-auto-source/pieces/pipstrap.py')]) + command = [py_venv, os.path.abspath('tools/pip_install.py')] + command.extend(pip_args) + subprocess_with_print(command) if os.path.isdir(os.path.join(venv_name, 'bin')): # Linux/OSX specific @@ -176,9 +209,3 @@ def main(venv_name, venv_args, args): print('---------------------------------------------------------------------------') else: raise ValueError('Error, directory {0} is not a valid venv.'.format(venv_name)) - - -if __name__ == '__main__': - main('venv', - '', - sys.argv[1:]) diff --git a/tools/deactivate.py b/tools/deactivate.py index d43b84552..10c9ecd35 100644 --- a/tools/deactivate.py +++ b/tools/deactivate.py @@ -16,8 +16,8 @@ import os import sys from cryptography.hazmat.backends import default_backend -from cryptography.hazmat.primitives.asymmetric import rsa from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric import rsa import josepy as jose from acme import client as acme_client diff --git a/tools/deps.sh b/tools/deps.sh deleted file mode 100755 index e12f201a5..000000000 --- a/tools/deps.sh +++ /dev/null @@ -1,15 +0,0 @@ -#!/bin/sh -# -# Find all Python imports. -# -# ./tools/deps.sh certbot -# ./tools/deps.sh acme -# ./tools/deps.sh certbot-apache -# ... -# -# Manually compare the output with deps in setup.py. - -git grep -h -E '^(import|from.*import)' $1/ | \ - awk '{print $2}' | \ - grep -vE "^$1" | \ - sort -u diff --git a/tools/dev_constraints.txt b/tools/dev_constraints.txt index 88340cb00..265d967d8 100644 --- a/tools/dev_constraints.txt +++ b/tools/dev_constraints.txt @@ -1,77 +1,114 @@ -# Specifies Python package versions for development. +# Specifies Python package versions for development and building Docker images. # It includes in particular packages not specified in letsencrypt-auto's requirements file. # Some dev package versions specified here may be overridden by higher level constraints # files during tests (eg. letsencrypt-auto-source/pieces/dependency-requirements.txt). alabaster==0.7.10 apipkg==1.4 +appnope==0.1.0 asn1crypto==0.22.0 -astroid==1.3.5 +astroid==2.3.3 attrs==17.3.0 Babel==2.5.1 +backports.functools-lru-cache==1.5 backports.shutil-get-terminal-size==1.0.0 -boto3==1.9.36 -botocore==1.12.36 -cloudflare==1.5.1 -coverage==4.4.2 -decorator==4.1.2 -dns-lexicon==3.0.8 +backports.ssl-match-hostname==3.7.0.1 +bcrypt==3.1.6 +boto3==1.11.7 +botocore==1.14.7 +cached-property==1.5.1 +cloudflare==2.3.1 +codecov==2.0.15 +configparser==3.7.4 +contextlib2==0.6.0.post1 +coverage==4.5.4 +decorator==4.4.1 +dns-lexicon==3.3.17 dnspython==1.15.0 -docutils==0.12 +docker==3.7.2 +docker-compose==1.25.0 +docker-pycreds==0.4.0 +dockerpty==0.4.1 +docopt==0.6.2 +docutils==0.15.2 execnet==1.5.0 +functools32==3.2.3.post2 future==0.16.0 -futures==3.1.1 -google-api-python-client==1.5 +futures==3.3.0 +filelock==3.0.12 +google-api-python-client==1.5.5 httplib2==0.10.3 imagesize==0.7.1 -ipdb==0.10.2 -ipython==5.5.0 +importlib-metadata==0.23 +ipdb==0.12.3 +ipython==5.8.0 ipython-genutils==0.2.0 +isort==4.3.21 Jinja2==2.9.6 -jmespath==0.9.3 +jmespath==0.9.4 josepy==1.1.0 +jsonschema==2.6.0 +lazy-object-proxy==1.4.3 logger==1.4 logilab-common==1.4.1 MarkupSafe==1.0 -mypy==0.600 +mccabe==0.6.1 +more-itertools==5.0.0 +mypy==0.710 +mypy-extensions==0.4.3 ndg-httpsclient==0.3.2 -oauth2client==2.0.0 -pathlib2==2.3.0 -pexpect==4.2.1 -pickleshare==0.7.4 +oauth2client==4.0.0 +packaging==19.2 +paramiko==2.4.2 +pathlib2==2.3.5 +pexpect==4.7.0 +pickleshare==0.7.5 pkginfo==1.4.2 -pluggy==0.5.2 -prompt-toolkit==1.0.15 -ptyprocess==0.5.2 -py==1.4.34 +pluggy==0.13.0 +prompt-toolkit==1.0.18 +ptyprocess==0.6.0 +py==1.8.0 pyasn1==0.1.9 pyasn1-modules==0.0.10 Pygments==2.2.0 -pylint==1.4.2 +pylint==2.4.3 +# If pynsist version is upgraded, our NSIS template windows-installer/template.nsi +# must be upgraded if necessary using the new built-in one from pynsist. +pynacl==1.3.0 +pynsist==2.4 pytest==3.2.5 pytest-cov==2.5.1 pytest-forked==0.2 pytest-xdist==1.22.5 -python-dateutil==2.6.1 +pytest-sugar==0.9.2 +pytest-rerunfailures==4.2 +python-dateutil==2.8.1 python-digitalocean==1.11 +pywin32==227 PyYAML==3.13 repoze.sphinx.autointerface==0.8 requests-file==1.4.2 requests-toolbelt==0.8.0 rsa==3.4.2 -s3transfer==0.1.11 -scandir==1.6 +s3transfer==0.3.1 +scandir==1.10.0 simplegeneric==0.8.1 +singledispatch==3.4.0.3 snowballstemmer==1.2.1 Sphinx==1.7.5 sphinx-rtd-theme==0.2.4 sphinxcontrib-websupport==1.0.1 +texttable==0.9.1 tldextract==2.2.0 -tox==2.9.1 +toml==0.10.0 +tox==3.14.0 tqdm==4.19.4 -traitlets==4.3.2 +traitlets==4.3.3 twine==1.11.0 -typed-ast==1.1.0 +typed-ast==1.4.0 typing==3.6.4 -uritemplate==0.6 -virtualenv==15.1.0 -wcwidth==0.1.7 +uritemplate==3.0.0 +virtualenv==16.6.2 +wcwidth==0.1.8 +websocket-client==0.56.0 +wrapt==1.11.2 +zipp==0.6.0 diff --git a/tools/docker-warning.sh b/tools/docker-warning.sh deleted file mode 100755 index e4f5f40ee..000000000 --- a/tools/docker-warning.sh +++ /dev/null @@ -1,4 +0,0 @@ -#!/bin/sh -e -echo "Warning: This Docker image will soon be switching to Alpine Linux." >&2 -echo "You can switch now using the certbot/certbot repo on Docker Hub." >&2 -exec /opt/certbot/venv/bin/certbot $@ diff --git a/tools/extract_changelog.py b/tools/extract_changelog.py new file mode 100755 index 000000000..fb0b849aa --- /dev/null +++ b/tools/extract_changelog.py @@ -0,0 +1,41 @@ +#!/usr/bin/env python +from __future__ import print_function + +import os +import re +import sys + +CERTBOT_ROOT = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + +NEW_SECTION_PATTERN = re.compile(r'^##\s*[\d.]+\s*-\s*[\d-]+$') + + +def main(): + version = sys.argv[1] + + section_pattern = re.compile(r'^##\s*{0}\s*-\s*[\d-]+$' + .format(version.replace('.', '\\.'))) + + with open(os.path.join(CERTBOT_ROOT, 'certbot', 'CHANGELOG.md')) as file_h: + lines = file_h.read().splitlines() + + changelog = [] + + i = 0 + while i < len(lines): + if section_pattern.match(lines[i]): + i = i + 1 + while i < len(lines): + if NEW_SECTION_PATTERN.match(lines[i]): + break + changelog.append(lines[i]) + i = i + 1 + i = i + 1 + + changelog = [entry for entry in changelog if entry] + + print('\n'.join(changelog)) + + +if __name__ == '__main__': + main() diff --git a/tools/install_and_test.py b/tools/install_and_test.py index b15c8eca5..192708957 100755 --- a/tools/install_and_test.py +++ b/tools/install_and_test.py @@ -8,19 +8,16 @@ from __future__ import print_function import os -import sys -import tempfile -import shutil -import subprocess import re +import subprocess +import sys -SKIP_PROJECTS_ON_WINDOWS = [ - 'certbot-apache', 'certbot-nginx', 'certbot-postfix', 'letshelp-certbot'] +SKIP_PROJECTS_ON_WINDOWS = ['certbot-apache', 'letshelp-certbot'] -def call_with_print(command, cwd=None): +def call_with_print(command): print(command) - subprocess.check_call(command, shell=True, cwd=cwd or os.getcwd()) + subprocess.check_call(command, shell=True) def main(args): @@ -42,16 +39,8 @@ def main(args): call_with_print(' '.join(current_command)) pkg = re.sub(r'\[\w+\]', '', requirement) - if pkg == '.': - pkg = 'certbot' - - temp_cwd = tempfile.mkdtemp() - shutil.copy2("pytest.ini", temp_cwd) - try: - call_with_print(' '.join([ - sys.executable, '-m', 'pytest', '--pyargs', pkg.replace('-', '_')]), cwd=temp_cwd) - finally: - shutil.rmtree(temp_cwd) + call_with_print(' '.join([ + sys.executable, '-m', 'pytest', pkg])) if __name__ == '__main__': main(sys.argv[1:]) diff --git a/tools/merge_requirements.py b/tools/merge_requirements.py index 4205e6bcf..0d41d12c4 100755 --- a/tools/merge_requirements.py +++ b/tools/merge_requirements.py @@ -10,27 +10,36 @@ from __future__ import print_function import sys -def read_file(file_path): - """Reads in a Python requirements file. +def process_entries(entries): + """ Ignore empty lines, comments and editable requirements - :param str file_path: path to requirements file + :param list entries: List of entries :returns: mapping from a project to its pinned version :rtype: dict - """ data = {} - with open(file_path) as file_h: - for line in file_h: - line = line.strip() - if line and not line.startswith('#') and not line.startswith('-e'): - project, version = line.split('==') - if not version: - raise ValueError("Unexpected syntax '{0}'".format(line)) - data[project] = version + for e in entries: + e = e.strip() + if e and not e.startswith('#') and not e.startswith('-e'): + project, version = e.split('==') + if not version: + raise ValueError("Unexpected syntax '{0}'".format(e)) + data[project] = version return data +def read_file(file_path): + """Reads in a Python requirements file. + + :param str file_path: path to requirements file + + :returns: list of entries in the file + :rtype: list + + """ + with open(file_path) as file_h: + return file_h.readlines() def output_requirements(requirements): """Prepare print requirements to stdout. @@ -46,14 +55,25 @@ def main(*paths): """Merges multiple requirements files together and prints the result. Requirement files specified later in the list take precedence over earlier - files. + files. Files are read from file paths passed from the command line arguments. - :param tuple paths: paths to requirements files + If no command line arguments are defined, data is read from stdin instead. + + :param tuple paths: paths to requirements files provided on command line """ data = {} - for path in paths: - data.update(read_file(path)) + if paths: + for path in paths: + data.update(process_entries(read_file(path))) + else: + # Need to check if interactive to avoid blocking if nothing is piped + if not sys.stdin.isatty(): + stdin_data = [] + for line in sys.stdin: + stdin_data.append(line) + data.update(process_entries(stdin_data)) + return output_requirements(data) diff --git a/tools/oldest_constraints.txt b/tools/oldest_constraints.txt index e48d6b13c..c5a5c5aa0 100644 --- a/tools/oldest_constraints.txt +++ b/tools/oldest_constraints.txt @@ -16,6 +16,7 @@ pyOpenSSL==0.13.1 pyparsing==1.5.6 pyRFC3339==1.0 python-augeas==0.5.0 +oauth2client==4.0.0 six==1.9.0 # setuptools 0.9.8 is the actual version packaged, but some other dependencies # in this file require setuptools>=1.0 and there are no relevant changes for us @@ -35,11 +36,12 @@ idna==2.0 pbr==1.8.0 pytz==2012rc0 +# Debian Buster constraints +google-api-python-client==1.5.5 + # Our setup.py constraints cloudflare==1.5.1 cryptography==1.2.3 -google-api-python-client==1.5 -oauth2client==2.0 parsedatetime==1.3 pyparsing==1.5.5 python-digitalocean==1.11 @@ -51,6 +53,7 @@ funcsigs==0.4 zope.hookable==4.0.4 # Ubuntu Bionic constraints. +distro==1.0.1 # Lexicon oldest constraint is overridden appropriately on relevant DNS provider plugins # using their local-oldest-requirements.txt dns-lexicon==2.2.1 diff --git a/tools/pip_install.py b/tools/pip_install.py index dd6302b48..0a3961384 100755 --- a/tools/pip_install.py +++ b/tools/pip_install.py @@ -8,17 +8,19 @@ # CERTBOT_OLDEST is set, this script must be run with `-e ` and # no other arguments. -from __future__ import print_function, absolute_import +from __future__ import absolute_import +from __future__ import print_function -import subprocess import os -import sys import re import shutil +import subprocess +import sys import tempfile import merge_requirements as merge_module import readlink +import strip_hashes def find_tools_path(): @@ -47,10 +49,8 @@ def certbot_normal_processing(tools_path, test_constraints): with open(certbot_requirements, 'r') as fd: data = fd.readlines() with open(test_constraints, 'w') as fd: - for line in data: - search = re.search(r'^(\S*==\S*).*$', line) - if search: - fd.write('{0}{1}'.format(search.group(1), os.linesep)) + data = "\n".join(strip_hashes.process_entries(data)) + fd.write(data) def merge_requirements(tools_path, requirements, test_constraints, all_constraints): @@ -70,9 +70,15 @@ def merge_requirements(tools_path, requirements, test_constraints, all_constrain fd.write(merged_requirements) -def call_with_print(command, cwd=None): +def call_with_print(command): print(command) - subprocess.check_call(command, shell=True, cwd=cwd or os.getcwd()) + subprocess.check_call(command, shell=True) + + +def pip_install_with_print(args_str): + command = '"{0}" -m pip install --disable-pip-version-check {1}'.format(sys.executable, + args_str) + call_with_print(command) def main(args): @@ -90,8 +96,7 @@ def main(args): if os.environ.get('CERTBOT_NO_PIN') == '1': # With unpinned dependencies, there is no constraint - call_with_print('"{0}" -m pip install {1}' - .format(sys.executable, ' '.join(args))) + pip_install_with_print(' '.join(args)) else: # Otherwise, we merge requirements to build the constraints and pin dependencies requirements = None @@ -101,12 +106,19 @@ def main(args): certbot_normal_processing(tools_path, test_constraints) merge_requirements(tools_path, requirements, test_constraints, all_constraints) - if requirements: - call_with_print('"{0}" -m pip install --constraint "{1}" --requirement "{2}"' - .format(sys.executable, all_constraints, requirements)) + if requirements: # This branch is executed during the oldest tests + # First step, install the transitive dependencies of oldest requirements + # in respect with oldest constraints. + pip_install_with_print('--constraint "{0}" --requirement "{1}"' + .format(all_constraints, requirements)) + # Second step, ensure that oldest requirements themselves are effectively + # installed using --force-reinstall, and avoid corner cases like the one described + # in https://github.com/certbot/certbot/issues/7014. + pip_install_with_print('--force-reinstall --no-deps --requirement "{0}"' + .format(requirements)) - call_with_print('"{0}" -m pip install --constraint "{1}" {2}' - .format(sys.executable, all_constraints, ' '.join(args))) + pip_install_with_print('--constraint "{0}" {1}'.format( + all_constraints, ' '.join(args))) finally: if os.environ.get('TRAVIS'): print('travis_fold:end:install_certbot_deps') diff --git a/tools/pip_install_editable.py b/tools/pip_install_editable.py index 8eaf3a9fa..3f7c02ba9 100755 --- a/tools/pip_install_editable.py +++ b/tools/pip_install_editable.py @@ -8,6 +8,7 @@ import sys import pip_install + def main(args): new_args = [] for arg in args: diff --git a/tools/readlink.py b/tools/readlink.py index 0199ce184..446c8ebdc 100755 --- a/tools/readlink.py +++ b/tools/readlink.py @@ -11,6 +11,7 @@ from __future__ import print_function import os import sys + def main(link): return os.path.realpath(link) diff --git a/tools/simple_http_server.py b/tools/simple_http_server.py index 14ac9a3d3..24c55962d 100755 --- a/tools/simple_http_server.py +++ b/tools/simple_http_server.py @@ -1,9 +1,13 @@ #!/usr/bin/env python -"""A version of Python 2.x's SimpleHTTPServer that flushes its output.""" -from BaseHTTPServer import HTTPServer -from SimpleHTTPServer import SimpleHTTPRequestHandler +"""A version of Python's SimpleHTTPServer that flushes its output.""" import sys +try: + from http.server import HTTPServer, SimpleHTTPRequestHandler +except ImportError: + from BaseHTTPServer import HTTPServer + from SimpleHTTPServer import SimpleHTTPRequestHandler + def serve_forever(port=0): """Spins up an HTTP server on all interfaces and the given port. diff --git a/tools/sphinx-quickstart.sh b/tools/sphinx-quickstart.sh index 72dc9e200..35a7f7fad 100755 --- a/tools/sphinx-quickstart.sh +++ b/tools/sphinx-quickstart.sh @@ -14,7 +14,7 @@ sed -i -e "s|\# import os|import os|" conf.py sed -i -e "s|\# needs_sphinx = '1.0'|needs_sphinx = '1.0'|" conf.py sed -i -e "s|intersphinx_mapping = {'https://docs.python.org/': None}|intersphinx_mapping = {\n 'python': ('https://docs.python.org/', None),\n 'acme': ('https://acme-python.readthedocs.org/en/latest/', None),\n 'certbot': ('https://certbot.eff.org/docs/', None),\n}|" conf.py sed -i -e "s|html_theme = 'alabaster'|\n# http://docs.readthedocs.org/en/latest/theme.html#how-do-i-use-this-locally-and-on-read-the-docs\n# on_rtd is whether we are on readthedocs.org\non_rtd = os.environ.get('READTHEDOCS', None) == 'True'\nif not on_rtd: # only import and set the theme if we're building docs locally\n import sphinx_rtd_theme\n html_theme = 'sphinx_rtd_theme'\n html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]\n# otherwise, readthedocs.org uses their theme by default, so no need to specify it|" conf.py -sed -i -e "s|# Add any paths that contain templates here, relative to this directory.|autodoc_member_order = 'bysource'\nautodoc_default_flags = ['show-inheritance', 'private-members']\n\n# Add any paths that contain templates here, relative to this directory.|" conf.py +sed -i -e "s|# Add any paths that contain templates here, relative to this directory.|autodoc_member_order = 'bysource'\nautodoc_default_flags = ['show-inheritance']\n\n# Add any paths that contain templates here, relative to this directory.|" conf.py sed -i -e "s|# The name of the Pygments (syntax highlighting) style to use.|default_role = 'py:obj'\n\n# The name of the Pygments (syntax highlighting) style to use.|" conf.py echo "/_build/" >> .gitignore echo "================= diff --git a/tools/strip_hashes.py b/tools/strip_hashes.py new file mode 100755 index 000000000..988e72eb8 --- /dev/null +++ b/tools/strip_hashes.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python +"""Removes hash information from requirement files passed to it as file path +arguments or simply piped to stdin.""" + +import re +import sys + + +def process_entries(entries): + """Strips off hash strings from dependencies. + + :param list entries: List of entries + + :returns: list of dependencies without hashes + :rtype: list + """ + out_lines = [] + for e in entries: + e = e.strip() + search = re.search(r'^(\S*==\S*).*$', e) + if search: + out_lines.append(search.group(1)) + return out_lines + +def main(*paths): + """ + Reads dependency definitions from a (list of) file(s) provided on the + command line. If no command line arguments are present, data is read from + stdin instead. + + Hashes are removed from returned entries. + """ + + deps = [] + if paths: + for path in paths: + with open(path) as file_h: + deps += process_entries(file_h.readlines()) + else: + # Need to check if interactive to avoid blocking if nothing is piped + if not sys.stdin.isatty(): + stdin_data = [] + for line in sys.stdin: + stdin_data.append(line) + deps += process_entries(stdin_data) + + return "\n".join(deps) + +if __name__ == '__main__': + print(main(*sys.argv[1:])) # pylint: disable=star-args diff --git a/tools/venv.py b/tools/venv.py index 93b012e76..f99386eff 100755 --- a/tools/venv.py +++ b/tools/venv.py @@ -1,41 +1,37 @@ #!/usr/bin/env python # Developer virtualenv setup for Certbot client import os +import sys import _venv_common -REQUIREMENTS = [ - '-e acme[dev]', - '-e .[dev,docs]', - '-e certbot-apache', - '-e certbot-dns-cloudflare', - '-e certbot-dns-cloudxns', - '-e certbot-dns-digitalocean', - '-e certbot-dns-dnsimple', - '-e certbot-dns-dnsmadeeasy', - '-e certbot-dns-gehirn', - '-e certbot-dns-google', - '-e certbot-dns-linode', - '-e certbot-dns-luadns', - '-e certbot-dns-nsone', - '-e certbot-dns-ovh', - '-e certbot-dns-rfc2136', - '-e certbot-dns-route53', - '-e certbot-dns-sakuracloud', - '-e certbot-nginx', - '-e certbot-postfix', - '-e letshelp-certbot', - '-e certbot-compatibility-test', -] + +def create_venv(venv_path): + """Create a Python 2 virtual environment at venv_path. + + :param str venv_path: path where the venv should be created + + """ + python2 = _venv_common.find_python_executable(2) + command = [sys.executable, '-m', 'virtualenv', '--python', python2, venv_path] + + environ = os.environ.copy() + environ['VIRTUALENV_NO_DOWNLOAD'] = '1' + _venv_common.subprocess_with_print(command, environ) -def main(): +def main(pip_args=None): if os.name == 'nt': raise ValueError('Certbot for Windows is not supported on Python 2.x.') - venv_args = '--python "{0}"'.format(_venv_common.find_python_executable(2)) - _venv_common.main('venv', venv_args, REQUIREMENTS) + venv_path = _venv_common.prepare_venv_path('venv') + create_venv(venv_path) + + if not pip_args: + pip_args = _venv_common.REQUIREMENTS + + _venv_common.install_packages(venv_path, pip_args) if __name__ == '__main__': - main() + main(sys.argv[1:]) diff --git a/tools/venv3.py b/tools/venv3.py index c2374ba5a..7ead82bd5 100755 --- a/tools/venv3.py +++ b/tools/venv3.py @@ -1,36 +1,30 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # Developer virtualenv setup for Certbot client +import sys + import _venv_common -REQUIREMENTS = [ - '-e acme[dev]', - '-e .[dev,docs]', - '-e certbot-apache', - '-e certbot-dns-cloudflare', - '-e certbot-dns-cloudxns', - '-e certbot-dns-digitalocean', - '-e certbot-dns-dnsimple', - '-e certbot-dns-dnsmadeeasy', - '-e certbot-dns-gehirn', - '-e certbot-dns-google', - '-e certbot-dns-linode', - '-e certbot-dns-luadns', - '-e certbot-dns-nsone', - '-e certbot-dns-ovh', - '-e certbot-dns-rfc2136', - '-e certbot-dns-route53', - '-e certbot-dns-sakuracloud', - '-e certbot-nginx', - '-e certbot-postfix', - '-e letshelp-certbot', - '-e certbot-compatibility-test', -] + +def create_venv(venv_path): + """Create a Python 3 virtual environment at venv_path. + + :param str venv_path: path where the venv should be created + + """ + python3 = _venv_common.find_python_executable(3) + command = [python3, '-m', 'venv', venv_path] + _venv_common.subprocess_with_print(command) -def main(): - venv_args = '--python "{0}"'.format(_venv_common.find_python_executable(3)) - _venv_common.main('venv3', venv_args, REQUIREMENTS) +def main(pip_args=None): + venv_path = _venv_common.prepare_venv_path('venv3') + create_venv(venv_path) + + if not pip_args: + pip_args = _venv_common.REQUIREMENTS + ['-e certbot[dev3]'] + + _venv_common.install_packages(venv_path, pip_args) if __name__ == '__main__': - main() + main(sys.argv[1:]) diff --git a/tox.cover.py b/tox.cover.py index 008424641..0ef5c0d07 100755 --- a/tox.cover.py +++ b/tox.cover.py @@ -1,7 +1,7 @@ #!/usr/bin/env python import argparse -import subprocess import os +import subprocess import sys DEFAULT_PACKAGES = [ @@ -9,14 +9,14 @@ DEFAULT_PACKAGES = [ 'certbot_dns_digitalocean', 'certbot_dns_dnsimple', 'certbot_dns_dnsmadeeasy', 'certbot_dns_gehirn', 'certbot_dns_google', 'certbot_dns_linode', 'certbot_dns_luadns', 'certbot_dns_nsone', 'certbot_dns_ovh', 'certbot_dns_rfc2136', 'certbot_dns_route53', - 'certbot_dns_sakuracloud', 'certbot_nginx', 'certbot_postfix', 'letshelp_certbot'] + 'certbot_dns_sakuracloud', 'certbot_nginx', 'letshelp_certbot'] COVER_THRESHOLDS = { - 'certbot': {'linux': 98, 'windows': 94}, + 'certbot': {'linux': 96, 'windows': 96}, 'acme': {'linux': 100, 'windows': 99}, 'certbot_apache': {'linux': 100, 'windows': 100}, 'certbot_dns_cloudflare': {'linux': 98, 'windows': 98}, - 'certbot_dns_cloudxns': {'linux': 99, 'windows': 99}, + 'certbot_dns_cloudxns': {'linux': 98, 'windows': 98}, 'certbot_dns_digitalocean': {'linux': 98, 'windows': 98}, 'certbot_dns_dnsimple': {'linux': 98, 'windows': 98}, 'certbot_dns_dnsmadeeasy': {'linux': 99, 'windows': 99}, @@ -30,12 +30,11 @@ COVER_THRESHOLDS = { 'certbot_dns_route53': {'linux': 92, 'windows': 92}, 'certbot_dns_sakuracloud': {'linux': 97, 'windows': 97}, 'certbot_nginx': {'linux': 97, 'windows': 97}, - 'certbot_postfix': {'linux': 100, 'windows': 100}, 'letshelp_certbot': {'linux': 100, 'windows': 100} } -SKIP_PROJECTS_ON_WINDOWS = [ - 'certbot-apache', 'certbot-nginx', 'certbot-postfix', 'letshelp-certbot'] +SKIP_PROJECTS_ON_WINDOWS = ['certbot-apache', 'letshelp-certbot'] + def cover(package): threshold = COVER_THRESHOLDS.get(package)['windows' if os.name == 'nt' else 'linux'] @@ -48,17 +47,18 @@ def cover(package): .format(pkg_dir))) return - subprocess.check_call([sys.executable, '-m', 'pytest', '--pyargs', - '--cov', pkg_dir, '--cov-append', '--cov-report=', package]) + subprocess.check_call([sys.executable, '-m', 'pytest', + '--cov', pkg_dir, '--cov-append', '--cov-report=', pkg_dir]) subprocess.check_call([ sys.executable, '-m', 'coverage', 'report', '--fail-under', str(threshold), '--include', '{0}/*'.format(pkg_dir), '--show-missing']) + def main(): description = """ -This script is used by tox.ini (and thus by Travis CI and AppVeyor) in order -to generate separate stats for each package. It should be removed once those -packages are moved to a separate repo. +This script is used by tox.ini (and thus by Travis CI and Azure Pipelines) in +order to generate separate stats for each package. It should be removed once +those packages are moved to a separate repo. Option -e makes sure we fail fast and don't submit to codecov.""" parser = argparse.ArgumentParser(description=description) @@ -77,5 +77,6 @@ Option -e makes sure we fail fast and don't submit to codecov.""" for package in packages: cover(package) + if __name__ == '__main__': main() diff --git a/tox.ini b/tox.ini index b66e330da..6d9814192 100644 --- a/tox.ini +++ b/tox.ini @@ -31,17 +31,16 @@ dns_packages = certbot-dns-sakuracloud all_packages = acme[dev] \ - .[dev] \ + certbot[dev] \ certbot-apache \ {[base]dns_packages} \ certbot-nginx \ - certbot-postfix \ letshelp-certbot install_packages = python {toxinidir}/tools/pip_install_editable.py {[base]all_packages} source_paths = acme/acme - certbot + certbot/certbot certbot-apache/certbot_apache certbot-compatibility-test/certbot_compatibility_test certbot-dns-cloudflare/certbot_dns_cloudflare @@ -59,7 +58,6 @@ source_paths = certbot-dns-route53/certbot_dns_route53 certbot-dns-sakuracloud/certbot_dns_sakuracloud certbot-nginx/certbot_nginx - certbot-postfix/certbot_postfix letshelp-certbot/letshelp_certbot tests/lock_test.py @@ -69,6 +67,9 @@ passenv = commands = {[base]install_and_test} {[base]all_packages} python tests/lock_test.py +# We always recreate the virtual environment to avoid problems like +# https://github.com/certbot/certbot/issues/7745. +recreate = true setenv = PYTEST_ADDOPTS = {env:PYTEST_ADDOPTS:--numprocesses auto} PYTHONHASHSEED = 0 @@ -94,7 +95,7 @@ setenv = [testenv:py27-certbot-oldest] commands = - {[base]install_and_test} .[dev] + {[base]install_and_test} certbot[dev] setenv = {[testenv:py27-oldest]setenv} @@ -111,17 +112,6 @@ commands = setenv = {[testenv:py27-oldest]setenv} -[testenv:py27-postfix-oldest] -commands = - {[base]install_and_test} certbot-postfix -setenv = - {[testenv:py27-oldest]setenv} - -[testenv:py27_install] -basepython = python2.7 -commands = - {[base]install_packages} - [testenv:py27-cover] basepython = python2.7 commands = @@ -135,44 +125,37 @@ commands = python tox.cover.py [testenv:lint] -basepython = python2.7 +basepython = python3 # separating into multiple invocations disables cross package # duplicate code checking; if one of the commands fails, others will # continue, but tox return code will reflect previous error commands = {[base]install_packages} + {[base]pip_install} certbot[dev3] python -m pylint --reports=n --rcfile=.pylintrc {[base]source_paths} [testenv:mypy] basepython = python3 commands = {[base]install_packages} - {[base]pip_install} .[dev3] + {[base]pip_install} certbot[dev3] mypy {[base]source_paths} [testenv:apacheconftest] -#basepython = python2.7 commands = - {[base]pip_install} acme . certbot-apache certbot-compatibility-test - {toxinidir}/certbot-apache/certbot_apache/tests/apache-conf-files/apache-conf-test --debian-modules + {[base]pip_install} acme certbot certbot-apache certbot-compatibility-test + {toxinidir}/certbot-apache/tests/apache-conf-files/apache-conf-test --debian-modules passenv = SERVER [testenv:apacheconftest-with-pebble] commands = - {toxinidir}/tests/pebble-fetch.sh - {[testenv:apacheconftest]commands} -passenv = - HOME - GOPATH - PEBBLEPATH - PEBBLE_STRICT -setenv = - SERVER=https://localhost:14000/dir + {[base]pip_install} acme certbot certbot-apache certbot-ci certbot-compatibility-test + {toxinidir}/certbot-apache/tests/apache-conf-files/apache-conf-test-pebble.py --debian-modules [testenv:nginxroundtrip] commands = - {[base]pip_install} acme . certbot-apache certbot-nginx + {[base]pip_install} acme certbot certbot-apache certbot-nginx python certbot-compatibility-test/nginx/roundtrip.py certbot-compatibility-test/nginx/nginx-roundtrip-testdata # This is a duplication of the command line in testenv:le_auto to @@ -201,12 +184,11 @@ whitelist_externals = passenv = DOCKER_* -[testenv:le_auto_trusty] -# At the moment, this tests under Python 2.7 only, as only that version is -# readily available on the Trusty Docker image. +[testenv:le_auto_xenial] +# At the moment, this tests under Python 2.7 only. commands = python {toxinidir}/tests/modification-check.py - docker build -f letsencrypt-auto-source/Dockerfile.trusty -t lea letsencrypt-auto-source + docker build -f letsencrypt-auto-source/Dockerfile.xenial -t lea letsencrypt-auto-source docker run --rm -t -i lea whitelist_externals = docker @@ -214,15 +196,6 @@ passenv = DOCKER_* TRAVIS_BRANCH -[testenv:le_auto_xenial] -# At the moment, this tests under Python 2.7 only. -commands = - docker build -f letsencrypt-auto-source/Dockerfile.xenial -t lea letsencrypt-auto-source - docker run --rm -t -i lea -whitelist_externals = - docker -passenv = DOCKER_* - [testenv:le_auto_jessie] # At the moment, this tests under Python 2.7 only, as only that version is # readily available on the Wheezy Docker image. @@ -237,7 +210,17 @@ passenv = DOCKER_* # At the moment, this tests under Python 2.6 only, as only that version is # readily available on the CentOS 6 Docker image. commands = - docker build -f letsencrypt-auto-source/Dockerfile.centos6 -t lea letsencrypt-auto-source + docker build -f letsencrypt-auto-source/Dockerfile.redhat6 --build-arg REDHAT_DIST_FLAVOR=centos -t lea letsencrypt-auto-source + docker run --rm -t -i lea +whitelist_externals = + docker +passenv = DOCKER_* + +[testenv:le_auto_oraclelinux6] +# At the moment, this tests under Python 2.6 only, as only that version is +# readily available on the Oracle Linux 6 Docker image. +commands = + docker build -f letsencrypt-auto-source/Dockerfile.redhat6 --build-arg REDHAT_DIST_FLAVOR=oraclelinux -t lea letsencrypt-auto-source docker run --rm -t -i lea whitelist_externals = docker @@ -251,3 +234,88 @@ commands = whitelist_externals = docker-compose passenv = DOCKER_* + +[testenv:integration] +commands = + {[base]pip_install} acme certbot certbot-nginx certbot-ci + pytest certbot-ci/certbot_integration_tests \ + --acme-server={env:ACME_SERVER:pebble} \ + --cov=acme --cov=certbot --cov=certbot_nginx --cov-report= \ + --cov-config=certbot-ci/certbot_integration_tests/.coveragerc + coverage report --include 'certbot/*' --show-missing --fail-under=65 + coverage report --include 'certbot-nginx/*' --show-missing --fail-under=74 +passenv = DOCKER_* + +[testenv:integration-certbot] +commands = + {[base]pip_install} acme certbot certbot-ci + pytest certbot-ci/certbot_integration_tests/certbot_tests \ + --acme-server={env:ACME_SERVER:pebble} \ + --cov=acme --cov=certbot --cov-report= \ + --cov-config=certbot-ci/certbot_integration_tests/.coveragerc + coverage report --include 'certbot/*' --show-missing --fail-under=62 + +[testenv:integration-certbot-oldest] +commands = + {[base]pip_install} certbot + {[base]pip_install} certbot-ci + pytest certbot-ci/certbot_integration_tests/certbot_tests \ + --acme-server={env:ACME_SERVER:pebble} +passenv = DOCKER_* +setenv = {[testenv:py27-oldest]setenv} + +[testenv:integration-nginx-oldest] +commands = + {[base]pip_install} certbot-nginx + {[base]pip_install} certbot-ci + pytest certbot-ci/certbot_integration_tests/nginx_tests \ + --acme-server={env:ACME_SERVER:pebble} +passenv = DOCKER_* +setenv = {[testenv:py27-oldest]setenv} + +[testenv:travis-test-farm-tests-base] +changedir = tests/letstest +commands = + ./travis-setup.sh +deps = -rtests/letstest/requirements.txt +passenv = + AWS_* + TRAVIS_* + encrypted_* +setenv = AWS_DEFAULT_REGION=us-east-1 + +[testenv:travis-test-farm-apache2] +changedir = {[testenv:travis-test-farm-tests-base]changedir} +commands = + {[testenv:travis-test-farm-tests-base]commands} + python multitester.py apache2_targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_apache2.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH} +deps = {[testenv:travis-test-farm-tests-base]deps} +passenv = {[testenv:travis-test-farm-tests-base]passenv} +setenv = {[testenv:travis-test-farm-tests-base]setenv} + +[testenv:travis-test-farm-leauto-upgrades] +changedir = {[testenv:travis-test-farm-tests-base]changedir} +commands = + {[testenv:travis-test-farm-tests-base]commands} + python multitester.py targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_leauto_upgrades.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH} +deps = {[testenv:travis-test-farm-tests-base]deps} +passenv = {[testenv:travis-test-farm-tests-base]passenv} +setenv = {[testenv:travis-test-farm-tests-base]setenv} + +[testenv:travis-test-farm-certonly-standalone] +changedir = {[testenv:travis-test-farm-tests-base]changedir} +commands = + {[testenv:travis-test-farm-tests-base]commands} + python multitester.py targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_letsencrypt_auto_certonly_standalone.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH} +deps = {[testenv:travis-test-farm-tests-base]deps} +passenv = {[testenv:travis-test-farm-tests-base]passenv} +setenv = {[testenv:travis-test-farm-tests-base]setenv} + +[testenv:travis-test-farm-sdists] +changedir = {[testenv:travis-test-farm-tests-base]changedir} +commands = + {[testenv:travis-test-farm-tests-base]commands} + python multitester.py targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_sdists.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH} +deps = {[testenv:travis-test-farm-tests-base]deps} +passenv = {[testenv:travis-test-farm-tests-base]passenv} +setenv = {[testenv:travis-test-farm-tests-base]setenv} diff --git a/windows-installer/.gitignore b/windows-installer/.gitignore new file mode 100644 index 000000000..a1a48d6b8 --- /dev/null +++ b/windows-installer/.gitignore @@ -0,0 +1,2 @@ +build +build.* diff --git a/windows-installer/certbot.ico b/windows-installer/certbot.ico new file mode 100644 index 000000000..364c32098 Binary files /dev/null and b/windows-installer/certbot.ico differ diff --git a/windows-installer/construct.py b/windows-installer/construct.py new file mode 100644 index 000000000..f0724f5f4 --- /dev/null +++ b/windows-installer/construct.py @@ -0,0 +1,192 @@ +#!/usr/bin/env python3 +import contextlib +import ctypes +import os +import shutil +import struct +import subprocess +import sys +import tempfile +import time + +PYTHON_VERSION = (3, 7, 4) +PYTHON_BITNESS = 32 +PYWIN32_VERSION = 227 # do not forget to edit pywin32 dependency accordingly in setup.py +NSIS_VERSION = '3.04' + + +def main(): + build_path, repo_path, venv_path, venv_python = _prepare_environment() + + _copy_assets(build_path, repo_path) + + installer_cfg_path = _generate_pynsist_config(repo_path, build_path) + + _prepare_build_tools(venv_path, venv_python, repo_path) + _compile_wheels(repo_path, build_path, venv_python) + _build_installer(installer_cfg_path, venv_path) + + print('Done') + + +def _build_installer(installer_cfg_path, venv_path): + print('Build the installer') + subprocess.check_call([os.path.join(venv_path, 'Scripts', 'pynsist.exe'), installer_cfg_path]) + + +def _compile_wheels(repo_path, build_path, venv_python): + print('Compile wheels') + + wheels_path = os.path.join(build_path, 'wheels') + os.makedirs(wheels_path) + + certbot_packages = ['acme', 'certbot'] + # Uncomment following line to include all DNS plugins in the installer + # certbot_packages.extend([name for name in os.listdir(repo_path) if name.startswith('certbot-dns-')]) + wheels_project = [os.path.join(repo_path, package) for package in certbot_packages] + + with _prepare_constraints(repo_path) as constraints_file_path: + command = [venv_python, '-m', 'pip', 'wheel', '-w', wheels_path, '--constraint', constraints_file_path] + command.extend(wheels_project) + subprocess.check_call(command) + + +def _prepare_build_tools(venv_path, venv_python, repo_path): + print('Prepare build tools') + subprocess.check_call([sys.executable, '-m', 'venv', venv_path]) + subprocess.check_call([venv_python, os.path.join(repo_path, 'letsencrypt-auto-source', 'pieces', 'pipstrap.py')]) + subprocess.check_call([venv_python, os.path.join(repo_path, 'tools', 'pip_install.py'), 'pynsist']) + subprocess.check_call(['choco', 'upgrade', '--allow-downgrade', '-y', 'nsis', '--version', NSIS_VERSION]) + + +@contextlib.contextmanager +def _prepare_constraints(repo_path): + requirements = os.path.join(repo_path, 'letsencrypt-auto-source', 'pieces', 'dependency-requirements.txt') + constraints = subprocess.check_output( + [sys.executable, os.path.join(repo_path, 'tools', 'strip_hashes.py'), requirements], + universal_newlines=True) + workdir = tempfile.mkdtemp() + try: + constraints_file_path = os.path.join(workdir, 'constraints.txt') + with open(constraints_file_path, 'a') as file_h: + file_h.write(constraints) + file_h.write('pywin32=={0}'.format(PYWIN32_VERSION)) + yield constraints_file_path + finally: + shutil.rmtree(workdir) + + +def _copy_assets(build_path, repo_path): + print('Copy assets') + if os.path.exists(build_path): + os.rename(build_path, '{0}.{1}.bak'.format(build_path, int(time.time()))) + os.makedirs(build_path) + shutil.copy(os.path.join(repo_path, 'windows-installer', 'certbot.ico'), build_path) + shutil.copy(os.path.join(repo_path, 'windows-installer', 'run.bat'), build_path) + shutil.copy(os.path.join(repo_path, 'windows-installer', 'template.nsi'), build_path) + shutil.copy(os.path.join(repo_path, 'windows-installer', 'renew-up.ps1'), build_path) + shutil.copy(os.path.join(repo_path, 'windows-installer', 'renew-down.ps1'), build_path) + + +def _generate_pynsist_config(repo_path, build_path): + print('Generate pynsist configuration') + + pywin32_paths_file = os.path.join(build_path, 'pywin32_paths.py') + + # Pywin32 uses non-standard folders to hold its packages. We need to instruct pynsist bootstrap + # explicitly to add them into sys.path. This is done with a custom "pywin32_paths.py" that is + # referred in the pynsist configuration as an "extra_preamble". + # Reference example: https://github.com/takluyver/pynsist/tree/master/examples/pywebview + with open(pywin32_paths_file, 'w') as file_h: + file_h.write('''\ +pkgdir = os.path.join(os.path.dirname(installdir), 'pkgs') + +sys.path.extend([ + os.path.join(pkgdir, 'win32'), + os.path.join(pkgdir, 'win32', 'lib'), +]) + +# Preload pywintypes and pythoncom +pwt = os.path.join(pkgdir, 'pywin32_system32', 'pywintypes{0}{1}.dll') +pcom = os.path.join(pkgdir, 'pywin32_system32', 'pythoncom{0}{1}.dll') +import warnings +with warnings.catch_warnings(): + warnings.simplefilter("ignore") + import imp +imp.load_dynamic('pywintypes', pwt) +imp.load_dynamic('pythoncom', pcom) +'''.format(PYTHON_VERSION[0], PYTHON_VERSION[1])) + + installer_cfg_path = os.path.join(build_path, 'installer.cfg') + + certbot_pkg_path = os.path.join(repo_path, 'certbot') + certbot_version = subprocess.check_output([sys.executable, '-c', 'import certbot; print(certbot.__version__)'], + universal_newlines=True, cwd=certbot_pkg_path).strip() + + with open(installer_cfg_path, 'w') as file_h: + file_h.write('''\ +[Application] +name=Certbot +version={certbot_version} +icon=certbot.ico +publisher=Electronic Frontier Foundation +target=$INSTDIR\\run.bat + +[Build] +directory=nsis +nsi_template=template.nsi +installer_name=certbot-beta-installer-{installer_suffix}.exe + +[Python] +version={python_version} +bitness={python_bitness} + +[Include] +local_wheels=wheels\\*.whl +files=run.bat + renew-up.ps1 + renew-down.ps1 + +[Command certbot] +entry_point=certbot.main:main +extra_preamble=pywin32_paths.py +'''.format(certbot_version=certbot_version, + installer_suffix='win_amd64' if PYTHON_BITNESS == 64 else 'win32', + python_bitness=PYTHON_BITNESS, + python_version='.'.join([str(item) for item in PYTHON_VERSION]))) + + return installer_cfg_path + + +def _prepare_environment(): + print('Prepare environment') + try: + subprocess.check_output(['choco', '--version']) + except subprocess.CalledProcessError: + raise RuntimeError('Error: Chocolatey (https://chocolatey.org/) needs ' + 'to be installed to run this script.') + script_path = os.path.realpath(__file__) + repo_path = os.path.dirname(os.path.dirname(script_path)) + build_path = os.path.join(repo_path, 'windows-installer', 'build') + venv_path = os.path.join(build_path, 'venv-config') + venv_python = os.path.join(venv_path, 'Scripts', 'python.exe') + + return build_path, repo_path, venv_path, venv_python + + +if __name__ == '__main__': + if not os.name == 'nt': + raise RuntimeError('This script must be run under Windows.') + + if ctypes.windll.shell32.IsUserAnAdmin() == 0: + # Administrator privileges are required to properly install NSIS through Chocolatey + raise RuntimeError('This script must be run with administrator privileges.') + + if sys.version_info[:2] != PYTHON_VERSION[:2]: + raise RuntimeError('This script must be run with Python {0}' + .format('.'.join([str(item) for item in PYTHON_VERSION[0:2]]))) + + if struct.calcsize('P') * 8 != PYTHON_BITNESS: + raise RuntimeError('This script must be run with a {0} bit version of Python.' + .format(PYTHON_BITNESS)) + main() diff --git a/windows-installer/renew-down.ps1 b/windows-installer/renew-down.ps1 new file mode 100644 index 000000000..60dc4d9e6 --- /dev/null +++ b/windows-installer/renew-down.ps1 @@ -0,0 +1,6 @@ +$taskName = "Certbot Renew Task" + +$exists = Get-ScheduledTask | Where-Object {$_.TaskName -like $taskName} +if ($exists) { + Unregister-ScheduledTask -TaskName $taskName -Confirm:$false +} diff --git a/windows-installer/renew-up.ps1 b/windows-installer/renew-up.ps1 new file mode 100644 index 000000000..224458748 --- /dev/null +++ b/windows-installer/renew-up.ps1 @@ -0,0 +1,17 @@ +function Get-ScriptDirectory { Split-Path $MyInvocation.ScriptName } +$down = Join-Path (Get-ScriptDirectory) 'renew-down.ps1' +& $down + +$taskName = "Certbot Renew Task" + +$action = New-ScheduledTaskAction -Execute 'Powershell.exe' -Argument '-NoProfile -WindowStyle Hidden -Command "certbot renew"' +$delay = New-TimeSpan -Hours 12 +$triggerAM = New-ScheduledTaskTrigger -Daily -At 12am -RandomDelay $delay +$triggerPM = New-ScheduledTaskTrigger -Daily -At 12pm -RandomDelay $delay +# NB: For now scheduled task is set up under Administrators group account because Certbot Installer installs Certbot for all users. +# If in the future we allow the Installer to install Certbot for one specific user, the scheduled task will need to +# switch to this user, since Certbot will be available only for him. +$adminsSID = New-Object System.Security.Principal.SecurityIdentifier("S-1-5-32-544") +$adminsGroupID = $adminsSID.Translate([System.Security.Principal.NTAccount]).Value +$principal = New-ScheduledTaskPrincipal -GroupId $adminsGroupID -RunLevel Highest +Register-ScheduledTask -Action $action -Trigger $triggerAM,$triggerPM -TaskName $taskName -Description "Execute twice a day the 'certbot renew' command, to renew managed certificates if needed." -Principal $principal diff --git a/windows-installer/run.bat b/windows-installer/run.bat new file mode 100644 index 000000000..efba28800 --- /dev/null +++ b/windows-installer/run.bat @@ -0,0 +1,31 @@ +@echo off + +:: BatchGotAdmin +:------------------------------------- +REM --> Check for permissions + IF "%PROCESSOR_ARCHITECTURE%" EQU "amd64" ( +>nul 2>&1 "%SYSTEMROOT%\SysWOW64\cacls.exe" "%SYSTEMROOT%\SysWOW64\config\system" +) ELSE ( +>nul 2>&1 "%SYSTEMROOT%\system32\cacls.exe" "%SYSTEMROOT%\system32\config\system" +) + +REM --> If error flag set, we do not have admin. +if '%errorlevel%' NEQ '0' ( + echo Requesting administrative privileges... + goto UACPrompt +) else ( goto gotAdmin ) + +:UACPrompt + echo Set UAC = CreateObject^("Shell.Application"^) > "%temp%\getadmin.vbs" + set params= %* + echo UAC.ShellExecute "cmd.exe", "/c ""%~s0"" %params:"=""%", "", "runas", 1 >> "%temp%\getadmin.vbs" + + "%temp%\getadmin.vbs" + del "%temp%\getadmin.vbs" + exit /B + +:gotAdmin + pushd "%CD%" + CD /D "%~dp0" +:-------------------------------------- +cmd.exe /k echo You can run 'certbot' commands here. Type 'certbot --help' for more information. diff --git a/windows-installer/template.nsi b/windows-installer/template.nsi new file mode 100644 index 000000000..50a03865f --- /dev/null +++ b/windows-installer/template.nsi @@ -0,0 +1,260 @@ +; This NSIS template is based on the built-in one in pynsist 2.3. +; Added lines are enclosed within "CERTBOT CUSTOM BEGIN/END" comments. +; If pynsist is upgraded, this template must be updated if necessary using the new built-in one. +; Original file can be found here: https://github.com/takluyver/pynsist/blob/2.4/nsist/pyapp.nsi + +!define PRODUCT_NAME "[[ib.appname]]" +!define PRODUCT_VERSION "[[ib.version]]" +!define PY_VERSION "[[ib.py_version]]" +!define PY_MAJOR_VERSION "[[ib.py_major_version]]" +!define BITNESS "[[ib.py_bitness]]" +!define ARCH_TAG "[[arch_tag]]" +!define INSTALLER_NAME "[[ib.installer_name]]" +!define PRODUCT_ICON "[[icon]]" + +; Marker file to tell the uninstaller that it's a user installation +!define USER_INSTALL_MARKER _user_install_marker + +SetCompressor lzma + +; CERTBOT CUSTOM BEGIN +; Administrator privileges are required to insert a new task in Windows Scheduler. +; Also comment out some options to disable ability to choose AllUsers/CurrentUser install mode. +; As a result, installer run always with admin privileges (because of MULTIUSER_EXECUTIONLEVEL), +; using the AllUsers installation mode by default (because of MULTIUSER_INSTALLMODE_DEFAULT_CURRENTUSER +; not set), and this default behavior cannot be overridden (because of MULTIUSER_MUI not set). +; See https://nsis.sourceforge.io/Docs/MultiUser/Readme.html +!define MULTIUSER_EXECUTIONLEVEL Admin +;!define MULTIUSER_EXECUTIONLEVEL Highest +;!define MULTIUSER_INSTALLMODE_DEFAULT_CURRENTUSER +;!define MULTIUSER_MUI +;!define MULTIUSER_INSTALLMODE_COMMANDLINE +; CERTBOT CUSTOM END +!define MULTIUSER_INSTALLMODE_INSTDIR "[[ib.appname]]" +[% if ib.py_bitness == 64 %] +!define MULTIUSER_INSTALLMODE_FUNCTION correct_prog_files +[% endif %] +!include MultiUser.nsh + +[% block modernui %] +; Modern UI installer stuff +!include "MUI2.nsh" +!define MUI_ABORTWARNING +!define MUI_ICON "[[icon]]" +!define MUI_UNICON "[[icon]]" + +; UI pages +[% block ui_pages %] +!insertmacro MUI_PAGE_WELCOME +[% if license_file %] +!insertmacro MUI_PAGE_LICENSE [[license_file]] +[% endif %] +; CERTBOT CUSTOM BEGIN +; Disable the installation mode page (AllUsers/CurrentUser) +;!insertmacro MULTIUSER_PAGE_INSTALLMODE +; CERTBOT CUSTOM END +!insertmacro MUI_PAGE_DIRECTORY +!insertmacro MUI_PAGE_INSTFILES +!insertmacro MUI_PAGE_FINISH +[% endblock ui_pages %] +!insertmacro MUI_LANGUAGE "English" +[% endblock modernui %] + +; CERTBOT CUSTOM BEGIN +Name "${PRODUCT_NAME} (beta) ${PRODUCT_VERSION}" +;Name "${PRODUCT_NAME} ${PRODUCT_VERSION}" +; CERTBOT CUSTOM END +OutFile "${INSTALLER_NAME}" +ShowInstDetails show + +Section -SETTINGS + SetOutPath "$INSTDIR" + SetOverwrite ifnewer +SectionEnd + +[% block sections %] + +Section "!${PRODUCT_NAME}" sec_app + SetRegView [[ib.py_bitness]] + SectionIn RO + File ${PRODUCT_ICON} + SetOutPath "$INSTDIR\pkgs" + File /r "pkgs\*.*" + SetOutPath "$INSTDIR" + + ; Marker file for per-user install + StrCmp $MultiUser.InstallMode CurrentUser 0 +3 + FileOpen $0 "$INSTDIR\${USER_INSTALL_MARKER}" w + FileClose $0 + SetFileAttributes "$INSTDIR\${USER_INSTALL_MARKER}" HIDDEN + + [% block install_files %] + ; Install files + [% for destination, group in grouped_files %] + SetOutPath "[[destination]]" + [% for file in group %] + File "[[ file ]]" + [% endfor %] + [% endfor %] + + ; Install directories + [% for dir, destination in ib.install_dirs %] + SetOutPath "[[ pjoin(destination, dir) ]]" + File /r "[[dir]]\*.*" + [% endfor %] + [% endblock install_files %] + + [% block install_shortcuts %] + ; Install shortcuts + ; The output path becomes the working directory for shortcuts + SetOutPath "%HOMEDRIVE%\%HOMEPATH%" + [% if single_shortcut %] + [% for scname, sc in ib.shortcuts.items() %] + CreateShortCut "$SMPROGRAMS\[[scname]].lnk" "[[sc['target'] ]]" \ + '[[ sc['parameters'] ]]' "$INSTDIR\[[ sc['icon'] ]]" + [% endfor %] + [% else %] + [# Multiple shortcuts: create a directory for them #] + CreateDirectory "$SMPROGRAMS\${PRODUCT_NAME}" + [% for scname, sc in ib.shortcuts.items() %] + CreateShortCut "$SMPROGRAMS\${PRODUCT_NAME}\[[scname]].lnk" "[[sc['target'] ]]" \ + '[[ sc['parameters'] ]]' "$INSTDIR\[[ sc['icon'] ]]" + [% endfor %] + [% endif %] + SetOutPath "$INSTDIR" + [% endblock install_shortcuts %] + + [% block install_commands %] + [% if has_commands %] + DetailPrint "Setting up command-line launchers..." + nsExec::ExecToLog '[[ python ]] -Es "$INSTDIR\_assemble_launchers.py" [[ python ]] "$INSTDIR\bin"' + + StrCmp $MultiUser.InstallMode CurrentUser 0 AddSysPathSystem + ; Add to PATH for current user + nsExec::ExecToLog '[[ python ]] -Es "$INSTDIR\_system_path.py" add_user "$INSTDIR\bin"' + GoTo AddedSysPath + AddSysPathSystem: + ; Add to PATH for all users + nsExec::ExecToLog '[[ python ]] -Es "$INSTDIR\_system_path.py" add "$INSTDIR\bin"' + AddedSysPath: + [% endif %] + [% endblock install_commands %] + + ; Byte-compile Python files. + DetailPrint "Byte-compiling Python modules..." + nsExec::ExecToLog '[[ python ]] -m compileall -q "$INSTDIR\pkgs"' + WriteUninstaller $INSTDIR\uninstall.exe + ; Add ourselves to Add/remove programs + WriteRegStr SHCTX "Software\Microsoft\Windows\CurrentVersion\Uninstall\${PRODUCT_NAME}" \ + "DisplayName" "${PRODUCT_NAME}" + WriteRegStr SHCTX "Software\Microsoft\Windows\CurrentVersion\Uninstall\${PRODUCT_NAME}" \ + "UninstallString" '"$INSTDIR\uninstall.exe"' + WriteRegStr SHCTX "Software\Microsoft\Windows\CurrentVersion\Uninstall\${PRODUCT_NAME}" \ + "InstallLocation" "$INSTDIR" + WriteRegStr SHCTX "Software\Microsoft\Windows\CurrentVersion\Uninstall\${PRODUCT_NAME}" \ + "DisplayIcon" "$INSTDIR\${PRODUCT_ICON}" + [% if ib.publisher is not none %] + WriteRegStr SHCTX "Software\Microsoft\Windows\CurrentVersion\Uninstall\${PRODUCT_NAME}" \ + "Publisher" "[[ib.publisher]]" + [% endif %] + WriteRegStr SHCTX "Software\Microsoft\Windows\CurrentVersion\Uninstall\${PRODUCT_NAME}" \ + "DisplayVersion" "${PRODUCT_VERSION}" + WriteRegDWORD SHCTX "Software\Microsoft\Windows\CurrentVersion\Uninstall\${PRODUCT_NAME}" \ + "NoModify" 1 + WriteRegDWORD SHCTX "Software\Microsoft\Windows\CurrentVersion\Uninstall\${PRODUCT_NAME}" \ + "NoRepair" 1 + + ; CERTBOT CUSTOM BEGIN + ; Execute ps script to create the certbot renew task + DetailPrint "Setting up certbot renew scheduled task" + nsExec::ExecToStack 'powershell -inputformat none -ExecutionPolicy RemoteSigned -File "$INSTDIR\renew-up.ps1"' + ; CERTBOT CUSTOM END + + ; Check if we need to reboot + IfRebootFlag 0 noreboot + MessageBox MB_YESNO "A reboot is required to finish the installation. Do you wish to reboot now?" \ + /SD IDNO IDNO noreboot + Reboot + noreboot: +SectionEnd + +Section "Uninstall" + ; CERTBOT CUSTOM BEGIN + ; Execute ps script to remove the certbot renew task + nsExec::ExecToStack 'powershell -inputformat none -ExecutionPolicy RemoteSigned -File "$INSTDIR\renew-down.ps1"' + ; CERTBOT CUSTOM END + + SetRegView [[ib.py_bitness]] + SetShellVarContext all + IfFileExists "$INSTDIR\${USER_INSTALL_MARKER}" 0 +3 + SetShellVarContext current + Delete "$INSTDIR\${USER_INSTALL_MARKER}" + + Delete $INSTDIR\uninstall.exe + Delete "$INSTDIR\${PRODUCT_ICON}" + RMDir /r "$INSTDIR\pkgs" + + ; Remove ourselves from %PATH% + [% block uninstall_commands %] + [% if has_commands %] + nsExec::ExecToLog '[[ python ]] -Es "$INSTDIR\_system_path.py" remove "$INSTDIR\bin"' + [% endif %] + [% endblock uninstall_commands %] + + [% block uninstall_files %] + ; Uninstall files + [% for file, destination in ib.install_files %] + Delete "[[pjoin(destination, file)]]" + [% endfor %] + ; Uninstall directories + [% for dir, destination in ib.install_dirs %] + RMDir /r "[[pjoin(destination, dir)]]" + [% endfor %] + [% endblock uninstall_files %] + + [% block uninstall_shortcuts %] + ; Uninstall shortcuts + [% if single_shortcut %] + [% for scname in ib.shortcuts %] + Delete "$SMPROGRAMS\[[scname]].lnk" + [% endfor %] + [% else %] + RMDir /r "$SMPROGRAMS\${PRODUCT_NAME}" + [% endif %] + [% endblock uninstall_shortcuts %] + RMDir $INSTDIR + DeleteRegKey SHCTX "Software\Microsoft\Windows\CurrentVersion\Uninstall\${PRODUCT_NAME}" +SectionEnd + +[% endblock sections %] + +; Functions + +Function .onMouseOverSection + ; Find which section the mouse is over, and set the corresponding description. + FindWindow $R0 "#32770" "" $HWNDPARENT + GetDlgItem $R0 $R0 1043 ; description item (must be added to the UI) + + [% block mouseover_messages %] + StrCmp $0 ${sec_app} "" +2 + SendMessage $R0 ${WM_SETTEXT} 0 "STR:${PRODUCT_NAME}" + + [% endblock mouseover_messages %] +FunctionEnd + +Function .onInit + !insertmacro MULTIUSER_INIT +FunctionEnd + +Function un.onInit + !insertmacro MULTIUSER_UNINIT +FunctionEnd + +[% if ib.py_bitness == 64 %] +Function correct_prog_files + ; The multiuser machinery doesn't know about the different Program files + ; folder for 64-bit applications. Override the install dir it set. + StrCmp $MultiUser.InstallMode AllUsers 0 +2 + StrCpy $INSTDIR "$PROGRAMFILES64\${MULTIUSER_INSTALLMODE_INSTDIR}" +FunctionEnd +[% endif %] \ No newline at end of file