Merge branch 'master' into centos6-migrate-python36

.azure-pipelines/INSTALL.md

@@ -0,0 +1,119 @@
+# Configuring Azure Pipelines with Certbot
+
+Let's begin. All pipelines are defined in `.azure-pipelines`. Currently there are two:
+* `.azure-pipelines/main.yml` is the main one, executed on PRs for master, and pushes to master,
+* `.azure-pipelines/advanced.yml` add installer testing on top of the main pipeline, and is executed for `test-*` branches, release branches, and nightly run for master.
+
+Several templates are defined in `.azure-pipelines/templates`. These YAML files aggregate common jobs configuration that can be reused in several pipelines.
+
+Unlike Travis, where CodeCov is working without any action required, CodeCov supports Azure Pipelines
+using the coverage-bash utility (not python-coverage for now) only if you provide the Codecov repo token
+using the `CODECOV_TOKEN` environment variable. So `CODECOV_TOKEN` needs to be set as a secured
+environment variable to allow the main pipeline to publish coverage reports to CodeCov.
+
+This INSTALL.md file explains how to configure Azure Pipelines with Certbot in order to execute the CI/CD logic defined in `.azure-pipelines` folder with it.
+During this installation step, warnings describing user access and legal comitments will be displayed like this:
+```
+!!! ACCESS REQUIRED !!!
+```
+
+This document suppose that the Azure DevOps organization is named _certbot_, and the Azure DevOps project is also _certbot_.
+
+## Useful links
+
+* https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema
+* https://www.azuredevopslabs.com/labs/azuredevops/github-integration/
+* https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/python?view=azure-devops
+
+## Prerequisites
+
+### Having a GitHub account
+
+Use your GitHub user for a normal GitHub account, or a user that has administrative rights to the GitHub organization if relevant.
+
+### Having an Azure DevOps account
+- Go to https://dev.azure.com/, click "Start free with GitHub"
+- Login to GitHub
+
+```
+!!! ACCESS REQUIRED !!!
+Personal user data (email + profile info, in read-only)
+```
+
+- Microsoft will create a Live account using the email referenced for the GitHub account. This account is also linked to GitHub account (meaning you can log it using GitHub authentication)
+- Proceed with account registration (birth date, country), add details about name and email contact
+
+```
+!!! ACCESS REQUIRED !!!
+Microsoft proposes to send commercial links to this mail
+Azure DevOps terms of service need to be accepted
+```
+
+_Logged to Azure DevOps, account is ready._
+
+### Installing Azure Pipelines to GitHub
+
+- On GitHub, go to Marketplace
+- Select Azure Pipeline, and "Set up a plan"
+- Select Free, then "Install it for free"
+- Click "Complete order and begin installation"
+
+```
+!!! ACCESS !!!
+Azure Pipeline needs RW on code, RO on metadata, RW on checks, commit statuses, deployments, issues, pull requests.
+RW access here is required to allow update of the pipelines YAML files from Azure DevOps interface, and to
+update the status of builds and PRs on GitHub side when Azure Pipelines are triggered.
+Note however that no admin access is defined here: this means that Azure Pipelines cannot do anything with
+protected branches, like master, and cannot modify the security context around this on GitHub.
+Access can be defined for all or only selected repositories, which is nice.
+```
+
+- Redirected to Azure DevOps, select the account created in _Having an Azure DevOps account_ section.
+- Select the organization, and click "Create a new project" (let's name it the same than the targetted github repo)
+- The Visibility is public, to profit from 10 parallel jobs
+
+```
+!!! ACCESS !!!
+Azure Pipelines needs access to the GitHub account (in term of beeing able to check it is valid), and the Resources shared between the GitHub account and Azure Pipelines.
+```
+
+_Done. We can move to pipelines configuration._
+
+## Import an existing pipelines from `.azure-pipelines` folder
+
+- On Azure DevOps, go to your organization (eg. _certbot_) then your project (eg. _certbot_)
+- Click "Pipelines" tab
+- Click "New pipeline"
+- Where is your code?: select "__Use the classic editor__"
+
+__Warning: Do not choose the GitHub option in Where is your code? section. Indeed, this option will trigger an OAuth
+grant permissions from Azure Pipelines to GitHub in order to setup a GitHub OAuth Application. The permissions asked
+then are way too large (admin level on almost everything), while the classic approach does not add any more
+permissions, and works perfectly well.__
+
+- Select GitHub in "Select your repository section", choose certbot/certbot in Repository, master in default branch.
+- Click on YAML option for "Select a template"
+- Choose a name for the pipeline (eg. test-pipeline), and browse to the actual pipeline YAML definition in the
+  "YAML file path" input (eg. `.azure-pipelines/test-pipeline.yml`)
+- Click "Save & queue", choose the master branch to build the first pipeline, and click "Save and run" button.
+
+_Done. Pipeline is operational. Repeat to add more pipelines from existing YAML files in `.azure-pipelines`._
+
+## Add a secret variable to a pipeline (like `CODECOV_TOKEN`)
+
+__NB: Following steps suppose that you already setup the YAML pipeline file to
+consume the secret variable that these steps will create as an environment variable.
+For a variable named `CODECOV_TOKEN` consuming the variable `codecov_token`,
+in the YAML file this setup would take the form of the following:
+```
+steps:
+    - script: ./do_something_that_consumes_CODECOV_TOKEN  # Eg. `codecov -F windows`
+      env:
+        CODECOV_TOKEN: $(codecov_token)
+```
+
+To set up a variable that is shared between pipelines, follow the instructions
+at
+https://docs.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups.
+When adding variables to a group, don't forget to tick "Keep this value secret"
+if it shouldn't be shared publcily.

.azure-pipelines/advanced.yml

@@ -0,0 +1,18 @@
+# Advanced pipeline for isolated checks and release purpose
+trigger:
+  - test-*
+  - '*.x'
+pr:
+  - test-*
+  - '*.x'
+# This pipeline is also nightly run on master
+schedules:
+  - cron: "4 0 * * *"
+    displayName: Nightly build
+    branches:
+      include:
+      - master
+
+jobs:
+  - template: templates/tests-suite.yml
+  - template: templates/installer-tests.yml

.azure-pipelines/main.yml

@@ -0,0 +1,11 @@
+trigger:
+  # apache-parser-v2 is a temporary branch for doing work related to
+  # rewriting the parser in the Apache plugin.
+  - apache-parser-v2
+  - master
+pr:
+  - apache-parser-v2
+  - master
+
+jobs:
+  - template: templates/tests-suite.yml

.azure-pipelines/templates/installer-tests.yml

@@ -0,0 +1,31 @@
+jobs:
+  - job: installer
+    pool:
+      vmImage: vs2017-win2016
+    steps:
+      - task: UsePythonVersion@0
+        inputs:
+          versionSpec: 3.7
+          architecture: x86
+          addToPath: true
+      - script: python windows-installer/construct.py
+        displayName: Build Certbot installer
+      - task: CopyFiles@2
+        inputs:
+          sourceFolder: $(System.DefaultWorkingDirectory)/windows-installer/build/nsis
+          contents: '*.exe'
+          targetFolder: $(Build.ArtifactStagingDirectory)
+      - task: PublishPipelineArtifact@1
+        inputs:
+          path: $(Build.ArtifactStagingDirectory)
+          artifact: WindowsInstaller
+      - script: $(Build.ArtifactStagingDirectory)\certbot-installer-win32.exe /S
+        displayName: Install Certbot
+      - script: |
+          python -m venv venv
+          venv\Scripts\python tools\pip_install.py -e certbot-ci
+        displayName: Prepare Certbot-CI
+      - script: |
+          set PATH=%ProgramFiles(x86)%\Certbot\bin;%PATH%
+          venv\Scripts\python -m pytest certbot-ci\certbot_integration_tests\certbot_tests -n 4
+        displayName: Run integration tests

.azure-pipelines/templates/tests-suite.yml

@@ -0,0 +1,38 @@
+jobs:
+  - job: test
+    pool:
+      vmImage: vs2017-win2016
+    strategy:
+      matrix:
+        py35:
+          PYTHON_VERSION: 3.5
+          TOXENV: py35
+        py37-cover:
+          PYTHON_VERSION: 3.7
+          TOXENV: py37-cover
+        integration-certbot:
+          PYTHON_VERSION: 3.7
+          TOXENV: integration-certbot
+          PYTEST_ADDOPTS: --numprocesses 4
+    variables:
+    - group: certbot-common
+    steps:
+    - task: UsePythonVersion@0
+      inputs:
+        versionSpec: $(PYTHON_VERSION)
+        addToPath: true
+    - script: python tools/pip_install.py -U tox coverage
+      displayName: Install dependencies
+    - script: python -m tox
+      displayName: Run tox
+    # We do not require codecov report upload to succeed. So to avoid to break the pipeline if
+    # something goes wrong, each command is suffixed with a command that hides any non zero exit
+    # codes and echoes an informative message instead.
+    - bash: |
+        curl -s https://codecov.io/bash -o codecov-bash || echo "Failed to download codecov-bash"
+        chmod +x codecov-bash || echo "Failed to apply execute permissions on codecov-bash"
+        ./codecov-bash -F windows  || echo "Codecov did not collect coverage reports"
+      condition: in(variables['TOXENV'], 'py37-cover', 'integration-certbot')
+      env:
+        CODECOV_TOKEN: $(codecov_token)
+      displayName: Publish coverage

AUTHORS.md

@@ -18,6 +18,7 @@ Authors
 * [Alex Zorin](https://github.com/alexzorin)
 * [Amjad Mashaal](https://github.com/TheNavigat)
 * [Andrew Murray](https://github.com/radarhere)
+* [Andrzej Górski](https://github.com/andrzej3393)
 * [Anselm Levskaya](https://github.com/levskaya)
 * [Antoine Jacoutot](https://github.com/ajacoutot)
 * [asaph](https://github.com/asaph)

CHANGELOG.md

@@ -10,7 +10,9 @@ Certbot adheres to [Semantic Versioning](https://semver.org/).
 
 ### Changed
 
-*
+* Removed `--fast` flag from the test farm tests
+* `--server` may now be combined with `--dry-run`. Certbot will, as before, use the
+  staging server instead of the live server when `--dry-run` is used.
 
 ### Fixed
 

certbot/cli.py

@@ -649,13 +649,20 @@ class HelpfulArgumentParser(object):
     def set_test_server(self, parsed_args):
         """We have --staging/--dry-run; perform sanity check and set config.server"""
 
-        if parsed_args.server not in (flag_default("server"), constants.STAGING_URI):
-            conflicts = ["--staging"] if parsed_args.staging else []
-            conflicts += ["--dry-run"] if parsed_args.dry_run else []
-            raise errors.Error("--server value conflicts with {0}".format(
-                " and ".join(conflicts)))
+        # Flag combinations should produce these results:
+        #                             | --staging      | --dry-run   |
+        # ------------------------------------------------------------
+        # | --server acme-v02         | Use staging    | Use staging |
+        # | --server acme-staging-v02 | Use staging    | Use staging |
+        # | --server <other>          | Conflict error | Use <other> |
 
-        parsed_args.server = constants.STAGING_URI
+        default_servers = (flag_default("server"), constants.STAGING_URI)
+
+        if parsed_args.staging and parsed_args.server not in default_servers:
+            raise errors.Error("--server value conflicts with --staging")
+
+        if parsed_args.server in default_servers:
+            parsed_args.server = constants.STAGING_URI
 
         if parsed_args.dry_run:
             if self.verb not in ["certonly", "renew"]:

certbot/compat/filesystem.py

@@ -546,13 +546,7 @@ def _generate_windows_flags(rights_desc):
     if rights_desc['write']:
         flag = flag | (ntsecuritycon.FILE_ALL_ACCESS
                        ^ ntsecuritycon.FILE_GENERIC_READ
-                       ^ ntsecuritycon.FILE_GENERIC_EXECUTE
-                       # Despite bit `512` being present in ntsecuritycon.FILE_ALL_ACCESS, it is
-                       # not effectively applied to the file or the directory.
-                       # As _generate_windows_flags is also used to compare two dacls, we remove
-                       # it right now to have flags that contain only the bits effectively applied
-                       # by Windows.
-                       ^ 512)
+                       ^ ntsecuritycon.FILE_GENERIC_EXECUTE)
     if rights_desc['execute']:
         flag = flag | ntsecuritycon.FILE_GENERIC_EXECUTE
 

certbot/tests/cli_test.py

@@ -333,16 +333,26 @@ class ParseTest(unittest.TestCase):  # pylint: disable=too-many-public-methods
 
         self._assert_dry_run_flag_worked(self.parse(short_args + ['auth']), True)
         self._assert_dry_run_flag_worked(self.parse(short_args + ['renew']), True)
+        self._assert_dry_run_flag_worked(self.parse(short_args + ['certonly']), True)
+
         short_args += ['certonly']
-        self._assert_dry_run_flag_worked(self.parse(short_args), True)
 
-        short_args += '--server example.com'.split()
-        conflicts = ['--dry-run']
-        self._check_server_conflict_message(short_args, '--dry-run')
+        # `--dry-run --server example.com` should emit example.com
+        self.assertEqual(self.parse(short_args + ['--server', 'example.com']).server,
+                         'example.com')
 
-        short_args += ['--staging']
-        conflicts += ['--staging']
-        self._check_server_conflict_message(short_args, conflicts)
+        # `--dry-run --server STAGING_URI` should emit STAGING_URI
+        self.assertEqual(self.parse(short_args + ['--server', constants.STAGING_URI]).server,
+                         constants.STAGING_URI)
+
+        # `--dry-run --server LIVE` should emit STAGING_URI
+        self.assertEqual(self.parse(short_args + ['--server', cli.flag_default("server")]).server,
+                         constants.STAGING_URI)
+
+        # `--dry-run --server example.com --staging` should emit an error
+        conflicts = ['--staging']
+        self._check_server_conflict_message(short_args + ['--server', 'example.com', '--staging'],
+                                            conflicts)
 
     def test_option_was_set(self):
         key_size_option = 'rsa_key_size'

certbot/tests/compat/filesystem_test.py

@@ -89,8 +89,8 @@ class WindowsChmodTests(TempDirTestCase):
         self.assertEqual(len(system_aces), 1)
         self.assertEqual(len(admin_aces), 1)
 
-        self.assertEqual(system_aces[0][1], ntsecuritycon.FILE_ALL_ACCESS ^ 512)
-        self.assertEqual(admin_aces[0][1], ntsecuritycon.FILE_ALL_ACCESS ^ 512)
+        self.assertEqual(system_aces[0][1], ntsecuritycon.FILE_ALL_ACCESS)
+        self.assertEqual(admin_aces[0][1], ntsecuritycon.FILE_ALL_ACCESS)
 
     def test_read_flag(self):
         self._test_flag(4, ntsecuritycon.FILE_GENERIC_READ)
@@ -101,12 +101,10 @@ class WindowsChmodTests(TempDirTestCase):
     def test_write_flag(self):
         self._test_flag(2, (ntsecuritycon.FILE_ALL_ACCESS
                             ^ ntsecuritycon.FILE_GENERIC_READ
-                            ^ ntsecuritycon.FILE_GENERIC_EXECUTE
-                            ^ 512))
+                            ^ ntsecuritycon.FILE_GENERIC_EXECUTE))
 
     def test_full_flag(self):
-        self._test_flag(7, (ntsecuritycon.FILE_ALL_ACCESS
-                            ^ 512))
+        self._test_flag(7, ntsecuritycon.FILE_ALL_ACCESS)
 
     def _test_flag(self, everyone_mode, windows_flag):
         # Note that flag is tested against `everyone`, not `user`, because practically these unit

setup.py

@@ -59,7 +59,7 @@ install_requires = [
 # However environment markers are supported only with setuptools >= 36.2.
 # So this dependency is not added for old Linux distributions with old setuptools,
 # in order to allow these systems to build certbot from sources.
-pywin32_req = 'pywin32>=224'
+pywin32_req = 'pywin32>=225'
 if StrictVersion(setuptools_version) >= StrictVersion('36.2'):
     install_requires.append(pywin32_req + " ; sys_platform == 'win32'")
 elif 'bdist_wheel' in sys.argv[1:]:

tests/letstest/multitester.py

@@ -84,9 +84,6 @@ parser.add_argument('--killboulder',
 parser.add_argument('--boulderonly',
                     action='store_true',
                     help="only make a boulder server")
-parser.add_argument('--fast',
-                    action='store_true',
-                    help="use larger instance types to run faster (saves about a minute, probably not worth it)")
 cl_args = parser.parse_args()
 
 # Credential Variables
@@ -310,10 +307,10 @@ def create_client_instance(ec2_client, target, security_group_id, subnet_id):
     if 'machine_type' in target:
         machine_type = target['machine_type']
     elif target['virt'] == 'hvm':
-        machine_type = 't2.medium' if cl_args.fast else 't2.micro'
+        machine_type = 't2.medium'
     else:
         # 32 bit systems
-        machine_type = 'c1.medium' if cl_args.fast else 't1.micro'
+        machine_type = 'c1.medium'
     if 'userdata' in target.keys():
         userdata = target['userdata']
     else:

tools/dev_constraints.txt

@@ -64,7 +64,7 @@ pytest-sugar==0.9.2
 pytest-rerunfailures==4.2
 python-dateutil==2.6.1
 python-digitalocean==1.11
-pywin32==224
+pywin32==225
 PyYAML==3.13
 repoze.sphinx.autointerface==0.8
 requests-file==1.4.2

tox.ini

@@ -284,7 +284,7 @@ setenv = AWS_DEFAULT_REGION=us-east-1
 changedir = {[testenv:travis-test-farm-tests-base]changedir}
 commands =
     {[testenv:travis-test-farm-tests-base]commands}
-    python multitester.py apache2_targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_apache2.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH} --fast
+    python multitester.py apache2_targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_apache2.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH}
 deps = {[testenv:travis-test-farm-tests-base]deps}
 passenv = {[testenv:travis-test-farm-tests-base]passenv}
 setenv = {[testenv:travis-test-farm-tests-base]setenv}
@@ -293,7 +293,7 @@ setenv = {[testenv:travis-test-farm-tests-base]setenv}
 changedir = {[testenv:travis-test-farm-tests-base]changedir}
 commands =
     {[testenv:travis-test-farm-tests-base]commands}
-    python multitester.py targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_leauto_upgrades.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH} --fast
+    python multitester.py targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_leauto_upgrades.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH}
 deps = {[testenv:travis-test-farm-tests-base]deps}
 passenv = {[testenv:travis-test-farm-tests-base]passenv}
 setenv = {[testenv:travis-test-farm-tests-base]setenv}
@@ -302,7 +302,7 @@ setenv = {[testenv:travis-test-farm-tests-base]setenv}
 changedir = {[testenv:travis-test-farm-tests-base]changedir}
 commands =
     {[testenv:travis-test-farm-tests-base]commands}
-    python multitester.py targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_letsencrypt_auto_certonly_standalone.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH} --fast
+    python multitester.py targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_letsencrypt_auto_certonly_standalone.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH}
 deps = {[testenv:travis-test-farm-tests-base]deps}
 passenv = {[testenv:travis-test-farm-tests-base]passenv}
 setenv = {[testenv:travis-test-farm-tests-base]setenv}
@@ -311,7 +311,7 @@ setenv = {[testenv:travis-test-farm-tests-base]setenv}
 changedir = {[testenv:travis-test-farm-tests-base]changedir}
 commands =
     {[testenv:travis-test-farm-tests-base]commands}
-    python multitester.py targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_sdists.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH} --fast
+    python multitester.py targets.yaml travis-test-farm.pem SET_BY_ENV scripts/test_sdists.sh --repo {env:TRAVIS_BUILD_DIR} --branch {env:TRAVIS_BRANCH}
 deps = {[testenv:travis-test-farm-tests-base]deps}
 passenv = {[testenv:travis-test-farm-tests-base]passenv}
 setenv = {[testenv:travis-test-farm-tests-base]setenv}

windows-installer/construct.py

@@ -70,13 +70,39 @@ def _copy_assets(build_path, repo_path):
 def _generate_pynsist_config(repo_path, build_path):
     print('Generate pynsist configuration')
 
+    pywin32_paths_file = os.path.join(build_path, 'pywin32_paths.py')
+
+    # Pywin32 uses non-standard folders to hold its packages. We need to instruct pynsist bootstrap
+    # explicitly to add them into sys.path. This is done with a custom "pywin32_paths.py" that is
+    # referred in the pynsist configuration as an "extra_preamble".
+    # Reference example: https://github.com/takluyver/pynsist/tree/master/examples/pywebview
+    with open(pywin32_paths_file, 'w') as file_h:
+        file_h.write('''\
+pkgdir = os.path.join(os.path.dirname(installdir), 'pkgs')
+
+sys.path.extend([
+    os.path.join(pkgdir, 'win32'),
+    os.path.join(pkgdir, 'win32', 'lib'),
+])
+
+# Preload pywintypes and pythoncom
+pwt = os.path.join(pkgdir, 'pywin32_system32', 'pywintypes{0}{1}.dll')
+pcom = os.path.join(pkgdir, 'pywin32_system32', 'pythoncom{0}{1}.dll')
+import warnings
+with warnings.catch_warnings():
+    warnings.simplefilter("ignore")
+    import imp
+imp.load_dynamic('pywintypes', pwt)
+imp.load_dynamic('pythoncom', pcom)
+'''.format(PYTHON_VERSION[0], PYTHON_VERSION[1]))
+
     installer_cfg_path = os.path.join(build_path, 'installer.cfg')
 
     certbot_version = subprocess.check_output([sys.executable, '-c', 'import certbot; print(certbot.__version__)'],
                                               universal_newlines=True, cwd=repo_path).strip()
 
-    with open(os.path.join(installer_cfg_path), 'w') as file_h:
-        file_h.write("""\
+    with open(installer_cfg_path, 'w') as file_h:
+        file_h.write('''\
 [Application]
 name=Certbot
 version={certbot_version}
@@ -87,7 +113,7 @@ target=$INSTDIR\\run.bat
 [Build]
 directory=nsis
 nsi_template=template.nsi
-installer_name=certbot-{certbot_version}-installer-{installer_suffix}.exe
+installer_name=certbot-installer-{installer_suffix}.exe
 
 [Python]
 version={python_version}
@@ -101,7 +127,8 @@ files=run.bat
 
 [Command certbot]
 entry_point=certbot.main:main
-""".format(certbot_version=certbot_version,
+extra_preamble=pywin32_paths.py
+'''.format(certbot_version=certbot_version,
            installer_suffix='win_amd64' if PYTHON_BITNESS == 64 else 'win32',
            python_bitness=PYTHON_BITNESS,
            python_version='.'.join([str(item) for item in PYTHON_VERSION])))