Merge branch 'master' into ecdsa

.travis.yml

@@ -78,13 +78,33 @@ matrix:
       env: TOXENV=apacheconftest
       sudo: required
     - python: "3.3"
-      env: TOXENV=py33
+      env: TOXENV=py33 BOULDER_INTEGRATION=1
+      sudo: required
+      after_failure:
+        - sudo cat /var/log/mysql/error.log
+        - ps aux | grep mysql
+      services: docker
     - python: "3.4"
-      env: TOXENV=py34
+      env: TOXENV=py34 BOULDER_INTEGRATION=1
+      sudo: required
+      after_failure:
+        - sudo cat /var/log/mysql/error.log
+        - ps aux | grep mysql
+      services: docker
     - python: "3.5"
-      env: TOXENV=py35
+      env: TOXENV=py35 BOULDER_INTEGRATION=1
+      sudo: required
+      after_failure:
+        - sudo cat /var/log/mysql/error.log
+        - ps aux | grep mysql
+      services: docker
     - python: "3.6"
-      env: TOXENV=py36
+      env: TOXENV=py36 BOULDER_INTEGRATION=1
+      sudo: required
+      after_failure:
+        - sudo cat /var/log/mysql/error.log
+        - ps aux | grep mysql
+      services: docker
     - python: "2.7"
       env: TOXENV=nginxroundtrip
 

CHANGELOG.md

@@ -0,0 +1,427 @@
+# Certbot change log
+
+Certbot adheres to [Semantic Versioning](http://semver.org/).
+
+## 0.13.0 - 2017-04-06
+
+### Added
+
+* `--debug-challenges` now pauses Certbot after setting up challenges for debugging.
+* The Nginx parser can now handle all valid directives in configuration files.
+* Nginx ciphersuites have changed to Mozilla Intermediate.
+* `certbot-auto --no-bootstrap` provides the option to not install OS dependencies.
+
+### Fixed
+
+* `--register-unsafely-without-email` now respects `--quiet`.
+* Hyphenated renewal parameters are now saved in renewal config files.
+* `--dry-run` no longer persists keys and csrs.
+* Certbot no longer hangs when trying to start Nginx in Arch Linux.
+* Apache rewrite rules no longer double-encode characters.
+
+A full list of changes is available on GitHub:
+https://github.com/certbot/certbot/issues?q=is%3Aissue%20milestone%3A0.13.0%20is%3Aclosed%20
+
+## 0.12.0 - 2017-03-02
+
+### Added
+
+* Certbot now allows non-camelcase Apache VirtualHost names.
+* Certbot now allows more log messages to be silenced.
+
+### Fixed
+
+* Fixed a regression around using `--cert-name` when getting new certificates
+
+More information about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/issues?q=is%3Aissue%20milestone%3A0.12.0
+
+## 0.11.1 - 2017-02-01
+
+### Fixed
+
+* Resolved a problem where Certbot would crash while parsing command line
+arguments in some cases.
+* Fixed a typo.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/pulls?q=is%3Apr%20milestone%3A0.11.1%20is%3Aclosed
+
+## 0.11.0 - 2017-02-01
+
+### Added
+
+* When using the standalone plugin while running Certbot interactively 
+and a required port is bound by another process, Certbot will give you
+the option to retry to grab the port rather than immediately exiting.
+* You are now able to deactivate your account with the Let's Encrypt
+server using the `unregister` subcommand.
+* When revoking a certificate using the `revoke` subcommand, you now
+have the option to provide the reason the certificate is being revoked
+to Let's Encrypt with `--reason`.
+
+### Changed
+
+* Providing `--quiet` to `certbot-auto` now silences package manager output.
+
+### Removed
+
+* Removed the optional `dnspython` dependency in our `acme` package.
+Now the library does not support client side verification of the DNS
+challenge.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/issues?q=is%3Aissue+milestone%3A0.11.0+is%3Aclosed
+
+## 0.10.2 - 2017-01-25
+
+### Added
+
+* If Certbot receives a request with a `badNonce` error, it now
+automatically retries the request. Since nonces from Let's Encrypt expire,
+this helps people performing the DNS challenge with the `manual` plugin
+who may have to wait an extended period of time for their DNS changes to
+propagate.
+
+### Fixed
+
+* Certbot now saves the `--preferred-challenges` values for renewal. Previously
+these values were discarded causing a different challenge type to be used when
+renewing certs in some cases.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/issues?q=is%3Aissue+milestone%3A0.10.2+is%3Aclosed
+
+## 0.10.1 - 2017-01-13
+
+### Fixed
+
+* Resolve problems where when asking Certbot to update a certificate at
+an existing path to include different domain names, the old names would
+continue to be used.
+* Fix issues successfully running our unit test suite on some systems.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/issues?q=is%3Aissue+milestone%3A0.10.1+is%3Aclosed
+
+## 0.10.0 - 2017-01-11
+
+## Added
+
+* Added the ability to customize and automatically complete DNS and HTTP
+domain validation challenges with the manual plugin. The flags
+`--manual-auth-hook` and `--manual-cleanup-hook` can now be provided
+when using the manual plugin to execute commands provided by the user to
+perform and clean up challenges provided by the CA. This is best used in
+complicated setups where the DNS challenge must be used or Certbot's
+existing plugins cannot be used to perform HTTP challenges. For more
+information on how this works, see `certbot --help manual`.
+* Added a `--cert-name` flag for specifying the name to use for the
+certificate in Certbot's configuration directory. Using this flag in
+combination with `-d/--domains`, a user can easily request a new
+certificate with different domains and save it with the name provided by
+`--cert-name`. Additionally, `--cert-name` can be used to select a
+certificate with the `certonly` and `run` subcommands so a full list of
+domains in the certificate does not have to be provided.
+* Added subcommand `certificates` for listing the certificates managed by
+Certbot and their properties.
+* Added the `delete` subcommand for removing certificates managed by Certbot
+from the configuration directory.
+* Certbot now supports requesting internationalized domain names (IDNs).
+* Hooks provided to Certbot are now saved to be reused during renewal.
+If you run Certbot with `--pre-hook`, `--renew-hook`, or `--post-hook`
+flags when obtaining a certificate, the provided commands will
+automatically be saved and executed again when renewing the certificate.
+A pre-hook and/or post-hook can also be given to the `certbot renew`
+command either on the command line or in a [configuration
+file](https://certbot.eff.org/docs/using.html#configuration-file) to run
+an additional command before/after any certificate is renewed. Hooks
+will only be run if a certificate is renewed.
+* Support Busybox in certbot-auto.
+
+### Changed
+
+* Recategorized `-h/--help` output to improve documentation and
+discoverability.
+
+### Removed
+
+* Removed the ncurses interface. This change solves problems people
+were having on many systems, reduces the number of Certbot
+dependencies, and simplifies our code. Certbot's only interface now is
+the text interface which was available by providing `-t/--text` to
+earlier versions of Certbot.
+
+### Fixed
+
+* Many small bug fixes.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/issues?q=is%3Aissue+milestone%3A0.10.0is%3Aclosed
+
+## 0.9.3 - 2016-10-13
+
+### Added
+
+* The Apache plugin uses information about your OS to help determine the
+layout of your Apache configuration directory. We added a patch to
+ensure this code behaves the same way when testing on different systems
+as the tests were failing in some cases.
+
+### Changed
+
+* Certbot adopted more conservative behavior about reporting a needed port as
+unavailable when using the standalone plugin.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/27?closed=1
+
+## 0.9.2 - 2016-10-12
+
+### Added
+
+* Certbot stopped requiring that all possibly required ports are available when
+using the standalone plugin. It now only verifies that the ports are available
+when they are necessary.
+
+### Fixed
+
+* Certbot now verifies that our optional dependencies version matches what is
+required by Certbot.
+* Certnot now properly copies the `ssl on;` directives as necessary when
+performing domain validation in the Nginx plugin.
+* Fixed problem where symlinks were becoming files when they were
+packaged, causing errors during testing and OS packaging.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/26?closed=1
+
+## 0.9.1 - 2016-10-06
+
+### Fixed
+
+* Fixed a bug that was introduced in version 0.9.0 where the command
+line flag -q/--quiet wasn't respected in some cases.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/25?closed=1
+
+## 0.9.0 - 2016-10-05
+
+### Added
+
+* Added an alpha version of the Nginx plugin. This plugin fully automates the
+process of obtaining and installing certificates with Nginx.
+Additionally, it is able to automatically configure security
+enhancements such as an HTTP to HTTPS redirect and OCSP stapling. To use
+this plugin, you must have the `certbot-nginx` package installed (which
+is installed automatically when using `certbot-auto`) and provide
+`--nginx` on the command line. This plugin is still in its early stages
+so we recommend you use it with some caution and make sure you have a
+backup of your Nginx configuration.
+* Added support for the `DNS` challenge in the `acme` library and `DNS` in
+Certbot's `manual` plugin. This allows you to create DNS records to
+prove to Let's Encrypt you control the requested domain name. To use
+this feature, include `--manual --preferred-challenges dns` on the
+command line.
+* Certbot now helps with enabling Extra Packages for Enterprise Linux (EPEL) on
+CentOS 6 when using `certbot-auto`. To use `certbot-auto` on CentOS 6,
+the EPEL repository has to be enabled. `certbot-auto` will now prompt
+users asking them if they would like the script to enable this for them
+automatically. This is done without prompting users when using
+`letsencrypt-auto` or if `-n/--non-interactive/--noninteractive` is
+included on the command line.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/issues?q=is%3Aissue+milestone%3A0.9.0+is%3Aclosed
+
+## 0.8.1 - 2016-06-14
+
+### Added
+
+* Certbot now preserves a certificate's common name when using `renew`.
+* Certbot now saves webroot values for renewal when they are entered interactively.
+* Certbot now gracefully reports that the Apache plugin isn't usable when Augeas is not installed.
+* Added experimental support for Mageia has been added to `certbot-auto`.
+
+### Fixed
+
+* Fixed problems with an invalid user-agent string on OS X.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/issues?q=is%3Aissue+milestone%3A0.8.1+
+
+## 0.8.0 - 2016-06-02
+
+### Added
+
+* Added the `register` subcommand which can be used to register an account
+with the Let's Encrypt CA.
+* You can now run `certbot register --update-registration` to
+change the e-mail address associated with your registration.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/issues?q=is%3Aissue+milestone%3A0.8.0+
+
+## 0.7.0 - 2016-05-27
+
+### Added
+
+* Added `--must-staple` to request certificates from Let's Encrypt
+with the OCSP must staple extension.
+* Certbot now automatically configures OSCP stapling for Apache.
+* Certbot now allows requesting certificates for domains found in the common name
+of a custom CSR.
+
+### Fixed
+
+* Fixed a number of miscellaneous bugs
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/issues?q=milestone%3A0.7.0+is%3Aissue
+
+## 0.6.0 - 2016-05-12
+
+### Added
+
+* Versioned the datetime dependency in setup.py.
+
+### Changed
+
+* Renamed the client from `letsencrypt` to `certbot`.
+
+### Fixed
+
+* Fixed a small json deserialization error.
+* Certbot now preserves domain order in generated CSRs.
+* Fixed some minor bugs.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/issues?q=is%3Aissue%20milestone%3A0.6.0%20is%3Aclosed%20
+
+## 0.5.0 - 2016-04-05
+
+### Added
+
+* Added the ability to use the webroot plugin interactively.
+* Added the flags --pre-hook, --post-hook, and --renew-hook which can be used with
+the renew subcommand to register shell commands to run in response to
+renewal events. Pre-hook commands will be run before any certs are
+renewed, post-hook commands will be run after any certs are renewed,
+and renew-hook commands will be run after each cert is renewed. If no
+certs are due for renewal, no command is run.
+* Added a -q/--quiet flag which silences all output except errors.
+* Added an --allow-subset-of-domains flag which can be used with the renew
+command to prevent renewal failures for a subset of the requested
+domains from causing the client to exit.
+
+### Changed
+
+* Certbot now uses renewal configuration files. In /etc/letsencrypt/renewal
+by default, these files can be used to control what parameters are
+used when renewing a specific certificate.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/letsencrypt/letsencrypt/issues?q=milestone%3A0.5.0+is%3Aissue
+
+## 0.4.2 - 2016-03-03
+
+### Fixed
+
+* Resolved problems encountered when compiling letsencrypt
+against the new OpenSSL release.
+* Fixed problems encountered when using `letsencrypt renew` with configuration files
+from the private beta.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/letsencrypt/letsencrypt/issues?q=is%3Aissue+milestone%3A0.4.2
+
+## 0.4.1 - 2016-02-29
+
+### Fixed
+
+* Fixed Apache parsing errors encountered with some configurations.
+* Fixed Werkzeug dependency problems encountered on some Red Hat systems.
+* Fixed bootstrapping failures when using letsencrypt-auto with --no-self-upgrade.
+* Fixed problems with parsing renewal config files from private beta.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/letsencrypt/letsencrypt/issues?q=is:issue+milestone:0.4.1
+
+## 0.4.0 - 2016-02-10
+
+### Added
+
+* Added the verb/subcommand `renew` which can be used to renew your existing
+certificates as they approach expiration. Running `letsencrypt renew`
+will examine all existing certificate lineages and determine if any are
+less than 30 days from expiration. If so, the client will use the
+settings provided when you previously obtained the certificate to renew
+it. The subcommand finishes by printing a summary of which renewals were
+successful, failed, or not yet due.
+* Added a `--dry-run` flag to help with testing configuration
+without affecting production rate limits. Currently supported by the
+`renew` and `certonly` subcommands, providing `--dry-run` on the command
+line will obtain certificates from the staging server without saving the
+resulting certificates to disk.
+* Added major improvements to letsencrypt-auto. This script
+has been rewritten to include full support for Python 2.6, the ability
+for letsencrypt-auto to update itself, and improvements to the
+stability, security, and performance of the script.
+* Added support for Apache 2.2 to the Apache plugin.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/letsencrypt/letsencrypt/issues?q=is%3Aissue+milestone%3A0.4.0
+
+## 0.3.0 - 2016-01-27
+
+### Added
+
+* Added a non-interactive mode which can be enabled by including `-n` or
+`--non-interactive` on the command line. This can be used to guarantee
+the client will not prompt when run automatically using cron/systemd.
+* Added preparation for the new letsencrypt-auto script. Over the past
+couple months, we've been working on increasing the reliability and
+security of letsencrypt-auto. A number of changes landed in this
+release to prepare for the new version of this script.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/letsencrypt/letsencrypt/issues?q=is%3Aissue+milestone%3A0.3.0
+
+## 0.2.0 - 2016-01-14
+
+### Added
+
+* Added Apache plugin support for non-Debian based systems. Support has been
+added for modern Red Hat based systems such as Fedora 23, Red Hat 7,
+and CentOS 7 running Apache 2.4. In theory, this plugin should be
+able to be configured to run on any Unix-like OS running Apache 2.4.
+* Relaxed PyOpenSSL version requirements. This adds support for systems
+with PyOpenSSL versions 0.13 or 0.14.
+* Improved error messages from the client.
+
+### Fixed
+
+* Resolved issues with the Apache plugin enabling an HTTP to HTTPS
+redirect on some systems.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/letsencrypt/letsencrypt/issues?q=is%3Aissue+milestone%3A0.2.0
+
+## 0.1.1 - 2015-12-15
+
+### Added
+
+* Added a check that avoids attempting to issue for unqualified domain names like
+"localhost".
+
+### Fixed
+
+* Fixed a confusing UI path that caused some users to repeatedly renew
+their certs while experimenting with the client, in some cases hitting
+issuance rate limits.
+* Fixed numerous Apache configuration parser problems
+* Fixed --webroot permission handling for non-root users
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/letsencrypt/letsencrypt/issues?q=milestone%3A0.1.1

Dockerfile

@@ -1,70 +1,26 @@
-# https://github.com/letsencrypt/letsencrypt/pull/431#issuecomment-103659297
-# it is more likely developers will already have ubuntu:trusty rather
-# than e.g. debian:jessie and image size differences are negligible
-FROM ubuntu:trusty
-MAINTAINER Jakub Warmuz <jakub@warmuz.org>
-MAINTAINER William Budington <bill@eff.org>
-
-# Note: this only exposes the port to other docker containers. You
-# still have to bind to 443@host at runtime, as per the ACME spec.
-EXPOSE 443
-
-# TODO: make sure --config-dir and --work-dir cannot be changed
-# through the CLI (certbot-docker wrapper that uses standalone
-# authenticator and text mode only?)
-VOLUME /etc/letsencrypt /var/lib/letsencrypt
-
-WORKDIR /opt/certbot
-
-# no need to mkdir anything:
-# https://docs.docker.com/reference/builder/#copy
-# If <dest> doesn't exist, it is created along with all missing
-# directories in its path.
-
-ENV DEBIAN_FRONTEND=noninteractive
-
-COPY letsencrypt-auto-source/letsencrypt-auto /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto
-RUN /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto --os-packages-only && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists/* \
-           /tmp/* \
-           /var/tmp/*
-
-# the above is not likely to change, so by putting it further up the
-# Dockerfile we make sure we cache as much as possible
-
-
-COPY setup.py README.rst CHANGES.rst MANIFEST.in letsencrypt-auto-source/pieces/pipstrap.py /opt/certbot/src/
-
-# all above files are necessary for setup.py and venv setup, however,
-# package source code directory has to be copied separately to a
-# subdirectory...
-# https://docs.docker.com/reference/builder/#copy: "If <src> is a
-# directory, the entire contents of the directory are copied,
-# including filesystem metadata. Note: The directory itself is not
-# copied, just its contents." Order again matters, three files are far
-# more likely to be cached than the whole project directory
-
-COPY certbot /opt/certbot/src/certbot/
-COPY acme /opt/certbot/src/acme/
-COPY certbot-apache /opt/certbot/src/certbot-apache/
-COPY certbot-nginx /opt/certbot/src/certbot-nginx/
-
-
-RUN virtualenv --no-site-packages -p python2 /opt/certbot/venv
-
-# PATH is set now so pipstrap upgrades the correct (v)env
-ENV PATH /opt/certbot/venv/bin:$PATH
-RUN /opt/certbot/venv/bin/python /opt/certbot/src/pipstrap.py && \
-    /opt/certbot/venv/bin/pip install \
-    -e /opt/certbot/src/acme \
-    -e /opt/certbot/src \
-    -e /opt/certbot/src/certbot-apache \
-    -e /opt/certbot/src/certbot-nginx
-
-# install in editable mode (-e) to save space: it's not possible to
-# "rm -rf /opt/certbot/src" (it's stays in the underlaying image);
-# this might also help in debugging: you can "docker run --entrypoint
-# bash" and investigate, apply patches, etc.
+FROM python:2-alpine
 
 ENTRYPOINT [ "certbot" ]
+EXPOSE 80 443
+VOLUME /etc/letsencrypt /var/lib/letsencrypt
+WORKDIR /opt/certbot
+
+COPY CHANGES.rst README.rst setup.py src/
+COPY acme src/acme
+COPY certbot src/certbot
+
+RUN apk add --no-cache --virtual .certbot-deps \
+        libffi \
+        libssl1.0 \
+        ca-certificates \
+        binutils
+RUN apk add --no-cache --virtual .build-deps \
+        gcc \
+        linux-headers \
+        openssl-dev \
+        musl-dev \
+        libffi-dev \
+    && pip install --no-cache-dir \
+        --editable /opt/certbot/src/acme \
+        --editable /opt/certbot/src \
+    && apk del .build-deps

Dockerfile-dev

@@ -13,17 +13,17 @@ EXPOSE 443
 # authenticator and text mode only?)
 VOLUME /etc/letsencrypt /var/lib/letsencrypt
 
-WORKDIR /opt/certbot
+WORKDIR /opt/certbot/src
 
 # no need to mkdir anything:
 # https://docs.docker.com/reference/builder/#copy
 # If <dest> doesn't exist, it is created along with all missing
 # directories in its path.
 
-# TODO: Install non-default Python versions for tox.
 # TODO: Install Apache/Nginx for plugin development.
 COPY letsencrypt-auto-source/letsencrypt-auto /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto
 RUN /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto --os-packages-only && \
+    apt-get install python3-dev -y && \
     apt-get clean && \
     rm -rf /var/lib/apt/lists/* \
            /tmp/* \
@@ -51,6 +51,8 @@ COPY certbot-compatibility-test /opt/certbot/src/certbot-compatibility-test/
 COPY tests /opt/certbot/src/tests/
 
 RUN virtualenv --no-site-packages -p python2 /opt/certbot/venv && \
+    /opt/certbot/venv/bin/pip install -U pip && \
+    /opt/certbot/venv/bin/pip install -U setuptools && \
     /opt/certbot/venv/bin/pip install \
     -e /opt/certbot/src/acme \
     -e /opt/certbot/src \

Dockerfile-old

@@ -0,0 +1,75 @@
+# https://github.com/letsencrypt/letsencrypt/pull/431#issuecomment-103659297
+# it is more likely developers will already have ubuntu:trusty rather
+# than e.g. debian:jessie and image size differences are negligible
+FROM ubuntu:trusty
+MAINTAINER Jakub Warmuz <jakub@warmuz.org>
+MAINTAINER William Budington <bill@eff.org>
+
+# Note: this only exposes the port to other docker containers. You
+# still have to bind to 443@host at runtime, as per the ACME spec.
+EXPOSE 443
+
+# TODO: make sure --config-dir and --work-dir cannot be changed
+# through the CLI (certbot-docker wrapper that uses standalone
+# authenticator and text mode only?)
+VOLUME /etc/letsencrypt /var/lib/letsencrypt
+
+WORKDIR /opt/certbot
+
+# no need to mkdir anything:
+# https://docs.docker.com/reference/builder/#copy
+# If <dest> doesn't exist, it is created along with all missing
+# directories in its path.
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+COPY letsencrypt-auto-source/letsencrypt-auto /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto
+RUN /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto --os-packages-only && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/* \
+           /tmp/* \
+           /var/tmp/*
+
+# the above is not likely to change, so by putting it further up the
+# Dockerfile we make sure we cache as much as possible
+
+
+COPY setup.py README.rst CHANGES.rst MANIFEST.in letsencrypt-auto-source/pieces/pipstrap.py /opt/certbot/src/
+
+# all above files are necessary for setup.py and venv setup, however,
+# package source code directory has to be copied separately to a
+# subdirectory...
+# https://docs.docker.com/reference/builder/#copy: "If <src> is a
+# directory, the entire contents of the directory are copied,
+# including filesystem metadata. Note: The directory itself is not
+# copied, just its contents." Order again matters, three files are far
+# more likely to be cached than the whole project directory
+
+COPY certbot /opt/certbot/src/certbot/
+COPY acme /opt/certbot/src/acme/
+COPY certbot-apache /opt/certbot/src/certbot-apache/
+COPY certbot-nginx /opt/certbot/src/certbot-nginx/
+
+
+RUN virtualenv --no-site-packages -p python2 /opt/certbot/venv
+
+# PATH is set now so pipstrap upgrades the correct (v)env
+ENV PATH /opt/certbot/venv/bin:$PATH
+RUN /opt/certbot/venv/bin/python /opt/certbot/src/pipstrap.py && \
+    /opt/certbot/venv/bin/pip install \
+    -e /opt/certbot/src/acme \
+    -e /opt/certbot/src \
+    -e /opt/certbot/src/certbot-apache \
+    -e /opt/certbot/src/certbot-nginx
+
+# install in editable mode (-e) to save space: it's not possible to
+# "rm -rf /opt/certbot/src" (it's stays in the underlaying image);
+# this might also help in debugging: you can "docker run --entrypoint
+# bash" and investigate, apply patches, etc.
+
+# set up certbot/letsencrypt wrapper to warn people about Dockerfile changes
+COPY tools/docker-warning.sh /opt/certbot/bin/certbot
+RUN ln -s /opt/certbot/bin/certbot /opt/certbot/bin/letsencrypt
+ENV PATH /opt/certbot/bin:$PATH
+
+ENTRYPOINT [ "certbot" ]

ISSUE_TEMPLATE.md

@@ -0,0 +1,14 @@
+## My operating system is (include version):
+
+
+## I installed Certbot with (certbot-auto, OS package manager, pip, etc):
+
+
+## I ran this command and it produced this output:
+
+
+## Certbot's behavior differed from what I expected because:
+
+
+## Here is a Certbot log showing the issue (if available):
+###### Logs are stored in `/var/log/letsencrypt` by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

README.rst

@@ -1,12 +1,12 @@
 .. This file contains a series of comments that are used to include sections of this README in other files. Do not modify these comments unless you know what you are doing. tag:intro-begin
 
-Certbot is part of EFF’s effort to encrypt the entire Internet. Secure communication over the Web relies on HTTPS, which requires the use of a digital certificate that lets browsers verify the identify of web servers (e.g., is that really google.com?). Web servers obtain their certificates from trusted third parties called certificate authorities (CAs). Certbot is an easy-to-use client that fetches a certificate from Let’s Encrypt—an open certificate authority launched by the EFF, Mozilla, and others—and deploys it to a web server.
+Certbot is part of EFF’s effort to encrypt the entire Internet. Secure communication over the Web relies on HTTPS, which requires the use of a digital certificate that lets browsers verify the identity of web servers (e.g., is that really google.com?). Web servers obtain their certificates from trusted third parties called certificate authorities (CAs). Certbot is an easy-to-use client that fetches a certificate from Let’s Encrypt—an open certificate authority launched by the EFF, Mozilla, and others—and deploys it to a web server.
 
 Anyone who has gone through the trouble of setting up a secure website knows what a hassle getting and maintaining a certificate is. Certbot and Let’s Encrypt can automate away the pain and let you turn on and manage HTTPS with simple commands. Using Certbot and Let's Encrypt is free, so there’s no need to arrange payment.
 
-How you use Certbot depends on the configuration of your web server. The best way to get started is to use our `interactive guide <https://certbot.eff.org>`_. It generates instructions based on your configuration settings. In most cases, you’ll need `root or administrator access <https://certbot.eff.org/faq/#does-certbot-require-root-privileges>`_ to your web server to run Certbot.
+How you use Certbot depends on the configuration of your web server. The best way to get started is to use our `interactive guide <https://certbot.eff.org>`_. It generates instructions based on your configuration settings. In most cases, you’ll need `root or administrator access <https://certbot.eff.org/faq/#does-certbot-require-root-administrator-privileges>`_ to your web server to run Certbot.
 
-If you’re using a hosted service and don’t have direct access to your web server, you might not be able to use Certbot. Check with your hosting provider for documentation about uploading certificates or using certificates issues by Let’s Encrypt.
+If you’re using a hosted service and don’t have direct access to your web server, you might not be able to use Certbot. Check with your hosting provider for documentation about uploading certificates or using certificates issued by Let’s Encrypt.
 
 Certbot is a fully-featured, extensible client for the Let's
 Encrypt CA (or any other CA that speaks the `ACME
@@ -88,7 +88,7 @@ Main Website: https://certbot.eff.org
 
 Let's Encrypt Website: https://letsencrypt.org
 
-IRC Channel: #letsencrypt on `Freenode`_ or #certbot on `OFTC`_
+IRC Channel: #letsencrypt on `Freenode`_
 
 Community: https://community.letsencrypt.org
 
@@ -129,19 +129,7 @@ email to client-dev+subscribe@letsencrypt.org)
 System Requirements
 ===================
 
-The Let's Encrypt Client presently only runs on Unix-ish OSes that include
-Python 2.6 or 2.7; Python 3.x support will hopefully be added in the future. The
-client requires root access in order to write to ``/etc/letsencrypt``,
-``/var/log/letsencrypt``, ``/var/lib/letsencrypt``; to bind to ports 80 and 443
-(if you use the ``standalone`` plugin) and to read and modify webserver
-configurations (if you use the ``apache`` or ``nginx`` plugins).  If none of
-these apply to you, it is theoretically possible to run without root privileges,
-but for most users who want to avoid running an ACME client as root, either
-`letsencrypt-nosudo <https://github.com/diafygi/letsencrypt-nosudo>`_ or
-`simp_le <https://github.com/kuba/simp_le>`_ are more appropriate choices.
-
-The Apache plugin currently requires a Debian-based OS with augeas version
-1.0; this includes Ubuntu 12.04+ and Debian 7+.
+See https://certbot.eff.org/docs/install.html#system-requirements.
 
 .. Do not modify this comment unless you know what you're doing. tag:intro-end
 

acme/acme/challenges.py

@@ -5,7 +5,7 @@ import hashlib
 import logging
 import socket
 
-from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives import hashes  # type: ignore
 import OpenSSL
 import requests
 
@@ -23,7 +23,7 @@ logger = logging.getLogger(__name__)
 class Challenge(jose.TypedJSONObjectWithFields):
     # _fields_to_partial_json | pylint: disable=abstract-method
     """ACME challenge."""
-    TYPES = {}
+    TYPES = {}  # type: dict
 
     @classmethod
     def from_json(cls, jobj):
@@ -37,7 +37,7 @@ class Challenge(jose.TypedJSONObjectWithFields):
 class ChallengeResponse(jose.TypedJSONObjectWithFields):
     # _fields_to_partial_json | pylint: disable=abstract-method
     """ACME challenge response."""
-    TYPES = {}
+    TYPES = {}  # type: dict
     resource_type = 'challenge'
     resource = fields.Resource(resource_type)
 
@@ -425,7 +425,7 @@ class TLSSNI01Response(KeyAuthorizationChallengeResponse):
         # TODO: domain is not necessary if host is provided
         if "host" not in kwargs:
             host = socket.gethostbyname(domain)
-            logging.debug('%s resolved to %s', domain, host)
+            logger.debug('%s resolved to %s', domain, host)
             kwargs["host"] = host
 
         kwargs.setdefault("port", self.PORT)
@@ -445,7 +445,7 @@ class TLSSNI01Response(KeyAuthorizationChallengeResponse):
         """
         # pylint: disable=protected-access
         sans = crypto_util._pyopenssl_cert_or_req_san(cert)
-        logging.debug('Certificate %s. SANs: %s', cert.digest('sha1'), sans)
+        logger.debug('Certificate %s. SANs: %s', cert.digest('sha256'), sans)
         return self.z_domain.decode() in sans
 
     def simple_verify(self, chall, domain, account_public_key,

acme/acme/client.py

@@ -28,11 +28,13 @@ logger = logging.getLogger(__name__)
 # https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning
 if sys.version_info < (2, 7, 9):  # pragma: no cover
     try:
-        requests.packages.urllib3.contrib.pyopenssl.inject_into_urllib3()
+        requests.packages.urllib3.contrib.pyopenssl.inject_into_urllib3()  # type: ignore
     except AttributeError:
         import urllib3.contrib.pyopenssl  # pylint: disable=import-error
         urllib3.contrib.pyopenssl.inject_into_urllib3()
 
+DEFAULT_NETWORK_TIMEOUT = 45
+
 DER_CONTENT_TYPE = 'application/pkix-cert'
 
 
@@ -127,8 +129,6 @@ class Client(object):  # pylint: disable=too-many-instance-attributes
         update = regr.body if update is None else update
         body = messages.UpdateRegistration(**dict(update))
         updated_regr = self._send_recv_regr(regr, body=body)
-        if updated_regr != regr:
-            raise errors.UnexpectedUpdate(regr)
         return updated_regr
 
     def deactivate_registration(self, regr):
@@ -175,22 +175,25 @@ class Client(object):  # pylint: disable=too-many-instance-attributes
             raise errors.UnexpectedUpdate(authzr)
         return authzr
 
-    def request_challenges(self, identifier):
+    def request_challenges(self, identifier, new_authzr_uri=None):
         """Request challenges.
 
         :param .messages.Identifier identifier: Identifier to be challenged.
+        :param str new_authzr_uri: Deprecated. Do not use.
 
         :returns: Authorization Resource.
         :rtype: `.AuthorizationResource`
 
         """
+        if new_authzr_uri is not None:
+            logger.debug("request_challenges with new_authzr_uri deprecated.")
         new_authz = messages.NewAuthorization(identifier=identifier)
         response = self.net.post(self.directory.new_authz, new_authz)
         # TODO: handle errors
         assert response.status_code == http_client.CREATED
         return self._authzr_from_response(response, identifier)
 
-    def request_domain_challenges(self, domain):
+    def request_domain_challenges(self, domain, new_authzr_uri=None):
         """Request challenges for domain names.
 
         This is simply a convenience function that wraps around
@@ -199,13 +202,14 @@ class Client(object):  # pylint: disable=too-many-instance-attributes
         documentation.
 
         :param str domain: Domain name to be challenged.
+        :param str new_authzr_uri: Deprecated. Do not use.
 
         :returns: Authorization Resource.
         :rtype: `.AuthorizationResource`
 
         """
         return self.request_challenges(messages.Identifier(
-            typ=messages.IDENTIFIER_FQDN, value=domain))
+            typ=messages.IDENTIFIER_FQDN, value=domain), new_authzr_uri)
 
     def answer_challenge(self, challb, response):
         """Answer challenge.
@@ -281,7 +285,6 @@ class Client(object):  # pylint: disable=too-many-instance-attributes
         response = self.net.get(authzr.uri)
         updated_authzr = self._authzr_from_response(
             response, authzr.body.identifier, authzr.uri)
-        # TODO: check and raise UnexpectedUpdate
         return updated_authzr, response
 
     def request_issuance(self, csr, authzrs):
@@ -357,14 +360,18 @@ class Client(object):  # pylint: disable=too-many-instance-attributes
 
         # priority queue with datetime.datetime (based on Retry-After) as key,
         # and original Authorization Resource as value
-        waiting = [(datetime.datetime.now(), authzr) for authzr in authzrs]
+        waiting = [
+            (datetime.datetime.now(), index, authzr)
+            for index, authzr in enumerate(authzrs)
+        ]
+        heapq.heapify(waiting)
         # mapping between original Authorization Resource and the most
         # recently updated one
         updated = dict((authzr, authzr) for authzr in authzrs)
 
         while waiting:
             # find the smallest Retry-After, and sleep if necessary
-            when, authzr = heapq.heappop(waiting)
+            when, index, authzr = heapq.heappop(waiting)
             now = datetime.datetime.now()
             if when > now:
                 seconds = (when - now).seconds
@@ -383,7 +390,7 @@ class Client(object):  # pylint: disable=too-many-instance-attributes
                 if attempts[authzr] < max_attempts:
                     # push back to the priority queue, with updated retry_after
                     heapq.heappush(waiting, (self.retry_after(
-                        response, default=mintime), authzr))
+                        response, default=mintime), index, authzr))
                 else:
                     exhausted.add(authzr)
 
@@ -502,13 +509,14 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
     REPLAY_NONCE_HEADER = 'Replay-Nonce'
 
     def __init__(self, key, alg=jose.RS256, verify_ssl=True,
-                 user_agent='acme-python'):
+                 user_agent='acme-python', timeout=DEFAULT_NETWORK_TIMEOUT):
         self.key = key
         self.alg = alg
         self.verify_ssl = verify_ssl
         self._nonces = set()
         self.user_agent = user_agent
         self.session = requests.Session()
+        self._default_timeout = timeout
 
     def __del__(self):
         self.session.close()
@@ -600,13 +608,14 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
 
         """
         if method == "POST":
-            logging.debug('Sending POST request to %s:\n%s',
+            logger.debug('Sending POST request to %s:\n%s',
                           url, kwargs['data'])
         else:
-            logging.debug('Sending %s request to %s.', method, url)
+            logger.debug('Sending %s request to %s.', method, url)
         kwargs['verify'] = self.verify_ssl
         kwargs.setdefault('headers', {})
         kwargs['headers'].setdefault('User-Agent', self.user_agent)
+        kwargs.setdefault('timeout', self._default_timeout)
         response = self.session.request(method, url, *args, **kwargs)
         # If content is DER, log the base64 of it instead of raw bytes, to keep
         # binary data out of the logs.
@@ -650,7 +659,7 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
 
     def _get_nonce(self, url):
         if not self._nonces:
-            logging.debug('Requesting fresh nonce')
+            logger.debug('Requesting fresh nonce')
             self._add_nonce(self.head(url))
         return self._nonces.pop()
 

acme/acme/client_test.py

@@ -115,8 +115,6 @@ class ClientTest(unittest.TestCase):
         # TODO: split here and separate test
         self.response.json.return_value = self.regr.body.update(
             contact=()).to_json()
-        self.assertRaises(
-            errors.UnexpectedUpdate, self.client.update_registration, self.regr)
 
     def test_deactivate_account(self):
         self.response.headers['Location'] = self.regr.uri
@@ -124,14 +122,6 @@ class ClientTest(unittest.TestCase):
         self.assertEqual(self.regr,
                          self.client.deactivate_registration(self.regr))
 
-    def test_deactivate_account_bad_registration_returned(self):
-        self.response.headers['Location'] = self.regr.uri
-        self.response.json.return_value = "some wrong registration thing"
-        self.assertRaises(
-            errors.UnexpectedUpdate,
-            self.client.deactivate_registration,
-            self.regr)
-
     def test_query_registration(self):
         self.response.json.return_value = self.regr.body.to_json()
         self.assertEqual(self.regr, self.client.query_registration(self.regr))
@@ -154,6 +144,13 @@ class ClientTest(unittest.TestCase):
             self.directory.new_authz,
             messages.NewAuthorization(identifier=self.identifier))
 
+    def test_request_challenges_deprecated_arg(self):
+        self._prepare_response_for_request_challenges()
+        self.client.request_challenges(self.identifier, new_authzr_uri="hi")
+        self.net.post.assert_called_once_with(
+            self.directory.new_authz,
+            messages.NewAuthorization(identifier=self.identifier))
+
     def test_request_challenges_custom_uri(self):
         self._prepare_response_for_request_challenges()
         self.client.request_challenges(self.identifier)
@@ -532,7 +529,7 @@ class ClientNetworkTest(unittest.TestCase):
             'HEAD', 'http://example.com/', 'foo', bar='baz'))
         self.net.session.request.assert_called_once_with(
             'HEAD', 'http://example.com/', 'foo',
-            headers=mock.ANY, verify=mock.ANY, bar='baz')
+            headers=mock.ANY, verify=mock.ANY, timeout=mock.ANY, bar='baz')
 
     @mock.patch('acme.client.logger')
     def test_send_request_get_der(self, mock_logger):
@@ -542,8 +539,9 @@ class ClientNetworkTest(unittest.TestCase):
             headers={"Content-Type": "application/pkix-cert"},
             content=b"hi")
         # pylint: disable=protected-access
-        self.net._send_request('HEAD', 'http://example.com/', 'foo', bar='baz')
-        mock_logger.debug.assert_called_once_with(
+        self.net._send_request('HEAD', 'http://example.com/', 'foo',
+          timeout=mock.ANY, bar='baz')
+        mock_logger.debug.assert_called_with(
             'Received response:\nHTTP %d\n%s\n\n%s', 200,
             'Content-Type: application/pkix-cert', b'aGk=')
 
@@ -555,7 +553,7 @@ class ClientNetworkTest(unittest.TestCase):
             'POST', 'http://example.com/', 'foo', data='qux', bar='baz'))
         self.net.session.request.assert_called_once_with(
             'POST', 'http://example.com/', 'foo',
-            headers=mock.ANY, verify=mock.ANY, data='qux', bar='baz')
+            headers=mock.ANY, verify=mock.ANY, timeout=mock.ANY, data='qux', bar='baz')
 
     def test_send_request_verify_ssl(self):
         # pylint: disable=protected-access
@@ -568,7 +566,8 @@ class ClientNetworkTest(unittest.TestCase):
                 self.response,
                 self.net._send_request('GET', 'http://example.com/'))
             self.net.session.request.assert_called_once_with(
-                'GET', 'http://example.com/', verify=verify, headers=mock.ANY)
+                'GET', 'http://example.com/', verify=verify,
+                timeout=mock.ANY, headers=mock.ANY)
 
     def test_send_request_user_agent(self):
         self.net.session = mock.MagicMock()
@@ -577,13 +576,23 @@ class ClientNetworkTest(unittest.TestCase):
                                headers={'bar': 'baz'})
         self.net.session.request.assert_called_once_with(
             'GET', 'http://example.com/', verify=mock.ANY,
+            timeout=mock.ANY,
             headers={'User-Agent': 'acme-python-test', 'bar': 'baz'})
 
         self.net._send_request('GET', 'http://example.com/',
                                headers={'User-Agent': 'foo2'})
         self.net.session.request.assert_called_with(
             'GET', 'http://example.com/',
-            verify=mock.ANY, headers={'User-Agent': 'foo2'})
+            verify=mock.ANY, timeout=mock.ANY, headers={'User-Agent': 'foo2'})
+
+    def test_send_request_timeout(self):
+        self.net.session = mock.MagicMock()
+        # pylint: disable=protected-access
+        self.net._send_request('GET', 'http://example.com/',
+                               headers={'bar': 'baz'})
+        self.net.session.request.assert_called_once_with(
+            mock.ANY, mock.ANY, verify=mock.ANY, headers=mock.ANY,
+            timeout=45)
 
     def test_del(self):
         sess = mock.MagicMock()

acme/acme/crypto_util.py

@@ -23,7 +23,7 @@ logger = logging.getLogger(__name__)
 # https://www.openssl.org/docs/ssl/SSLv23_method.html). _serve_sni
 # should be changed to use "set_options" to disable SSLv2 and SSLv3,
 # in case it's used for things other than probing/serving!
-_DEFAULT_TLSSNI01_SSL_METHOD = OpenSSL.SSL.SSLv23_METHOD
+_DEFAULT_TLSSNI01_SSL_METHOD = OpenSSL.SSL.SSLv23_METHOD  # type: ignore
 
 
 class SSLSocket(object):  # pylint: disable=too-few-public-methods
@@ -149,6 +149,36 @@ def probe_sni(name, host, port=443, timeout=300,
             raise errors.Error(error)
     return client_ssl.get_peer_certificate()
 
+def make_csr(private_key_pem, domains, must_staple=False):
+    """Generate a CSR containing a list of domains as subjectAltNames.
+
+    :param buffer private_key_pem: Private key, in PEM PKCS#8 format.
+    :param list domains: List of DNS names to include in subjectAltNames of CSR.
+    :param bool must_staple: Whether to include the TLS Feature extension (aka
+        OCSP Must Staple: https://tools.ietf.org/html/rfc7633).
+    :returns: buffer PEM-encoded Certificate Signing Request.
+    """
+    private_key = OpenSSL.crypto.load_privatekey(
+        OpenSSL.crypto.FILETYPE_PEM, private_key_pem)
+    csr = OpenSSL.crypto.X509Req()
+    extensions = [
+        OpenSSL.crypto.X509Extension(
+            b'subjectAltName',
+            critical=False,
+            value=', '.join('DNS:' + d for d in domains).encode('ascii')
+        ),
+    ]
+    if must_staple:
+        extensions.append(OpenSSL.crypto.X509Extension(
+            b"1.3.6.1.5.5.7.1.24",
+            critical=False,
+            value=b"DER:30:03:02:01:05"))
+    csr.add_extensions(extensions)
+    csr.set_pubkey(private_key)
+    csr.set_version(2)
+    csr.sign(private_key, 'sha256')
+    return OpenSSL.crypto.dump_certificate_request(
+        OpenSSL.crypto.FILETYPE_PEM, csr)
 
 def _pyopenssl_cert_or_req_san(cert_or_req):
     """Get Subject Alternative Names from certificate or CSR using pyOpenSSL.

acme/acme/crypto_util_test.py

@@ -6,7 +6,7 @@ import time
 import unittest
 
 import six
-from six.moves import socketserver  # pylint: disable=import-error
+from six.moves import socketserver  #type: ignore  # pylint: disable=import-error
 
 import OpenSSL
 
@@ -151,6 +151,53 @@ class RandomSnTest(unittest.TestCase):
             self.serial_num.append(cert.get_serial_number())
         self.assertTrue(len(set(self.serial_num)) > 1)
 
+class MakeCSRTest(unittest.TestCase):
+    """Test for standalone functions."""
+
+    @classmethod
+    def _call_with_key(cls, *args, **kwargs):
+        privkey = OpenSSL.crypto.PKey()
+        privkey.generate_key(OpenSSL.crypto.TYPE_RSA, 2048)
+        privkey_pem = OpenSSL.crypto.dump_privatekey(OpenSSL.crypto.FILETYPE_PEM, privkey)
+        from acme.crypto_util import make_csr
+        return make_csr(privkey_pem, *args, **kwargs)
+
+    def test_make_csr(self):
+        csr_pem = self._call_with_key(["a.example", "b.example"])
+        self.assertTrue(b'--BEGIN CERTIFICATE REQUEST--' in csr_pem)
+        self.assertTrue(b'--END CERTIFICATE REQUEST--' in csr_pem)
+        csr = OpenSSL.crypto.load_certificate_request(
+            OpenSSL.crypto.FILETYPE_PEM, csr_pem)
+        # In pyopenssl 0.13 (used with TOXENV=py26-oldest and py27-oldest), csr
+        # objects don't have a get_extensions() method, so we skip this test if
+        # the method isn't available.
+        if hasattr(csr, 'get_extensions'):
+            self.assertEquals(len(csr.get_extensions()), 1)
+            self.assertEquals(csr.get_extensions()[0].get_data(),
+                OpenSSL.crypto.X509Extension(
+                    b'subjectAltName',
+                    critical=False,
+                    value=b'DNS:a.example, DNS:b.example',
+                ).get_data(),
+            )
+
+    def test_make_csr_must_staple(self):
+        csr_pem = self._call_with_key(["a.example"], must_staple=True)
+        csr = OpenSSL.crypto.load_certificate_request(
+            OpenSSL.crypto.FILETYPE_PEM, csr_pem)
+
+        # In pyopenssl 0.13 (used with TOXENV=py26-oldest and py27-oldest), csr
+        # objects don't have a get_extensions() method, so we skip this test if
+        # the method isn't available.
+        if hasattr(csr, 'get_extensions'):
+            self.assertEquals(len(csr.get_extensions()), 2)
+            # NOTE: Ideally we would filter by the TLS Feature OID, but
+            # OpenSSL.crypto.X509Extension doesn't give us the extension's raw OID,
+            # and the shortname field is just "UNDEF"
+            must_staple_exts = [e for e in csr.get_extensions()
+                if e.get_data() == b"0\x03\x02\x01\x05"]
+            self.assertEqual(len(must_staple_exts), 1,
+                "Expected exactly one Must Staple extension")
 
 if __name__ == '__main__':
     unittest.main()  # pragma: no cover

acme/acme/jose/jwa.py

@@ -9,9 +9,9 @@ import logging
 
 import cryptography.exceptions
 from cryptography.hazmat.backends import default_backend
-from cryptography.hazmat.primitives import hashes
-from cryptography.hazmat.primitives import hmac
-from cryptography.hazmat.primitives.asymmetric import padding
+from cryptography.hazmat.primitives import hashes  # type: ignore
+from cryptography.hazmat.primitives import hmac  # type: ignore
+from cryptography.hazmat.primitives.asymmetric import padding  # type: ignore
 
 from acme.jose import errors
 from acme.jose import interfaces
@@ -28,9 +28,9 @@ class JWA(interfaces.JSONDeSerializable):  # pylint: disable=abstract-method
     """JSON Web Algorithm."""
 
 
-class JWASignature(JWA, collections.Hashable):
+class JWASignature(JWA, collections.Hashable):  # type: ignore
     """JSON Web Signature Algorithm."""
-    SIGNATURES = {}
+    SIGNATURES = {}  # type: dict
 
     def __init__(self, name):
         self.name = name

acme/acme/jose/jwk.py

@@ -6,9 +6,9 @@ import logging
 
 import cryptography.exceptions
 from cryptography.hazmat.backends import default_backend
-from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives import hashes  # type: ignore
 from cryptography.hazmat.primitives import serialization
-from cryptography.hazmat.primitives.asymmetric import ec
+from cryptography.hazmat.primitives.asymmetric import ec  # type: ignore
 from cryptography.hazmat.primitives.asymmetric import rsa
 
 import six
@@ -25,8 +25,8 @@ class JWK(json_util.TypedJSONObjectWithFields):
     # pylint: disable=too-few-public-methods
     """JSON Web Key."""
     type_field_name = 'kty'
-    TYPES = {}
-    cryptography_key_types = ()
+    TYPES = {}  # type: dict
+    cryptography_key_types = ()  # type: tuple
     """Subclasses should override."""
 
     required = NotImplemented

acme/acme/jose/jws.py

@@ -121,12 +121,12 @@ class Header(json_util.JSONObjectWithFields):
 
     # x5c does NOT use JOSE Base64 (4.1.6)
 
-    @x5c.encoder
+    @x5c.encoder  # type: ignore
     def x5c(value):  # pylint: disable=missing-docstring,no-self-argument
         return [base64.b64encode(OpenSSL.crypto.dump_certificate(
             OpenSSL.crypto.FILETYPE_ASN1, cert.wrapped)) for cert in value]
 
-    @x5c.decoder
+    @x5c.decoder  # type: ignore
     def x5c(value):  # pylint: disable=missing-docstring,no-self-argument
         try:
             return tuple(util.ComparableX509(OpenSSL.crypto.load_certificate(
@@ -157,12 +157,12 @@ class Signature(json_util.JSONObjectWithFields):
         'signature', decoder=json_util.decode_b64jose,
         encoder=json_util.encode_b64jose)
 
-    @protected.encoder
+    @protected.encoder  # type: ignore
     def protected(value):  # pylint: disable=missing-docstring,no-self-argument
         # wrong type guess (Signature, not bytes) | pylint: disable=no-member
         return json_util.encode_b64jose(value.encode('utf-8'))
 
-    @protected.decoder
+    @protected.decoder  # type: ignore
     def protected(value):  # pylint: disable=missing-docstring,no-self-argument
         return json_util.decode_b64jose(value).decode('utf-8')
 

acme/acme/jose/util.py

@@ -134,7 +134,7 @@ class ComparableRSAKey(ComparableKey):  # pylint: disable=too-few-public-methods
             return hash((self.__class__, pub.n, pub.e))
 
 
-class ImmutableMap(collections.Mapping, collections.Hashable):
+class ImmutableMap(collections.Mapping, collections.Hashable):  # type: ignore
     # pylint: disable=too-few-public-methods
     """Immutable key to value mapping with attribute access."""
 
@@ -180,7 +180,7 @@ class ImmutableMap(collections.Mapping, collections.Hashable):
             for key, value in six.iteritems(self)))
 
 
-class frozendict(collections.Mapping, collections.Hashable):
+class frozendict(collections.Mapping, collections.Hashable):  # type: ignore
     # pylint: disable=invalid-name,too-few-public-methods
     """Frozen dictionary."""
     __slots__ = ('_items', '_keys')

acme/acme/jws.py

@@ -1,14 +1,18 @@
-"""ACME JOSE JWS."""
+"""ACME-specific JWS.
+
+The JWS implementation in acme.jose only implements the base JOSE standard. In
+order to support the new header fields defined in ACME, this module defines some
+ACME-specific classes that layer on top of acme.jose.
+"""
 from acme import jose
 
 
 class Header(jose.Header):
-    """ACME JOSE Header.
-
-    .. todo:: Implement ``acmePath``.
-
+    """ACME-specific JOSE Header. Implements nonce, kid, and url.
     """
     nonce = jose.Field('nonce', omitempty=True, encoder=jose.encode_b64jose)
+    kid = jose.Field('kid', omitempty=True)
+    url = jose.Field('url', omitempty=True)
 
     @nonce.decoder
     def nonce(value):  # pylint: disable=missing-docstring,no-self-argument
@@ -20,7 +24,7 @@ class Header(jose.Header):
 
 
 class Signature(jose.Signature):
-    """ACME Signature."""
+    """ACME-specific Signature. Uses ACME-specific Header for customer fields."""
     __slots__ = jose.Signature._orig_slots  # pylint: disable=no-member
 
     # TODO: decoder/encoder should accept cls? Otherwise, subclassing
@@ -34,11 +38,17 @@ class Signature(jose.Signature):
 
 
 class JWS(jose.JWS):
-    """ACME JWS."""
+    """ACME-specific JWS. Includes none, url, and kid in protected header."""
     signature_cls = Signature
     __slots__ = jose.JWS._orig_slots  # pylint: disable=no-member
 
     @classmethod
-    def sign(cls, payload, key, alg, nonce):  # pylint: disable=arguments-differ
+    # pylint: disable=arguments-differ,too-many-arguments
+    def sign(cls, payload, key, alg, nonce, url=None, kid=None):
+        # Per ACME spec, jwk and kid are mutually exclusive, so only include a
+        # jwk field if kid is not provided.
+        include_jwk = kid is None
         return super(JWS, cls).sign(payload, key=key, alg=alg,
-                                    protect=frozenset(['nonce']), nonce=nonce)
+                                    protect=frozenset(['nonce', 'url', 'kid']),
+                                    nonce=nonce, url=url, kid=kid,
+                                    include_jwk=include_jwk)

acme/acme/jws_test.py

@@ -37,16 +37,30 @@ class JWSTest(unittest.TestCase):
         self.privkey = KEY
         self.pubkey = self.privkey.public_key()
         self.nonce = jose.b64encode(b'Nonce')
+        self.url = 'hi'
+        self.kid = 'baaaaa'
 
-    def test_it(self):
+    def test_kid_serialize(self):
         from acme.jws import JWS
         jws = JWS.sign(payload=b'foo', key=self.privkey,
-                       alg=jose.RS256, nonce=self.nonce)
+                       alg=jose.RS256, nonce=self.nonce,
+                       url=self.url, kid=self.kid)
         self.assertEqual(jws.signature.combined.nonce, self.nonce)
+        self.assertEqual(jws.signature.combined.url, self.url)
+        self.assertEqual(jws.signature.combined.kid, self.kid)
+        self.assertEqual(jws.signature.combined.jwk, None)
         # TODO: check that nonce is in protected header
 
         self.assertEqual(jws, JWS.from_json(jws.to_json()))
 
+    def test_jwk_serialize(self):
+        from acme.jws import JWS
+        jws = JWS.sign(payload=b'foo', key=self.privkey,
+                       alg=jose.RS256, nonce=self.nonce,
+                       url=self.url)
+        self.assertEqual(jws.signature.combined.kid, None)
+        self.assertEqual(jws.signature.combined.jwk, self.pubkey)
+
 
 if __name__ == '__main__':
     unittest.main()  # pragma: no cover

acme/acme/messages.py

@@ -98,7 +98,7 @@ class Error(jose.JSONObjectWithFields, errors.Error):
             if part is not None)
 
 
-class _Constant(jose.JSONDeSerializable, collections.Hashable):
+class _Constant(jose.JSONDeSerializable, collections.Hashable):  # type: ignore
     """ACME constant."""
     __slots__ = ('name',)
     POSSIBLE_NAMES = NotImplemented
@@ -132,7 +132,7 @@ class _Constant(jose.JSONDeSerializable, collections.Hashable):
 
 class Status(_Constant):
     """ACME "status" field."""
-    POSSIBLE_NAMES = {}
+    POSSIBLE_NAMES = {}  # type: dict
 STATUS_UNKNOWN = Status('unknown')
 STATUS_PENDING = Status('pending')
 STATUS_PROCESSING = Status('processing')
@@ -143,7 +143,7 @@ STATUS_REVOKED = Status('revoked')
 
 class IdentifierType(_Constant):
     """ACME identifier type."""
-    POSSIBLE_NAMES = {}
+    POSSIBLE_NAMES = {}  # type: dict
 IDENTIFIER_FQDN = IdentifierType('dns')  # IdentifierDNS in Boulder
 
 
@@ -161,7 +161,7 @@ class Identifier(jose.JSONObjectWithFields):
 class Directory(jose.JSONDeSerializable):
     """Directory."""
 
-    _REGISTERED_TYPES = {}
+    _REGISTERED_TYPES = {}  # type: dict
 
     class Meta(jose.JSONObjectWithFields):
         """Directory Meta."""
@@ -237,10 +237,6 @@ class Registration(ResourceBody):
     :ivar tuple contact: Contact information following ACME spec,
         `tuple` of `unicode`.
     :ivar unicode agreement:
-    :ivar unicode authorizations: URI where
-        `messages.Registration.Authorizations` can be found.
-    :ivar unicode certificates: URI where
-        `messages.Registration.Certificates` can be found.
 
     """
     # on new-reg key server ignores 'key' and populates it based on
@@ -248,26 +244,8 @@ class Registration(ResourceBody):
     key = jose.Field('key', omitempty=True, decoder=jose.JWK.from_json)
     contact = jose.Field('contact', omitempty=True, default=())
     agreement = jose.Field('agreement', omitempty=True)
-    authorizations = jose.Field('authorizations', omitempty=True)
-    certificates = jose.Field('certificates', omitempty=True)
     status = jose.Field('status', omitempty=True)
 
-    class Authorizations(jose.JSONObjectWithFields):
-        """Authorizations granted to Account in the process of registration.
-
-        :ivar tuple authorizations: URIs to Authorization Resources.
-
-        """
-        authorizations = jose.Field('authorizations')
-
-    class Certificates(jose.JSONObjectWithFields):
-        """Certificates granted to Account in the process of registration.
-
-        :ivar tuple certificates: URIs to Certificate Resources.
-
-        """
-        certificates = jose.Field('certificates')
-
     phone_prefix = 'tel:'
     email_prefix = 'mailto:'
 
@@ -315,10 +293,12 @@ class RegistrationResource(ResourceWithURI):
     """Registration Resource.
 
     :ivar acme.messages.Registration body:
+    :ivar unicode new_authzr_uri: Deprecated. Do not use.
     :ivar unicode terms_of_service: URL for the CA TOS.
 
     """
     body = jose.Field('body', decoder=Registration.from_json)
+    new_authzr_uri = jose.Field('new_authzr_uri', omitempty=True)
     terms_of_service = jose.Field('terms_of_service', omitempty=True)
 
 
@@ -423,9 +403,11 @@ class AuthorizationResource(ResourceWithURI):
     """Authorization Resource.
 
     :ivar acme.messages.Authorization body:
+    :ivar unicode new_cert_uri: Deprecated. Do not use.
 
     """
     body = jose.Field('body', decoder=Authorization.from_json)
+    new_cert_uri = jose.Field('new_cert_uri', omitempty=True)
 
 
 @Directory.register

acme/acme/messages_test.py

@@ -170,8 +170,7 @@ class RegistrationTest(unittest.TestCase):
 
         from acme.messages import Registration
         self.reg = Registration(key=key, contact=contact, agreement=agreement)
-        self.reg_none = Registration(authorizations='uri/authorizations',
-                                     certificates='uri/certificates')
+        self.reg_none = Registration()
 
         self.jobj_to = {
             'contact': contact,

acme/acme/standalone.py

@@ -6,9 +6,9 @@ import logging
 import os
 import sys
 
-from six.moves import BaseHTTPServer  # pylint: disable=import-error
+from six.moves import BaseHTTPServer  # type: ignore  # pylint: disable=import-error
 from six.moves import http_client  # pylint: disable=import-error
-from six.moves import socketserver  # pylint: disable=import-error
+from six.moves import socketserver  # type: ignore  # pylint: disable=import-error
 
 import OpenSSL
 

acme/acme/standalone_test.py

@@ -7,7 +7,7 @@ import time
 import unittest
 
 from six.moves import http_client  # pylint: disable=import-error
-from six.moves import socketserver  # pylint: disable=import-error
+from six.moves import socketserver  # type: ignore  # pylint: disable=import-error
 
 import requests
 

acme/setup.py

@@ -4,35 +4,30 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.12.0.dev0'
+version = '0.14.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [
+    'argparse',
     # load_pem_private/public_key (>=0.6)
     # rsa_recover_prime_factors (>=0.8)
     'cryptography>=0.8',
     # Connection.set_tlsext_host_name (>=0.13)
+    'mock',
     'PyOpenSSL>=0.13',
     'pyrfc3339',
     'pytz',
-    'requests[security]>=2.4.1',  # security extras added in 2.4.1
+    # requests>=2.10 is required to fix
+    # https://github.com/shazow/urllib3/issues/556. This requirement can be
+    # relaxed to 'requests[security]>=2.4.1', however, less useful errors
+    # will be raised for some network/SSL errors.
+    'requests[security]>=2.10',
     # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
     # will tolerate; see #2599:
     'setuptools>=1.0',
     'six',
 ]
 
-# env markers in extras_require cause problems with older pip: #517
-# Keep in sync with conditional_requirements.py.
-if sys.version_info < (2, 7):
-    install_requires.extend([
-        # only some distros recognize stdlib argparse as already satisfying
-        'argparse',
-        'mock<1.1.0',
-    ])
-else:
-    install_requires.append('mock')
-
 dev_extras = [
     'nose',
     'tox',

certbot-apache/certbot_apache/configurator.py

@@ -174,6 +174,8 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         # Set Version
         if self.version is None:
             self.version = self.get_version()
+            logger.debug('Apache version is %s',
+                         '.'.join(str(i) for i in self.version))
         if self.version < (2, 2):
             raise errors.NotSupportedError(
                 "Apache Version %s not supported.", str(self.version))
@@ -254,9 +256,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             raise errors.PluginError(
                 "Unable to find cert and/or key directives")
 
-        logger.info("Deploying Certificate to VirtualHost %s", vhost.filep)
-        logger.debug("Apache version is %s",
-                     ".".join(str(i) for i in self.version))
+        logger.info("Deploying Certificate for %s to VirtualHost %s", domain, vhost.filep)
 
         if self.version < (2, 4, 8) or (chain_path and not fullchain_path):
             # install SSLCertificateFile, SSLCertificateKeyFile,
@@ -580,7 +580,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
                 ("/files%s//*[label()=~regexp('%s')]" %
                     (vhost_path, parser.case_i("VirtualHost"))))
             paths = [path for path in paths if
-                     os.path.basename(path) == "VirtualHost"]
+                     os.path.basename(path.lower()) == "virtualhost"]
             for path in paths:
                 new_vhost = self._create_vhost(path)
                 if not new_vhost:
@@ -1315,18 +1315,15 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             #     even with save() and load()
             if not self._is_rewrite_engine_on(general_vh):
                 self.parser.add_dir(general_vh.path, "RewriteEngine", "on")
+
             names = ssl_vhost.get_names()
             for idx, name in enumerate(names):
                 args = ["%{SERVER_NAME}", "={0}".format(name), "[OR]"]
                 if idx == len(names) - 1:
                     args.pop()
                 self.parser.add_dir(general_vh.path, "RewriteCond", args)
-            if self.get_version() >= (2, 3, 9):
-                self.parser.add_dir(general_vh.path, "RewriteRule",
-                                    constants.REWRITE_HTTPS_ARGS_WITH_END)
-            else:
-                self.parser.add_dir(general_vh.path, "RewriteRule",
-                                    constants.REWRITE_HTTPS_ARGS)
+
+            self._set_https_redirection_rewrite_rule(general_vh)
 
             self.save_notes += ("Redirecting host in %s to ssl vhost in %s\n" %
                                 (general_vh.filep, ssl_vhost.filep))
@@ -1336,12 +1333,24 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             logger.info("Redirecting vhost in %s to ssl vhost in %s",
                         general_vh.filep, ssl_vhost.filep)
 
+    def _set_https_redirection_rewrite_rule(self, vhost):
+        if self.get_version() >= (2, 3, 9):
+            self.parser.add_dir(vhost.path, "RewriteRule",
+                    constants.REWRITE_HTTPS_ARGS_WITH_END)
+        else:
+            self.parser.add_dir(vhost.path, "RewriteRule",
+                    constants.REWRITE_HTTPS_ARGS)
+
+
     def _verify_no_certbot_redirect(self, vhost):
         """Checks to see if a redirect was already installed by certbot.
 
         Checks to see if virtualhost already contains a rewrite rule that is
         identical to Certbot's redirection rewrite rule.
 
+        For graceful transition to new rewrite rules for HTTPS redireciton we
+        delete certbot's old rewrite rules and set the new one instead.
+
         :param vhost: vhost to check
         :type vhost: :class:`~certbot_apache.obj.VirtualHost`
 
@@ -1355,19 +1364,29 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         # rewrite_args_dict keys are directive ids and the corresponding value
         # for each is a list of arguments to that directive.
         rewrite_args_dict = defaultdict(list)
-        pat = r'.*(directive\[\d+\]).*'
+        pat = r'(.*directive\[\d+\]).*'
         for match in rewrite_path:
             m = re.match(pat, match)
             if m:
-                dir_id = m.group(1)
-                rewrite_args_dict[dir_id].append(match)
+                dir_path = m.group(1)
+                rewrite_args_dict[dir_path].append(match)
 
         if rewrite_args_dict:
             redirect_args = [constants.REWRITE_HTTPS_ARGS,
                              constants.REWRITE_HTTPS_ARGS_WITH_END]
 
-            for matches in rewrite_args_dict.values():
-                if [self.aug.get(x) for x in matches] in redirect_args:
+            for dir_path, args_paths in rewrite_args_dict.items():
+                arg_vals = [self.aug.get(x) for x in args_paths]
+
+                # Search for past redirection rule, delete it, set the new one
+                if arg_vals in constants.OLD_REWRITE_HTTPS_ARGS:
+                    self.aug.remove(dir_path)
+                    self._set_https_redirection_rewrite_rule(vhost)
+                    self.save()
+                    raise errors.PluginEnhancementAlreadyPresent(
+                        "Certbot has already enabled redirection")
+
+                if arg_vals in redirect_args:
                     raise errors.PluginEnhancementAlreadyPresent(
                         "Certbot has already enabled redirection")
 

certbot-apache/certbot_apache/constants.py

@@ -136,15 +136,19 @@ AUGEAS_LENS_DIR = pkg_resources.resource_filename(
 """Path to the Augeas lens directory"""
 
 REWRITE_HTTPS_ARGS = [
-    "^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[L,QSA,R=permanent]"]
+    "^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[L,NE,R=permanent]"]
 """Apache version<2.3.9 rewrite rule arguments used for redirections to
 https vhost"""
 
 REWRITE_HTTPS_ARGS_WITH_END = [
-    "^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[END,QSA,R=permanent]"]
+    "^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[END,NE,R=permanent]"]
 """Apache version >= 2.3.9 rewrite rule arguments used for redirections to
     https vhost"""
 
+OLD_REWRITE_HTTPS_ARGS = [
+    ["^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[L,QSA,R=permanent]"],
+    ["^", "https://%{SERVER_NAME}%{REQUEST_URI}", "[END,QSA,R=permanent]"]]
+
 HSTS_ARGS = ["always", "set", "Strict-Transport-Security",
              "\"max-age=31536000\""]
 """Apache header arguments for HSTS"""

certbot-apache/certbot_apache/obj.py

@@ -25,6 +25,11 @@ class Addr(common.Addr):
     def __repr__(self):
         return "certbot_apache.obj.Addr(" + repr(self.tup) + ")"
 
+    def __hash__(self):
+        # Python 3 requires explicit overridden for __hash__ if __eq__ or
+        # __cmp__ is overridden. See https://bugs.python.org/issue2235
+        return super(Addr, self).__hash__()
+
     def _addr_less_specific(self, addr):
         """Returns if addr.get_addr() is more specific than self.get_addr()."""
         # pylint: disable=protected-access
@@ -174,6 +179,11 @@ class VirtualHost(object):  # pylint: disable=too-few-public-methods
     def __ne__(self, other):
         return not self.__eq__(other)
 
+    def __hash__(self):
+        return hash((self.filep, self.path,
+                     tuple(self.addrs), tuple(self.get_names()),
+                     self.ssl, self.enabled, self.modmacro))
+
     def conflicts(self, addrs):
         """See if vhost conflicts with any of the addrs.
 

certbot-apache/certbot_apache/parser.py

@@ -1,10 +1,12 @@
 """ApacheParser is a member object of the ApacheConfigurator class."""
 import fnmatch
-import itertools
 import logging
 import os
 import re
 import subprocess
+import sys
+
+import six
 
 from certbot import errors
 
@@ -87,7 +89,7 @@ class ApacheParser(object):
         while len(self.modules) != prev_size:
             prev_size = len(self.modules)
 
-            for match_name, match_filename in itertools.izip(
+            for match_name, match_filename in six.moves.zip(
                     iterator, iterator):
                 self.modules.add(self.get_arg(match_name))
                 self.modules.add(
@@ -134,7 +136,8 @@ class ApacheParser(object):
             proc = subprocess.Popen(
                 constants.os_constant("define_cmd"),
                 stdout=subprocess.PIPE,
-                stderr=subprocess.PIPE)
+                stderr=subprocess.PIPE,
+                universal_newlines=True)
             stdout, stderr = proc.communicate()
 
         except (OSError, ValueError):
@@ -460,8 +463,12 @@ class ApacheParser(object):
         :rtype: str
 
         """
-        # This strips off final /Z(?ms)
-        return fnmatch.translate(clean_fn_match)[:-7]
+        if sys.version_info < (3, 6):
+            # This strips off final /Z(?ms)
+            return fnmatch.translate(clean_fn_match)[:-7]
+        else:  # pragma: no cover
+            # Since Python 3.6, it returns a different pattern like (?s:.*\.load)\Z
+            return fnmatch.translate(clean_fn_match)[4:-3]
 
     def _parse_file(self, filepath):
         """Parse file with Augeas

certbot-apache/certbot_apache/tests/configurator_test.py

@@ -6,6 +6,8 @@ import socket
 import unittest
 
 import mock
+# six is used in mock.patch()
+import six  # pylint: disable=unused-import
 
 from acme import challenges
 
@@ -16,6 +18,7 @@ from certbot.tests import acme_util
 from certbot.tests import util as certbot_util
 
 from certbot_apache import configurator
+from certbot_apache import constants
 from certbot_apache import parser
 from certbot_apache import obj
 
@@ -517,12 +520,12 @@ class MultipleVhostsTest(util.ApacheTest):
         # Test
         self.config.prepare_server_https("8080", temp=True)
         self.assertEqual(mock_add_dir.call_count, 3)
-        self.assertEqual(mock_add_dir.call_args_list[0][0][2],
-                         ["1.2.3.4:8080", "https"])
-        self.assertEqual(mock_add_dir.call_args_list[1][0][2],
-                         ["[::1]:8080", "https"])
-        self.assertEqual(mock_add_dir.call_args_list[2][0][2],
-                         ["1.1.1.1:8080", "https"])
+        call_args_list = [mock_add_dir.call_args_list[i][0][2] for i in range(3)]
+        self.assertEqual(
+            sorted(call_args_list),
+            sorted([["1.2.3.4:8080", "https"],
+                    ["[::1]:8080", "https"],
+                    ["1.1.1.1:8080", "https"]]))
 
         # mock_get.side_effect = ["1.2.3.4:80", "[::1]:80"]
         # mock_find.return_value = ["test1", "test2", "test3"]
@@ -662,7 +665,7 @@ class MultipleVhostsTest(util.ApacheTest):
         # This calls open
         self.config.reverter.register_file_creation = mock.Mock()
         mock_open.side_effect = IOError
-        with mock.patch("__builtin__.open", mock_open):
+        with mock.patch("six.moves.builtins.open", mock_open):
             self.assertRaises(
                 errors.PluginError,
                 self.config.make_vhost_ssl, self.vh_truth[0])
@@ -1045,6 +1048,36 @@ class MultipleVhostsTest(util.ApacheTest):
 
         self.assertTrue("rewrite_module" in self.config.parser.modules)
 
+    @mock.patch("certbot.util.run_script")
+    @mock.patch("certbot.util.exe_exists")
+    def test_redirect_with_old_https_redirection(self, mock_exe, _):
+        self.config.parser.update_runtime_variables = mock.Mock()
+        mock_exe.return_value = True
+        self.config.get_version = mock.Mock(return_value=(2, 2, 0))
+
+        ssl_vhost = self.config.choose_vhost("certbot.demo")
+
+        # pylint: disable=protected-access
+        http_vhost = self.config._get_http_vhost(ssl_vhost)
+
+        # Create an old (previously suppoorted) https redirectoin rewrite rule
+        self.config.parser.add_dir(
+            http_vhost.path, "RewriteRule",
+            ["^",
+             "https://%{SERVER_NAME}%{REQUEST_URI}",
+             "[L,QSA,R=permanent]"])
+
+        self.config.save()
+
+        try:
+            self.config.enhance("certbot.demo", "redirect")
+        except errors.PluginEnhancementAlreadyPresent:
+            args_paths = self.config.parser.find_dir(
+                "RewriteRule", None, http_vhost.path, False)
+            arg_vals = [self.config.aug.get(x) for x in args_paths]
+            self.assertEqual(arg_vals, constants.REWRITE_HTTPS_ARGS)
+
+
     def test_redirect_with_conflict(self):
         self.config.parser.modules.add("rewrite_module")
         ssl_vh = obj.VirtualHost(
@@ -1132,7 +1165,7 @@ class MultipleVhostsTest(util.ApacheTest):
             http_vhost.path, "RewriteRule",
             ["^",
              "https://%{SERVER_NAME}%{REQUEST_URI}",
-             "[L,QSA,R=permanent]"])
+             "[L,NE,R=permanent]"])
         self.config.save()
 
         ssl_vhost = self.config.make_vhost_ssl(self.vh_truth[0])
@@ -1143,7 +1176,7 @@ class MultipleVhostsTest(util.ApacheTest):
         conf_text = open(ssl_vhost.filep).read()
         commented_rewrite_rule = ("# RewriteRule ^ "
                                   "https://%{SERVER_NAME}%{REQUEST_URI} "
-                                  "[L,QSA,R=permanent]")
+                                  "[L,NE,R=permanent]")
         self.assertTrue(commented_rewrite_rule in conf_text)
         mock_get_utility().add_message.assert_called_once_with(mock.ANY,
 
@@ -1162,7 +1195,7 @@ class MultipleVhostsTest(util.ApacheTest):
                 "RewriteCond", ["%{DOCUMENT_ROOT}/%{REQUEST_FILENAME}", "!-f"])
         self.config.parser.add_dir(
             http_vhost.path, "RewriteRule",
-            ["^(.*)$", "b://u%{REQUEST_URI}", "[P,QSA,L]"])
+            ["^(.*)$", "b://u%{REQUEST_URI}", "[P,NE,L]"])
 
         # Add a chunk that should be commented out.
         self.config.parser.add_dir(http_vhost.path,
@@ -1173,7 +1206,7 @@ class MultipleVhostsTest(util.ApacheTest):
             http_vhost.path, "RewriteRule",
             ["^",
              "https://%{SERVER_NAME}%{REQUEST_URI}",
-             "[L,QSA,R=permanent]"])
+             "[L,NE,R=permanent]"])
 
         self.config.save()
 
@@ -1184,13 +1217,13 @@ class MultipleVhostsTest(util.ApacheTest):
         not_commented_cond1 = ("RewriteCond "
                 "%{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f")
         not_commented_rewrite_rule = ("RewriteRule "
-            "^(.*)$ b://u%{REQUEST_URI} [P,QSA,L]")
+            "^(.*)$ b://u%{REQUEST_URI} [P,NE,L]")
 
         commented_cond1 = "# RewriteCond %{HTTPS} !=on"
         commented_cond2 = "# RewriteCond %{HTTPS} !^$"
         commented_rewrite_rule = ("# RewriteRule ^ "
                                   "https://%{SERVER_NAME}%{REQUEST_URI} "
-                                  "[L,QSA,R=permanent]")
+                                  "[L,NE,R=permanent]")
 
         self.assertTrue(not_commented_cond1 in conf_line_set)
         self.assertTrue(not_commented_rewrite_rule in conf_line_set)
@@ -1208,13 +1241,13 @@ class MultipleVhostsTest(util.ApacheTest):
         achall1 = achallenges.KeyAuthorizationAnnotatedChallenge(
             challb=acme_util.chall_to_challb(
                 challenges.TLSSNI01(
-                    token="jIq_Xy1mXGN37tb4L6Xj_es58fW571ZNyXekdZzhh7Q"),
+                    token=b"jIq_Xy1mXGN37tb4L6Xj_es58fW571ZNyXekdZzhh7Q"),
                 "pending"),
             domain="encryption-example.demo", account_key=account_key)
         achall2 = achallenges.KeyAuthorizationAnnotatedChallenge(
             challb=acme_util.chall_to_challb(
                 challenges.TLSSNI01(
-                    token="uqnaPzxtrndteOqtrXb0Asl5gOJfWAnnx6QJyvcmlDU"),
+                    token=b"uqnaPzxtrndteOqtrXb0Asl5gOJfWAnnx6QJyvcmlDU"),
                 "pending"),
             domain="certbot.demo", account_key=account_key)
 

certbot-apache/certbot_apache/tests/display_ops_test.py

@@ -38,7 +38,7 @@ class SelectVhostTest(unittest.TestCase):
         try:
             self._call(self.vhosts)
         except errors.MissingCommandlineFlag as e:
-            self.assertTrue("vhost ambiguity" in e.message)
+            self.assertTrue("vhost ambiguity" in str(e))
 
     @certbot_util.patch_get_utility()
     def test_more_info_cancel(self, mock_util):

certbot-apache/certbot_apache/tests/tls_sni_01_test.py

@@ -105,7 +105,7 @@ class TlsSniPerformTest(util.ApacheTest):
         for achall in self.achalls:
             self.sni.add_chall(achall)
             z_domain = achall.response(self.auth_key).z_domain
-            z_domains.append(set([z_domain]))
+            z_domains.append(set([z_domain.decode('ascii')]))
 
         self.sni._mod_config()  # pylint: disable=protected-access
         self.sni.configurator.save()

certbot-apache/certbot_apache/tls_sni_01.py

@@ -184,7 +184,7 @@ class ApacheTlsSni01(common.TLSSNI01):
         # https://docs.python.org/2.7/reference/lexical_analysis.html
         return self.VHOST_TEMPLATE.format(
             vhost=ips,
-            server_name=achall.response(achall.account_key).z_domain,
+            server_name=achall.response(achall.account_key).z_domain.decode('ascii'),
             ssl_options_conf_path=self.configurator.mod_ssl_conf,
             cert_path=self.get_cert_path(achall),
             key_path=self.get_key_path(achall),

certbot-apache/setup.py

@@ -4,13 +4,14 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.12.0.dev0'
+version = '0.14.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [
     'acme=={0}'.format(version),
     'certbot=={0}'.format(version),
-    'python-augeas',
+    'mock',
+    'python-augeas<=0.5.0',
     # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
     # will tolerate; see #2599:
     'setuptools>=1.0',
@@ -18,11 +19,6 @@ install_requires = [
     'zope.interface',
 ]
 
-if sys.version_info < (2, 7):
-    install_requires.append('mock<1.1.0')
-else:
-    install_requires.append('mock')
-
 docs_extras = [
     'Sphinx>=1.0',  # autodoc_member_order = 'bysource', autodoc_default_flags
     'sphinx_rtd_theme',

certbot-auto

@@ -23,7 +23,7 @@ if [ -z "$VENV_PATH" ]; then
   VENV_PATH="$XDG_DATA_HOME/$VENV_NAME"
 fi
 VENV_BIN="$VENV_PATH/bin"
-LE_AUTO_VERSION="0.11.1"
+LE_AUTO_VERSION="0.13.0"
 BASENAME=$(basename $0)
 USAGE="Usage: $BASENAME [OPTIONS]
 A self-updating wrapper script for the Certbot ACME client. When run, updates
@@ -36,6 +36,7 @@ Help for certbot itself cannot be provided until it is installed.
   --debug                                   attempt experimental installation
   -h, --help                                print this help
   -n, --non-interactive, --noninteractive   run without asking for user input
+  --no-bootstrap                            do not install OS dependencies
   --no-self-upgrade                         do not download updates
   --os-packages-only                        install OS dependencies and exit
   -v, --verbose                             provide more output
@@ -54,6 +55,8 @@ for arg in "$@" ; do
       # Do not upgrade this script (also prevents client upgrades, because each
       # copy of the script pins a hash of the python client)
       NO_SELF_UPGRADE=1;;
+    --no-bootstrap)
+      NO_BOOTSTRAP=1;;
     --help)
       HELP=1;;
     --noninteractive|--non-interactive)
@@ -160,17 +163,24 @@ else
   fi
 fi
 
+BootstrapMessage() {
+  # Arguments: Platform name
+  echo "Bootstrapping dependencies for $1... (you can skip this with --no-bootstrap)"
+}
+
 ExperimentalBootstrap() {
   # Arguments: Platform name, bootstrap function name
   if [ "$DEBUG" = 1 ]; then
     if [ "$2" != "" ]; then
-      echo "Bootstrapping dependencies via $1..."
+      BootstrapMessage $1
       $2
     fi
   else
     echo "FATAL: $1 support is very experimental at present..."
     echo "if you would like to work on improving it, please ensure you have backups"
     echo "and then run this script again with the --debug flag!"
+    echo "Alternatively, you can install OS dependencies yourself and run this script"
+    echo "again with --no-bootstrap."
     exit 1
   fi
 }
@@ -576,21 +586,23 @@ BootstrapMageiaCommon() {
 
 # Install required OS packages:
 Bootstrap() {
-  if [ -f /etc/debian_version ]; then
-    echo "Bootstrapping dependencies for Debian-based OSes..."
+  if [ "$NO_BOOTSTRAP" = 1 ]; then
+      return
+  elif [ -f /etc/debian_version ]; then
+    BootstrapMessage "Debian-based OSes"
     BootstrapDebCommon
-  elif [ -f /etc/mageia-release ] ; then
+  elif [ -f /etc/mageia-release ]; then
     # Mageia has both /etc/mageia-release and /etc/redhat-release
     ExperimentalBootstrap "Mageia" BootstrapMageiaCommon
   elif [ -f /etc/redhat-release ]; then
-    echo "Bootstrapping dependencies for RedHat-based OSes..."
+    BootstrapMessage "RedHat-based OSes"
     BootstrapRpmCommon
   elif [ -f /etc/os-release ] && `grep -q openSUSE /etc/os-release` ; then
-    echo "Bootstrapping dependencies for openSUSE-based OSes..."
+    BootstrapMessage "openSUSE-based OSes"
     BootstrapSuseCommon
   elif [ -f /etc/arch-release ]; then
     if [ "$DEBUG" = 1 ]; then
-      echo "Bootstrapping dependencies for Archlinux..."
+      BootstrapMessage "Archlinux"
       BootstrapArchCommon
     else
       echo "Please use pacman to install letsencrypt packages:"
@@ -615,7 +627,7 @@ Bootstrap() {
   else
     echo "Sorry, I don't know how to bootstrap Certbot on your operating system!"
     echo
-    echo "You will need to bootstrap, configure virtualenv, and run pip install manually."
+    echo "You will need to install OS dependencies, configure virtualenv, and run pip install manually."
     echo "Please see https://letsencrypt.readthedocs.org/en/latest/contributing.html#prerequisites"
     echo "for more info."
     exit 1
@@ -833,18 +845,18 @@ letsencrypt==0.7.0 \
 
 # THE LINES BELOW ARE EDITED BY THE RELEASE SCRIPT; ADD ALL DEPENDENCIES ABOVE.
 
-acme==0.11.1 \
-    --hash=sha256:9f4efac6dc4477a3baa7eb2392d4f7583f974e4ad336439aa1961ef805622a77 \
-    --hash=sha256:db35258edfc13dfe5839215898fe2d5d3caafc9a084f631a032f3fdf712c694e
-certbot==0.11.1 \
-    --hash=sha256:ba80552df0f390dbc5fcd14b4ea4b1499ea866f5f78c8c1a375abc25101dedf1 \
-    --hash=sha256:6c1724486d500c5163c9313d6a14af5af9f4515f79553627303a6b86df2c3af2
-certbot-apache==0.11.1 \
-    --hash=sha256:70132d9013509011b9edeba64fc208961f50ef78457f58d3b80a61094102efcd \
-    --hash=sha256:efe2224b531595edee366423c115e2874a3c9011890321d3ccda0367efc776c0
-certbot-nginx==0.11.1 \
-    --hash=sha256:1895eea1de92ab3dfd762998a4be7868ec3ec4d42cce7772995e4e9b2e488e6a \
-    --hash=sha256:e5e5ffe8930ba10139bb61c2a05a30e84d9a69a7d8fc6a7b391f707eae8bfce5
+acme==0.13.0 \
+    --hash=sha256:103ce8bed43aad1a9655ed815df09bbeab86ee16cc82137b44d9dac68faa394f \
+    --hash=sha256:7489b3e20d02da0a389aedb82408ffb6b76294e41d833db85591b9f779539815
+certbot==0.13.0 \
+    --hash=sha256:65d0d9d158972aff7746d4ef80a20465a14c54ae8bcb879216970c2a1b34503c \
+    --hash=sha256:f63ad7747edaca2fb7d60c28882e44d2f48ff1cca9b9c7c251ad47e2189c00f3
+certbot-apache==0.13.0 \
+    --hash=sha256:22f7c1dc93439384c0874960081d66957910c6dc737a9facbd9fcbc46c545874 \
+    --hash=sha256:b43b04b53005e7218a09a0ba4d97581fab369e929472fa49fb55d29d0ab54589
+certbot-nginx==0.13.0 \
+    --hash=sha256:9d0ab4eeb98b0ebad70ba116b32268342ad343d82d64990a652ff8072959b044 \
+    --hash=sha256:f026a8faee8397a22c5d4a7623a6ef7c7e780ed63a3bdf9940f43f7823aa2a72
 
 UNLIKELY_EOF
     # -------------------------------------------------------------------------
@@ -1093,6 +1105,9 @@ else
 On failure, return non-zero.
 
 """
+
+from __future__ import print_function
+
 from distutils.version import LooseVersion
 from json import loads
 from os import devnull, environ
@@ -1194,12 +1209,12 @@ def main():
     flag = argv[1]
     try:
         if flag == '--latest-version':
-            print latest_stable_version(get)
+            print(latest_stable_version(get))
         elif flag == '--le-auto-script':
             tag = argv[2]
             verified_new_le_auto(get, tag, dirname(argv[0]))
     except ExpectedError as exc:
-        print exc.args[0], exc.args[1]
+        print(exc.args[0], exc.args[1])
         return 1
     else:
         return 0

certbot-compatibility-test/certbot_compatibility_test/interfaces.py

@@ -20,20 +20,20 @@ class IPluginProxy(zope.interface.Interface):
     def __init__(args):
         """Initializes the plugin with the given command line args"""
 
-    def cleanup_from_tests():
+    def cleanup_from_tests():  # type: ignore
         """Performs any necessary cleanup from running plugin tests.
 
         This is guaranteed to be called before the program exits.
 
         """
 
-    def has_more_configs():
+    def has_more_configs():  # type: ignore
         """Returns True if there are more configs to test"""
 
-    def load_config():
+    def load_config():  # type: ignore
         """Loads the next config and returns its name"""
 
-    def get_testable_domain_names():
+    def get_testable_domain_names():  # type: ignore
         """Returns the domain names that can be used in testing"""
 
 
@@ -44,7 +44,7 @@ class IAuthenticatorProxy(IPluginProxy, certbot.interfaces.IAuthenticator):
 class IInstallerProxy(IPluginProxy, certbot.interfaces.IInstaller):
     """Wraps a Certbot installer"""
 
-    def get_all_names_answer():
+    def get_all_names_answer():  # type: ignore
         """Returns all names that should be found by the installer"""
 
 

certbot-compatibility-test/nginx/nginx-roundtrip-testdata/simplepythonfcgi/weird-spacing.conf

@@ -1,16 +1,16 @@
  # static files
  location ~ ^/(images|javascript|js|css|flash|media|static)/ {
-   root ${PROJECTBASE}/${PROJECTNAME}/static;
+   root "${PROJECTBASE}/${PROJECTNAME}/static";
  }
 
  location = /favicon.ico {
-   root ${PROJECTBASE}/${PROJECTNAME}/static/images;
+   root "${PROJECTBASE}/${PROJECTNAME}/static/images";
  }
 
  # pass all requests to FastCGI TG server listening on ${HOST}:${PORT}
  #
  location / {
-   fastcgi_pass ${HOST}:${PORT};
+   fastcgi_pass "${HOST}:${PORT}";
    fastcgi_index index;
    fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
    include conf/fastcgi_params;

certbot-compatibility-test/setup.py

@@ -4,21 +4,17 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.12.0.dev0'
+version = '0.14.0.dev0'
 
 install_requires = [
     'certbot',
     'certbot-apache',
+    'mock',
     'six',
     'requests',
     'zope.interface',
 ]
 
-if sys.version_info < (2, 7):
-    install_requires.append('mock<1.1.0')
-else:
-    install_requires.append('mock')
-
 if sys.version_info < (2, 7, 9):
     # For secure SSL connexion with Python 2.7 (InsecurePlatformWarning)
     install_requires.append('ndg-httpsclient')

certbot-nginx/certbot_nginx/configurator.py

@@ -5,9 +5,11 @@ import re
 import shutil
 import socket
 import subprocess
+import tempfile
 import time
 
 import OpenSSL
+import six
 import zope.interface
 
 from acme import challenges
@@ -30,16 +32,16 @@ from certbot_nginx import parser
 logger = logging.getLogger(__name__)
 
 REDIRECT_BLOCK = [[
-    ['\n    ', 'if', ' ', '($scheme != "https") '],
-    [['\n        ', 'return', ' ', '301 https://$host$request_uri'],
+    ['\n    ', 'if', ' ', '($scheme', ' ', '!=', ' ', '"https") '],
+    [['\n        ', 'return', ' ', '301', ' ', 'https://$host$request_uri'],
      '\n    ']
 ], ['\n']]
 
 TEST_REDIRECT_BLOCK = [
     [
-        ['if', '($scheme != "https")'],
+        ['if', '($scheme', '!=', '"https")'],
         [
-            ['return', '301 https://$host$request_uri']
+            ['return', '301', 'https://$host$request_uri']
         ]
     ],
     ['#', ' managed by Certbot']
@@ -262,7 +264,7 @@ class NginxConfigurator(common.Plugin):
         """
         if not matches:
             return None
-        elif matches[0]['rank'] in xrange(2, 6):
+        elif matches[0]['rank'] in six.moves.range(2, 6):
             # Wildcard match - need to find the longest one
             rank = matches[0]['rank']
             wildcards = [x for x in matches if x['rank'] == rank]
@@ -628,10 +630,11 @@ class NginxConfigurator(common.Plugin):
             proc = subprocess.Popen(
                 [self.conf('ctl'), "-c", self.nginx_conf, "-V"],
                 stdout=subprocess.PIPE,
-                stderr=subprocess.PIPE)
+                stderr=subprocess.PIPE,
+                universal_newlines=True)
             text = proc.communicate()[1]  # nginx prints output to stderr
         except (OSError, ValueError) as error:
-            logging.debug(error, exc_info=True)
+            logger.debug(error, exc_info=True)
             raise errors.PluginError(
                 "Unable to run %s -V" % self.conf('ctl'))
 
@@ -819,7 +822,7 @@ class NginxConfigurator(common.Plugin):
             self.restart()
 
 
-def nginx_restart(nginx_ctl, nginx_conf="/etc/nginx.conf"):
+def nginx_restart(nginx_ctl, nginx_conf):
     """Restarts the Nginx Server.
 
     .. todo:: Nginx restart is fatal if the configuration references
@@ -830,22 +833,22 @@ def nginx_restart(nginx_ctl, nginx_conf="/etc/nginx.conf"):
 
     """
     try:
-        proc = subprocess.Popen([nginx_ctl, "-c", nginx_conf, "-s", "reload"],
-                                stdout=subprocess.PIPE,
-                                stderr=subprocess.PIPE)
-        stdout, stderr = proc.communicate()
+        proc = subprocess.Popen([nginx_ctl, "-c", nginx_conf, "-s", "reload"])
+        proc.communicate()
 
         if proc.returncode != 0:
             # Maybe Nginx isn't running
-            nginx_proc = subprocess.Popen([nginx_ctl, "-c", nginx_conf],
-                                          stdout=subprocess.PIPE,
-                                          stderr=subprocess.PIPE)
-            stdout, stderr = nginx_proc.communicate()
-
-            if nginx_proc.returncode != 0:
-                # Enter recovery routine...
-                raise errors.MisconfigurationError(
-                    "nginx restart failed:\n%s\n%s" % (stdout, stderr))
+            # Write to temporary files instead of piping because of communication issues on Arch
+            # https://github.com/certbot/certbot/issues/4324
+            with tempfile.TemporaryFile() as out:
+                with tempfile.TemporaryFile() as err:
+                    nginx_proc = subprocess.Popen([nginx_ctl, "-c", nginx_conf],
+                        stdout=out, stderr=err)
+                    nginx_proc.communicate()
+                    if nginx_proc.returncode != 0:
+                        # Enter recovery routine...
+                        raise errors.MisconfigurationError(
+                            "nginx restart failed:\n%s\n%s" % (out.read(), err.read()))
 
     except (OSError, ValueError):
         raise errors.MisconfigurationError("nginx restart failed")

certbot-nginx/certbot_nginx/nginxparser.py

@@ -2,11 +2,9 @@
 # Forked from https://github.com/fatiherikli/nginxparser (MIT Licensed)
 import copy
 import logging
-import string
 
 from pyparsing import (
-    Literal, White, Word, alphanums, CharsNotIn, Combine, Forward, Group,
-    Optional, OneOrMore, Regex, ZeroOrMore)
+    Literal, White, Forward, Group, Optional, OneOrMore, QuotedString, Regex, ZeroOrMore, Combine)
 from pyparsing import stringEnd
 from pyparsing import restOfLine
 
@@ -14,73 +12,42 @@ logger = logging.getLogger(__name__)
 
 class RawNginxParser(object):
     # pylint: disable=expression-not-assigned
+    # pylint: disable=pointless-statement
     """A class that parses nginx configuration with pyparsing."""
 
     # constants
-    space = Optional(White())
-    nonspace = Regex(r"\S+")
+    space = Optional(White()).leaveWhitespace()
+    required_space = White().leaveWhitespace()
+
     left_bracket = Literal("{").suppress()
-    right_bracket = space.leaveWhitespace() + Literal("}").suppress()
+    right_bracket = space + Literal("}").suppress()
     semicolon = Literal(";").suppress()
-    key = Word(alphanums + "_/+-.")
-    dollar_var = Combine(Literal('$') + Regex(r"[^\{\};,\s]+"))
-    condition = Regex(r"\(.+\)")
-    # Matches anything that is not a special character, and ${SHELL_VARS}, AND
-    # any chars in single or double quotes
-    # All of these COULD be upgraded to something like
-    # https://stackoverflow.com/a/16130746
-    dquoted = Regex(r'(\".*\")')
-    squoted = Regex(r"(\'.*\')")
-    nonspecial = Regex(r"[^\{\};,]")
-    varsub = Regex(r"(\$\{\w+\})")
-    # nonspecial nibbles one character at a time, but the other objects take
-    # precedence.  We use ZeroOrMore to allow entries like "break ;" to be
-    # parsed as assignments
-    value = Combine(ZeroOrMore(dquoted | squoted | varsub | nonspecial))
+    dquoted = QuotedString('"', multiline=True, unquoteResults=False, escChar='\\')
+    squoted = QuotedString("'", multiline=True, unquoteResults=False, escChar='\\')
+    quoted = dquoted | squoted
+    head_tokenchars = Regex(r"[^{};\s'\"]") # if (last_space)
+    tail_tokenchars = Regex(r"(\$\{)|[^{;\s]") # else
+    tokenchars = Combine(head_tokenchars + ZeroOrMore(tail_tokenchars))
+    paren_quote_extend = Combine(quoted + Literal(')') + ZeroOrMore(tail_tokenchars))
+    # note: ')' allows extension, but then we fall into else, not last_space.
 
-    location = CharsNotIn("{};," + string.whitespace)
-    # modifier for location uri [ = | ~ | ~* | ^~ ]
-    modifier = Literal("=") | Literal("~*") | Literal("~") | Literal("^~")
+    token = paren_quote_extend | tokenchars | quoted
+
+    whitespace_token_group = space + token + ZeroOrMore(required_space + token) + space
+    assignment = whitespace_token_group + semicolon
 
-    # rules
     comment = space + Literal('#') + restOfLine
 
-    assignment = space + key + Optional(space + value, default=None) + semicolon
-    location_statement = space + Optional(modifier) + Optional(space + location + space)
-    if_statement = space + Literal("if") + space + condition + space
-    charset_map_statement = space + Literal("charset_map") + space + value + space + value
-
-    map_statement = space + Literal("map") + space + nonspace + space + dollar_var + space
-    # This is NOT an accurate way to parse nginx map entries; it's almost
-    # certainly too permissive and may be wrong in other ways, but it should
-    # preserve things correctly in mmmmost or all cases.
-    #
-    #    - I can neither prove nor disprove that it is correct wrt all escaped
-    #      semicolon situations
-    # Addresses https://github.com/fatiherikli/nginxparser/issues/19
-    map_pattern = Regex(r'".*"') | Regex(r"'.*'") | nonspace
-    map_entry = space + map_pattern + space + value + space + semicolon
-    map_block = Group(
-        Group(map_statement).leaveWhitespace() +
-        left_bracket +
-        Group(ZeroOrMore(Group(comment | map_entry)) + space).leaveWhitespace() +
-        right_bracket)
-
     block = Forward()
 
-    # key could for instance be "server" or "http", or "location" (in which case
-    # location_statement needs to have a non-empty location)
+    # order matters! see issue 518, and also http { # server { \n}
+    contents = Group(comment) | Group(block) | Group(assignment)
 
-    block_begin = (Group(space + key + location_statement) ^
-                   Group(if_statement) ^
-                   Group(charset_map_statement)).leaveWhitespace()
+    block_begin = Group(whitespace_token_group)
+    block_innards = Group(ZeroOrMore(contents) + space).leaveWhitespace()
+    block << block_begin + left_bracket + block_innards + right_bracket
 
-    block_innards = Group(ZeroOrMore(Group(comment | assignment) | block | map_block)
-                          + space).leaveWhitespace()
-
-    block << Group(block_begin + left_bracket + block_innards + right_bracket)
-
-    script = OneOrMore(Group(comment | assignment) ^ block ^ map_block) + space + stringEnd
+    script = OneOrMore(contents) + space + stringEnd
     script.parseWithTabs().leaveWhitespace()
 
     def __init__(self, source):
@@ -107,30 +74,23 @@ class RawNginxDumper(object):
             if isinstance(b0, str):
                 yield b0
                 continue
-            b = copy.deepcopy(b0)
-            if spacey(b[0]):
-                yield b.pop(0) # indentation
-                if not b:
+            item = copy.deepcopy(b0)
+            if spacey(item[0]):
+                yield item.pop(0) # indentation
+                if not item:
                     continue
-            key, values = b.pop(0), b.pop(0)
 
-            if isinstance(key, list):
-                yield "".join(key) + '{'
-                for parameter in values:
+            if isinstance(item[0], list): # block
+                yield "".join(item.pop(0)) + '{'
+                for parameter in item.pop(0):
                     for line in self.__iter__([parameter]): # negate "for b0 in blocks"
                         yield line
                 yield '}'
-            else:
-                if isinstance(key, str) and key.strip() == '#':  # comment
-                    yield key + values
-                else:                                            # assignment
-                    gap = ""
-                    # Sometimes the parser has stuck some gap whitespace in here;
-                    # if so rotate it into gap
-                    if values and spacey(values):
-                        gap = values
-                        values = b.pop(0)
-                    yield key + gap + values + ';'
+            else: # not a block - list of strings
+                semicolon = ";"
+                if isinstance(item[0], str) and item[0].strip() == '#': # comment
+                    semicolon = ""
+                yield "".join(item) + semicolon
 
     def __str__(self):
         """Return the parsed block as a string."""

certbot-nginx/certbot_nginx/obj.py

@@ -1,6 +1,8 @@
 """Module contains classes used by the Nginx Configurator."""
 import re
 
+import six
+
 from certbot.plugins import common
 
 REDIRECT_DIRECTIVES = ['return', 'rewrite']
@@ -97,6 +99,11 @@ class Addr(common.Addr):
     def __repr__(self):
         return "Addr(" + self.__str__() + ")"
 
+    def __hash__(self):
+        # Python 3 requires explicit overridden for __hash__
+        # See certbot-apache/certbot_apache/obj.py for more information
+        return super(Addr, self).__hash__()
+
     def super_eq(self, other):
         """Check ip/port equality, with IPv6 support.
         """
@@ -147,13 +154,15 @@ class VirtualHost(object):  # pylint: disable=too-few-public-methods
         self.path = path
 
     def __str__(self):
-        addr_str = ", ".join(str(addr) for addr in self.addrs)
+        addr_str = ", ".join(str(addr) for addr in sorted(self.addrs, key=str))
+        # names might be a set, and it has different representations in Python
+        # 2 and 3. Force it to be a list here for consistent outputs
         return ("file: %s\n"
                 "addrs: %s\n"
                 "names: %s\n"
                 "ssl: %s\n"
                 "enabled: %s" % (self.filep, addr_str,
-                                 self.names, self.ssl, self.enabled))
+                                 list(self.names), self.ssl, self.enabled))
 
     def __repr__(self):
         return "VirtualHost(" + self.__str__().replace("\n", ", ") + ")\n"
@@ -161,7 +170,7 @@ class VirtualHost(object):  # pylint: disable=too-few-public-methods
     def __eq__(self, other):
         if isinstance(other, self.__class__):
             return (self.filep == other.filep and
-                    list(self.addrs) == list(other.addrs) and
+                    sorted(self.addrs, key=str) == sorted(other.addrs, key=str) and
                     self.names == other.names and
                     self.ssl == other.ssl and
                     self.enabled == other.enabled and
@@ -181,7 +190,7 @@ class VirtualHost(object):  # pylint: disable=too-few-public-methods
     def contains_list(self, test):
         """Determine if raw server block contains test list at top level
         """
-        for i in xrange(0, len(self.raw) - len(test)):
+        for i in six.moves.range(0, len(self.raw) - len(test)):
             if self.raw[i:i + len(test)] == test:
                 return True
         return False

certbot-nginx/certbot_nginx/options-ssl-nginx.conf

@@ -4,4 +4,4 @@ ssl_session_timeout 1440m;
 ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
 ssl_prefer_server_ciphers on;
 
-ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES128-SHA ECDHE-ECDSA-AES256-SHA ECDHE-ECDSA-AES128-SHA256 ECDHE-ECDSA-AES256-SHA384 ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES128-SHA ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES256-SHA384 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA DHE-RSA-AES128-SHA256 DHE-RSA-AES256-SHA256 EDH-RSA-DES-CBC3-SHA";
+ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS";

certbot-nginx/certbot_nginx/parser.py

@@ -205,8 +205,8 @@ class NginxParser(object):
                     trees.append(parsed)
             except IOError:
                 logger.warning("Could not open file: %s", item)
-            except pyparsing.ParseException:
-                logger.debug("Could not parse file: %s", item)
+            except pyparsing.ParseException as err:
+                logger.debug("Could not parse file: %s due to %s", item, err)
         return trees
 
     def _parse_ssl_options(self, ssl_options):
@@ -216,8 +216,8 @@ class NginxParser(object):
                     return nginxparser.load(_file).spaced
             except IOError:
                 logger.warn("Missing NGINX TLS options file: %s", ssl_options)
-            except pyparsing.ParseBaseException:
-                logger.debug("Could not parse file: %s", ssl_options)
+            except pyparsing.ParseBaseException as err:
+                logger.debug("Could not parse file: %s due to %s", ssl_options, err)
         return []
 
     def _set_locations(self, ssl_options):
@@ -298,9 +298,9 @@ class NginxParser(object):
         """
         server = vhost.raw
         for directive in server:
-            if not directive or len(directive) < 2:
+            if not directive:
                 continue
-            elif directive[0] == 'ssl' and directive[1] == 'on':
+            elif _is_ssl_on_directive(directive):
                 return True
 
         return False
@@ -342,7 +342,7 @@ class NginxParser(object):
             vhost.names = parsed_server['names']
             vhost.raw = new_server
         except errors.MisconfigurationError as err:
-            raise errors.MisconfigurationError("Problem in %s: %s" % (filename, err.message))
+            raise errors.MisconfigurationError("Problem in %s: %s" % (filename, str(err)))
 
 
 def _do_for_subarray(entry, condition, func, path=None):
@@ -468,17 +468,17 @@ def _is_include_directive(entry):
             len(entry) == 2 and entry[0] == 'include' and
             isinstance(entry[1], str))
 
+def _is_ssl_on_directive(entry):
+    """Checks if an nginx parsed entry is an 'ssl on' directive.
 
-def _get_servernames(names):
-    """Turns a server_name string into a list of server names
-
-    :param str names: server names
-    :rtype: list
+    :param list entry: the parsed entry
+    :returns: Whether it's an 'ssl on' directive
+    :rtype: bool
 
     """
-    whitespace_re = re.compile(r'\s+')
-    names = re.sub(whitespace_re, ' ', names)
-    return names.split(' ')
+    return (isinstance(entry, list) and
+            len(entry) == 2 and entry[0] == 'ssl' and
+            entry[1] == 'on')
 
 def _add_directives(block, directives, replace):
     """Adds or replaces directives in a config block.
@@ -550,12 +550,11 @@ def _add_directive(block, directive, replace):
         # and there is already a copy of that directive with a different value
         # in the config file.
         directive_name = directive[0]
-        directive_value = directive[1]
         if location is None or (isinstance(directive_name, str) and
                                 directive_name in REPEATABLE_DIRECTIVES):
             block.append(directive)
             _comment_directive(block, len(block) - 1)
-        elif block[location][1] != directive_value:
+        elif block[location] != directive:
             raise errors.MisconfigurationError(
                 'tried to insert directive "{0}" but found '
                 'conflicting "{1}".'.format(directive, block[location]))
@@ -585,14 +584,14 @@ def _parse_server_raw(server):
         if not directive:
             continue
         if directive[0] == 'listen':
-            addr = obj.Addr.fromstring(directive[1])
-            parsed_server['addrs'].add(addr)
-            if addr.ssl:
-                parsed_server['ssl'] = True
+            addr = obj.Addr.fromstring(" ".join(directive[1:]))
+            if addr:
+                parsed_server['addrs'].add(addr)
+                if addr.ssl:
+                    parsed_server['ssl'] = True
         elif directive[0] == 'server_name':
-            parsed_server['names'].update(
-                _get_servernames(directive[1]))
-        elif directive[0] == 'ssl' and directive[1] == 'on':
+            parsed_server['names'].update(directive[1:])
+        elif _is_ssl_on_directive(directive):
             parsed_server['ssl'] = True
             apply_ssl_to_all_addrs = True
 

certbot-nginx/certbot_nginx/tests/configurator_test.py

@@ -27,12 +27,13 @@ class NginxConfiguratorTest(util.NginxTest):
         super(NginxConfiguratorTest, self).setUp()
 
         self.config = util.get_nginx_configurator(
-            self.config_path, self.config_dir, self.work_dir)
+            self.config_path, self.config_dir, self.work_dir, self.logs_dir)
 
     def tearDown(self):
         shutil.rmtree(self.temp_dir)
         shutil.rmtree(self.config_dir)
         shutil.rmtree(self.work_dir)
+        shutil.rmtree(self.logs_dir)
 
     @mock.patch("certbot_nginx.configurator.util.exe_exists")
     def test_prepare_no_install(self, mock_exe_exists):
@@ -93,7 +94,7 @@ class NginxConfiguratorTest(util.NginxTest):
                                      None, [0])
         self.config.parser.add_server_directives(
             mock_vhost,
-            [['listen', ' ', '5001 ssl']],
+            [['listen', ' ', '5001', ' ', 'ssl']],
             replace=False)
         self.config.save()
 
@@ -104,7 +105,7 @@ class NginxConfiguratorTest(util.NginxTest):
                             ['listen', '127.0.0.1'],
                             ['server_name', '.example.com'],
                             ['server_name', 'example.*'],
-                            ['listen', '5001 ssl'],
+                            ['listen', '5001', 'ssl'],
                             ['#', parser.COMMENT]]]],
                          parsed[0])
 
@@ -204,13 +205,13 @@ class NginxConfiguratorTest(util.NginxTest):
                             ['server_name', '.example.com'],
                             ['server_name', 'example.*'],
 
-                            ['listen', '5001 ssl'],
+                            ['listen', '5001', 'ssl'],
                             ['ssl_certificate', 'example/fullchain.pem'],
                             ['ssl_certificate_key', 'example/key.pem']] +
                             util.filter_comments(self.config.parser.loc["ssl_options"])
                             ]],
                          parsed_example_conf)
-        self.assertEqual([['server_name', 'somename  alias  another.alias']],
+        self.assertEqual([['server_name', 'somename', 'alias', 'another.alias']],
                          parsed_server_conf)
         self.assertTrue(util.contains_at_depth(
             parsed_nginx_conf,
@@ -221,8 +222,8 @@ class NginxConfiguratorTest(util.NginxTest):
               ['include', 'server.conf'],
               [['location', '/'],
                [['root', 'html'],
-                ['index', 'index.html index.htm']]],
-              ['listen', '5001 ssl'],
+                ['index', 'index.html', 'index.htm']]],
+              ['listen', '5001', 'ssl'],
               ['ssl_certificate', '/etc/nginx/fullchain.pem'],
               ['ssl_certificate_key', '/etc/nginx/key.pem']] +
              util.filter_comments(self.config.parser.loc["ssl_options"])
@@ -246,7 +247,7 @@ class NginxConfiguratorTest(util.NginxTest):
                            ['server_name', 'summer.com'],
 
                            ['listen', '80'],
-                           ['listen', '5001 ssl'],
+                           ['listen', '5001', 'ssl'],
                            ['ssl_certificate', 'summer/fullchain.pem'],
                            ['ssl_certificate_key', 'summer/key.pem']] +
                            util.filter_comments(self.config.parser.loc["ssl_options"])
@@ -261,13 +262,13 @@ class NginxConfiguratorTest(util.NginxTest):
         # Note: As more challenges are offered this will have to be expanded
         achall1 = achallenges.KeyAuthorizationAnnotatedChallenge(
             challb=messages.ChallengeBody(
-                chall=challenges.TLSSNI01(token="kNdwjwOeX0I_A8DXt9Msmg"),
+                chall=challenges.TLSSNI01(token=b"kNdwjwOeX0I_A8DXt9Msmg"),
                 uri="https://ca.org/chall0_uri",
                 status=messages.Status("pending"),
             ), domain="localhost", account_key=self.rsa512jwk)
         achall2 = achallenges.KeyAuthorizationAnnotatedChallenge(
             challb=messages.ChallengeBody(
-                chall=challenges.TLSSNI01(token="m8TdO1qik4JVFtgPPurJmg"),
+                chall=challenges.TLSSNI01(token=b"m8TdO1qik4JVFtgPPurJmg"),
                 uri="https://ca.org/chall1_uri",
                 status=messages.Status("pending"),
             ), domain="example.com", account_key=self.rsa512jwk)
@@ -407,8 +408,8 @@ class NginxConfiguratorTest(util.NginxTest):
         # Test that we successfully add a redirect when there is
         # a listen directive
         expected = [
-            ['if', '($scheme != "https") '],
-            [['return', '301 https://$host$request_uri']]
+            ['if', '($scheme', '!=', '"https") '],
+            [['return', '301', 'https://$host$request_uri']]
         ]
 
         example_conf = self.config.parser.abs_path('sites-enabled/example.com')

certbot-nginx/certbot_nginx/tests/nginxparser_test.py

@@ -25,15 +25,15 @@ class TestRawNginxParser(unittest.TestCase):
 
     def test_blocks(self):
         parsed = RawNginxParser.block.parseString('foo {}').asList()
-        self.assertEqual(parsed, [[['foo', ' '], []]])
+        self.assertEqual(parsed, [['foo', ' '], []])
         parsed = RawNginxParser.block.parseString('location /foo{}').asList()
-        self.assertEqual(parsed, [[['location', ' ', '/foo'], []]])
+        self.assertEqual(parsed, [['location', ' ', '/foo'], []])
         parsed = RawNginxParser.block.parseString('foo { bar foo ; }').asList()
-        self.assertEqual(parsed, [[['foo', ' '], [[' ', 'bar', ' ', 'foo '], ' ']]])
+        self.assertEqual(parsed, [['foo', ' '], [[' ', 'bar', ' ', 'foo', ' '], ' ']])
 
     def test_nested_blocks(self):
         parsed = RawNginxParser.block.parseString('foo { bar {} }').asList()
-        block, content = FIRST(parsed)
+        block, content = parsed
         self.assertEqual(FIRST(content), [[' ', 'bar', ' '], []])
         self.assertEqual(FIRST(block), 'foo')
 
@@ -72,8 +72,8 @@ class TestRawNginxParser(unittest.TestCase):
             [['user', 'www-data'],
              [['http'],
               [[['server'], [
-                  ['listen', '*:80 default_server ssl'],
-                  ['server_name', '*.www.foo.com *.www.example.com'],
+                  ['listen', '*:80', 'default_server', 'ssl'],
+                  ['server_name', '*.www.foo.com', '*.www.example.com'],
                   ['root', '/home/ubuntu/sites/foo/'],
                   [['location', '/status'], [
                       [['types'], [['image/jpeg', 'jpg']]],
@@ -97,17 +97,35 @@ class TestRawNginxParser(unittest.TestCase):
              [['server'],
               [['server_name', 'with.if'],
                [['location', '~', '^/services/.+$'],
-                [[['if', '($request_filename ~* \\.(ttf|woff)$)'],
-                  [['add_header', 'Access-Control-Allow-Origin "*"']]]]]]],
+                [[['if', '($request_filename', '~*', '\\.(ttf|woff)$)'],
+                  [['add_header', 'Access-Control-Allow-Origin', '"*"']]]]]]],
              [['server'],
               [['server_name', 'with.complicated.headers'],
                [['location', '~*', '\\.(?:gif|jpe?g|png)$'],
-                [['add_header', 'Pragma public'],
+                [['add_header', 'Pragma', 'public'],
                  ['add_header',
-                  'Cache-Control  \'public, must-revalidate, proxy-revalidate\''
-                  ' "test,;{}" foo'],
+                  'Cache-Control', '\'public, must-revalidate, proxy-revalidate\'',
+                  '"test,;{}"', 'foo'],
                  ['blah', '"hello;world"'],
-                 ['try_files', '$uri @rewrites']]]]]])
+                 ['try_files', '$uri', '@rewrites']]]]]])
+
+    def test_parse_from_file3(self):
+        with open(util.get_data_filename('multiline_quotes.conf')) as handle:
+            parsed = util.filter_comments(load(handle))
+        self.assertEqual(
+            parsed,
+            [[['http'],
+                [[['server'],
+                    [['listen', '*:443'],
+                    [['location', '/'],
+                        [['body_filter_by_lua',
+                          '\'ngx.ctx.buffered = (ngx.ctx.buffered or "")'
+                          ' .. string.sub(ngx.arg[1], 1, 1000)\n'
+                          '                            '
+                          'if ngx.arg[2] then\n'
+                          '                              '
+                          'ngx.var.resp_body = ngx.ctx.buffered\n'
+                          '                            end\'']]]]]]]])
 
     def test_abort_on_parse_failure(self):
         with open(util.get_data_filename('broken.conf')) as handle:
@@ -117,7 +135,7 @@ class TestRawNginxParser(unittest.TestCase):
         with open(util.get_data_filename('nginx.conf')) as handle:
             parsed = load(handle)
         parsed[-1][-1].append(UnspacedList([['server'],
-                               [['listen', ' ', '443 ssl'],
+                               [['listen', ' ', '443', ' ', 'ssl'],
                                 ['server_name', ' ', 'localhost'],
                                 ['ssl_certificate', ' ', 'cert.pem'],
                                 ['ssl_certificate_key', ' ', 'cert.key'],
@@ -126,9 +144,9 @@ class TestRawNginxParser(unittest.TestCase):
                                 ['ssl_ciphers', ' ', 'HIGH:!aNULL:!MD5'],
                                 [['location', ' ', '/'],
                                  [['root', ' ', 'html'],
-                                  ['index', ' ', 'index.html index.htm']]]]]))
+                                  ['index', ' ', 'index.html', ' ', 'index.htm']]]]]))
 
-        with tempfile.TemporaryFile() as f:
+        with tempfile.TemporaryFile(mode='w+t') as f:
             dump(parsed, f)
             f.seek(0)
             parsed_new = load(f)
@@ -138,7 +156,7 @@ class TestRawNginxParser(unittest.TestCase):
         with open(util.get_data_filename('minimalistic_comments.conf')) as handle:
             parsed = load(handle)
 
-        with tempfile.TemporaryFile() as f:
+        with tempfile.TemporaryFile(mode='w+t') as f:
             dump(parsed, f)
             f.seek(0)
             parsed_new = load(f)
@@ -161,10 +179,177 @@ class TestRawNginxParser(unittest.TestCase):
         parsed = loads('if ($http_accept ~* "webp") { set $webp "true"; }')
 
         self.assertEqual(parsed, [
-            [['if', '($http_accept ~* "webp")'],
-             [['set', '$webp "true"']]]
+            [['if', '($http_accept', '~*', '"webp")'],
+             [['set', '$webp', '"true"']]]
         ])
 
+    def test_comment_in_block(self):
+        parsed = loads("""http {
+          # server{
+          }""")
+
+        self.assertEqual(parsed, [
+            [['http'],
+             [['#', ' server{']]]
+        ])
+
+    def test_access_log(self):
+        # see issue #3798
+        parsed = loads('access_log syslog:server=unix:/dev/log,facility=auth,'
+            'tag=nginx_post,severity=info custom;')
+
+        self.assertEqual(parsed, [
+            ['access_log',
+             'syslog:server=unix:/dev/log,facility=auth,tag=nginx_post,severity=info',
+             'custom']
+        ])
+
+    def test_add_header(self):
+        # see issue #3798
+        parsed = loads('add_header Cache-Control no-cache,no-store,must-revalidate,max-age=0;')
+
+        self.assertEqual(parsed, [
+            ['add_header', 'Cache-Control', 'no-cache,no-store,must-revalidate,max-age=0']
+        ])
+
+    def test_map_then_assignment_in_block(self):
+        # see issue #3798
+        test_str = """http {
+            map $http_upgrade $connection_upgrade {
+              default upgrade;
+              ''      close;
+              "~Opera Mini" 1;
+              *.example.com 1;
+            }
+            one;
+        }"""
+        parsed = loads(test_str)
+        self.assertEqual(parsed, [
+            [['http'], [
+                [['map', '$http_upgrade', '$connection_upgrade'], [
+                    ['default', 'upgrade'],
+                    ["''", 'close'],
+                    ['"~Opera Mini"', '1'],
+                    ['*.example.com', '1']
+                ]],
+                ['one']
+            ]]
+        ])
+
+    def test_variable_name(self):
+        parsed = loads('try_files /typo3temp/tx_ncstaticfilecache/'
+            '$host${request_uri}index.html @nocache;')
+
+        self.assertEqual(parsed, [
+            ['try_files',
+             '/typo3temp/tx_ncstaticfilecache/$host${request_uri}index.html',
+             '@nocache']
+        ])
+
+    def test_weird_blocks(self):
+        test = r"""
+            if ($http_user_agent ~ MSIE) {
+                rewrite ^(.*)$ /msie/$1 break;
+            }
+
+            if ($http_cookie ~* "id=([^;]+)(?:;|$)") {
+               set $id $1;
+            }
+
+            if ($request_method = POST) {
+               return 405;
+            }
+
+            if ($request_method) {
+               return 403;
+            }
+
+            if ($args ~ post=140){
+              rewrite ^ http://example.com/;
+            }
+
+            location ~ ^/users/(.+\.(?:gif|jpe?g|png))$ {
+              alias /data/w3/images/$1;
+            }
+        """
+        parsed = loads(test)
+        self.assertEqual(parsed, [[['if', '($http_user_agent', '~', 'MSIE)'],
+            [['rewrite', '^(.*)$', '/msie/$1', 'break']]],
+            [['if', '($http_cookie', '~*', '"id=([^;]+)(?:;|$)")'], [['set', '$id', '$1']]],
+            [['if', '($request_method', '=', 'POST)'], [['return', '405']]],
+            [['if', '($request_method)'],
+            [['return', '403']]], [['if', '($args', '~', 'post=140)'],
+            [['rewrite', '^', 'http://example.com/']]],
+            [['location', '~', '^/users/(.+\\.(?:gif|jpe?g|png))$'],
+            [['alias', '/data/w3/images/$1']]]]
+        )
+
+    def test_edge_cases(self):
+        # quotes
+        parsed = loads(r'"hello\""; # blah "heh heh"')
+        self.assertEqual(parsed, [['"hello\\""'], ['#', ' blah "heh heh"']])
+
+        # empty var as block
+        parsed = loads(r"${}")
+        self.assertEqual(parsed, [[['$'], []]])
+
+        # if with comment
+        parsed = loads("""if ($http_cookie ~* "id=([^;]+)(?:;|$)") { # blah )
+            }""")
+        self.assertEqual(parsed, [[['if', '($http_cookie', '~*', '"id=([^;]+)(?:;|$)")'],
+            [['#', ' blah )']]]])
+
+        # end paren
+        test = """
+            one"test";
+            ("two");
+            "test")red;
+            "test")"blue";
+            "test")"three;
+            (one"test")one;
+            one";
+            one"test;
+            one"test"one;
+        """
+        parsed = loads(test)
+        self.assertEqual(parsed, [
+            ['one"test"'],
+            ['("two")'],
+            ['"test")red'],
+            ['"test")"blue"'],
+            ['"test")"three'],
+            ['(one"test")one'],
+            ['one"'],
+            ['one"test'],
+            ['one"test"one']
+        ])
+        self.assertRaises(ParseException, loads, r'"test"one;') # fails
+        self.assertRaises(ParseException, loads, r'"test;') # fails
+
+        # newlines
+        test = """
+            server_name foo.example.com bar.example.com \
+                        baz.example.com qux.example.com;
+            server_name foo.example.com bar.example.com
+                        baz.example.com qux.example.com;
+        """
+        parsed = loads(test)
+        self.assertEqual(parsed, [
+            ['server_name', 'foo.example.com', 'bar.example.com',
+                'baz.example.com', 'qux.example.com'],
+            ['server_name', 'foo.example.com', 'bar.example.com',
+                'baz.example.com', 'qux.example.com']
+        ])
+
+        # variable weirdness
+        parsed = loads("directive $var;")
+        self.assertEqual(parsed, [['directive', '$var']])
+        self.assertRaises(ParseException, loads, "server {server_name test.com};")
+        self.assertRaises(ParseException, loads, "directive ${var};")
+        self.assertEqual(loads("blag${dfgdfg};"), [['blag${dfgdfg}']])
+        self.assertRaises(ParseException, loads, "blag${dfgdf{g};")
+
+
 class TestUnspacedList(unittest.TestCase):
     """Test the UnspacedList data structure"""
     def setUp(self):
@@ -219,18 +404,18 @@ class TestUnspacedList(unittest.TestCase):
                 ['\n    ', 'listen', '       ', '127.0.0.1'],
                 ['\n    ', 'server_name', ' ', '.example.com'],
                 ['\n    ', 'server_name', ' ', 'example.*'], '\n',
-                ['listen', ' ', '5001 ssl']])
+                ['listen', ' ', '5001', ' ', 'ssl']])
         x.insert(5, "FROGZ")
         self.assertEqual(x,
             [['listen', '69.50.225.155:9000'], ['listen', '127.0.0.1'],
             ['server_name', '.example.com'], ['server_name', 'example.*'],
-            ['listen', '5001 ssl'], 'FROGZ'])
+            ['listen', '5001', 'ssl'], 'FROGZ'])
         self.assertEqual(x.spaced,
             [['\n    ', 'listen', '       ', '69.50.225.155:9000'],
             ['\n    ', 'listen', '       ', '127.0.0.1'],
             ['\n    ', 'server_name', ' ', '.example.com'],
             ['\n    ', 'server_name', ' ', 'example.*'], '\n',
-            ['listen', ' ', '5001 ssl'],
+            ['listen', ' ', '5001', ' ', 'ssl'],
             'FROGZ'])
 
     def test_rawlists(self):

certbot-nginx/certbot_nginx/tests/obj_test.py

@@ -108,8 +108,8 @@ class VirtualHostTest(unittest.TestCase):
         from certbot_nginx.obj import Addr
         raw1 = [
             ['listen', '69.50.225.155:9000'],
-            [['if', '($scheme != "https") '],
-                [['return', '301 https://$host$request_uri']]
+            [['if', '($scheme', '!=', '"https") '],
+                [['return', '301', 'https://$host$request_uri']]
             ],
             ['#', ' managed by Certbot']
         ]
@@ -119,8 +119,8 @@ class VirtualHostTest(unittest.TestCase):
             set(['localhost']), raw1, [])
         raw2 = [
             ['listen', '69.50.225.155:9000'],
-            [['if', '($scheme != "https") '],
-                [['return', '301 https://$host$request_uri']]
+            [['if', '($scheme', '!=', '"https") '],
+                [['return', '301', 'https://$host$request_uri']]
             ]
         ]
         self.vhost2 = VirtualHost(
@@ -129,7 +129,7 @@ class VirtualHostTest(unittest.TestCase):
             set(['localhost']), raw2, [])
         raw3 = [
             ['listen', '69.50.225.155:9000'],
-            ['rewrite', '^(.*)$ $scheme://www.domain.com$1 permanent;']
+            ['rewrite', '^(.*)$', '$scheme://www.domain.com$1', 'permanent']
         ]
         self.vhost3 = VirtualHost(
             "filep",
@@ -158,7 +158,7 @@ class VirtualHostTest(unittest.TestCase):
 
     def test_str(self):
         stringified = '\n'.join(['file: filep', 'addrs: localhost',
-                                 "names: set(['localhost'])", 'ssl: False',
+                                 "names: ['localhost']", 'ssl: False',
                                  'enabled: False'])
         self.assertEqual(stringified, str(self.vhost1))
 
@@ -181,7 +181,9 @@ class VirtualHostTest(unittest.TestCase):
             ['#', ' managed by Certbot'],
             ['ssl_certificate_key', '/etc/letsencrypt/live/two.functorkitten.xyz/privkey.pem'],
             ['#', ' managed by Certbot'],
-            [['if', '($scheme != "https")'], [['return', '301 https://$host$request_uri']]],
+            [['if', '($scheme', '!=', '"https")'],
+             [['return', '301', 'https://$host$request_uri']]
+            ],
             ['#', ' managed by Certbot'], []]
         vhost_haystack = VirtualHost(
             "filp",
@@ -195,7 +197,9 @@ class VirtualHostTest(unittest.TestCase):
             ['#', ' managed by Certbot'],
             ['ssl_certificate_key', '/etc/letsencrypt/live/two.functorkitten.xyz/privkey.pem'],
             ['#', ' managed by Certbot'],
-            [['if', '($scheme != "https")'], [['return', '302 https://$host$request_uri']]],
+            [['if', '($scheme', '!=', '"https")'],
+             [['return', '302', 'https://$host$request_uri']]
+            ],
             ['#', ' managed by Certbot'], []]
         vhost_bad_haystack = VirtualHost(
             "filp",

certbot-nginx/certbot_nginx/tests/parser_test.py

@@ -52,7 +52,7 @@ class NginxParserTest(util.NginxTest):
                                'sites-enabled/sslon.com',
                                'sites-enabled/globalssl.com']]),
                          set(nparser.parsed.keys()))
-        self.assertEqual([['server_name', 'somename  alias  another.alias']],
+        self.assertEqual([['server_name', 'somename', 'alias', 'another.alias']],
                          nparser.parsed[nparser.abs_path('server.conf')])
         self.assertEqual([[['server'], [['listen', '69.50.225.155:9000'],
                                         ['listen', '127.0.0.1'],
@@ -168,16 +168,16 @@ class NginxParserTest(util.NginxTest):
                [['location', '/'], [['root', 'html'], ['index', 'index.html index.htm']]]
                ], None)
         self.assertFalse(nparser.has_ssl_on_directive(mock_vhost))
-        mock_vhost.raw = [['listen', '*:80 default_server ssl'],
-                          ['server_name', '*.www.foo.com *.www.example.com'],
+        mock_vhost.raw = [['listen', '*:80', 'default_server', 'ssl'],
+                          ['server_name', '*.www.foo.com', '*.www.example.com'],
                           ['root', '/home/ubuntu/sites/foo/']]
         self.assertFalse(nparser.has_ssl_on_directive(mock_vhost))
         mock_vhost.raw = [['listen', '80 ssl'],
-                          ['server_name', '*.www.foo.com *.www.example.com']]
+                          ['server_name', '*.www.foo.com', '*.www.example.com']]
         self.assertFalse(nparser.has_ssl_on_directive(mock_vhost))
         mock_vhost.raw = [['listen', '80'],
                           ['ssl', 'on'],
-                          ['server_name', '*.www.foo.com *.www.example.com']]
+                          ['server_name', '*.www.foo.com', '*.www.example.com']]
         self.assertTrue(nparser.has_ssl_on_directive(mock_vhost))
 
     def test_add_server_directives(self):
@@ -309,7 +309,7 @@ class NginxParserTest(util.NginxTest):
         self.assertFalse(server['ssl'])
 
         server = parser._parse_server_raw([ #pylint: disable=protected-access
-            ['listen', '443 ssl']
+            ['listen', '443', 'ssl']
         ])
         self.assertTrue(server['ssl'])
 
@@ -323,6 +323,12 @@ class NginxParserTest(util.NginxTest):
         ])
         self.assertTrue(server['ssl'])
 
+    def test_parse_server_raw_unix(self):
+        server = parser._parse_server_raw([ #pylint: disable=protected-access
+            ['listen', 'unix:/var/run/nginx.sock']
+        ])
+        self.assertEqual(len(server['addrs']), 0)
+
     def test_parse_server_global_ssl_applied(self):
         nparser = parser.NginxParser(self.config_path, self.ssl_options)
         server = nparser.parse_server([
@@ -335,16 +341,21 @@ class NginxParserTest(util.NginxTest):
         self.assertEqual(nginxparser.UnspacedList(nparser.loc["ssl_options"]),
                          [['ssl_session_cache', 'shared:le_nginx_SSL:1m'],
                           ['ssl_session_timeout', '1440m'],
-                          ['ssl_protocols', 'TLSv1 TLSv1.1 TLSv1.2'],
+                          ['ssl_protocols', 'TLSv1', 'TLSv1.1', 'TLSv1.2'],
                           ['ssl_prefer_server_ciphers', 'on'],
-                          ['ssl_ciphers', '"ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-ECDSA-'+
-                           'AES256-GCM-SHA384 ECDHE-ECDSA-AES128-SHA ECDHE-ECDSA-AES256'+
-                           '-SHA ECDHE-ECDSA-AES128-SHA256 ECDHE-ECDSA-AES256-SHA384'+
-                           ' ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-GCM-SHA384'+
-                           ' ECDHE-RSA-AES128-SHA ECDHE-RSA-AES128-SHA256 ECDHE-RSA-'+
-                           'AES256-SHA384 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM'+
-                           '-SHA384 DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA DHE-RSA-'+
-                           'AES128-SHA256 DHE-RSA-AES256-SHA256 EDH-RSA-DES-CBC3-SHA"']
+                          ['ssl_ciphers', '"ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-'+
+                          'RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:'+
+                          'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-'+
+                          'SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-'+
+                          'SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-'+
+                          'SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:'+
+                          'ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-'+
+                          'AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:'+
+                          'DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-'+
+                          'SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-'+
+                          'RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:'+
+                          'AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:'+
+                          'AES256-SHA:DES-CBC3-SHA:!DSS"']
                          ])
 
 if __name__ == "__main__":

certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/multiline_quotes.conf

@@ -0,0 +1,16 @@
+# Test nginx configuration file with multiline quoted strings.
+# Good example of usage for multilined quoted values is when
+# using Openresty's Lua directives and you wish to keep the
+# inline Lua code readable.
+http {
+  server {
+      listen *:443; # because there should be no other port open.
+
+      location / {
+        body_filter_by_lua 'ngx.ctx.buffered = (ngx.ctx.buffered or "") .. string.sub(ngx.arg[1], 1, 1000)
+                            if ngx.arg[2] then
+                              ngx.var.resp_body = ngx.ctx.buffered
+                            end';
+      }
+  }
+}

certbot-nginx/certbot_nginx/tests/tls_sni_01_test.py

@@ -3,6 +3,7 @@ import unittest
 import shutil
 
 import mock
+import six
 
 from acme import challenges
 
@@ -23,25 +24,25 @@ class TlsSniPerformTest(util.NginxTest):
     achalls = [
         achallenges.KeyAuthorizationAnnotatedChallenge(
             challb=acme_util.chall_to_challb(
-                challenges.TLSSNI01(token="kNdwjwOeX0I_A8DXt9Msmg"), "pending"),
+                challenges.TLSSNI01(token=b"kNdwjwOeX0I_A8DXt9Msmg"), "pending"),
             domain="www.example.com", account_key=account_key),
         achallenges.KeyAuthorizationAnnotatedChallenge(
             challb=acme_util.chall_to_challb(
                 challenges.TLSSNI01(
-                    token="\xba\xa9\xda?<m\xaewmx\xea\xad\xadv\xf4\x02\xc9y"
-                          "\x80\xe2_X\t\xe7\xc7\xa4\t\xca\xf7&\x945"
+                    token=b"\xba\xa9\xda?<m\xaewmx\xea\xad\xadv\xf4\x02\xc9y"
+                          b"\x80\xe2_X\t\xe7\xc7\xa4\t\xca\xf7&\x945"
                 ), "pending"),
             domain="another.alias", account_key=account_key),
         achallenges.KeyAuthorizationAnnotatedChallenge(
             challb=acme_util.chall_to_challb(
                 challenges.TLSSNI01(
-                    token="\x8c\x8a\xbf_-f\\cw\xee\xd6\xf8/\xa5\xe3\xfd"
-                          "\xeb9\xf1\xf5\xb9\xefVM\xc9w\xa4u\x9c\xe1\x87\xb4"
+                    token=b"\x8c\x8a\xbf_-f\\cw\xee\xd6\xf8/\xa5\xe3\xfd"
+                          b"\xeb9\xf1\xf5\xb9\xefVM\xc9w\xa4u\x9c\xe1\x87\xb4"
                 ), "pending"),
             domain="www.example.org", account_key=account_key),
         achallenges.KeyAuthorizationAnnotatedChallenge(
             challb=acme_util.chall_to_challb(
-                challenges.TLSSNI01(token="kNdwjxOeX0I_A8DXt9Msmg"), "pending"),
+                challenges.TLSSNI01(token=b"kNdwjxOeX0I_A8DXt9Msmg"), "pending"),
             domain="sslon.com", account_key=account_key),
     ]
 
@@ -49,7 +50,7 @@ class TlsSniPerformTest(util.NginxTest):
         super(TlsSniPerformTest, self).setUp()
 
         config = util.get_nginx_configurator(
-            self.config_path, self.config_dir, self.work_dir)
+            self.config_path, self.config_dir, self.work_dir, self.logs_dir)
 
         from certbot_nginx import tls_sni_01
         self.sni = tls_sni_01.NginxTlsSni01(config)
@@ -117,7 +118,7 @@ class TlsSniPerformTest(util.NginxTest):
             util.contains_at_depth(http, ['server_name', 'another.alias'], 3))
 
         self.assertEqual(len(sni_responses), 4)
-        for i in xrange(4):
+        for i in six.moves.range(4):
             self.assertEqual(sni_responses[i], acme_responses[i])
 
     def test_mod_config(self):
@@ -148,7 +149,7 @@ class TlsSniPerformTest(util.NginxTest):
             else:
                 response = self.achalls[2].response(self.account_key)
                 self.assertEqual(vhost.addrs, set(v_addr2_print))
-            self.assertEqual(vhost.names, set([response.z_domain]))
+            self.assertEqual(vhost.names, set([response.z_domain.decode('ascii')]))
 
         self.assertEqual(len(vhs), 2)
 

certbot-nginx/certbot_nginx/tests/util.py

@@ -2,6 +2,7 @@
 import copy
 import os
 import pkg_resources
+import tempfile
 import unittest
 
 import mock
@@ -27,6 +28,7 @@ class NginxTest(unittest.TestCase):  # pylint: disable=too-few-public-methods
 
         self.temp_dir, self.config_dir, self.work_dir = common.dir_setup(
             "etc_nginx", "certbot_nginx.tests")
+        self.logs_dir = tempfile.mkdtemp('logs')
 
         self.ssl_options = common.setup_ssl_options(
             self.config_dir, constants.MOD_SSL_CONF_SRC,
@@ -46,7 +48,7 @@ def get_data_filename(filename):
 
 
 def get_nginx_configurator(
-        config_path, config_dir, work_dir, version=(1, 6, 2)):
+        config_path, config_dir, work_dir, logs_dir, version=(1, 6, 2)):
     """Create an Nginx Configurator with the specified options."""
 
     backups = os.path.join(work_dir, "backups")
@@ -62,11 +64,13 @@ def get_nginx_configurator(
                     le_vhost_ext="-le-ssl.conf",
                     config_dir=config_dir,
                     work_dir=work_dir,
+                    logs_dir=logs_dir,
                     backup_dir=backups,
                     temp_checkpoint_dir=os.path.join(work_dir, "temp_checkpoints"),
                     in_progress_dir=os.path.join(backups, "IN_PROGRESS"),
                     server="https://acme-server.org:443/new",
                     tls_sni_01_port=5001,
+                    dry_run=False,
                 ),
                 name="nginx",
                 version=version)
@@ -111,7 +115,7 @@ def contains_at_depth(haystack, needle, n):
     """
     # Specifically use hasattr rather than isinstance(..., collections.Iterable)
     # because we want to include lists but reject strings.
-    if not hasattr(haystack, '__iter__'):
+    if not hasattr(haystack, '__iter__') or hasattr(haystack, 'strip'):
         return False
     if n == 0:
         return needle in haystack

certbot-nginx/certbot_nginx/tls_sni_01.py

@@ -1,9 +1,10 @@
 """A class that performs TLS-SNI-01 challenges for Nginx"""
 
-import itertools
 import logging
 import os
 
+import six
+
 from certbot import errors
 from certbot.plugins import common
 
@@ -95,11 +96,12 @@ class NginxTlsSni01(common.TLSSNI01):
         bucket_directive = ['\n', 'server_names_hash_bucket_size', ' ', '128']
 
         main = self.configurator.parser.parsed[root]
-        for key, body in main:
-            if key == ['http']:
+        for line in main:
+            if line[0] == ['http']:
+                body = line[1]
                 found_bucket = False
-                for k, _ in body:
-                    if k == bucket_directive[1]:
+                for inner_line in body:
+                    if inner_line[0] == bucket_directive[1]:
                         found_bucket = True
                 if not found_bucket:
                     body.insert(0, bucket_directive)
@@ -113,7 +115,7 @@ class NginxTlsSni01(common.TLSSNI01):
                 'TLS-SNI-01 challenges in %s.' % root)
 
         config = [self._make_server_block(pair[0], pair[1])
-                  for pair in itertools.izip(self.achalls, ll_addrs)]
+                  for pair in six.moves.zip(self.achalls, ll_addrs)]
         config = nginxparser.UnspacedList(config)
 
         self.configurator.reverter.register_file_creation(
@@ -142,7 +144,7 @@ class NginxTlsSni01(common.TLSSNI01):
         block = [['listen', ' ', addr.to_string(include_default=False)] for addr in addrs]
 
         block.extend([['server_name', ' ',
-                       achall.response(achall.account_key).z_domain],
+                       achall.response(achall.account_key).z_domain.decode('ascii')],
                       # access and error logs necessary for
                       # integration testing (non-root)
                       ['access_log', ' ', os.path.join(

certbot-nginx/setup.py

@@ -4,12 +4,13 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.12.0.dev0'
+version = '0.14.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [
     'acme=={0}'.format(version),
     'certbot=={0}'.format(version),
+    'mock',
     'PyOpenSSL',
     'pyparsing>=1.5.5',  # Python3 support; perhaps unnecessary?
     # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
@@ -18,11 +19,6 @@ install_requires = [
     'zope.interface',
 ]
 
-if sys.version_info < (2, 7):
-    install_requires.append('mock<1.1.0')
-else:
-    install_requires.append('mock')
-
 docs_extras = [
     'Sphinx>=1.0',  # autodoc_member_order = 'bysource', autodoc_default_flags
     'sphinx_rtd_theme',

certbot/__init__.py

@@ -1,4 +1,4 @@
 """Certbot client."""
 
 # version number like 1.2.3a0, must have at least 2 parts, like 1.2
-__version__ = '0.12.0.dev0'
+__version__ = '0.14.0.dev0'

certbot/account.py

@@ -74,7 +74,8 @@ class Account(object):  # pylint: disable=too-few-public-methods
             self.meta.creation_dt), self.meta.creation_host, self.id[:4])
 
     def __repr__(self):
-        return "<{0}({1})>".format(self.__class__.__name__, self.id)
+        return "<{0}({1}, {2}, {3})>".format(
+            self.__class__.__name__, self.regr, self.id, self.meta)
 
     def __eq__(self, other):
         return (isinstance(other, self.__class__) and
@@ -98,7 +99,7 @@ def report_new_account(config):
 
 
 class AccountMemoryStorage(interfaces.AccountStorage):
-    """In-memory account strage."""
+    """In-memory account storage."""
 
     def __init__(self, initial_accounts=None):
         self.accounts = initial_accounts if initial_accounts is not None else {}
@@ -106,7 +107,8 @@ class AccountMemoryStorage(interfaces.AccountStorage):
     def find_all(self):
         return list(six.itervalues(self.accounts))
 
-    def save(self, account):
+    def save(self, account, acme):
+        # pylint: disable=unused-argument
         if account.id in self.accounts:
             logger.debug("Overwriting account: %s", account.id)
         self.accounts[account.id] = account
@@ -117,6 +119,16 @@ class AccountMemoryStorage(interfaces.AccountStorage):
         except KeyError:
             raise errors.AccountNotFound(account_id)
 
+class RegistrationResourceWithNewAuthzrURI(messages.RegistrationResource):
+    """A backwards-compatible RegistrationResource with a new-authz URI.
+
+       Hack: Certbot versions pre-0.11.1 expect to load
+       new_authzr_uri as part of the account. Because people
+       sometimes switch between old and new versions, we will
+       continue to write out this field for some time so older
+       clients don't crash in that scenario.
+    """
+    new_authzr_uri = jose.Field('new_authzr_uri')
 
 class AccountFileStorage(interfaces.AccountStorage):
     """Accounts file storage.
@@ -181,16 +193,16 @@ class AccountFileStorage(interfaces.AccountStorage):
                     account_id, acc.id))
         return acc
 
-    def save(self, account):
-        self._save(account, regr_only=False)
+    def save(self, account, acme):
+        self._save(account, acme, regr_only=False)
 
-    def save_regr(self, account):
+    def save_regr(self, account, acme):
         """Save the registration resource.
 
         :param Account account: account whose regr should be saved
 
         """
-        self._save(account, regr_only=True)
+        self._save(account, acme, regr_only=True)
 
     def delete(self, account_id):
         """Delete registration info from disk
@@ -204,13 +216,19 @@ class AccountFileStorage(interfaces.AccountStorage):
                 "Account at %s does not exist" % account_dir_path)
         shutil.rmtree(account_dir_path)
 
-    def _save(self, account, regr_only):
+    def _save(self, account, acme, regr_only):
         account_dir_path = self._account_dir_path(account.id)
         util.make_or_verify_dir(account_dir_path, 0o700, os.geteuid(),
                                 self.config.strict_permissions)
         try:
             with open(self._regr_path(account_dir_path), "w") as regr_file:
-                regr_file.write(account.regr.json_dumps())
+                regr = account.regr
+                with_uri = RegistrationResourceWithNewAuthzrURI(
+                    new_authzr_uri=acme.directory.new_authz,
+                    body=regr.body,
+                    uri=regr.uri,
+                    terms_of_service=regr.terms_of_service)
+                regr_file.write(with_uri.json_dumps())
             if not regr_only:
                 with util.safe_open(self._key_path(account_dir_path),
                                     "w", chmod=0o400) as key_file:

certbot/auth_handler.py

@@ -66,11 +66,16 @@ class AuthHandler(object):
             self.authzr[domain] = self.acme.request_domain_challenges(domain)
 
         self._choose_challenges(domains)
+        config = zope.component.getUtility(interfaces.IConfig)
+        notify = zope.component.getUtility(interfaces.IDisplay).notification
 
         # While there are still challenges remaining...
         while self.achalls:
             resp = self._solve_challenges()
             logger.info("Waiting for verification...")
+            if config.debug_challenges:
+                notify('Challenges loaded. Press continue to submit to CA. '
+                       'Pass "-v" for more info about challenges.', pause=True)
 
             # Send all Responses - this modifies achalls
             self._respond(resp, best_effort)

certbot/cert_manager.py

@@ -100,7 +100,10 @@ def lineage_for_certname(cli_config, certname):
     configs_dir = cli_config.renewal_configs_dir
     # Verify the directory is there
     util.make_or_verify_dir(configs_dir, mode=0o755, uid=os.geteuid())
-    renewal_file = storage.renewal_file_for_certname(cli_config, certname)
+    try:
+        renewal_file = storage.renewal_file_for_certname(cli_config, certname)
+    except errors.CertStorageError:
+        return None
     try:
         return storage.RenewableCert(renewal_file, cli_config)
     except (errors.CertStorageError, IOError):

certbot/cli.py

@@ -19,6 +19,7 @@ import certbot
 from certbot import constants
 from certbot import crypto_util
 from certbot import errors
+from certbot import hooks
 from certbot import interfaces
 from certbot import util
 
@@ -207,7 +208,7 @@ def set_by_cli(var):
 
     return False
 # static housekeeping var
-set_by_cli.detector = None
+set_by_cli.detector = None  # type: ignore
 
 
 def has_default_value(option, value):
@@ -515,6 +516,13 @@ class HelpfulArgumentParser(object):
 
         return usage
 
+    def remove_config_file_domains_for_renewal(self, parsed_args):
+        """Make "certbot renew" safe if domains are set in cli.ini."""
+        # Works around https://github.com/certbot/certbot/issues/4096
+        if self.verb == "renew":
+            for source, flags in self.parser._source_to_settings.items(): # pylint: disable=protected-access
+                if source.startswith("config_file") and "domains" in flags:
+                    parsed_args.domains = _Default() if self.detect_defaults else []
 
     def parse_args(self):
         """Parses command line arguments and returns the result.
@@ -527,6 +535,8 @@ class HelpfulArgumentParser(object):
         parsed_args.func = self.VERBS[self.verb]
         parsed_args.verb = self.verb
 
+        self.remove_config_file_domains_for_renewal(parsed_args)
+
         if self.detect_defaults:
             return parsed_args
 
@@ -556,6 +566,11 @@ class HelpfulArgumentParser(object):
         if parsed_args.must_staple:
             parsed_args.staple = True
 
+        if parsed_args.validate_hooks:
+            hooks.validate_hooks(parsed_args)
+
+        possible_deprecation_warning(parsed_args)
+
         return parsed_args
 
     def set_test_server(self, parsed_args):
@@ -895,7 +910,7 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
              "'run' subcommand this means reinstall the existing cert). (default: Ask)")
     helpful.add(
         "automation", "--expand", action="store_true",
-        help="If an existing cert covers some subset of the requested names, "
+        help="If an existing cert is a strict subset of the requested names, "
              "always expand and replace it with the additional names. (default: Ask)")
     helpful.add(
         "automation", "--version", action="version",
@@ -941,6 +956,11 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         help="(certbot-auto only) prevent the certbot-auto script from"
              " upgrading itself to newer released versions (default: Upgrade"
              " automatically)")
+    helpful.add(
+        "automation", "--no-bootstrap", action="store_true",
+        help="(certbot-auto only) prevent the certbot-auto script from"
+             " installing OS-level dependencies (default: Prompt to install "
+             " OS-wide dependencies, but exit if the user says 'No')")
     helpful.add(
         ["automation", "renew", "certonly", "run"],
         "-q", "--quiet", dest="quiet", action="store_true",
@@ -955,6 +975,11 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         "testing", "--debug", action="store_true",
         help="Show tracebacks in case of errors, and allow certbot-auto "
              "execution on experimental platforms")
+    helpful.add(
+        [None, "certonly", "renew", "run"], "--debug-challenges", action="store_true",
+        default=flag_default("debug_challenges"),
+        help="After setting up challenges, wait for user input before "
+             "submitting to CA")
     helpful.add(
         "testing", "--no-verify-ssl", action="store_true",
         help=config_help("no_verify_ssl"),
@@ -1051,9 +1076,11 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         "renew", "--renew-hook",
         help="Command to be run in a shell once for each successfully renewed"
         " certificate. For this command, the shell variable $RENEWED_LINEAGE"
-        " will point to the config live subdirectory containing the new certs"
+        " will point to the config live subdirectory (for example,"
+        " \"/etc/letsencrypt/live/example.com\") containing the new certs"
         " and keys; the shell variable $RENEWED_DOMAINS will contain a"
-        " space-delimited list of renewed cert domains")
+        " space-delimited list of renewed cert domains (for example,"
+        " \"example.com www.example.com\"")
     helpful.add(
         "renew", "--disable-hook-validation",
         action='store_false', dest='validate_hooks', default=True,
@@ -1156,7 +1183,7 @@ def _paths_parser(helpful):
     default_cp = None
     if verb == "certonly":
         default_cp = flag_default("auth_chain_path")
-    add(["install", "paths"], "--fullchain-path", default=default_cp, type=os.path.abspath,
+    add(["paths", "install"], "--fullchain-path", default=default_cp, type=os.path.abspath,
         help="Accompanying path to a full certificate chain (cert plus chain).")
     add("paths", "--chain-path", default=default_cp, type=os.path.abspath,
         help="Accompanying path to a certificate chain.")

certbot/client.py

@@ -8,6 +8,7 @@ import OpenSSL
 import zope.component
 
 from acme import client as acme_client
+from acme import errors as acme_errors
 from acme import jose
 from acme import messages
 
@@ -117,7 +118,7 @@ def register(config, account_storage, tos_cb=None):
             logger.warning(msg)
             raise errors.Error(msg)
         if not config.dry_run:
-            logger.warning("Registering without email!")
+            logger.info("Registering without email!")
 
     # Each new registration shall use a fresh new key
     key = jose.JWKRSA(key=jose.ComparableRSAKey(
@@ -138,7 +139,7 @@ def register(config, account_storage, tos_cb=None):
 
     acc = account.Account(regr, key)
     account.report_new_account(config)
-    account_storage.save(acc)
+    account_storage.save(acc, acme)
 
     eff.handle_subscription(config)
 
@@ -155,8 +156,6 @@ def perform_registration(acme, config):
 
     :returns: Registration Resource.
     :rtype: `acme.messages.RegistrationResource`
-
-    :raises .UnexpectedUpdate:
     """
     try:
         return acme.register(messages.NewRegistration.from_data(email=config.email))
@@ -168,7 +167,7 @@ def perform_registration(acme, config):
                        "registration again." % config.email)
                 raise errors.Error(msg)
             else:
-                config.namespace.email = display_ops.get_email(invalid=True)
+                config.email = display_ops.get_email(invalid=True)
                 return perform_registration(acme, config)
         else:
             raise
@@ -208,15 +207,14 @@ class Client(object):
         else:
             self.auth_handler = None
 
-    def obtain_certificate_from_csr(self, domains, csr,
-        typ=OpenSSL.crypto.FILETYPE_ASN1, authzr=None):
+    def obtain_certificate_from_csr(self, domains, csr, authzr=None):
         """Obtain certificate.
 
         Internal function with precondition that `domains` are
         consistent with identifiers present in the `csr`.
 
         :param list domains: Domain names.
-        :param .util.CSR csr: DER-encoded Certificate Signing
+        :param .util.CSR csr: PEM-encoded Certificate Signing
             Request. The key used to generate this CSR can be different
             than `authkey`.
         :param list authzr: List of
@@ -242,9 +240,30 @@ class Client(object):
 
         certr = self.acme.request_issuance(
             jose.ComparableX509(
-                OpenSSL.crypto.load_certificate_request(typ, csr.data)),
+                OpenSSL.crypto.load_certificate_request(OpenSSL.crypto.FILETYPE_PEM, csr.data)),
                 authzr)
-        return certr, self.acme.fetch_chain(certr)
+
+        notify = zope.component.getUtility(interfaces.IDisplay).notification
+        retries = 0
+        chain = None
+
+        while retries <= 1:
+            if retries:
+                notify('Failed to fetch chain, please check your network '
+                       'and continue', pause=True)
+            try:
+                chain = self.acme.fetch_chain(certr)
+                break
+            except acme_errors.Error:
+                logger.debug('Failed to fetch chain', exc_info=True)
+                retries += 1
+
+        if chain is None:
+            raise acme_errors.Error(
+                'Failed to fetch chain. You should not deploy the generated '
+                'certificate, please rerun the command for a new one.')
+
+        return certr, chain
 
     def obtain_certificate(self, domains):
         """Obtains a certificate from the ACME server.
@@ -305,10 +324,12 @@ class Client(object):
         key = crypto_util.save_key(
             key_pem, self.config.key_dir)
         csr = crypto_util.init_save_csr(key, domains, self.config.csr_dir)
+        certr, chain = self.obtain_certificate_from_csr(
+            domains, csr, authzr=authzr)
 
-        return (self.obtain_certificate_from_csr(domains, csr, authzr=authzr)
-                                                                + (key, csr))
+        return certr, chain, key, csr
 
+    # pylint: disable=no-member
     def obtain_and_enroll_certificate(self, domains, certname):
         """Obtain and enroll certificate.
 

certbot/colored_logging.py

@@ -1,45 +0,0 @@
-"""A formatter and StreamHandler for colorizing logging output."""
-import logging
-import sys
-
-from certbot import util
-
-
-class StreamHandler(logging.StreamHandler):
-    """Sends colored logging output to a stream.
-
-    If the specified stream is not a tty, the class works like the
-    standard logging.StreamHandler. Default red_level is logging.WARNING.
-
-    :ivar bool colored: True if output should be colored
-    :ivar bool red_level: The level at which to output
-
-    """
-
-    def __init__(self, stream=None):
-        if sys.version_info < (2, 7):
-            # pragma: no cover
-            # pylint: disable=non-parent-init-called
-            logging.StreamHandler.__init__(self, stream)
-        else:
-            super(StreamHandler, self).__init__(stream)
-        self.colored = (sys.stderr.isatty() if stream is None else
-                        stream.isatty())
-        self.red_level = logging.WARNING
-
-    def format(self, record):
-        """Formats the string representation of record.
-
-        :param logging.LogRecord record: Record to be formatted
-
-        :returns: Formatted, string representation of record
-        :rtype: str
-
-        """
-        out = (logging.StreamHandler.format(self, record)
-               if sys.version_info < (2, 7)
-               else super(StreamHandler, self).format(record))
-        if self.colored and record.levelno >= self.red_level:
-            return ''.join((util.ANSI_SGR_RED, out, util.ANSI_SGR_RESET))
-        else:
-            return out

certbot/configuration.py

@@ -42,7 +42,7 @@ class NamespaceConfig(object):
     """
 
     def __init__(self, namespace):
-        self.namespace = namespace
+        object.__setattr__(self, 'namespace', namespace)
 
         self.namespace.config_dir = os.path.abspath(self.namespace.config_dir)
         self.namespace.work_dir = os.path.abspath(self.namespace.work_dir)
@@ -54,6 +54,9 @@ class NamespaceConfig(object):
     def __getattr__(self, name):
         return getattr(self.namespace, name)
 
+    def __setattr__(self, name, value):
+        setattr(self.namespace, name, value)
+
     @property
     def server_path(self):
         """File path based on ``server``."""

certbot/constants.py

@@ -18,6 +18,7 @@ CLI_DEFAULTS = dict(
         os.path.join(os.environ.get("XDG_CONFIG_HOME", "~/.config"),
                      "letsencrypt", "cli.ini"),
     ],
+    dry_run=False,
     verbose_count=-int(logging.INFO / 10),
     server="https://acme-v01.api.letsencrypt.org/directory",
     rsa_key_size=2048,
@@ -34,6 +35,7 @@ CLI_DEFAULTS = dict(
     auth_cert_path="./cert.pem",
     auth_chain_path="./chain.pem",
     strict_permissions=False,
+    debug_challenges=False,
 )
 STAGING_URI = "https://acme-staging.api.letsencrypt.org/directory"
 

certbot/crypto_util.py

@@ -6,7 +6,6 @@
 """
 import logging
 import os
-import traceback
 
 import OpenSSL
 import pyrfc3339
@@ -49,16 +48,20 @@ def save_key(key_pem, key_dir, keyname="key-certbot.pem"):
     # Save file
     util.make_or_verify_dir(key_dir, 0o700, os.geteuid(),
                             config.strict_permissions)
-    key_f, key_path = util.unique_file(
-        os.path.join(key_dir, keyname), 0o600, "wb")
-    with key_f:
-        logger.info("Saving key to: %s", key_path)
-        key_f.write(key_pem)
+    if config.dry_run:
+        key_path = None
+        logger.info("Dry run, not saving private key to file")
+    else:
+        key_f, key_path = util.unique_file(
+            os.path.join(key_dir, keyname), 0o600, "wb")
+        with key_f:
+            key_f.write(key_pem)
+        logger.info("Saving private key: %s", key_path)
 
     return util.Key(key_path, key_pem)
 
 
-def init_save_csr(privkey, names, path, csrname="csr-certbot.pem"):
+def init_save_csr(privkey, names, path):
     """Initialize a CSR with the given private key.
 
     :param privkey: Key to include in the CSR
@@ -74,61 +77,23 @@ def init_save_csr(privkey, names, path, csrname="csr-certbot.pem"):
     """
     config = zope.component.getUtility(interfaces.IConfig)
 
-    csr_pem, csr_der = make_csr(privkey.pem, names,
-        must_staple=config.must_staple)
+    csr_pem = acme_crypto_util.make_csr(
+        privkey.pem, names, must_staple=config.must_staple)
 
     # Save CSR
     util.make_or_verify_dir(path, 0o755, os.geteuid(),
                                config.strict_permissions)
-    csr_f, csr_filename = util.unique_file(
-        os.path.join(path, csrname), 0o644, "wb")
-    csr_f.write(csr_pem)
-    csr_f.close()
+    if config.dry_run:
+        csr_filename = None
+        logger.info("Creating CSR: not saving to file")
+    else:
+        csr_f, csr_filename = util.unique_file(
+            os.path.join(path, "csr-certbot.pem"), 0o644, "wb")
+        with csr_f:
+            csr_f.write(csr_pem)
+        logger.info("Creating CSR: %s", csr_filename)
 
-    logger.info("Creating CSR: %s", csr_filename)
-
-    return util.CSR(csr_filename, csr_der, "der")
-
-
-# Lower level functions
-def make_csr(key_str, domains, must_staple=False):
-    """Generate a CSR.
-
-    :param str key_str: PEM-encoded RSA key.
-    :param list domains: Domains included in the certificate.
-
-    .. todo:: Detect duplicates in `domains`? Using a set doesn't
-              preserve order...
-
-    :returns: new CSR in PEM and DER form containing all domains
-    :rtype: tuple
-
-    """
-    assert domains, "Must provide one or more hostnames for the CSR."
-    pkey = OpenSSL.crypto.load_privatekey(OpenSSL.crypto.FILETYPE_PEM, key_str)
-    req = OpenSSL.crypto.X509Req()
-    req.get_subject().CN = domains[0]
-    # TODO: what to put into req.get_subject()?
-    # TODO: put SAN if len(domains) > 1
-    extensions = [
-        OpenSSL.crypto.X509Extension(
-            b"subjectAltName",
-            critical=False,
-            value=", ".join("DNS:%s" % d for d in domains).encode('ascii')
-        )
-    ]
-    if must_staple:
-        extensions.append(OpenSSL.crypto.X509Extension(
-            b"1.3.6.1.5.5.7.1.24",
-            critical=False,
-            value=b"DER:30:03:02:01:05"))
-    req.add_extensions(extensions)
-    req.set_version(2)
-    req.set_pubkey(pkey)
-    req.sign(pkey, "sha256")
-    return tuple(OpenSSL.crypto.dump_certificate_request(method, req)
-                 for method in (OpenSSL.crypto.FILETYPE_PEM,
-                                OpenSSL.crypto.FILETYPE_ASN1))
+    return util.CSR(csr_filename, csr_pem, "pem")
 
 
 # WARNING: the csr and private key file are possible attack vectors for TOCTOU
@@ -182,22 +147,27 @@ def import_csr_file(csrfile, data):
     :param str csrfile: CSR filename
     :param str data: contents of the CSR file
 
-    :returns: (`OpenSSL.crypto.FILETYPE_PEM` or `OpenSSL.crypto.FILETYPE_ASN1`,
+    :returns: (`OpenSSL.crypto.FILETYPE_PEM`,
                util.CSR object representing the CSR,
                list of domains requested in the CSR)
     :rtype: tuple
 
     """
-    for form, typ in (("der", OpenSSL.crypto.FILETYPE_ASN1,),
-                      ("pem", OpenSSL.crypto.FILETYPE_PEM,),):
+    PEM = OpenSSL.crypto.FILETYPE_PEM
+    load = OpenSSL.crypto.load_certificate_request
+    try:
+        # Try to parse as DER first, then fall back to PEM.
+        csr = load(OpenSSL.crypto.FILETYPE_ASN1, data)
+    except OpenSSL.crypto.Error:
         try:
-            domains = get_names_from_csr(data, typ)
+            csr = load(PEM, data)
         except OpenSSL.crypto.Error:
-            logger.debug("CSR parse error (form=%s, typ=%s):", form, typ)
-            logger.debug(traceback.format_exc())
-            continue
-        return typ, util.CSR(file=csrfile, data=data, form=form), domains
-    raise errors.Error("Failed to parse CSR file: {0}".format(csrfile))
+            raise errors.Error("Failed to parse CSR file: {0}".format(csrfile))
+
+    domains = _get_names_from_loaded_cert_or_req(csr)
+    # Internally we always use PEM, so re-encode as PEM before returning.
+    data_pem = OpenSSL.crypto.dump_certificate_request(PEM, csr)
+    return PEM, util.CSR(file=csrfile, data=data_pem, form="pem"), domains
 
 
 def make_key_rsa(bits):
@@ -303,22 +273,12 @@ def get_sans_from_cert(cert, typ=OpenSSL.crypto.FILETYPE_PEM):
         cert, OpenSSL.crypto.load_certificate, typ)
 
 
-def get_sans_from_csr(csr, typ=OpenSSL.crypto.FILETYPE_PEM):
-    """Get a list of Subject Alternative Names from a CSR.
-
-    :param str csr: CSR (encoded).
-    :param typ: `OpenSSL.crypto.FILETYPE_PEM` or `OpenSSL.crypto.FILETYPE_ASN1`
-
-    :returns: A list of Subject Alternative Names.
-    :rtype: list
-
-    """
-    return _get_sans_from_cert_or_req(
-        csr, OpenSSL.crypto.load_certificate_request, typ)
-
-
 def _get_names_from_cert_or_req(cert_or_req, load_func, typ):
     loaded_cert_or_req = _load_cert_or_req(cert_or_req, load_func, typ)
+    return _get_names_from_loaded_cert_or_req(loaded_cert_or_req)
+
+
+def _get_names_from_loaded_cert_or_req(loaded_cert_or_req):
     common_name = loaded_cert_or_req.get_subject().CN
     # pylint: disable=protected-access
     sans = acme_crypto_util._pyopenssl_cert_or_req_san(loaded_cert_or_req)
@@ -343,20 +303,6 @@ def get_names_from_cert(csr, typ=OpenSSL.crypto.FILETYPE_PEM):
         csr, OpenSSL.crypto.load_certificate, typ)
 
 
-def get_names_from_csr(csr, typ=OpenSSL.crypto.FILETYPE_PEM):
-    """Get a list of domains from a CSR, including the CN if it is set.
-
-    :param str csr: CSR (encoded).
-    :param typ: `OpenSSL.crypto.FILETYPE_PEM` or `OpenSSL.crypto.FILETYPE_ASN1`
-
-    :returns: A list of domain names.
-    :rtype: list
-
-    """
-    return _get_names_from_cert_or_req(
-        csr, OpenSSL.crypto.load_certificate_request, typ)
-
-
 def dump_pyopenssl_chain(chain, filetype=OpenSSL.crypto.FILETYPE_PEM):
     """Dump certificate chain into a bundle.
 

certbot/display/completer.py

@@ -4,7 +4,7 @@ import glob
 try:
     import readline
 except ImportError:
-    import certbot.display.dummy_readline as readline
+    import certbot.display.dummy_readline as readline  # type: ignore
 
 
 class Completer(object):

certbot/display/ops.py

@@ -192,12 +192,7 @@ def _choose_names_manually(prompt_prefix=""):
             try:
                 domain_list[i] = util.enforce_domain_sanity(domain)
             except errors.ConfigurationError as e:
-                try:  # Python 2
-                    # pylint: disable=no-member
-                    err_msg = e.message.encode('utf-8')
-                except AttributeError:
-                    err_msg = str(e)
-                invalid_domains[domain] = err_msg
+                invalid_domains[domain] = str(e)
 
         if len(invalid_domains):
             retry_message = (

certbot/hooks.py

@@ -13,12 +13,14 @@ from certbot.plugins import util as plug_util
 
 logger = logging.getLogger(__name__)
 
+
 def validate_hooks(config):
     """Check hook commands are executable."""
     validate_hook(config.pre_hook, "pre")
     validate_hook(config.post_hook, "post")
     validate_hook(config.renew_hook, "renew")
 
+
 def _prog(shell_cmd):
     """Extract the program run by a shell command.
 
@@ -44,10 +46,15 @@ def validate_hook(shell_cmd, hook_name):
         cmd = shell_cmd.split(None, 1)[0]
         if not _prog(cmd):
             path = os.environ["PATH"]
-            msg = "Unable to find {2}-hook command {0} in the PATH.\n(PATH is {1})".format(
-                cmd, path, hook_name)
+            if os.path.exists(cmd):
+                msg = "{1}-hook command {0} exists, but is not executable.".format(cmd, hook_name)
+            else:
+                msg = "Unable to find {2}-hook command {0} in the PATH.\n(PATH is {1})".format(
+                    cmd, path, hook_name)
+
             raise errors.HookCommandNotFound(msg)
 
+
 def pre_hook(config):
     "Run pre-hook if it's defined and hasn't been run."
     cmd = config.pre_hook
@@ -58,7 +65,7 @@ def pre_hook(config):
     elif cmd:
         logger.info("Pre-hook command already run, skipping: %s", cmd)
 
-pre_hook.already = set()
+pre_hook.already = set()  # type: ignore
 
 
 def post_hook(config):
@@ -78,7 +85,8 @@ def post_hook(config):
         logger.info("Running post-hook command: %s", cmd)
         _run_hook(cmd)
 
-post_hook.eventually = []
+post_hook.eventually = []  # type: ignore
+
 
 def run_saved_post_hooks():
     """Run any post hooks that were saved up in the course of the 'renew' verb"""

certbot/interfaces.py

@@ -32,7 +32,7 @@ class AccountStorage(object):
         raise NotImplementedError()
 
     @abc.abstractmethod
-    def save(self, account):  # pragma: no cover
+    def save(self, account, client):  # pragma: no cover
         """Save account.
 
         :raises .AccountStorageError: if account could not be saved
@@ -99,7 +99,7 @@ class IPluginFactory(zope.interface.Interface):
 class IPlugin(zope.interface.Interface):
     """Certbot plugin."""
 
-    def prepare():
+    def prepare():  # type: ignore
         """Prepare the plugin.
 
         Finish up any additional initialization.
@@ -118,7 +118,7 @@ class IPlugin(zope.interface.Interface):
 
         """
 
-    def more_info():
+    def more_info():  # type: ignore
         """Human-readable string to help the user.
 
         Should describe the steps taken and any relevant info to help the user
@@ -243,6 +243,22 @@ class IConfig(zope.interface.Interface):
         "This only affects the port Certbot listens on. "
         "A conforming ACME server will still attempt to connect on port 80.")
 
+    pref_challs = zope.interface.Attribute(
+        "Sorted user specified preferred challenges"
+        "type strings with the most preferred challenge listed first")
+
+    allow_subset_of_names = zope.interface.Attribute(
+        "When performing domain validation, do not consider it a failure "
+        "if authorizations can not be obtained for a strict subset of "
+        "the requested domains. This may be useful for allowing renewals for "
+        "multiple domains to succeed even if some domains no longer point "
+        "at this system. This is a boolean")
+
+    strict_permissions = zope.interface.Attribute(
+        "Require that all configuration files are owned by the current "
+        "user; only needed if your config is somewhere unsafe like /tmp/."
+        "This is a boolean")
+
 
 class IInstaller(IPlugin):
     """Generic Certbot Installer Interface.
@@ -259,7 +275,7 @@ class IInstaller(IPlugin):
 
     """
 
-    def get_all_names():
+    def get_all_names():  # type: ignore
         """Returns all names that may be authenticated.
 
         :rtype: `collections.Iterable` of `str`
@@ -296,7 +312,7 @@ class IInstaller(IPlugin):
 
         """
 
-    def supported_enhancements():
+    def supported_enhancements():  # type: ignore
         """Returns a `collections.Iterable` of supported enhancements.
 
         :returns: supported enhancements which should be a subset of
@@ -334,7 +350,7 @@ class IInstaller(IPlugin):
 
         """
 
-    def recovery_routine():
+    def recovery_routine():  # type: ignore
         """Revert configuration to most recent finalized checkpoint.
 
         Remove all changes (temporary and permanent) that have not been
@@ -345,21 +361,21 @@ class IInstaller(IPlugin):
 
         """
 
-    def view_config_changes():
+    def view_config_changes():  # type: ignore
         """Display all of the LE config changes.
 
         :raises .PluginError: when config changes cannot be parsed
 
         """
 
-    def config_test():
+    def config_test():  # type: ignore
         """Make sure the configuration is valid.
 
         :raises .MisconfigurationError: when the config is not in a usable state
 
         """
 
-    def restart():
+    def restart():  # type: ignore
         """Restart or refresh the server content.
 
         :raises .PluginError: when server cannot be restarted

certbot/log.py

@@ -0,0 +1,321 @@
+"""Logging utilities for Certbot.
+
+The best way to use this module is through `pre_arg_parse_setup` and
+`post_arg_parse_setup`. `pre_arg_parse_setup` configures a minimal
+terminal logger and ensures a detailed log is written to a secure
+temporary file if Certbot exits before `post_arg_parse_setup` is called.
+`post_arg_parse_setup` relies on the parsed command line arguments and
+does the full logging setup with terminal and rotating file handling as
+configured by the user. Any logged messages before
+`post_arg_parse_setup` is called are sent to the rotating file handler.
+Special care is taken by both methods to ensure all errors are logged
+and properly flushed before program exit.
+
+"""
+from __future__ import print_function
+import functools
+import logging
+import logging.handlers
+import os
+import sys
+import tempfile
+import time
+import traceback
+
+from acme import messages
+
+from certbot import constants
+from certbot import errors
+from certbot import util
+
+# Logging format
+CLI_FMT = "%(message)s"
+FILE_FMT = "%(asctime)s:%(levelname)s:%(name)s:%(message)s"
+
+
+logger = logging.getLogger(__name__)
+
+
+def pre_arg_parse_setup():
+    """Setup logging before command line arguments are parsed.
+
+    Terminal logging is setup using
+    `certbot.constants.QUIET_LOGGING_LEVEL` so Certbot is as quiet as
+    possible. File logging is setup so that logging messages are
+    buffered in memory. If Certbot exits before `post_arg_parse_setup`
+    is called, these buffered messages are written to a temporary file.
+    If Certbot doesn't exit, `post_arg_parse_setup` writes the messages
+    to the normal log files.
+
+    This function also sets `logging.shutdown` to be called on program
+    exit which automatically flushes logging handlers and
+    `sys.excepthook` to properly log/display fatal exceptions.
+
+    """
+    temp_handler = TempHandler()
+    temp_handler.setFormatter(logging.Formatter(FILE_FMT))
+    temp_handler.setLevel(logging.DEBUG)
+    memory_handler = MemoryHandler(temp_handler)
+
+    stream_handler = ColoredStreamHandler()
+    stream_handler.setFormatter(logging.Formatter(CLI_FMT))
+    stream_handler.setLevel(constants.QUIET_LOGGING_LEVEL)
+
+    root_logger = logging.getLogger()
+    root_logger.setLevel(logging.DEBUG)  # send all records to handlers
+    root_logger.addHandler(memory_handler)
+    root_logger.addHandler(stream_handler)
+
+    # logging.shutdown will flush the memory handler because flush() and
+    # close() are explicitly called
+    util.atexit_register(logging.shutdown)
+    sys.excepthook = functools.partial(
+        except_hook, debug='--debug' in sys.argv, log_path=temp_handler.path)
+
+
+def post_arg_parse_setup(config):
+    """Setup logging after command line arguments are parsed.
+
+    This function assumes `pre_arg_parse_setup` was called earlier and
+    the root logging configuration has not been modified. A rotating
+    file logging handler is created and the buffered log messages are
+    sent to that handler. Terminal logging output is set to the level
+    requested by the user.
+
+    :param certbot.interface.IConfig config: Configuration object
+
+    """
+    file_handler, file_path = setup_log_file_handler(
+        config, 'letsencrypt.log', FILE_FMT)
+    logs_dir = os.path.dirname(file_path)
+
+    root_logger = logging.getLogger()
+    memory_handler = stderr_handler = None
+    for handler in root_logger.handlers:
+        if isinstance(handler, ColoredStreamHandler):
+            stderr_handler = handler
+        elif isinstance(handler, MemoryHandler):
+            memory_handler = handler
+    msg = 'Previously configured logging handlers have been removed!'
+    assert memory_handler is not None and stderr_handler is not None, msg
+
+    root_logger.addHandler(file_handler)
+    root_logger.removeHandler(memory_handler)
+    temp_handler = memory_handler.target
+    memory_handler.setTarget(file_handler)
+    memory_handler.close()
+    temp_handler.delete_and_close()
+
+    if config.quiet:
+        level = constants.QUIET_LOGGING_LEVEL
+    else:
+        level = -config.verbose_count * 10
+    stderr_handler.setLevel(level)
+    logger.debug('Root logging level set at %d', level)
+    logger.info('Saving debug log to %s', file_path)
+
+    sys.excepthook = functools.partial(
+        except_hook, debug=config.debug, log_path=logs_dir)
+
+
+def setup_log_file_handler(config, logfile, fmt):
+    """Setup file debug logging.
+
+    :param certbot.interface.IConfig config: Configuration object
+    :param str logfile: basename for the log file
+    :param str fmt: logging format string
+
+    :returns: file handler and absolute path to the log file
+    :rtype: tuple
+
+    """
+    # TODO: logs might contain sensitive data such as contents of the
+    # private key! #525
+    util.make_or_verify_core_dir(
+        config.logs_dir, 0o700, os.geteuid(), config.strict_permissions)
+    log_file_path = os.path.join(config.logs_dir, logfile)
+    try:
+        handler = logging.handlers.RotatingFileHandler(
+            log_file_path, maxBytes=2 ** 20, backupCount=1000)
+    except IOError as error:
+        raise errors.Error(util.PERM_ERR_FMT.format(error))
+    # rotate on each invocation, rollover only possible when maxBytes
+    # is nonzero and backupCount is nonzero, so we set maxBytes as big
+    # as possible not to overrun in single CLI invocation (1MB).
+    handler.doRollover()  # TODO: creates empty letsencrypt.log.1 file
+    handler.setLevel(logging.DEBUG)
+    handler_formatter = logging.Formatter(fmt=fmt)
+    handler_formatter.converter = time.gmtime  # don't use localtime
+    handler.setFormatter(handler_formatter)
+    return handler, log_file_path
+
+
+class ColoredStreamHandler(logging.StreamHandler):
+    """Sends colored logging output to a stream.
+
+    If the specified stream is not a tty, the class works like the
+    standard `logging.StreamHandler`. Default red_level is
+    `logging.WARNING`.
+
+    :ivar bool colored: True if output should be colored
+    :ivar bool red_level: The level at which to output
+
+    """
+    def __init__(self, stream=None):
+        # logging handlers use old style classes in Python 2.6 so
+        # super() cannot be used
+        if sys.version_info < (2, 7):  # pragma: no cover
+            logging.StreamHandler.__init__(self, stream)
+        else:
+            super(ColoredStreamHandler, self).__init__(stream)
+        self.colored = (sys.stderr.isatty() if stream is None else
+                        stream.isatty())
+        self.red_level = logging.WARNING
+
+    def format(self, record):
+        """Formats the string representation of record.
+
+        :param logging.LogRecord record: Record to be formatted
+
+        :returns: Formatted, string representation of record
+        :rtype: str
+
+        """
+        out = (logging.StreamHandler.format(self, record)
+               if sys.version_info < (2, 7)
+               else super(ColoredStreamHandler, self).format(record))
+        if self.colored and record.levelno >= self.red_level:
+            return ''.join((util.ANSI_SGR_RED, out, util.ANSI_SGR_RESET))
+        else:
+            return out
+
+
+class MemoryHandler(logging.handlers.MemoryHandler):
+    """Buffers logging messages in memory until the buffer is flushed.
+
+    This differs from `logging.handlers.MemoryHandler` in that flushing
+    only happens when it is done explicitly by calling flush() or
+    close().
+
+    """
+    def __init__(self, target=None):
+        # capacity doesn't matter because should_flush() is overridden
+        capacity = float('inf')
+        # logging handlers use old style classes in Python 2.6 so
+        # super() cannot be used
+        if sys.version_info < (2, 7):  # pragma: no cover
+            logging.handlers.MemoryHandler.__init__(
+                self, capacity, target=target)
+        else:
+            super(MemoryHandler, self).__init__(capacity, target=target)
+
+    def shouldFlush(self, record):
+        """Should the buffer be automatically flushed?
+
+        :param logging.LogRecord record: log record to be considered
+
+        :returns: False because the buffer should never be auto-flushed
+        :rtype: bool
+
+        """
+        return False
+
+
+class TempHandler(logging.StreamHandler):
+    """Safely logs messages to a temporary file.
+
+    The file is created with permissions 600.
+
+    :ivar str path: file system path to the temporary log file
+
+    """
+    def __init__(self):
+        stream = tempfile.NamedTemporaryFile('w', delete=False)
+        # logging handlers use old style classes in Python 2.6 so
+        # super() cannot be used
+        if sys.version_info < (2, 7):  # pragma: no cover
+            logging.StreamHandler.__init__(self, stream)
+        else:
+            super(TempHandler, self).__init__(stream)
+        self.path = stream.name
+
+    def delete_and_close(self):
+        """Close the handler and delete the temporary log file."""
+        self._close(delete=True)
+
+    def close(self):
+        """Close the handler and the temporary log file."""
+        self._close(delete=False)
+
+    def _close(self, delete):
+        """Close the handler and the temporary log file.
+
+        :param bool delete: True if the log file should be deleted
+
+        """
+        self.acquire()
+        try:
+            # StreamHandler.close() doesn't close the stream to allow a
+            # stream like stderr to be used
+            self.stream.close()
+            if delete:
+                os.remove(self.path)
+            if sys.version_info < (2, 7):  # pragma: no cover
+                logging.StreamHandler.close(self)
+            else:
+                super(TempHandler, self).close()
+        finally:
+            self.release()
+
+
+def except_hook(exc_type, exc_value, trace, debug, log_path):
+    """Logs fatal exceptions and reports them to the user.
+
+    If debug is True, the full exception and traceback is shown to the
+    user, otherwise, it is suppressed. sys.exit is always called with a
+    nonzero status.
+
+    :param type exc_type: type of the raised exception
+    :param BaseException exc_value: raised exception
+    :param traceback trace: traceback of where the exception was raised
+    :param bool debug: True if the traceback should be shown to the user
+    :param str log_path: path to file or directory containing the log
+
+    """
+    exc_info = (exc_type, exc_value, trace)
+    # constants.QUIET_LOGGING_LEVEL or higher should be used to
+    # display message the user, otherwise, a lower level like
+    # logger.DEBUG should be used
+    if debug or not issubclass(exc_type, Exception):
+        assert constants.QUIET_LOGGING_LEVEL <= logging.ERROR
+        logger.error('Exiting abnormally:', exc_info=exc_info)
+    else:
+        logger.debug('Exiting abnormally:', exc_info=exc_info)
+        if issubclass(exc_type, errors.Error):
+            sys.exit(exc_value)
+        print('An unexpected error occurred:', file=sys.stderr)
+        if messages.is_acme_error(exc_value):
+            # Remove the ACME error prefix from the exception
+            _, _, exc_str = str(exc_value).partition(':: ')
+            print(exc_str, file=sys.stderr)
+        else:
+            traceback.print_exception(exc_type, exc_value, None)
+    exit_with_log_path(log_path)
+
+
+def exit_with_log_path(log_path):
+    """Print a message about the log location and exit.
+
+    The message is printed to stderr and the program will exit with a
+    nonzero status.
+
+    :param str log_path: path to file or directory containing the log
+
+    """
+    msg = 'Please see the '
+    if os.path.isdir(log_path):
+        msg += 'logfiles in {0} '.format(log_path)
+    else:
+        msg += "logfile '{0}' ".format(log_path)
+    msg += 'for more details.'
+    sys.exit(msg)

certbot/main.py

@@ -1,47 +1,37 @@
 """Certbot main entry point."""
 from __future__ import print_function
-import atexit
-import functools
 import logging.handlers
 import os
 import sys
-import time
-import traceback
 
 import zope.component
 
 from acme import jose
-from acme import messages
 from acme import errors as acme_errors
 
 import certbot
 
 from certbot import account
 from certbot import cert_manager
-from certbot import client
 from certbot import cli
-from certbot import crypto_util
-from certbot import colored_logging
+from certbot import client
 from certbot import configuration
 from certbot import constants
+from certbot import crypto_util
 from certbot import eff
 from certbot import errors
 from certbot import hooks
 from certbot import interfaces
-from certbot import util
-from certbot import reporter
+from certbot import log
 from certbot import renewal
+from certbot import reporter
+from certbot import util
 
 from certbot.display import util as display_util, ops as display_ops
 from certbot.plugins import disco as plugins_disco
 from certbot.plugins import selection as plug_sel
 
 
-_PERM_ERR_FMT = os.linesep.join((
-    "The following error was encountered:", "{0}",
-    "If running as non-root, set --config-dir, "
-    "--work-dir, and --logs-dir to writeable paths."))
-
 USER_CANCELLED = ("User chose to cancel the operation and may "
                   "reinvoke the client.")
 
@@ -359,7 +349,7 @@ def _determine_account(config):
             acc = accounts[0]
         else:  # no account registered yet
             if config.email is None and not config.register_unsafely_without_email:
-                config.namespace.email = display_ops.get_email()
+                config.email = display_ops.get_email()
 
             def _tos_cb(regr):
                 if config.tos:
@@ -382,7 +372,7 @@ def _determine_account(config):
                 raise errors.Error(
                     "Unable to register an account with ACME server")
 
-    config.namespace.account = acc.id
+    config.account = acc.id
     return acc, acme
 
 
@@ -417,10 +407,10 @@ def unregister(config, unused_plugins):
         return "Deactivation aborted."
 
     acc, acme = _determine_account(config)
-    acme_client = client.Client(config, acc, None, None, acme=acme)
+    cb_client = client.Client(config, acc, None, None, acme=acme)
 
     # delete on boulder
-    acme_client.acme.deactivate_registration(acc.regr)
+    cb_client.acme.deactivate_registration(acc.regr)
     account_files = account.AccountFileStorage(config)
     # delete local account files
     account_files.delete(config.account)
@@ -459,14 +449,14 @@ def register(config, unused_plugins):
             return ("--register-unsafely-without-email provided, however, a "
                     "new e-mail address must\ncurrently be provided when "
                     "updating a registration.")
-        config.namespace.email = display_ops.get_email(optional=False)
+        config.email = display_ops.get_email(optional=False)
 
     acc, acme = _determine_account(config)
-    acme_client = client.Client(config, acc, None, None, acme=acme)
+    cb_client = client.Client(config, acc, None, None, acme=acme)
     # We rely on an exception to interrupt this process if it didn't work.
-    acc.regr = acme_client.acme.update_registration(acc.regr.update(
+    acc.regr = cb_client.acme.update_registration(acc.regr.update(
         body=acc.regr.body.update(contact=('mailto:' + config.email,))))
-    account_storage.save_regr(acc)
+    account_storage.save_regr(acc, cb_client.acme)
     eff.handle_subscription(config)
     add_msg("Your e-mail address was updated to {0}.".format(config.email))
 
@@ -487,7 +477,7 @@ def install(config, plugins):
     try:
         installer, _ = plug_sel.choose_configurator_plugins(config, plugins, "install")
     except errors.PluginSelectionError as e:
-        return e.message
+        return str(e)
 
     domains, _ = _find_domains_or_certname(config, installer)
     le_client = _init_le_client(config, authenticator=None, installer=installer)
@@ -565,7 +555,7 @@ def certificates(config, unused_plugins):
 def revoke(config, unused_plugins):  # TODO: coop with renewal config
     """Revoke a previously obtained certificate."""
     # For user-agent construction
-    config.namespace.installer = config.namespace.authenticator = "None"
+    config.installer = config.authenticator = "None"
     if config.key_path is not None:  # revocation by cert key
         logger.debug("Revoking %s using cert key %s",
                      config.cert_path[0], config.key_path[0])
@@ -581,7 +571,7 @@ def revoke(config, unused_plugins):  # TODO: coop with renewal config
     try:
         acme.revoke(jose.ComparableX509(cert), config.reason)
     except acme_errors.ClientError as e:
-        return e.message
+        return str(e)
 
     display_ops.success_revocation(config.cert_path[0])
 
@@ -593,7 +583,7 @@ def run(config, plugins):  # pylint: disable=too-many-branches,too-many-locals
     try:
         installer, authenticator = plug_sel.choose_configurator_plugins(config, plugins, "run")
     except errors.PluginSelectionError as e:
-        return e.message
+        return str(e)
 
     # TODO: Handle errors from _init_le_client?
     le_client = _init_le_client(config, authenticator, installer)
@@ -627,8 +617,8 @@ def _csr_get_and_save_cert(config, le_client):
     have the privkey, and therefore can't construct the files for a lineage.
     So we just save the cert & chain to disk :/
     """
-    csr, typ = config.actual_csr
-    certr, chain = le_client.obtain_certificate_from_csr(config.domains, csr, typ)
+    csr, _ = config.actual_csr
+    certr, chain = le_client.obtain_certificate_from_csr(config.domains, csr)
     if config.dry_run:
         logger.debug(
             "Dry run: skipping saving certificate to %s", config.cert_path)
@@ -704,138 +694,11 @@ def renew(config, unused_plugins):
         hooks.run_saved_post_hooks()
 
 
-def setup_log_file_handler(config, logfile, fmt):
-    """Setup file debug logging."""
-    log_file_path = os.path.join(config.logs_dir, logfile)
-    try:
-        handler = logging.handlers.RotatingFileHandler(
-            log_file_path, maxBytes=2 ** 20, backupCount=1000)
-    except IOError as error:
-        raise errors.Error(_PERM_ERR_FMT.format(error))
-    # rotate on each invocation, rollover only possible when maxBytes
-    # is nonzero and backupCount is nonzero, so we set maxBytes as big
-    # as possible not to overrun in single CLI invocation (1MB).
-    handler.doRollover()  # TODO: creates empty letsencrypt.log.1 file
-    handler.setLevel(logging.DEBUG)
-    handler_formatter = logging.Formatter(fmt=fmt)
-    handler_formatter.converter = time.gmtime  # don't use localtime
-    handler.setFormatter(handler_formatter)
-    return handler, log_file_path
-
-
-def _cli_log_handler(level, fmt):
-    handler = colored_logging.StreamHandler()
-    handler.setFormatter(logging.Formatter(fmt))
-    handler.setLevel(level)
-    return handler
-
-
-def setup_logging(config):
-    """Sets up logging to logfiles and the terminal.
-
-    :param certbot.interface.IConfig config: Configuration object
-
-    """
-    cli_fmt = "%(message)s"
-    file_fmt = "%(asctime)s:%(levelname)s:%(name)s:%(message)s"
-    logfile = "letsencrypt.log"
-    if config.quiet:
-        level = constants.QUIET_LOGGING_LEVEL
-    else:
-        level = -config.verbose_count * 10
-    file_handler, log_file_path = setup_log_file_handler(
-        config, logfile=logfile, fmt=file_fmt)
-    cli_handler = _cli_log_handler(level, cli_fmt)
-
-    # TODO: use fileConfig?
-
-    root_logger = logging.getLogger()
-    root_logger.setLevel(logging.DEBUG)  # send all records to handlers
-    root_logger.addHandler(cli_handler)
-    root_logger.addHandler(file_handler)
-
-    logger.debug("Root logging level set at %d", level)
-    logger.info("Saving debug log to %s", log_file_path)
-
-
-def _handle_exception(exc_type, exc_value, trace, config):
-    """Logs exceptions and reports them to the user.
-
-    Config is used to determine how to display exceptions to the user. In
-    general, if config.debug is True, then the full exception and traceback is
-    shown to the user, otherwise it is suppressed. If config itself is None,
-    then the traceback and exception is attempted to be written to a logfile.
-    If this is successful, the traceback is suppressed, otherwise it is shown
-    to the user. sys.exit is always called with a nonzero status.
-
-    """
-    tb_str = "".join(traceback.format_exception(exc_type, exc_value, trace))
-    logger.debug("Exiting abnormally:%s%s", os.linesep, tb_str)
-
-    if issubclass(exc_type, Exception) and (config is None or not config.debug):
-        if config is None:
-            logfile = "certbot.log"
-            try:
-                with open(logfile, "w") as logfd:
-                    traceback.print_exception(
-                        exc_type, exc_value, trace, file=logfd)
-                assert "--debug" not in sys.argv  # config is None if this explodes
-            except:  # pylint: disable=bare-except
-                sys.exit(tb_str)
-            if "--debug" in sys.argv:
-                sys.exit(tb_str)
-
-        if issubclass(exc_type, errors.Error):
-            sys.exit(exc_value)
-        else:
-            # Here we're passing a client or ACME error out to the client at the shell
-            # Tell the user a bit about what happened, without overwhelming
-            # them with a full traceback
-            err = traceback.format_exception_only(exc_type, exc_value)[0]
-            # Typical error from the ACME module:
-            # acme.messages.Error: urn:ietf:params:acme:error:malformed :: The
-            # request message was malformed :: Error creating new registration
-            # :: Validation of contact mailto:none@longrandomstring.biz failed:
-            # Server failure at resolver
-            if (messages.is_acme_error(err) and ":: " in err and
-                 config.verbose_count <= cli.flag_default("verbose_count")):
-                # prune ACME error code, we have a human description
-                _code, _sep, err = err.partition(":: ")
-            msg = "An unexpected error occurred:\n" + err + "Please see the "
-            if config is None:
-                msg += "logfile '{0}' for more details.".format(logfile)
-            else:
-                msg += "logfiles in {0} for more details.".format(config.logs_dir)
-            sys.exit(msg)
-    else:
-        sys.exit(tb_str)
-
-
-def make_or_verify_core_dir(directory, mode, uid, strict):
-    """Make sure directory exists with proper permissions.
-
-    :param str directory: Path to a directory.
-    :param int mode: Directory mode.
-    :param int uid: Directory owner.
-    :param bool strict: require directory to be owned by current user
-
-    :raises .errors.Error: if the directory cannot be made or verified
-
-    """
-    try:
-        util.make_or_verify_dir(directory, mode, uid, strict)
-    except OSError as error:
-        raise errors.Error(_PERM_ERR_FMT.format(error))
-
 def make_or_verify_needed_dirs(config):
-    """Create or verify existence of config, work, or logs directories"""
-    make_or_verify_core_dir(config.config_dir, constants.CONFIG_DIRS_MODE,
+    """Create or verify existence of config and work directories"""
+    util.make_or_verify_core_dir(config.config_dir, constants.CONFIG_DIRS_MODE,
                             os.geteuid(), config.strict_permissions)
-    make_or_verify_core_dir(config.work_dir, constants.CONFIG_DIRS_MODE,
-                            os.geteuid(), config.strict_permissions)
-    # TODO: logs might contain sensitive data such as contents of the
-    # private key! #525
-    make_or_verify_core_dir(config.logs_dir, 0o700,
+    util.make_or_verify_core_dir(config.work_dir, constants.CONFIG_DIRS_MODE,
                             os.geteuid(), config.strict_permissions)
 
 
@@ -851,47 +714,30 @@ def set_displayer(config):
                                              config.force_interactive)
     zope.component.provideUtility(displayer)
 
-def _post_logging_setup(config, plugins, cli_args):
-    """Perform any setup or configuration tasks that require a logger."""
 
-    # This needs logging, but would otherwise be in HelpfulArgumentParser
-    if config.validate_hooks:
-        hooks.validate_hooks(config)
-
-    cli.possible_deprecation_warning(config)
+def main(cli_args=sys.argv[1:]):
+    """Command line argument parsing and main script execution."""
+    log.pre_arg_parse_setup()
 
+    plugins = plugins_disco.PluginsRegistry.find_all()
     logger.debug("certbot version: %s", certbot.__version__)
     # do not log `config`, as it contains sensitive data (e.g. revoke --key)!
     logger.debug("Arguments: %r", cli_args)
     logger.debug("Discovered plugins: %r", plugins)
 
-
-def main(cli_args=sys.argv[1:]):
-    """Command line argument parsing and main script execution."""
-    sys.excepthook = functools.partial(_handle_exception, config=None)
-    plugins = plugins_disco.PluginsRegistry.find_all()
-
     # note: arg parser internally handles --help (and exits afterwards)
     args = cli.prepare_and_parse_args(plugins, cli_args)
     config = configuration.NamespaceConfig(args)
     zope.component.provideUtility(config)
 
+    log.post_arg_parse_setup(config)
     make_or_verify_needed_dirs(config)
-
-    # Setup logging ASAP, otherwise "No handlers could be found for
-    # logger ..." TODO: this should be done before plugins discovery
-    setup_logging(config)
-
-    _post_logging_setup(config, plugins, cli_args)
-
-    sys.excepthook = functools.partial(_handle_exception, config=config)
-
     set_displayer(config)
 
     # Reporter
     report = reporter.Reporter(config)
     zope.component.provideUtility(report)
-    atexit.register(report.atexit_print_messages)
+    util.atexit_register(report.print_messages)
 
     return config.func(config, plugins)
 

certbot/ocsp.py

@@ -16,7 +16,7 @@ class RevocationChecker(object):
         self.broken = False
 
         if not util.exe_exists("openssl"):
-            logging.info("openssl not installed, can't check revocation")
+            logger.info("openssl not installed, can't check revocation")
             self.broken = True
             return
 
@@ -61,7 +61,7 @@ class RevocationChecker(object):
         logger.debug("Querying OCSP for %s", cert_path)
         logger.debug(" ".join(cmd))
         try:
-            output, err = util.run_script(cmd, log=logging.debug)
+            output, err = util.run_script(cmd, log=logger.debug)
         except errors.SubprocessError:
             logger.info("OCSP check failed for %s (are we offline?)", cert_path)
             return False
@@ -80,7 +80,7 @@ class RevocationChecker(object):
         try:
             url, _err = util.run_script(
                 ["openssl", "x509", "-in", cert_path, "-noout", "-ocsp_uri"],
-                log=logging.debug)
+                log=logger.debug)
         except errors.SubprocessError:
             logger.info("Cannot extract OCSP URI from %s", cert_path)
             return None, None

certbot/plugins/disco.py

@@ -27,7 +27,7 @@ class PluginEntryPoint(object):
     """Distributions for which prefix will be omitted."""
 
     # this object is mutable, don't allow it to be hashed!
-    __hash__ = None
+    __hash__ = None  # type: ignore
 
     def __init__(self, entry_point):
         self.name = self.entry_point_to_plugin_name(entry_point)

certbot/plugins/selection.py

@@ -137,9 +137,8 @@ noninstaller_plugins = ["webroot", "manual", "standalone"]
 
 def record_chosen_plugins(config, plugins, auth, inst):
     "Update the config entries to reflect the plugins we actually selected."
-    cn = config.namespace
-    cn.authenticator = plugins.find_init(auth).name if auth else "None"
-    cn.installer = plugins.find_init(inst).name if inst else "None"
+    config.authenticator = plugins.find_init(auth).name if auth else "None"
+    config.installer = plugins.find_init(inst).name if inst else "None"
 
 
 def choose_configurator_plugins(config, plugins, verb):

certbot/plugins/util.py

@@ -33,6 +33,6 @@ def path_surgery(cmd):
         return True
     else:
         expanded = " expanded" if any(added) else ""
-        logger.warning("Failed to find %s in%s PATH: %s", cmd,
+        logger.warning("Failed to find executable %s in%s PATH: %s", cmd,
                        expanded, path)
         return False

certbot/renewal.py

@@ -80,7 +80,7 @@ def _reconstitute(config, full_path):
     except (ValueError, errors.Error) as error:
         logger.warning(
             "An error occurred while parsing %s. The error was %s. "
-            "Skipping the file.", full_path, error.message)
+            "Skipping the file.", full_path, str(error))
         logger.debug("Traceback was:\n%s", traceback.format_exc())
         return None
 
@@ -104,13 +104,13 @@ def _restore_webroot_config(config, renewalparams):
     """
     if "webroot_map" in renewalparams:
         if not cli.set_by_cli("webroot_map"):
-            config.namespace.webroot_map = renewalparams["webroot_map"]
+            config.webroot_map = renewalparams["webroot_map"]
     elif "webroot_path" in renewalparams:
         logger.debug("Ancient renewal conf file without webroot-map, restoring webroot-path")
         wp = renewalparams["webroot_path"]
         if isinstance(wp, str):  # prior to 0.1.0, webroot_path was a string
             wp = [wp]
-        config.namespace.webroot_path = wp
+        config.webroot_path = wp
 
 
 def _restore_plugin_configs(config, renewalparams):
@@ -149,10 +149,10 @@ def _restore_plugin_configs(config, renewalparams):
                 if config_value in ("None", "True", "False"):
                     # bool("False") == True
                     # pylint: disable=eval-used
-                    setattr(config.namespace, config_item, eval(config_value))
+                    setattr(config, config_item, eval(config_value))
                 else:
                     cast = cli.argparse_type(config_item)
-                    setattr(config.namespace, config_item, cast(config_value))
+                    setattr(config, config_item, cast(config_value))
 
 
 def restore_required_config_elements(config, renewalparams):
@@ -173,7 +173,7 @@ def restore_required_config_elements(config, renewalparams):
     for item_name, restore_func in required_items:
         if item_name in renewalparams and not cli.set_by_cli(item_name):
             value = restore_func(item_name, renewalparams[item_name])
-            setattr(config.namespace, item_name, value)
+            setattr(config, item_name, value)
 
 
 def _restore_pref_challs(unused_name, value):

certbot/reporter.py

@@ -3,11 +3,10 @@ from __future__ import print_function
 
 import collections
 import logging
-import os
 import sys
 import textwrap
 
-from six.moves import queue  # pylint: disable=import-error
+from six.moves import queue  # type: ignore  # pylint: disable=import-error
 import zope.interface
 
 from certbot import interfaces
@@ -16,11 +15,6 @@ from certbot import util
 
 logger = logging.getLogger(__name__)
 
-# Store the pid of the process that first imported this module so that
-# atexit_print_messages side-effects such as error reporting can be limited to
-# this process and not any fork()'d children.
-INITIAL_PID = os.getpid()
-
 
 @zope.interface.implementer(interfaces.IReporter)
 class Reporter(object):
@@ -60,19 +54,6 @@ class Reporter(object):
         self.messages.put(self._msg_type(priority, msg, on_crash))
         logger.debug("Reporting to user: %s", msg)
 
-    def atexit_print_messages(self, pid=None):
-        """Function to be registered with atexit to print messages.
-
-        :param int pid: Process ID
-
-        """
-        if pid is None:
-            pid = INITIAL_PID
-        # This ensures that messages are only printed from the process that
-        # created the Reporter.
-        if pid == os.getpid():
-            self.print_messages()
-
     def print_messages(self):
         """Prints messages to the user and clears the message queue.
 

certbot/storage.py

@@ -19,6 +19,9 @@ from certbot import errors
 from certbot import error_handler
 from certbot import util
 
+from certbot.plugins import common as plugins_common
+from certbot.plugins import disco as plugins_disco
+
 logger = logging.getLogger(__name__)
 
 ALL_FOUR = ("cert", "privkey", "chain", "fullchain")
@@ -27,7 +30,14 @@ CURRENT_VERSION = util.get_strict_version(certbot.__version__)
 
 
 def renewal_conf_files(config):
-    """Return /path/to/*.conf in the renewal conf directory"""
+    """Build a list of all renewal configuration files.
+
+    :param certbot.interfaces.IConfig config: Configuration object
+
+    :returns: list of renewal configuration files
+    :rtype: `list` of `str`
+
+    """
     return glob.glob(os.path.join(config.renewal_configs_dir, "*.conf"))
 
 def renewal_file_for_certname(config, certname):
@@ -179,13 +189,12 @@ def _relevant(option):
 
     :rtype: bool
     """
-    # The list() here produces a list of the plugin names as strings.
     from certbot import renewal
-    from certbot.plugins import disco as plugins_disco
-    plugins = list(plugins_disco.PluginsRegistry.find_all())
+    plugins = plugins_disco.PluginsRegistry.find_all()
+    namespaces = [plugins_common.dest_namespace(plugin) for plugin in plugins]
 
     return (option in renewal.CONFIG_ITEMS or
-            any(option.startswith(x + "_") for x in plugins))
+            any(option.startswith(namespace) for namespace in namespaces))
 
 
 def relevant_values(all_values):
@@ -1038,10 +1047,10 @@ class RenewableCert(object):
             is regarded as a successor (used to choose a privkey, if the
             key has not changed, but otherwise this information is not
             permanently recorded anywhere)
-        :param str new_cert: the new certificate, in PEM format
-        :param str new_privkey: the new private key, in PEM format,
+        :param bytes new_cert: the new certificate, in PEM format
+        :param bytes new_privkey: the new private key, in PEM format,
             or ``None``, if the private key has not changed
-        :param str new_chain: the new chain, in PEM format
+        :param bytes new_chain: the new chain, in PEM format
         :param .NamespaceConfig cli_config: parsed command line
             arguments
 
@@ -1077,18 +1086,18 @@ class RenewableCert(object):
             logger.debug("Writing symlink to old private key, %s.", old_privkey)
             os.symlink(old_privkey, target["privkey"])
         else:
-            with open(target["privkey"], "w") as f:
+            with open(target["privkey"], "wb") as f:
                 logger.debug("Writing new private key to %s.", target["privkey"])
                 f.write(new_privkey)
 
         # Save everything else
-        with open(target["cert"], "w") as f:
+        with open(target["cert"], "wb") as f:
             logger.debug("Writing certificate to %s.", target["cert"])
             f.write(new_cert)
-        with open(target["chain"], "w") as f:
+        with open(target["chain"], "wb") as f:
             logger.debug("Writing chain to %s.", target["chain"])
             f.write(new_chain)
-        with open(target["fullchain"], "w") as f:
+        with open(target["fullchain"], "wb") as f:
             logger.debug("Writing full chain to %s.", target["fullchain"])
             f.write(new_cert + new_chain)
 

certbot/tests/account_test.py

@@ -1,9 +1,9 @@
 """Tests for certbot.account."""
 import datetime
+import json
 import os
 import shutil
 import stat
-import tempfile
 import unittest
 
 import mock
@@ -16,6 +16,8 @@ from certbot import errors
 
 from certbot.tests import util
 
+from certbot.tests.util import TempDirTestCase
+
 
 KEY = jose.JWKRSA.load(util.load_vector("rsa512_key_2.pem"))
 
@@ -31,6 +33,7 @@ class AccountTest(unittest.TestCase):
             creation_dt=datetime.datetime(
                 2015, 7, 4, 14, 4, 10, tzinfo=pytz.UTC))
         self.acc = Account(self.regr, KEY, self.meta)
+        self.regr.__repr__ = mock.MagicMock(return_value="i_am_a_regr")
 
         with mock.patch("certbot.account.socket") as mock_socket:
             mock_socket.getfqdn.return_value = "test.certbot.org"
@@ -52,10 +55,8 @@ class AccountTest(unittest.TestCase):
             self.acc.slug, "test.certbot.org@2015-07-04T14:04:10Z (bca5)")
 
     def test_repr(self):
-        self.assertEqual(
-            repr(self.acc),
-            "<Account(bca5889f66457d5b62fbba7b25f9ab6f)>")
-
+        self.assertTrue(repr(self.acc).startswith(
+          "<Account(i_am_a_regr, bca5889f66457d5b62fbba7b25f9ab6f, Meta("))
 
 class ReportNewAccountTest(unittest.TestCase):
     """Tests for certbot.account.report_new_account."""
@@ -90,37 +91,39 @@ class AccountMemoryStorageTest(unittest.TestCase):
         account = mock.Mock(id="x")
         self.assertEqual([], self.storage.find_all())
         self.assertRaises(errors.AccountNotFound, self.storage.load, "x")
-        self.storage.save(account)
+        self.storage.save(account, None)
         self.assertEqual([account], self.storage.find_all())
         self.assertEqual(account, self.storage.load("x"))
-        self.storage.save(account)
+        self.storage.save(account, None)
         self.assertEqual([account], self.storage.find_all())
 
 
-class AccountFileStorageTest(unittest.TestCase):
+class AccountFileStorageTest(TempDirTestCase):
     """Tests for certbot.account.AccountFileStorage."""
 
     def setUp(self):
-        self.tmp = tempfile.mkdtemp()
+        super(AccountFileStorageTest, self).setUp()
+
         self.config = mock.MagicMock(
-            accounts_dir=os.path.join(self.tmp, "accounts"))
+            accounts_dir=os.path.join(self.tempdir, "accounts"))
         from certbot.account import AccountFileStorage
         self.storage = AccountFileStorage(self.config)
 
         from certbot.account import Account
+        new_authzr_uri = "hi"
         self.acc = Account(
             regr=messages.RegistrationResource(
-                uri=None, body=messages.Registration()),
+                uri=None, body=messages.Registration(),
+                new_authzr_uri=new_authzr_uri),
             key=KEY)
-
-    def tearDown(self):
-        shutil.rmtree(self.tmp)
+        self.mock_client = mock.MagicMock()
+        self.mock_client.directory.new_authz = new_authzr_uri
 
     def test_init_creates_dir(self):
         self.assertTrue(os.path.isdir(self.config.accounts_dir))
 
     def test_save_and_restore(self):
-        self.storage.save(self.acc)
+        self.storage.save(self.acc, self.mock_client)
         account_path = os.path.join(self.config.accounts_dir, self.acc.id)
         self.assertTrue(os.path.exists(account_path))
         for file_name in "regr.json", "meta.json", "private_key.json":
@@ -130,10 +133,19 @@ class AccountFileStorageTest(unittest.TestCase):
             account_path, "private_key.json"))[stat.ST_MODE] & 0o777) in ("0400", "0o400"))
 
         # restore
-        self.assertEqual(self.acc, self.storage.load(self.acc.id))
+        loaded = self.storage.load(self.acc.id)
+        self.assertEqual(self.acc, loaded)
+
+    def test_save_and_restore_old_version(self):
+        """Saved regr should include a new_authzr_uri for older Certbots"""
+        self.storage.save(self.acc, self.mock_client)
+        path = os.path.join(self.config.accounts_dir, self.acc.id, "regr.json")
+        with open(path, "r") as f:
+            regr = json.load(f)
+        self.assertTrue("new_authzr_uri" in regr)
 
     def test_save_regr(self):
-        self.storage.save_regr(self.acc)
+        self.storage.save_regr(self.acc, self.mock_client)
         account_path = os.path.join(self.config.accounts_dir, self.acc.id)
         self.assertTrue(os.path.exists(account_path))
         self.assertTrue(os.path.exists(os.path.join(
@@ -143,7 +155,7 @@ class AccountFileStorageTest(unittest.TestCase):
                 os.path.join(account_path, file_name)))
 
     def test_find_all(self):
-        self.storage.save(self.acc)
+        self.storage.save(self.acc, self.mock_client)
         self.assertEqual([self.acc], self.storage.find_all())
 
     def test_find_all_none_empty_list(self):
@@ -164,14 +176,14 @@ class AccountFileStorageTest(unittest.TestCase):
         self.assertRaises(errors.AccountNotFound, self.storage.load, "missing")
 
     def test_load_id_mismatch_raises_error(self):
-        self.storage.save(self.acc)
+        self.storage.save(self.acc, self.mock_client)
         shutil.move(os.path.join(self.config.accounts_dir, self.acc.id),
                     os.path.join(self.config.accounts_dir, "x" + self.acc.id))
         self.assertRaises(errors.AccountStorageError, self.storage.load,
                           "x" + self.acc.id)
 
     def test_load_ioerror(self):
-        self.storage.save(self.acc)
+        self.storage.save(self.acc, self.mock_client)
         mock_open = mock.mock_open()
         mock_open.side_effect = IOError
         with mock.patch("six.moves.builtins.open", mock_open):
@@ -183,10 +195,11 @@ class AccountFileStorageTest(unittest.TestCase):
         mock_open.side_effect = IOError  # TODO: [None, None, IOError]
         with mock.patch("six.moves.builtins.open", mock_open):
             self.assertRaises(
-                errors.AccountStorageError, self.storage.save, self.acc)
+                errors.AccountStorageError, self.storage.save,
+                    self.acc, self.mock_client)
 
     def test_delete(self):
-        self.storage.save(self.acc)
+        self.storage.save(self.acc, self.mock_client)
         self.storage.delete(self.acc.id)
         self.assertRaises(errors.AccountNotFound, self.storage.load, self.acc.id)
 

certbot/tests/auth_handler_test.py

@@ -5,6 +5,7 @@ import unittest
 
 import mock
 import six
+import zope.component
 
 from acme import challenges
 from acme import client as acme_client
@@ -12,6 +13,7 @@ from acme import messages
 
 from certbot import achallenges
 from certbot import errors
+from certbot import interfaces
 from certbot import util
 
 from certbot.tests import acme_util
@@ -65,6 +67,12 @@ class GetAuthorizationsTest(unittest.TestCase):
     def setUp(self):
         from certbot.auth_handler import AuthHandler
 
+        self.mock_display = mock.Mock()
+        zope.component.provideUtility(
+            self.mock_display, interfaces.IDisplay)
+        zope.component.provideUtility(
+            mock.Mock(debug_challenges=False), interfaces.IConfig)
+
         self.mock_auth = mock.MagicMock(name="ApacheConfigurator")
 
         self.mock_auth.get_chall_pref.return_value = [challenges.TLSSNI01]
@@ -157,6 +165,20 @@ class GetAuthorizationsTest(unittest.TestCase):
 
         self.assertEqual(len(authzr), 3)
 
+    @mock.patch("certbot.auth_handler.AuthHandler._poll_challenges")
+    def test_debug_challenges(self, mock_poll):
+        zope.component.provideUtility(
+            mock.Mock(debug_challenges=True), interfaces.IConfig)
+        self.mock_net.request_domain_challenges.side_effect = functools.partial(
+            gen_dom_authzr, challs=acme_util.CHALLENGES)
+
+        mock_poll.side_effect = self._validate_all
+
+        self.handler.get_authorizations(["0"])
+
+        self.assertEqual(self.mock_net.answer_challenge.call_count, 1)
+        self.assertEqual(self.mock_display.notification.call_count, 1)
+
     def test_perform_failure(self):
         self.mock_net.request_domain_challenges.side_effect = functools.partial(
             gen_dom_authzr, challs=acme_util.CHALLENGES)

certbot/tests/cert_manager_test.py

@@ -18,11 +18,14 @@ from certbot.storage import ALL_FOUR
 from certbot.tests import storage_test
 from certbot.tests import util as test_util
 
-class BaseCertManagerTest(unittest.TestCase):
+from certbot.tests.util import TempDirTestCase
+
+
+class BaseCertManagerTest(TempDirTestCase):
     """Base class for setting up Cert Manager tests.
     """
     def setUp(self):
-        self.tempdir = tempfile.mkdtemp()
+        super(BaseCertManagerTest, self).setUp()
 
         os.makedirs(os.path.join(self.tempdir, "renewal"))
 
@@ -68,9 +71,6 @@ class BaseCertManagerTest(unittest.TestCase):
         config.write()
         return config
 
-    def tearDown(self):
-        shutil.rmtree(self.tempdir)
-
 
 class UpdateLiveSymlinksTest(BaseCertManagerTest):
     """Tests for certbot.cert_manager.update_live_symlinks
@@ -290,6 +290,16 @@ class LineageForCertnameTest(BaseCertManagerTest):
             None)
         self.assertTrue(mock_make_or_verify_dir.called)
 
+    @mock.patch('certbot.util.make_or_verify_dir')
+    @mock.patch('certbot.storage.renewal_file_for_certname')
+    def test_no_renewal_file(self, mock_renewal_conf_file,
+        mock_make_or_verify_dir):
+        mock_renewal_conf_file.side_effect = errors.CertStorageError()
+        from certbot import cert_manager
+        self.assertEqual(cert_manager.lineage_for_certname(self.cli_config, "example.com"),
+            None)
+        self.assertTrue(mock_make_or_verify_dir.called)
+
 
 class DomainsForCertnameTest(BaseCertManagerTest):
     """Tests for certbot.cert_manager.domains_for_certname"""
@@ -427,9 +437,6 @@ class DuplicativeCertsTest(storage_test.BaseRenewableCertTest):
         self.config.write()
         self._write_out_ex_kinds()
 
-    def tearDown(self):
-        shutil.rmtree(self.tempdir)
-
     @mock.patch('certbot.util.make_or_verify_dir')
     def test_find_duplicative_names(self, unused_makedir):
         from certbot.cert_manager import find_duplicative_certs

certbot/tests/cli_test.py

@@ -15,17 +15,18 @@ from certbot import constants
 from certbot import errors
 from certbot.plugins import disco
 
+from certbot.tests.util import TempDirTestCase
+
 PLUGINS = disco.PluginsRegistry.find_all()
 
 
-class TestReadFile(unittest.TestCase):
+class TestReadFile(TempDirTestCase):
     '''Test cli.read_file'''
 
     _multiprocess_can_split_ = True
 
     def test_read_file(self):
-        tmp_dir = tempfile.mkdtemp()
-        rel_test_path = os.path.relpath(os.path.join(tmp_dir, 'foo'))
+        rel_test_path = os.path.relpath(os.path.join(self.tempdir, 'foo'))
         self.assertRaises(
             argparse.ArgumentTypeError, cli.read_file, rel_test_path)
 
@@ -38,6 +39,7 @@ class TestReadFile(unittest.TestCase):
         self.assertEqual(contents, test_contents)
 
 
+
 class ParseTest(unittest.TestCase):
     '''Test the cli args entrypoint'''
 
@@ -60,6 +62,22 @@ class ParseTest(unittest.TestCase):
                 self.assertRaises(SystemExit, self.parse, args, output)
         return output.getvalue()
 
+    @mock.patch("certbot.cli.flag_default")
+    def test_cli_ini_domains(self, mock_flag_default):
+        tmp_config = tempfile.NamedTemporaryFile()
+        # use a shim to get ConfigArgParse to pick up tmp_config
+        shim = lambda v: constants.CLI_DEFAULTS[v] if v != "config_files" else [tmp_config.name]
+        mock_flag_default.side_effect = shim
+
+        namespace = self.parse(["certonly"])
+        self.assertEqual(namespace.domains, [])
+        tmp_config.write(b"domains = example.com")
+        tmp_config.flush()
+        namespace = self.parse(["certonly"])
+        self.assertEqual(namespace.domains, ["example.com"])
+        namespace = self.parse(["renew"])
+        self.assertEqual(namespace.domains, [])
+
     def test_no_args(self):
         namespace = self.parse([])
         for d in ('config_dir', 'logs_dir', 'work_dir'):

certbot/tests/client_test.py

@@ -7,6 +7,7 @@ import unittest
 import OpenSSL
 import mock
 
+from acme import errors as acme_errors
 from acme import jose
 
 from certbot import account
@@ -17,7 +18,7 @@ import certbot.tests.util as test_util
 
 
 KEY = test_util.load_vector("rsa512_key.pem")
-CSR_SAN = test_util.load_vector("csr-san.der")
+CSR_SAN = test_util.load_vector("csr-san.pem")
 
 
 class ConfigHelper(object):
@@ -102,7 +103,7 @@ class RegisterTest(unittest.TestCase):
                     self.config.register_unsafely_without_email = True
                     self.config.dry_run = False
                     self._call()
-                    mock_logger.warning.assert_called_once_with(mock.ANY)
+                    mock_logger.info.assert_called_once_with(mock.ANY)
                     self.assertTrue(mock_handle.called)
 
     def test_unsupported_error(self):
@@ -167,15 +168,17 @@ class ClientTest(ClientTestCommon):
 
         self.acme.request_issuance.assert_called_once_with(
             jose.ComparableX509(OpenSSL.crypto.load_certificate_request(
-                OpenSSL.crypto.FILETYPE_ASN1, CSR_SAN)),
+                OpenSSL.crypto.FILETYPE_PEM, CSR_SAN)),
             authzr)
 
         self.acme.fetch_chain.assert_called_once_with(mock.sentinel.certr)
 
     @mock.patch("certbot.client.logger")
-    def test_obtain_certificate_from_csr(self, mock_logger):
+    @test_util.patch_get_utility()
+    def test_obtain_certificate_from_csr(self, unused_mock_get_utility,
+                                         mock_logger):
         self._mock_obtain_certificate()
-        test_csr = util.CSR(form="der", file=None, data=CSR_SAN)
+        test_csr = util.CSR(form="pem", file=None, data=CSR_SAN)
         auth_handler = self.client.auth_handler
 
         authzr = auth_handler.get_authorizations(self.eg_domains, False)
@@ -206,11 +209,47 @@ class ClientTest(ClientTestCommon):
             test_csr)
         mock_logger.warning.assert_called_once_with(mock.ANY)
 
+    @test_util.patch_get_utility()
+    def test_obtain_certificate_from_csr_retry_succeeded(
+            self, mock_get_utility):
+        self._mock_obtain_certificate()
+        self.acme.fetch_chain.side_effect = [acme_errors.Error,
+                                             mock.sentinel.chain]
+        test_csr = util.CSR(form="der", file=None, data=CSR_SAN)
+        auth_handler = self.client.auth_handler
+
+        authzr = auth_handler.get_authorizations(self.eg_domains, False)
+        self.assertEqual(
+            (mock.sentinel.certr, mock.sentinel.chain),
+            self.client.obtain_certificate_from_csr(
+                self.eg_domains,
+                test_csr,
+                authzr=authzr))
+        self.assertEqual(1, mock_get_utility().notification.call_count)
+
+    @test_util.patch_get_utility()
+    def test_obtain_certificate_from_csr_retry_failed(self, mock_get_utility):
+        self._mock_obtain_certificate()
+        self.acme.fetch_chain.side_effect = acme_errors.Error
+        test_csr = util.CSR(form="der", file=None, data=CSR_SAN)
+        auth_handler = self.client.auth_handler
+
+        authzr = auth_handler.get_authorizations(self.eg_domains, False)
+        self.assertRaises(
+            acme_errors.Error,
+            self.client.obtain_certificate_from_csr,
+            self.eg_domains,
+            test_csr,
+            authzr=authzr)
+        self.assertEqual(1, mock_get_utility().notification.call_count)
+
     @mock.patch("certbot.client.crypto_util")
-    def test_obtain_certificate_rsa(self, mock_crypto_util):
+    @test_util.patch_get_utility()
+    def test_obtain_certificate_rsa(self, unused_mock_get_utility,
+                                    mock_crypto_util):
         self._mock_obtain_certificate()
 
-        csr = util.CSR(form="der", file=None, data=CSR_SAN)
+        csr = util.CSR(form="pem", file=None, data=CSR_SAN)
         mock_crypto_util.init_save_csr.return_value = csr
         mock_crypto_util.save_key.return_value = mock.sentinel.key
         domains = ["example.com", "www.example.com"]
@@ -241,12 +280,14 @@ class ClientTest(ClientTestCommon):
         self._check_obtain_certificate()
 
     @mock.patch("certbot.client.crypto_util")
-    def test_obtain_certificate_ecdsa_p256(self, mock_crypto_util):
+    @test_util.patch_get_utility()
+    def test_obtain_certificate_ecdsa_p256(self, unused_mock_get_utility,
+                                           mock_crypto_util):
         self._mock_obtain_certificate()
 
         self.config.key_types = "ecdsa"
         self.config.ecdsa_curve = "p-256"
-        csr = util.CSR(form="der", file=None, data=CSR_SAN)
+        csr = util.CSR(form="pem", file=None, data=CSR_SAN)
         mock_crypto_util.init_save_csr.return_value = csr
         mock_crypto_util.save_key.return_value = mock.sentinel.key
         domains = ["example.com", "www.example.com"]

certbot/tests/colored_logging_test.py

@@ -1,41 +0,0 @@
-"""Tests for certbot.colored_logging."""
-import logging
-import unittest
-
-import six
-
-from certbot import util
-
-
-class StreamHandlerTest(unittest.TestCase):
-    """Tests for certbot.colored_logging."""
-
-    def setUp(self):
-        from certbot import colored_logging
-
-        self.stream = six.StringIO()
-        self.stream.isatty = lambda: True
-        self.handler = colored_logging.StreamHandler(self.stream)
-
-        self.logger = logging.getLogger()
-        self.logger.setLevel(logging.DEBUG)
-        self.logger.addHandler(self.handler)
-
-    def test_format(self):
-        msg = 'I did a thing'
-        self.logger.debug(msg)
-        self.assertEqual(self.stream.getvalue(), '{0}\n'.format(msg))
-
-    def test_format_and_red_level(self):
-        msg = 'I did another thing'
-        self.handler.red_level = logging.DEBUG
-        self.logger.debug(msg)
-
-        self.assertEqual(self.stream.getvalue(),
-                         '{0}{1}{2}\n'.format(util.ANSI_SGR_RED,
-                                              msg,
-                                              util.ANSI_SGR_RESET))
-
-
-if __name__ == "__main__":
-    unittest.main()  # pragma: no cover

certbot/tests/configuration_test.py

@@ -120,6 +120,12 @@ class NamespaceConfigTest(unittest.TestCase):
         self.assertTrue(os.path.isabs(config.live_dir))
         self.assertTrue(os.path.isabs(config.renewal_configs_dir))
 
+    def test_get_and_set_attr(self):
+        self.config.foo = 42
+        self.assertEqual(self.config.namespace.foo, 42)
+        self.config.namespace.bar = 1337
+        self.assertEqual(self.config.bar, 1337)
+
 
 if __name__ == '__main__':
     unittest.main()  # pragma: no cover

certbot/tests/crypto_util_test.py

@@ -1,7 +1,6 @@
 """Tests for certbot.crypto_util."""
 import logging
-import shutil
-import tempfile
+import os
 import unittest
 
 import OpenSSL
@@ -20,17 +19,20 @@ CERT = test_util.load_vector('cert.pem')
 SAN_CERT = test_util.load_vector('cert-san.pem')
 
 
-class SaveKeyTest(unittest.TestCase):
+class SaveKeyTest(test_util.TempDirTestCase):
     """Tests for certbot.crypto_util.save_key."""
     def setUp(self):
+        super(SaveKeyTest, self).setUp()
+
         logging.disable(logging.CRITICAL)
         zope.component.provideUtility(
-            mock.Mock(strict_permissions=True), interfaces.IConfig)
-        self.key_dir = tempfile.mkdtemp('key_dir')
+            mock.Mock(strict_permissions=True, dry_run=False),
+            interfaces.IConfig)
 
     def tearDown(self):
+        super(SaveKeyTest, self).tearDown()
+
         logging.disable(logging.NOTSET)
-        shutil.rmtree(self.key_dir)
 
     @classmethod
     def _call(cls, key_pem, key_dir):
@@ -38,74 +40,50 @@ class SaveKeyTest(unittest.TestCase):
         return save_key(key_pem, key_dir, 'key-certbot.pem')
 
     def test_success(self):
-        key = self._call(RSA512_KEY, self.key_dir)
+        key = self._call(RSA512_KEY, self.tempdir)
         self.assertEqual(key.pem, RSA512_KEY)
         self.assertTrue('key-certbot.pem' in key.file)
+        self.assertTrue(os.path.exists(os.path.join(self.tempdir, key.file)))
 
 
-class InitSaveCSRTest(unittest.TestCase):
+class InitSaveCSRTest(test_util.TempDirTestCase):
     """Tests for certbot.crypto_util.init_save_csr."""
 
     def setUp(self):
+        super(InitSaveCSRTest, self).setUp()
+
         zope.component.provideUtility(
-            mock.Mock(strict_permissions=True), interfaces.IConfig)
-        self.csr_dir = tempfile.mkdtemp('csr_dir')
+            mock.Mock(strict_permissions=True, dry_run=False),
+            interfaces.IConfig)
 
-    def tearDown(self):
-        shutil.rmtree(self.csr_dir)
-
-    @mock.patch('certbot.crypto_util.make_csr')
+    @mock.patch('acme.crypto_util.make_csr')
     @mock.patch('certbot.crypto_util.util.make_or_verify_dir')
-    def test_it(self, unused_mock_verify, mock_csr):
+    def test_success(self, unused_mock_verify, mock_csr):
         from certbot.crypto_util import init_save_csr
 
-        mock_csr.return_value = (b'csr_pem', b'csr_der')
+        mock_csr.return_value = b'csr_pem'
 
         csr = init_save_csr(
-            mock.Mock(pem='dummy_key'), 'example.com', self.csr_dir,
-            'csr-certbot.pem')
+            mock.Mock(pem='dummy_key'), 'example.com', self.tempdir)
 
-        self.assertEqual(csr.data, b'csr_der')
+        self.assertEqual(csr.data, b'csr_pem')
         self.assertTrue('csr-certbot.pem' in csr.file)
 
+    @mock.patch('acme.crypto_util.make_csr')
+    @mock.patch('certbot.crypto_util.util.make_or_verify_dir')
+    def test_success_dry_run(self, unused_mock_verify, mock_csr):
+        from certbot.crypto_util import init_save_csr
 
-class MakeCSRTest(unittest.TestCase):
-    """Tests for certbot.crypto_util.make_csr."""
+        zope.component.provideUtility(
+            mock.Mock(strict_permissions=True, dry_run=True),
+            interfaces.IConfig)
+        mock_csr.return_value = b'csr_pem'
 
-    @classmethod
-    def _call(cls, *args, **kwargs):
-        from certbot.crypto_util import make_csr
-        return make_csr(*args, **kwargs)
+        csr = init_save_csr(
+            mock.Mock(pem='dummy_key'), 'example.com', self.tempdir)
 
-    def test_san(self):
-        from certbot.crypto_util import get_sans_from_csr
-        # TODO: Fails for RSA256_KEY
-        csr_pem, csr_der = self._call(
-            RSA512_KEY, ['example.com', 'www.example.com'])
-        self.assertEqual(
-            ['example.com', 'www.example.com'], get_sans_from_csr(csr_pem))
-        self.assertEqual(
-            ['example.com', 'www.example.com'], get_sans_from_csr(
-                csr_der, OpenSSL.crypto.FILETYPE_ASN1))
-
-    def test_must_staple(self):
-        # TODO: Fails for RSA256_KEY
-        csr_pem, _ = self._call(
-            RSA512_KEY, ['example.com', 'www.example.com'], must_staple=True)
-        csr = OpenSSL.crypto.load_certificate_request(
-            OpenSSL.crypto.FILETYPE_PEM, csr_pem)
-
-        # In pyopenssl 0.13 (used with TOXENV=py26-oldest and py27-oldest), csr
-        # objects don't have a get_extensions() method, so we skip this test if
-        # the method isn't available.
-        if hasattr(csr, 'get_extensions'):
-            # NOTE: Ideally we would filter by the TLS Feature OID, but
-            # OpenSSL.crypto.X509Extension doesn't give us the extension's raw OID,
-            # and the shortname field is just "UNDEF"
-            must_staple_exts = [e for e in csr.get_extensions()
-                if e.get_data() == b"0\x03\x02\x01\x05"]
-            self.assertEqual(len(must_staple_exts), 1,
-                "Expected exactly one Must Staple extension")
+        self.assertEqual(csr.data, b'csr_pem')
+        self.assertTrue(csr.file is None)
 
 
 class ValidCSRTest(unittest.TestCase):
@@ -125,9 +103,6 @@ class ValidCSRTest(unittest.TestCase):
     def test_valid_der_false(self):
         self.assertFalse(self._call(test_util.load_vector('csr.der')))
 
-    def test_valid_der_san_false(self):
-        self.assertFalse(self._call(test_util.load_vector('csr-san.der')))
-
     def test_empty_false(self):
         self.assertFalse(self._call(''))
 
@@ -163,12 +138,13 @@ class ImportCSRFileTest(unittest.TestCase):
     def test_der_csr(self):
         csrfile = test_util.vector_path('csr.der')
         data = test_util.load_vector('csr.der')
+        data_pem = test_util.load_vector('csr.pem')
 
         self.assertEqual(
-            (OpenSSL.crypto.FILETYPE_ASN1,
+            (OpenSSL.crypto.FILETYPE_PEM,
              util.CSR(file=csrfile,
-                      data=data,
-                      form="der"),
+                      data=data_pem,
+                      form="pem"),
              ["example.com"],),
             self._call(csrfile, data))
 
@@ -252,36 +228,6 @@ class GetSANsFromCertTest(unittest.TestCase):
             self._call(test_util.load_vector('cert-san.pem')))
 
 
-class GetSANsFromCSRTest(unittest.TestCase):
-    """Tests for certbot.crypto_util.get_sans_from_csr."""
-
-    @classmethod
-    def _call(cls, *args, **kwargs):
-        from certbot.crypto_util import get_sans_from_csr
-        return get_sans_from_csr(*args, **kwargs)
-
-    def test_extract_one_san(self):
-        self.assertEqual(['example.com'], self._call(
-            test_util.load_vector('csr.pem')))
-
-    def test_extract_two_sans(self):
-        self.assertEqual(['example.com', 'www.example.com'], self._call(
-            test_util.load_vector('csr-san.pem')))
-
-    def test_extract_six_sans(self):
-        self.assertEqual(self._call(test_util.load_vector('csr-6sans.pem')),
-                         ["example.com", "example.org", "example.net",
-                          "example.info", "subdomain.example.com",
-                          "other.subdomain.example.com"])
-
-    def test_parse_non_csr(self):
-        self.assertRaises(OpenSSL.crypto.Error, self._call, "hello there")
-
-    def test_parse_no_sans(self):
-        self.assertEqual(
-            [], self._call(test_util.load_vector('csr-nosans.pem')))
-
-
 class GetNamesFromCertTest(unittest.TestCase):
     """Tests for certbot.crypto_util.get_names_from_cert."""
 
@@ -307,36 +253,9 @@ class GetNamesFromCertTest(unittest.TestCase):
             ['example.com'] + ['{0}.example.com'.format(c) for c in 'abcd'],
             self._call(test_util.load_vector('cert-5sans.pem')))
 
-
-class GetNamesFromCSRTest(unittest.TestCase):
-    """Tests for certbot.crypto_util.get_names_from_csr."""
-    @classmethod
-    def _call(cls, *args, **kwargs):
-        from certbot.crypto_util import get_names_from_csr
-        return get_names_from_csr(*args, **kwargs)
-
-    def test_extract_one_san(self):
-        self.assertEqual(['example.com'], self._call(
-            test_util.load_vector('csr.pem')))
-
-    def test_extract_two_sans(self):
-        self.assertEqual(set(('example.com', 'www.example.com',)), set(
-            self._call(test_util.load_vector('csr-san.pem'))))
-
-    def test_extract_six_sans(self):
-        self.assertEqual(
-            set(self._call(test_util.load_vector('csr-6sans.pem'))),
-            set(("example.com", "example.org", "example.net",
-                 "example.info", "subdomain.example.com",
-                 "other.subdomain.example.com",)))
-
-    def test_parse_non_csr(self):
+    def test_parse_non_cert(self):
         self.assertRaises(OpenSSL.crypto.Error, self._call, "hello there")
 
-    def test_parse_no_sans(self):
-        self.assertEqual(["example.org"],
-                         self._call(test_util.load_vector('csr-nosans.pem')))
-
 
 class CertLoaderTest(unittest.TestCase):
     """Tests for certbot.crypto_util.pyopenssl_load_certificate"""
@@ -345,8 +264,8 @@ class CertLoaderTest(unittest.TestCase):
         from certbot.crypto_util import pyopenssl_load_certificate
 
         cert, file_type = pyopenssl_load_certificate(CERT)
-        self.assertEqual(cert.digest('sha1'),
-                         OpenSSL.crypto.load_certificate(file_type, CERT).digest('sha1'))
+        self.assertEqual(cert.digest('sha256'),
+                         OpenSSL.crypto.load_certificate(file_type, CERT).digest('sha256'))
 
     def test_load_invalid_cert(self):
         from certbot.crypto_util import pyopenssl_load_certificate

certbot/tests/display/completer_test.py

@@ -1,31 +1,30 @@
 """Test certbot.display.completer."""
 import os
 import readline
-import shutil
 import string
 import sys
-import tempfile
 import unittest
 
 import mock
 from six.moves import reload_module  # pylint: disable=import-error
 
+from certbot.tests.util import TempDirTestCase
 
-class CompleterTest(unittest.TestCase):
+class CompleterTest(TempDirTestCase):
     """Test certbot.display.completer.Completer."""
 
     def setUp(self):
-        self.temp_dir = tempfile.mkdtemp()
+        super(CompleterTest, self).setUp()
 
         # directories must end with os.sep for completer to
         # search inside the directory for possible completions
-        if self.temp_dir[-1] != os.sep:
-            self.temp_dir += os.sep
+        if self.tempdir[-1] != os.sep:
+            self.tempdir += os.sep
 
         self.paths = []
         # create some files and directories in temp_dir
         for c in string.ascii_lowercase:
-            path = os.path.join(self.temp_dir, c)
+            path = os.path.join(self.tempdir, c)
             self.paths.append(path)
             if ord(c) % 2:
                 os.mkdir(path)
@@ -33,21 +32,18 @@ class CompleterTest(unittest.TestCase):
                 with open(path, 'w'):
                     pass
 
-    def tearDown(self):
-        shutil.rmtree(self.temp_dir)
-
     def test_complete(self):
         from certbot.display import completer
         my_completer = completer.Completer()
         num_paths = len(self.paths)
 
         for i in range(num_paths):
-            completion = my_completer.complete(self.temp_dir, i)
+            completion = my_completer.complete(self.tempdir, i)
             self.assertTrue(completion in self.paths)
             self.paths.remove(completion)
 
         self.assertFalse(self.paths)
-        completion = my_completer.complete(self.temp_dir, num_paths)
+        completion = my_completer.complete(self.tempdir, num_paths)
         self.assertEqual(completion, None)
 
     def test_import_error(self):

certbot/tests/display/ops_test.py

@@ -2,7 +2,6 @@
 """Test certbot.display.ops."""
 import os
 import sys
-import tempfile
 import unittest
 
 import mock
@@ -87,18 +86,19 @@ class GetEmailTest(unittest.TestCase):
             self.assertTrue(invalid_txt in mock_input.call_args[0][0])
 
 
-class ChooseAccountTest(unittest.TestCase):
+class ChooseAccountTest(test_util.TempDirTestCase):
     """Tests for certbot.display.ops.choose_account."""
     def setUp(self):
+        super(ChooseAccountTest, self).setUp()
+
         zope.component.provideUtility(display_util.FileDisplay(sys.stdout,
                                                                False))
 
-        self.accounts_dir = tempfile.mkdtemp("accounts")
-        self.account_keys_dir = os.path.join(self.accounts_dir, "keys")
+        self.account_keys_dir = os.path.join(self.tempdir, "keys")
         os.makedirs(self.account_keys_dir, 0o700)
 
         self.config = mock.MagicMock(
-            accounts_dir=self.accounts_dir,
+            accounts_dir=self.tempdir,
             account_keys_dir=self.account_keys_dir,
             server="certbot-demo.org")
         self.key = KEY

certbot/tests/log_test.py

@@ -0,0 +1,374 @@
+"""Tests for certbot.log."""
+import logging
+import logging.handlers
+import os
+import sys
+import time
+import unittest
+
+import mock
+import six
+
+from acme import messages
+
+from certbot import constants
+from certbot import errors
+from certbot import util
+from certbot.tests import util as test_util
+
+
+class PreArgParseSetupTest(unittest.TestCase):
+    """Tests for certbot.log.pre_arg_parse_setup."""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.log import pre_arg_parse_setup
+        return pre_arg_parse_setup(*args, **kwargs)
+
+    @mock.patch('certbot.log.sys')
+    @mock.patch('certbot.log.except_hook')
+    @mock.patch('certbot.log.logging.getLogger')
+    @mock.patch('certbot.log.util.atexit_register')
+    def test_it(self, mock_register, mock_get, mock_except_hook, mock_sys):
+        mock_sys.argv = ['--debug']
+        mock_sys.version_info = sys.version_info
+        self._call()
+
+        mock_register.assert_called_once_with(logging.shutdown)
+        mock_sys.excepthook(1, 2, 3)
+        mock_except_hook.assert_called_once_with(
+            1, 2, 3, debug=True, log_path=mock.ANY)
+
+        mock_root_logger = mock_get()
+        mock_root_logger.setLevel.assert_called_once_with(logging.DEBUG)
+        self.assertEqual(mock_root_logger.addHandler.call_count, 2)
+
+        MemoryHandler = logging.handlers.MemoryHandler
+        memory_handler = None
+        for call in mock_root_logger.addHandler.call_args_list:
+            handler = call[0][0]
+            if memory_handler is None and isinstance(handler, MemoryHandler):
+                memory_handler = handler
+            else:
+                self.assertTrue(isinstance(handler, logging.StreamHandler))
+        self.assertTrue(
+            isinstance(memory_handler.target, logging.StreamHandler))
+
+
+class PostArgParseSetupTest(test_util.TempDirTestCase):
+    """Tests for certbot.log.post_arg_parse_setup."""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.log import post_arg_parse_setup
+        return post_arg_parse_setup(*args, **kwargs)
+
+    def setUp(self):
+        super(PostArgParseSetupTest, self).setUp()
+        self.config = mock.MagicMock(
+            debug=False, logs_dir=self.tempdir, quiet=False,
+            verbose_count=constants.CLI_DEFAULTS['verbose_count'])
+        self.devnull = open(os.devnull, 'w')
+
+        from certbot.log import ColoredStreamHandler
+        self.stream_handler = ColoredStreamHandler(six.StringIO())
+        from certbot.log import MemoryHandler, TempHandler
+        self.temp_handler = TempHandler()
+        self.temp_path = self.temp_handler.path
+        self.memory_handler = MemoryHandler(self.temp_handler)
+        self.root_logger = mock.MagicMock(
+            handlers=[self.memory_handler, self.stream_handler])
+
+    def tearDown(self):
+        self.memory_handler.close()
+        self.stream_handler.close()
+        self.temp_handler.close()
+        super(PostArgParseSetupTest, self).tearDown()
+
+    def test_common(self):
+        with mock.patch('certbot.log.logging.getLogger') as mock_get_logger:
+            mock_get_logger.return_value = self.root_logger
+            with mock.patch('certbot.log.except_hook') as mock_except_hook:
+                with mock.patch('certbot.log.sys') as mock_sys:
+                    mock_sys.version_info = sys.version_info
+                    self._call(self.config)
+
+        self.root_logger.removeHandler.assert_called_once_with(
+            self.memory_handler)
+        self.assertTrue(self.root_logger.addHandler.called)
+        self.assertTrue(os.path.exists(os.path.join(
+            self.config.logs_dir, 'letsencrypt.log')))
+        self.assertFalse(os.path.exists(self.temp_path))
+        mock_sys.excepthook(1, 2, 3)
+        mock_except_hook.assert_called_once_with(
+            1, 2, 3, debug=self.config.debug, log_path=self.tempdir)
+
+        level = self.stream_handler.level
+        if self.config.quiet:
+            self.assertEqual(level, constants.QUIET_LOGGING_LEVEL)
+        else:
+            self.assertEqual(level, -self.config.verbose_count * 10)
+
+    def test_debug(self):
+        self.config.debug = True
+        self.test_common()
+
+    def test_quiet(self):
+        self.config.quiet = True
+        self.test_common()
+
+
+class SetupLogFileHandlerTest(test_util.TempDirTestCase):
+    """Tests for certbot.log.setup_log_file_handler."""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.log import setup_log_file_handler
+        return setup_log_file_handler(*args, **kwargs)
+
+    def setUp(self):
+        super(SetupLogFileHandlerTest, self).setUp()
+        self.config = mock.MagicMock(logs_dir=self.tempdir)
+
+    @mock.patch('certbot.main.logging.handlers.RotatingFileHandler')
+    def test_failure(self, mock_handler):
+        mock_handler.side_effect = IOError
+
+        try:
+            self._call(self.config, 'test.log', '%(message)s')
+        except errors.Error as err:
+            self.assertTrue('--logs-dir' in str(err))
+        else:  # pragma: no cover
+            self.fail('Error not raised.')
+
+    def test_success(self):
+        log_file = 'test.log'
+        handler, log_path = self._call(self.config, log_file, '%(message)s')
+        self.assertEqual(handler.level, logging.DEBUG)
+        self.assertEqual(handler.formatter.converter, time.gmtime)
+
+        expected_path = os.path.join(self.config.logs_dir, log_file)
+        self.assertEqual(log_path, expected_path)
+        handler.close()
+
+
+class ColoredStreamHandlerTest(unittest.TestCase):
+    """Tests for certbot.log.ColoredStreamHandler"""
+
+    def setUp(self):
+        self.stream = six.StringIO()
+        self.stream.isatty = lambda: True
+        self.logger = logging.getLogger()
+        self.logger.setLevel(logging.DEBUG)
+
+        from certbot.log import ColoredStreamHandler
+        self.handler = ColoredStreamHandler(self.stream)
+        self.logger.addHandler(self.handler)
+
+    def tearDown(self):
+        self.handler.close()
+
+    def test_format(self):
+        msg = 'I did a thing'
+        self.logger.debug(msg)
+        self.assertEqual(self.stream.getvalue(), '{0}\n'.format(msg))
+
+    def test_format_and_red_level(self):
+        msg = 'I did another thing'
+        self.handler.red_level = logging.DEBUG
+        self.logger.debug(msg)
+
+        self.assertEqual(self.stream.getvalue(),
+                         '{0}{1}{2}\n'.format(util.ANSI_SGR_RED,
+                                              msg,
+                                              util.ANSI_SGR_RESET))
+
+
+class MemoryHandlerTest(unittest.TestCase):
+    """Tests for certbot.log.MemoryHandler"""
+    def setUp(self):
+        self.logger = logging.getLogger(__name__)
+        self.logger.setLevel(logging.DEBUG)
+        self.msg = 'hi there'
+        self.stream = six.StringIO()
+
+        self.stream_handler = logging.StreamHandler(self.stream)
+        from certbot.log import MemoryHandler
+        self.handler = MemoryHandler(self.stream_handler)
+        self.logger.addHandler(self.handler)
+
+    def tearDown(self):
+        self.handler.close()
+        self.stream_handler.close()
+
+    def test_flush(self):
+        self._test_log_debug()
+        self.handler.flush()
+        self.assertEqual(self.stream.getvalue(), self.msg + '\n')
+
+    def test_not_flushed(self):
+        # By default, logging.ERROR messages and higher are flushed
+        self.logger.critical(self.msg)
+        self.assertEqual(self.stream.getvalue(), '')
+
+    def test_target_reset(self):
+        self._test_log_debug()
+
+        new_stream = six.StringIO()
+        new_stream_handler = logging.StreamHandler(new_stream)
+        self.handler.setTarget(new_stream_handler)
+        self.handler.flush()
+        self.assertEqual(self.stream.getvalue(), '')
+        self.assertEqual(new_stream.getvalue(), self.msg + '\n')
+        new_stream_handler.close()
+
+    def _test_log_debug(self):
+        self.logger.debug(self.msg)
+
+
+class TempHandlerTest(unittest.TestCase):
+    """Tests for certbot.log.TempHandler."""
+    def setUp(self):
+        self.closed = False
+        from certbot.log import TempHandler
+        self.handler = TempHandler()
+
+    def tearDown(self):
+        if not self.closed:
+            self.handler.delete_and_close()
+
+    def test_permissions(self):
+        self.assertTrue(
+            util.check_permissions(self.handler.path, 0o600, os.getuid()))
+
+    def test_delete(self):
+        self.handler.delete_and_close()
+        self.closed = True
+        self.assertFalse(os.path.exists(self.handler.path))
+
+    def test_no_delete(self):
+        self.handler.close()
+        self.closed = True
+        self.assertTrue(os.path.exists(self.handler.path))
+        os.remove(self.handler.path)
+
+
+class ExceptHookTest(unittest.TestCase):
+    """Tests for certbot.log.except_hook."""
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.log import except_hook
+        return except_hook(*args, **kwargs)
+
+    def setUp(self):
+        self.error_msg = 'test error message'
+        self.log_path = 'foo.log'
+
+    def test_base_exception(self):
+        exc_type = KeyboardInterrupt
+        mock_logger, output = self._test_common(exc_type, debug=False)
+        self._assert_exception_logged(mock_logger.error, exc_type)
+        self._assert_logfile_output(output)
+
+    def test_debug(self):
+        exc_type = ValueError
+        mock_logger, output = self._test_common(exc_type, debug=True)
+        self._assert_exception_logged(mock_logger.error, exc_type)
+        self._assert_logfile_output(output)
+
+    def test_custom_error(self):
+        exc_type = errors.PluginError
+        mock_logger, output = self._test_common(exc_type, debug=False)
+        self._assert_exception_logged(mock_logger.debug, exc_type)
+        self._assert_quiet_output(mock_logger, output)
+
+    def test_acme_error(self):
+        # Get an arbitrary error code
+        acme_code = next(six.iterkeys(messages.ERROR_CODES))
+
+        def get_acme_error(msg):
+            """Wraps ACME errors so the constructor takes only a msg."""
+            return messages.Error.with_code(acme_code, detail=msg)
+
+        mock_logger, output = self._test_common(get_acme_error, debug=False)
+        self._assert_exception_logged(mock_logger.debug, messages.Error)
+        self._assert_quiet_output(mock_logger, output)
+        self.assertFalse(messages.ERROR_PREFIX in output)
+
+    def test_other_error(self):
+        exc_type = ValueError
+        mock_logger, output = self._test_common(exc_type, debug=False)
+        self._assert_exception_logged(mock_logger.debug, exc_type)
+        self._assert_quiet_output(mock_logger, output)
+
+    def _test_common(self, error_type, debug):
+        """Returns the mocked logger and stderr output."""
+        mock_err = six.StringIO()
+        try:
+            raise error_type(self.error_msg)
+        except BaseException:
+            exc_info = sys.exc_info()
+            with mock.patch('certbot.log.logger') as mock_logger:
+                with mock.patch('certbot.log.sys.stderr', mock_err):
+                    try:
+                        # pylint: disable=star-args
+                        self._call(
+                            *exc_info, debug=debug, log_path=self.log_path)
+                    except SystemExit as exit_err:
+                        mock_err.write(str(exit_err))
+                    else:  # pragma: no cover
+                        self.fail('SystemExit not raised.')
+
+        output = mock_err.getvalue()
+        return mock_logger, output
+
+    def _assert_exception_logged(self, log_func, exc_type):
+        self.assertTrue(log_func.called)
+        call_kwargs = log_func.call_args[1]
+        self.assertTrue('exc_info' in call_kwargs)
+
+        actual_exc_info = call_kwargs['exc_info']
+        expected_exc_info = (exc_type, mock.ANY, mock.ANY)
+        self.assertEqual(actual_exc_info, expected_exc_info)
+
+    def _assert_logfile_output(self, output):
+        self.assertTrue('Please see the logfile' in output)
+        self.assertTrue(self.log_path in output)
+
+    def _assert_quiet_output(self, mock_logger, output):
+        self.assertFalse(mock_logger.exception.called)
+        self.assertTrue(mock_logger.debug.called)
+        self.assertTrue(self.error_msg in output)
+
+
+class ExitWithLogPathTest(test_util.TempDirTestCase):
+    """Tests for certbot.log.exit_with_log_path."""
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.log import exit_with_log_path
+        return exit_with_log_path(*args, **kwargs)
+
+    def test_log_file(self):
+        log_file = os.path.join(self.tempdir, 'test.log')
+        open(log_file, 'w').close()
+
+        err_str = self._test_common(log_file)
+        self.assertTrue('logfiles' not in err_str)
+        self.assertTrue(log_file in err_str)
+
+    def test_log_dir(self):
+        err_str = self._test_common(self.tempdir)
+        self.assertTrue('logfiles' in err_str)
+        self.assertTrue(self.tempdir in err_str)
+
+    def _test_common(self, *args, **kwargs):
+        try:
+            self._call(*args, **kwargs)
+        except SystemExit as err:
+            return str(err)
+        else:  # pragma: no cover
+            self.fail('SystemExit was not raised.')
+
+
+if __name__ == "__main__":
+    unittest.main()  # pragma: no cover

certbot/tests/main_test.py

@@ -6,7 +6,6 @@ import itertools
 import mock
 import os
 import shutil
-import tempfile
 import traceback
 import unittest
 import datetime
@@ -19,7 +18,6 @@ from acme import jose
 
 from certbot import account
 from certbot import cli
-from certbot import colored_logging
 from certbot import constants
 from certbot import configuration
 from certbot import crypto_util
@@ -219,13 +217,14 @@ class FindDomainsOrCertnameTest(unittest.TestCase):
             (["one.com", "two.com"], "one.com"))
 
 
-class RevokeTest(unittest.TestCase):
+class RevokeTest(test_util.TempDirTestCase):
     """Tests for certbot.main.revoke."""
 
     def setUp(self):
-        self.tempdir_path = tempfile.mkdtemp()
-        shutil.copy(CERT_PATH, self.tempdir_path)
-        self.tmp_cert_path = os.path.abspath(os.path.join(self.tempdir_path,
+        super(RevokeTest, self).setUp()
+
+        shutil.copy(CERT_PATH, self.tempdir)
+        self.tmp_cert_path = os.path.abspath(os.path.join(self.tempdir,
             'cert.pem'))
 
         self.patches = [
@@ -251,7 +250,8 @@ class RevokeTest(unittest.TestCase):
         self.mock_determine_account.return_value = (self.acc, None)
 
     def tearDown(self):
-        shutil.rmtree(self.tempdir_path)
+        super(RevokeTest, self).tearDown()
+
         for patch in self.patches:
             patch.stop()
 
@@ -287,89 +287,6 @@ class RevokeTest(unittest.TestCase):
         self.mock_success_revoke.assert_not_called()
 
 
-class SetupLogFileHandlerTest(unittest.TestCase):
-    """Tests for certbot.main.setup_log_file_handler."""
-
-    def setUp(self):
-        self.config = mock.Mock(spec_set=['logs_dir'],
-                                logs_dir=tempfile.mkdtemp())
-
-    def tearDown(self):
-        shutil.rmtree(self.config.logs_dir)
-
-    def _call(self, *args, **kwargs):
-        from certbot.main import setup_log_file_handler
-        return setup_log_file_handler(*args, **kwargs)
-
-    @mock.patch('certbot.main.logging.handlers.RotatingFileHandler')
-    def test_ioerror(self, mock_handler):
-        mock_handler.side_effect = IOError
-        self.assertRaises(errors.Error, self._call,
-                          self.config, "test.log", "%s")
-
-
-class SetupLoggingTest(unittest.TestCase):
-    """Tests for certbot.main.setup_logging."""
-
-    def setUp(self):
-        self.config = mock.Mock(
-            logs_dir=tempfile.mkdtemp(),
-            noninteractive_mode=False, quiet=False,
-            verbose_count=constants.CLI_DEFAULTS['verbose_count'])
-
-    def tearDown(self):
-        shutil.rmtree(self.config.logs_dir)
-
-    @classmethod
-    def _call(cls, *args, **kwargs):
-        from certbot.main import setup_logging
-        return setup_logging(*args, **kwargs)
-
-    @mock.patch('certbot.main.logging.getLogger')
-    def test_defaults(self, mock_get_logger):
-        self._call(self.config)
-
-        cli_handler = mock_get_logger().addHandler.call_args_list[0][0][0]
-        self.assertEqual(cli_handler.level, -self.config.verbose_count * 10)
-        self.assertTrue(
-            isinstance(cli_handler, colored_logging.StreamHandler))
-
-    @mock.patch('certbot.main.logging.getLogger')
-    def test_quiet_mode(self, mock_get_logger):
-        self.config.quiet = self.config.noninteractive_mode = True
-        self._call(self.config)
-
-        cli_handler = mock_get_logger().addHandler.call_args_list[0][0][0]
-        self.assertEqual(cli_handler.level, constants.QUIET_LOGGING_LEVEL)
-        self.assertTrue(
-            isinstance(cli_handler, colored_logging.StreamHandler))
-
-
-class MakeOrVerifyCoreDirTest(unittest.TestCase):
-    """Tests for certbot.main.make_or_verify_core_dir."""
-
-    def setUp(self):
-        self.dir = tempfile.mkdtemp()
-
-    def tearDown(self):
-        shutil.rmtree(self.dir)
-
-    def _call(self, *args, **kwargs):
-        from certbot.main import make_or_verify_core_dir
-        return make_or_verify_core_dir(*args, **kwargs)
-
-    def test_success(self):
-        new_dir = os.path.join(self.dir, 'new')
-        self._call(new_dir, 0o700, os.geteuid(), False)
-        self.assertTrue(os.path.exists(new_dir))
-
-    @mock.patch('certbot.main.util.make_or_verify_dir')
-    def test_failure(self, mock_make_or_verify):
-        mock_make_or_verify.side_effect = OSError
-        self.assertRaises(errors.Error, self._call,
-                          self.dir, 0o700, os.geteuid(), False)
-
-
 class DetermineAccountTest(unittest.TestCase):
     """Tests for certbot.main._determine_account."""
 
@@ -382,6 +299,9 @@ class DetermineAccountTest(unittest.TestCase):
         self.config = configuration.NamespaceConfig(self.args)
         self.accs = [mock.MagicMock(id='x'), mock.MagicMock(id='y')]
         self.account_storage = account.AccountMemoryStorage()
+        # For use in saving accounts: fake out the new_authz URL.
+        self.mock_client = mock.MagicMock()
+        self.mock_client.directory.new_authz = "hi"
 
     def _call(self):
         # pylint: disable=protected-access
@@ -391,14 +311,14 @@ class DetermineAccountTest(unittest.TestCase):
             return _determine_account(self.config)
 
     def test_args_account_set(self):
-        self.account_storage.save(self.accs[1])
+        self.account_storage.save(self.accs[1], self.mock_client)
         self.config.account = self.accs[1].id
         self.assertEqual((self.accs[1], None), self._call())
         self.assertEqual(self.accs[1].id, self.config.account)
         self.assertTrue(self.config.email is None)
 
     def test_single_account(self):
-        self.account_storage.save(self.accs[0])
+        self.account_storage.save(self.accs[0], self.mock_client)
         self.assertEqual((self.accs[0], None), self._call())
         self.assertEqual(self.accs[0].id, self.config.account)
         self.assertTrue(self.config.email is None)
@@ -406,7 +326,7 @@ class DetermineAccountTest(unittest.TestCase):
     @mock.patch('certbot.client.display_ops.choose_account')
     def test_multiple_accounts(self, mock_choose_accounts):
         for acc in self.accs:
-            self.account_storage.save(acc)
+            self.account_storage.save(acc, self.mock_client)
         mock_choose_accounts.return_value = self.accs[1]
         self.assertEqual((self.accs[1], None), self._call())
         self.assertEqual(
@@ -437,14 +357,15 @@ class DetermineAccountTest(unittest.TestCase):
         self.assertEqual('other email', self.config.email)
 
 
-class MainTest(unittest.TestCase):  # pylint: disable=too-many-public-methods
+class MainTest(test_util.TempDirTestCase):  # pylint: disable=too-many-public-methods
     """Tests for different commands."""
 
     def setUp(self):
-        self.tmp_dir = tempfile.mkdtemp()
-        self.config_dir = os.path.join(self.tmp_dir, 'config')
-        self.work_dir = os.path.join(self.tmp_dir, 'work')
-        self.logs_dir = os.path.join(self.tmp_dir, 'logs')
+        super(MainTest, self).setUp()
+
+        self.config_dir = os.path.join(self.tempdir, 'config')
+        self.work_dir = os.path.join(self.tempdir, 'work')
+        self.logs_dir = os.path.join(self.tempdir, 'logs')
         os.mkdir(self.logs_dir)
         self.standard_args = ['--config-dir', self.config_dir,
                               '--work-dir', self.work_dir,
@@ -453,7 +374,8 @@ class MainTest(unittest.TestCase):  # pylint: disable=too-many-public-methods
     def tearDown(self):
         # Reset globals in cli
         reload_module(cli)
-        shutil.rmtree(self.tmp_dir)
+
+        super(MainTest, self).tearDown()
 
     def _call(self, args, stdout=None):
         "Run the cli with output streams and actual client mocked out"
@@ -1171,8 +1093,8 @@ class MainTest(unittest.TestCase):  # pylint: disable=too-many-public-methods
                         mocked_account.AccountFileStorage.return_value = mocked_storage
                         mocked_storage.find_all.return_value = ["an account"]
                         mocked_det.return_value = (mock.MagicMock(), "foo")
-                        acme_client = mock.MagicMock()
-                        mocked_client.Client.return_value = acme_client
+                        cb_client = mock.MagicMock()
+                        mocked_client.Client.return_value = cb_client
                         x = self._call_no_clientmock(
                             ["register", "--update-registration"])
                         # When registration change succeeds, the return value
@@ -1181,7 +1103,7 @@ class MainTest(unittest.TestCase):  # pylint: disable=too-many-public-methods
                         # and we got supposedly did update the registration from
                         # the server
                         self.assertTrue(
-                            acme_client.acme.update_registration.called)
+                            cb_client.acme.update_registration.called)
                         # and we saved the updated registration on disk
                         self.assertTrue(mocked_storage.save_regr.called)
                         self.assertTrue(
@@ -1221,8 +1143,8 @@ class UnregisterTest(unittest.TestCase):
         self.mocks['account'].AccountFileStorage.return_value = mocked_storage
         self.mocks['_determine_account'].return_value = (mock.MagicMock(), "foo")
 
-        acme_client = mock.MagicMock()
-        self.mocks['client'].Client.return_value = acme_client
+        cb_client = mock.MagicMock()
+        self.mocks['client'].Client.return_value = cb_client
 
         config = mock.MagicMock()
         unused_plugins = mock.MagicMock()
@@ -1230,7 +1152,7 @@ class UnregisterTest(unittest.TestCase):
         res = main.unregister(config, unused_plugins)
 
         self.assertTrue(res is None)
-        self.assertTrue(acme_client.acme.deactivate_registration.called)
+        self.assertTrue(cb_client.acme.deactivate_registration.called)
         m = "Account deactivated."
         self.assertTrue(m in self.mocks['get_utility']().add_message.call_args[0][0])
 
@@ -1239,8 +1161,8 @@ class UnregisterTest(unittest.TestCase):
         mocked_storage.find_all.return_value = []
         self.mocks['account'].AccountFileStorage.return_value = mocked_storage
 
-        acme_client = mock.MagicMock()
-        self.mocks['client'].Client.return_value = acme_client
+        cb_client = mock.MagicMock()
+        self.mocks['client'].Client.return_value = cb_client
 
         config = mock.MagicMock()
         unused_plugins = mock.MagicMock()
@@ -1248,61 +1170,7 @@ class UnregisterTest(unittest.TestCase):
         res = main.unregister(config, unused_plugins)
         m = "Could not find existing account to deactivate."
         self.assertEqual(res, m)
-        self.assertFalse(acme_client.acme.deactivate_registration.called)
-
-
-class TestHandleException(unittest.TestCase):
-    """Test main._handle_exception"""
-    @mock.patch('certbot.main.sys')
-    def test_handle_exception(self, mock_sys):
-        # pylint: disable=protected-access
-        from acme import messages
-
-        config = mock.MagicMock()
-        mock_open = mock.mock_open()
-
-        with mock.patch('certbot.main.open', mock_open, create=True):
-            exception = Exception('detail')
-            config.verbose_count = 1
-            main._handle_exception(
-                Exception, exc_value=exception, trace=None, config=None)
-            mock_open().write.assert_any_call(''.join(
-                traceback.format_exception_only(Exception, exception)))
-            error_msg = mock_sys.exit.call_args_list[0][0][0]
-            self.assertTrue('unexpected error' in error_msg)
-
-        with mock.patch('certbot.main.open', mock_open, create=True):
-            mock_open.side_effect = [KeyboardInterrupt]
-            error = errors.Error('detail')
-            main._handle_exception(
-                errors.Error, exc_value=error, trace=None, config=None)
-            # assert_any_call used because sys.exit doesn't exit in cli.py
-            mock_sys.exit.assert_any_call(''.join(
-                traceback.format_exception_only(errors.Error, error)))
-
-        bad_typ = messages.ERROR_PREFIX + 'triffid'
-        exception = messages.Error(detail='alpha', typ=bad_typ, title='beta')
-        config = mock.MagicMock(debug=False, verbose_count=-3)
-        main._handle_exception(
-            messages.Error, exc_value=exception, trace=None, config=config)
-        error_msg = mock_sys.exit.call_args_list[-1][0][0]
-        self.assertTrue('unexpected error' in error_msg)
-        self.assertTrue('acme:error' not in error_msg)
-        self.assertTrue('alpha' in error_msg)
-        self.assertTrue('beta' in error_msg)
-        config = mock.MagicMock(debug=False, verbose_count=1)
-        main._handle_exception(
-            messages.Error, exc_value=exception, trace=None, config=config)
-        error_msg = mock_sys.exit.call_args_list[-1][0][0]
-        self.assertTrue('unexpected error' in error_msg)
-        self.assertTrue('acme:error' in error_msg)
-        self.assertTrue('alpha' in error_msg)
-
-        interrupt = KeyboardInterrupt('detail')
-        main._handle_exception(
-            KeyboardInterrupt, exc_value=interrupt, trace=None, config=None)
-        mock_sys.exit.assert_called_with(''.join(
-            traceback.format_exception_only(KeyboardInterrupt, interrupt)))
+        self.assertFalse(cb_client.acme.deactivate_registration.called)
 
 
 if __name__ == '__main__':

certbot/tests/ocsp_test.py

@@ -28,7 +28,7 @@ class OCSPTest(unittest.TestCase):
     def tearDown(self):
         pass
 
-    @mock.patch('certbot.ocsp.logging.info')
+    @mock.patch('certbot.ocsp.logger.info')
     @mock.patch('certbot.ocsp.Popen')
     @mock.patch('certbot.util.exe_exists')
     def test_init(self, mock_exists, mock_popen, mock_log):

certbot/tests/renewal_test.py

@@ -2,8 +2,6 @@
 import os
 import mock
 import unittest
-import shutil
-import tempfile
 
 from acme import challenges
 
@@ -14,13 +12,11 @@ from certbot import storage
 from certbot.tests import util
 
 
-class RenewalTest(unittest.TestCase):
+class RenewalTest(util.TempDirTestCase):
     def setUp(self):
-        self.tmp_dir = tempfile.mkdtemp()
-        self.config_dir = os.path.join(self.tmp_dir, 'config')
+        super(RenewalTest, self).setUp()
 
-    def tearDown(self):
-        shutil.rmtree(self.tmp_dir)
+        self.config_dir = os.path.join(self.tempdir, 'config')
 
     @mock.patch('certbot.cli.set_by_cli')
     def test_ancient_webroot_renewal_conf(self, mock_set_by_cli):
@@ -52,7 +48,7 @@ class RestoreRequiredConfigElementsTest(unittest.TestCase):
     def test_allow_subset_of_names_success(self, mock_set_by_cli):
         mock_set_by_cli.return_value = False
         self._call(self.config, {'allow_subset_of_names': 'True'})
-        self.assertTrue(self.config.namespace.allow_subset_of_names is True)
+        self.assertTrue(self.config.allow_subset_of_names is True)
 
     @mock.patch('certbot.renewal.cli.set_by_cli')
     def test_allow_subset_of_names_failure(self, mock_set_by_cli):
@@ -68,7 +64,7 @@ class RestoreRequiredConfigElementsTest(unittest.TestCase):
         self._call(self.config, renewalparams)
         expected = [challenges.TLSSNI01.typ,
                     challenges.HTTP01.typ, challenges.DNS01.typ]
-        self.assertEqual(self.config.namespace.pref_challs, expected)
+        self.assertEqual(self.config.pref_challs, expected)
 
     @mock.patch('certbot.renewal.cli.set_by_cli')
     def test_pref_challs_str(self, mock_set_by_cli):
@@ -76,7 +72,7 @@ class RestoreRequiredConfigElementsTest(unittest.TestCase):
         renewalparams = {'pref_challs': 'dns'}
         self._call(self.config, renewalparams)
         expected = [challenges.DNS01.typ]
-        self.assertEqual(self.config.namespace.pref_challs, expected)
+        self.assertEqual(self.config.pref_challs, expected)
 
     @mock.patch('certbot.renewal.cli.set_by_cli')
     def test_pref_challs_failure(self, mock_set_by_cli):
@@ -88,7 +84,7 @@ class RestoreRequiredConfigElementsTest(unittest.TestCase):
     def test_must_staple_success(self, mock_set_by_cli):
         mock_set_by_cli.return_value = False
         self._call(self.config, {'must_staple': 'True'})
-        self.assertTrue(self.config.namespace.must_staple is True)
+        self.assertTrue(self.config.must_staple is True)
 
     @mock.patch('certbot.renewal.cli.set_by_cli')
     def test_must_staple_failure(self, mock_set_by_cli):

certbot/tests/reporter_test.py

@@ -38,18 +38,6 @@ class ReporterTest(unittest.TestCase):
             self.reporter.print_messages()
         self.assertEqual(sys.stdout.getvalue(), "")
 
-    @mock.patch('certbot.reporter.os.getpid')
-    def test_atexit_print_messages(self, mock_getpid):
-        self._add_messages()
-        mock_getpid.return_value = 42
-        with mock.patch('certbot.reporter.INITIAL_PID', 42):
-            self.reporter.atexit_print_messages()
-        output = sys.stdout.getvalue()
-        self.assertTrue("IMPORTANT NOTES:" in output)
-        self.assertTrue("High" in output)
-        self.assertTrue("Med" in output)
-        self.assertTrue("Low" in output)
-
     def test_tty_successful_exit(self):
         sys.stdout.isatty = lambda: True
         self._successful_exit_common()

certbot/tests/storage_test.py

@@ -3,7 +3,6 @@
 import datetime
 import os
 import shutil
-import tempfile
 import unittest
 
 import configobj
@@ -36,7 +35,7 @@ def fill_with_sample_data(rc_object):
             f.write(kind)
 
 
-class BaseRenewableCertTest(unittest.TestCase):
+class BaseRenewableCertTest(util.TempDirTestCase):
     """Base class for setting up Renewable Cert tests.
 
     .. note:: It may be required to write out self.config for
@@ -47,7 +46,8 @@ class BaseRenewableCertTest(unittest.TestCase):
 
     def setUp(self):
         from certbot import storage
-        self.tempdir = tempfile.mkdtemp()
+
+        super(BaseRenewableCertTest, self).setUp()
 
         self.cli_config = configuration.NamespaceConfig(
             namespace=mock.MagicMock(
@@ -91,9 +91,6 @@ class BaseRenewableCertTest(unittest.TestCase):
             check.return_value = True
             self.test_rc = storage.RenewableCert(config.filename, self.cli_config)
 
-    def tearDown(self):
-        shutil.rmtree(self.tempdir)
-
     def _write_out_kind(self, kind, ver, value=None):
         link = getattr(self.test_rc, kind)
         if os.path.lexists(link):
@@ -492,8 +489,8 @@ class RenewableCertTests(BaseRenewableCertTest):
                 self._write_out_kind(kind, ver)
         self.test_rc.update_all_links_to(3)
         self.assertEqual(
-            6, self.test_rc.save_successor(3, "new cert", None,
-                                           "new chain", self.cli_config))
+            6, self.test_rc.save_successor(3, b'new cert', None,
+                                           b'new chain', self.cli_config))
         with open(self.test_rc.version("cert", 6)) as f:
             self.assertEqual(f.read(), "new cert")
         with open(self.test_rc.version("chain", 6)) as f:
@@ -505,11 +502,11 @@ class RenewableCertTests(BaseRenewableCertTest):
         self.assertTrue(os.path.islink(self.test_rc.version("privkey", 6)))
         # Let's try two more updates
         self.assertEqual(
-            7, self.test_rc.save_successor(6, "again", None,
-                                           "newer chain", self.cli_config))
+            7, self.test_rc.save_successor(6, b'again', None,
+                                           b'newer chain', self.cli_config))
         self.assertEqual(
-            8, self.test_rc.save_successor(7, "hello", None,
-                                           "other chain", self.cli_config))
+            8, self.test_rc.save_successor(7, b'hello', None,
+                                           b'other chain', self.cli_config))
         # All of the subsequent versions should link directly to the original
         # privkey.
         for i in (6, 7, 8):
@@ -523,8 +520,8 @@ class RenewableCertTests(BaseRenewableCertTest):
         # Test updating from latest version rather than old version
         self.test_rc.update_all_links_to(8)
         self.assertEqual(
-            9, self.test_rc.save_successor(8, "last", None,
-                                           "attempt", self.cli_config))
+            9, self.test_rc.save_successor(8, b'last', None,
+                                           b'attempt', self.cli_config))
         for kind in ALL_FOUR:
             self.assertEqual(self.test_rc.available_versions(kind),
                              list(six.moves.range(1, 10)))
@@ -538,8 +535,8 @@ class RenewableCertTests(BaseRenewableCertTest):
         # Test updating when providing a new privkey.  The key should
         # be saved in a new file rather than creating a new symlink.
         self.assertEqual(
-            10, self.test_rc.save_successor(9, "with", "a",
-                                            "key", self.cli_config))
+            10, self.test_rc.save_successor(9, b'with', b'a',
+                                            b'key', self.cli_config))
         self.assertTrue(os.path.exists(self.test_rc.version("privkey", 10)))
         self.assertFalse(os.path.islink(self.test_rc.version("privkey", 10)))
         self.assertFalse(os.path.exists(temp_config_file))
@@ -581,6 +578,15 @@ class RenewableCertTests(BaseRenewableCertTest):
         self.assertEqual(
             self._test_relevant_values_common(values), values)
 
+    @mock.patch("certbot.cli.set_by_cli")
+    @mock.patch("certbot.plugins.disco.PluginsRegistry.find_all")
+    def test_relevant_values_namespace(self, mock_find_all, mock_set_by_cli):
+        mock_set_by_cli.return_value = True
+        mock_find_all.return_value = ["certbot-foo:bar"]
+        values = {"certbot_foo:bar_baz": 42}
+        self.assertEqual(
+            self._test_relevant_values_common(values), values)
+
     @mock.patch("certbot.storage.relevant_values")
     def test_new_lineage(self, mock_rv):
         """Test for new_lineage() class method."""
@@ -789,6 +795,7 @@ class DeleteFilesTest(BaseRenewableCertTest):
     """Tests for certbot.storage.delete_files"""
     def setUp(self):
         super(DeleteFilesTest, self).setUp()
+
         for kind in ALL_FOUR:
             kind_path = os.path.join(self.tempdir, "live", "example.org",
                                         kind + ".pem")

certbot/tests/util.py

@@ -6,6 +6,7 @@
 import os
 import pkg_resources
 import shutil
+import tempfile
 import unittest
 
 from cryptography.hazmat.backends import default_backend
@@ -230,3 +231,13 @@ def _assert_valid_call(*args, **kwargs):
 
     # pylint: disable=star-args
     display_util.assert_valid_call(*assert_args, **assert_kwargs)
+
+
+class TempDirTestCase(unittest.TestCase):
+    """Base test class which sets up and tears down a temporary directory"""
+
+    def setUp(self):
+        self.tempdir = tempfile.mkdtemp()
+
+    def tearDown(self):
+        shutil.rmtree(self.tempdir)

certbot/tests/util_test.py

@@ -2,9 +2,7 @@
 import argparse
 import errno
 import os
-import shutil
 import stat
-import tempfile
 import unittest
 
 import mock
@@ -75,7 +73,26 @@ class ExeExistsTest(unittest.TestCase):
         self.assertFalse(self._call("exe"))
 
 
-class MakeOrVerifyDirTest(unittest.TestCase):
+class MakeOrVerifyCoreDirTest(test_util.TempDirTestCase):
+    """Tests for certbot.util.make_or_verify_core_dir."""
+
+    def _call(self, *args, **kwargs):
+        from certbot.util import make_or_verify_core_dir
+        return make_or_verify_core_dir(*args, **kwargs)
+
+    def test_success(self):
+        new_dir = os.path.join(self.tempdir, 'new')
+        self._call(new_dir, 0o700, os.geteuid(), False)
+        self.assertTrue(os.path.exists(new_dir))
+
+    @mock.patch('certbot.main.util.make_or_verify_dir')
+    def test_failure(self, mock_make_or_verify):
+        mock_make_or_verify.side_effect = OSError
+        self.assertRaises(errors.Error, self._call,
+                          self.tempdir, 0o700, os.geteuid(), False)
+
+
+class MakeOrVerifyDirTest(test_util.TempDirTestCase):
     """Tests for certbot.util.make_or_verify_dir.
 
     Note that it is not possible to test for a wrong directory owner,
@@ -84,21 +101,19 @@ class MakeOrVerifyDirTest(unittest.TestCase):
     """
 
     def setUp(self):
-        self.root_path = tempfile.mkdtemp()
-        self.path = os.path.join(self.root_path, "foo")
+        super(MakeOrVerifyDirTest, self).setUp()
+
+        self.path = os.path.join(self.tempdir, "foo")
         os.mkdir(self.path, 0o400)
 
         self.uid = os.getuid()
 
-    def tearDown(self):
-        shutil.rmtree(self.root_path, ignore_errors=True)
-
     def _call(self, directory, mode):
         from certbot.util import make_or_verify_dir
         return make_or_verify_dir(directory, mode, self.uid, strict=True)
 
     def test_creates_dir_when_missing(self):
-        path = os.path.join(self.root_path, "bar")
+        path = os.path.join(self.tempdir, "bar")
         self._call(path, 0o650)
         self.assertTrue(os.path.isdir(path))
         self.assertEqual(stat.S_IMODE(os.stat(path).st_mode), 0o650)
@@ -116,7 +131,7 @@ class MakeOrVerifyDirTest(unittest.TestCase):
             self.assertRaises(OSError, self._call, "bar", 12312312)
 
 
-class CheckPermissionsTest(unittest.TestCase):
+class CheckPermissionsTest(test_util.TempDirTestCase):
     """Tests for certbot.util.check_permissions.
 
     Note that it is not possible to test for a wrong file owner,
@@ -125,34 +140,30 @@ class CheckPermissionsTest(unittest.TestCase):
     """
 
     def setUp(self):
-        _, self.path = tempfile.mkstemp()
-        self.uid = os.getuid()
+        super(CheckPermissionsTest, self).setUp()
 
-    def tearDown(self):
-        os.remove(self.path)
+        self.uid = os.getuid()
 
     def _call(self, mode):
         from certbot.util import check_permissions
-        return check_permissions(self.path, mode, self.uid)
+        return check_permissions(self.tempdir, mode, self.uid)
 
     def test_ok_mode(self):
-        os.chmod(self.path, 0o600)
+        os.chmod(self.tempdir, 0o600)
         self.assertTrue(self._call(0o600))
 
     def test_wrong_mode(self):
-        os.chmod(self.path, 0o400)
+        os.chmod(self.tempdir, 0o400)
         self.assertFalse(self._call(0o600))
 
 
-class UniqueFileTest(unittest.TestCase):
+class UniqueFileTest(test_util.TempDirTestCase):
     """Tests for certbot.util.unique_file."""
 
     def setUp(self):
-        self.root_path = tempfile.mkdtemp()
-        self.default_name = os.path.join(self.root_path, "foo.txt")
+        super(UniqueFileTest, self).setUp()
 
-    def tearDown(self):
-        shutil.rmtree(self.root_path, ignore_errors=True)
+        self.default_name = os.path.join(self.tempdir, "foo.txt")
 
     def _call(self, mode=0o600):
         from certbot.util import unique_file
@@ -177,9 +188,9 @@ class UniqueFileTest(unittest.TestCase):
         self.assertNotEqual(name1, name3)
         self.assertNotEqual(name2, name3)
 
-        self.assertEqual(os.path.dirname(name1), self.root_path)
-        self.assertEqual(os.path.dirname(name2), self.root_path)
-        self.assertEqual(os.path.dirname(name3), self.root_path)
+        self.assertEqual(os.path.dirname(name1), self.tempdir)
+        self.assertEqual(os.path.dirname(name2), self.tempdir)
+        self.assertEqual(os.path.dirname(name3), self.tempdir)
 
         basename1 = os.path.basename(name2)
         self.assertTrue(basename1.endswith("foo.txt"))
@@ -193,26 +204,20 @@ try:
     file_type = file
 except NameError:
     import io
-    file_type = io.TextIOWrapper
+    file_type = io.TextIOWrapper  # type: ignore
 
 
-class UniqueLineageNameTest(unittest.TestCase):
+class UniqueLineageNameTest(test_util.TempDirTestCase):
     """Tests for certbot.util.unique_lineage_name."""
 
-    def setUp(self):
-        self.root_path = tempfile.mkdtemp()
-
-    def tearDown(self):
-        shutil.rmtree(self.root_path, ignore_errors=True)
-
     def _call(self, filename, mode=0o777):
         from certbot.util import unique_lineage_name
-        return unique_lineage_name(self.root_path, filename, mode)
+        return unique_lineage_name(self.tempdir, filename, mode)
 
     def test_basic(self):
         f, path = self._call("wow")
         self.assertTrue(isinstance(f, file_type))
-        self.assertEqual(os.path.join(self.root_path, "wow.conf"), path)
+        self.assertEqual(os.path.join(self.tempdir, "wow.conf"), path)
 
     def test_multiple(self):
         for _ in six.moves.range(10):
@@ -237,15 +242,13 @@ class UniqueLineageNameTest(unittest.TestCase):
         self.assertRaises(OSError, self._call, "wow")
 
 
-class SafelyRemoveTest(unittest.TestCase):
+class SafelyRemoveTest(test_util.TempDirTestCase):
     """Tests for certbot.util.safely_remove."""
 
     def setUp(self):
-        self.tmp = tempfile.mkdtemp()
-        self.path = os.path.join(self.tmp, "foo")
+        super(SafelyRemoveTest, self).setUp()
 
-    def tearDown(self):
-        shutil.rmtree(self.tmp)
+        self.path = os.path.join(self.tempdir, "foo")
 
     def _call(self):
         from certbot.util import safely_remove
@@ -489,5 +492,37 @@ class OsInfoTest(unittest.TestCase):
                                      ("windows", "95"))
 
 
+class AtexitRegisterTest(unittest.TestCase):
+    """Tests for certbot.util.atexit_register."""
+    def setUp(self):
+        self.func = mock.MagicMock()
+        self.args = ('hi',)
+        self.kwargs = {'answer': 42}
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.util import atexit_register
+        return atexit_register(*args, **kwargs)
+
+    def test_called(self):
+        self._test_common(os.getpid())
+        self.func.assert_called_with(*self.args, **self.kwargs)
+
+    def test_not_called(self):
+        self._test_common(initial_pid=-1)
+        self.assertFalse(self.func.called)
+
+    def _test_common(self, initial_pid):
+        with mock.patch('certbot.util._INITIAL_PID', initial_pid):
+            with mock.patch('certbot.util.atexit') as mock_atexit:
+                self._call(self.func, *self.args, **self.kwargs)
+
+            # _INITAL_PID must be mocked when calling atexit_func
+            self.assertTrue(mock_atexit.register.called)
+            args, kwargs = mock_atexit.register.call_args
+            atexit_func = args[0]
+            atexit_func(*args[1:], **kwargs)  # pylint: disable=star-args
+
+
 if __name__ == "__main__":
     unittest.main()  # pragma: no cover

certbot/util.py

@@ -1,5 +1,6 @@
 """Utilities for all Certbot."""
 import argparse
+import atexit
 import collections
 # distutils.version under virtualenv confuses pylint
 # For more info, see: https://github.com/PyCQA/pylint/issues/73
@@ -38,6 +39,16 @@ ANSI_SGR_RED = "\033[31m"
 ANSI_SGR_RESET = "\033[0m"
 
 
+PERM_ERR_FMT = os.linesep.join((
+    "The following error was encountered:", "{0}",
+    "If running as non-root, set --config-dir, "
+    "--work-dir, and --logs-dir to writeable paths."))
+
+
+# Stores importing process ID to be used by atexit_register()
+_INITIAL_PID = os.getpid()
+
+
 def run_script(params, log=logger.error):
     """Run the script with the given params.
 
@@ -92,6 +103,23 @@ def exe_exists(exe):
     return False
 
 
+def make_or_verify_core_dir(directory, mode, uid, strict):
+    """Make sure directory exists with proper permissions.
+
+    :param str directory: Path to a directory.
+    :param int mode: Directory mode.
+    :param int uid: Directory owner.
+    :param bool strict: require directory to be owned by current user
+
+    :raises .errors.Error: if the directory cannot be made or verified
+
+    """
+    try:
+        make_or_verify_dir(directory, mode, uid, strict)
+    except OSError as error:
+        raise errors.Error(PERM_ERR_FMT.format(error))
+
+
 def make_or_verify_dir(directory, mode=0o755, uid=0, strict=False):
     """Make sure directory exists with proper permissions.
 
@@ -533,3 +561,20 @@ def is_staging(srv):
     :rtype bool:
     """
     return srv == constants.STAGING_URI or "staging" in srv
+
+
+def atexit_register(func, *args, **kwargs):
+    """Sets func to be called before the program exits.
+
+    Special care is taken to ensure func is only called when the process
+    that first imports this module exits rather than any child processes.
+
+    :param function func: function to be called in case of an error
+
+    """
+    atexit.register(_atexit_call, func, *args, **kwargs)
+
+
+def _atexit_call(func, *args, **kwargs):
+    if _INITIAL_PID == os.getpid():
+        func(*args, **kwargs)

docker-compose.yml

@@ -1,14 +1,18 @@
-production:
-  build: .
-  ports:
-    - "443:443"
+version: '2'
+services:
+  production:
+    build: .
+    ports:
+      - "443:443"
 
 # For development, mount git root to /opt/certbot/src in order to
 # make the dev workflow more vagrant-like.
-development:
-  build: .
-  ports:
-    - "443:443"
-  volumes:
-    - .:/opt/certbot/src
-    - /opt/certbot/venv
+  development:
+    build:
+      context: .
+      dockerfile: Dockerfile-dev
+    ports:
+      - "443:443"
+    volumes:
+      - .:/opt/certbot/src
+      - /opt/certbot/venv

docs/challenges.rst

@@ -0,0 +1,156 @@
+Digital certificates can only be issued to people who are entitled to them. For example, assuming you don't run google.com, you're not entitled to a certificate for it. Nor is someone else entitled to receive a certificate for your web site.
+
+In order to receive a certificate from Let's Encrypt certificate authority (CA), you have to prove your control over each of the domain names that will be listed in the certificate. You can do so by making certain publicly-visible changes, proving that the person who's requested a particular certificate is the same person who controls the site(s) that the certificate will refer to.
+
+Let’s Encrypt specifies three different ways to prove your control over a domain (each of which Certbot may be able to do for you). These are called "challenges," because you are being challenged to perform tasks that only someone who controls the domain should be able to accomplish.
+
+When you use Certbot, it will attempt to help you prove control over your domains automatically in a way that's acceptable to the CA. Especially if this doesn't work the way you expected, it can be helpful to understand what Certbot is trying to do in each case.
+
+The three ways to prove your control over a domain for the Let’s Encrypt CA are:
+
+* Posting a specified file on a web site
+
+This method is called the HTTP-01 challenge.  In this challenge, the certificate authority will expect a specified file to be posted in a specified location on a web site.  The file will be downloaded using an HTTP request on TCP port 80.  Since part of what this challenge shows is the ability to create a file at an arbitrary location, you cannot choose a different location or port number.
+
+* Offering a specified certificate on a web site
+
+This method is called the TLS-SNI-01 challenge.  In this challenge, the certificate authority will expect a specified digital certificate to be provided by the web server in response to an HTTPS request using a particular made-up domain name.  The request will be made using HTTPS on TCP port 443.  You cannot choose a different port number.
+
+This certificate is a self-signed certificate created by Certbot.  You use it only temporarily to prove your control over a domain name.  It’s not the same as the certificate for your site that will later be issued by Let's Encrypt once you've proven that you control the site.
+
+* Posting a specified DNS record in the domain name system
+
+This method is called the DNS-01 challenge.  In this challenge, the certificate authority will expect a specified DNS record to be present in your DNS zone when queried for.  The record will be a TXT record for a specific subdomain of the name you're proving your control over.
+
+For each kind of challenge, the challenge can potentially be completed *automatically* (Certbot directly makes the necessary changes itself, or runs another program that does so), or *manually* (Certbot tells you to make a certain change, and you edit a configuration file of some kind in order to accomplish it).  Certbot's design emphasizes performing challenges *automatically*, and this is the normal case for most uses of Certbot.
+
+Some Certbot *plugins* offer the functionality of an *authenticator*, which simply means that they can satisfy challenges. Different plugins can satisfy different kinds of challenges, as follows:
+
+apache plugin: Can only use TLS-SNI-01.  Tries to edit your Apache configuration files in order to temporarily serve a specified Certbot-generated certificate for a specified name.  This can work when you're running Certbot on a web server with an existing installation of Apache that is able to listen on port 443. This makes certain assumptions about your Apache configuration.
+
+nginx plugin: Can only use TLS-SNI-01.  Tries to edit your nginx configuration files in order to temporarily serve a specified Certbot-generated certificate for a specified name.  This can work when you're running Certbot on a web server with an existing installation of nginx that is able to listen on port 443. This makes certain assumptions about your nginx configuration.
+
+webroot plugin: Can only use HTTP-01.  Tries to place a file into an appropriate place in order for that file to be served over HTTP on port 80 by an existing web server running on your system.  This can work when you're running Certbot on a web server with any existing server application that already listens to web requests on port 80, and that serves files from disk in response.
+
+standalone plugin: Can use either TLS-SNI-01 or HTTP-01.  (You can choose with the `--preferred-challenges` option.)  Tries to run its own temporary web server which will speak either HTTP on port 80 (for HTTP-01) or HTTPS on port 443 (for TLS-SNI-01).  This can work if either of these ports is free to receive incoming connections at the moment that you run Certbot, because there's no existing program listening to them or because you've temporarily shut down any server application that was listening to them.
+
+manual plugin: Can use either DNS-01 or HTTP-01.  May tell you what changes you are expected to make to your configuration.  Or, using an external script, can update your DNS records (for DNS-01) or your webroot (for HTTP-01).  This can work if you have appropriate technical knowledge of how to make these kinds of changes yourself when asked to do so.  Note that this will prevent automated renewal of your certificate using `certbot renew`.  [Can manual also use TLS-SNI-01??]
+
+
+Common problems with passing different challenges
+
+HTTP-01 challenge:
+* (With webroot plugin) You aren't running Certbot on your web server
+
+  Most people should install and run Certbot on their web server hosting their website, not on their laptops or some other computer.  While you can use Certbot in manual mode on a laptop and then separately set up the appropriate files on your webserver, it's not likely to be the most convenient way to get a certificate for most users.
+
+* A domain name you're requesting a certificate for isn't correctly pointed at that web server
+
+  In most cases, every name you're requesting a certificate for should already exist and be pointed to the public IP address of the server where you're requesting that certificate.  (Some alternatives exist for complex network configurations, but they're the exception rather than the rule.)
+
+* A firewall is blocking access to port 80
+
+  The certificate authority needs to be able to connect to port 80 of your server in order to confirm that you satisfied the HTTP-01 challenge.  So that needs to be publicly reachable from the Internet, and not blocked by a router or firewall.
+
+* (With webroot plugin) You specified the webroot directory incorrectly
+
+  If you used `--webroot`, you need to tell Certbot where it can put
+  files in order to have them served by your existing web server.
+  If you said your webroot for example.com was /var/www/example.com,
+  then a file placed in /var/www/example.com/.well-known/acme-challenge/testfile should appear on
+  your web site at http://example.com/.well-known/acme-challenge/testfile (which you can test using a web browser). (A redirection to HTTPS
+  is OK here and should not stop the challenge from working.)
+
+  Note that you should *not* specify the .well-known/acme-challenge directory itself.  Instead, you should specify the top level directory that web content is served from.
+
+* (With webroot plugin) You don't have a webroot directory at all
+
+  In some web server configurations, all pages are dynamically generated by some kind of framework, usually using a database backend.  In this case, there might not be a particular directory that files can be directly served from by the existing web server application.  Using the webroot plugin in this case requires making a change to your web server configuration first.
+
+* (With manual plugin) You updated the webroot directory incorrectly
+
+  If you used `--manual`, you need to know where you can put files in order to have them served by your existing web server. If you think your webroot for example.com is /var/www/example.com, then a file placed in /var/www/example.com/.well-known/acme-challenge/testfile should appear on
+  your web site at http://example.com/.well-known/acme-challenge/testfile.  (A redirection to HTTPS
+  is OK here and should not stop the challenge from working.) You should also make sure that you don't make a typo in the name of the file when creating it.
+
+* Your existing web server's configuration refuses to serve files
+  from /.well-known/acme-challenge, or doesn't serve them at the
+  /.well-known/acme-challenge location on your site, or serves them
+  with a header or footer, or serves them with an unusual MIME type.
+
+* (With standalone plugin)
+  You tried to use `--standalone` when there was already some other
+  program on your server listening to port 80
+
+* (With webroot plugin)
+  You tried to use `--webroot` when you don't have an existing web
+  server listening on port 80
+
+* Your DNS records aren't valid
+  Try checking your DNS records with a tool like the DNSchecker at
+  http://www.dnsstuff.com/ to make sure there are no serious errors.
+  Sometimes a DNS error still allows your site to load in a web
+  browser, but prevents the certificate authority from issuing a
+  certificate.
+
+TLS-SNI-01 challenge:
+* You aren't running Certbot on your web server
+
+  Most people should install and run Certbot on their web server hosting their website, not on their laptops or some other computer.  While you can use Certbot in manual mode on a laptop and then separately set up the appropriate files on your webserver, it's not likely to be the most convenient way to get a certificate for most users.
+
+* A domain name you're requesting a certificate for isn't correctly
+  pointed at that web server
+
+  In most cases, every name you're requesting a certificate for should
+  already exist and be pointed to the server where you're requesting
+  that certificate.  (Some alternatives exist for complex network
+  configurations, but they're the exception rather than the rule.)
+
+* You're using a content delivery network (CDN)
+
+  TLS-SNI-01 doesn't work with CDNs (like CloudFlare and Akamai).  You
+  have to use a different challenge type.  (This is a special case of
+  the previous problem: the domain name is pointed at the CDN, not
+  directly at your server.)
+
+* A firewall is blocking access to port 443
+
+  The certificate authority needs to be able to connect to port 443 of
+  your server in order to confirm that you satisfied the TLS-SNI-01
+  challenge.  So that needs to be publicly reachable from the Internet,
+  and not blocked by a router or firewall.
+
+* (With apache plugin)
+  Certbot thinks you're running Apache, but you aren't running it, or
+  you're running a different server of some kind on port 443
+
+* (With nginx plugin)
+  Certbot thinks you're running nginx, but you aren't running it, or
+  you're running a different server of some kind on port 443
+
+* (With apache or nginx plugin)
+  Certbot doesn't know how to modify your web server configuration correctly
+
+* (With standalone plugin)
+  You tried to use `--standalone` when there was already some other
+  program on your server listening to port 443
+
+* Your DNS records aren't valid
+  Try checking your DNS records with a tool like the DNSchecker at
+  http://www.dnsstuff.com/ to make sure there are no serious errors.
+  Sometimes a DNS error still allows your site to load in a web
+  browser, but prevents the certificate authority from issuing a
+  certificate.
+
+DNS-01 challenge:
+
+* (With manual plugin) Your DNS records weren't correctly updated.
+  You need to be able to make appropriate changes to your DNS zone
+  in order to pass the challenge.
+
+* Your DNS records aren't valid.
+  Try checking your DNS records with a tool like the DNSchecker at
+  http://www.dnsstuff.com/ to make sure there are no serious errors.
+  Sometimes a DNS error still allows your site to load in a web
+  browser, but prevents the certificate authority from issuing a
+  certificate.

docs/ciphers.rst

@@ -255,7 +255,7 @@ I have access to an English-language summary of the recommendations.
 Keylength.com
 ~~~~~~~~~~~~~
 
-Damien Giry collects recommendations by academic researchers and standards organizations about keylengths for particular cryptoperiods, years, or security levels.  The keylength recommendations of the various sources are summarized in a chart.  This site has been updated over time and includes expert guidance from eight sources published between 2000 and 2015.
+Damien Giry collects recommendations by academic researchers and standards organizations about keylengths for particular cryptoperiods, years, or security levels.  The keylength recommendations of the various sources are summarized in a chart.  This site has been updated over time and includes expert guidance from eight sources published between 2000 and 2017.
 
 http://www.keylength.com/
 

docs/cli-help.txt

@@ -70,6 +70,8 @@ optional arguments:
                         because they may be necessary to accurately simulate
                         renewal. --renew-hook commands are not called.
                         (default: False)
+  --debug-challenges    After setting up challenges, wait for user input
+                        before submitting to CA (default: False)
   --preferred-challenges PREF_CHALLS
                         A sorted, comma delimited list of the preferred
                         challenge to use during authorization with the most
@@ -86,7 +88,7 @@ optional arguments:
                         statistics about success rates by OS and plugin. If
                         you wish to hide your server OS version from the Let's
                         Encrypt server, set this to "". (default:
-                        CertbotACMEClient/0.11.1 (Ubuntu 16.04.1 LTS)
+                        CertbotACMEClient/0.13.0 (Ubuntu 16.04.2 LTS)
                         Authenticator/XXX Installer/YYY)
 
 automation:
@@ -97,7 +99,7 @@ automation:
                         keep the existing one until it is due for renewal (for
                         the 'run' subcommand this means reinstall the existing
                         cert). (default: Ask)
-  --expand              If an existing cert covers some subset of the
+  --expand              If an existing cert is a strict subset of the
                         requested names, always expand and replace it with the
                         additional names. (default: Ask)
   --version             show program's version number and exit
@@ -128,6 +130,10 @@ automation:
   --no-self-upgrade     (certbot-auto only) prevent the certbot-auto script
                         from upgrading itself to newer released versions
                         (default: Upgrade automatically)
+  --no-bootstrap        (certbot-auto only) prevent the certbot-auto script
+                        from installing OS-level dependencies (default: Prompt
+                        to install OS-wide dependencies, but exit if the user
+                        says 'No')
   -q, --quiet           Silence all output except errors. Useful for
                         automation via cron. Implies --non-interactive.
                         (default: False)
@@ -193,6 +199,9 @@ paths:
                         installed from, or revoked. (default: None)
   --key-path KEY_PATH   Path to private key for cert installation or
                         revocation (if account key is missing) (default: None)
+  --fullchain-path FULLCHAIN_PATH
+                        Accompanying path to a full certificate chain (cert
+                        plus chain). (default: None)
   --chain-path CHAIN_PATH
                         Accompanying path to a certificate chain. (default:
                         None)
@@ -256,10 +265,12 @@ renew:
                         Command to be run in a shell once for each
                         successfully renewed certificate. For this command,
                         the shell variable $RENEWED_LINEAGE will point to the
-                        config live subdirectory containing the new certs and
-                        keys; the shell variable $RENEWED_DOMAINS will contain
-                        a space-delimited list of renewed cert domains
-                        (default: None)
+                        config live subdirectory (for example,
+                        "/etc/letsencrypt/live/example.com") containing the
+                        new certs and keys; the shell variable
+                        $RENEWED_DOMAINS will contain a space-delimited list
+                        of renewed cert domains (for example,
+                        "example.com www.example.com") (default: None)
   --disable-hook-validation
                         Ordinarily the commands specified for --pre-hook
                         /--post-hook/--renew-hook will be checked for
@@ -315,10 +326,6 @@ unregister:
 install:
   Options for modifying how a cert is deployed
 
-  --fullchain-path FULLCHAIN_PATH
-                        Accompanying path to a full certificate chain (cert
-                        plus chain). (default: None)
-
 config_changes:
   Options for controlling which changes are displayed
 

docs/contributing.rst

@@ -14,15 +14,25 @@ Getting Started
 Running a local copy of the client
 ----------------------------------
 
-Running the client in developer mode from your local tree is a little
-different than running ``certbot-auto``.  To get set up, do these things
-once:
+Running the client in developer mode from your local tree is a little different
+than running Certbot as a user. To get set up, clone our git repository by
+running:
 
 .. code-block:: shell
 
    git clone https://github.com/certbot/certbot
+
+If you're on macOS, we recommend you skip the rest of this section and instead
+run Certbot in Docker. You can find instructions for how to do this :ref:`here
+<docker>`. If you're running on Linux, you can run the following commands to
+install dependencies and set up a virtual environment where you can run
+Certbot. You will need to repeat this when Certbot's dependencies change or when
+a new plugin is introduced.
+
+.. code-block:: shell
+
    cd certbot
-   ./letsencrypt-auto-source/letsencrypt-auto --os-packages-only
+   ./certbot-auto --os-packages-only
    ./tools/venv.sh
 
 Then in each shell where you're working on the client, do:
@@ -30,17 +40,18 @@ Then in each shell where you're working on the client, do:
 .. code-block:: shell
 
    source ./venv/bin/activate
+   export SERVER=https://acme-staging.api.letsencrypt.org/directory
+   source tests/integration/_common.sh
 
 After that, your shell will be using the virtual environment, and you run the
-client by typing:
+client by typing `certbot` or `certbot_test`. The latter is an alias that
+includes several flags useful for testing. For instance, it sets various output
+directories to point to /tmp/, and uses non-privileged ports for challenges, so
+root privileges are not required.
 
-.. code-block:: shell
-
-   certbot
-
-Activating a shell in this way makes it easier to run unit tests
-with ``tox`` and integration tests, as described below. To reverse this, you
-can type ``deactivate``.  More information can be found in the `virtualenv docs`_.
+Activating a shell with `venv/bin/activate` sets environment variables so that
+Python pulls in the correct versions of various packages needed by Certbot.
+More information can be found in the `virtualenv docs`_.
 
 .. _`virtualenv docs`: https://virtualenv.pypa.io
 
@@ -69,10 +80,8 @@ either in the same directory as ``foo.py`` or in the ``tests`` subdirectory
 (if there isn't, make one). While you are working on your code and tests, run
 ``python foo_test.py`` to run the relevant tests.
 
-For debugging, we recommend running ``pip install ipdb`` and putting
-``import ipdb; ipdb.set_trace()`` statements inside the source
-code. Alternatively, you can use Python's standard library `pdb`,
-but you won't get TAB completion.
+For debugging, we recommend putting
+``import ipdb; ipdb.set_trace()`` statements inside the source code.
 
 Once you are done with your code changes, and the tests in ``foo_test.py`` pass,
 run all of the unittests for Certbot with ``tox -e py27`` (this uses Python
@@ -109,6 +118,14 @@ and working. Fetch and start Boulder using:
 If you have problems with Docker, you may want to try `removing all containers and
 volumes`_ and making sure you have at least 1GB of memory.
 
+Set up a certbot_test alias that enables easily running against the local
+Boulder:
+
+.. code-block:: shell
+
+   export SERVER=http://localhost:4000/directory
+   source tests/integration/_common.sh
+
 Run the integration tests using:
 
 .. code-block:: shell
@@ -138,13 +155,15 @@ different webservers, other TLS servers, and operating systems.
 The interfaces available for plugins to implement are defined in
 `interfaces.py`_ and `plugins/common.py`_.
 
-The most common kind of plugin is a "Configurator", which is likely to
-implement the `~certbot.interfaces.IAuthenticator` and
-`~certbot.interfaces.IInstaller` interfaces (though some
-Configurators may implement just one of those).
+The main two plugin interfaces are `~certbot.interfaces.IAuthenticator`, which
+implements various ways of proving domain control to a certificate authority,
+and `~certbot.interfaces.IInstaller`, which configures a server to use a
+certificate once it is issued. Some plugins, like the built-in Apache and Nginx
+plugins, implement both interfaces and perform both tasks. Others, like the
+built-in Standalone authenticator, implement just one interface.
 
 There are also `~certbot.interfaces.IDisplay` plugins,
-which implement bindings to alternative UI libraries.
+which can change how prompts are displayed to a user.
 
 .. _interfaces.py: https://github.com/certbot/certbot/blob/master/certbot/interfaces.py
 .. _plugins/common.py: https://github.com/certbot/certbot/blob/master/certbot/plugins/common.py#L34
@@ -153,27 +172,20 @@ which implement bindings to alternative UI libraries.
 Authenticators
 --------------
 
-Authenticators are plugins designed to prove that this client deserves a
-certificate for some domain name by solving challenges received from
-the ACME server. From the protocol, there are essentially two
-different types of challenges. Challenges that must be solved by
-individual plugins in order to satisfy domain validation (subclasses
-of `~.DVChallenge`, i.e. `~.challenges.TLSSNI01`,
-`~.challenges.HTTP01`, `~.challenges.DNS`) and continuity specific
-challenges (subclasses of `~.ContinuityChallenge`,
-i.e. `~.challenges.RecoveryToken`, `~.challenges.RecoveryContact`,
-`~.challenges.ProofOfPossession`). Continuity challenges are
-always handled by the `~.ContinuityAuthenticator`, while plugins are
-expected to handle `~.DVChallenge` types.
-Right now, we have two authenticator plugins, the `~.ApacheConfigurator`
-and the `~.StandaloneAuthenticator`. The Standalone and Apache
-authenticators only solve the `~.challenges.TLSSNI01` challenge currently.
-(You can set which challenges your authenticator can handle through the
-:meth:`~.IAuthenticator.get_chall_pref`.
+Authenticators are plugins that prove control of a domain name by solving a
+challenge provided by the ACME server. ACME currently defines three types of
+challenges: HTTP, TLS-SNI, and DNS, represented by classes in `acme.challenges`.
+An authenticator plugin should implement support for at least one challenge type.
 
-(FYI: We also have a partial implementation for a `~.DNSAuthenticator`
-in a separate branch).
+An Authenticator indicates which challenges it supports by implementing
+`get_chall_pref(domain)` to return a sorted list of challenge types in
+preference order.
 
+An Authenticator must also implement `perform(achalls)`, which "performs" a list
+of challenges by, for instance, provisioning a file on an HTTP server, or
+setting a TXT record in DNS. Once all challenges have succeeded or failed,
+Certbot will call the plugin's `cleanup(achalls)` method to remove any files or
+DNS records that were needed only during authentication.
 
 Installer
 ---------
@@ -211,24 +223,40 @@ Augeas may still find the `~.Reverter` class helpful in handling
 configuration checkpoints and rollback.
 
 
-Display
-~~~~~~~
-
-We currently only offer a "text" mode for displays. Display plugins
-implement the `~certbot.interfaces.IDisplay` interface.
-
 .. _dev-plugin:
 
 Writing your own plugin
-=======================
+~~~~~~~~~~~~~~~~~~~~~~~
 
 Certbot client supports dynamic discovery of plugins through the
-`setuptools entry points`_. This way you can, for example, create a
-custom implementation of `~certbot.interfaces.IAuthenticator` or
-the `~certbot.interfaces.IInstaller` without having to merge it
+`setuptools entry points`_ using the `certbot.plugins` group. This
+way you can, for example, create a custom implementation of
+`~certbot.interfaces.IAuthenticator` or the
+`~certbot.interfaces.IInstaller` without having to merge it
 with the core upstream source code. An example is provided in
 ``examples/plugins/`` directory.
 
+While developing, you can install your plugin into a Certbot development
+virtualenv like this:
+
+.. code-block:: shell
+
+  . venv/bin/activate
+  . tests/integration/_common.sh
+  pip install -e examples/plugins/
+  certbot_test plugins
+
+Your plugin should show up in the output of the last command. If not,
+it was not installed properly.
+
+Once you've finished your plugin and published it, you can have your
+users install it system-wide with `pip install`. Note that this will
+only work for users who have Certbot installed from OS packages or via
+pip. Users who run `certbot-auto` are currently unable to use third-party
+plugins. It's technically possible to install third-party plugins into
+the virtualenv used by `certbot-auto`, but they will be wiped away when
+`certbot-auto` upgrades.
+
 .. warning:: Please be aware though that as this client is still in a
    developer-preview stage, the API may undergo a few changes. If you
    believe the plugin will be beneficial to the community, please
@@ -285,8 +313,7 @@ Steps:
    including coverage. The ``--skip-missing-interpreters`` argument ignores
    missing versions of Python needed for running the tests. Fix any errors.
 5. If your code touches communication with an ACME server/Boulder, you
-   should run the integration tests, see `integration`_. See `Known Issues`_
-   for some common failures that have nothing to do with your code.
+   should run the integration tests, see `integration`_.
 6. Submit the PR.
 7. Did your tests pass on Travis? If they didn't, fix any errors.
 
@@ -346,56 +373,36 @@ This should generate documentation in the ``docs/_build/html``
 directory.
 
 
-Other methods for running the client
-====================================
+.. _docker:
 
-Vagrant
--------
+Running the client with Docker
+==============================
 
-If you are a Vagrant user, Certbot comes with a Vagrantfile that
-automates setting up a development environment in an Ubuntu 14.04
-LTS VM. To set it up, simply run ``vagrant up``. The repository is
-synced to ``/vagrant``, so you can get started with:
+You can use Docker Compose to quickly set up an environment for running and
+testing Certbot. This is especially useful for macOS users. To install Docker
+Compose, follow the instructions at https://docs.docker.com/compose/install/.
 
-.. code-block:: shell
+.. note:: Linux users can simply run ``pip install docker-compose`` to get
+  Docker Compose after installing Docker Engine and activating your shell as
+  described in the :ref:`Getting Started <getting_started>` section.
 
-  vagrant ssh
-  cd /vagrant
-  sudo ./venv/bin/certbot
+Now you can develop on your host machine, but run Certbot and test your changes
+in Docker. When using ``docker-compose`` make sure you are inside your clone of
+the Certbot repository. As an example, you can run the following command to
+check for linting errors::
 
-Support for other Linux distributions coming soon.
+  docker-compose run --rm --service-ports development bash -c 'tox -e lint'
 
-.. note::
-   Unfortunately, Python distutils and, by extension, setup.py and
-   tox, use hard linking quite extensively. Hard linking is not
-   supported by the default sync filesystem in Vagrant. As a result,
-   all actions with these commands are *significantly slower* in
-   Vagrant. One potential fix is to `use NFS`_ (`related issue`_).
+You can also leave a terminal open running a shell in the Docker container and
+modify Certbot code in another window. The Certbot repo on your host machine is
+mounted inside of the container so any changes you make immediately take
+effect. To do this, run::
 
-.. _use NFS: http://docs.vagrantup.com/v2/synced-folders/nfs.html
-.. _related issue: https://github.com/ClusterHQ/flocker/issues/516
+  docker-compose run --rm --service-ports development bash
 
+Now running the check for linting errors described above is as easy as::
 
-Docker
-------
-
-OSX users will probably find it easiest to set up a Docker container for
-development. Certbot comes with a Dockerfile (``Dockerfile-dev``)
-for doing so. To use Docker on OSX, install and setup docker-machine using the
-instructions at https://docs.docker.com/installation/mac/.
-
-To build the development Docker image::
-
-  docker build -t certbot -f Dockerfile-dev .
-
-Now run tests inside the Docker image:
-
-.. code-block:: shell
-
-  docker run -it certbot bash
-  cd src
-  tox -e py27
-
+  tox -e lint
 
 .. _prerequisites:
 
@@ -432,8 +439,12 @@ For squeeze you will need to:
 FreeBSD
 -------
 
-Package installation for FreeBSD uses ``pkg``, not ports.
+Packages can be installed on FreeBSD using ``pkg``, 
+or any other port-management tool (``portupgrade``, ``portmanager``, etc.) 
+from the pre-built package or can be built and installed from ports. 
+Either way will ensure proper installation of all the dependencies required 
+for the package.
 
 FreeBSD by default uses ``tcsh``. In order to activate virtualenv (see
-below), you will need a compatible shell, e.g. ``pkg install bash &&
+above), you will need a compatible shell, e.g. ``pkg install bash &&
 bash``.

docs/install.rst

@@ -22,9 +22,8 @@ your system.
 System Requirements
 ===================
 
-The Let's Encrypt Client presently only runs on Unix-ish OSes that include
-Python 2.6 or 2.7; Python 3.x support will hopefully be added in the future. The
-client requires root access in order to write to ``/etc/letsencrypt``,
+Certbot currently requires Python 2.6 or 2.7. By default, it requires root
+access in order to write to ``/etc/letsencrypt``,
 ``/var/log/letsencrypt``, ``/var/lib/letsencrypt``; to bind to ports 80 and 443
 (if you use the ``standalone`` plugin) and to read and modify webserver
 configurations (if you use the ``apache`` or ``nginx`` plugins).  If none of
@@ -33,11 +32,16 @@ but for most users who want to avoid running an ACME client as root, either
 `letsencrypt-nosudo <https://github.com/diafygi/letsencrypt-nosudo>`_ or
 `simp_le <https://github.com/kuba/simp_le>`_ are more appropriate choices.
 
-The Apache plugin currently requires OS with augeas version 1.0; currently `it
+The Apache plugin currently requires an OS with augeas version 1.0; currently `it
 supports
 <https://github.com/certbot/certbot/blob/master/certbot-apache/certbot_apache/constants.py>`_
 modern OSes based on Debian, Fedora, SUSE, Gentoo and Darwin.
 
+Installing with ``certbot-auto`` requires 512MB of RAM in order to build some
+of the dependencies. Installing from pre-built OS packages avoids this
+requirement. You can also temporarily set a swap file. See "Problems with
+Python virtual environment" below for details.
+
 Alternate installation methods
 ================================
 
@@ -76,7 +80,7 @@ For full command line help, you can type::
 Problems with Python virtual environment
 ----------------------------------------
 
-On a low memory system such as VPS with only 256MB of RAM, the required dependencies of Certbot will failed to build.
+On a low memory system such as VPS with less than 512MB of RAM, the required dependencies of Certbot will failed to build.
 This can be identified if the pip outputs contains something like ``internal compiler error: Killed (program cc1)``.
 You can workaround this restriction by creating a temporary swapfile::
 
@@ -120,7 +124,7 @@ to, `install Docker`_, then issue the following command:
    sudo docker run -it --rm -p 443:443 -p 80:80 --name certbot \
                -v "/etc/letsencrypt:/etc/letsencrypt" \
                -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
-               quay.io/letsencrypt/letsencrypt:latest certonly
+               certbot/certbot certonly
 
 Running Certbot with the ``certonly`` command will obtain a certificate and place it in the directory
 ``/etc/letsencrypt/live`` on your system. Because Certonly cannot install the certificate from
@@ -136,16 +140,6 @@ of the ``/etc/letsencrypt`` directory, see :ref:`where-certs`.
 Operating System Packages
 -------------------------
 
-**FreeBSD**
-
-  * Port: ``cd /usr/ports/security/py-certbot && make install clean``
-  * Package: ``pkg install py27-certbot``
-
-**OpenBSD**
-
-  * Port: ``cd /usr/ports/security/letsencrypt/client && make install clean``
-  * Package: ``pkg_add letsencrypt``
-
 **Arch Linux**
 
 .. code-block:: shell
@@ -178,6 +172,11 @@ repo, if you have not already done so. Then run:
 
     sudo dnf install certbot python2-certbot-apache
 
+**FreeBSD**
+
+  * Port: ``cd /usr/ports/security/py-certbot && make install clean``
+  * Package: ``pkg install py27-certbot``
+
 **Gentoo**
 
 The official Certbot client is available in Gentoo Portage. If you
@@ -213,6 +212,16 @@ For the time being, this is the only way for the Apache plugin to recognise
 the appropriate directives when installing the certificate.
 Note: this change is not required for the other plugins.
 
+**NetBSD**
+
+  * Build from source: ``cd /usr/pkgsrc/security/py-certbot && make install clean``
+  * Install pre-compiled package: ``pkg_add py27-certbot``
+
+**OpenBSD**
+
+  * Port: ``cd /usr/ports/security/letsencrypt/client && make install clean``
+  * Package: ``pkg_add letsencrypt``
+
 **Other Operating Systems**
 
 OS packaging is an ongoing effort. If you'd like to package

docs/packaging.rst

@@ -16,21 +16,19 @@ The following scripts are used in the process:
 
 - https://github.com/letsencrypt/letsencrypt/blob/master/tools/release.sh
 
-We currently version with the following scheme:
+We use git tags to identify releases, using `Semantic Versioning`_. For
+example: `v0.11.1`.
 
-- ``0.1.0``
-- ``0.2.0dev`` for developement in ``master``
-- ``0.2.0`` (only temporarily in ``master``)
-- ...
+.. _`Semantic Versioning`: http://semver.org/
 
 Notes for package maintainers
 =============================
 
-0. Please use our releases, not ``master``!
+0. Please use our tagged releases, not ``master``!
 
 1. Do not package ``certbot-compatibility-test`` or ``letshelp-certbot`` - it's only used internally.
 
-2. If you'd like to include automated renewal in your package ``certbot renew -q`` should be added to crontab or systemd timer.
+2. If you'd like to include automated renewal in your package ``certbot renew -q`` should be added to crontab or systemd timer. Additionally you should include a random per-machine time offset to avoid having a large number of your clients hit Let's Encrypt's servers simultaneously.
 
 3. ``jws`` is an internal script for ``acme`` module and it doesn't have to be packaged - it's mostly for debugging: you can use it as ``echo foo | jws sign | jws verify``.
 
@@ -44,34 +42,43 @@ Arch
 ----
 
 From our official releases:
+
 - https://www.archlinux.org/packages/community/any/python2-acme
 - https://www.archlinux.org/packages/community/any/certbot
 - https://www.archlinux.org/packages/community/any/certbot-apache
 - https://www.archlinux.org/packages/community/any/certbot-nginx
-- https://www.archlinux.org/packages/community/any/letshelp-certbot
 
 From ``master``: https://aur.archlinux.org/packages/certbot-git
 
 Debian (and its derivatives, including Ubuntu)
 ------
 
-https://packages.debian.org/sid/certbot
-https://packages.debian.org/sid/python-certbot
-https://packages.debian.org/sid/python-certbot-apache
+- https://packages.debian.org/sid/certbot
+- https://packages.debian.org/sid/python-certbot
+- https://packages.debian.org/sid/python-certbot-apache
 
 Fedora
 ------
 
 In Fedora 23+.
 
-- https://admin.fedoraproject.org/pkgdb/package/letsencrypt/
 - https://admin.fedoraproject.org/pkgdb/package/certbot/
 - https://admin.fedoraproject.org/pkgdb/package/python-acme/
 
 FreeBSD
 -------
 
-https://svnweb.freebsd.org/ports/head/security/py-certbot/
+- https://svnweb.freebsd.org/ports/head/security/py-certbot/
+
+Gentoo
+------
+
+Currently, all ``certbot`` related packages are in the testing branch:
+
+- https://packages.gentoo.org/packages/app-crypt/certbot
+- https://packages.gentoo.org/packages/app-crypt/certbot-apache
+- https://packages.gentoo.org/packages/app-crypt/certbot-nginx
+- https://packages.gentoo.org/packages/app-crypt/acme
 
 GNU Guix
 --------