Merge https://github.com/certbot/certbot

Merging so I can begin restructuring work.

.travis.yml

@@ -5,7 +5,7 @@ cache:
         - $HOME/.cache/pip
 
 before_install:
-  - '[ $TRAVIS_OS_NAME == linux ] && dpkg -s libaugeas0 || brew install augeas python3'
+  - '([ $TRAVIS_OS_NAME == linux ] && dpkg -s libaugeas0) || (brew update && brew install augeas python3)'
 
 before_script:
   - 'if [ $TRAVIS_OS_NAME = osx ] ; then ulimit -n 1024 ; fi'
@@ -161,7 +161,9 @@ addons:
     - libapache2-mod-macro
 
 install: "travis_retry pip install tox coveralls"
-script: 'travis_retry tox && ([ "xxx$BOULDER_INTEGRATION" = "xxx" ] || ./tests/travis-integration.sh)'
+script:
+    - travis_retry tox
+    - '[ -z "${BOULDER_INTEGRATION+x}" ] || (travis_retry tests/boulder-fetch.sh && tests/tox-boulder-integration.sh)'
 
 after_success: '[ "$TOXENV" == "cover" ] && coveralls'
 
@@ -169,7 +171,7 @@ notifications:
   email: false
   irc:
     channels:
-      - "chat.freenode.net#letsencrypt-dev"
+      - secure: "SGWZl3ownKx9xKVV2VnGt7DqkTmutJ89oJV9tjKhSs84kLijU6EYdPnllqISpfHMTxXflNZuxtGo0wTDYHXBuZL47w1O32W6nzuXdra5zC+i4sYQwYULUsyfOv9gJX8zWAULiK0Z3r0oho45U+FR5ZN6TPCidi8/eGU+EEPwaAw="
     on_success: never
     on_failure: always
     use_notice: true

CHANGELOG.md

@@ -2,6 +2,114 @@
 
 Certbot adheres to [Semantic Versioning](http://semver.org/).
 
+## 0.19.0 - 2017-10-04
+
+### Added
+
+* Certbot now has renewal hook directories where executable files can be placed
+  for Certbot to run with the renew subcommand. Pre-hooks, deploy-hooks, and
+  post-hooks can be specified in the renewal-hooks/pre, renewal-hooks/deploy,
+  and renewal-hooks/post directories respectively in Certbot's configuration
+  directory (which is /etc/letsencrypt by default). Certbot will automatically
+  create these directories when it is run if they do not already exist.
+* After revoking a certificate with the revoke subcommand, Certbot will offer
+  to delete the lineage associated with the certificate. When Certbot is run
+  with --non-interactive, it will automatically try to delete the associated
+  lineage.
+* When using Certbot's Google Cloud DNS plugin on Google Compute Engine, you no
+  longer have to provide a credential file to Certbot if you have configured
+  sufficient permissions for the instance which Certbot can automatically
+  obtain using Google's metadata service.
+
+### Changed
+
+* When deleting certificates interactively using the delete subcommand, Certbot
+  will now allow you to select multiple lineages to be deleted at once.
+* Certbot's Apache plugin no longer always parses Apache's sites-available on
+  Debian based systems and instead only parses virtual hosts included in your
+  Apache configuration. You can provide an additional directory for Certbot to
+  parse using the command line flag --apache-vhost-root.
+
+### Fixed
+
+* The plugins subcommand can now be run without root access.
+* certbot-auto now includes a timeout when updating itself so it no longer
+  hangs indefinitely when it is unable to connect to the external server.
+* An issue where Certbot's Apache plugin would sometimes fail to deploy a
+  certificate on Debian based systems if mod_ssl wasn't already enabled has
+  been resolved.
+* A bug in our Docker image where the certificates subcommand could not report
+  if certificates maintained by Certbot had been revoked has been fixed.
+* Certbot's RFC 2136 DNS plugin (for use with software like BIND) now properly
+  performs DNS challenges when the domain being verified contains a CNAME
+  record.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/43?closed=1
+
+## 0.18.2 - 2017-09-20
+
+### Fixed
+
+* An issue where Certbot's ACME module would raise an AttributeError trying to
+  create self-signed certificates when used with pyOpenSSL 17.3.0 has been
+  resolved. For Certbot users with this version of pyOpenSSL, this caused
+  Certbot to crash when performing a TLS SNI challenge or when the Nginx plugin
+  tried to create an SSL server block.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/46?closed=1
+
+## 0.18.1 - 2017-09-08
+
+### Fixed
+
+* If certbot-auto was running as an unprivileged user and it upgraded from
+  0.17.0 to 0.18.0, it would crash with a permissions error and would need to
+  be run again to successfully complete the upgrade. This has been fixed and
+  certbot-auto should upgrade cleanly to 0.18.1.
+* Certbot usually uses "certbot-auto" or "letsencrypt-auto" in error messages
+  and the User-Agent string instead of "certbot" when you are using one of
+  these wrapper scripts. Proper detection of this was broken with Certbot's new
+  installation path in /opt in 0.18.0 but this problem has been resolved.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/45?closed=1
+
+## 0.18.0 - 2017-09-06
+
+### Added
+
+* The Nginx plugin now configures Nginx to use 2048-bit Diffie-Hellman
+  parameters. Java 6 clients do not support Diffie-Hellman parameters larger
+  than 1024 bits, so if you need to support these clients you will need to
+  manually modify your Nginx configuration after using the Nginx installer.
+
+### Changed
+
+* certbot-auto now installs Certbot in directories under `/opt/eff.org`. If you
+  had an existing installation from certbot-auto, a symlink is created to the
+  new directory. You can configure certbot-auto to use a different path by
+  setting the environment variable VENV_PATH.
+* The Nginx plugin can now be selected in Certbot's interactive output.
+* Output verbosity of renewal failures when running with `--quiet` has been
+  reduced.
+* The default revocation reason shown in Certbot help output now is a human
+  readable string instead of a numerical code.
+* Plugin selection is now included in normal terminal output.
+
+### Fixed
+
+* A newer version of ConfigArgParse is now installed when using certbot-auto
+  causing values set to false in a Certbot INI configuration file to be handled
+  intuitively. Setting a boolean command line flag to false is equivalent to
+  not including it in the configuration file at all.
+* New naming conventions preventing certbot-auto from installing OS
+  dependencies on Fedora 26 have been resolved.
+
+More details about these changes can be found on our GitHub repo:
+https://github.com/certbot/certbot/milestone/42?closed=1
+
 ## 0.17.0 - 2017-08-02
 
 ### Added

Dockerfile

@@ -12,6 +12,7 @@ COPY certbot src/certbot
 RUN apk add --no-cache --virtual .certbot-deps \
         libffi \
         libssl1.0 \
+        openssl \
         ca-certificates \
         binutils
 RUN apk add --no-cache --virtual .build-deps \

MANIFEST.in

@@ -6,3 +6,4 @@ include linter_plugin.py
 recursive-include docs *
 recursive-include examples *
 recursive-include certbot/tests/testdata *
+include certbot/ssl-dhparams.pem

README.rst

@@ -15,6 +15,9 @@ protocol) that can automate the tasks of obtaining certificates and
 configuring webservers to use them. This client runs on Unix-based operating
 systems.
 
+To see the changes made to Certbot between versions please refer to our
+`changelog <https://github.com/certbot/certbot/blob/master/CHANGELOG.md>`_.
+
 Until May 2016, Certbot was named simply ``letsencrypt`` or ``letsencrypt-auto``,
 depending on install method. Instructions on the Internet, and some pieces of the
 software, may still refer to this older name.

acme/acme/client.py

@@ -11,6 +11,7 @@ import six
 from six.moves import http_client  # pylint: disable=import-error
 
 import OpenSSL
+import re
 import requests
 import sys
 
@@ -599,6 +600,7 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
         return response
 
     def _send_request(self, method, url, *args, **kwargs):
+        # pylint: disable=too-many-locals
         """Send HTTP request.
 
         Makes sure that `verify_ssl` is respected. Logs request and
@@ -624,7 +626,32 @@ class ClientNetwork(object):  # pylint: disable=too-many-instance-attributes
         kwargs.setdefault('headers', {})
         kwargs['headers'].setdefault('User-Agent', self.user_agent)
         kwargs.setdefault('timeout', self._default_timeout)
-        response = self.session.request(method, url, *args, **kwargs)
+        try:
+            response = self.session.request(method, url, *args, **kwargs)
+        except requests.exceptions.RequestException as e:
+            # pylint: disable=pointless-string-statement
+            """Requests response parsing
+
+            The requests library emits exceptions with a lot of extra text.
+            We parse them with a regexp to raise a more readable exceptions.
+
+            Example:
+            HTTPSConnectionPool(host='acme-v01.api.letsencrypt.org',
+            port=443): Max retries exceeded with url: /directory
+            (Caused by NewConnectionError('
+            <requests.packages.urllib3.connection.VerifiedHTTPSConnection
+            object at 0x108356c50>: Failed to establish a new connection:
+            [Errno 65] No route to host',))"""
+
+            # pylint: disable=line-too-long
+            err_regex = r".*host='(\S*)'.*Max retries exceeded with url\: (\/\w*).*(\[Errno \d+\])([A-Za-z ]*)"
+            m = re.match(err_regex, str(e))
+            if m is None:
+                raise # pragma: no cover
+            else:
+                host, path, _err_no, err_msg = m.groups()
+                raise ValueError("Requesting {0}{1}:{2}".format(host, path, err_msg))
+
         # If content is DER, log the base64 of it instead of raw bytes, to keep
         # binary data out of the logs.
         if response.headers.get("Content-Type") == DER_CONTENT_TYPE:

acme/acme/client_test.py

@@ -621,6 +621,21 @@ class ClientNetworkTest(unittest.TestCase):
         self.assertRaises(requests.exceptions.RequestException,
                           self.net._send_request, 'GET', 'uri')
 
+    def test_urllib_error(self):
+        # Using a connection error to test a properly formatted error message
+        try:
+            # pylint: disable=protected-access
+            self.net._send_request('GET', "http://localhost:19123/nonexistent.txt")
+
+        # Value Error Generated Exceptions
+        except ValueError as y:
+            self.assertEqual("Requesting localhost/nonexistent: "
+                             "Connection refused", str(y))
+
+        # Requests Library Exceptions
+        except requests.exceptions.ConnectionError as z: #pragma: no cover
+            self.assertEqual("('Connection aborted.', "
+                             "error(111, 'Connection refused'))", str(z))
 
 class ClientNetworkWithMockedResponseTest(unittest.TestCase):
     """Tests for acme.client.ClientNetwork which mock out response."""

acme/acme/crypto_util.py

@@ -2,6 +2,7 @@
 import binascii
 import contextlib
 import logging
+import os
 import re
 import socket
 import sys
@@ -243,7 +244,7 @@ def gen_ss_cert(key, domains, not_before=None,
     """
     assert domains, "Must provide one or more hostnames for the cert."
     cert = OpenSSL.crypto.X509()
-    cert.set_serial_number(int(binascii.hexlify(OpenSSL.rand.bytes(16)), 16))
+    cert.set_serial_number(int(binascii.hexlify(os.urandom(16)), 16))
     cert.set_version(2)
 
     extensions = [

acme/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [
@@ -20,7 +20,7 @@ install_requires = [
     # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
     # will tolerate; see #2599:
     'setuptools>=1.0',
-    'six',
+    'six>=1.9.0',  # needed for python_2_unicode_compatible
 ]
 
 # env markers cause problems with older pip and setuptools

certbot-apache/certbot_apache/augeas_configurator.py

@@ -3,7 +3,6 @@ import logging
 
 
 from certbot import errors
-from certbot import reverter
 from certbot.plugins import common
 
 from certbot_apache import constants
@@ -11,7 +10,7 @@ from certbot_apache import constants
 logger = logging.getLogger(__name__)
 
 
-class AugeasConfigurator(common.Plugin):
+class AugeasConfigurator(common.Installer):
     """Base Augeas Configurator class.
 
     :ivar config: Configuration.
@@ -33,11 +32,6 @@ class AugeasConfigurator(common.Plugin):
 
         self.save_notes = ""
 
-        # See if any temporary changes need to be recovered
-        # This needs to occur before VirtualHost objects are setup...
-        # because this will change the underlying configuration and potential
-        # vhosts
-        self.reverter = reverter.Reverter(self.config)
 
     def init_augeas(self):
         """ Initialize the actual Augeas instance """
@@ -50,6 +44,10 @@ class AugeasConfigurator(common.Plugin):
             flags=(augeas.Augeas.NONE |
                    augeas.Augeas.NO_MODL_AUTOLOAD |
                    augeas.Augeas.ENABLE_SPAN))
+        # See if any temporary changes need to be recovered
+        # This needs to occur before VirtualHost objects are setup...
+        # because this will change the underlying configuration and potential
+        # vhosts
         self.recovery_routine()
 
     def check_parsing_errors(self, lens):
@@ -78,26 +76,26 @@ class AugeasConfigurator(common.Plugin):
                     self.aug.get(path + "/message")))
                 raise errors.PluginError(msg)
 
-    # TODO: Cleanup this function
-    def save(self, title=None, temporary=False):
-        """Saves all changes to the configuration files.
+    def ensure_augeas_state(self):
+        """Makes sure that all Augeas dom changes are written to files to avoid
+        loss of configuration directives when doing additional augeas parsing,
+        causing a possible augeas.load() resulting dom reset
+        """
 
-        This function first checks for save errors, if none are found,
-        all configuration changes made will be saved. According to the
-        function parameters. If an exception is raised, a new checkpoint
-        was not created.
+        if self.unsaved_files():
+            self.save_notes += "(autosave)"
+            self.save()
 
-        :param str title: The title of the save. If a title is given, the
-            configuration will be saved as a new checkpoint and put in a
-            timestamped directory.
-
-        :param bool temporary: Indicates whether the changes made will
-            be quickly reversed in the future (ie. challenges)
+    def unsaved_files(self):
+        """Lists files that have modified Augeas DOM but the changes have not
+        been written to the filesystem yet, used by `self.save()` and
+        ApacheConfigurator to check the file state.
 
         :raises .errors.PluginError: If there was an error in Augeas, in
             an attempt to save the configuration, or an error creating a
             checkpoint
 
+        :returns: `set` of unsaved files
         """
         save_state = self.aug.get("/augeas/save")
         self.aug.set("/augeas/save", "noop")
@@ -113,30 +111,41 @@ class AugeasConfigurator(common.Plugin):
             raise errors.PluginError(
                 "Error saving files, check logs for more info.")
 
+        # Return the original save method
+        self.aug.set("/augeas/save", save_state)
+
         # Retrieve list of modified files
         # Note: Noop saves can cause the file to be listed twice, I used a
         # set to remove this possibility. This is a known augeas 0.10 error.
         save_paths = self.aug.match("/augeas/events/saved")
 
-        # If the augeas tree didn't change, no files were saved and a backup
-        # should not be created
         save_files = set()
         if save_paths:
             for path in save_paths:
                 save_files.add(self.aug.get(path)[6:])
+        return save_files
 
-            try:
-                # Create Checkpoint
-                if temporary:
-                    self.reverter.add_to_temp_checkpoint(
-                        save_files, self.save_notes)
-                else:
-                    self.reverter.add_to_checkpoint(save_files,
-                                                    self.save_notes)
-            except errors.ReverterError as err:
-                raise errors.PluginError(str(err))
+    def save(self, title=None, temporary=False):
+        """Saves all changes to the configuration files.
+
+        This function first checks for save errors, if none are found,
+        all configuration changes made will be saved. According to the
+        function parameters. If an exception is raised, a new checkpoint
+        was not created.
+
+        :param str title: The title of the save. If a title is given, the
+            configuration will be saved as a new checkpoint and put in a
+            timestamped directory.
+
+        :param bool temporary: Indicates whether the changes made will
+            be quickly reversed in the future (ie. challenges)
+
+        """
+        save_files = self.unsaved_files()
+        if save_files:
+            self.add_to_checkpoint(save_files,
+                                   self.save_notes, temporary=temporary)
 
-        self.aug.set("/augeas/save", save_state)
         self.save_notes = ""
         self.aug.save()
 
@@ -147,10 +156,7 @@ class AugeasConfigurator(common.Plugin):
                 self.aug.remove("/files/"+sf)
             self.aug.load()
         if title and not temporary:
-            try:
-                self.reverter.finalize_checkpoint(title)
-            except errors.ReverterError as err:
-                raise errors.PluginError(str(err))
+            self.finalize_checkpoint(title)
 
     def _log_save_errors(self, ex_errs):
         """Log errors due to bad Augeas save.
@@ -175,10 +181,7 @@ class AugeasConfigurator(common.Plugin):
         :raises .errors.PluginError: If unable to recover the configuration
 
         """
-        try:
-            self.reverter.recovery_routine()
-        except errors.ReverterError as err:
-            raise errors.PluginError(str(err))
+        super(AugeasConfigurator, self).recovery_routine()
         # Need to reload configuration after these changes take effect
         self.aug.load()
 
@@ -188,10 +191,7 @@ class AugeasConfigurator(common.Plugin):
         :raises .errors.PluginError: If unable to revert the challenge config.
 
         """
-        try:
-            self.reverter.revert_temporary_config()
-        except errors.ReverterError as err:
-            raise errors.PluginError(str(err))
+        self.revert_temporary_config()
         self.aug.load()
 
     def rollback_checkpoints(self, rollback=1):
@@ -203,20 +203,5 @@ class AugeasConfigurator(common.Plugin):
             the function is unable to correctly revert the configuration
 
         """
-        try:
-            self.reverter.rollback_checkpoints(rollback)
-        except errors.ReverterError as err:
-            raise errors.PluginError(str(err))
+        super(AugeasConfigurator, self).rollback_checkpoints(rollback)
         self.aug.load()
-
-    def view_config_changes(self):
-        """Show all of the configuration changes that have taken place.
-
-        :raises .errors.PluginError: If there is a problem while processing
-            the checkpoints directories.
-
-        """
-        try:
-            self.reverter.view_config_changes()
-        except errors.ReverterError as err:
-            raise errors.PluginError(str(err))

certbot-apache/certbot_apache/configurator.py

@@ -1,6 +1,5 @@
 """Apache Configuration based off of Augeas Configurator."""
 # pylint: disable=too-many-lines
-import filecmp
 import fnmatch
 import logging
 import os
@@ -96,7 +95,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             help="SSL vhost configuration extension.")
         add("server-root", default=constants.os_constant("server_root"),
             help="Apache server root directory.")
-        add("vhost-root", default=constants.os_constant("vhost_root"),
+        add("vhost-root", default=None,
             help="Apache server VirtualHost configuration root")
         add("logs-root", default=constants.os_constant("logs_root"),
             help="Apache server logs directory")
@@ -134,6 +133,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         self.parser = None
         self.version = version
         self.vhosts = None
+        self.vhostroot = None
         self._enhance_func = {"redirect": self._enable_redirect,
                               "ensure-http-header": self._set_http_header,
                               "staple-ocsp": self._enable_ocsp_stapling}
@@ -190,9 +190,15 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
                 "version 1.2.0 or higher, please make sure you have you have "
                 "those installed.")
 
+        # Parse vhost-root if defined on cli
+        if not self.conf("vhost-root"):
+            self.vhostroot = constants.os_constant("vhost_root")
+        else:
+            self.vhostroot = os.path.abspath(self.conf("vhost-root"))
+
         self.parser = parser.ApacheParser(
             self.aug, self.conf("server-root"), self.conf("vhost-root"),
-            self.version)
+            self.version, configurator=self)
         # Check for errors in parsing files with Augeas
         self.check_parsing_errors("httpd.aug")
 
@@ -242,13 +248,18 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             a lack of directives
 
         """
+        # Choose vhost before (possible) enabling of mod_ssl, to keep the
+        # vhost choice namespace similar with the pre-validation one.
         vhost = self.choose_vhost(domain)
-        self._clean_vhost(vhost)
 
         # This is done first so that ssl module is enabled and cert_path,
         # cert_key... can all be parsed appropriately
         self.prepare_server_https("443")
 
+        # Add directives and remove duplicates
+        self._add_dummy_ssl_directives(vhost.path)
+        self._clean_vhost(vhost)
+
         path = {"cert_path": self.parser.find_dir("SSLCertificateFile",
                                                   None, vhost.path),
                 "cert_key": self.parser.find_dir("SSLCertificateKeyFile",
@@ -290,6 +301,10 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             self.aug.set(path["cert_path"][-1], fullchain_path)
             self.aug.set(path["cert_key"][-1], key_path)
 
+        # Enable the new vhost if needed
+        if not vhost.enabled:
+            self.enable_site(vhost)
+
         # Save notes about the transaction that took place
         self.save_notes += ("Changed vhost at %s with addresses of %s\n"
                             "\tSSLCertificateFile %s\n"
@@ -300,11 +315,6 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         if chain_path is not None:
             self.save_notes += "\tSSLCertificateChainFile %s\n" % chain_path
 
-        # Make sure vhost is enabled if distro with enabled / available
-        if self.conf("handle-sites"):
-            if not vhost.enabled:
-                self.enable_site(vhost)
-
     def choose_vhost(self, target_name, temp=False):
         """Chooses a virtual host based on the given domain name.
 
@@ -579,17 +589,14 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         if filename is None:
             return None
 
-        if self.conf("handle-sites"):
-            is_enabled = self.is_site_enabled(filename)
-        else:
-            is_enabled = True
-
         macro = False
         if "/macro/" in path.lower():
             macro = True
 
+        vhost_enabled = self.parser.parsed_in_original(filename)
+
         vhost = obj.VirtualHost(filename, path, addrs, is_ssl,
-                                is_enabled, modmacro=macro)
+                                vhost_enabled, modmacro=macro)
         self._add_servernames(vhost)
         return vhost
 
@@ -644,7 +651,6 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
                 elif internal_path not in internal_paths[realpath]:
                     internal_paths[realpath].add(internal_path)
                     vhs.append(new_vhost)
-
         return vhs
 
     def is_name_vhost(self, target_addr):
@@ -855,14 +861,22 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         vh_p = self._get_new_vh_path(orig_matches, new_matches)
 
         if not vh_p:
-            raise errors.PluginError(
-                "Could not reverse map the HTTPS VirtualHost to the original")
+            # The vhost was not found on the currently parsed paths
+            # Make Augeas aware of the new vhost
+            self.parser.parse_file(ssl_fp)
+            # Try to search again
+            new_matches = self.aug.match(
+                "/files%s//* [label()=~regexp('%s')]" %
+                (self._escape(ssl_fp),
+                 parser.case_i("VirtualHost")))
+            vh_p = self._get_new_vh_path(orig_matches, new_matches)
+            if not vh_p:
+                raise errors.PluginError(
+                    "Could not reverse map the HTTPS VirtualHost to the original")
+
 
         # Update Addresses
         self._update_ssl_vhosts_addrs(vh_p)
-        # Add directives
-        self._add_dummy_ssl_directives(vh_p)
-        self.save()
 
         # Log actions and create save notes
         logger.info("Created an SSL vhost at %s", ssl_fp)
@@ -873,6 +887,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         # Create the Vhost object
         ssl_vhost = self._create_vhost(vh_p)
         ssl_vhost.ancestor = nonssl_vhost
+
         self.vhosts.append(ssl_vhost)
 
         # NOTE: Searches through Augeas seem to ruin changes to directives
@@ -901,11 +916,29 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         return None
 
     def _get_ssl_vhost_path(self, non_ssl_vh_fp):
-        # Get filepath of new ssl_vhost
-        if non_ssl_vh_fp.endswith(".conf"):
-            return non_ssl_vh_fp[:-(len(".conf"))] + self.conf("le_vhost_ext")
+        """ Get a file path for SSL vhost, uses user defined path as priority,
+        but if the value is invalid or not defined, will fall back to non-ssl
+        vhost filepath.
+
+        :param str non_ssl_vh_fp: Filepath of non-SSL vhost
+
+        :returns: Filepath for SSL vhost
+        :rtype: str
+        """
+
+        if self.conf("vhost-root") and os.path.exists(self.conf("vhost-root")):
+            # Defined by user on CLI
+
+            fp = os.path.join(os.path.realpath(self.vhostroot),
+                              os.path.basename(non_ssl_vh_fp))
         else:
-            return non_ssl_vh_fp + self.conf("le_vhost_ext")
+            # Use non-ssl filepath
+            fp = os.path.realpath(non_ssl_vh_fp)
+
+        if fp.endswith(".conf"):
+            return fp[:-(len(".conf"))] + self.conf("le_vhost_ext")
+        else:
+            return fp + self.conf("le_vhost_ext")
 
     def _sift_rewrite_rule(self, line):
         """Decides whether a line should be copied to a SSL vhost.
@@ -970,6 +1003,10 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
                 # The content does not include the closing tag, so add it
                 new_file.write("</VirtualHost>\n")
                 new_file.write("</IfModule>\n")
+            # Add new file to augeas paths if we're supposed to handle
+            # activation (it's not included as default)
+            if not self.parser.parsed_in_current(ssl_fp):
+                self.parser.parse_file(ssl_fp)
         except IOError:
             logger.fatal("Error writing/reading to file in make_vhost_ssl")
             raise errors.PluginError("Unable to write/read in make_vhost_ssl")
@@ -1258,13 +1295,13 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         .. note:: This function saves the configuration
 
         :param ssl_vhost: Destination of traffic, an ssl enabled vhost
-        :type ssl_vhost: :class:`~letsencrypt_apache.obj.VirtualHost`
+        :type ssl_vhost: :class:`~certbot_apache.obj.VirtualHost`
 
         :param unused_options: Not currently used
         :type unused_options: Not Available
 
         :returns: Success, general_vhost (HTTP vhost)
-        :rtype: (bool, :class:`~letsencrypt_apache.obj.VirtualHost`)
+        :rtype: (bool, :class:`~certbot_apache.obj.VirtualHost`)
 
         """
         min_apache_ver = (2, 3, 3)
@@ -1610,7 +1647,7 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             if len(ssl_vhost.name) < (255 - (len(redirect_filename) + 1)):
                 redirect_filename = "le-redirect-%s.conf" % ssl_vhost.name
 
-        redirect_filepath = os.path.join(self.conf("vhost-root"),
+        redirect_filepath = os.path.join(self.vhostroot,
                                          redirect_filename)
 
         # Register the new file that will be created
@@ -1621,6 +1658,11 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
         # Write out file
         with open(redirect_filepath, "w") as redirect_file:
             redirect_file.write(text)
+
+        # Add new include to configuration if it doesn't exist yet
+        if not self.parser.parsed_in_current(redirect_filepath):
+            self.parser.parse_file(redirect_filepath)
+
         logger.info("Created redirect file: %s", redirect_filename)
 
         return redirect_filepath
@@ -1660,32 +1702,6 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
 
         return redirects
 
-    def is_site_enabled(self, avail_fp):
-        """Checks to see if the given site is enabled.
-
-        .. todo:: fix hardcoded sites-enabled, check os.path.samefile
-
-        :param str avail_fp: Complete file path of available site
-
-        :returns: Success
-        :rtype: bool
-
-        """
-
-        enabled_dir = os.path.join(self.parser.root, "sites-enabled")
-        if not os.path.isdir(enabled_dir):
-            error_msg = ("Directory '{0}' does not exist. Please ensure "
-                         "that the values for --apache-handle-sites and "
-                         "--apache-server-root are correct for your "
-                         "environment.".format(enabled_dir))
-            raise errors.ConfigurationError(error_msg)
-        for entry in os.listdir(enabled_dir):
-            try:
-                if filecmp.cmp(avail_fp, os.path.join(enabled_dir, entry)):
-                    return True
-            except OSError:
-                pass
-        return False
 
     def enable_site(self, vhost):
         """Enables an available site, Apache reload required.
@@ -1705,21 +1721,40 @@ class ApacheConfigurator(augeas_configurator.AugeasConfigurator):
             supported.
 
         """
-        if self.is_site_enabled(vhost.filep):
+        if vhost.enabled:
             return
 
-        if "/sites-available/" in vhost.filep:
-            enabled_path = ("%s/sites-enabled/%s" %
-                            (self.parser.root, os.path.basename(vhost.filep)))
-            self.reverter.register_file_creation(False, enabled_path)
+        # Handle non-debian systems
+        if not self.conf("handle-sites"):
+            if not self.parser.parsed_in_original(vhost.filep):
+                # Add direct include to root conf
+                self.parser.add_include(self.parser.loc["default"], vhost.filep)
+                vhost.enabled = True
+            return
+
+        enabled_path = ("%s/sites-enabled/%s" %
+                        (self.parser.root, os.path.basename(vhost.filep)))
+        self.reverter.register_file_creation(False, enabled_path)
+        try:
             os.symlink(vhost.filep, enabled_path)
-            vhost.enabled = True
-            logger.info("Enabling available site: %s", vhost.filep)
-            self.save_notes += "Enabled site %s\n" % vhost.filep
-        else:
-            raise errors.NotSupportedError(
-                "Unsupported filesystem layout. "
-                "sites-available/enabled expected.")
+        except OSError as err:
+            if os.path.islink(enabled_path) and os.path.realpath(
+               enabled_path) == vhost.filep:
+                # Already in shape
+                vhost.enabled = True
+                return
+            else:
+                logger.warning(
+                    "Could not symlink %s to %s, got error: %s", enabled_path,
+                    vhost.filep, err.strerror)
+                errstring = ("Encountered error while trying to enable a " +
+                             "newly created VirtualHost located at {0} by " +
+                             "linking to it from {1}")
+                raise errors.NotSupportedError(errstring.format(vhost.filep,
+                                                                enabled_path))
+        vhost.enabled = True
+        logger.info("Enabling available site: %s", vhost.filep)
+        self.save_notes += "Enabled site %s\n" % vhost.filep
 
     def enable_mod(self, mod_name, temp=False):
         """Enables module in Apache.
@@ -1989,5 +2024,5 @@ def install_ssl_options_conf(options_ssl, options_ssl_digest):
     # XXX if we ever try to enforce a local privilege boundary (eg, running
     # certbot for unprivileged users via setuid), this function will need
     # to be modified.
-    return common.install_ssl_options_conf(options_ssl, options_ssl_digest,
+    return common.install_version_controlled_file(options_ssl, options_ssl_digest,
         constants.os_constant("MOD_SSL_CONF_SRC"), constants.ALL_SSL_OPTIONS_HASHES)

certbot-apache/certbot_apache/display_ops.py

@@ -86,10 +86,11 @@ def _vhost_menu(domain, vhosts):
             choices, force_interactive=True)
     except errors.MissingCommandlineFlag:
         msg = (
-            "Encountered vhost ambiguity but unable to ask for user "
+            "Encountered vhost ambiguity when trying to find a vhost for "
+            "{0} but was unable to ask for user "
             "guidance in non-interactive mode. Certbot may need "
             "vhosts to be explicitly labelled with ServerName or "
-            "ServerAlias directives.")
+            "ServerAlias directives.".format(domain))
         logger.warning(msg)
         raise errors.MissingCommandlineFlag(msg)
 

certbot-apache/certbot_apache/parser.py

@@ -1,4 +1,5 @@
 """ApacheParser is a member object of the ApacheConfigurator class."""
+import copy
 import fnmatch
 import logging
 import os
@@ -30,9 +31,15 @@ class ApacheParser(object):
     arg_var_interpreter = re.compile(r"\$\{[^ \}]*}")
     fnmatch_chars = set(["*", "?", "\\", "[", "]"])
 
-    def __init__(self, aug, root, vhostroot, version=(2, 4)):
+    def __init__(self, aug, root, vhostroot=None, version=(2, 4),
+                 configurator=None):
         # Note: Order is important here.
 
+        # Needed for calling save() with reverter functionality that resides in
+        # AugeasConfigurator superclass of ApacheConfigurator. This resolves
+        # issues with aug.load() after adding new files / defines to parse tree
+        self.configurator = configurator
+
         # This uses the binary, so it can be done first.
         # https://httpd.apache.org/docs/2.4/mod/core.html#define
         # https://httpd.apache.org/docs/2.4/mod/core.html#ifdefine
@@ -46,9 +53,7 @@ class ApacheParser(object):
         # Find configuration root and make sure augeas can parse it.
         self.root = os.path.abspath(root)
         self.loc = {"root": self._find_config_root()}
-        self._parse_file(self.loc["root"])
-
-        self.vhostroot = os.path.abspath(vhostroot)
+        self.parse_file(self.loc["root"])
 
         # This problem has been fixed in Augeas 1.0
         self.standardize_excl()
@@ -62,15 +67,42 @@ class ApacheParser(object):
         # Set up rest of locations
         self.loc.update(self._set_locations())
 
-        # Must also attempt to parse virtual host root
-        self._parse_file(self.vhostroot + "/" +
-                         constants.os_constant("vhost_files"))
+        self.existing_paths = copy.deepcopy(self.parser_paths)
+
+        # Must also attempt to parse additional virtual host root
+        if vhostroot:
+            self.parse_file(os.path.abspath(vhostroot) + "/" +
+                            constants.os_constant("vhost_files"))
 
         # check to see if there were unparsed define statements
         if version < (2, 4):
             if self.find_dir("Define", exclude=False):
                 raise errors.PluginError("Error parsing runtime variables")
 
+    def add_include(self, main_config, inc_path):
+        """Add Include for a new configuration file if one does not exist
+
+        :param str main_config: file path to main Apache config file
+        :param str inc_path: path of file to include
+
+        """
+        if len(self.find_dir(case_i("Include"), inc_path)) == 0:
+            logger.debug("Adding Include %s to %s",
+                         inc_path, get_aug_path(main_config))
+            self.add_dir(
+                get_aug_path(main_config),
+                "Include", inc_path)
+
+            # Add new path to parser paths
+            new_dir = os.path.dirname(inc_path)
+            new_file = os.path.basename(inc_path)
+            if new_dir in self.existing_paths.keys():
+                # Add to existing path
+                self.existing_paths[new_dir].append(new_file)
+            else:
+                # Create a new path
+                self.existing_paths[new_dir] = [new_file]
+
     def init_modules(self):
         """Iterates on the configuration until no new modules are loaded.
 
@@ -91,9 +123,14 @@ class ApacheParser(object):
 
             for match_name, match_filename in six.moves.zip(
                     iterator, iterator):
-                self.modules.add(self.get_arg(match_name))
-                self.modules.add(
-                    os.path.basename(self.get_arg(match_filename))[:-2] + "c")
+                mod_name = self.get_arg(match_name)
+                mod_filename = self.get_arg(match_filename)
+                if mod_name and mod_filename:
+                    self.modules.add(mod_name)
+                    self.modules.add(os.path.basename(mod_filename)[:-2] + "c")
+                else:
+                    logger.debug("Could not read LoadModule directive from " +
+                                 "Augeas path: {0}".format(match_name[6:]))
 
     def update_runtime_variables(self):
         """"
@@ -339,7 +376,10 @@ class ApacheParser(object):
 
         # Note: normal argument may be a quoted variable
         # e.g. strip now, not later
-        value = value.strip("'\"")
+        if not value:
+            return None
+        else:
+            value = value.strip("'\"")
 
         variables = ApacheParser.arg_var_interpreter.findall(value)
 
@@ -428,9 +468,9 @@ class ApacheParser(object):
 
         # Attempts to add a transform to the file if one does not already exist
         if os.path.isdir(arg):
-            self._parse_file(os.path.join(arg, "*"))
+            self.parse_file(os.path.join(arg, "*"))
         else:
-            self._parse_file(arg)
+            self.parse_file(arg)
 
         # Argument represents an fnmatch regular expression, convert it
         # Split up the path and convert each into an Augeas accepted regex
@@ -470,7 +510,7 @@ class ApacheParser(object):
             # Since Python 3.6, it returns a different pattern like (?s:.*\.load)\Z
             return fnmatch.translate(clean_fn_match)[4:-3]
 
-    def _parse_file(self, filepath):
+    def parse_file(self, filepath):
         """Parse file with Augeas
 
         Checks to see if file_path is parsed by Augeas
@@ -480,6 +520,10 @@ class ApacheParser(object):
 
         """
         use_new, remove_old = self._check_path_actions(filepath)
+        # Ensure that we have the latest Augeas DOM state on disk before
+        # calling aug.load() which reloads the state from disk
+        if self.configurator:
+            self.configurator.ensure_augeas_state()
         # Test if augeas included file for Httpd.lens
         # Note: This works for augeas globs, ie. *.conf
         if use_new:
@@ -494,6 +538,39 @@ class ApacheParser(object):
                 self._add_httpd_transform(filepath)
                 self.aug.load()
 
+    def parsed_in_current(self, filep):
+        """Checks if the file path is parsed by current Augeas parser config
+        ie. returns True if the file is found on a path that's found in live
+        Augeas configuration.
+
+        :param str filep: Path to match
+
+        :returns: True if file is parsed in existing configuration tree
+        :rtype: bool
+        """
+        return self._parsed_by_parser_paths(filep, self.parser_paths)
+
+    def parsed_in_original(self, filep):
+        """Checks if the file path is parsed by existing Apache config.
+        ie. returns True if the file is found on a path that matches Include or
+        IncludeOptional statement in the Apache configuration.
+
+        :param str filep: Path to match
+
+        :returns: True if file is parsed in existing configuration tree
+        :rtype: bool
+        """
+        return self._parsed_by_parser_paths(filep, self.existing_paths)
+
+    def _parsed_by_parser_paths(self, filep, paths):
+        """Helper function that searches through provided paths and returns
+        True if file path is found in the set"""
+        for directory in paths.keys():
+            for filename in paths[directory]:
+                if fnmatch.fnmatch(filep, os.path.join(directory, filename)):
+                    return True
+        return False
+
     def _check_path_actions(self, filepath):
         """Determine actions to take with a new augeas path
 
@@ -622,7 +699,6 @@ class ApacheParser(object):
         for name in location:
             if os.path.isfile(os.path.join(self.root, name)):
                 return os.path.join(self.root, name)
-
         raise errors.NoInstallationError("Could not find configuration root")
 
 

certbot-apache/certbot_apache/tests/apache-conf-files/apache-conf-test

@@ -26,6 +26,7 @@ function Setup() {
     ErrorLog /tmp/error.log
     CustomLog /tmp/requests.log combined
 </VirtualHost>" | sudo tee $EA/sites-available/throwaway-example.conf >/dev/null
+        sudo ln -sf $EA/sites-available/throwaway-example.conf $EA/sites-enabled/throwaway-example.conf
     else
         TMP="/tmp/`basename \"$APPEND_APACHECONF\"`.$$"
         sudo cp -a "$APPEND_APACHECONF" "$TMP"
@@ -37,6 +38,7 @@ function Cleanup() {
     if [ "$APPEND_APACHECONF" = "" ] ; then
         sudo rm /etc/apache2/sites-{enabled,available}/"$f"
         sudo rm $EA/sites-available/throwaway-example.conf
+        sudo rm $EA/sites-enabled/throwaway-example.conf
     else
         sudo mv "$TMP" "$APPEND_APACHECONF"
     fi

certbot-apache/certbot_apache/tests/augeas_configurator_test.py

@@ -31,7 +31,7 @@ class AugeasConfiguratorTest(util.ApacheTest):
 
     def test_bad_parse(self):
         # pylint: disable=protected-access
-        self.config.parser._parse_file(os.path.join(
+        self.config.parser.parse_file(os.path.join(
             self.config.parser.root, "conf-available", "bad_conf_file.conf"))
         self.assertRaises(
             errors.PluginError, self.config.check_parsing_errors, "httpd.aug")

certbot-apache/certbot_apache/tests/configurator_test.py

@@ -8,6 +8,7 @@ import unittest
 import mock
 # six is used in mock.patch()
 import six  # pylint: disable=unused-import
+import tempfile
 
 from acme import challenges
 
@@ -34,9 +35,20 @@ class MultipleVhostsTest(util.ApacheTest):
     def setUp(self):  # pylint: disable=arguments-differ
         super(MultipleVhostsTest, self).setUp()
 
-        self.config = util.get_apache_configurator(
-            self.config_path, self.vhost_path, self.config_dir, self.work_dir)
-        self.config = self.mock_deploy_cert(self.config)
+        from certbot_apache.constants import os_constant
+        orig_os_constant = os_constant
+        def mock_os_constant(key, vhost_path=self.vhost_path):
+            """Mock default vhost path"""
+            if key == "vhost_root":
+                return vhost_path
+            else:
+                return orig_os_constant(key)
+
+        with mock.patch("certbot_apache.constants.os_constant") as mock_c:
+            mock_c.side_effect = mock_os_constant
+            self.config = util.get_apache_configurator(
+                self.config_path, None, self.config_dir, self.work_dir)
+            self.config = self.mock_deploy_cert(self.config)
         self.vh_truth = util.get_vh_truth(
             self.temp_dir, "debian_apache_2_4/multiple_vhosts")
 
@@ -121,19 +133,20 @@ class MultipleVhostsTest(util.ApacheTest):
 
     @certbot_util.patch_get_utility()
     def test_get_all_names(self, mock_getutility):
-        mock_getutility.notification = mock.MagicMock(return_value=True)
+        mock_utility = mock_getutility()
+        mock_utility.notification = mock.MagicMock(return_value=True)
         names = self.config.get_all_names()
         self.assertEqual(names, set(
-            ["certbot.demo", "ocspvhost.com", "encryption-example.demo"]
+            ["certbot.demo", "ocspvhost.com", "encryption-example.demo",
+             "nonsym.link", "vhost.in.rootconf"]
         ))
 
     @certbot_util.patch_get_utility()
     @mock.patch("certbot_apache.configurator.socket.gethostbyaddr")
     def test_get_all_names_addrs(self, mock_gethost, mock_getutility):
         mock_gethost.side_effect = [("google.com", "", ""), socket.error]
-        notification = mock.Mock()
-        notification.notification = mock.Mock(return_value=True)
-        mock_getutility.return_value = notification
+        mock_utility = mock_getutility()
+        mock_utility.notification.return_value = True
         vhost = obj.VirtualHost(
             "fp", "ap",
             set([obj.Addr(("8.8.8.8", "443")),
@@ -145,7 +158,7 @@ class MultipleVhostsTest(util.ApacheTest):
 
         names = self.config.get_all_names()
         # Names get filtered, only 5 are returned
-        self.assertEqual(len(names), 5)
+        self.assertEqual(len(names), 7)
         self.assertTrue("zombo.com" in names)
         self.assertTrue("google.com" in names)
         self.assertTrue("certbot.demo" in names)
@@ -159,7 +172,7 @@ class MultipleVhostsTest(util.ApacheTest):
     def test_get_aug_internal_path(self):
         from certbot_apache.configurator import get_internal_aug_path
         internal_paths = [
-            "VirtualHost", "IfModule/VirtualHost", "VirtualHost", "VirtualHost",
+            "Virtualhost", "IfModule/VirtualHost", "VirtualHost", "VirtualHost",
             "Macro/VirtualHost", "IfModule/VirtualHost", "VirtualHost",
             "IfModule/VirtualHost"]
 
@@ -185,14 +198,9 @@ class MultipleVhostsTest(util.ApacheTest):
             self.vh_truth[2].get_names(), set(["*.le.co", "ip-172-30-0-17"]))
 
     def test_get_virtual_hosts(self):
-        """Make sure all vhosts are being properly found.
-
-        .. note:: If test fails, only finding 1 Vhost... it is likely that
-            it is a problem with is_enabled.  If finding only 3, likely is_ssl
-
-        """
+        """Make sure all vhosts are being properly found."""
         vhs = self.config.get_virtual_hosts()
-        self.assertEqual(len(vhs), 8)
+        self.assertEqual(len(vhs), 10)
         found = 0
 
         for vhost in vhs:
@@ -203,7 +211,7 @@ class MultipleVhostsTest(util.ApacheTest):
             else:
                 raise Exception("Missed: %s" % vhost)  # pragma: no cover
 
-        self.assertEqual(found, 8)
+        self.assertEqual(found, 10)
 
         # Handle case of non-debian layout get_virtual_hosts
         with mock.patch(
@@ -211,7 +219,7 @@ class MultipleVhostsTest(util.ApacheTest):
         ) as mock_conf:
             mock_conf.return_value = False
             vhs = self.config.get_virtual_hosts()
-            self.assertEqual(len(vhs), 8)
+            self.assertEqual(len(vhs), 10)
 
     @mock.patch("certbot_apache.display_ops.select_vhost")
     def test_choose_vhost_none_avail(self, mock_select):
@@ -226,8 +234,10 @@ class MultipleVhostsTest(util.ApacheTest):
             self.vh_truth[1], self.config.choose_vhost("none.com"))
 
     @mock.patch("certbot_apache.display_ops.select_vhost")
-    def test_choose_vhost_select_vhost_non_ssl(self, mock_select):
+    @mock.patch("certbot_apache.obj.VirtualHost.conflicts")
+    def test_choose_vhost_select_vhost_non_ssl(self, mock_conf, mock_select):
         mock_select.return_value = self.vh_truth[0]
+        mock_conf.return_value = False
         chosen_vhost = self.config.choose_vhost("none.com")
         self.vh_truth[0].aliases.add("none.com")
         self.assertEqual(
@@ -237,6 +247,15 @@ class MultipleVhostsTest(util.ApacheTest):
         self.assertFalse(self.vh_truth[0].ssl)
         self.assertTrue(chosen_vhost.ssl)
 
+    @mock.patch("certbot_apache.configurator.ApacheConfigurator._find_best_vhost")
+    @mock.patch("certbot_apache.parser.ApacheParser.add_dir")
+    def test_choose_vhost_and_servername_addition(self, mock_add, mock_find):
+        ret_vh = self.vh_truth[8]
+        ret_vh.enabled = False
+        mock_find.return_value = self.vh_truth[8]
+        self.config.choose_vhost("whatever.com")
+        self.assertTrue(mock_add.called)
+
     @mock.patch("certbot_apache.display_ops.select_vhost")
     def test_choose_vhost_select_vhost_with_temp(self, mock_select):
         mock_select.return_value = self.vh_truth[0]
@@ -288,9 +307,9 @@ class MultipleVhostsTest(util.ApacheTest):
         # Assume only the two default vhosts.
         self.config.vhosts = [
             vh for vh in self.config.vhosts
-            if vh.name not in ["certbot.demo",
+            if vh.name not in ["certbot.demo", "nonsym.link",
                 "encryption-example.demo",
-                "ocspvhost.com"]
+                "ocspvhost.com", "vhost.in.rootconf"]
             and "*.blue.purple.com" not in vh.aliases
         ]
         self.assertEqual(
@@ -299,26 +318,7 @@ class MultipleVhostsTest(util.ApacheTest):
 
     def test_non_default_vhosts(self):
         # pylint: disable=protected-access
-        self.assertEqual(len(self.config._non_default_vhosts()), 6)
-
-    def test_is_site_enabled(self):
-        """Test if site is enabled.
-
-        .. note:: This test currently fails for hard links
-            (which may happen if you move dirs incorrectly)
-        .. warning:: This test does not work when running using the
-            unittest.main() function. It incorrectly copies symlinks.
-
-        """
-        self.assertTrue(self.config.is_site_enabled(self.vh_truth[0].filep))
-        self.assertFalse(self.config.is_site_enabled(self.vh_truth[1].filep))
-        self.assertTrue(self.config.is_site_enabled(self.vh_truth[2].filep))
-        self.assertTrue(self.config.is_site_enabled(self.vh_truth[3].filep))
-        with mock.patch("os.path.isdir") as mock_isdir:
-            mock_isdir.return_value = False
-            self.assertRaises(errors.ConfigurationError,
-                              self.config.is_site_enabled,
-                              "irrelevant")
+        self.assertEqual(len(self.config._non_default_vhosts()), 8)
 
     @mock.patch("certbot.util.run_script")
     @mock.patch("certbot.util.exe_exists")
@@ -345,21 +345,59 @@ class MultipleVhostsTest(util.ApacheTest):
         self.assertRaises(
             errors.MisconfigurationError, self.config.enable_mod, "ssl")
 
-    def test_enable_site(self):
-        # Default 443 vhost
-        self.assertFalse(self.vh_truth[1].enabled)
-        self.config.enable_site(self.vh_truth[1])
+    def test_enable_site_already_enabled(self):
         self.assertTrue(self.vh_truth[1].enabled)
-
-        # Go again to make sure nothing fails
         self.config.enable_site(self.vh_truth[1])
 
     def test_enable_site_failure(self):
+        self.config.parser.root = "/tmp/nonexistent"
         self.assertRaises(
             errors.NotSupportedError,
             self.config.enable_site,
             obj.VirtualHost("asdf", "afsaf", set(), False, False))
 
+    def test_enable_site_nondebian(self):
+        mock_c = "certbot_apache.configurator.ApacheConfigurator.conf"
+        def conf_side_effect(arg):
+            """ Mock function for ApacheConfigurator.conf """
+            confvars = {"handle-sites": False}
+            if arg in confvars:
+                return confvars[arg]
+        inc_path = "/path/to/wherever"
+        vhost = self.vh_truth[0]
+        with mock.patch(mock_c) as mock_conf:
+            mock_conf.side_effect = conf_side_effect
+            vhost.enabled = False
+            vhost.filep = inc_path
+            self.assertFalse(self.config.parser.find_dir("Include", inc_path))
+            self.assertFalse(
+                os.path.dirname(inc_path) in self.config.parser.existing_paths)
+            self.config.enable_site(vhost)
+            self.assertTrue(self.config.parser.find_dir("Include", inc_path))
+            self.assertTrue(
+                os.path.dirname(inc_path) in self.config.parser.existing_paths)
+            self.assertTrue(
+                os.path.basename(inc_path) in self.config.parser.existing_paths[
+                    os.path.dirname(inc_path)])
+
+    def test_deploy_cert_enable_new_vhost(self):
+        # Create
+        ssl_vhost = self.config.make_vhost_ssl(self.vh_truth[0])
+        self.config.parser.modules.add("ssl_module")
+        self.config.parser.modules.add("mod_ssl.c")
+        self.assertFalse(ssl_vhost.enabled)
+        self.config.deploy_cert(
+            "encryption-example.demo", "example/cert.pem", "example/key.pem",
+            "example/cert_chain.pem", "example/fullchain.pem")
+        self.assertTrue(ssl_vhost.enabled)
+        # Make sure that we don't error out if symlink already exists
+        ssl_vhost.enabled = False
+        self.assertFalse(ssl_vhost.enabled)
+        self.config.deploy_cert(
+            "encryption-example.demo", "example/cert.pem", "example/key.pem",
+            "example/cert_chain.pem", "example/fullchain.pem")
+        self.assertTrue(ssl_vhost.enabled)
+
     def test_deploy_cert_newssl(self):
         self.config = util.get_apache_configurator(
             self.config_path, self.vhost_path, self.config_dir,
@@ -388,12 +426,14 @@ class MultipleVhostsTest(util.ApacheTest):
 
         # Verify one directive was found in the correct file
         self.assertEqual(len(loc_cert), 1)
-        self.assertEqual(configurator.get_file_path(loc_cert[0]),
-                         self.vh_truth[1].filep)
+        self.assertEqual(
+            configurator.get_file_path(loc_cert[0]),
+            self.vh_truth[1].filep)
 
         self.assertEqual(len(loc_key), 1)
-        self.assertEqual(configurator.get_file_path(loc_key[0]),
-                         self.vh_truth[1].filep)
+        self.assertEqual(
+            configurator.get_file_path(loc_key[0]),
+            self.vh_truth[1].filep)
 
     def test_deploy_cert_newssl_no_fullchain(self):
         self.config = util.get_apache_configurator(
@@ -427,10 +467,75 @@ class MultipleVhostsTest(util.ApacheTest):
                               "random.demo", "example/cert.pem",
                               "example/key.pem"))
 
+    def test_deploy_cert_not_parsed_path(self):
+        # Make sure that we add include to root config for vhosts when
+        # handle-sites is false
+        self.config.parser.modules.add("ssl_module")
+        self.config.parser.modules.add("mod_ssl.c")
+        tmp_path = os.path.realpath(tempfile.mkdtemp("vhostroot"))
+        os.chmod(tmp_path, 0o755)
+        mock_p = "certbot_apache.configurator.ApacheConfigurator._get_ssl_vhost_path"
+        mock_a = "certbot_apache.parser.ApacheParser.add_include"
+        mock_c = "certbot_apache.configurator.ApacheConfigurator.conf"
+        orig_conf = self.config.conf
+        def conf_side_effect(arg):
+            """ Mock function for ApacheConfigurator.conf """
+            confvars = {"handle-sites": False}
+            if arg in confvars:
+                return confvars[arg]
+            else:
+                return orig_conf("arg")
+
+        with mock.patch(mock_c) as mock_conf:
+            mock_conf.side_effect = conf_side_effect
+            with mock.patch(mock_p) as mock_path:
+                mock_path.return_value = os.path.join(tmp_path, "whatever.conf")
+                with mock.patch(mock_a) as mock_add:
+                    self.config.deploy_cert(
+                        "encryption-example.demo",
+                        "example/cert.pem", "example/key.pem",
+                        "example/cert_chain.pem")
+                    # Test that we actually called add_include
+                    self.assertTrue(mock_add.called)
+        shutil.rmtree(tmp_path)
+
+
     def test_deploy_cert(self):
         self.config.parser.modules.add("ssl_module")
         self.config.parser.modules.add("mod_ssl.c")
 
+        # Patch _add_dummy_ssl_directives to make sure we write them correctly
+        # pylint: disable=protected-access
+        orig_add_dummy = self.config._add_dummy_ssl_directives
+        def mock_add_dummy_ssl(vhostpath):
+            """Mock method for _add_dummy_ssl_directives"""
+            def find_args(path, directive):
+                """Return list of arguments in requested directive at path"""
+                f_args = []
+                dirs = self.config.parser.find_dir(directive, None,
+                                                   path)
+                for d in dirs:
+                    f_args.append(self.config.parser.get_arg(d))
+                return f_args
+            # Verify that the dummy directives do not exist
+            self.assertFalse(
+                "insert_cert_file_path" in find_args(vhostpath,
+                                                     "SSLCertificateFile"))
+            self.assertFalse(
+                "insert_key_file_path" in find_args(vhostpath,
+                                                    "SSLCertificateKeyFile"))
+            orig_add_dummy(vhostpath)
+            # Verify that the dummy directives exist
+            self.assertTrue(
+                "insert_cert_file_path" in find_args(vhostpath,
+                                                     "SSLCertificateFile"))
+            self.assertTrue(
+                "insert_key_file_path" in find_args(vhostpath,
+                                                    "SSLCertificateKeyFile"))
+
+        # pylint: disable=protected-access
+        self.config._add_dummy_ssl_directives = mock_add_dummy_ssl
+
         # Get the default 443 vhost
         self.config.assoc["random.demo"] = self.vh_truth[1]
         self.config.deploy_cert(
@@ -452,16 +557,19 @@ class MultipleVhostsTest(util.ApacheTest):
 
         # Verify one directive was found in the correct file
         self.assertEqual(len(loc_cert), 1)
-        self.assertEqual(configurator.get_file_path(loc_cert[0]),
-                         self.vh_truth[1].filep)
+        self.assertEqual(
+            configurator.get_file_path(loc_cert[0]),
+            self.vh_truth[1].filep)
 
         self.assertEqual(len(loc_key), 1)
-        self.assertEqual(configurator.get_file_path(loc_key[0]),
-                         self.vh_truth[1].filep)
+        self.assertEqual(
+            configurator.get_file_path(loc_key[0]),
+            self.vh_truth[1].filep)
 
         self.assertEqual(len(loc_chain), 1)
-        self.assertEqual(configurator.get_file_path(loc_chain[0]),
-                         self.vh_truth[1].filep)
+        self.assertEqual(
+            configurator.get_file_path(loc_chain[0]),
+            self.vh_truth[1].filep)
 
         # One more time for chain directive setting
         self.config.deploy_cert(
@@ -614,6 +722,30 @@ class MultipleVhostsTest(util.ApacheTest):
             mock_span.return_value = return_value
             self.test_make_vhost_ssl()
 
+    def test_make_vhost_ssl_nonsymlink(self):
+        ssl_vhost_slink = self.config.make_vhost_ssl(self.vh_truth[8])
+        self.assertTrue(ssl_vhost_slink.ssl)
+        self.assertTrue(ssl_vhost_slink.enabled)
+        self.assertEqual(ssl_vhost_slink.name, "nonsym.link")
+
+    def test_make_vhost_ssl_nonexistent_vhost_path(self):
+        def conf_side_effect(arg):
+            """ Mock function for ApacheConfigurator.conf """
+            confvars = {
+                "vhost-root": "/tmp/nonexistent",
+                "le_vhost_ext": "-le-ssl.conf",
+                "handle-sites": True}
+            return confvars[arg]
+
+        with mock.patch(
+                "certbot_apache.configurator.ApacheConfigurator.conf"
+        ) as mock_conf:
+            mock_conf.side_effect = conf_side_effect
+            ssl_vhost = self.config.make_vhost_ssl(self.vh_truth[1])
+            self.assertEqual(os.path.dirname(ssl_vhost.filep),
+                             os.path.dirname(os.path.realpath(
+                                 self.vh_truth[1].filep)))
+
     def test_make_vhost_ssl(self):
         ssl_vhost = self.config.make_vhost_ssl(self.vh_truth[0])
 
@@ -623,22 +755,17 @@ class MultipleVhostsTest(util.ApacheTest):
                          "encryption-example-le-ssl.conf"))
 
         self.assertEqual(ssl_vhost.path,
-                         "/files" + ssl_vhost.filep + "/IfModule/VirtualHost")
+                         "/files" + ssl_vhost.filep + "/IfModule/Virtualhost")
         self.assertEqual(len(ssl_vhost.addrs), 1)
         self.assertEqual(set([obj.Addr.fromstring("*:443")]), ssl_vhost.addrs)
         self.assertEqual(ssl_vhost.name, "encryption-example.demo")
         self.assertTrue(ssl_vhost.ssl)
         self.assertFalse(ssl_vhost.enabled)
 
-        self.assertTrue(self.config.parser.find_dir(
-            "SSLCertificateFile", None, ssl_vhost.path, False))
-        self.assertTrue(self.config.parser.find_dir(
-            "SSLCertificateKeyFile", None, ssl_vhost.path, False))
-
         self.assertEqual(self.config.is_name_vhost(self.vh_truth[0]),
                          self.config.is_name_vhost(ssl_vhost))
 
-        self.assertEqual(len(self.config.vhosts), 9)
+        self.assertEqual(len(self.config.vhosts), 11)
 
     def test_clean_vhost_ssl(self):
         # pylint: disable=protected-access
@@ -688,17 +815,17 @@ class MultipleVhostsTest(util.ApacheTest):
         DIRECTIVES = ["Foo", "Bar"]
         for directive in DIRECTIVES:
             for _ in range(10):
-                self.config.parser.add_dir(self.vh_truth[1].path,
+                self.config.parser.add_dir(self.vh_truth[2].path,
                                            directive, ["baz"])
         self.config.save()
 
-        self.config._remove_directives(self.vh_truth[1].path, DIRECTIVES)
+        self.config._remove_directives(self.vh_truth[2].path, DIRECTIVES)
         self.config.save()
 
         for directive in DIRECTIVES:
             self.assertEqual(
                 len(self.config.parser.find_dir(
-                    directive, None, self.vh_truth[1].path, False)), 0)
+                    directive, None, self.vh_truth[2].path, False)), 0)
 
     def test_make_vhost_ssl_bad_write(self):
         mock_open = mock.mock_open()
@@ -717,10 +844,10 @@ class MultipleVhostsTest(util.ApacheTest):
 
     def test_add_name_vhost_if_necessary(self):
         # pylint: disable=protected-access
-        self.config.save = mock.Mock()
+        self.config.add_name_vhost = mock.Mock()
         self.config.version = (2, 2)
         self.config._add_name_vhost_if_necessary(self.vh_truth[0])
-        self.assertTrue(self.config.save.called)
+        self.assertTrue(self.config.add_name_vhost.called)
 
         new_addrs = set()
         for addr in self.vh_truth[0].addrs:
@@ -728,7 +855,7 @@ class MultipleVhostsTest(util.ApacheTest):
 
         self.vh_truth[0].addrs = new_addrs
         self.config._add_name_vhost_if_necessary(self.vh_truth[0])
-        self.assertEqual(self.config.save.call_count, 2)
+        self.assertEqual(self.config.add_name_vhost.call_count, 2)
 
     @mock.patch("certbot_apache.configurator.tls_sni_01.ApacheTlsSni01.perform")
     @mock.patch("certbot_apache.configurator.ApacheConfigurator.restart")
@@ -915,7 +1042,6 @@ class MultipleVhostsTest(util.ApacheTest):
             "SSLUseStapling", "on", ssl_vhost.path)
 
         self.assertEqual(len(ssl_use_stapling_aug_path), 1)
-
         ssl_vhost_aug_path = parser.get_aug_path(ssl_vhost.filep)
         stapling_cache_aug_path = self.config.parser.find_dir('SSLStaplingCache',
                     "shmcb:/var/run/apache2/stapling_cache(128000)",
@@ -1177,7 +1303,7 @@ class MultipleVhostsTest(util.ApacheTest):
 
         # pylint: disable=protected-access
         self.config._enable_redirect(self.vh_truth[1], "")
-        self.assertEqual(len(self.config.vhosts), 9)
+        self.assertEqual(len(self.config.vhosts), 11)
 
     def test_create_own_redirect_for_old_apache_version(self):
         self.config.parser.modules.add("rewrite_module")
@@ -1188,7 +1314,7 @@ class MultipleVhostsTest(util.ApacheTest):
 
         # pylint: disable=protected-access
         self.config._enable_redirect(self.vh_truth[1], "")
-        self.assertEqual(len(self.config.vhosts), 9)
+        self.assertEqual(len(self.config.vhosts), 11)
 
     def test_sift_rewrite_rule(self):
         # pylint: disable=protected-access
@@ -1241,7 +1367,7 @@ class MultipleVhostsTest(util.ApacheTest):
         self.assertFalse(self.config._check_aug_version())
 
 class AugeasVhostsTest(util.ApacheTest):
-    """Test vhosts with illegal names dependant on augeas version."""
+    """Test vhosts with illegal names dependent on augeas version."""
     # pylint: disable=protected-access
     _multiprocess_can_split_ = True
 
@@ -1285,13 +1411,17 @@ class AugeasVhostsTest(util.ApacheTest):
         for name in names:
             self.assertFalse(name in self.config.choose_vhost(name).aliases)
 
-    def test_choose_vhost_without_matching_wildcard(self):
+    @mock.patch("certbot_apache.obj.VirtualHost.conflicts")
+    def test_choose_vhost_without_matching_wildcard(self, mock_conflicts):
+        mock_conflicts.return_value = False
         mock_path = "certbot_apache.display_ops.select_vhost"
         with mock.patch(mock_path, lambda _, vhosts: vhosts[0]):
             for name in ("a.example.net", "other.example.net"):
                 self.assertTrue(name in self.config.choose_vhost(name).aliases)
 
-    def test_choose_vhost_wildcard_not_found(self):
+    @mock.patch("certbot_apache.obj.VirtualHost.conflicts")
+    def test_choose_vhost_wildcard_not_found(self, mock_conflicts):
+        mock_conflicts.return_value = False
         mock_path = "certbot_apache.display_ops.select_vhost"
         names = (
             "abc.example.net", "not.there.tld", "aa.wildcard.tld"
@@ -1321,7 +1451,7 @@ class AugeasVhostsTest(util.ApacheTest):
                           broken_vhost)
 
 class MultiVhostsTest(util.ApacheTest):
-    """Test vhosts with illegal names dependant on augeas version."""
+    """Test vhosts with illegal names dependent on augeas version."""
     # pylint: disable=protected-access
 
     def setUp(self):  # pylint: disable=arguments-differ
@@ -1358,10 +1488,6 @@ class MultiVhostsTest(util.ApacheTest):
         self.assertTrue(ssl_vhost.ssl)
         self.assertFalse(ssl_vhost.enabled)
 
-        self.assertTrue(self.config.parser.find_dir(
-            "SSLCertificateFile", None, ssl_vhost.path, False))
-        self.assertTrue(self.config.parser.find_dir(
-            "SSLCertificateKeyFile", None, ssl_vhost.path, False))
 
         self.assertEqual(self.config.is_name_vhost(self.vh_truth[1]),
                          self.config.is_name_vhost(ssl_vhost))
@@ -1497,7 +1623,7 @@ class InstallSslOptionsConfTest(util.ApacheTest):
         with mock.patch("certbot.plugins.common.logger") as mock_logger:
             self._call()
             self.assertEqual(mock_logger.warning.call_args[0][0],
-                "%s has been manually modified; updated ssl configuration options "
+                "%s has been manually modified; updated file "
                 "saved to %s. We recommend updating %s for security purposes.")
         self.assertEqual(crypto_util.sha256sum(constants.os_constant("MOD_SSL_CONF_SRC")),
             self._current_ssl_options_hash())

certbot-apache/certbot_apache/tests/parser_test.py

@@ -38,7 +38,7 @@ class BasicParserTest(util.ParserTest):
         file_path = os.path.join(
             self.config_path, "not-parsed-by-default", "certbot.conf")
 
-        self.parser._parse_file(file_path)  # pylint: disable=protected-access
+        self.parser.parse_file(file_path)  # pylint: disable=protected-access
 
         # search for the httpd incl
         matches = self.parser.aug.match(
@@ -52,7 +52,7 @@ class BasicParserTest(util.ParserTest):
         test2 = self.parser.find_dir("documentroot")
 
         self.assertEqual(len(test), 1)
-        self.assertEqual(len(test2), 4)
+        self.assertEqual(len(test2), 7)
 
     def test_add_dir(self):
         aug_default = "/files" + self.parser.loc["default"]
@@ -66,6 +66,10 @@ class BasicParserTest(util.ParserTest):
         for i, match in enumerate(matches):
             self.assertEqual(self.parser.aug.get(match), str(i + 1))
 
+    def test_empty_arg(self):
+        self.assertEquals(None,
+                          self.parser.get_arg("/files/whatever/nonexistent"))
+
     def test_add_dir_to_ifmodssl(self):
         """test add_dir_to_ifmodssl.
 
@@ -114,6 +118,16 @@ class BasicParserTest(util.ParserTest):
             self.assertEqual(results["default"], results["listen"])
             self.assertEqual(results["default"], results["name"])
 
+    @mock.patch("certbot_apache.parser.ApacheParser.find_dir")
+    @mock.patch("certbot_apache.parser.ApacheParser.get_arg")
+    def test_init_modules_bad_syntax(self, mock_arg, mock_find):
+        mock_find.return_value = ["1", "2", "3", "4", "5", "6", "7", "8"]
+        mock_arg.return_value = None
+        with mock.patch("certbot_apache.parser.logger") as mock_logger:
+            self.parser.init_modules()
+            # Make sure that we got None return value and logged the file
+            self.assertTrue(mock_logger.debug.called)
+
     @mock.patch("certbot_apache.parser.ApacheParser._get_runtime_cfg")
     def test_update_runtime_variables(self, mock_cfg):
         mock_cfg.return_value = (

certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/another_wildcard.conf

@@ -0,0 +1 @@
+../sites-available/another_wildcard.conf

certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/old,default.conf

@@ -0,0 +1 @@
+../sites-available/old,default.conf

certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/augeas_vhosts/apache2/sites-enabled/wildcard.conf

@@ -0,0 +1 @@
+../sites-available/wildcard.conf

certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/apache2.conf

@@ -193,4 +193,15 @@ IncludeOptional conf-enabled/*.conf
 # Include the virtual host configurations:
 IncludeOptional sites-enabled/*.conf
 
+<VirtualHost *:80>
+
+	ServerName vhost.in.rootconf
+	ServerAdmin webmaster@localhost
+	DocumentRoot /var/www/html
+
+	ErrorLog ${APACHE_LOG_DIR}/error.log
+	CustomLog ${APACHE_LOG_DIR}/access.log combined
+
+</VirtualHost>
+
 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet

certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-available/encryption-example.conf

@@ -1,4 +1,4 @@
-<VirtualHost *:80>
+<Virtualhost *:80>
              ServerName encryption-example.demo
 	     ServerAdmin webmaster@localhost
 
@@ -39,4 +39,4 @@
 	      Allow from 127.0.0.0/255.0.0.0 ::1/128
     </Directory>
 
-</VirtualHost>
+</Virtualhost>

certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/default-ssl-port-only.conf

@@ -0,0 +1 @@
+../sites-available/default-ssl-port-only.conf

certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/default-ssl.conf

@@ -0,0 +1 @@
+../sites-available/default-ssl.conf

certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/non-symlink.conf

@@ -0,0 +1,9 @@
+<VirtualHost *:80>
+ServerName nonsym.link
+ServerAdmin webmaster@localhost
+
+DocumentRoot /var/www-certbot-reworld/static/
+
+ErrorLog ${APACHE_LOG_DIR}/error.log
+CustomLog ${APACHE_LOG_DIR}/access.log combined
+</VirtualHost>

certbot-apache/certbot_apache/tests/testdata/debian_apache_2_4/multiple_vhosts/apache2/sites-enabled/wildcard.conf

@@ -0,0 +1 @@
+../sites-available/wildcard.conf

certbot-apache/certbot_apache/tests/util.py

@@ -45,6 +45,9 @@ class ApacheTest(unittest.TestCase):  # pylint: disable=too-few-public-methods
             return
 
         for vhost_basename in os.listdir(sites_enabled):
+            # Keep the one non-symlink test vhost in place
+            if vhost_basename == "non-symlink.conf":
+                continue
             vhost = os.path.join(sites_enabled, vhost_basename)
             if not os.path.islink(vhost):  # pragma: no cover
                 os.remove(vhost)
@@ -115,18 +118,20 @@ def get_vh_truth(temp_dir, config_name):
     """Return the ground truth for the specified directory."""
     if config_name == "debian_apache_2_4/multiple_vhosts":
         prefix = os.path.join(
-            temp_dir, config_name, "apache2/sites-available")
+            temp_dir, config_name, "apache2/sites-enabled")
+
         aug_pre = "/files" + prefix
         vh_truth = [
             obj.VirtualHost(
                 os.path.join(prefix, "encryption-example.conf"),
-                os.path.join(aug_pre, "encryption-example.conf/VirtualHost"),
+                os.path.join(aug_pre, "encryption-example.conf/Virtualhost"),
                 set([obj.Addr.fromstring("*:80")]),
                 False, True, "encryption-example.demo"),
             obj.VirtualHost(
                 os.path.join(prefix, "default-ssl.conf"),
-                os.path.join(aug_pre, "default-ssl.conf/IfModule/VirtualHost"),
-                set([obj.Addr.fromstring("_default_:443")]), True, False),
+                os.path.join(aug_pre,
+                             "default-ssl.conf/IfModule/VirtualHost"),
+                set([obj.Addr.fromstring("_default_:443")]), True, True),
             obj.VirtualHost(
                 os.path.join(prefix, "000-default.conf"),
                 os.path.join(aug_pre, "000-default.conf/VirtualHost"),
@@ -148,17 +153,34 @@ def get_vh_truth(temp_dir, config_name):
                 os.path.join(prefix, "default-ssl-port-only.conf"),
                 os.path.join(aug_pre, ("default-ssl-port-only.conf/"
                                        "IfModule/VirtualHost")),
-                set([obj.Addr.fromstring("_default_:443")]), True, False),
+                set([obj.Addr.fromstring("_default_:443")]), True, True),
             obj.VirtualHost(
                 os.path.join(prefix, "wildcard.conf"),
                 os.path.join(aug_pre, "wildcard.conf/VirtualHost"),
-                set([obj.Addr.fromstring("*:80")]), False, False,
+                set([obj.Addr.fromstring("*:80")]), False, True,
                 "ip-172-30-0-17", aliases=["*.blue.purple.com"]),
             obj.VirtualHost(
                 os.path.join(prefix, "ocsp-ssl.conf"),
                 os.path.join(aug_pre, "ocsp-ssl.conf/IfModule/VirtualHost"),
                 set([obj.Addr.fromstring("10.2.3.4:443")]), True, True,
-                "ocspvhost.com")]
+                "ocspvhost.com"),
+            obj.VirtualHost(
+                os.path.join(prefix, "non-symlink.conf"),
+                os.path.join(aug_pre, "non-symlink.conf/VirtualHost"),
+                set([obj.Addr.fromstring("*:80")]), False, True,
+                "nonsym.link"),
+            obj.VirtualHost(
+                os.path.join(prefix, "default-ssl-port-only.conf"),
+                os.path.join(aug_pre,
+                             "default-ssl-port-only.conf/VirtualHost"),
+                set([obj.Addr.fromstring("*:80")]), True, True, ""),
+            obj.VirtualHost(
+                os.path.join(temp_dir, config_name,
+                             "apache2/apache2.conf"),
+                "/files" + os.path.join(temp_dir, config_name,
+                                        "apache2/apache2.conf/VirtualHost"),
+                set([obj.Addr.fromstring("*:80")]), False, True,
+                "vhost.in.rootconf")]
         return vh_truth
     if config_name == "debian_apache_2_4/multi_vhosts":
         prefix = os.path.join(

certbot-apache/certbot_apache/tls_sni_01.py

@@ -7,7 +7,6 @@ from certbot.plugins import common
 from certbot.errors import PluginError, MissingCommandlineFlag
 
 from certbot_apache import obj
-from certbot_apache import parser
 
 logger = logging.getLogger(__name__)
 
@@ -105,7 +104,8 @@ class ApacheTlsSni01(common.TLSSNI01):
 
         config_text += "</IfModule>\n"
 
-        self._conf_include_check(self.configurator.parser.loc["default"])
+        self.configurator.parser.add_include(
+            self.configurator.parser.loc["default"], self.challenge_conf)
         self.configurator.reverter.register_file_creation(
             True, self.challenge_conf)
 
@@ -142,24 +142,6 @@ class ApacheTlsSni01(common.TLSSNI01):
 
         return addrs
 
-    def _conf_include_check(self, main_config):
-        """Add TLS-SNI-01 challenge conf file into configuration.
-
-        Adds TLS-SNI-01 challenge include file if it does not already exist
-        within mainConfig
-
-        :param str main_config: file path to main user apache config file
-
-        """
-        if len(self.configurator.parser.find_dir(
-                parser.case_i("Include"), self.challenge_conf)) == 0:
-            # print "Including challenge virtual host(s)"
-            logger.debug("Adding Include %s to %s",
-                         self.challenge_conf, parser.get_aug_path(main_config))
-            self.configurator.parser.add_dir(
-                parser.get_aug_path(main_config),
-                "Include", self.challenge_conf)
-
     def _get_config_text(self, achall, ip_addrs):
         """Chocolate virtual server configuration text
 

certbot-apache/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [

certbot-auto

@@ -23,12 +23,15 @@ fi
 if [ -z "$XDG_DATA_HOME" ]; then
   XDG_DATA_HOME=~/.local/share
 fi
-VENV_NAME="letsencrypt"
 if [ -z "$VENV_PATH" ]; then
-  VENV_PATH="$XDG_DATA_HOME/$VENV_NAME"
+  # We export these values so they are preserved properly if this script is
+  # rerun with sudo/su where $HOME/$XDG_DATA_HOME may have a different value.
+  export OLD_VENV_PATH="$XDG_DATA_HOME/letsencrypt"
+  export VENV_PATH="/opt/eff.org/certbot/venv"
 fi
 VENV_BIN="$VENV_PATH/bin"
-LE_AUTO_VERSION="0.17.0"
+BOOTSTRAP_VERSION_PATH="$VENV_PATH/certbot-auto-bootstrap-version.txt"
+LE_AUTO_VERSION="0.19.0"
 BASENAME=$(basename $0)
 USAGE="Usage: $BASENAME [OPTIONS]
 A self-updating wrapper script for the Certbot ACME client. When run, updates
@@ -49,6 +52,7 @@ Help for certbot itself cannot be provided until it is installed.
                                             implies --non-interactive
 
 All arguments are accepted and forwarded to the Certbot client when run."
+export CERTBOT_AUTO="$0"
 
 for arg in "$@" ; do
   case "$arg" in
@@ -77,7 +81,7 @@ for arg in "$@" ; do
           h)
             HELP=1;;
           n)
-            ASSUME_YES=1;;
+            NONINTERACTIVE=1;;
           q)
             QUIET=1;;
           v)
@@ -93,8 +97,8 @@ if [ $BASENAME = "letsencrypt-auto" ]; then
   HELP=0
 fi
 
-# Set ASSUME_YES to 1 if QUIET (i.e. --quiet implies --non-interactive)
-if [ "$QUIET" = 1 ]; then
+# Set ASSUME_YES to 1 if QUIET or NONINTERACTIVE
+if [ "$QUIET" = 1 -o "$NONINTERACTIVE" = 1 ]; then
   ASSUME_YES=1
 fi
 
@@ -119,16 +123,18 @@ else
   exit 1
 fi
 
-# certbot-auto needs root access to bootstrap OS dependencies, and
-# certbot itself needs root access for almost all modes of operation
-# The "normal" case is that sudo is used for the steps that need root, but
-# this script *can* be run as root (not recommended), or fall back to using
-# `su`. Auto-detection can be overridden by explicitly setting the
-# environment variable LE_AUTO_SUDO to 'sudo', 'sudo_su' or '' as used below.
+# Certbot itself needs root access for almost all modes of operation.
+# certbot-auto needs root access to bootstrap OS dependencies and install
+# Certbot at a protected path so it can be safely run as root. To accomplish
+# this, this script will attempt to run itself as root if it doesn't have the
+# necessary privileges by using `sudo` or falling back to `su` if it is not
+# available. The mechanism used to obtain root access can be set explicitly by
+# setting the environment variable LE_AUTO_SUDO to 'sudo', 'su', 'su_sudo',
+# 'SuSudo', or '' as used below.
 
 # Because the parameters in `su -c` has to be a string,
 # we need to properly escape it.
-su_sudo() {
+SuSudo() {
   args=""
   # This `while` loop iterates over all parameters given to this function.
   # For each parameter, all `'` will be replace by `'"'"'`, and the escaped string
@@ -147,37 +153,57 @@ su_sudo() {
   su root -c "$args"
 }
 
-SUDO_ENV=""
-export CERTBOT_AUTO="$0"
-if [ -n "${LE_AUTO_SUDO+x}" ]; then
-  case "$LE_AUTO_SUDO" in
-    su_sudo|su)
-      SUDO=su_sudo
-      ;;
-    sudo)
-      SUDO=sudo
-      SUDO_ENV="CERTBOT_AUTO=$0"
-      ;;
-    '') ;; # Nothing to do for plain root method.
-    *)
-      error "Error: unknown root authorization mechanism '$LE_AUTO_SUDO'."
-      exit 1
-  esac
-  say "Using preset root authorization mechanism '$LE_AUTO_SUDO'."
-else
-  if test "`id -u`" -ne "0" ; then
-    if $EXISTS sudo 1>/dev/null 2>&1; then
-      SUDO=sudo
-      SUDO_ENV="CERTBOT_AUTO=$0"
-    else
-      say \"sudo\" is not available, will use \"su\" for installation steps...
-      SUDO=su_sudo
-    fi
+# Sets the environment variable SUDO to be the name of the program or function
+# to call to get root access. If this script already has root privleges, SUDO
+# is set to an empty string. The value in SUDO should be run with the command
+# to called with root privileges as arguments.
+SetRootAuthMechanism() {
+  SUDO=""
+  if [ -n "${LE_AUTO_SUDO+x}" ]; then
+    case "$LE_AUTO_SUDO" in
+      SuSudo|su_sudo|su)
+        SUDO=SuSudo
+        ;;
+      sudo)
+        SUDO="sudo -E"
+        ;;
+      '') ;; # Nothing to do for plain root method.
+      *)
+        error "Error: unknown root authorization mechanism '$LE_AUTO_SUDO'."
+        exit 1
+    esac
+    say "Using preset root authorization mechanism '$LE_AUTO_SUDO'."
   else
-    SUDO=
+    if test "`id -u`" -ne "0" ; then
+      if $EXISTS sudo 1>/dev/null 2>&1; then
+        SUDO="sudo -E"
+      else
+        say \"sudo\" is not available, will use \"su\" for installation steps...
+        SUDO=SuSudo
+      fi
+    fi
+  fi
+}
+
+if [ "$1" = "--cb-auto-has-root" ]; then
+  shift 1
+else
+  SetRootAuthMechanism
+  if [ -n "$SUDO" ]; then
+    echo "Requesting to rerun $0 with root privileges..."
+    $SUDO "$0" --cb-auto-has-root "$@"
+    exit 0
   fi
 fi
 
+# Runs this script again with the given arguments. --cb-auto-has-root is added
+# to the command line arguments to ensure we don't try to acquire root a
+# second time. After the script is rerun, we exit the current script.
+RerunWithArgs() {
+    "$0" --cb-auto-has-root "$@"
+    exit 0
+}
+
 BootstrapMessage() {
   # Arguments: Platform name
   say "Bootstrapping dependencies for $1... (you can skip this with --no-bootstrap)"
@@ -238,6 +264,10 @@ DeterminePythonVersion() {
   fi
 }
 
+# If new packages are installed by BootstrapDebCommon below, this version
+# number must be increased.
+BOOTSTRAP_DEB_COMMON_VERSION=1
+
 BootstrapDebCommon() {
   # Current version tested with:
   #
@@ -261,7 +291,7 @@ BootstrapDebCommon() {
     QUIET_FLAG='-qq'
   fi
 
-  $SUDO apt-get $QUIET_FLAG update || error apt-get update hit problems but continuing anyway...
+  apt-get $QUIET_FLAG update || error apt-get update hit problems but continuing anyway...
 
   # virtualenv binary can be found in different packages depending on
   # distro version (#346)
@@ -311,13 +341,13 @@ BootstrapDebCommon() {
           esac
         fi
         if [ "$add_backports" = 1 ]; then
-          $SUDO sh -c "echo $BACKPORT_SOURCELINE >> /etc/apt/sources.list.d/$BACKPORT_NAME.list"
-          $SUDO apt-get $QUIET_FLAG update
+          sh -c "echo $BACKPORT_SOURCELINE >> /etc/apt/sources.list.d/$BACKPORT_NAME.list"
+          apt-get $QUIET_FLAG update
         fi
       fi
     fi
     if [ "$add_backports" != 0 ]; then
-      $SUDO apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends -t "$BACKPORT_NAME" $augeas_pkg
+      apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends -t "$BACKPORT_NAME" $augeas_pkg
       augeas_pkg=
     fi
   }
@@ -336,7 +366,7 @@ BootstrapDebCommon() {
     # XXX add a case for ubuntu PPAs
   fi
 
-  $SUDO apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends \
+  apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends \
     python \
     python-dev \
     $virtualenv \
@@ -354,6 +384,10 @@ BootstrapDebCommon() {
   fi
 }
 
+# If new packages are installed by BootstrapRpmCommon below, this version
+# number must be increased.
+BOOTSTRAP_RPM_COMMON_VERSION=1
+
 BootstrapRpmCommon() {
   # Tested with:
   #   - Fedora 20, 21, 22, 23 (x64)
@@ -380,9 +414,9 @@ BootstrapRpmCommon() {
     QUIET_FLAG='--quiet'
   fi
 
-  if ! $SUDO $tool list *virtualenv >/dev/null 2>&1; then
+  if ! $tool list *virtualenv >/dev/null 2>&1; then
     echo "To use Certbot, packages from the EPEL repository need to be installed."
-    if ! $SUDO $tool list epel-release >/dev/null 2>&1; then
+    if ! $tool list epel-release >/dev/null 2>&1; then
       error "Enable the EPEL repository and try running Certbot again."
       exit 1
     fi
@@ -394,7 +428,7 @@ BootstrapRpmCommon() {
       /bin/echo -e "\e[0K\rEnabling the EPEL repository in 1 seconds..."
       sleep 1s
     fi
-    if ! $SUDO $tool install $yes_flag $QUIET_FLAG epel-release; then
+    if ! $tool install $yes_flag $QUIET_FLAG epel-release; then
       error "Could not enable EPEL. Aborting bootstrap!"
       exit 1
     fi
@@ -410,9 +444,8 @@ BootstrapRpmCommon() {
     ca-certificates
   "
 
-  # Some distros and older versions of current distros use a "python27"
-  # instead of "python" naming convention. Try both conventions.
-  if $SUDO $tool list python >/dev/null 2>&1; then
+  # Most RPM distros use the "python" or "python-" naming convention.  Let's try that first.
+  if $tool list python >/dev/null 2>&1; then
     pkgs="$pkgs
       python
       python-devel
@@ -420,6 +453,20 @@ BootstrapRpmCommon() {
       python-tools
       python-pip
     "
+  # Fedora 26 starts to use the prefix python2 for python2 based packages.
+  # this elseif is theoretically for any Fedora over version 26:
+  elif $tool list python2 >/dev/null 2>&1; then
+    pkgs="$pkgs
+      python2
+      python2-libs
+      python2-setuptools
+      python2-devel
+      python2-virtualenv
+      python2-tools
+      python2-pip
+    "
+  # Some distros and older versions of current distros use a "python27"
+  # instead of the "python" or "python-" naming convention.
   else
     pkgs="$pkgs
       python27
@@ -430,18 +477,22 @@ BootstrapRpmCommon() {
     "
   fi
 
-  if $SUDO $tool list installed "httpd" >/dev/null 2>&1; then
+  if $tool list installed "httpd" >/dev/null 2>&1; then
     pkgs="$pkgs
       mod_ssl
     "
   fi
 
-  if ! $SUDO $tool install $yes_flag $QUIET_FLAG $pkgs; then
+  if ! $tool install $yes_flag $QUIET_FLAG $pkgs; then
     error "Could not install OS dependencies. Aborting bootstrap!"
     exit 1
   fi
 }
 
+# If new packages are installed by BootstrapSuseCommon below, this version
+# number must be increased.
+BOOTSTRAP_SUSE_COMMON_VERSION=1
+
 BootstrapSuseCommon() {
   # SLE12 don't have python-virtualenv
 
@@ -454,7 +505,7 @@ BootstrapSuseCommon() {
     QUIET_FLAG='-qq'
   fi
 
-  $SUDO zypper $QUIET_FLAG $zypper_flags in $install_flags \
+  zypper $QUIET_FLAG $zypper_flags in $install_flags \
     python \
     python-devel \
     python-virtualenv \
@@ -465,6 +516,10 @@ BootstrapSuseCommon() {
     ca-certificates
 }
 
+# If new packages are installed by BootstrapArchCommon below, this version
+# number must be increased.
+BOOTSTRAP_ARCH_COMMON_VERSION=1
+
 BootstrapArchCommon() {
   # Tested with:
   #   - ArchLinux (x86_64)
@@ -485,21 +540,25 @@ BootstrapArchCommon() {
   "
 
   # pacman -T exits with 127 if there are missing dependencies
-  missing=$($SUDO pacman -T $deps) || true
+  missing=$(pacman -T $deps) || true
 
   if [ "$ASSUME_YES" = 1 ]; then
     noconfirm="--noconfirm"
   fi
 
   if [ "$missing" ]; then
-    if [ "$QUIET" = 1]; then
-      $SUDO pacman -S --needed $missing $noconfirm > /dev/null
+    if [ "$QUIET" = 1 ]; then
+      pacman -S --needed $missing $noconfirm > /dev/null
     else
-      $SUDO pacman -S --needed $missing $noconfirm
+      pacman -S --needed $missing $noconfirm
     fi
   fi
 }
 
+# If new packages are installed by BootstrapGentooCommon below, this version
+# number must be increased.
+BOOTSTRAP_GENTOO_COMMON_VERSION=1
+
 BootstrapGentooCommon() {
   PACKAGES="
     dev-lang/python:2.7
@@ -517,29 +576,37 @@ BootstrapGentooCommon() {
 
   case "$PACKAGE_MANAGER" in
     (paludis)
-      $SUDO cave resolve --preserve-world --keep-targets if-possible $PACKAGES -x
+      cave resolve --preserve-world --keep-targets if-possible $PACKAGES -x
       ;;
     (pkgcore)
-      $SUDO pmerge --noreplace --oneshot $ASK_OPTION $PACKAGES
+      pmerge --noreplace --oneshot $ASK_OPTION $PACKAGES
       ;;
     (portage|*)
-      $SUDO emerge --noreplace --oneshot $ASK_OPTION $PACKAGES
+      emerge --noreplace --oneshot $ASK_OPTION $PACKAGES
       ;;
   esac
 }
 
+# If new packages are installed by BootstrapFreeBsd below, this version number
+# must be increased.
+BOOTSTRAP_FREEBSD_VERSION=1
+
 BootstrapFreeBsd() {
   if [ "$QUIET" = 1 ]; then
     QUIET_FLAG="--quiet"
   fi
 
-  $SUDO pkg install -Ay $QUIET_FLAG \
+  pkg install -Ay $QUIET_FLAG \
     python \
     py27-virtualenv \
     augeas \
     libffi
 }
 
+# If new packages are installed by BootstrapMac below, this version number must
+# be increased.
+BOOTSTRAP_MAC_VERSION=1
+
 BootstrapMac() {
   if hash brew 2>/dev/null; then
     say "Using Homebrew to install dependencies..."
@@ -548,7 +615,7 @@ BootstrapMac() {
   elif hash port 2>/dev/null; then
     say "Using MacPorts to install dependencies..."
     pkgman=port
-    pkgcmd="$SUDO port install"
+    pkgcmd="port install"
   else
     say "No Homebrew/MacPorts; installing Homebrew..."
     ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
@@ -568,8 +635,8 @@ BootstrapMac() {
   # Workaround for _dlopen not finding augeas on macOS
   if [ "$pkgman" = "port" ] && ! [ -e "/usr/local/lib/libaugeas.dylib" ] && [ -e "/opt/local/lib/libaugeas.dylib" ]; then
     say "Applying augeas workaround"
-    $SUDO mkdir -p /usr/local/lib/
-    $SUDO ln -s /opt/local/lib/libaugeas.dylib /usr/local/lib/
+    mkdir -p /usr/local/lib/
+    ln -s /opt/local/lib/libaugeas.dylib /usr/local/lib/
   fi
 
   if ! hash pip 2>/dev/null; then
@@ -585,17 +652,25 @@ BootstrapMac() {
   fi
 }
 
+# If new packages are installed by BootstrapSmartOS below, this version number
+# must be increased.
+BOOTSTRAP_SMARTOS_VERSION=1
+
 BootstrapSmartOS() {
   pkgin update
   pkgin -y install 'gcc49' 'py27-augeas' 'py27-virtualenv'
 }
 
+# If new packages are installed by BootstrapMageiaCommon below, this version
+# number must be increased.
+BOOTSTRAP_MAGEIA_COMMON_VERSION=1
+
 BootstrapMageiaCommon() {
   if [ "$QUIET" = 1 ]; then
     QUIET_FLAG='--quiet'
   fi
 
-  if ! $SUDO urpmi --force $QUIET_FLAG \
+  if ! urpmi --force $QUIET_FLAG \
       python \
       libpython-devel \
       python-virtualenv
@@ -604,7 +679,7 @@ BootstrapMageiaCommon() {
       exit 1
   fi
 
-  if ! $SUDO urpmi --force $QUIET_FLAG \
+  if ! urpmi --force $QUIET_FLAG \
       git \
       gcc \
       python-augeas \
@@ -618,23 +693,41 @@ BootstrapMageiaCommon() {
 }
 
 
-# Install required OS packages:
-Bootstrap() {
-  if [ "$NO_BOOTSTRAP" = 1 ]; then
-      return
-  elif [ -f /etc/debian_version ]; then
+# Set Bootstrap to the function that installs OS dependencies on this system
+# and BOOTSTRAP_VERSION to the unique identifier for the current version of
+# that function. If Bootstrap is set to a function that doesn't install any
+# packages (either because --no-bootstrap was included on the command line or
+# we don't know how to bootstrap on this system), BOOTSTRAP_VERSION is not set.
+if [ "$NO_BOOTSTRAP" = 1 ]; then
+  Bootstrap() {
+    :
+  }
+elif [ -f /etc/debian_version ]; then
+  Bootstrap() {
     BootstrapMessage "Debian-based OSes"
     BootstrapDebCommon
-  elif [ -f /etc/mageia-release ]; then
-    # Mageia has both /etc/mageia-release and /etc/redhat-release
+  }
+  BOOTSTRAP_VERSION="BootstrapDebCommon $BOOTSTRAP_DEB_COMMON_VERSION"
+elif [ -f /etc/mageia-release ]; then
+  # Mageia has both /etc/mageia-release and /etc/redhat-release
+  Bootstrap() {
     ExperimentalBootstrap "Mageia" BootstrapMageiaCommon
-  elif [ -f /etc/redhat-release ]; then
+  }
+  BOOTSTRAP_VERSION="BootstrapMageiaCommon $BOOTSTRAP_MAGEIA_COMMON_VERSION"
+elif [ -f /etc/redhat-release ]; then
+  Bootstrap() {
     BootstrapMessage "RedHat-based OSes"
     BootstrapRpmCommon
-  elif [ -f /etc/os-release ] && `grep -q openSUSE /etc/os-release` ; then
+  }
+  BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION"
+elif [ -f /etc/os-release ] && `grep -q openSUSE /etc/os-release` ; then
+  Bootstrap() {
     BootstrapMessage "openSUSE-based OSes"
     BootstrapSuseCommon
-  elif [ -f /etc/arch-release ]; then
+  }
+  BOOTSTRAP_VERSION="BootstrapSuseCommon $BOOTSTRAP_SUSE_COMMON_VERSION"
+elif [ -f /etc/arch-release ]; then
+  Bootstrap() {
     if [ "$DEBUG" = 1 ]; then
       BootstrapMessage "Archlinux"
       BootstrapArchCommon
@@ -646,25 +739,76 @@ Bootstrap() {
       error "--debug flag."
       exit 1
     fi
-  elif [ -f /etc/manjaro-release ]; then
+  }
+  BOOTSTRAP_VERSION="BootstrapArchCommon $BOOTSTRAP_ARCH_COMMON_VERSION"
+elif [ -f /etc/manjaro-release ]; then
+  Bootstrap() {
     ExperimentalBootstrap "Manjaro Linux" BootstrapArchCommon
-  elif [ -f /etc/gentoo-release ]; then
+  }
+  BOOTSTRAP_VERSION="BootstrapArchCommon $BOOTSTRAP_ARCH_COMMON_VERSION"
+elif [ -f /etc/gentoo-release ]; then
+  Bootstrap() {
     DeprecationBootstrap "Gentoo" BootstrapGentooCommon
-  elif uname | grep -iq FreeBSD ; then
+  }
+  BOOTSTRAP_VERSION="BootstrapGentooCommon $BOOTSTRAP_GENTOO_COMMON_VERSION"
+elif uname | grep -iq FreeBSD ; then
+  Bootstrap() {
     DeprecationBootstrap "FreeBSD" BootstrapFreeBsd
-  elif uname | grep -iq Darwin ; then
+  }
+  BOOTSTRAP_VERSION="BootstrapFreeBsd $BOOTSTRAP_FREEBSD_VERSION"
+elif uname | grep -iq Darwin ; then
+  Bootstrap() {
     DeprecationBootstrap "macOS" BootstrapMac
-  elif [ -f /etc/issue ] && grep -iq "Amazon Linux" /etc/issue ; then
+  }
+  BOOTSTRAP_VERSION="BootstrapMac $BOOTSTRAP_MAC_VERSION"
+elif [ -f /etc/issue ] && grep -iq "Amazon Linux" /etc/issue ; then
+  Bootstrap() {
     ExperimentalBootstrap "Amazon Linux" BootstrapRpmCommon
-  elif [ -f /etc/product ] && grep -q "Joyent Instance" /etc/product ; then
+  }
+  BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION"
+elif [ -f /etc/product ] && grep -q "Joyent Instance" /etc/product ; then
+  Bootstrap() {
     ExperimentalBootstrap "Joyent SmartOS Zone" BootstrapSmartOS
-  else
+  }
+  BOOTSTRAP_VERSION="BootstrapSmartOS $BOOTSTRAP_SMARTOS_VERSION"
+else
+  Bootstrap() {
     error "Sorry, I don't know how to bootstrap Certbot on your operating system!"
     error
     error "You will need to install OS dependencies, configure virtualenv, and run pip install manually."
     error "Please see https://letsencrypt.readthedocs.org/en/latest/contributing.html#prerequisites"
     error "for more info."
     exit 1
+  }
+fi
+
+# Sets PREV_BOOTSTRAP_VERSION to the identifier for the bootstrap script used
+# to install OS dependencies on this system. PREV_BOOTSTRAP_VERSION isn't set
+# if it is unknown how OS dependencies were installed on this system.
+SetPrevBootstrapVersion() {
+  if [ -f $BOOTSTRAP_VERSION_PATH ]; then
+    PREV_BOOTSTRAP_VERSION=$(cat "$BOOTSTRAP_VERSION_PATH")
+  # The list below only contains bootstrap version strings that existed before
+  # we started writing them to disk.
+  #
+  # DO NOT MODIFY THIS LIST UNLESS YOU KNOW WHAT YOU'RE DOING!
+  elif grep -Fqx "$BOOTSTRAP_VERSION" << "UNLIKELY_EOF"
+BootstrapDebCommon 1
+BootstrapMageiaCommon 1
+BootstrapRpmCommon 1
+BootstrapSuseCommon 1
+BootstrapArchCommon 1
+BootstrapGentooCommon 1
+BootstrapFreeBsd 1
+BootstrapMac 1
+BootstrapSmartOS 1
+UNLIKELY_EOF
+  then
+    # If there's no bootstrap version saved to disk, but the currently selected
+    # bootstrap script is from before we started saving the version number,
+    # return the currently selected version to prevent us from rebootstrapping
+    # unnecessarily.
+    PREV_BOOTSTRAP_VERSION="$BOOTSTRAP_VERSION"
   fi
 }
 
@@ -678,18 +822,38 @@ if [ "$1" = "--le-auto-phase2" ]; then
   # Phase 2: Create venv, install LE, and run.
 
   shift 1  # the --le-auto-phase2 arg
-  if [ -f "$VENV_BIN/letsencrypt" ]; then
-    # --version output ran through grep due to python-cryptography DeprecationWarnings
-    # grep for both certbot and letsencrypt until certbot and shim packages have been released
-    INSTALLED_VERSION=$("$VENV_BIN/letsencrypt" --version 2>&1 | grep "^certbot\|^letsencrypt" | cut -d " " -f 2)
-    if [ -z "$INSTALLED_VERSION" ]; then
-        error "Error: couldn't get currently installed version for $VENV_BIN/letsencrypt: " 1>&2
-        "$VENV_BIN/letsencrypt" --version
-        exit 1
+  SetPrevBootstrapVersion
+
+  INSTALLED_VERSION="none"
+  if [ -d "$VENV_PATH" ]; then
+    # If the selected Bootstrap function isn't a noop and it differs from the
+    # previously used version
+    if [ -n "$BOOTSTRAP_VERSION" -a "$BOOTSTRAP_VERSION" != "$PREV_BOOTSTRAP_VERSION" ]; then
+      # if non-interactive mode or stdin and stdout are connected to a terminal
+      if [ \( "$NONINTERACTIVE" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then
+        rm -rf "$VENV_PATH"
+        RerunWithArgs "$@"
+      else
+        error "Skipping upgrade because new OS dependencies may need to be installed."
+        error
+        error "To upgrade to a newer version, please run this script again manually so you can"
+        error "approve changes or with --non-interactive on the command line to automatically"
+        error "install any required packages."
+        # Set INSTALLED_VERSION to be the same so we don't update the venv
+        INSTALLED_VERSION="$LE_AUTO_VERSION"
+      fi
+    elif [ -f "$VENV_BIN/letsencrypt" ]; then
+      # --version output ran through grep due to python-cryptography DeprecationWarnings
+      # grep for both certbot and letsencrypt until certbot and shim packages have been released
+      INSTALLED_VERSION=$("$VENV_BIN/letsencrypt" --version 2>&1 | grep "^certbot\|^letsencrypt" | cut -d " " -f 2)
+      if [ -z "$INSTALLED_VERSION" ]; then
+          error "Error: couldn't get currently installed version for $VENV_BIN/letsencrypt: " 1>&2
+          "$VENV_BIN/letsencrypt" --version
+          exit 1
+      fi
     fi
-  else
-    INSTALLED_VERSION="none"
   fi
+
   if [ "$LE_AUTO_VERSION" != "$INSTALLED_VERSION" ]; then
     say "Creating virtual environment..."
     DeterminePythonVersion
@@ -700,6 +864,12 @@ if [ "$1" = "--le-auto-phase2" ]; then
       virtualenv --no-site-packages --python "$LE_PYTHON" "$VENV_PATH" > /dev/null
     fi
 
+    if [ -n "$BOOTSTRAP_VERSION" ]; then
+      echo "$BOOTSTRAP_VERSION" > "$BOOTSTRAP_VERSION_PATH"
+    elif [ -n "$PREV_BOOTSTRAP_VERSION" ]; then
+      echo "$PREV_BOOTSTRAP_VERSION" > "$BOOTSTRAP_VERSION_PATH"
+    fi
+
     say "Installing Python packages..."
     TEMP_DIR=$(TempDir)
     trap 'rm -rf "$TEMP_DIR"' EXIT
@@ -766,8 +936,8 @@ cffi==1.10.0 \
     --hash=sha256:285ab352552f52f1398c912556d4d36d4ea9b8450e5c65d03809bf9886755533 \
     --hash=sha256:5576644b859197da7bbd8f8c7c2fb5dcc6cd505cadb42992d5f104c013f8a214 \
     --hash=sha256:b3b02911eb1f6ada203b0763ba924234629b51586f72a21faacc638269f4ced5
-ConfigArgParse==0.10.0 \
-    --hash=sha256:3b50a83dd58149dfcee98cb6565265d10b53e9c0a2bca7eeef7fb5f5524890a7
+ConfigArgParse==0.12.0 \
+    --hash=sha256:28cd7d67669651f2a4518367838c49539457504584a139709b2b8f6c208ef339
 configobj==5.0.6 \
     --hash=sha256:a2f5650770e1c87fb335af19a9b7eb73fc05ccf22144eb68db7d00cd2bcb0902
 cryptography==2.0.2 \
@@ -907,18 +1077,18 @@ letsencrypt==0.7.0 \
     --hash=sha256:105a5fb107e45bcd0722eb89696986dcf5f08a86a321d6aef25a0c7c63375ade \
     --hash=sha256:c36e532c486a7e92155ee09da54b436a3c420813ec1c590b98f635d924720de9
 
-certbot==0.17.0 \
-    --hash=sha256:64c25c7123357feffded6408660bc6f5c7d493dd635ae172081d21473075a86a \
-    --hash=sha256:43f5b26c3f314d14babf79a3bdf3522e4fc9eef867a0681c426f113c650a669c
-acme==0.17.0 \
-    --hash=sha256:501710171633af13fc52aa61d0277a6fe335f7477db5810e72239aaf4f3a09e7 \
-    --hash=sha256:3ccbe4aaeb98c77b98ee4093b4e4adb76a1a24cbdfec0130c489c206f1d9b66e
-certbot-apache==0.17.0 \
-    --hash=sha256:17a7e8d7526d838610e68b96cf052af17c4055655b76b06d1cbc74857d90a216 \
-    --hash=sha256:29b9e7bc5eaaff6dc4bce8398e35eeacdf346126aad68cac3d41bb87df20a6b9
-certbot-nginx==0.17.0 \
-    --hash=sha256:980c9a33a79ab839a089a0085ff0c5414f01f47b6db26ed342df25916658cec9 \
-    --hash=sha256:e573f8b4283172755c07b9cca8a8da7ef2d31b4df763881394b5339b2d42994a
+certbot==0.19.0 \
+    --hash=sha256:3207ee5319bfc37e855c25a43148275fcfb37869eefde9087405012049734a20 \
+    --hash=sha256:a7230791dff5d085738119fc22d88ad9d8a35d0b6a3d67806fe33990c7c79d53
+acme==0.19.0 \
+    --hash=sha256:c612eafe234d722d97bb5d3dbc49e5522f44be29611f7577954eb893e5c2d6de \
+    --hash=sha256:1fa23d64d494aaf001e6fe857c461fcfff10f75a1c2c35ec831447f641e1e822
+certbot-apache==0.19.0 \
+    --hash=sha256:fadb28b33bfabc85cdb962b5b149bef58b98f0606b78581db7895fe38323f37c \
+    --hash=sha256:70306ca2d5be7f542af68d46883c0ae39527cf202f17ef92cd256fb0bc3f1619
+certbot-nginx==0.19.0 \
+    --hash=sha256:4909cb3db49919fb35590793cac28e1c0b6dbd29cbedf887b9106e5fcef5362c \
+    --hash=sha256:cb5a224a3f277092555c25096d1678fc735306fd3a43447649ebe524c7ca79e1
 
 UNLIKELY_EOF
     # -------------------------------------------------------------------------
@@ -1131,20 +1301,15 @@ UNLIKELY_EOF
       rm -rf "$VENV_PATH"
       exit 1
     fi
+
+    if [ -d "$OLD_VENV_PATH" -a ! -L "$OLD_VENV_PATH" ]; then
+      rm -rf "$OLD_VENV_PATH"
+      ln -s "$VENV_PATH" "$OLD_VENV_PATH"
+    fi
+
     say "Installation succeeded."
   fi
-  if [ -n "$SUDO" ]; then
-    # SUDO is su wrapper or sudo
-    say "Requesting root privileges to run certbot..."
-    say "  $VENV_BIN/letsencrypt" "$@"
-  fi
-  if [ -z "$SUDO_ENV" ] ; then
-    # SUDO is su wrapper / noop
-    $SUDO "$VENV_BIN/letsencrypt" "$@"
-  else
-    # sudo
-    $SUDO "$SUDO_ENV" "$VENV_BIN/letsencrypt" "$@"
-  fi
+  "$VENV_BIN/letsencrypt" "$@"
 
 else
   # Phase 1: Upgrade certbot-auto if necessary, then self-invoke.
@@ -1155,12 +1320,14 @@ else
   # package). Phase 2 checks the version of the locally installed certbot.
 
   if [ ! -f "$VENV_BIN/letsencrypt" ]; then
-    if [ "$HELP" = 1 ]; then
-      echo "$USAGE"
-      exit 0
+    if [ -z "$OLD_VENV_PATH" -o ! -f "$OLD_VENV_PATH/bin/letsencrypt" ]; then
+      if [ "$HELP" = 1 ]; then
+        echo "$USAGE"
+        exit 0
+      fi
+      # If it looks like we've never bootstrapped before, bootstrap:
+      Bootstrap
     fi
-    # If it looks like we've never bootstrapped before, bootstrap:
-    Bootstrap
   fi
   if [ "$OS_PACKAGES_ONLY" = 1 ]; then
     say "OS packages installed."
@@ -1194,7 +1361,8 @@ from os.path import dirname, join
 import re
 from subprocess import check_call, CalledProcessError
 from sys import argv, exit
-from urllib2 import build_opener, HTTPHandler, HTTPSHandler, HTTPError
+from urllib2 import build_opener, HTTPHandler, HTTPSHandler
+from urllib2 import HTTPError, URLError
 
 PUBLIC_KEY = environ.get('LE_AUTO_PUBLIC_KEY', """-----BEGIN PUBLIC KEY-----
 MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6MR8W/galdxnpGqBsYbq
@@ -1230,7 +1398,9 @@ class HttpsGetter(object):
 
         """
         try:
-            return self._opener.open(url).read()
+            # socket module docs say default timeout is None: that is, no
+            # timeout
+            return self._opener.open(url, timeout=30).read()
         except (HTTPError, IOError) as exc:
             raise ExpectedError("Couldn't download %s." % url, exc)
 
@@ -1320,15 +1490,15 @@ UNLIKELY_EOF
       say "Replacing certbot-auto..."
       # Clone permissions with cp. chmod and chown don't have a --reference
       # option on macOS or BSD, and stat -c on Linux is stat -f on macOS and BSD:
-      $SUDO cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone"
-      $SUDO cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone"
+      cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone"
+      cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone"
       # Using mv rather than cp leaves the old file descriptor pointing to the
       # original copy so the shell can continue to read it unmolested. mv across
       # filesystems is non-atomic, doing `rm dest, cp src dest, rm src`, but the
-      # cp is unlikely to fail (esp. under sudo) if the rm doesn't.
-      $SUDO mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0"
+      # cp is unlikely to fail if the rm doesn't.
+      mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0"
     fi  # A newer version is available.
   fi  # Self-upgrading is allowed.
 
-  "$0" --le-auto-phase2 "$@"
+  RerunWithArgs --le-auto-phase2 "$@"
 fi

certbot-compatibility-test/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 install_requires = [
     'certbot',

certbot-dns-cloudflare/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [

certbot-dns-cloudxns/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [

certbot-dns-digitalocean/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [

certbot-dns-dnsimple/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [

certbot-dns-dnsmadeeasy/setup.py

@@ -4,13 +4,15 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [
     'acme=={0}'.format(version),
     'certbot=={0}'.format(version),
-    'dns-lexicon',
+    # new versions of lexicon require that we install dnsmadeeasy extras and
+    # 2.1.11 is the first version that defines them.
+    'dns-lexicon[dnsmadeeasy]>=2.1.11',
     'mock',
     # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
     # will tolerate; see #2599:

certbot-dns-google/certbot_dns_google/__init__.py

@@ -10,7 +10,7 @@ Named Arguments
 ========================================  =====================================
 ``--dns-google-credentials``              Google Cloud Platform credentials_
                                           JSON file.
-                                          (Required)
+                                          (Required - Optional on Google Compute Engine)
 ``--dns-google-propagation-seconds``      The number of seconds to wait for DNS
                                           to propagate before asking the ACME
                                           server to verify the DNS record.
@@ -21,8 +21,8 @@ Named Arguments
 Credentials
 -----------
 
-Use of this plugin requires a configuration file containing Google Cloud
-Platform API credentials for an account with the following permissions:
+Use of this plugin requires Google Cloud Platform API credentials
+for an account with the following permissions:
 
 * ``dns.changes.create``
 * ``dns.changes.get``
@@ -33,7 +33,12 @@ Platform API credentials for an account with the following permissions:
 Google provides instructions for `creating a service account <https://developers
 .google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount>`_ and
 `information about the required permissions <https://cloud.google.com/dns/access
--control#permissions_and_roles>`_.
+-control#permissions_and_roles>`_. If you're running on Google Compute Engine,
+you can `assign the service account to the instance <https://cloud.google.com/
+compute/docs/access/create-enable-service-accounts-for-instances>`_ which
+is running certbot. A credentials file is not required in this case, as they
+are automatically obtained by certbot through the `metadata service
+<https://cloud.google.com/compute/docs/storing-retrieving-metadata>`_ .
 
 .. code-block:: json
    :name: credentials.json

certbot-dns-google/certbot_dns_google/dns_google.py

@@ -2,6 +2,7 @@
 import json
 import logging
 
+import httplib2
 import zope.interface
 from googleapiclient import discovery
 from googleapiclient import errors as googleapiclient_errors
@@ -15,6 +16,8 @@ logger = logging.getLogger(__name__)
 
 ACCT_URL = 'https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount'
 PERMISSIONS_URL = 'https://cloud.google.com/dns/access-control#permissions_and_roles'
+METADATA_URL = 'http://metadata.google.internal/computeMetadata/v1/'
+METADATA_HEADERS = {'Metadata-Flavor': 'Google'}
 
 
 @zope.interface.implementer(interfaces.IAuthenticator)
@@ -39,16 +42,29 @@ class Authenticator(dns_common.DNSAuthenticator):
         add('credentials',
             help=('Path to Google Cloud DNS service account JSON file. (See {0} for' +
                   'information about creating a service account and {1} for information about the' +
-                  'required permissions.)').format(ACCT_URL, PERMISSIONS_URL))
+                  'required permissions.)').format(ACCT_URL, PERMISSIONS_URL),
+            default=None)
 
     def more_info(self): # pylint: disable=missing-docstring,no-self-use
         return 'This plugin configures a DNS TXT record to respond to a dns-01 challenge using ' + \
                'the Google Cloud DNS API.'
 
     def _setup_credentials(self):
-        self._configure_file('credentials', 'path to Google Cloud DNS service account JSON file')
+        if self.conf('credentials') is None:
+            try:
+                # use project_id query to check for availability of google metadata server
+                # we won't use the result but know we're not on GCP when an exception is thrown
+                _GoogleClient.get_project_id()
+            except (ValueError, httplib2.ServerNotFoundError):
+                raise errors.PluginError('Unable to get Google Cloud Metadata and no credentials'
+                                         ' specified. Automatic credential lookup is only '
+                                         'available on Google Cloud Platform. Please configure'
+                                         ' credentials using --dns-google-credentials <file>')
+        else:
+            self._configure_file('credentials',
+                                 'path to Google Cloud DNS service account JSON file')
 
-        dns_common.validate_file_permissions(self.conf('credentials'))
+            dns_common.validate_file_permissions(self.conf('credentials'))
 
     def _perform(self, domain, validation_name, validation):
         self._get_google_client().add_txt_record(domain, validation_name, validation, self.ttl)
@@ -65,13 +81,18 @@ class _GoogleClient(object):
     Encapsulates all communication with the Google Cloud DNS API.
     """
 
-    def __init__(self, account_json):
+    def __init__(self, account_json=None):
 
         scopes = ['https://www.googleapis.com/auth/ndev.clouddns.readwrite']
-        credentials = ServiceAccountCredentials.from_json_keyfile_name(account_json, scopes)
+        if account_json is not None:
+            credentials = ServiceAccountCredentials.from_json_keyfile_name(account_json, scopes)
+            with open(account_json) as account:
+                self.project_id = json.load(account)['project_id']
+        else:
+            credentials = None
+            self.project_id = self.get_project_id()
+
         self.dns = discovery.build('dns', 'v1', credentials=credentials, cache_discovery=False)
-        with open(account_json) as account:
-            self.project_id = json.load(account)['project_id']
 
     def add_txt_record(self, domain, record_name, record_content, record_ttl):
         """
@@ -183,3 +204,24 @@ class _GoogleClient(object):
 
         raise errors.PluginError('Unable to determine managed zone for {0} using zone names: {1}.'
                                  .format(domain, zone_dns_name_guesses))
+
+    @staticmethod
+    def get_project_id():
+        """
+        Query the google metadata service for the current project ID
+
+        This only works on Google Cloud Platform
+
+        :raises ServerNotFoundError: Not running on Google Compute or DNS not available
+        :raises ValueError: Server is found, but response code is not 200
+        :returns: project id
+        """
+        url = '{0}project/project-id'.format(METADATA_URL)
+
+        # Request an access token from the metadata server.
+        http = httplib2.Http()
+        r, content = http.request(url, headers=METADATA_HEADERS)
+        if r.status != 200:
+            raise ValueError("Invalid status code: {0}".format(r))
+
+        return content

certbot-dns-google/certbot_dns_google/dns_google_test.py

@@ -5,8 +5,10 @@ import unittest
 
 import mock
 from googleapiclient.errors import Error
+from httplib2 import ServerNotFoundError
 
 from certbot import errors
+from certbot.errors import PluginError
 from certbot.plugins import dns_test_common
 from certbot.plugins.dns_test_common import DOMAIN
 from certbot.tests import util as test_util
@@ -50,6 +52,11 @@ class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthentic
         expected = [mock.call.del_txt_record(DOMAIN, '_acme-challenge.'+DOMAIN, mock.ANY, mock.ANY)]
         self.assertEqual(expected, self.mock_client.mock_calls)
 
+    @mock.patch('httplib2.Http.request', side_effect=ServerNotFoundError)
+    def test_without_auth(self, unused_mock):
+        self.config.google_credentials = None
+        self.assertRaises(PluginError, self.auth.perform, [self.achall])
+
 
 class GoogleClientTest(unittest.TestCase):
     record_name = "foo"
@@ -74,11 +81,24 @@ class GoogleClientTest(unittest.TestCase):
 
         return client, mock_changes
 
+    @mock.patch('googleapiclient.discovery.build')
+    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('certbot_dns_google.dns_google._GoogleClient.get_project_id')
+    def test_client_without_credentials(self, get_project_id_mock, credential_mock,
+                                        unused_discovery_mock):
+        from certbot_dns_google.dns_google import _GoogleClient
+        _GoogleClient(None)
+        self.assertFalse(credential_mock.called)
+        self.assertTrue(get_project_id_mock.called)
+
     @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
     @mock.patch('certbot_dns_google.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_add_txt_record(self, unused_credential_mock):
+    @mock.patch('certbot_dns_google.dns_google._GoogleClient.get_project_id')
+    def test_add_txt_record(self, get_project_id_mock, credential_mock):
         client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}])
+        credential_mock.assert_called_once_with('/not/a/real/path.json', mock.ANY)
+        self.assertFalse(get_project_id_mock.called)
 
         client.add_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
@@ -197,6 +217,34 @@ class GoogleClientTest(unittest.TestCase):
 
         client.del_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
+    def test_get_project_id(self):
+        from certbot_dns_google.dns_google import _GoogleClient
+
+        response = DummyResponse()
+        response.status = 200
+
+        with mock.patch('httplib2.Http.request', return_value=(response, 1234)):
+            project_id = _GoogleClient.get_project_id()
+            self.assertEqual(project_id, 1234)
+
+        failed_response = DummyResponse()
+        failed_response.status = 404
+
+        with mock.patch('httplib2.Http.request',
+                        return_value=(failed_response, "some detailed http error response")):
+            self.assertRaises(ValueError, _GoogleClient.get_project_id)
+
+        with mock.patch('httplib2.Http.request', side_effect=ServerNotFoundError):
+            self.assertRaises(ServerNotFoundError, _GoogleClient.get_project_id)
+
+
+class DummyResponse(object):
+    """
+    Dummy object to create a fake HTTPResponse (the actual one requires a socket and we only
+     need the status attribute)
+    """
+    def __init__(self):
+        self.status = 200
 
 if __name__ == "__main__":
     unittest.main()  # pragma: no cover

certbot-dns-google/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [
@@ -19,6 +19,8 @@ install_requires = [
     # will tolerate; see #2599:
     'setuptools>=1.0',
     'zope.interface',
+    # already a dependency of google-api-python-client, but added for consistency
+    'httplib2'
 ]
 
 docs_extras = [

certbot-dns-luadns/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [

certbot-dns-nsone/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [

certbot-dns-rfc2136/certbot_dns_rfc2136/dns_rfc2136.py

@@ -208,8 +208,8 @@ class _RFC2136Client(object):
             rcode = response.rcode()
 
             # Authoritative Answer bit should be set
-            if (rcode == dns.rcode.NOERROR and len(response.answer) > 0 and
-                    response.flags & dns.flags.AA):
+            if (rcode == dns.rcode.NOERROR and response.get_rrset(response.answer,
+                domain, dns.rdataclass.IN, dns.rdatatype.SOA) and response.flags & dns.flags.AA):
                 logger.debug('Received authoritative SOA response for %s', domain_name)
                 return True
 

certbot-dns-rfc2136/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [

certbot-dns-route53/setup.py

@@ -3,7 +3,7 @@ import sys
 from distutils.core import setup
 from setuptools import find_packages
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 install_requires = [
     'acme=={0}'.format(version),

certbot-nginx/certbot_nginx/configurator.py

@@ -19,7 +19,6 @@ from certbot import crypto_util
 from certbot import errors
 from certbot import interfaces
 from certbot import util
-from certbot import reverter
 
 from certbot.plugins import common
 
@@ -31,7 +30,7 @@ from certbot_nginx import parser
 logger = logging.getLogger(__name__)
 
 REDIRECT_BLOCK = [[
-    ['\n    ', 'if', ' ', '($scheme', ' ', '!=', ' ', '"https") '],
+    ['\n    ', 'if', ' ', '($scheme', ' ', '!=', ' ', '"https")'],
     [['\n        ', 'return', ' ', '301', ' ', 'https://$host$request_uri'],
      '\n    ']
 ], ['\n']]
@@ -63,7 +62,7 @@ TEST_REDIRECT_COMMENT_BLOCK = [
 
 @zope.interface.implementer(interfaces.IAuthenticator, interfaces.IInstaller)
 @zope.interface.provider(interfaces.IPluginFactory)
-class NginxConfigurator(common.Plugin):
+class NginxConfigurator(common.Installer):
     # pylint: disable=too-many-instance-attributes,too-many-public-methods
     """Nginx configurator.
 
@@ -118,6 +117,9 @@ class NginxConfigurator(common.Plugin):
         # Files to save
         self.save_notes = ""
 
+        # For creating new vhosts if no names match
+        self.new_vhost = None
+
         # Add number of outstanding challenges
         self._chall_out = 0
 
@@ -127,8 +129,6 @@ class NginxConfigurator(common.Plugin):
         self._enhance_func = {"redirect": self._enable_redirect,
                               "staple-ocsp": self._enable_ocsp_stapling}
 
-        # Set up reverter
-        self.reverter = reverter.Reverter(self.config)
         self.reverter.recovery_routine()
 
     @property
@@ -160,6 +160,8 @@ class NginxConfigurator(common.Plugin):
 
         install_ssl_options_conf(self.mod_ssl_conf, self.updated_mod_ssl_conf_digest)
 
+        self.install_ssl_dhparams()
+
         # Set Version
         if self.version is None:
             self.version = self.get_version()
@@ -192,9 +194,11 @@ class NginxConfigurator(common.Plugin):
                 "The nginx plugin currently requires --fullchain-path to "
                 "install a cert.")
 
-        vhost = self.choose_vhost(domain)
-        cert_directives = [['\n', 'ssl_certificate', ' ', fullchain_path],
-                           ['\n', 'ssl_certificate_key', ' ', key_path]]
+        vhost = self.choose_vhost(domain, raise_if_no_match=False)
+        if vhost is None:
+            vhost = self._vhost_from_duplicated_default(domain)
+        cert_directives = [['\n    ', 'ssl_certificate', ' ', fullchain_path],
+                           ['\n    ', 'ssl_certificate_key', ' ', key_path]]
 
         self.parser.add_server_directives(vhost,
                                           cert_directives, replace=True)
@@ -210,7 +214,7 @@ class NginxConfigurator(common.Plugin):
     #######################
     # Vhost parsing methods
     #######################
-    def choose_vhost(self, target_name):
+    def choose_vhost(self, target_name, raise_if_no_match=True):
         """Chooses a virtual host based on the given domain name.
 
         .. note:: This makes the vhost SSL-enabled if it isn't already. Follows
@@ -224,6 +228,8 @@ class NginxConfigurator(common.Plugin):
             hostname. Currently we just ignore this.
 
         :param str target_name: domain name
+        :param bool raise_if_no_match: True iff not finding a match is an error;
+                                       otherwise, return None
 
         :returns: ssl vhost associated with name
         :rtype: :class:`~certbot_nginx.obj.VirtualHost`
@@ -234,9 +240,16 @@ class NginxConfigurator(common.Plugin):
         matches = self._get_ranked_matches(target_name)
         vhost = self._select_best_name_match(matches)
         if not vhost:
-            # No matches. Raise a misconfiguration error.
-            raise errors.MisconfigurationError(
-                        "Cannot find a VirtualHost matching domain %s." % (target_name))
+            if raise_if_no_match:
+                # No matches. Raise a misconfiguration error.
+                raise errors.MisconfigurationError(
+                            ("Cannot find a VirtualHost matching domain %s. "
+                             "In order for Certbot to correctly perform the challenge "
+                             "please add a corresponding server_name directive to your "
+                             "nginx configuration: "
+                             "https://nginx.org/en/docs/http/server_names.html") % (target_name))
+            else:
+                return None
         else:
             # Note: if we are enhancing with ocsp, vhost should already be ssl.
             if not vhost.ssl:
@@ -244,6 +257,65 @@ class NginxConfigurator(common.Plugin):
 
         return vhost
 
+
+    def ipv6_info(self, port):
+        """Returns tuple of booleans (ipv6_active, ipv6only_present)
+        ipv6_active is true if any server block listens ipv6 address in any port
+
+        ipv6only_present is true if ipv6only=on option exists in any server
+        block ipv6 listen directive for the specified port.
+
+        :param str port: Port to check ipv6only=on directive for
+
+        :returns: Tuple containing information if IPv6 is enabled in the global
+            configuration, and existence of ipv6only directive for specified port
+        :rtype: tuple of type (bool, bool)
+        """
+        vhosts = self.parser.get_vhosts()
+        ipv6_active = False
+        ipv6only_present = False
+        for vh in vhosts:
+            for addr in vh.addrs:
+                if addr.ipv6:
+                    ipv6_active = True
+                if addr.ipv6only and addr.get_port() == port:
+                    ipv6only_present = True
+        return (ipv6_active, ipv6only_present)
+
+    def _vhost_from_duplicated_default(self, domain):
+        if self.new_vhost is None:
+            default_vhost = self._get_default_vhost()
+            self.new_vhost = self.parser.create_new_vhost_from_default(default_vhost)
+            if not self.new_vhost.ssl:
+                self._make_server_ssl(self.new_vhost)
+            self.new_vhost.names = set()
+
+        self.new_vhost.names.add(domain)
+        name_block = [['\n    ', 'server_name']]
+        for name in self.new_vhost.names:
+            name_block[0].append(' ')
+            name_block[0].append(name)
+        self.parser.add_server_directives(self.new_vhost, name_block, replace=True)
+        return self.new_vhost
+
+    def _get_default_vhost(self):
+        vhost_list = self.parser.get_vhosts()
+        # if one has default_server set, return that one
+        default_vhosts = []
+        for vhost in vhost_list:
+            for addr in vhost.addrs:
+                if addr.default:
+                    default_vhosts.append(vhost)
+                    break
+
+        if len(default_vhosts) == 1:
+            return default_vhosts[0]
+
+        # TODO: present a list of vhosts for user to choose from
+
+        raise errors.MisconfigurationError("Could not automatically find a matching server"
+            " block. Set the `server_name` directive to use the Nginx installer.")
+
     def _get_ranked_matches(self, target_name):
         """Returns a ranked list of vhosts that match target_name.
         The ranking gives preference to SSL vhosts.
@@ -402,9 +474,12 @@ class NginxConfigurator(common.Plugin):
                     all_names.add(host)
                 elif not common.private_ips_regex.match(host):
                     # If it isn't a private IP, do a reverse DNS lookup
-                    # TODO: IPv6 support
                     try:
-                        socket.inet_aton(host)
+                        if addr.ipv6:
+                            host = addr.get_ipv6_exploded()
+                            socket.inet_pton(socket.AF_INET6, host)
+                        else:
+                            socket.inet_pton(socket.AF_INET, host)
                         all_names.add(socket.gethostbyaddr(host)[0])
                     except (socket.error, socket.herror, socket.timeout):
                         continue
@@ -440,19 +515,43 @@ class NginxConfigurator(common.Plugin):
         :type vhost: :class:`~certbot_nginx.obj.VirtualHost`
 
         """
+        ipv6info = self.ipv6_info(self.config.tls_sni_01_port)
+        ipv6_block = ['']
+        ipv4_block = ['']
+
         # If the vhost was implicitly listening on the default Nginx port,
         # have it continue to do so.
         if len(vhost.addrs) == 0:
             listen_block = [['\n    ', 'listen', ' ', self.DEFAULT_LISTEN_PORT]]
             self.parser.add_server_directives(vhost, listen_block, replace=False)
 
+        if vhost.ipv6_enabled():
+            ipv6_block = ['\n    ',
+                          'listen',
+                          ' ',
+                          '[::]:{0} ssl'.format(self.config.tls_sni_01_port)]
+            if not ipv6info[1]:
+                # ipv6only=on is absent in global config
+                ipv6_block.append(' ')
+                ipv6_block.append('ipv6only=on')
+
+        if vhost.ipv4_enabled():
+            ipv4_block = ['\n    ',
+                          'listen',
+                          ' ',
+                          '{0} ssl'.format(self.config.tls_sni_01_port)]
+
+
         snakeoil_cert, snakeoil_key = self._get_snakeoil_paths()
 
-        ssl_block = (
-            [['\n    ', 'listen', ' ', '{0} ssl'.format(self.config.tls_sni_01_port)],
-             ['\n    ', 'ssl_certificate', ' ', snakeoil_cert],
-             ['\n    ', 'ssl_certificate_key', ' ', snakeoil_key],
-             ['\n    ', 'include', ' ', self.mod_ssl_conf]])
+        ssl_block = ([
+            ipv6_block,
+            ipv4_block,
+            ['\n    ', 'ssl_certificate', ' ', snakeoil_cert],
+            ['\n    ', 'ssl_certificate_key', ' ', snakeoil_key],
+            ['\n    ', 'include', ' ', self.mod_ssl_conf],
+            ['\n    ', 'ssl_dhparam', ' ', self.ssl_dhparams],
+        ])
 
         self.parser.add_server_directives(
             vhost, ssl_block, replace=False)
@@ -693,31 +792,13 @@ class NginxConfigurator(common.Plugin):
 
         """
         save_files = set(self.parser.parsed.keys())
-
-        try:  # TODO: make a common base for Apache and Nginx plugins
-            # Create Checkpoint
-            if temporary:
-                self.reverter.add_to_temp_checkpoint(
-                    save_files, self.save_notes)
-                # how many comments does it take
-            else:
-                self.reverter.add_to_checkpoint(save_files,
-                                            self.save_notes)
-                # to confuse a linter?
-        except errors.ReverterError as err:
-            raise errors.PluginError(str(err))
-
+        self.add_to_checkpoint(save_files, self.save_notes, temporary)
         self.save_notes = ""
 
         # Change 'ext' to something else to not override existing conf files
         self.parser.filedump(ext='')
         if title and not temporary:
-            try:
-                self.reverter.finalize_checkpoint(title)
-            except errors.ReverterError as err:
-                raise errors.PluginError(str(err))
-
-        return True
+            self.finalize_checkpoint(title)
 
     def recovery_routine(self):
         """Revert all previously modified files.
@@ -727,10 +808,7 @@ class NginxConfigurator(common.Plugin):
         :raises .errors.PluginError: If unable to recover the configuration
 
         """
-        try:
-            self.reverter.recovery_routine()
-        except errors.ReverterError as err:
-            raise errors.PluginError(str(err))
+        super(NginxConfigurator, self).recovery_routine()
         self.parser.load()
 
     def revert_challenge_config(self):
@@ -739,10 +817,7 @@ class NginxConfigurator(common.Plugin):
         :raises .errors.PluginError: If unable to revert the challenge config.
 
         """
-        try:
-            self.reverter.revert_temporary_config()
-        except errors.ReverterError as err:
-            raise errors.PluginError(str(err))
+        self.revert_temporary_config()
         self.parser.load()
 
     def rollback_checkpoints(self, rollback=1):
@@ -754,24 +829,9 @@ class NginxConfigurator(common.Plugin):
             the function is unable to correctly revert the configuration
 
         """
-        try:
-            self.reverter.rollback_checkpoints(rollback)
-        except errors.ReverterError as err:
-            raise errors.PluginError(str(err))
+        super(NginxConfigurator, self).rollback_checkpoints(rollback)
         self.parser.load()
 
-    def view_config_changes(self):
-        """Show all of the configuration changes that have taken place.
-
-        :raises .errors.PluginError: If there is a problem while processing
-            the checkpoints directories.
-
-        """
-        try:
-            self.reverter.view_config_changes()
-        except errors.ReverterError as err:
-            raise errors.PluginError(str(err))
-
     ###########################################################################
     # Challenges Section for IAuthenticator
     ###########################################################################
@@ -860,5 +920,5 @@ def nginx_restart(nginx_ctl, nginx_conf):
 
 def install_ssl_options_conf(options_ssl, options_ssl_digest):
     """Copy Certbot's SSL options file into the system's config dir if required."""
-    return common.install_ssl_options_conf(options_ssl, options_ssl_digest,
+    return common.install_version_controlled_file(options_ssl, options_ssl_digest,
         constants.MOD_SSL_CONF_SRC, constants.ALL_SSL_OPTIONS_HASHES)

certbot-nginx/certbot_nginx/nginxparser.py

@@ -7,6 +7,7 @@ from pyparsing import (
     Literal, White, Forward, Group, Optional, OneOrMore, QuotedString, Regex, ZeroOrMore, Combine)
 from pyparsing import stringEnd
 from pyparsing import restOfLine
+import six
 
 logger = logging.getLogger(__name__)
 
@@ -71,7 +72,7 @@ class RawNginxDumper(object):
         """Iterates the dumped nginx content."""
         blocks = blocks or self.blocks
         for b0 in blocks:
-            if isinstance(b0, str):
+            if isinstance(b0, six.string_types):
                 yield b0
                 continue
             item = copy.deepcopy(b0)
@@ -88,7 +89,7 @@ class RawNginxDumper(object):
                 yield '}'
             else: # not a block - list of strings
                 semicolon = ";"
-                if isinstance(item[0], str) and item[0].strip() == '#': # comment
+                if isinstance(item[0], six.string_types) and item[0].strip() == '#': # comment
                     semicolon = ""
                 yield "".join(item) + semicolon
 
@@ -145,7 +146,7 @@ def dump(blocks, _file):
     return _file.write(dumps(blocks))
 
 
-spacey = lambda x: (isinstance(x, str) and x.isspace()) or x == ''
+spacey = lambda x: (isinstance(x, six.string_types) and x.isspace()) or x == ''
 
 class UnspacedList(list):
     """Wrap a list [of lists], making any whitespace entries magically invisible"""
@@ -189,13 +190,15 @@ class UnspacedList(list):
         item, spaced_item = self._coerce(x)
         slicepos = self._spaced_position(i) if i < len(self) else len(self.spaced)
         self.spaced.insert(slicepos, spaced_item)
-        list.insert(self, i, item)
+        if not spacey(item):
+            list.insert(self, i, item)
         self.dirty = True
 
     def append(self, x):
         item, spaced_item = self._coerce(x)
         self.spaced.append(spaced_item)
-        list.append(self, item)
+        if not spacey(item):
+            list.append(self, item)
         self.dirty = True
 
     def extend(self, x):
@@ -226,7 +229,8 @@ class UnspacedList(list):
             raise NotImplementedError("Slice operations on UnspacedLists not yet implemented")
         item, spaced_item = self._coerce(value)
         self.spaced.__setitem__(self._spaced_position(i), spaced_item)
-        list.__setitem__(self, i, item)
+        if not spacey(item):
+            list.__setitem__(self, i, item)
         self.dirty = True
 
     def __delitem__(self, i):
@@ -235,8 +239,8 @@ class UnspacedList(list):
         self.dirty = True
 
     def __deepcopy__(self, memo):
-        l = UnspacedList(self[:])
-        l.spaced = copy.deepcopy(self.spaced, memo=memo)
+        new_spaced = copy.deepcopy(self.spaced, memo=memo)
+        l = UnspacedList(new_spaced)
         l.dirty = self.dirty
         return l
 

certbot-nginx/certbot_nginx/obj.py

@@ -34,10 +34,13 @@ class Addr(common.Addr):
     UNSPECIFIED_IPV4_ADDRESSES = ('', '*', '0.0.0.0')
     CANONICAL_UNSPECIFIED_ADDRESS = UNSPECIFIED_IPV4_ADDRESSES[0]
 
-    def __init__(self, host, port, ssl, default):
+    def __init__(self, host, port, ssl, default, ipv6, ipv6only):
+        # pylint: disable=too-many-arguments
         super(Addr, self).__init__((host, port))
         self.ssl = ssl
         self.default = default
+        self.ipv6 = ipv6
+        self.ipv6only = ipv6only
         self.unspecified_address = host in self.UNSPECIFIED_IPV4_ADDRESSES
 
     @classmethod
@@ -46,6 +49,8 @@ class Addr(common.Addr):
         parts = str_addr.split(' ')
         ssl = False
         default = False
+        ipv6 = False
+        ipv6only = False
         host = ''
         port = ''
 
@@ -56,15 +61,25 @@ class Addr(common.Addr):
         if addr.startswith('unix:'):
             return None
 
-        tup = addr.partition(':')
-        if re.match(r'^\d+$', tup[0]):
-            # This is a bare port, not a hostname. E.g. listen 80
-            host = ''
-            port = tup[0]
+        # IPv6 check
+        ipv6_match = re.match(r'\[.*\]', addr)
+        if ipv6_match:
+            ipv6 = True
+            # IPv6 handling
+            host = ipv6_match.group()
+            # The rest of the addr string will be the port, if any
+            port = addr[ipv6_match.end()+1:]
         else:
-            # This is a host-port tuple. E.g. listen 127.0.0.1:*
-            host = tup[0]
-            port = tup[2]
+            # IPv4 handling
+            tup = addr.partition(':')
+            if re.match(r'^\d+$', tup[0]):
+                # This is a bare port, not a hostname. E.g. listen 80
+                host = ''
+                port = tup[0]
+            else:
+                # This is a host-port tuple. E.g. listen 127.0.0.1:*
+                host = tup[0]
+                port = tup[2]
 
         # The rest of the parts are options; we only care about ssl and default
         while len(parts) > 0:
@@ -73,8 +88,10 @@ class Addr(common.Addr):
                 ssl = True
             elif nextpart == 'default_server':
                 default = True
+            elif nextpart == "ipv6only=on":
+                ipv6only = True
 
-        return cls(host, port, ssl, default)
+        return cls(host, port, ssl, default, ipv6, ipv6only)
 
     def to_string(self, include_default=True):
         """Return string representation of Addr"""
@@ -114,8 +131,6 @@ class Addr(common.Addr):
                                 self.tup[1]), self.ipv6) == \
                    common.Addr((other.CANONICAL_UNSPECIFIED_ADDRESS,
                                 other.tup[1]), other.ipv6)
-        # Nginx plugin currently doesn't support IPv6 but this will
-        # future-proof it
         return super(Addr, self).__eq__(other)
 
     def __eq__(self, other):
@@ -195,10 +210,24 @@ class VirtualHost(object):  # pylint: disable=too-few-public-methods
                 return True
         return False
 
+    def ipv6_enabled(self):
+        """Return true if one or more of the listen directives in vhost supports
+        IPv6"""
+        for a in self.addrs:
+            if a.ipv6:
+                return True
+
+    def ipv4_enabled(self):
+        """Return true if one or more of the listen directives in vhost are IPv4
+        only"""
+        for a in self.addrs:
+            if not a.ipv6:
+                return True
+
 def _find_directive(directives, directive_name):
     """Find a directive of type directive_name in directives
     """
-    if not directives or isinstance(directives, str) or len(directives) == 0:
+    if not directives or isinstance(directives, six.string_types) or len(directives) == 0:
         return None
 
     if directives[0] == directive_name:

certbot-nginx/certbot_nginx/parser.py

@@ -6,6 +6,8 @@ import os
 import pyparsing
 import re
 
+import six
+
 from certbot import errors
 
 from certbot_nginx import obj
@@ -312,6 +314,32 @@ class NginxParser(object):
         except errors.MisconfigurationError as err:
             raise errors.MisconfigurationError("Problem in %s: %s" % (filename, str(err)))
 
+    def create_new_vhost_from_default(self, vhost_template):
+        """Duplicate the default vhost in the configuration files.
+
+        :param :class:`~certbot_nginx.obj.VirtualHost` vhost_template: The vhost
+            whose information we copy
+
+        :returns: A vhost object for the newly created vhost
+        :rtype: :class:`~certbot_nginx.obj.VirtualHost`
+        """
+        # TODO: https://github.com/certbot/certbot/issues/5185
+        # put it in the same file as the template, at the same level
+        enclosing_block = self.parsed[vhost_template.filep]
+        for index in vhost_template.path[:-1]:
+            enclosing_block = enclosing_block[index]
+        new_location = vhost_template.path[-1] + 1
+        raw_in_parsed = copy.deepcopy(enclosing_block[vhost_template.path[-1]])
+        enclosing_block.insert(new_location, raw_in_parsed)
+        new_vhost = copy.deepcopy(vhost_template)
+        new_vhost.path[-1] = new_location
+        for addr in new_vhost.addrs:
+            addr.default = False
+        for directive in enclosing_block[new_vhost.path[-1]][1]:
+            if len(directive) > 0 and directive[0] == 'listen' and 'default_server' in directive:
+                del directive[directive.index('default_server')]
+        return new_vhost
+
 def _parse_ssl_options(ssl_options):
     if ssl_options is not None:
         try:
@@ -444,7 +472,7 @@ def _is_include_directive(entry):
     """
     return (isinstance(entry, list) and
             len(entry) == 2 and entry[0] == 'include' and
-            isinstance(entry[1], str))
+            isinstance(entry[1], six.string_types))
 
 def _is_ssl_on_directive(entry):
     """Checks if an nginx parsed entry is an 'ssl on' directive.
@@ -561,7 +589,8 @@ def _add_directive(block, directive, replace):
     directive_name = directive[0]
     def can_append(loc, dir_name):
         """ Can we append this directive to the block? """
-        return loc is None or (isinstance(dir_name, str) and dir_name in REPEATABLE_DIRECTIVES)
+        return loc is None or (isinstance(dir_name, six.string_types)
+            and dir_name in REPEATABLE_DIRECTIVES)
 
     err_fmt = 'tried to insert directive "{0}" but found conflicting "{1}".'
 

certbot-nginx/certbot_nginx/tests/configurator_test.py

@@ -46,7 +46,7 @@ class NginxConfiguratorTest(util.NginxTest):
 
     def test_prepare(self):
         self.assertEqual((1, 6, 2), self.config.version)
-        self.assertEqual(8, len(self.config.parser.parsed))
+        self.assertEqual(10, len(self.config.parser.parsed))
 
     @mock.patch("certbot_nginx.configurator.util.exe_exists")
     @mock.patch("certbot_nginx.configurator.subprocess.Popen")
@@ -90,7 +90,7 @@ class NginxConfiguratorTest(util.NginxTest):
         self.assertEqual(names, set(
             ["155.225.50.69.nephoscale.net", "www.example.org", "another.alias",
              "migration.com", "summer.com", "geese.com", "sslon.com",
-             "globalssl.com", "globalsslsetssl.com"]))
+             "globalssl.com", "globalsslsetssl.com", "ipv6.com", "ipv6ssl.com"]))
 
     def test_supported_enhancements(self):
         self.assertEqual(['redirect', 'staple-ocsp'],
@@ -132,6 +132,7 @@ class NginxConfiguratorTest(util.NginxTest):
         server_conf = set(['somename', 'another.alias', 'alias'])
         example_conf = set(['.example.com', 'example.*'])
         foo_conf = set(['*.www.foo.com', '*.www.example.com'])
+        ipv6_conf = set(['ipv6.com'])
 
         results = {'localhost': localhost_conf,
                    'alias': server_conf,
@@ -140,7 +141,8 @@ class NginxConfiguratorTest(util.NginxTest):
                    'www.example.com': example_conf,
                    'test.www.example.com': foo_conf,
                    'abc.www.foo.com': foo_conf,
-                   'www.bar.co.uk': localhost_conf}
+                   'www.bar.co.uk': localhost_conf,
+                   'ipv6.com': ipv6_conf}
 
         conf_path = {'localhost': "etc_nginx/nginx.conf",
                    'alias': "etc_nginx/nginx.conf",
@@ -149,7 +151,8 @@ class NginxConfiguratorTest(util.NginxTest):
                    'www.example.com': "etc_nginx/sites-enabled/example.com",
                    'test.www.example.com': "etc_nginx/foo.conf",
                    'abc.www.foo.com': "etc_nginx/foo.conf",
-                   'www.bar.co.uk': "etc_nginx/nginx.conf"}
+                   'www.bar.co.uk': "etc_nginx/nginx.conf",
+                   'ipv6.com': "etc_nginx/sites-enabled/ipv6.com"}
 
         bad_results = ['www.foo.com', 'example', 't.www.bar.co',
                        '69.255.225.155']
@@ -160,11 +163,24 @@ class NginxConfiguratorTest(util.NginxTest):
 
             self.assertEqual(results[name], vhost.names)
             self.assertEqual(conf_path[name], path)
+            # IPv6 specific checks
+            if name == "ipv6.com":
+                self.assertTrue(vhost.ipv6_enabled())
+                # Make sure that we have SSL enabled also for IPv6 addr
+                self.assertTrue(
+                    any([True for x in vhost.addrs if x.ssl and x.ipv6]))
 
         for name in bad_results:
             self.assertRaises(errors.MisconfigurationError,
                               self.config.choose_vhost, name)
 
+    def test_ipv6only(self):
+        # ipv6_info: (ipv6_active, ipv6only_present)
+        self.assertEquals((True, False), self.config.ipv6_info("80"))
+        # Port 443 has ipv6only=on because of ipv6ssl.com vhost
+        self.assertEquals((True, True), self.config.ipv6_info("443"))
+
+
     def test_more_info(self):
         self.assertTrue('nginx.conf' in self.config.more_info())
 
@@ -226,8 +242,9 @@ class NginxConfiguratorTest(util.NginxTest):
                             ['listen', '5001', 'ssl'],
                             ['ssl_certificate', 'example/fullchain.pem'],
                             ['ssl_certificate_key', 'example/key.pem'],
-                            ['include', self.config.mod_ssl_conf]]
-                            ]],
+                            ['include', self.config.mod_ssl_conf],
+                            ['ssl_dhparam', self.config.ssl_dhparams],
+                            ]]],
                          parsed_example_conf)
         self.assertEqual([['server_name', 'somename', 'alias', 'another.alias']],
                          parsed_server_conf)
@@ -244,8 +261,9 @@ class NginxConfiguratorTest(util.NginxTest):
               ['listen', '5001', 'ssl'],
               ['ssl_certificate', '/etc/nginx/fullchain.pem'],
               ['ssl_certificate_key', '/etc/nginx/key.pem'],
-              ['include', self.config.mod_ssl_conf]]
-            ],
+              ['include', self.config.mod_ssl_conf],
+              ['ssl_dhparam', self.config.ssl_dhparams],
+            ]],
             2))
 
     def test_deploy_cert_add_explicit_listen(self):
@@ -268,8 +286,9 @@ class NginxConfiguratorTest(util.NginxTest):
                            ['listen', '5001', 'ssl'],
                            ['ssl_certificate', 'summer/fullchain.pem'],
                            ['ssl_certificate_key', 'summer/key.pem'],
-                           ['include', self.config.mod_ssl_conf]]
-                           ],
+                           ['include', self.config.mod_ssl_conf],
+                           ['ssl_dhparam', self.config.ssl_dhparams],
+                           ]],
                          parsed_migration_conf[0])
 
     @mock.patch("certbot_nginx.configurator.tls_sni_01.NginxTlsSni01.perform")
@@ -426,7 +445,7 @@ class NginxConfiguratorTest(util.NginxTest):
         # Test that we successfully add a redirect when there is
         # a listen directive
         expected = [
-            ['if', '($scheme', '!=', '"https") '],
+            ['if', '($scheme', '!=', '"https")'],
             [['return', '301', 'https://$host$request_uri']]
         ]
 
@@ -509,6 +528,23 @@ class NginxConfiguratorTest(util.NginxTest):
         self.assertEqual(mock_logger.info.call_args[0][0],
                 'No matching insecure server blocks listening on port %s found.')
 
+    def test_no_double_redirect(self):
+        # Test that we don't also add the commented redirect if we've just added
+        # a redirect to that vhost this run
+        example_conf = self.config.parser.abs_path('sites-enabled/example.com')
+        self.config.enhance("example.com", "redirect")
+        self.config.enhance("example.org", "redirect")
+
+        unexpected = [
+            ['#', ' Redirect non-https traffic to https'],
+            ['#', ' if ($scheme != "https") {'],
+            ['#', '     return 301 https://$host$request_uri;'],
+            ['#', ' } # managed by Certbot']
+        ]
+        generated_conf = self.config.parser.parsed[example_conf]
+        for line in unexpected:
+            self.assertFalse(util.contains_at_depth(generated_conf, line, 2))
+
     def test_staple_ocsp_bad_version(self):
         self.config.version = (1, 3, 1)
         self.assertRaises(errors.PluginError, self.config.enhance,
@@ -539,6 +575,145 @@ class NginxConfiguratorTest(util.NginxTest):
         self.assertTrue(util.contains_at_depth(
             generated_conf, ['ssl_stapling_verify', 'on'], 2))
 
+    def test_deploy_no_match_default_set(self):
+        default_conf = self.config.parser.abs_path('sites-enabled/default')
+        foo_conf = self.config.parser.abs_path('foo.conf')
+        del self.config.parser.parsed[foo_conf][2][1][0][1][0] # remove default_server
+        self.config.version = (1, 3, 1)
+
+        self.config.deploy_cert(
+            "www.nomatch.com",
+            "example/cert.pem",
+            "example/key.pem",
+            "example/chain.pem",
+            "example/fullchain.pem")
+        self.config.save()
+
+        self.config.parser.load()
+
+        parsed_default_conf = util.filter_comments(self.config.parser.parsed[default_conf])
+
+        self.assertEqual([[['server'],
+                           [['listen', 'myhost', 'default_server'],
+                            ['listen', 'otherhost', 'default_server'],
+                            ['server_name', 'www.example.org'],
+                            [['location', '/'],
+                             [['root', 'html'],
+                              ['index', 'index.html', 'index.htm']]]]],
+                          [['server'],
+                           [['listen', 'myhost'],
+                            ['listen', 'otherhost'],
+                            ['server_name', 'www.nomatch.com'],
+                            [['location', '/'],
+                             [['root', 'html'],
+                              ['index', 'index.html', 'index.htm']]],
+                            ['listen', '5001', 'ssl'],
+                            ['ssl_certificate', 'example/fullchain.pem'],
+                            ['ssl_certificate_key', 'example/key.pem'],
+                            ['include', self.config.mod_ssl_conf],
+                            ['ssl_dhparam', self.config.ssl_dhparams]]]],
+                         parsed_default_conf)
+
+        self.config.deploy_cert(
+            "nomatch.com",
+            "example/cert.pem",
+            "example/key.pem",
+            "example/chain.pem",
+            "example/fullchain.pem")
+        self.config.save()
+
+        self.config.parser.load()
+
+        parsed_default_conf = util.filter_comments(self.config.parser.parsed[default_conf])
+
+        self.assertTrue(util.contains_at_depth(parsed_default_conf, "nomatch.com", 3))
+
+    def test_deploy_no_match_default_set_multi_level_path(self):
+        default_conf = self.config.parser.abs_path('sites-enabled/default')
+        foo_conf = self.config.parser.abs_path('foo.conf')
+        del self.config.parser.parsed[default_conf][0][1][0]
+        del self.config.parser.parsed[default_conf][0][1][0]
+        self.config.version = (1, 3, 1)
+
+        self.config.deploy_cert(
+            "www.nomatch.com",
+            "example/cert.pem",
+            "example/key.pem",
+            "example/chain.pem",
+            "example/fullchain.pem")
+        self.config.save()
+
+        self.config.parser.load()
+
+        parsed_foo_conf = util.filter_comments(self.config.parser.parsed[foo_conf])
+
+        self.assertEqual([['server'],
+                          [['listen', '*:80', 'ssl'],
+                          ['server_name', 'www.nomatch.com'],
+                          ['root', '/home/ubuntu/sites/foo/'],
+                          [['location', '/status'], [[['types'], [['image/jpeg', 'jpg']]]]],
+                          [['location', '~', 'case_sensitive\\.php$'], [['index', 'index.php'],
+                           ['root', '/var/root']]],
+                          [['location', '~*', 'case_insensitive\\.php$'], []],
+                          [['location', '=', 'exact_match\\.php$'], []],
+                          [['location', '^~', 'ignore_regex\\.php$'], []],
+                          ['ssl_certificate', 'example/fullchain.pem'],
+                          ['ssl_certificate_key', 'example/key.pem']]],
+                         parsed_foo_conf[1][1][1])
+
+    def test_deploy_no_match_no_default_set(self):
+        default_conf = self.config.parser.abs_path('sites-enabled/default')
+        foo_conf = self.config.parser.abs_path('foo.conf')
+        del self.config.parser.parsed[default_conf][0][1][0]
+        del self.config.parser.parsed[default_conf][0][1][0]
+        del self.config.parser.parsed[foo_conf][2][1][0][1][0]
+        self.config.version = (1, 3, 1)
+
+        self.assertRaises(errors.MisconfigurationError, self.config.deploy_cert,
+            "www.nomatch.com", "example/cert.pem", "example/key.pem",
+            "example/chain.pem", "example/fullchain.pem")
+
+    def test_deploy_no_match_fail_multiple_defaults(self):
+        self.config.version = (1, 3, 1)
+        self.assertRaises(errors.MisconfigurationError, self.config.deploy_cert,
+            "www.nomatch.com", "example/cert.pem", "example/key.pem",
+            "example/chain.pem", "example/fullchain.pem")
+
+    def test_deploy_no_match_add_redirect(self):
+        default_conf = self.config.parser.abs_path('sites-enabled/default')
+        foo_conf = self.config.parser.abs_path('foo.conf')
+        del self.config.parser.parsed[foo_conf][2][1][0][1][0] # remove default_server
+        self.config.version = (1, 3, 1)
+
+        self.config.deploy_cert(
+            "www.nomatch.com",
+            "example/cert.pem",
+            "example/key.pem",
+            "example/chain.pem",
+            "example/fullchain.pem")
+
+        self.config.deploy_cert(
+            "nomatch.com",
+            "example/cert.pem",
+            "example/key.pem",
+            "example/chain.pem",
+            "example/fullchain.pem")
+
+        self.config.enhance("www.nomatch.com", "redirect")
+
+        self.config.save()
+
+        self.config.parser.load()
+
+        expected = [
+            ['if', '($scheme', '!=', '"https")'],
+            [['return', '301', 'https://$host$request_uri']]
+        ]
+
+        generated_conf = self.config.parser.parsed[default_conf]
+        self.assertTrue(util.contains_at_depth(generated_conf, expected, 2))
+
+
 class InstallSslOptionsConfTest(util.NginxTest):
     """Test that the options-ssl-nginx.conf file is installed and updated properly."""
 
@@ -601,7 +776,7 @@ class InstallSslOptionsConfTest(util.NginxTest):
         with mock.patch("certbot.plugins.common.logger") as mock_logger:
             self._call()
             self.assertEqual(mock_logger.warning.call_args[0][0],
-                "%s has been manually modified; updated ssl configuration options "
+                "%s has been manually modified; updated file "
                 "saved to %s. We recommend updating %s for security purposes.")
         self.assertEqual(crypto_util.sha256sum(constants.MOD_SSL_CONF_SRC),
             self._current_ssl_options_hash())

certbot-nginx/certbot_nginx/tests/parser_test.py

@@ -50,7 +50,9 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods
                                'sites-enabled/example.com',
                                'sites-enabled/migration.com',
                                'sites-enabled/sslon.com',
-                               'sites-enabled/globalssl.com']]),
+                               'sites-enabled/globalssl.com',
+                               'sites-enabled/ipv6.com',
+                               'sites-enabled/ipv6ssl.com']]),
                          set(nparser.parsed.keys()))
         self.assertEqual([['server_name', 'somename', 'alias', 'another.alias']],
                          nparser.parsed[nparser.abs_path('server.conf')])
@@ -74,7 +76,7 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods
         parsed = nparser._parse_files(nparser.abs_path(
             'sites-enabled/example.com.test'))
         self.assertEqual(3, len(glob.glob(nparser.abs_path('*.test'))))
-        self.assertEqual(5, len(
+        self.assertEqual(7, len(
             glob.glob(nparser.abs_path('sites-enabled/*.test'))))
         self.assertEqual([[['server'], [['listen', '69.50.225.155:9000'],
                                         ['listen', '127.0.0.1'],
@@ -110,7 +112,8 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods
         vhosts = nparser.get_vhosts()
 
         vhost = obj.VirtualHost(nparser.abs_path('sites-enabled/globalssl.com'),
-                                [obj.Addr('4.8.2.6', '57', True, False)],
+                                [obj.Addr('4.8.2.6', '57', True, False,
+                                          False, False)],
                                 True, True, set(['globalssl.com']), [], [0])
 
         globalssl_com = [x for x in vhosts if 'globalssl.com' in x.filep][0]
@@ -121,34 +124,42 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods
         vhosts = nparser.get_vhosts()
 
         vhost1 = obj.VirtualHost(nparser.abs_path('nginx.conf'),
-                                 [obj.Addr('', '8080', False, False)],
+                                 [obj.Addr('', '8080', False, False,
+                                           False, False)],
                                  False, True,
                                  set(['localhost',
                                       r'~^(www\.)?(example|bar)\.']),
                                  [], [10, 1, 9])
         vhost2 = obj.VirtualHost(nparser.abs_path('nginx.conf'),
-                                 [obj.Addr('somename', '8080', False, False),
-                                  obj.Addr('', '8000', False, False)],
+                                 [obj.Addr('somename', '8080', False, False,
+                                           False, False),
+                                  obj.Addr('', '8000', False, False,
+                                           False, False)],
                                  False, True,
                                  set(['somename', 'another.alias', 'alias']),
                                  [], [10, 1, 12])
         vhost3 = obj.VirtualHost(nparser.abs_path('sites-enabled/example.com'),
                                  [obj.Addr('69.50.225.155', '9000',
-                                           False, False),
-                                  obj.Addr('127.0.0.1', '', False, False)],
+                                           False, False, False, False),
+                                  obj.Addr('127.0.0.1', '', False, False,
+                                           False, False)],
                                  False, True,
                                  set(['.example.com', 'example.*']), [], [0])
         vhost4 = obj.VirtualHost(nparser.abs_path('sites-enabled/default'),
-                                 [obj.Addr('myhost', '', False, True)],
+                                 [obj.Addr('myhost', '', False, True,
+                                           False, False),
+                                  obj.Addr('otherhost', '', False, True,
+                                           False, False)],
                                  False, True, set(['www.example.org']),
                                  [], [0])
         vhost5 = obj.VirtualHost(nparser.abs_path('foo.conf'),
-                                 [obj.Addr('*', '80', True, True)],
+                                 [obj.Addr('*', '80', True, True,
+                                           False, False)],
                                  True, True, set(['*.www.foo.com',
                                                   '*.www.example.com']),
                                  [], [2, 1, 0])
 
-        self.assertEqual(10, len(vhosts))
+        self.assertEqual(12, len(vhosts))
         example_com = [x for x in vhosts if 'example.com' in x.filep][0]
         self.assertEqual(vhost3, example_com)
         default = [x for x in vhosts if 'default' in x.filep][0]
@@ -395,6 +406,29 @@ class NginxParserTest(util.NginxTest): #pylint: disable=too-many-public-methods
         ])
         self.assertTrue(server['ssl'])
 
+    def test_create_new_vhost_from_default(self):
+        nparser = parser.NginxParser(self.config_path)
+
+        vhosts = nparser.get_vhosts()
+        default = [x for x in vhosts if 'default' in x.filep][0]
+        new_vhost = nparser.create_new_vhost_from_default(default)
+        nparser.filedump(ext='')
+
+        # check properties of new vhost
+        self.assertFalse(next(iter(new_vhost.addrs)).default)
+        self.assertNotEqual(new_vhost.path, default.path)
+
+        # check that things are written to file correctly
+        new_nparser = parser.NginxParser(self.config_path)
+        new_vhosts = new_nparser.get_vhosts()
+        new_defaults = [x for x in new_vhosts if 'default' in x.filep]
+        self.assertEqual(len(new_defaults), 2)
+        new_vhost_parsed = new_defaults[1]
+        self.assertFalse(next(iter(new_vhost_parsed.addrs)).default)
+        self.assertEqual(next(iter(default.names)), next(iter(new_vhost_parsed.names)))
+        self.assertEqual(len(default.raw), len(new_vhost_parsed.raw))
+        self.assertTrue(next(iter(default.addrs)).super_eq(next(iter(new_vhost_parsed.addrs))))
+
 
 if __name__ == "__main__":
     unittest.main()  # pragma: no cover

certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/default

@@ -1,5 +1,6 @@
 server {
     listen       myhost default_server;
+    listen       otherhost default_server;
     server_name www.example.org;
 
     location / {

certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/ipv6.com

@@ -0,0 +1,5 @@
+server {
+    listen 80;
+    listen [::]:80;
+    server_name ipv6.com;
+}

certbot-nginx/certbot_nginx/tests/testdata/etc_nginx/sites-enabled/ipv6ssl.com

@@ -0,0 +1,5 @@
+server {
+    listen 443 ssl;
+    listen [::]:443 ssl ipv6only=on;
+    server_name ipv6ssl.com;
+}

certbot-nginx/certbot_nginx/tests/tls_sni_01_test.py

@@ -66,7 +66,7 @@ class TlsSniPerformTest(util.NginxTest):
         self.sni.add_chall(self.achalls[1])
         mock_choose.return_value = None
         result = self.sni.perform()
-        self.assertTrue(result is None)
+        self.assertFalse(result is None)
 
     def test_perform0(self):
         responses = self.sni.perform()
@@ -125,10 +125,10 @@ class TlsSniPerformTest(util.NginxTest):
         self.sni.add_chall(self.achalls[0])
         self.sni.add_chall(self.achalls[2])
 
-        v_addr1 = [obj.Addr("69.50.225.155", "9000", True, False),
-                   obj.Addr("127.0.0.1", "", False, False)]
-        v_addr2 = [obj.Addr("myhost", "", False, True)]
-        v_addr2_print = [obj.Addr("myhost", "", False, False)]
+        v_addr1 = [obj.Addr("69.50.225.155", "9000", True, False, False, False),
+                   obj.Addr("127.0.0.1", "", False, False, False, False)]
+        v_addr2 = [obj.Addr("myhost", "", False, True, False, False)]
+        v_addr2_print = [obj.Addr("myhost", "", False, False, False, False)]
         ll_addr = [v_addr1, v_addr2]
         self.sni._mod_config(ll_addr)  # pylint: disable=protected-access
 

certbot-nginx/certbot_nginx/tls_sni_01.py

@@ -51,19 +51,32 @@ class NginxTlsSni01(common.TLSSNI01):
         default_addr = "{0} ssl".format(
             self.configurator.config.tls_sni_01_port)
 
-        for achall in self.achalls:
-            vhost = self.configurator.choose_vhost(achall.domain)
-            if vhost is None:
-                logger.error(
-                    "No nginx vhost exists with server_name matching: %s. "
-                    "Please specify server_names in the Nginx config.",
-                    achall.domain)
-                return None
+        ipv6, ipv6only = self.configurator.ipv6_info(
+            self.configurator.config.tls_sni_01_port)
 
-            if vhost.addrs:
+        for achall in self.achalls:
+            vhost = self.configurator.choose_vhost(achall.domain, raise_if_no_match=False)
+
+            if vhost is not None and vhost.addrs:
                 addresses.append(list(vhost.addrs))
             else:
-                addresses.append([obj.Addr.fromstring(default_addr)])
+                if ipv6:
+                    # If IPv6 is active in Nginx configuration
+                    ipv6_addr = "[::]:{0} ssl".format(
+                        self.configurator.config.tls_sni_01_port)
+                    if not ipv6only:
+                        # If ipv6only=on is not already present in the config
+                        ipv6_addr = ipv6_addr + " ipv6only=on"
+                    addresses.append([obj.Addr.fromstring(default_addr),
+                                      obj.Addr.fromstring(ipv6_addr)])
+                    logger.info(("Using default addresses %s and %s for " +
+                                 "TLSSNI01 authentication."),
+                                default_addr,
+                                ipv6_addr)
+                else:
+                    addresses.append([obj.Addr.fromstring(default_addr)])
+                    logger.info("Using default address %s for TLSSNI01 authentication.",
+                                default_addr)
 
         # Create challenge certs
         responses = [self._setup_challenge_cert(x) for x in self.achalls]
@@ -115,9 +128,8 @@ class NginxTlsSni01(common.TLSSNI01):
                 break
         if not included:
             raise errors.MisconfigurationError(
-                'LetsEncrypt could not find an HTTP block to include '
+                'Certbot could not find an HTTP block to include '
                 'TLS-SNI-01 challenges in %s.' % root)
-
         config = [self._make_server_block(pair[0], pair[1])
                   for pair in six.moves.zip(self.achalls, ll_addrs)]
         config = nginxparser.UnspacedList(config)

certbot-nginx/setup.py

@@ -4,7 +4,7 @@ from setuptools import setup
 from setuptools import find_packages
 
 
-version = '0.18.0.dev0'
+version = '0.20.0.dev0'
 
 # Please update tox.ini when modifying dependency version requirements
 install_requires = [

certbot/__init__.py

@@ -1,4 +1,4 @@
 """Certbot client."""
 
 # version number like 1.2.3a0, must have at least 2 parts, like 1.2
-__version__ = '0.18.0.dev0'
+__version__ = '0.20.0.dev0'

certbot/cert_manager.py

@@ -3,6 +3,7 @@ import datetime
 import logging
 import os
 import pytz
+import re
 import traceback
 import zope.component
 
@@ -45,7 +46,7 @@ def rename_lineage(config):
     """
     disp = zope.component.getUtility(interfaces.IDisplay)
 
-    certname = _get_certname(config, "rename")
+    certname = _get_certnames(config, "rename")[0]
 
     new_certname = config.new_certname
     if not new_certname:
@@ -87,11 +88,12 @@ def certificates(config):
 
 def delete(config):
     """Delete Certbot files associated with a certificate lineage."""
-    certname = _get_certname(config, "delete")
-    storage.delete_files(config, certname)
-    disp = zope.component.getUtility(interfaces.IDisplay)
-    disp.notification("Deleted all files relating to certificate {0}."
-        .format(certname), pause=False)
+    certnames = _get_certnames(config, "delete", allow_multiple=True)
+    for certname in certnames:
+        storage.delete_files(config, certname)
+        disp = zope.component.getUtility(interfaces.IDisplay)
+        disp.notification("Deleted all files relating to certificate {0}."
+            .format(certname), pause=False)
 
 ###################
 # Public Helpers
@@ -141,28 +143,162 @@ def find_duplicative_certs(config, domains):
 
     return _search_lineages(config, update_certs_for_domain_matches, (None, None))
 
+def _archive_files(candidate_lineage, filetype):
+    """ In order to match things like:
+        /etc/letsencrypt/archive/example.com/chain1.pem.
+
+        Anonymous functions which call this function are eventually passed (in a list) to
+        `match_and_check_overlaps` to help specify the acceptable_matches.
+
+        :param `.storage.RenewableCert` candidate_lineage: Lineage whose archive dir is to
+            be searched.
+        :param str filetype: main file name prefix e.g. "fullchain" or "chain".
+
+        :returns: Files in candidate_lineage's archive dir that match the provided filetype.
+        :rtype: list of str or None
+    """
+    archive_dir = candidate_lineage.archive_dir
+    pattern = [os.path.join(archive_dir, f) for f in os.listdir(archive_dir)
+                    if re.match("{0}[0-9]*.pem".format(filetype), f)]
+    if len(pattern) > 0:
+        return pattern
+    else:
+        return None
+
+def _acceptable_matches():
+    """ Generates the list that's passed to match_and_check_overlaps. Is its own function to
+    make unit testing easier.
+
+    :returns: list of functions
+    :rtype: list
+    """
+    return [lambda x: x.fullchain_path, lambda x: x.cert_path,
+            lambda x: _archive_files(x, "cert"), lambda x: _archive_files(x, "fullchain")]
+
+def cert_path_to_lineage(cli_config):
+    """ If config.cert_path is defined, try to find an appropriate value for config.certname.
+
+    :param `configuration.NamespaceConfig` cli_config: parsed command line arguments
+
+    :returns: a lineage name
+    :rtype: str
+
+    :raises `errors.Error`: If the specified cert path can't be matched to a lineage name.
+    :raises `errors.OverlappingMatchFound`: If the matched lineage's archive is shared.
+    """
+    acceptable_matches = _acceptable_matches()
+    match = match_and_check_overlaps(cli_config, acceptable_matches,
+            lambda x: cli_config.cert_path[0], lambda x: x.lineagename)
+    return match[0]
+
+def match_and_check_overlaps(cli_config, acceptable_matches, match_func, rv_func):
+    """ Searches through all lineages for a match, and checks for duplicates.
+    If a duplicate is found, an error is raised, as performing operations on lineages
+    that have their properties incorrectly duplicated elsewhere is probably a bad idea.
+
+    :param `configuration.NamespaceConfig` cli_config: parsed command line arguments
+    :param list acceptable_matches: a list of functions that specify acceptable matches
+    :param function match_func: specifies what to match
+    :param function rv_func: specifies what to return
+
+    """
+    def find_matches(candidate_lineage, return_value, acceptable_matches):
+        """Returns a list of matches using _search_lineages."""
+        acceptable_matches = [func(candidate_lineage) for func in acceptable_matches]
+        acceptable_matches_rv = []
+        for item in acceptable_matches:
+            if isinstance(item, list):
+                acceptable_matches_rv += item
+            else:
+                acceptable_matches_rv.append(item)
+        match = match_func(candidate_lineage)
+        if match in acceptable_matches_rv:
+            return_value.append(rv_func(candidate_lineage))
+        return return_value
+
+    matched = _search_lineages(cli_config, find_matches, [], acceptable_matches)
+    if not matched:
+        raise errors.Error("No match found for cert-path {0}!".format(cli_config.cert_path[0]))
+    elif len(matched) > 1:
+        raise errors.OverlappingMatchFound()
+    else:
+        return matched
+
+def human_readable_cert_info(config, cert, skip_filter_checks=False):
+    """ Returns a human readable description of info about a RenewableCert object"""
+    certinfo = []
+    checker = ocsp.RevocationChecker()
+
+    if config.certname and cert.lineagename != config.certname and not skip_filter_checks:
+        return ""
+    if config.domains and not set(config.domains).issubset(cert.names()):
+        return ""
+    now = pytz.UTC.fromutc(datetime.datetime.utcnow())
+
+    reasons = []
+    if cert.is_test_cert:
+        reasons.append('TEST_CERT')
+    if cert.target_expiry <= now:
+        reasons.append('EXPIRED')
+    if checker.ocsp_revoked(cert.cert, cert.chain):
+        reasons.append('REVOKED')
+
+    if reasons:
+        status = "INVALID: " + ", ".join(reasons)
+    else:
+        diff = cert.target_expiry - now
+        if diff.days == 1:
+            status = "VALID: 1 day"
+        elif diff.days < 1:
+            status = "VALID: {0} hour(s)".format(diff.seconds // 3600)
+        else:
+            status = "VALID: {0} days".format(diff.days)
+
+    valid_string = "{0} ({1})".format(cert.target_expiry, status)
+    certinfo.append("  Certificate Name: {0}\n"
+                    "    Domains: {1}\n"
+                    "    Expiry Date: {2}\n"
+                    "    Certificate Path: {3}\n"
+                    "    Private Key Path: {4}".format(
+                         cert.lineagename,
+                         " ".join(cert.names()),
+                         valid_string,
+                         cert.fullchain,
+                         cert.privkey))
+    return "".join(certinfo)
 
 ###################
 # Private Helpers
 ###################
 
-def _get_certname(config, verb):
+def _get_certnames(config, verb, allow_multiple=False):
     """Get certname from flag, interactively, or error out.
     """
     certname = config.certname
-    if not certname:
+    if certname:
+        certnames = [certname]
+    else:
         disp = zope.component.getUtility(interfaces.IDisplay)
         filenames = storage.renewal_conf_files(config)
         choices = [storage.lineagename_for_filename(name) for name in filenames]
         if not choices:
             raise errors.Error("No existing certificates found.")
-        code, index = disp.menu("Which certificate would you like to {0}?".format(verb),
-                                choices, flag="--cert-name",
-                                force_interactive=True)
-        if code != display_util.OK or not index in range(0, len(choices)):
-            raise errors.Error("User ended interaction.")
-        certname = choices[index]
-    return certname
+        if allow_multiple:
+            code, certnames = disp.checklist(
+                                    "Which certificate(s) would you like to {0}?".format(verb),
+                                    choices, cli_flag="--cert-name",
+                                    force_interactive=True)
+            if code != display_util.OK:
+                raise errors.Error("User ended interaction.")
+        else:
+            code, index = disp.menu("Which certificate would you like to {0}?".format(verb),
+                                    choices, cli_flag="--cert-name",
+                                    force_interactive=True)
+
+            if code != display_util.OK or index not in range(0, len(choices)):
+                raise errors.Error("User ended interaction.")
+            certnames = [choices[index]]
+    return certnames
 
 def _report_lines(msgs):
     """Format a results report for a category of single-line renewal outcomes"""
@@ -171,44 +307,8 @@ def _report_lines(msgs):
 def _report_human_readable(config, parsed_certs):
     """Format a results report for a parsed cert"""
     certinfo = []
-    checker = ocsp.RevocationChecker()
     for cert in parsed_certs:
-        if config.certname and cert.lineagename != config.certname:
-            continue
-        if config.domains and not set(config.domains).issubset(cert.names()):
-            continue
-        now = pytz.UTC.fromutc(datetime.datetime.utcnow())
-
-        reasons = []
-        if cert.is_test_cert:
-            reasons.append('TEST_CERT')
-        if cert.target_expiry <= now:
-            reasons.append('EXPIRED')
-        if checker.ocsp_revoked(cert.cert, cert.chain):
-            reasons.append('REVOKED')
-
-        if reasons:
-            status = "INVALID: " + ", ".join(reasons)
-        else:
-            diff = cert.target_expiry - now
-            if diff.days == 1:
-                status = "VALID: 1 day"
-            elif diff.days < 1:
-                status = "VALID: {0} hour(s)".format(diff.seconds // 3600)
-            else:
-                status = "VALID: {0} days".format(diff.days)
-
-        valid_string = "{0} ({1})".format(cert.target_expiry, status)
-        certinfo.append("  Certificate Name: {0}\n"
-                        "    Domains: {1}\n"
-                        "    Expiry Date: {2}\n"
-                        "    Certificate Path: {3}\n"
-                        "    Private Key Path: {4}".format(
-                            cert.lineagename,
-                            ",".join(cert.names()),
-                            valid_string,
-                            cert.fullchain,
-                            cert.privkey))
+        certinfo.append(human_readable_cert_info(config, cert))
     return "\n".join(certinfo)
 
 def _describe_certs(config, parsed_certs, parse_failures):
@@ -232,11 +332,17 @@ def _describe_certs(config, parsed_certs, parse_failures):
     disp = zope.component.getUtility(interfaces.IDisplay)
     disp.notification("\n".join(out), pause=False, wrap=False)
 
-def _search_lineages(cli_config, func, initial_rv):
+def _search_lineages(cli_config, func, initial_rv, *args):
     """Iterate func over unbroken lineages, allowing custom return conditions.
 
     Allows flexible customization of return values, including multiple
     return values and complex checks.
+
+    :param `configuration.NamespaceConfig` cli_config: parsed command line arguments
+    :param function func: function used while searching over lineages
+    :param initial_rv: initial return value of the function (any type)
+
+    :returns: Whatever was specified by `func` if a match is found.
     """
     configs_dir = cli_config.renewal_configs_dir
     # Verify the directory is there
@@ -250,5 +356,5 @@ def _search_lineages(cli_config, func, initial_rv):
             logger.debug("Renewal conf file %s is broken. Skipping.", renewal_file)
             logger.debug("Traceback was:\n%s", traceback.format_exc())
             continue
-        rv = func(candidate_lineage, rv)
+        rv = func(candidate_lineage, rv, *args)
     return rv

certbot/cli.py

@@ -11,6 +11,9 @@ import sys
 
 import configargparse
 import six
+import zope.component
+
+from zope.interface import interfaces as zope_interfaces
 
 from acme import challenges
 
@@ -23,6 +26,7 @@ from certbot import hooks
 from certbot import interfaces
 from certbot import util
 
+from certbot.display import util as display_util
 from certbot.plugins import disco as plugins_disco
 import certbot.plugins.selection as plugin_selection
 
@@ -45,8 +49,13 @@ if "CERTBOT_AUTO" in os.environ:
     # user saved the script under a different name
     LEAUTO = os.path.basename(os.environ["CERTBOT_AUTO"])
 
-fragment = os.path.join(".local", "share", "letsencrypt")
-cli_command = LEAUTO if fragment in sys.argv[0] else "certbot"
+old_path_fragment = os.path.join(".local", "share", "letsencrypt")
+new_path_prefix = os.path.abspath(os.path.join(os.sep, "opt",
+                                               "eff.org", "certbot", "venv"))
+if old_path_fragment in sys.argv[0] or sys.argv[0].startswith(new_path_prefix):
+    cli_command = LEAUTO
+else:
+    cli_command = "certbot"
 
 # Argparse's help formatting has a lot of unhelpful peculiarities, so we want
 # to replace as much of it as we can...
@@ -120,6 +129,7 @@ ZERO_ARG_ACTIONS = set(("store_const", "store_true",
 # This dictionary is used recursively, so if A modifies B and B modifies C,
 # it is determined that C was modified by the user if A was modified.
 VAR_MODIFIERS = {"account": set(("server",)),
+                 "renew_hook": set(("deploy_hook",)),
                  "server": set(("dry_run", "staging",)),
                  "webroot_map": set(("webroot_path",))}
 
@@ -132,14 +142,14 @@ def report_config_interaction(modified, modifiers):
     between config options.
 
     :param modified: config options that can be modified by modifiers
-    :type modified: iterable or str
+    :type modified: iterable or str (string_types)
     :param modifiers: config options that modify modified
-    :type modifiers: iterable or str
+    :type modifiers: iterable or str (string_types)
 
     """
-    if isinstance(modified, str):
+    if isinstance(modified, six.string_types):
         modified = (modified,)
-    if isinstance(modifiers, str):
+    if isinstance(modifiers, six.string_types):
         modifiers = (modifiers,)
 
     for var in modified:
@@ -273,7 +283,7 @@ def flag_default(name):
     # argparse has been set up; it is not accurate for all flags.  Call it
     # with caution.  Plugin defaults are missing, and some things are using
     # defaults defined in this file, not in constants.py :(
-    return constants.CLI_DEFAULTS[name]
+    return copy.deepcopy(constants.CLI_DEFAULTS[name])
 
 
 def config_help(name, hidden=False):
@@ -347,7 +357,7 @@ VERB_HELP = [
                  " before and after renewal; see"
                  " https://certbot.eff.org/docs/using.html#renewal for more"
                  " information on these."),
-        "usage": "\n\n  certbot renew [--cert-name NAME] [options]\n\n"
+        "usage": "\n\n  certbot renew [--cert-name CERTNAME] [options]\n\n"
     }),
     ("certificates", {
         "short": "List certificates managed by Certbot",
@@ -439,6 +449,15 @@ class HelpfulArgumentParser(object):
             "delete": main.delete,
         }
 
+        # Get notification function for printing
+        try:
+            self.notify = zope.component.getUtility(
+                interfaces.IDisplay).notification
+        except zope_interfaces.ComponentLookupError:
+            self.notify = display_util.NoninteractiveDisplay(
+                sys.stdout).notification
+
+
         # List of topics for which additional help can be provided
         HELP_TOPICS = ["all", "security", "paths", "automation", "testing"]
         HELP_TOPICS += list(self.VERBS) + self.COMMANDS_TOPICS + ["manage"]
@@ -458,7 +477,7 @@ class HelpfulArgumentParser(object):
         if isinstance(help1, bool) and isinstance(help2, bool):
             self.help_arg = help1 or help2
         else:
-            self.help_arg = help1 if isinstance(help1, str) else help2
+            self.help_arg = help1 if isinstance(help1, six.string_types) else help2
 
         short_usage = self._usage_string(plugins, self.help_arg)
 
@@ -510,10 +529,10 @@ class HelpfulArgumentParser(object):
 
         usage = SHORT_USAGE
         if help_arg == True:
-            print(usage + COMMAND_OVERVIEW % (apache_doc, nginx_doc) + HELP_USAGE)
+            self.notify(usage + COMMAND_OVERVIEW % (apache_doc, nginx_doc) + HELP_USAGE)
             sys.exit(0)
         elif help_arg in self.COMMANDS_TOPICS:
-            print(usage + self._list_subcommands())
+            self.notify(usage + self._list_subcommands())
             sys.exit(0)
         elif help_arg == "all":
             # if we're doing --help all, the OVERVIEW is part of the SHORT_USAGE at
@@ -848,9 +867,10 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         "e.g. -vvv.")
     helpful.add(
         None, "-t", "--text", dest="text_mode", action="store_true",
-        help=argparse.SUPPRESS)
+        default=flag_default("text_mode"), help=argparse.SUPPRESS)
     helpful.add(
-        None, "--max-log-backups", type=nonnegative_int, default=1000,
+        None, "--max-log-backups", type=nonnegative_int,
+        default=flag_default("max_log_backups"),
         help="Specifies the maximum number of backup logs that should "
              "be kept by Certbot's built in log rotation. Setting this "
              "flag to 0 disables log rotation entirely, causing "
@@ -858,30 +878,37 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
     helpful.add(
         [None, "automation", "run", "certonly"], "-n", "--non-interactive", "--noninteractive",
         dest="noninteractive_mode", action="store_true",
+        default=flag_default("noninteractive_mode"),
         help="Run without ever asking for user input. This may require "
               "additional command line flags; the client will try to explain "
               "which ones are required if it finds one missing")
     helpful.add(
         [None, "register", "run", "certonly"],
         constants.FORCE_INTERACTIVE_FLAG, action="store_true",
+        default=flag_default("force_interactive"),
         help="Force Certbot to be interactive even if it detects it's not "
              "being run in a terminal. This flag cannot be used with the "
              "renew subcommand.")
     helpful.add(
         [None, "run", "certonly", "certificates"],
         "-d", "--domains", "--domain", dest="domains",
-        metavar="DOMAIN", action=_DomainsAction, default=[],
+        metavar="DOMAIN", action=_DomainsAction,
+        default=flag_default("domains"),
         help="Domain names to apply. For multiple domains you can use "
              "multiple -d flags or enter a comma separated list of domains "
-             "as a parameter. The first provided domain will be used in "
-             "some software user interfaces and file paths for the "
+             "as a parameter. The first domain provided will be the "
+             "subject CN of the certificate, and all domains will be "
+             "Subject Alternative Names on the certificate. "
+             "The first domain will also be used in "
+             "some software user interfaces and as the file paths for the "
              "certificate and related material unless otherwise "
-             "specified or you already have a certificate for the same "
-             "domains. (default: Ask)")
+             "specified or you already have a certificate with the same "
+             "name. In the case of a name collision it will append a number "
+             "like 0001 to the file path name. (default: Ask)")
     helpful.add(
-        [None, "run", "certonly", "manage", "delete", "certificates"],
+        [None, "run", "certonly", "manage", "delete", "certificates", "renew"],
         "--cert-name", dest="certname",
-        metavar="CERTNAME", default=None,
+        metavar="CERTNAME", default=flag_default("certname"),
         help="Certificate name to apply. This name is used by Certbot for housekeeping "
              "and in file paths; it doesn't affect the content of the certificate itself. "
              "To see certificate names, run 'certbot certificates'. "
@@ -891,6 +918,7 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
     helpful.add(
         [None, "testing", "renew", "certonly"],
         "--dry-run", action="store_true", dest="dry_run",
+        default=flag_default("dry_run"),
         help="Perform a test run of the client, obtaining test (invalid) certificates"
              " but not saving them to disk. This can currently only be used"
              " with the 'certonly' and 'renew' subcommands. \nNote: Although --dry-run"
@@ -903,6 +931,7 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
              " renewal. --deploy-hook commands are not called.")
     helpful.add(
         ["register", "automation"], "--register-unsafely-without-email", action="store_true",
+        default=flag_default("register_unsafely_without_email"),
         help="Specifying this flag enables registering an account with no "
              "email address. This is strongly discouraged, because in the "
              "event of key loss or account compromise you will irrevocably "
@@ -913,27 +942,29 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
              "update to the web site.")
     helpful.add(
         "register", "--update-registration", action="store_true",
+        default=flag_default("update_registration"),
         help="With the register verb, indicates that details associated "
              "with an existing registration, such as the e-mail address, "
              "should be updated, rather than registering a new account.")
     helpful.add(
         ["register", "unregister", "automation"], "-m", "--email",
+        default=flag_default("email"),
         help=config_help("email"))
     helpful.add(["register", "automation"], "--eff-email", action="store_true",
-                default=None, dest="eff_email",
+                default=flag_default("eff_email"), dest="eff_email",
                 help="Share your e-mail address with EFF")
     helpful.add(["register", "automation"], "--no-eff-email", action="store_false",
-                default=None, dest="eff_email",
+                default=flag_default("eff_email"), dest="eff_email",
                 help="Don't share your e-mail address with EFF")
     helpful.add(
         ["automation", "certonly", "run"],
         "--keep-until-expiring", "--keep", "--reinstall",
-        dest="reinstall", action="store_true",
+        dest="reinstall", action="store_true", default=flag_default("reinstall"),
         help="If the requested certificate matches an existing certificate, always keep the "
              "existing one until it is due for renewal (for the "
              "'run' subcommand this means reinstall the existing certificate). (default: Ask)")
     helpful.add(
-        "automation", "--expand", action="store_true",
+        "automation", "--expand", action="store_true", default=flag_default("expand"),
         help="If an existing certificate is a strict subset of the requested names, "
              "always expand and replace it with the additional names. (default: Ask)")
     helpful.add(
@@ -942,21 +973,24 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         help="show program's version number and exit")
     helpful.add(
         ["automation", "renew"],
-        "--force-renewal", "--renew-by-default",
-        action="store_true", dest="renew_by_default", help="If a certificate "
+        "--force-renewal", "--renew-by-default", dest="renew_by_default",
+        action="store_true", default=flag_default("renew_by_default"),
+        help="If a certificate "
              "already exists for the requested domains, renew it now, "
              "regardless of whether it is near expiry. (Often "
              "--keep-until-expiring is more appropriate). Also implies "
              "--expand.")
     helpful.add(
-        "automation", "--renew-with-new-domains",
-        action="store_true", dest="renew_with_new_domains", help="If a "
+        "automation", "--renew-with-new-domains", dest="renew_with_new_domains",
+        action="store_true", default=flag_default("renew_with_new_domains"),
+        help="If a "
              "certificate already exists for the requested certificate name "
              "but does not match the requested domains, renew it now, "
              "regardless of whether it is near expiry.")
     helpful.add(
         ["automation", "renew", "certonly"],
         "--allow-subset-of-names", action="store_true",
+        default=flag_default("allow_subset_of_names"),
         help="When performing domain validation, do not consider it a failure "
              "if authorizations can not be obtained for a strict subset of "
              "the requested domains. This may be useful for allowing renewals for "
@@ -964,39 +998,46 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
              "at this system. This option cannot be used with --csr.")
     helpful.add(
         "automation", "--agree-tos", dest="tos", action="store_true",
+        default=flag_default("tos"),
         help="Agree to the ACME Subscriber Agreement (default: Ask)")
     helpful.add(
         ["unregister", "automation"], "--account", metavar="ACCOUNT_ID",
+        default=flag_default("account"),
         help="Account ID to use")
     helpful.add(
         "automation", "--duplicate", dest="duplicate", action="store_true",
+        default=flag_default("duplicate"),
         help="Allow making a certificate lineage that duplicates an existing one "
              "(both can be renewed in parallel)")
     helpful.add(
         "automation", "--os-packages-only", action="store_true",
+        default=flag_default("os_packages_only"),
         help="(certbot-auto only) install OS package dependencies and then stop")
     helpful.add(
         "automation", "--no-self-upgrade", action="store_true",
+        default=flag_default("no_self_upgrade"),
         help="(certbot-auto only) prevent the certbot-auto script from"
              " upgrading itself to newer released versions (default: Upgrade"
              " automatically)")
     helpful.add(
         "automation", "--no-bootstrap", action="store_true",
+        default=flag_default("no_bootstrap"),
         help="(certbot-auto only) prevent the certbot-auto script from"
              " installing OS-level dependencies (default: Prompt to install "
              " OS-wide dependencies, but exit if the user says 'No')")
     helpful.add(
         ["automation", "renew", "certonly", "run"],
         "-q", "--quiet", dest="quiet", action="store_true",
+        default=flag_default("quiet"),
         help="Silence all output except errors. Useful for automation via cron."
              " Implies --non-interactive.")
     # overwrites server, handled in HelpfulArgumentParser.parse_args()
     helpful.add(["testing", "revoke", "run"], "--test-cert", "--staging",
-        action='store_true', dest='staging',
-        help='Use the staging server to obtain or revoke test (invalid) certificates; equivalent'
-             ' to --server ' + constants.STAGING_URI)
+        dest="staging", action="store_true", default=flag_default("staging"),
+        help="Use the staging server to obtain or revoke test (invalid) certificates; equivalent"
+             " to --server " + constants.STAGING_URI)
     helpful.add(
-        "testing", "--debug", action="store_true",
+        "testing", "--debug", action="store_true", default=flag_default("debug"),
         help="Show tracebacks in case of errors, and allow certbot-auto "
              "execution on experimental platforms")
     helpful.add(
@@ -1026,6 +1067,7 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         default=flag_default("http01_address"), help=config_help("http01_address"))
     helpful.add(
         "testing", "--break-my-certs", action="store_true",
+        default=flag_default("break_my_certs"),
         help="Be willing to replace or renew valid certificates with invalid "
              "(testing/staging) certificates")
     helpful.add(
@@ -1033,47 +1075,51 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         default=flag_default("rsa_key_size"), help=config_help("rsa_key_size"))
     helpful.add(
         "security", "--must-staple", action="store_true",
-        help=config_help("must_staple"), dest="must_staple", default=False)
+        dest="must_staple", default=flag_default("must_staple"),
+        help=config_help("must_staple"))
     helpful.add(
-        "security", "--redirect", action="store_true",
+        "security", "--redirect", action="store_true", dest="redirect",
+        default=flag_default("redirect"),
         help="Automatically redirect all HTTP traffic to HTTPS for the newly "
-             "authenticated vhost. (default: Ask)", dest="redirect", default=None)
+             "authenticated vhost. (default: Ask)")
     helpful.add(
-        "security", "--no-redirect", action="store_false",
+        "security", "--no-redirect", action="store_false", dest="redirect",
+        default=flag_default("redirect"),
         help="Do not automatically redirect all HTTP traffic to HTTPS for the newly "
-             "authenticated vhost. (default: Ask)", dest="redirect", default=None)
+             "authenticated vhost. (default: Ask)")
     helpful.add(
-        "security", "--hsts", action="store_true",
+        "security", "--hsts", action="store_true", dest="hsts", default=flag_default("hsts"),
         help="Add the Strict-Transport-Security header to every HTTP response."
              " Forcing browser to always use SSL for the domain."
-             " Defends against SSL Stripping.", dest="hsts", default=False)
+             " Defends against SSL Stripping.")
     helpful.add(
-        "security", "--no-hsts", action="store_false",
-        help=argparse.SUPPRESS, dest="hsts", default=False)
+        "security", "--no-hsts", action="store_false", dest="hsts",
+        default=flag_default("hsts"), help=argparse.SUPPRESS)
     helpful.add(
-        "security", "--uir", action="store_true",
-        help="Add the \"Content-Security-Policy: upgrade-insecure-requests\""
-             " header to every HTTP response. Forcing the browser to use"
-             " https:// for every http:// resource.", dest="uir", default=None)
+        "security", "--uir", action="store_true", dest="uir", default=flag_default("uir"),
+        help='Add the "Content-Security-Policy: upgrade-insecure-requests"'
+             ' header to every HTTP response. Forcing the browser to use'
+             ' https:// for every http:// resource.')
     helpful.add(
-        "security", "--no-uir", action="store_false",
-        help=argparse.SUPPRESS, dest="uir", default=None)
+        "security", "--no-uir", action="store_false", dest="uir", default=flag_default("uir"),
+        help=argparse.SUPPRESS)
     helpful.add(
-        "security", "--staple-ocsp", action="store_true",
+        "security", "--staple-ocsp", action="store_true", dest="staple",
+        default=flag_default("staple"),
         help="Enables OCSP Stapling. A valid OCSP response is stapled to"
-        " the certificate that the server offers during TLS.",
-        dest="staple", default=None)
+        " the certificate that the server offers during TLS.")
     helpful.add(
-        "security", "--no-staple-ocsp", action="store_false",
-        help=argparse.SUPPRESS, dest="staple", default=None)
+        "security", "--no-staple-ocsp", action="store_false", dest="staple",
+        default=flag_default("staple"), help=argparse.SUPPRESS)
     helpful.add(
         "security", "--strict-permissions", action="store_true",
+        default=flag_default("strict_permissions"),
         help="Require that all configuration files are owned by the current "
              "user; only needed if your config is somewhere unsafe like /tmp/")
     helpful.add(
         ["manual", "standalone", "certonly", "renew"],
         "--preferred-challenges", dest="pref_challs",
-        action=_PrefChallAction, default=[],
+        action=_PrefChallAction, default=flag_default("pref_challs"),
         help='A sorted, comma delimited list of the preferred challenge to '
              'use during authorization with the most preferred challenge '
              'listed first (Eg, "dns" or "tls-sni-01,http,dns"). '
@@ -1102,17 +1148,18 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
                 action=_RenewHookAction, help=argparse.SUPPRESS)
     helpful.add(
         "renew", "--deploy-hook", action=_DeployHookAction,
-        help="Command to be run in a shell once for each successfully"
-        " issued certificate. For this command, the shell variable"
-        " $RENEWED_LINEAGE will point to the config live subdirectory"
+        help='Command to be run in a shell once for each successfully'
+        ' issued certificate. For this command, the shell variable'
+        ' $RENEWED_LINEAGE will point to the config live subdirectory'
         ' (for example, "/etc/letsencrypt/live/example.com") containing'
-        " the new certificates and keys; the shell variable"
-        " $RENEWED_DOMAINS will contain a space-delimited list of"
+        ' the new certificates and keys; the shell variable'
+        ' $RENEWED_DOMAINS will contain a space-delimited list of'
         ' renewed certificate domains (for example, "example.com'
         ' www.example.com"')
     helpful.add(
         "renew", "--disable-hook-validation",
-        action='store_false', dest='validate_hooks', default=True,
+        action="store_false", dest="validate_hooks",
+        default=flag_default("validate_hooks"),
         help="Ordinarily the commands specified for"
         " --pre-hook/--post-hook/--deploy-hook will be checked for"
         " validity, to see if the programs being run are in the $PATH,"
@@ -1121,6 +1168,11 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
         " simplistic and fails if you use more advanced shell"
         " constructs, so you can use this switch to disable it."
         " (default: False)")
+    helpful.add(
+        "renew", "--no-directory-hooks", action="store_false",
+        default=flag_default("directory_hooks"), dest="directory_hooks",
+        help="Disable running executables found in Certbot's hook directories"
+        " during renewal. (default: False)")
 
     helpful.add_deprecated_argument("--agree-dev-preview", 0)
     helpful.add_deprecated_argument("--dialog", 0)
@@ -1138,48 +1190,53 @@ def prepare_and_parse_args(plugins, args, detect_defaults=False):  # pylint: dis
 
 
 def _create_subparsers(helpful):
-    helpful.add("config_changes", "--num", type=int,
+    helpful.add("config_changes", "--num", type=int, default=flag_default("num"),
                 help="How many past revisions you want to be displayed")
 
     from certbot.client import sample_user_agent # avoid import loops
     helpful.add(
-        None, "--user-agent", default=None,
-        help="Set a custom user agent string for the client. User agent strings allow "
-             "the CA to collect high level statistics about success rates by OS, "
-             "plugin and use case, and to know when to deprecate support for past Python "
+        None, "--user-agent", default=flag_default("user_agent"),
+        help='Set a custom user agent string for the client. User agent strings allow '
+             'the CA to collect high level statistics about success rates by OS, '
+             'plugin and use case, and to know when to deprecate support for past Python '
              "versions and flags. If you wish to hide this information from the Let's "
              'Encrypt server, set this to "". '
              '(default: {0}). The flags encoded in the user agent are: '
              '--duplicate, --force-renew, --allow-subset-of-names, -n, and '
              'whether any hooks are set.'.format(sample_user_agent()))
     helpful.add(
-        None, "--user-agent-comment", default=None, type=_user_agent_comment_type,
+        None, "--user-agent-comment", default=flag_default("user_agent_comment"),
+        type=_user_agent_comment_type,
         help="Add a comment to the default user agent string. May be used when repackaging Certbot "
              "or calling it from another tool to allow additional statistical data to be collected."
              " Ignored if --user-agent is set. (Example: Foo-Wrapper/1.0)")
     helpful.add("certonly",
-                "--csr", type=read_file,
+                "--csr", default=flag_default("csr"), type=read_file,
                 help="Path to a Certificate Signing Request (CSR) in DER or PEM format."
                 " Currently --csr only works with the 'certonly' subcommand.")
     helpful.add("revoke",
                 "--reason", dest="reason",
                 choices=CaseInsensitiveList(sorted(constants.REVOCATION_REASONS,
                                                    key=constants.REVOCATION_REASONS.get)),
-                action=_EncodeReasonAction, default=0,
+                action=_EncodeReasonAction, default=flag_default("reason"),
                 help="Specify reason for revoking certificate. (default: unspecified)")
     helpful.add("rollback",
                 "--checkpoints", type=int, metavar="N",
                 default=flag_default("rollback_checkpoints"),
                 help="Revert configuration N number of checkpoints.")
     helpful.add("plugins",
-                "--init", action="store_true", help="Initialize plugins.")
+                "--init", action="store_true", default=flag_default("init"),
+                help="Initialize plugins.")
     helpful.add("plugins",
-                "--prepare", action="store_true", help="Initialize and prepare plugins.")
+                "--prepare", action="store_true", default=flag_default("prepare"),
+                help="Initialize and prepare plugins.")
     helpful.add("plugins",
                 "--authenticators", action="append_const", dest="ifaces",
+                default=flag_default("ifaces"),
                 const=interfaces.IAuthenticator, help="Limit to authenticator plugins only.")
     helpful.add("plugins",
                 "--installers", action="append_const", dest="ifaces",
+                default=flag_default("ifaces"),
                 const=interfaces.IInstaller, help="Limit to installer plugins only.")
 
 
@@ -1245,53 +1302,68 @@ def _plugins_parsing(helpful, plugins):
         "a particular plugin by setting options provided below. Running "
         "--help <plugin_name> will list flags specific to that plugin.")
 
-    helpful.add("plugins", "--configurator",
+    helpful.add("plugins", "--configurator", default=flag_default("configurator"),
                 help="Name of the plugin that is both an authenticator and an installer."
                 " Should not be used together with --authenticator or --installer. "
                 "(default: Ask)")
-    helpful.add("plugins", "-a", "--authenticator", help="Authenticator plugin name.")
-    helpful.add("plugins", "-i", "--installer",
+    helpful.add("plugins", "-a", "--authenticator", default=flag_default("authenticator"),
+                help="Authenticator plugin name.")
+    helpful.add("plugins", "-i", "--installer", default=flag_default("installer"),
                 help="Installer plugin name (also used to find domains).")
     helpful.add(["plugins", "certonly", "run", "install", "config_changes"],
-                "--apache", action="store_true",
+                "--apache", action="store_true", default=flag_default("apache"),
                 help="Obtain and install certificates using Apache")
     helpful.add(["plugins", "certonly", "run", "install", "config_changes"],
-                "--nginx", action="store_true", help="Obtain and install certificates using Nginx")
+                "--nginx", action="store_true", default=flag_default("nginx"),
+                help="Obtain and install certificates using Nginx")
     helpful.add(["plugins", "certonly"], "--standalone", action="store_true",
+                default=flag_default("standalone"),
                 help='Obtain certificates using a "standalone" webserver.')
     helpful.add(["plugins", "certonly"], "--manual", action="store_true",
-                help='Provide laborious manual instructions for obtaining a certificate')
+                default=flag_default("manual"),
+                help="Provide laborious manual instructions for obtaining a certificate")
     helpful.add(["plugins", "certonly"], "--webroot", action="store_true",
-                help='Obtain certificates by placing files in a webroot directory.')
+                default=flag_default("webroot"),
+                help="Obtain certificates by placing files in a webroot directory.")
     helpful.add(["plugins", "certonly"], "--dns-cloudflare", action="store_true",
-                help=('Obtain certificates using a DNS TXT record (if you are '
-                      'using Cloudflare for DNS).'))
+                default=flag_default("dns_cloudflare"),
+                help=("Obtain certificates using a DNS TXT record (if you are "
+                      "using Cloudflare for DNS)."))
     helpful.add(["plugins", "certonly"], "--dns-cloudxns", action="store_true",
-                help=('Obtain certificates using a DNS TXT record (if you are '
-                     'using CloudXNS for DNS).'))
+                default=flag_default("dns_cloudxns"),
+                help=("Obtain certificates using a DNS TXT record (if you are "
+                     "using CloudXNS for DNS)."))
     helpful.add(["plugins", "certonly"], "--dns-digitalocean", action="store_true",
-                help=('Obtain certificates using a DNS TXT record (if you are '
-                      'using DigitalOcean for DNS).'))
+                default=flag_default("dns_digitalocean"),
+                help=("Obtain certificates using a DNS TXT record (if you are "
+                      "using DigitalOcean for DNS)."))
     helpful.add(["plugins", "certonly"], "--dns-dnsimple", action="store_true",
-                help=('Obtain certificates using a DNS TXT record (if you are '
-                      'using DNSimple for DNS).'))
+                default=flag_default("dns_dnsimple"),
+                help=("Obtain certificates using a DNS TXT record (if you are "
+                      "using DNSimple for DNS)."))
     helpful.add(["plugins", "certonly"], "--dns-dnsmadeeasy", action="store_true",
-                help=('Obtain certificates using a DNS TXT record (if you are'
-                      'using DNS Made Easy for DNS).'))
+                default=flag_default("dns_dnsmadeeasy"),
+                help=("Obtain certificates using a DNS TXT record (if you are"
+                      "using DNS Made Easy for DNS)."))
     helpful.add(["plugins", "certonly"], "--dns-google", action="store_true",
-                help=('Obtain certificates using a DNS TXT record (if you are '
-                      'using Google Cloud DNS).'))
+                default=flag_default("dns_google"),
+                help=("Obtain certificates using a DNS TXT record (if you are "
+                      "using Google Cloud DNS)."))
     helpful.add(["plugins", "certonly"], "--dns-luadns", action="store_true",
-                help=('Obtain certificates using a DNS TXT record (if you are '
-                      'using LuaDNS for DNS).'))
+                default=flag_default("dns_luadns"),
+                help=("Obtain certificates using a DNS TXT record (if you are "
+                      "using LuaDNS for DNS)."))
     helpful.add(["plugins", "certonly"], "--dns-nsone", action="store_true",
-                help=('Obtain certificates using a DNS TXT record (if you are '
-                      'using NS1 for DNS).'))
+                default=flag_default("dns_nsone"),
+                help=("Obtain certificates using a DNS TXT record (if you are "
+                      "using NS1 for DNS)."))
     helpful.add(["plugins", "certonly"], "--dns-rfc2136", action="store_true",
-                help='Obtain certificates using a DNS TXT record (if you are using BIND for DNS).')
+                default=flag_default("dns_rfc2136"),
+                help="Obtain certificates using a DNS TXT record (if you are using BIND for DNS).")
     helpful.add(["plugins", "certonly"], "--dns-route53", action="store_true",
-                help=('Obtain certificates using a DNS TXT record (if you are using Route53 for '
-                      'DNS).'))
+                default=flag_default("dns_route53"),
+                help=("Obtain certificates using a DNS TXT record (if you are using Route53 for "
+                      "DNS)."))
 
     # things should not be reorder past/pre this comment:
     # plugins_group should be displayed in --help before plugin

certbot/configuration.py

@@ -108,6 +108,30 @@ class NamespaceConfig(object):
         return os.path.join(
             self.namespace.config_dir, constants.RENEWAL_CONFIGS_DIR)
 
+    @property
+    def renewal_hooks_dir(self):
+        """Path to directory with hooks to run with the renew subcommand."""
+        return os.path.join(self.namespace.config_dir,
+                            constants.RENEWAL_HOOKS_DIR)
+
+    @property
+    def renewal_pre_hooks_dir(self):
+        """Path to the pre-hook directory for the renew subcommand."""
+        return os.path.join(self.renewal_hooks_dir,
+                            constants.RENEWAL_PRE_HOOKS_DIR)
+
+    @property
+    def renewal_deploy_hooks_dir(self):
+        """Path to the deploy-hook directory for the renew subcommand."""
+        return os.path.join(self.renewal_hooks_dir,
+                            constants.RENEWAL_DEPLOY_HOOKS_DIR)
+
+    @property
+    def renewal_post_hooks_dir(self):
+        """Path to the post-hook directory for the renew subcommand."""
+        return os.path.join(self.renewal_hooks_dir,
+                            constants.RENEWAL_POST_HOOKS_DIR)
+
 
 def check_config_sanity(config):
     """Validate command line options and display error message if

certbot/constants.py

@@ -1,6 +1,7 @@
 """Certbot constants."""
-import os
 import logging
+import os
+import pkg_resources
 
 from acme import challenges
 
@@ -18,23 +19,92 @@ CLI_DEFAULTS = dict(
         os.path.join(os.environ.get("XDG_CONFIG_HOME", "~/.config"),
                      "letsencrypt", "cli.ini"),
     ],
+
+    # Main parser
     verbose_count=-int(logging.INFO / 10),
-    server="https://acme-v01.api.letsencrypt.org/directory",
+    text_mode=False,
+    max_log_backups=1000,
+    noninteractive_mode=False,
+    force_interactive=False,
+    domains=[],
+    certname=None,
+    dry_run=False,
+    register_unsafely_without_email=False,
+    update_registration=False,
+    email=None,
+    eff_email=None,
+    reinstall=False,
+    expand=False,
+    renew_by_default=False,
+    renew_with_new_domains=False,
+    allow_subset_of_names=False,
+    tos=False,
+    account=None,
+    duplicate=False,
+    os_packages_only=False,
+    no_self_upgrade=False,
+    no_bootstrap=False,
+    quiet=False,
+    staging=False,
+    debug=False,
+    debug_challenges=False,
+    no_verify_ssl=False,
+    tls_sni_01_port=challenges.TLSSNI01Response.PORT,
+    tls_sni_01_address="",
+    http01_port=challenges.HTTP01Response.PORT,
+    http01_address="",
+    break_my_certs=False,
     rsa_key_size=2048,
+    must_staple=False,
+    redirect=None,
+    hsts=None,
+    uir=None,
+    staple=None,
+    strict_permissions=False,
+    pref_challs=[],
+    validate_hooks=True,
+    directory_hooks=True,
+
+    # Subparsers
+    num=None,
+    user_agent=None,
+    user_agent_comment=None,
+    csr=None,
+    reason=0,
     rollback_checkpoints=1,
+    init=False,
+    prepare=False,
+    ifaces=None,
+
+    # Path parsers
+    auth_cert_path="./cert.pem",
+    auth_chain_path="./chain.pem",
+    key_path=None,
     config_dir="/etc/letsencrypt",
     work_dir="/var/lib/letsencrypt",
     logs_dir="/var/log/letsencrypt",
-    no_verify_ssl=False,
-    http01_port=challenges.HTTP01Response.PORT,
-    http01_address="",
-    tls_sni_01_port=challenges.TLSSNI01Response.PORT,
-    tls_sni_01_address="",
+    server="https://acme-v01.api.letsencrypt.org/directory",
+
+    # Plugins parsers
+    configurator=None,
+    authenticator=None,
+    installer=None,
+    apache=False,
+    nginx=False,
+    standalone=False,
+    manual=False,
+    webroot=False,
+    dns_cloudflare=False,
+    dns_cloudxns=False,
+    dns_digitalocean=False,
+    dns_dnsimple=False,
+    dns_dnsmadeeasy=False,
+    dns_google=False,
+    dns_luadns=False,
+    dns_nsone=False,
+    dns_rfc2136=False,
+    dns_route53=False
 
-    auth_cert_path="./cert.pem",
-    auth_chain_path="./chain.pem",
-    strict_permissions=False,
-    debug_challenges=False,
 )
 STAGING_URI = "https://acme-staging.api.letsencrypt.org/directory"
 
@@ -107,8 +177,35 @@ TEMP_CHECKPOINT_DIR = "temp_checkpoint"
 RENEWAL_CONFIGS_DIR = "renewal"
 """Renewal configs directory, relative to `IConfig.config_dir`."""
 
+RENEWAL_HOOKS_DIR = "renewal-hooks"
+"""Basename of directory containing hooks to run with the renew command."""
+
+RENEWAL_PRE_HOOKS_DIR = "pre"
+"""Basename of directory containing pre-hooks to run with the renew command."""
+
+RENEWAL_DEPLOY_HOOKS_DIR = "deploy"
+"""Basename of directory containing deploy-hooks to run with the renew command."""
+
+RENEWAL_POST_HOOKS_DIR = "post"
+"""Basename of directory containing post-hooks to run with the renew command."""
+
 FORCE_INTERACTIVE_FLAG = "--force-interactive"
 """Flag to disable TTY checking in IDisplay."""
 
 EFF_SUBSCRIBE_URI = "https://supporters.eff.org/subscribe/certbot"
 """EFF URI used to submit the e-mail address of users who opt-in."""
+
+SSL_DHPARAMS_DEST = "ssl-dhparams.pem"
+"""Name of the ssl_dhparams file as saved in `IConfig.config_dir`."""
+
+SSL_DHPARAMS_SRC = pkg_resources.resource_filename(
+    "certbot", "ssl-dhparams.pem")
+"""Path to the nginx ssl_dhparams file found in the Certbot distribution."""
+
+UPDATED_SSL_DHPARAMS_DIGEST = ".updated-ssl-dhparams-pem-digest.txt"
+"""Name of the hash of the updated or informed ssl_dhparams as saved in `IConfig.config_dir`."""
+
+ALL_SSL_DHPARAMS_HASHES = [
+    '9ba6429597aeed2d8617a7705b56e96d044f64b07971659382e426675105654b',
+]
+"""SHA256 hashes of the contents of all versions of SSL_DHPARAMS_SRC"""

certbot/display/util.py

@@ -117,6 +117,7 @@ class FileDisplay(object):
         self.outfile.write(
             "{line}{frame}{line}{msg}{line}{frame}{line}".format(
                 line=os.linesep, frame=side_frame, msg=message))
+        self.outfile.flush()
         if pause:
             if self._can_interact(force_interactive):
                 input_with_timeout("Press Enter to Continue")
@@ -213,6 +214,7 @@ class FileDisplay(object):
 
         self.outfile.write("{0}{frame}{msg}{0}{frame}".format(
             os.linesep, frame=side_frame, msg=message))
+        self.outfile.flush()
 
         while True:
             ans = input_with_timeout("{yes}/{no}: ".format(
@@ -267,6 +269,7 @@ class FileDisplay(object):
                 else:
                     self.outfile.write(
                         "** Error - Invalid selection **%s" % os.linesep)
+                    self.outfile.flush()
             else:
                 return code, []
 
@@ -395,6 +398,7 @@ class FileDisplay(object):
             self.outfile.write(os.linesep)
 
         self.outfile.write(side_frame)
+        self.outfile.flush()
 
     def _get_valid_int_ans(self, max_):
         """Get a numerical selection.
@@ -428,6 +432,7 @@ class FileDisplay(object):
             except ValueError:
                 self.outfile.write(
                     "{0}** Invalid input **{0}".format(os.linesep))
+                self.outfile.flush()
 
         return OK, selection
 
@@ -483,6 +488,7 @@ class NoninteractiveDisplay(object):
         self.outfile.write(
             "{line}{frame}{line}{msg}{line}{frame}{line}".format(
                 line=os.linesep, frame=side_frame, msg=message))
+        self.outfile.flush()
 
     def menu(self, message, choices, ok_label=None, cancel_label=None,
              help_label=None, default=None, cli_flag=None, **unused_kwargs):

certbot/error_handler.py

@@ -29,18 +29,19 @@ class ErrorHandler(object):
     """Context manager for running code that must be cleaned up on failure.
 
     The context manager allows you to register functions that will be called
-    when an exception (excluding SystemExit) or signal is encountered. Usage:
+    when an exception (excluding SystemExit) or signal is encountered.
+    Usage::
 
-    handler = ErrorHandler(cleanup1_func, *cleanup1_args, **cleanup1_kwargs)
-    handler.register(cleanup2_func, *cleanup2_args, **cleanup2_kwargs)
+        handler = ErrorHandler(cleanup1_func, *cleanup1_args, **cleanup1_kwargs)
+        handler.register(cleanup2_func, *cleanup2_args, **cleanup2_kwargs)
 
-    with handler:
-        do_something()
+        with handler:
+            do_something()
 
-    Or for one cleanup function:
+    Or for one cleanup function::
 
-    with ErrorHandler(func, args, kwargs):
-        do_something()
+        with ErrorHandler(func, args, kwargs):
+            do_something()
 
     If an exception is raised out of do_something, the cleanup functions will
     be called in last in first out order. Then the exception is raised.
@@ -84,7 +85,7 @@ class ErrorHandler(object):
         return retval
 
     def register(self, func, *args, **kwargs):
-        """Sets func to be called with *args and **kwargs during cleanup
+        """Sets func to be run with the given arguments during cleanup.
 
         :param function func: function to be called in case of an error
 

certbot/errors.py

@@ -32,6 +32,8 @@ class HookCommandNotFound(Error):
 class SignalExit(Error):
     """A Unix signal was received while in the ErrorHandler context manager."""
 
+class OverlappingMatchFound(Error):
+    """Multiple lineages matched what should have been a unique result."""
 
 class LockError(Error):
     """File locking error."""

certbot/hooks.py

@@ -57,30 +57,71 @@ def validate_hook(shell_cmd, hook_name):
 
 
 def pre_hook(config):
-    "Run pre-hook if it's defined and hasn't been run."
+    """Run pre-hooks if they exist and haven't already been run.
+
+    When Certbot is running with the renew subcommand, this function
+    runs any hooks found in the config.renewal_pre_hooks_dir (if they
+    have not already been run) followed by any pre-hook in the config.
+    If hooks in config.renewal_pre_hooks_dir are run and the pre-hook in
+    the config is a path to one of these scripts, it is not run twice.
+
+    :param configuration.NamespaceConfig config: Certbot settings
+
+    """
+    if config.verb == "renew" and config.directory_hooks:
+        for hook in list_hooks(config.renewal_pre_hooks_dir):
+            _run_pre_hook_if_necessary(hook)
+
     cmd = config.pre_hook
-    if cmd and cmd not in pre_hook.already:
-        logger.info("Running pre-hook command: %s", cmd)
-        _run_hook(cmd)
-        pre_hook.already.add(cmd)
-    elif cmd:
-        logger.info("Pre-hook command already run, skipping: %s", cmd)
+    if cmd:
+        _run_pre_hook_if_necessary(cmd)
 
 pre_hook.already = set()  # type: ignore
 
 
-def post_hook(config):
-    """Run post hook if defined.
+def _run_pre_hook_if_necessary(command):
+    """Run the specified pre-hook if we haven't already.
+
+    If we've already run this exact command before, a message is logged
+    saying the pre-hook was skipped.
+
+    :param str command: pre-hook to be run
+
+    """
+    if command in pre_hook.already:
+        logger.info("Pre-hook command already run, skipping: %s", command)
+    else:
+        logger.info("Running pre-hook command: %s", command)
+        _run_hook(command)
+        pre_hook.already.add(command)
+
+
+def post_hook(config):
+    """Run post-hooks if defined.
+
+    This function also registers any executables found in
+    config.renewal_post_hooks_dir to be run when Certbot is used with
+    the renew subcommand.
+
+    If the verb is renew, we delay executing any post-hooks until
+    :func:`run_saved_post_hooks` is called. In this case, this function
+    registers all hooks found in config.renewal_post_hooks_dir to be
+    called followed by any post-hook in the config. If the post-hook in
+    the config is a path to an executable in the post-hook directory, it
+    is not scheduled to be run twice.
+
+    :param configuration.NamespaceConfig config: Certbot settings
 
-    If the verb is renew, we might have more certs to renew, so we wait until
-    run_saved_post_hooks() is called.
     """
 
     cmd = config.post_hook
     # In the "renew" case, we save these up to run at the end
     if config.verb == "renew":
-        if cmd and cmd not in post_hook.eventually:
-            post_hook.eventually.append(cmd)
+        if config.directory_hooks:
+            for hook in list_hooks(config.renewal_post_hooks_dir):
+                _run_eventually(hook)
+        if cmd:
+            _run_eventually(cmd)
     # certonly / run
     elif cmd:
         logger.info("Running post-hook command: %s", cmd)
@@ -89,6 +130,19 @@ def post_hook(config):
 post_hook.eventually = []  # type: ignore
 
 
+def _run_eventually(command):
+    """Registers a post-hook to be run eventually.
+
+    All commands given to this function will be run exactly once in the
+    order they were given when :func:`run_saved_post_hooks` is called.
+
+    :param str command: post-hook to register to be run
+
+    """
+    if command not in post_hook.eventually:
+        post_hook.eventually.append(command)
+
+
 def run_saved_post_hooks():
     """Run any post hooks that were saved up in the course of the 'renew' verb"""
     for cmd in post_hook.eventually:
@@ -106,20 +160,65 @@ def deploy_hook(config, domains, lineage_path):
 
     """
     if config.deploy_hook:
-        renew_hook(config, domains, lineage_path)
+        _run_deploy_hook(config.deploy_hook, domains,
+                         lineage_path, config.dry_run)
 
 
 def renew_hook(config, domains, lineage_path):
-    """Run post-renewal hook if defined."""
+    """Run post-renewal hooks.
+
+    This function runs any hooks found in
+    config.renewal_deploy_hooks_dir followed by any renew-hook in the
+    config. If the renew-hook in the config is a path to a script in
+    config.renewal_deploy_hooks_dir, it is not run twice.
+
+    If Certbot is doing a dry run, no hooks are run and messages are
+    logged saying that they were skipped.
+
+    :param configuration.NamespaceConfig config: Certbot settings
+    :param domains: domains in the obtained certificate
+    :type domains: `list` of `str`
+    :param str lineage_path: live directory path for the new cert
+
+    """
+    executed_dir_hooks = set()
+    if config.directory_hooks:
+        for hook in list_hooks(config.renewal_deploy_hooks_dir):
+            _run_deploy_hook(hook, domains, lineage_path, config.dry_run)
+            executed_dir_hooks.add(hook)
+
     if config.renew_hook:
-        if not config.dry_run:
-            os.environ["RENEWED_DOMAINS"] = " ".join(domains)
-            os.environ["RENEWED_LINEAGE"] = lineage_path
-            logger.info("Running deploy-hook command: %s", config.renew_hook)
-            _run_hook(config.renew_hook)
+        if config.renew_hook in executed_dir_hooks:
+            logger.info("Skipping deploy-hook '%s' as it was already run.",
+                        config.renew_hook)
         else:
-            logger.warning(
-                "Dry run: skipping deploy hook command: %s", config.renew_hook)
+            _run_deploy_hook(config.renew_hook, domains,
+                             lineage_path, config.dry_run)
+
+
+def _run_deploy_hook(command, domains, lineage_path, dry_run):
+    """Run the specified deploy-hook (if not doing a dry run).
+
+    If dry_run is True, command is not run and a message is logged
+    saying that it was skipped. If dry_run is False, the hook is run
+    after setting the appropriate environment variables.
+
+    :param str command: command to run as a deploy-hook
+    :param domains: domains in the obtained certificate
+    :type domains: `list` of `str`
+    :param str lineage_path: live directory path for the new cert
+    :param bool dry_run: True iff Certbot is doing a dry run
+
+    """
+    if dry_run:
+        logger.warning("Dry run: skipping deploy hook command: %s",
+                       command)
+        return
+
+    os.environ["RENEWED_DOMAINS"] = " ".join(domains)
+    os.environ["RENEWED_LINEAGE"] = lineage_path
+    logger.info("Running deploy-hook command: %s", command)
+    _run_hook(command)
 
 
 def _run_hook(shell_cmd):
@@ -151,3 +250,15 @@ def execute(shell_cmd):
         logger.error('Error output from %s:\n%s', base_cmd, err)
     return (err, out)
 
+
+def list_hooks(dir_path):
+    """List paths to all hooks found in dir_path in sorted order.
+
+    :param str dir_path: directory to search
+
+    :returns: `list` of `str`
+    :rtype: sorted list of absolute paths to executables in dir_path
+
+    """
+    paths = (os.path.join(dir_path, f) for f in os.listdir(dir_path))
+    return sorted(path for path in paths if util.is_exe(path))

certbot/log.py

@@ -359,11 +359,11 @@ def post_arg_parse_except_hook(exc_type, exc_value, trace, debug, log_path):
         logger.debug('Exiting abnormally:', exc_info=exc_info)
         if issubclass(exc_type, errors.Error):
             sys.exit(exc_value)
-        print('An unexpected error occurred:', file=sys.stderr)
+        logger.error('An unexpected error occurred:')
         if messages.is_acme_error(exc_value):
             # Remove the ACME error prefix from the exception
             _, _, exc_str = str(exc_value).partition(':: ')
-            print(exc_str, file=sys.stderr)
+            logger.error(exc_str)
         else:
             traceback.print_exception(exc_type, exc_value, None)
     exit_with_log_path(log_path)

certbot/main.py

@@ -1,9 +1,11 @@
 """Certbot main entry point."""
 from __future__ import print_function
+import functools
 import logging.handlers
 import os
 import sys
 
+import configobj
 import zope.component
 
 from acme import jose
@@ -25,6 +27,7 @@ from certbot import interfaces
 from certbot import log
 from certbot import renewal
 from certbot import reporter
+from certbot import storage
 from certbot import util
 
 from certbot.display import util as display_util, ops as display_ops
@@ -384,6 +387,92 @@ def _determine_account(config):
     return acc, acme
 
 
+def _delete_if_appropriate(config): # pylint: disable=too-many-locals,too-many-branches
+    """Does the user want to delete their now-revoked certs? If run in non-interactive mode,
+    deleting happens automatically, unless if both `--cert-name` and `--cert-path` were
+    specified with conflicting values.
+
+    :param `configuration.NamespaceConfig` config: parsed command line arguments
+
+    :raises `error.Errors`: If anything goes wrong, including bad user input, if an overlapping
+        archive dir is found for the specified lineage, etc ...
+    """
+    display = zope.component.getUtility(interfaces.IDisplay)
+    reporter_util = zope.component.getUtility(interfaces.IReporter)
+
+    msg = ("Would you like to delete the cert(s) you just revoked?")
+    attempt_deletion = display.yesno(msg, yes_label="Yes (recommended)", no_label="No",
+            force_interactive=True, default=True)
+
+    if not attempt_deletion:
+        reporter_util.add_message("Not deleting revoked certs.", reporter_util.LOW_PRIORITY)
+        return
+
+    if not (config.certname or config.cert_path):
+        raise errors.Error('At least one of --cert-path or --cert-name must be specified.')
+
+    if config.certname and config.cert_path:
+        # first, check if certname and cert_path imply the same certs
+        implied_cert_name = cert_manager.cert_path_to_lineage(config)
+
+        if implied_cert_name != config.certname:
+            cert_path_implied_cert_name = cert_manager.cert_path_to_lineage(config)
+            cert_path_implied_conf = storage.renewal_file_for_certname(config,
+                    cert_path_implied_cert_name)
+            cert_path_cert = storage.RenewableCert(cert_path_implied_conf, config)
+            cert_path_info = cert_manager.human_readable_cert_info(config, cert_path_cert,
+                    skip_filter_checks=True)
+
+            cert_name_implied_conf = storage.renewal_file_for_certname(config, config.certname)
+            cert_name_cert = storage.RenewableCert(cert_name_implied_conf, config)
+            cert_name_info = cert_manager.human_readable_cert_info(config, cert_name_cert)
+
+            msg = ("You specified conflicting values for --cert-path and --cert-name. "
+                    "Which did you mean to select?")
+            choices = [cert_path_info, cert_name_info]
+            try:
+                code, index = display.menu(msg,
+                        choices, ok_label="Select", force_interactive=True)
+            except errors.MissingCommandlineFlag:
+                error_msg = ('To run in non-interactive mode, you must either specify only one of '
+                '--cert-path or --cert-name, or both must point to the same certificate lineages.')
+                raise errors.Error(error_msg)
+
+            if code != display_util.OK or not index in range(0, len(choices)):
+                raise errors.Error("User ended interaction.")
+
+            if index == 0:
+                config.certname = cert_path_implied_cert_name
+            else:
+                config.cert_path = storage.cert_path_for_cert_name(config, config.certname)
+
+    elif config.cert_path:
+        config.certname = cert_manager.cert_path_to_lineage(config)
+
+    else: # if only config.certname was specified
+        config.cert_path = storage.cert_path_for_cert_name(config, config.certname)
+
+    # don't delete if the archive_dir is used by some other lineage
+    archive_dir = storage.full_archive_path(
+            configobj.ConfigObj(storage.renewal_file_for_certname(config, config.certname)),
+            config, config.certname)
+    try:
+        cert_manager.match_and_check_overlaps(config, [lambda x: archive_dir],
+            lambda x: x.archive_dir, lambda x: x)
+    except errors.OverlappingMatchFound:
+        msg = ('Not deleting revoked certs due to overlapping archive dirs. More than '
+                'one lineage is using {0}'.format(archive_dir))
+        reporter_util.add_message(''.join(msg), reporter_util.MEDIUM_PRIORITY)
+        return
+    except Exception as e:
+        msg = ('config.default_archive_dir: {0}, config.live_dir: {1}, archive_dir: {2},'
+        'original exception: {3}')
+        msg = msg.format(config.default_archive_dir, config.live_dir, archive_dir, e)
+        raise errors.Error(msg)
+
+    cert_manager.delete(config)
+
+
 def _init_le_client(config, authenticator, installer):
     if authenticator is not None:
         # if authenticator was given, then we will need account...
@@ -492,7 +581,7 @@ def install(config, plugins):
     _install_cert(config, le_client, domains)
 
 
-def plugins_cmd(config, plugins):  # TODO: Use IDisplay rather than print
+def plugins_cmd(config, plugins):
     """List server software plugins."""
     logger.debug("Expected interfaces: %s", config.ifaces)
 
@@ -500,8 +589,10 @@ def plugins_cmd(config, plugins):  # TODO: Use IDisplay rather than print
     filtered = plugins.visible().ifaces(ifaces)
     logger.debug("Filtered plugins: %r", filtered)
 
+    notify = functools.partial(zope.component.getUtility(
+        interfaces.IDisplay).notification, pause=False)
     if not config.init and not config.prepare:
-        print(str(filtered))
+        notify(str(filtered))
         return
 
     filtered.init(config)
@@ -509,13 +600,13 @@ def plugins_cmd(config, plugins):  # TODO: Use IDisplay rather than print
     logger.debug("Verified plugins: %r", verified)
 
     if not config.prepare:
-        print(str(verified))
+        notify(str(verified))
         return
 
     verified.prepare()
     available = verified.available()
     logger.debug("Prepared plugins: %s", available)
-    print(str(available))
+    notify(str(available))
 
 
 def rollback(config, plugins):
@@ -579,6 +670,7 @@ def revoke(config, unused_plugins):  # TODO: coop with renewal config
 
     try:
         acme.revoke(jose.ComparableX509(cert), config.reason)
+        _delete_if_appropriate(config)
     except acme_errors.ClientError as e:
         return str(e)
 
@@ -708,12 +800,20 @@ def renew(config, unused_plugins):
 
 
 def make_or_verify_needed_dirs(config):
-    """Create or verify existence of config and work directories"""
+    """Create or verify existence of config, work, and hook directories."""
     util.set_up_core_dir(config.config_dir, constants.CONFIG_DIRS_MODE,
                          os.geteuid(), config.strict_permissions)
     util.set_up_core_dir(config.work_dir, constants.CONFIG_DIRS_MODE,
                          os.geteuid(), config.strict_permissions)
 
+    hook_dirs = (config.renewal_pre_hooks_dir,
+                 config.renewal_deploy_hooks_dir,
+                 config.renewal_post_hooks_dir,)
+    for hook_dir in hook_dirs:
+        util.make_or_verify_dir(hook_dir,
+                                uid=os.geteuid(),
+                                strict=config.strict_permissions)
+
 
 def set_displayer(config):
     """Set the displayer"""
@@ -743,8 +843,14 @@ def main(cli_args=sys.argv[1:]):
     config = configuration.NamespaceConfig(args)
     zope.component.provideUtility(config)
 
-    log.post_arg_parse_setup(config)
-    make_or_verify_needed_dirs(config)
+    try:
+        log.post_arg_parse_setup(config)
+        make_or_verify_needed_dirs(config)
+    except errors.Error:
+        # Let plugins_cmd be run as un-privileged user.
+        if config.func != plugins_cmd:
+            raise
+
     set_displayer(config)
 
     # Reporter

certbot/plugins/common.py

@@ -13,7 +13,9 @@ from acme.jose import util as jose_util
 
 from certbot import constants
 from certbot import crypto_util
+from certbot import errors
 from certbot import interfaces
+from certbot import reverter
 from certbot import util
 
 logger = logging.getLogger(__name__)
@@ -100,6 +102,120 @@ class Plugin(object):
 # other
 
 
+class Installer(Plugin):
+    """An installer base class with reverter and ssl_dhparam methods defined.
+
+    Installer plugins do not have to inherit from this class.
+
+    """
+    def __init__(self, *args, **kwargs):
+        super(Installer, self).__init__(*args, **kwargs)
+        self.reverter = reverter.Reverter(self.config)
+
+    def add_to_checkpoint(self, save_files, save_notes, temporary=False):
+        """Add files to a checkpoint.
+
+        :param set save_files: set of filepaths to save
+        :param str save_notes: notes about changes during the save
+        :param bool temporary: True if the files should be added to a
+            temporary checkpoint rather than a permanent one. This is
+            usually used for changes that will soon be reverted.
+
+        :raises .errors.PluginError: when unable to add to checkpoint
+
+        """
+        if temporary:
+            checkpoint_func = self.reverter.add_to_temp_checkpoint
+        else:
+            checkpoint_func = self.reverter.add_to_checkpoint
+
+        try:
+            checkpoint_func(save_files, save_notes)
+        except errors.ReverterError as err:
+            raise errors.PluginError(str(err))
+
+    def finalize_checkpoint(self, title):
+        """Timestamp and save changes made through the reverter.
+
+        :param str title: Title describing checkpoint
+
+        :raises .errors.PluginError: when an error occurs
+
+        """
+        try:
+            self.reverter.finalize_checkpoint(title)
+        except errors.ReverterError as err:
+            raise errors.PluginError(str(err))
+
+    def recovery_routine(self):
+        """Revert all previously modified files.
+
+        Reverts all modified files that have not been saved as a checkpoint
+
+        :raises .errors.PluginError: If unable to recover the configuration
+
+        """
+        try:
+            self.reverter.recovery_routine()
+        except errors.ReverterError as err:
+            raise errors.PluginError(str(err))
+
+    def revert_temporary_config(self):
+        """Rollback temporary checkpoint.
+
+        :raises .errors.PluginError: when unable to revert config
+
+        """
+        try:
+            self.reverter.revert_temporary_config()
+        except errors.ReverterError as err:
+            raise errors.PluginError(str(err))
+
+    def rollback_checkpoints(self, rollback=1):
+        """Rollback saved checkpoints.
+
+        :param int rollback: Number of checkpoints to revert
+
+        :raises .errors.PluginError: If there is a problem with the input or
+            the function is unable to correctly revert the configuration
+
+        """
+        try:
+            self.reverter.rollback_checkpoints(rollback)
+        except errors.ReverterError as err:
+            raise errors.PluginError(str(err))
+
+    def view_config_changes(self):
+        """Show all of the configuration changes that have taken place.
+
+        :raises .errors.PluginError: If there is a problem while processing
+            the checkpoints directories.
+
+        """
+        try:
+            self.reverter.view_config_changes()
+        except errors.ReverterError as err:
+            raise errors.PluginError(str(err))
+
+    @property
+    def ssl_dhparams(self):
+        """Full absolute path to ssl_dhparams file."""
+        return os.path.join(self.config.config_dir, constants.SSL_DHPARAMS_DEST)
+
+    @property
+    def updated_ssl_dhparams_digest(self):
+        """Full absolute path to digest of updated ssl_dhparams file."""
+        return os.path.join(self.config.config_dir, constants.UPDATED_SSL_DHPARAMS_DIGEST)
+
+    def install_ssl_dhparams(self):
+        """Copy Certbot's ssl_dhparams file into the system's config dir if required."""
+        return install_version_controlled_file(
+            self.ssl_dhparams,
+            self.updated_ssl_dhparams_digest,
+            constants.SSL_DHPARAMS_SRC,
+            constants.ALL_SSL_DHPARAMS_HASHES)
+
+
 class Addr(object):
     r"""Represents an virtual host address.
 
@@ -135,7 +251,7 @@ class Addr(object):
         """Normalized representation of addr/port tuple
         """
         if self.ipv6:
-            return (self._normalize_ipv6(self.tup[0]), self.tup[1])
+            return (self.get_ipv6_exploded(), self.tup[1])
         return self.tup
 
     def __eq__(self, other):
@@ -270,51 +386,50 @@ class TLSSNI01(object):
         return response
 
 
-def install_ssl_options_conf(options_ssl, options_ssl_digest, mod_ssl_conf_src,
-    all_ssl_options_hashes):
-    """Copy Certbot's SSL options file into the system's config dir if required.
+def install_version_controlled_file(dest_path, digest_path, src_path, all_hashes):
+    """Copy a file into an active location (likely the system's config dir) if required.
 
-       :param str options_ssl: destination path for file containing ssl options
-       :param str options_ssl_digest: path to save a digest of options_ssl in
-       :param str mod_ssl_conf_src: path to file containing ssl options found in distribution
-       :param list all_ssl_options_hashes: hashes of every released version of options_ssl
+       :param str dest_path: destination path for version controlled file
+       :param str digest_path: path to save a digest of the file in
+       :param str src_path: path to version controlled file found in distribution
+       :param list all_hashes: hashes of every released version of the file
     """
-    current_ssl_options_hash = crypto_util.sha256sum(mod_ssl_conf_src)
+    current_hash = crypto_util.sha256sum(src_path)
 
     def _write_current_hash():
-        with open(options_ssl_digest, "w") as f:
-            f.write(current_ssl_options_hash)
+        with open(digest_path, "w") as f:
+            f.write(current_hash)
 
     def _install_current_file():
-        shutil.copyfile(mod_ssl_conf_src, options_ssl)
+        shutil.copyfile(src_path, dest_path)
         _write_current_hash()
 
     # Check to make sure options-ssl.conf is installed
-    if not os.path.isfile(options_ssl):
+    if not os.path.isfile(dest_path):
         _install_current_file()
         return
     # there's already a file there. if it's up to date, do nothing. if it's not but
     # it matches a known file hash, we can update it.
     # otherwise, print a warning once per new version.
-    active_file_digest = crypto_util.sha256sum(options_ssl)
-    if active_file_digest == current_ssl_options_hash: # already up to date
+    active_file_digest = crypto_util.sha256sum(dest_path)
+    if active_file_digest == current_hash: # already up to date
         return
-    elif active_file_digest in all_ssl_options_hashes: # safe to update
+    elif active_file_digest in all_hashes: # safe to update
         _install_current_file()
     else: # has been manually modified, not safe to update
         # did they modify the current version or an old version?
-        if os.path.isfile(options_ssl_digest):
-            with open(options_ssl_digest, "r") as f:
+        if os.path.isfile(digest_path):
+            with open(digest_path, "r") as f:
                 saved_digest = f.read()
             # they modified it after we either installed or told them about this version, so return
-            if saved_digest == current_ssl_options_hash:
+            if saved_digest == current_hash:
                 return
         # there's a new version but we couldn't update the file, or they deleted the digest.
         # save the current digest so we only print this once, and print a warning
         _write_current_hash()
-        logger.warning("%s has been manually modified; updated ssl configuration options "
+        logger.warning("%s has been manually modified; updated file "
             "saved to %s. We recommend updating %s for security purposes.",
-            options_ssl, mod_ssl_conf_src, options_ssl)
+            dest_path, src_path, dest_path)
 
 
 # test utils used by certbot_apache/certbot_nginx (hence

certbot/plugins/common_test.py

@@ -1,4 +1,5 @@
 """Tests for certbot.plugins.common."""
+import functools
 import os
 import shutil
 import tempfile
@@ -12,6 +13,7 @@ from acme import jose
 
 from certbot import achallenges
 from certbot import crypto_util
+from certbot import errors
 
 from certbot.tests import acme_util
 from certbot.tests import util as test_util
@@ -77,6 +79,107 @@ class PluginTest(unittest.TestCase):
             "--mock-foo-bar", dest="different_to_foo_bar", x=1, y=None)
 
 
+class InstallerTest(test_util.ConfigTestCase):
+    """Tests for certbot.plugins.common.Installer."""
+
+    def setUp(self):
+        super(InstallerTest, self).setUp()
+        os.mkdir(self.config.config_dir)
+        from certbot.plugins.common import Installer
+
+        with mock.patch("certbot.plugins.common.reverter.Reverter"):
+            self.installer = Installer(config=self.config,
+                                       name="Installer")
+        self.reverter = self.installer.reverter
+
+    def test_add_to_real_checkpoint(self):
+        files = set(("foo.bar", "baz.qux",))
+        save_notes = "foo bar baz qux"
+        self._test_wrapped_method("add_to_checkpoint", files, save_notes)
+
+    def test_add_to_real_checkpoint2(self):
+        self._test_add_to_checkpoint_common(False)
+
+    def test_add_to_temporary_checkpoint(self):
+        self._test_add_to_checkpoint_common(True)
+
+    def _test_add_to_checkpoint_common(self, temporary):
+        files = set(("foo.bar", "baz.qux",))
+        save_notes = "foo bar baz qux"
+
+        installer_func = functools.partial(self.installer.add_to_checkpoint,
+                                           temporary=temporary)
+
+        if temporary:
+            reverter_func = self.reverter.add_to_temp_checkpoint
+        else:
+            reverter_func = self.reverter.add_to_checkpoint
+
+        self._test_adapted_method(
+            installer_func, reverter_func, files, save_notes)
+
+    def test_finalize_checkpoint(self):
+        self._test_wrapped_method("finalize_checkpoint", "foo")
+
+    def test_recovery_routine(self):
+        self._test_wrapped_method("recovery_routine")
+
+    def test_revert_temporary_config(self):
+        self._test_wrapped_method("revert_temporary_config")
+
+    def test_rollback_checkpoints(self):
+        self._test_wrapped_method("rollback_checkpoints", 42)
+
+    def test_view_config_changes(self):
+        self._test_wrapped_method("view_config_changes")
+
+    def _test_wrapped_method(self, name, *args, **kwargs):
+        """Test a wrapped reverter method.
+
+        :param str name: name of the method to test
+        :param tuple args: position arguments to method
+        :param dict kwargs: keyword arguments to method
+
+        """
+        installer_func = getattr(self.installer, name)
+        reverter_func = getattr(self.reverter, name)
+        self._test_adapted_method(
+            installer_func, reverter_func, *args, **kwargs)
+
+    def _test_adapted_method(self, installer_func,
+                             reverter_func, *passed_args, **passed_kwargs):
+        """Test an adapted reverter method
+
+        :param callable installer_func: installer method to test
+        :param mock.MagicMock reverter_func: mocked adapated
+            reverter method
+        :param tuple passed_args: positional arguments passed from
+            installer method to the reverter method
+        :param dict passed_kargs: keyword arguments passed from
+            installer method to the reverter method
+
+        """
+        installer_func(*passed_args, **passed_kwargs)
+        reverter_func.assert_called_once_with(*passed_args, **passed_kwargs)
+        reverter_func.side_effect = errors.ReverterError
+        self.assertRaises(
+            errors.PluginError, installer_func, *passed_args, **passed_kwargs)
+
+    def test_install_ssl_dhparams(self):
+        self.installer.install_ssl_dhparams()
+        self.assertTrue(os.path.isfile(self.installer.ssl_dhparams))
+
+    def _current_ssl_dhparams_hash(self):
+        from certbot.constants import SSL_DHPARAMS_SRC
+        return crypto_util.sha256sum(SSL_DHPARAMS_SRC)
+
+    def test_current_file_hash_in_all_hashes(self):
+        from certbot.constants import ALL_SSL_DHPARAMS_HASHES
+        self.assertTrue(self._current_ssl_dhparams_hash() in ALL_SSL_DHPARAMS_HASHES,
+            "Constants.ALL_SSL_DHPARAMS_HASHES must be appended"
+            " with the sha256 hash of self.config.ssl_dhparams when it is updated.")
+
+
 class AddrTest(unittest.TestCase):
     """Tests for certbot.client.plugins.common.Addr."""
 
@@ -227,11 +330,11 @@ class TLSSNI01Test(unittest.TestCase):
             achall.response(achall.account_key).z_domain.decode("utf-8"))
 
 
-class InstallSslOptionsConfTest(test_util.TempDirTestCase):
-    """Tests for certbot.plugins.common.install_ssl_options_conf."""
+class InstallVersionControlledFileTest(test_util.TempDirTestCase):
+    """Tests for certbot.plugins.common.install_version_controlled_file."""
 
     def setUp(self):
-        super(InstallSslOptionsConfTest, self).setUp()
+        super(InstallVersionControlledFileTest, self).setUp()
         self.hashes = ["someotherhash"]
         self.dest_path = os.path.join(self.tempdir, "options-ssl-dest.conf")
         self.hash_path = os.path.join(self.tempdir, ".options-ssl-conf.txt")
@@ -243,19 +346,19 @@ class InstallSslOptionsConfTest(test_util.TempDirTestCase):
             self.hashes.append(crypto_util.sha256sum(path))
 
     def _call(self):
-        from certbot.plugins.common import install_ssl_options_conf
-        install_ssl_options_conf(self.dest_path,
-                                 self.hash_path,
-                                 self.source_path,
-                                 self.hashes)
+        from certbot.plugins.common import install_version_controlled_file
+        install_version_controlled_file(self.dest_path,
+                                        self.hash_path,
+                                        self.source_path,
+                                        self.hashes)
 
-    def _current_ssl_options_hash(self):
+    def _current_file_hash(self):
         return crypto_util.sha256sum(self.source_path)
 
     def _assert_current_file(self):
         self.assertTrue(os.path.isfile(self.dest_path))
         self.assertEqual(crypto_util.sha256sum(self.dest_path),
-            self._current_ssl_options_hash())
+            self._current_file_hash())
 
     def test_no_file(self):
         self.assertFalse(os.path.isfile(self.dest_path))
@@ -282,9 +385,9 @@ class InstallSslOptionsConfTest(test_util.TempDirTestCase):
             self.assertFalse(mock_logger.warning.called)
         self.assertTrue(os.path.isfile(self.dest_path))
         self.assertEqual(crypto_util.sha256sum(self.source_path),
-            self._current_ssl_options_hash())
+            self._current_file_hash())
         self.assertNotEqual(crypto_util.sha256sum(self.dest_path),
-            self._current_ssl_options_hash())
+            self._current_file_hash())
 
     def test_manually_modified_past_file_warns(self):
         with open(self.dest_path, "a") as mod_ssl_conf:
@@ -294,10 +397,10 @@ class InstallSslOptionsConfTest(test_util.TempDirTestCase):
         with mock.patch("certbot.plugins.common.logger") as mock_logger:
             self._call()
             self.assertEqual(mock_logger.warning.call_args[0][0],
-                "%s has been manually modified; updated ssl configuration options "
+                "%s has been manually modified; updated file "
                 "saved to %s. We recommend updating %s for security purposes.")
         self.assertEqual(crypto_util.sha256sum(self.source_path),
-            self._current_ssl_options_hash())
+            self._current_file_hash())
         # only print warning once
         with mock.patch("certbot.plugins.common.logger") as mock_logger:
             self._call()

certbot/plugins/null_test.py

@@ -1,5 +1,6 @@
 """Tests for certbot.plugins.null."""
 import unittest
+import six
 
 import mock
 
@@ -12,7 +13,7 @@ class InstallerTest(unittest.TestCase):
         self.installer = Installer(config=mock.MagicMock(), name="null")
 
     def test_it(self):
-        self.assertTrue(isinstance(self.installer.more_info(), str))
+        self.assertTrue(isinstance(self.installer.more_info(), six.string_types))
         self.assertEqual([], self.installer.get_all_names())
         self.assertEqual([], self.installer.supported_enhancements())
 

certbot/plugins/standalone_test.py

@@ -158,10 +158,11 @@ class AuthenticatorTest(unittest.TestCase):
 
     @test_util.patch_get_utility()
     def test_perform_eaddrinuse_retry(self, mock_get_utility):
+        mock_utility = mock_get_utility()
         errno = socket.errno.EADDRINUSE
         error = errors.StandaloneBindError(mock.MagicMock(errno=errno), -1)
         self.auth.servers.run.side_effect = [error] + 2 * [mock.MagicMock()]
-        mock_yesno = mock_get_utility.return_value.yesno
+        mock_yesno = mock_utility.yesno
         mock_yesno.return_value = True
 
         self.test_perform()
@@ -169,7 +170,8 @@ class AuthenticatorTest(unittest.TestCase):
 
     @test_util.patch_get_utility()
     def test_perform_eaddrinuse_no_retry(self, mock_get_utility):
-        mock_yesno = mock_get_utility.return_value.yesno
+        mock_utility = mock_get_utility()
+        mock_yesno = mock_utility.yesno
         mock_yesno.return_value = False
 
         errno = socket.errno.EADDRINUSE

certbot/plugins/webroot.py

@@ -102,10 +102,14 @@ to serve all files under specified web root ({0})."""
         webroot = None
 
         while webroot is None:
-            webroot = self._prompt_with_webroot_list(domain, known_webroots)
-
-            if webroot is None:
-                webroot = self._prompt_for_new_webroot(domain)
+            if known_webroots:
+                # Only show the menu if we have options for it
+                webroot = self._prompt_with_webroot_list(domain, known_webroots)
+                if webroot is None:
+                    webroot = self._prompt_for_new_webroot(domain)
+            else:
+                # Allow prompt to raise PluginError instead of looping forever
+                webroot = self._prompt_for_new_webroot(domain, True)
 
         return webroot
 
@@ -125,13 +129,18 @@ to serve all files under specified web root ({0})."""
             else:  # code == display_util.OK
                 return None if index == 0 else known_webroots[index - 1]
 
-    def _prompt_for_new_webroot(self, domain):
+    def _prompt_for_new_webroot(self, domain, allowraise=False):
         code, webroot = ops.validated_directory(
             _validate_webroot,
             "Input the webroot for {0}:".format(domain),
             force_interactive=True)
         if code == display_util.CANCEL:
-            return None
+            if not allowraise:
+                return None
+            else:
+                raise errors.PluginError(
+                    "Every requested domain must have a "
+                    "webroot when using the webroot plugin.")
         else:  # code == display_util.OK
             return _validate_webroot(webroot)
 

certbot/plugins/webroot_test.py

@@ -50,7 +50,7 @@ class AuthenticatorTest(unittest.TestCase):
 
     def test_more_info(self):
         more_info = self.auth.more_info()
-        self.assertTrue(isinstance(more_info, str))
+        self.assertTrue(isinstance(more_info, six.string_types))
         self.assertTrue(self.path in more_info)
 
     def test_add_parser_arguments(self):
@@ -96,7 +96,7 @@ class AuthenticatorTest(unittest.TestCase):
     @test_util.patch_get_utility()
     def test_new_webroot(self, mock_get_utility):
         self.config.webroot_path = []
-        self.config.webroot_map = {}
+        self.config.webroot_map = {"something.com": self.path}
 
         mock_display = mock_get_utility()
         mock_display.menu.return_value = (display_util.OK, 0,)
@@ -108,6 +108,19 @@ class AuthenticatorTest(unittest.TestCase):
 
         self.assertEqual(self.config.webroot_map[self.achall.domain], self.path)
 
+    @test_util.patch_get_utility()
+    def test_new_webroot_empty_map_cancel(self, mock_get_utility):
+        self.config.webroot_path = []
+        self.config.webroot_map = {}
+
+        mock_display = mock_get_utility()
+        mock_display.menu.return_value = (display_util.OK, 0,)
+        with mock.patch('certbot.display.ops.validated_directory') as m:
+            m.return_value = (display_util.CANCEL, -1)
+            self.assertRaises(errors.PluginError,
+                              self.auth.perform,
+                              [self.achall])
+
     def test_perform_missing_root(self):
         self.config.webroot_path = None
         self.config.webroot_map = {}
@@ -132,6 +145,22 @@ class AuthenticatorTest(unittest.TestCase):
         mock_chown.side_effect = OSError(errno.EACCES, "msg")
         self.auth.perform([self.achall])  # exception caught and logged
 
+
+    @test_util.patch_get_utility()
+    def test_perform_new_webroot_not_in_map(self, mock_get_utility):
+        new_webroot = tempfile.mkdtemp()
+        self.config.webroot_path = []
+        self.config.webroot_map = {"whatever.com": self.path}
+        mock_display = mock_get_utility()
+        mock_display.menu.side_effect = ((display_util.OK, 0),
+                                         (display_util.OK, new_webroot))
+        achall = achallenges.KeyAuthorizationAnnotatedChallenge(
+            challb=acme_util.HTTP01_P, domain="something.com", account_key=KEY)
+        with mock.patch('certbot.display.ops.validated_directory') as m:
+            m.return_value = (display_util.OK, new_webroot,)
+            self.auth.perform([achall])
+        self.assertEqual(self.config.webroot_map[achall.domain], new_webroot)
+
     def test_perform_permissions(self):
         self.auth.prepare()
 

certbot/renewal.py

@@ -108,7 +108,7 @@ def _restore_webroot_config(config, renewalparams):
     elif "webroot_path" in renewalparams:
         logger.debug("Ancient renewal conf file without webroot-map, restoring webroot-path")
         wp = renewalparams["webroot_path"]
-        if isinstance(wp, str):  # prior to 0.1.0, webroot_path was a string
+        if isinstance(wp, six.string_types):  # prior to 0.1.0, webroot_path was a string
             wp = [wp]
         config.webroot_path = wp
 
@@ -194,7 +194,7 @@ def _restore_pref_challs(unused_name, value):
     # If pref_challs has only one element, configobj saves the value
     # with a trailing comma so it's parsed as a list. If this comma is
     # removed by the user, the value is parsed as a str.
-    value = [value] if isinstance(value, str) else value
+    value = [value] if isinstance(value, six.string_types) else value
     return cli.parse_preferred_challenges(value)
 
 
@@ -320,6 +320,12 @@ def _renew_describe_results(config, renew_successes, renew_failures,
 
     out = []
     notify = out.append
+    disp = zope.component.getUtility(interfaces.IDisplay)
+
+    def notify_error(err):
+        """Notify and log errors."""
+        notify(err)
+        logger.error(err)
 
     if config.dry_run:
         notify("** DRY RUN: simulating 'certbot renew' close to cert expiry")
@@ -338,14 +344,14 @@ def _renew_describe_results(config, renew_successes, renew_failures,
                "have been renewed:")
         notify(report(renew_successes, "success"))
     elif renew_failures and not renew_successes:
-        notify("All renewal attempts failed. The following certs could not be "
-               "renewed:")
-        notify(report(renew_failures, "failure"))
+        notify_error("All renewal attempts failed. The following certs could "
+               "not be renewed:")
+        notify_error(report(renew_failures, "failure"))
     elif renew_failures and renew_successes:
         notify("The following certs were successfully renewed:")
-        notify(report(renew_successes, "success"))
-        notify("\nThe following certs could not be renewed:")
-        notify(report(renew_failures, "failure"))
+        notify(report(renew_successes, "success") + "\n")
+        notify_error("The following certs could not be renewed:")
+        notify_error(report(renew_failures, "failure"))
 
     if parse_failures:
         notify("\nAdditionally, the following renewal configuration files "
@@ -356,9 +362,7 @@ def _renew_describe_results(config, renew_successes, renew_failures,
         notify("** DRY RUN: simulating 'certbot renew' close to cert expiry")
         notify("**          (The test certificates above have not been saved.)")
 
-    if config.quiet and not (renew_failures or parse_failures):
-        return
-    print("\n".join(out))
+    disp.notification("\n".join(out), wrap=False)
 
 
 def handle_renewal_request(config):
@@ -372,8 +376,8 @@ def handle_renewal_request(config):
                            "renewing all installed certificates that are due "
                            "to be renewed or renewing a single certificate specified "
                            "by its name. If you would like to renew specific "
-                           "certificates by their domains, use the certonly "
-                           "command. The renew verb may provide other options "
+                           "certificates by their domains, use the certonly command "
+                           "instead. The renew verb may provide other options "
                            "for selecting certificates to renew in the future.")
 
     if config.certname:

certbot/ssl-dhparams.pem

@@ -0,0 +1,8 @@
+-----BEGIN DH PARAMETERS-----
+MIIBCAKCAQEA//////////+t+FRYortKmq/cViAnPTzx2LnFg84tNpWp4TZBFGQz
++8yTnc4kmz75fS/jY2MMddj2gbICrsRhetPfHtXV/WVhJDP1H18GbtCFY2VVPe0a
+87VXE15/V8k1mE8McODmi3fipona8+/och3xWKE2rec1MKzKT0g6eXq8CrGCsyT7
+YdEIqUuyyOP7uWrat2DX9GgdT0Kj3jlN9K5W7edjcrsZCwenyO4KbXCeAvzhzffi
+7MA0BM0oNC9hkXL+nOmFg/+OTxIy7vKBg8P+OxtMb61zO7X8vC7CIAXFjvGDfRaD
+ssbzSibBsu/6iGtCOGEoXJf//////////wIBAg==
+-----END DH PARAMETERS-----

certbot/storage.py

@@ -49,6 +49,21 @@ def renewal_file_for_certname(config, certname):
             "{1}).".format(certname, path))
     return path
 
+
+def cert_path_for_cert_name(config, cert_name):
+    """ If `--cert-name` was specified, but you need a value for `--cert-path`.
+
+    :param `configuration.NamespaceConfig` config: parsed command line arguments
+    :param str cert_name: cert name.
+
+    """
+    cert_name_implied_conf = renewal_file_for_certname(config, cert_name)
+    fullchain_path = configobj.ConfigObj(cert_name_implied_conf)["fullchain"]
+    with open(fullchain_path) as f:
+        cert_path = (fullchain_path, f.read())
+    return cert_path
+
+
 def config_with_defaults(config=None):
     """Merge supplied config, if provided, on top of builtin defaults."""
     defaults_copy = configobj.ConfigObj(constants.RENEWER_DEFAULTS)
@@ -246,7 +261,7 @@ def _relpath_from_file(archive_dir, from_file):
     """Path to a directory from a file"""
     return os.path.relpath(archive_dir, os.path.dirname(from_file))
 
-def _full_archive_path(config_obj, cli_config, lineagename):
+def full_archive_path(config_obj, cli_config, lineagename):
     """Returns the full archive path for a lineagename
 
     Uses cli_config to determine archive path if not available from config_obj.
@@ -271,7 +286,7 @@ def delete_files(config, certname):
     """
     renewal_filename = renewal_file_for_certname(config, certname)
     # file exists
-    full_default_archive_dir = _full_archive_path(None, config, certname)
+    full_default_archive_dir = full_archive_path(None, config, certname)
     full_default_live_dir = _full_live_path(config, certname)
     try:
         renewal_config = configobj.ConfigObj(renewal_filename)
@@ -323,7 +338,7 @@ def delete_files(config, certname):
 
     # archive directory
     try:
-        archive_path = _full_archive_path(renewal_config, config, certname)
+        archive_path = full_archive_path(renewal_config, config, certname)
         shutil.rmtree(archive_path)
         logger.debug("Removed %s", archive_path)
     except OSError:
@@ -450,7 +465,7 @@ class RenewableCert(object):
     @property
     def archive_dir(self):
         """Returns the default or specified archive directory"""
-        return _full_archive_path(self.configuration,
+        return full_archive_path(self.configuration,
             self.cli_config, self.lineagename)
 
     def relative_archive_dir(self, from_file):
@@ -992,7 +1007,7 @@ class RenewableCert(object):
         # lineagename will now potentially be modified based on which
         # renewal configuration file could actually be created
         lineagename = lineagename_for_filename(config_filename)
-        archive = _full_archive_path(None, cli_config, lineagename)
+        archive = full_archive_path(None, cli_config, lineagename)
         live_dir = _full_live_path(cli_config, lineagename)
         if os.path.exists(archive):
             raise errors.CertStorageError(

certbot/tests/cert_manager_test.py

@@ -1,3 +1,4 @@
+
 """Tests for certbot.cert_manager."""
 # pylint: disable=protected-access
 import os
@@ -107,16 +108,45 @@ class UpdateLiveSymlinksTest(BaseCertManagerTest):
 class DeleteTest(storage_test.BaseRenewableCertTest):
     """Tests for certbot.cert_manager.delete
     """
+
+    def _call(self):
+        from certbot import cert_manager
+        cert_manager.delete(self.config)
+
     @test_util.patch_get_utility()
     @mock.patch('certbot.cert_manager.lineage_for_certname')
     @mock.patch('certbot.storage.delete_files')
-    def test_delete(self, mock_delete_files, mock_lineage_for_certname, unused_get_utility):
+    def test_delete_from_config(self, mock_delete_files, mock_lineage_for_certname,
+        unused_get_utility):
         """Test delete"""
         mock_lineage_for_certname.return_value = self.test_rc
         self.config.certname = "example.org"
-        from certbot import cert_manager
-        cert_manager.delete(self.config)
-        self.assertTrue(mock_delete_files.called)
+        self._call()
+        mock_delete_files.assert_called_once_with(self.config, "example.org")
+
+    @test_util.patch_get_utility()
+    @mock.patch('certbot.cert_manager.lineage_for_certname')
+    @mock.patch('certbot.storage.delete_files')
+    def test_delete_interactive_single(self, mock_delete_files, mock_lineage_for_certname,
+        mock_util):
+        """Test delete"""
+        mock_lineage_for_certname.return_value = self.test_rc
+        mock_util().checklist.return_value = (display_util.OK, ["example.org"])
+        self._call()
+        mock_delete_files.assert_called_once_with(self.config, "example.org")
+
+    @test_util.patch_get_utility()
+    @mock.patch('certbot.cert_manager.lineage_for_certname')
+    @mock.patch('certbot.storage.delete_files')
+    def test_delete_interactive_multiple(self, mock_delete_files, mock_lineage_for_certname,
+        mock_util):
+        """Test delete"""
+        mock_lineage_for_certname.return_value = self.test_rc
+        mock_util().checklist.return_value = (display_util.OK, ["example.org", "other.org"])
+        self._call()
+        mock_delete_files.assert_any_call(self.config, "example.org")
+        mock_delete_files.assert_any_call(self.config, "other.org")
+        self.assertEqual(mock_delete_files.call_count, 2)
 
 
 class CertificatesTest(BaseCertManagerTest):
@@ -346,9 +376,8 @@ class RenameLineageTest(BaseCertManagerTest):
         self.assertRaises(errors.Error, self._call, self.config)
 
         mock_renewal_conf_files.return_value = ["one.conf"]
-        util_mock = mock.Mock()
+        util_mock = mock_get_utility()
         util_mock.menu.return_value = (display_util.CANCEL, 0)
-        mock_get_utility.return_value = util_mock
         self.assertRaises(errors.Error, self._call, self.config)
 
         util_mock.menu.return_value = (display_util.OK, -1)
@@ -359,14 +388,11 @@ class RenameLineageTest(BaseCertManagerTest):
         self.config.certname = "one"
         self.config.new_certname = None
 
-        util_mock = mock.Mock()
+        util_mock = mock_get_utility()
         util_mock.input.return_value = (display_util.CANCEL, "name")
-        mock_get_utility.return_value = util_mock
         self.assertRaises(errors.Error, self._call, self.config)
 
-        util_mock = mock.Mock()
         util_mock.input.return_value = (display_util.OK, None)
-        mock_get_utility.return_value = util_mock
         self.assertRaises(errors.Error, self._call, self.config)
 
     @test_util.patch_get_utility()
@@ -393,9 +419,8 @@ class RenameLineageTest(BaseCertManagerTest):
     def test_rename_cert_interactive_certname(self, mock_check, mock_get_utility):
         mock_check.return_value = True
         self.config.certname = None
-        util_mock = mock.Mock()
+        util_mock = mock_get_utility()
         util_mock.menu.return_value = (display_util.OK, 0)
-        mock_get_utility.return_value = util_mock
         self._call(self.config)
         from certbot import cert_manager
         updated_lineage = cert_manager.lineage_for_certname(self.config, self.config.new_certname)
@@ -453,5 +478,95 @@ class DuplicativeCertsTest(storage_test.BaseRenewableCertTest):
         self.assertEqual(result, (None, None))
 
 
+class CertPathToLineageTest(storage_test.BaseRenewableCertTest):
+    """Tests for certbot.cert_manager.cert_path_to_lineage"""
+
+    def setUp(self):
+        super(CertPathToLineageTest, self).setUp()
+        self.config_file.write()
+        self._write_out_ex_kinds()
+        self.fullchain = os.path.join(self.config.config_dir, 'live', 'example.org',
+                'fullchain.pem')
+        self.config.cert_path = (self.fullchain, '')
+
+    def _call(self, cli_config):
+        from certbot.cert_manager import cert_path_to_lineage
+        return cert_path_to_lineage(cli_config)
+
+    def _archive_files(self, cli_config, filetype):
+        from certbot.cert_manager import _archive_files
+        return _archive_files(cli_config, filetype)
+
+    def test_basic_match(self):
+        self.assertEqual('example.org', self._call(self.config))
+
+    def test_no_match_exists(self):
+        bad_test_config = self.config
+        bad_test_config.cert_path = os.path.join(self.config.config_dir, 'live',
+                'SailorMoon', 'fullchain.pem')
+        self.assertRaises(errors.Error, self._call, bad_test_config)
+
+    @mock.patch('certbot.cert_manager._acceptable_matches')
+    def test_options_fullchain(self, mock_acceptable_matches):
+        mock_acceptable_matches.return_value = [lambda x: x.fullchain_path]
+        self.config.fullchain_path = self.fullchain
+        self.assertEqual('example.org', self._call(self.config))
+
+    @mock.patch('certbot.cert_manager._acceptable_matches')
+    def test_options_cert_path(self, mock_acceptable_matches):
+        mock_acceptable_matches.return_value = [lambda x: x.cert_path]
+        test_cert_path = os.path.join(self.config.config_dir, 'live', 'example.org',
+                'cert.pem')
+        self.config.cert_path = (test_cert_path, '')
+        self.assertEqual('example.org', self._call(self.config))
+
+    @mock.patch('certbot.cert_manager._acceptable_matches')
+    def test_options_archive_cert(self, mock_acceptable_matches):
+        # Also this and the next test check that the regex of _archive_files is working.
+        self.config.cert_path = (os.path.join(self.config.config_dir, 'archive', 'example.org',
+            'cert11.pem'), '')
+        mock_acceptable_matches.return_value = [lambda x: self._archive_files(x, 'cert')]
+        self.assertEqual('example.org', self._call(self.config))
+
+    @mock.patch('certbot.cert_manager._acceptable_matches')
+    def test_options_archive_fullchain(self, mock_acceptable_matches):
+        self.config.cert_path = (os.path.join(self.config.config_dir, 'archive',
+            'example.org', 'fullchain11.pem'), '')
+        mock_acceptable_matches.return_value = [lambda x:
+                self._archive_files(x, 'fullchain')]
+        self.assertEqual('example.org', self._call(self.config))
+
+
+class MatchAndCheckOverlaps(storage_test.BaseRenewableCertTest):
+    """Tests for certbot.cert_manager.match_and_check_overlaps w/o overlapping archive dirs."""
+    # A test with real overlapping archive dirs can be found in tests/boulder_integration.sh
+    def setUp(self):
+        super(MatchAndCheckOverlaps, self).setUp()
+        self.config_file.write()
+        self._write_out_ex_kinds()
+        self.fullchain = os.path.join(self.config.config_dir, 'live', 'example.org',
+                'fullchain.pem')
+        self.config.cert_path = (self.fullchain, '')
+
+    def _call(self, cli_config, acceptable_matches, match_func, rv_func):
+        from certbot.cert_manager import match_and_check_overlaps
+        return match_and_check_overlaps(cli_config, acceptable_matches, match_func, rv_func)
+
+    def test_basic_match(self):
+        from certbot.cert_manager import _acceptable_matches
+        self.assertEqual(['example.org'], self._call(self.config, _acceptable_matches(),
+            lambda x: self.config.cert_path[0], lambda x: x.lineagename))
+
+    @mock.patch('certbot.cert_manager._search_lineages')
+    def test_no_matches(self, mock_search_lineages):
+        mock_search_lineages.return_value = []
+        self.assertRaises(errors.Error, self._call, self.config, None, None, None)
+
+    @mock.patch('certbot.cert_manager._search_lineages')
+    def test_too_many_matches(self, mock_search_lineages):
+        mock_search_lineages.return_value = ['spider', 'dance']
+        self.assertRaises(errors.OverlappingMatchFound, self._call, self.config, None, None, None)
+
+
 if __name__ == "__main__":
     unittest.main()  # pragma: no cover

certbot/tests/cli_test.py

@@ -3,6 +3,7 @@ import argparse
 import unittest
 import os
 import tempfile
+import copy
 
 import mock
 import six
@@ -15,6 +16,8 @@ from certbot import constants
 from certbot import errors
 from certbot.plugins import disco
 
+import certbot.tests.util as test_util
+
 from certbot.tests.util import TempDirTestCase
 
 PLUGINS = disco.PluginsRegistry.find_all()
@@ -49,24 +52,41 @@ class ParseTest(unittest.TestCase):  # pylint: disable=too-many-public-methods
         reload_module(cli)
 
     @staticmethod
-    def parse(*args, **kwargs):
+    def _unmocked_parse(*args, **kwargs):
         """Get result of cli.prepare_and_parse_args."""
         return cli.prepare_and_parse_args(PLUGINS, *args, **kwargs)
 
+    @staticmethod
+    def parse(*args, **kwargs):
+        """Mocks zope.component.getUtility and calls _unmocked_parse."""
+        with test_util.patch_get_utility():
+            return ParseTest._unmocked_parse(*args, **kwargs)
+
     def _help_output(self, args):
         "Run a command, and return the output string for scrutiny"
 
         output = six.StringIO()
+
+        def write_msg(message, *args, **kwargs): # pylint: disable=missing-docstring,unused-argument
+            output.write(message)
+
         with mock.patch('certbot.main.sys.stdout', new=output):
-            with mock.patch('certbot.main.sys.stderr'):
-                self.assertRaises(SystemExit, self.parse, args, output)
+            with test_util.patch_get_utility() as mock_get_utility:
+                mock_get_utility().notification.side_effect = write_msg
+                with mock.patch('certbot.main.sys.stderr'):
+                    self.assertRaises(SystemExit, self._unmocked_parse, args, output)
+
         return output.getvalue()
 
     @mock.patch("certbot.cli.flag_default")
     def test_cli_ini_domains(self, mock_flag_default):
         tmp_config = tempfile.NamedTemporaryFile()
         # use a shim to get ConfigArgParse to pick up tmp_config
-        shim = lambda v: constants.CLI_DEFAULTS[v] if v != "config_files" else [tmp_config.name]
+        shim = (
+                lambda v: copy.deepcopy(constants.CLI_DEFAULTS[v])
+                if v != "config_files"
+                else [tmp_config.name]
+                )
         mock_flag_default.side_effect = shim
 
         namespace = self.parse(["certonly"])
@@ -376,6 +396,24 @@ class ParseTest(unittest.TestCase):  # pylint: disable=too-many-public-methods
         namespace = self.parse(["--max-log-backups", value])
         self.assertEqual(namespace.max_log_backups, int(value))
 
+    def test_unchanging_defaults(self):
+        namespace = self.parse([])
+        self.assertEqual(namespace.domains, [])
+        self.assertEqual(namespace.pref_challs, [])
+
+        namespace.pref_challs = [challenges.HTTP01.typ]
+        namespace.domains = ['example.com']
+
+        namespace = self.parse([])
+        self.assertEqual(namespace.domains, [])
+        self.assertEqual(namespace.pref_challs, [])
+
+    def test_no_directory_hooks_set(self):
+        self.assertFalse(self.parse(["--no-directory-hooks"]).directory_hooks)
+
+    def test_no_directory_hooks_unset(self):
+        self.assertTrue(self.parse([]).directory_hooks)
+
 
 class DefaultTest(unittest.TestCase):
     """Tests for certbot.cli._Default."""
@@ -406,6 +444,10 @@ class SetByCliTest(unittest.TestCase):
     def setUp(self):
         reload_module(cli)
 
+    def test_deploy_hook(self):
+        self.assertTrue(_call_set_by_cli(
+            'renew_hook', '--deploy-hook foo'.split(), 'renew'))
+
     def test_webroot_map(self):
         args = '-w /var/www/html -d example.com'.split()
         verb = 'renew'
@@ -450,9 +492,10 @@ class SetByCliTest(unittest.TestCase):
 
 def _call_set_by_cli(var, args, verb):
     with mock.patch('certbot.cli.helpful_parser') as mock_parser:
-        mock_parser.args = args
-        mock_parser.verb = verb
-        return cli.set_by_cli(var)
+        with test_util.patch_get_utility():
+            mock_parser.args = args
+            mock_parser.verb = verb
+            return cli.set_by_cli(var)
 
 
 if __name__ == '__main__':

certbot/tests/client_test.py

@@ -28,6 +28,7 @@ class RegisterTest(test_util.ConfigTestCase):
         super(RegisterTest, self).setUp()
         self.config.rsa_key_size = 1024
         self.config.register_unsafely_without_email = False
+        self.config.email = "alias@example.com"
         self.account_storage = account.AccountMemoryStorage()
         self.tos_cb = mock.MagicMock()
 
@@ -75,6 +76,7 @@ class RegisterTest(test_util.ConfigTestCase):
     @mock.patch("certbot.account.report_new_account")
     def test_email_invalid_noninteractive(self, _rep):
         from acme import messages
+        self.config.noninteractive_mode = True
         msg = "DNS problem: NXDOMAIN looking up MX for example.com"
         mx_err = messages.Error.with_code('invalidContact', detail=msg)
         with mock.patch("certbot.client.acme_client.Client") as mock_client:

certbot/tests/configuration_test.py

@@ -4,6 +4,7 @@ import unittest
 
 import mock
 
+from certbot import constants
 from certbot import errors
 
 from certbot.tests import util as test_util
@@ -37,14 +38,14 @@ class NamespaceConfigTest(test_util.ConfigTestCase):
                          self.config.server_path.split(os.path.sep))
 
     @mock.patch('certbot.configuration.constants')
-    def test_dynamic_dirs(self, constants):
-        constants.ACCOUNTS_DIR = 'acc'
-        constants.BACKUP_DIR = 'backups'
-        constants.CSR_DIR = 'csr'
+    def test_dynamic_dirs(self, mock_constants):
+        mock_constants.ACCOUNTS_DIR = 'acc'
+        mock_constants.BACKUP_DIR = 'backups'
+        mock_constants.CSR_DIR = 'csr'
 
-        constants.IN_PROGRESS_DIR = '../p'
-        constants.KEY_DIR = 'keys'
-        constants.TEMP_CHECKPOINT_DIR = 't'
+        mock_constants.IN_PROGRESS_DIR = '../p'
+        mock_constants.KEY_DIR = 'keys'
+        mock_constants.TEMP_CHECKPOINT_DIR = 't'
 
         self.assertEqual(
             self.config.accounts_dir, os.path.join(
@@ -95,10 +96,10 @@ class NamespaceConfigTest(test_util.ConfigTestCase):
         self.assertTrue(os.path.isabs(config.temp_checkpoint_dir))
 
     @mock.patch('certbot.configuration.constants')
-    def test_renewal_dynamic_dirs(self, constants):
-        constants.ARCHIVE_DIR = 'a'
-        constants.LIVE_DIR = 'l'
-        constants.RENEWAL_CONFIGS_DIR = 'renewal_configs'
+    def test_renewal_dynamic_dirs(self, mock_constants):
+        mock_constants.ARCHIVE_DIR = 'a'
+        mock_constants.LIVE_DIR = 'l'
+        mock_constants.RENEWAL_CONFIGS_DIR = 'renewal_configs'
 
         self.assertEqual(
                 self.config.default_archive_dir, os.path.join(self.config.config_dir, 'a'))
@@ -134,6 +135,20 @@ class NamespaceConfigTest(test_util.ConfigTestCase):
         self.config.namespace.bar = 1337
         self.assertEqual(self.config.bar, 1337)
 
+    def test_hook_directories(self):
+        self.assertEqual(self.config.renewal_hooks_dir,
+                         os.path.join(self.config.config_dir,
+                                      constants.RENEWAL_HOOKS_DIR))
+        self.assertEqual(self.config.renewal_pre_hooks_dir,
+                         os.path.join(self.config.renewal_hooks_dir,
+                                      constants.RENEWAL_PRE_HOOKS_DIR))
+        self.assertEqual(self.config.renewal_deploy_hooks_dir,
+                         os.path.join(self.config.renewal_hooks_dir,
+                                      constants.RENEWAL_DEPLOY_HOOKS_DIR))
+        self.assertEqual(self.config.renewal_post_hooks_dir,
+                         os.path.join(self.config.renewal_hooks_dir,
+                                      constants.RENEWAL_POST_HOOKS_DIR))
+
 
 if __name__ == '__main__':
     unittest.main()  # pragma: no cover

certbot/tests/display/ops_test.py

@@ -310,10 +310,11 @@ class ChooseNamesTest(unittest.TestCase):
     @test_util.patch_get_utility("certbot.display.ops.z_util")
     def test_choose_manually(self, mock_util):
         from certbot.display.ops import _choose_names_manually
+        utility_mock = mock_util()
         # No retry
-        mock_util().yesno.return_value = False
+        utility_mock.yesno.return_value = False
         # IDN and no retry
-        mock_util().input.return_value = (display_util.OK,
+        utility_mock.input.return_value = (display_util.OK,
                                           "uniçodé.com")
         self.assertEqual(_choose_names_manually(), [])
         # IDN exception with previous mocks
@@ -324,7 +325,7 @@ class ChooseNamesTest(unittest.TestCase):
             mock_sli.side_effect = unicode_error
             self.assertEqual(_choose_names_manually(), [])
         # Valid domains
-        mock_util().input.return_value = (display_util.OK,
+        utility_mock.input.return_value = (display_util.OK,
                                           ("example.com,"
                                            "under_score.example.com,"
                                            "justtld,"
@@ -332,14 +333,17 @@ class ChooseNamesTest(unittest.TestCase):
         self.assertEqual(_choose_names_manually(),
                          ["example.com", "under_score.example.com",
                           "justtld", "valid.example.com"])
+
+    @test_util.patch_get_utility("certbot.display.ops.z_util")
+    def test_choose_manually_retry(self, mock_util):
+        from certbot.display.ops import _choose_names_manually
+        utility_mock = mock_util()
         # Three iterations
-        mock_util().input.return_value = (display_util.OK,
+        utility_mock.input.return_value = (display_util.OK,
                                           "uniçodé.com")
-        yn = mock.MagicMock()
-        yn.side_effect = [True, True, False]
-        mock_util().yesno = yn
+        utility_mock.yesno.side_effect = [True, True, False]
         _choose_names_manually()
-        self.assertEqual(mock_util().yesno.call_count, 3)
+        self.assertEqual(utility_mock.yesno.call_count, 3)
 
 
 class SuccessInstallationTest(unittest.TestCase):

certbot/tests/hook_test.py

@@ -1,136 +1,488 @@
-"""Tests for hooks.py"""
-# pylint: disable=protected-access
-
+"""Tests for certbot.hooks."""
 import os
+import stat
 import unittest
 
 import mock
-from six.moves import reload_module  # pylint: disable=import-error
 
 from certbot import errors
-from certbot import hooks
+from certbot.tests import util
 
-class HookTest(unittest.TestCase):
-    def setUp(self):
-        reload_module(hooks)
 
-    @mock.patch('certbot.hooks._prog')
-    def test_validate_hooks(self, mock_prog):
-        config = mock.MagicMock(deploy_hook=None, pre_hook="",
-                                post_hook="ls -lR", renew_hook="uptime")
-        hooks.validate_hooks(config)
-        self.assertEqual(mock_prog.call_count, 2)
-        self.assertEqual(mock_prog.call_args_list[1][0][0], 'uptime')
-        self.assertEqual(mock_prog.call_args_list[0][0][0], 'ls')
-        mock_prog.return_value = None
-        config = mock.MagicMock(pre_hook="explodinator", post_hook="", renew_hook="")
-        self.assertRaises(errors.HookCommandNotFound, hooks.validate_hooks, config)
+class ValidateHooksTest(unittest.TestCase):
+    """Tests for certbot.hooks.validate_hooks."""
 
-    @mock.patch('certbot.hooks.validate_hook')
-    def test_validation_order(self, mock_validate_hook):
-        # This ensures error messages are about deploy hook when appropriate
-        config = mock.Mock(deploy_hook=None, pre_hook=None,
-                           post_hook=None, renew_hook=None)
-        hooks.validate_hooks(config)
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.hooks import validate_hooks
+        return validate_hooks(*args, **kwargs)
 
-        order = [call[0][1] for call in mock_validate_hook.call_args_list]
-        self.assertTrue('pre' in order)
-        self.assertTrue('post' in order)
-        self.assertTrue('deploy' in order)
-        self.assertEqual(order[-1], 'renew')
+    @mock.patch("certbot.hooks.validate_hook")
+    def test_it(self, mock_validate_hook):
+        config = mock.MagicMock()
+        self._call(config)
 
-    @mock.patch('certbot.hooks.util.exe_exists')
-    @mock.patch('certbot.hooks.plug_util.path_surgery')
-    def test_prog(self, mock_ps, mock_exe_exists):
-        mock_exe_exists.return_value = True
-        self.assertEqual(hooks._prog("funky"), "funky")
-        self.assertEqual(mock_ps.call_count, 0)
+        types = [call[0][1] for call in mock_validate_hook.call_args_list]
+        self.assertEqual(set(("pre", "post", "deploy",)), set(types[:-1]))
+        # This ensures error messages are about deploy hooks when appropriate
+        self.assertEqual("renew", types[-1])
+
+
+class ValidateHookTest(util.TempDirTestCase):
+    """Tests for certbot.hooks.validate_hook."""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.hooks import validate_hook
+        return validate_hook(*args, **kwargs)
+
+    def test_not_executable(self):
+        file_path = os.path.join(self.tempdir, "foo")
+        # create a non-executable file
+        os.close(os.open(file_path, os.O_CREAT | os.O_WRONLY, 0o666))
+        # prevent unnecessary modifications to PATH
+        with mock.patch("certbot.hooks.plug_util.path_surgery"):
+            self.assertRaises(errors.HookCommandNotFound,
+                              self._call, file_path, "foo")
+
+    @mock.patch("certbot.hooks.util.exe_exists")
+    def test_not_found(self, mock_exe_exists):
         mock_exe_exists.return_value = False
-        self.assertEqual(hooks._prog("funky"), None)
-        self.assertEqual(mock_ps.call_count, 1)
+        with mock.patch("certbot.hooks.plug_util.path_surgery") as mock_ps:
+            self.assertRaises(errors.HookCommandNotFound,
+                              self._call, "foo", "bar")
+        self.assertTrue(mock_ps.called)
 
-    @mock.patch('certbot.hooks.renew_hook')
-    def test_deploy_hook(self, mock_renew_hook):
-        args = (mock.Mock(deploy_hook='foo'), ['example.org'], 'path',)
-        # pylint: disable=star-args
-        hooks.deploy_hook(*args)
-        mock_renew_hook.assert_called_once_with(*args)
+    @mock.patch("certbot.hooks._prog")
+    def test_unset(self, mock_prog):
+        self._call(None, "foo")
+        self.assertFalse(mock_prog.called)
 
-    @mock.patch('certbot.hooks.renew_hook')
-    def test_no_deploy_hook(self, mock_renew_hook):
-        args = (mock.Mock(deploy_hook=None), ['example.org'], 'path',)
-        hooks.deploy_hook(*args)  # pylint: disable=star-args
-        mock_renew_hook.assert_not_called()
 
-    def _test_a_hook(self, config, hook_function, calls_expected, **kwargs):
-        with mock.patch('certbot.hooks.logger') as mock_logger:
-            mock_logger.warning = mock.MagicMock()
-            with mock.patch('certbot.hooks._run_hook') as mock_run_hook:
-                hook_function(config, **kwargs)
-                hook_function(config, **kwargs)
-                self.assertEqual(mock_run_hook.call_count, calls_expected)
-            return mock_logger.warning
+class HookTest(util.ConfigTestCase):
+    """Common base class for hook tests."""
 
-    def test_pre_hook(self):
-        config = mock.MagicMock(pre_hook="true")
-        self._test_a_hook(config, hooks.pre_hook, 1)
-        self._test_a_hook(config, hooks.pre_hook, 0)
-        config = mock.MagicMock(pre_hook="more_true")
-        self._test_a_hook(config, hooks.pre_hook, 1)
-        self._test_a_hook(config, hooks.pre_hook, 0)
-        config = mock.MagicMock(pre_hook="")
-        self._test_a_hook(config, hooks.pre_hook, 0)
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        """Calls the method being tested with the given arguments."""
+        raise NotImplementedError
 
-    def _test_renew_post_hooks(self, expected_count):
-        with mock.patch('certbot.hooks.logger.info') as mock_info:
-            with mock.patch('certbot.hooks._run_hook') as mock_run:
-                hooks.run_saved_post_hooks()
-                self.assertEqual(mock_run.call_count, expected_count)
-                self.assertEqual(mock_info.call_count, expected_count)
+    @classmethod
+    def _call_with_mock_execute(cls, *args, **kwargs):
+        """Calls self._call after mocking out certbot.hooks.execute.
 
-    def test_post_hooks(self):
-        config = mock.MagicMock(post_hook="true", verb="splonk")
-        self._test_a_hook(config, hooks.post_hook, 2)
-        self._test_renew_post_hooks(0)
+        The mock execute object is returned rather than the return value
+        of self._call.
 
-        config = mock.MagicMock(post_hook="true", verb="renew")
-        self._test_a_hook(config, hooks.post_hook, 0)
-        self._test_renew_post_hooks(1)
-        self._test_a_hook(config, hooks.post_hook, 0)
-        self._test_renew_post_hooks(1)
+        """
+        with mock.patch("certbot.hooks.execute") as mock_execute:
+            mock_execute.return_value = ("", "")
+            cls._call(*args, **kwargs)
+        return mock_execute
 
-        config = mock.MagicMock(post_hook="more_true", verb="renew")
-        self._test_a_hook(config, hooks.post_hook, 0)
-        self._test_renew_post_hooks(2)
 
-    def test_renew_hook(self):
-        with mock.patch.dict('os.environ', {}):
-            domains = ["a", "b"]
-            lineage = "thing"
-            rhook = lambda x: hooks.renew_hook(x, domains, lineage)
+class PreHookTest(HookTest):
+    """Tests for certbot.hooks.pre_hook."""
 
-            config = mock.MagicMock(renew_hook="true", dry_run=False)
-            self._test_a_hook(config, rhook, 2)
-            self.assertEqual(os.environ["RENEWED_DOMAINS"], "a b")
-            self.assertEqual(os.environ["RENEWED_LINEAGE"], "thing")
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.hooks import pre_hook
+        return pre_hook(*args, **kwargs)
 
-            config = mock.MagicMock(renew_hook="true", dry_run=True)
-            mock_warn = self._test_a_hook(config, rhook, 0)
-            self.assertEqual(mock_warn.call_count, 2)
+    def setUp(self):
+        super(PreHookTest, self).setUp()
+        self.config.pre_hook = "foo"
 
-    @mock.patch('certbot.hooks.Popen')
-    def test_run_hook(self, mock_popen):
-        with mock.patch('certbot.hooks.logger.error') as mock_error:
-            mock_cmd = mock.MagicMock()
-            mock_cmd.returncode = 1
-            mock_cmd.communicate.return_value = ("", "")
-            mock_popen.return_value = mock_cmd
-            hooks._run_hook("ls")
-            self.assertEqual(mock_error.call_count, 1)
-        with mock.patch('certbot.hooks.logger.error') as mock_error:
-            mock_cmd.communicate.return_value = ("", "thing")
-            hooks._run_hook("ls")
-            self.assertEqual(mock_error.call_count, 2)
+        os.makedirs(self.config.renewal_pre_hooks_dir)
+        self.dir_hook = os.path.join(self.config.renewal_pre_hooks_dir, "bar")
+        create_hook(self.dir_hook)
+
+        # Reset this value as it may have been modified by past tests
+        self._reset_pre_hook_already()
+
+    def tearDown(self):
+        # Reset this value so it's unmodified for future tests
+        self._reset_pre_hook_already()
+        super(PreHookTest, self).tearDown()
+
+    def _reset_pre_hook_already(self):
+        from certbot.hooks import pre_hook
+        pre_hook.already.clear()
+
+    def test_certonly(self):
+        self.config.verb = "certonly"
+        self._test_nonrenew_common()
+
+    def test_run(self):
+        self.config.verb = "run"
+        self._test_nonrenew_common()
+
+    def _test_nonrenew_common(self):
+        mock_execute = self._call_with_mock_execute(self.config)
+        mock_execute.assert_called_once_with(self.config.pre_hook)
+        self._test_no_executions_common()
+
+    def test_no_hooks(self):
+        self.config.pre_hook = None
+        self.config.verb = "renew"
+        os.remove(self.dir_hook)
+
+        with mock.patch("certbot.hooks.logger") as mock_logger:
+            mock_execute = self._call_with_mock_execute(self.config)
+        self.assertFalse(mock_execute.called)
+        self.assertFalse(mock_logger.info.called)
+
+    def test_renew_disabled_dir_hooks(self):
+        self.config.directory_hooks = False
+        mock_execute = self._call_with_mock_execute(self.config)
+        mock_execute.assert_called_once_with(self.config.pre_hook)
+        self._test_no_executions_common()
+
+    def test_renew_no_overlap(self):
+        self.config.verb = "renew"
+        mock_execute = self._call_with_mock_execute(self.config)
+        mock_execute.assert_any_call(self.dir_hook)
+        mock_execute.assert_called_with(self.config.pre_hook)
+        self._test_no_executions_common()
+
+    def test_renew_with_overlap(self):
+        self.config.pre_hook = self.dir_hook
+        self.config.verb = "renew"
+        mock_execute = self._call_with_mock_execute(self.config)
+        mock_execute.assert_called_once_with(self.dir_hook)
+        self._test_no_executions_common()
+
+    def _test_no_executions_common(self):
+        with mock.patch("certbot.hooks.logger") as mock_logger:
+            mock_execute = self._call_with_mock_execute(self.config)
+        self.assertFalse(mock_execute.called)
+        self.assertTrue(mock_logger.info.called)
+
+
+class PostHookTest(HookTest):
+    """Tests for certbot.hooks.post_hook."""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.hooks import post_hook
+        return post_hook(*args, **kwargs)
+
+    def setUp(self):
+        super(PostHookTest, self).setUp()
+
+        self.config.post_hook = "bar"
+        os.makedirs(self.config.renewal_post_hooks_dir)
+        self.dir_hook = os.path.join(self.config.renewal_post_hooks_dir, "foo")
+        create_hook(self.dir_hook)
+
+        # Reset this value as it may have been modified by past tests
+        self._reset_post_hook_eventually()
+
+    def tearDown(self):
+        # Reset this value so it's unmodified for future tests
+        self._reset_post_hook_eventually()
+        super(PostHookTest, self).tearDown()
+
+    def _reset_post_hook_eventually(self):
+        from certbot.hooks import post_hook
+        post_hook.eventually = []
+
+    def test_certonly_and_run_with_hook(self):
+        for verb in ("certonly", "run",):
+            self.config.verb = verb
+            mock_execute = self._call_with_mock_execute(self.config)
+            mock_execute.assert_called_once_with(self.config.post_hook)
+            self.assertFalse(self._get_eventually())
+
+    def test_cert_only_and_run_without_hook(self):
+        self.config.post_hook = None
+        for verb in ("certonly", "run",):
+            self.config.verb = verb
+            self.assertFalse(self._call_with_mock_execute(self.config).called)
+            self.assertFalse(self._get_eventually())
+
+    def test_renew_disabled_dir_hooks(self):
+        self.config.directory_hooks = False
+        self._test_renew_common([self.config.post_hook])
+
+    def test_renew_no_config_hook(self):
+        self.config.post_hook = None
+        self._test_renew_common([self.dir_hook])
+
+    def test_renew_no_dir_hook(self):
+        os.remove(self.dir_hook)
+        self._test_renew_common([self.config.post_hook])
+
+    def test_renew_no_hooks(self):
+        self.config.post_hook = None
+        os.remove(self.dir_hook)
+        self._test_renew_common([])
+
+    def test_renew_no_overlap(self):
+        expected = [self.dir_hook, self.config.post_hook]
+        self._test_renew_common(expected)
+
+        self.config.post_hook = "baz"
+        expected.append(self.config.post_hook)
+        self._test_renew_common(expected)
+
+    def test_renew_with_overlap(self):
+        self.config.post_hook = self.dir_hook
+        self._test_renew_common([self.dir_hook])
+
+    def _test_renew_common(self, expected):
+        self.config.verb = "renew"
+
+        for _ in range(2):
+            self._call(self.config)
+            self.assertEqual(self._get_eventually(), expected)
+
+    def _get_eventually(self):
+        from certbot.hooks import post_hook
+        return post_hook.eventually
+
+
+class RunSavedPostHooksTest(HookTest):
+    """Tests for certbot.hooks.run_saved_post_hooks."""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.hooks import run_saved_post_hooks
+        return run_saved_post_hooks(*args, **kwargs)
+
+    def _call_with_mock_execute_and_eventually(self, *args, **kwargs):
+        """Call run_saved_post_hooks but mock out execute and eventually
+
+        certbot.hooks.post_hook.eventually is replaced with
+        self.eventually. The mock execute object is returned rather than
+        the return value of run_saved_post_hooks.
+
+        """
+        eventually_path = "certbot.hooks.post_hook.eventually"
+        with mock.patch(eventually_path, new=self.eventually):
+            return self._call_with_mock_execute(*args, **kwargs)
+
+    def setUp(self):
+        super(RunSavedPostHooksTest, self).setUp()
+        self.eventually = []
+
+    def test_empty(self):
+        self.assertFalse(self._call_with_mock_execute_and_eventually().called)
+
+    def test_multiple(self):
+        self.eventually = ["foo", "bar", "baz", "qux"]
+        mock_execute = self._call_with_mock_execute_and_eventually()
+
+        calls = mock_execute.call_args_list
+        for actual_call, expected_arg in zip(calls, self.eventually):
+            self.assertEqual(actual_call[0][0], expected_arg)
+
+    def test_single(self):
+        self.eventually = ["foo"]
+        mock_execute = self._call_with_mock_execute_and_eventually()
+        mock_execute.assert_called_once_with(self.eventually[0])
+
+
+class RenewalHookTest(HookTest):
+    """Common base class for testing deploy/renew hooks."""
+    # Needed for https://github.com/PyCQA/pylint/issues/179
+    # pylint: disable=abstract-method
+
+    def _call_with_mock_execute(self, *args, **kwargs):
+        """Calls self._call after mocking out certbot.hooks.execute.
+
+        The mock execute object is returned rather than the return value
+        of self._call. The mock execute object asserts that environment
+        variables were properly set.
+
+        """
+        domains = kwargs["domains"] if "domains" in kwargs else args[1]
+        lineage = kwargs["lineage"] if "lineage" in kwargs else args[2]
+
+        def execute_side_effect(*unused_args, **unused_kwargs):
+            """Assert environment variables are properly set.
+
+            :returns: two strings imitating no output from the hook
+            :rtype: `tuple` of `str`
+
+            """
+            self.assertEqual(os.environ["RENEWED_DOMAINS"], " ".join(domains))
+            self.assertEqual(os.environ["RENEWED_LINEAGE"], lineage)
+            return ("", "")
+
+        with mock.patch("certbot.hooks.execute") as mock_execute:
+            mock_execute.side_effect = execute_side_effect
+            self._call(*args, **kwargs)
+        return mock_execute
+
+    def setUp(self):
+        super(RenewalHookTest, self).setUp()
+        self.vars_to_clear = set(
+            var for var in ("RENEWED_DOMAINS", "RENEWED_LINEAGE",)
+            if var not in os.environ)
+
+    def tearDown(self):
+        for var in self.vars_to_clear:
+            os.environ.pop(var, None)
+        super(RenewalHookTest, self).tearDown()
+
+
+class DeployHookTest(RenewalHookTest):
+    """Tests for certbot.hooks.deploy_hook."""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.hooks import deploy_hook
+        return deploy_hook(*args, **kwargs)
+
+    @mock.patch("certbot.hooks.logger")
+    def test_dry_run(self, mock_logger):
+        self.config.deploy_hook = "foo"
+        self.config.dry_run = True
+        mock_execute = self._call_with_mock_execute(
+            self.config, ["example.org"], "/foo/bar")
+        self.assertFalse(mock_execute.called)
+        self.assertTrue(mock_logger.warning.called)
+
+    @mock.patch("certbot.hooks.logger")
+    def test_no_hook(self, mock_logger):
+        self.config.deploy_hook = None
+        mock_execute = self._call_with_mock_execute(
+            self.config, ["example.org"], "/foo/bar")
+        self.assertFalse(mock_execute.called)
+        self.assertFalse(mock_logger.info.called)
+
+    def test_success(self):
+        domains = ["example.org", "example.net"]
+        lineage = "/foo/bar"
+        self.config.deploy_hook = "foo"
+        mock_execute = self._call_with_mock_execute(
+            self.config, domains, lineage)
+        mock_execute.assert_called_once_with(self.config.deploy_hook)
+
+
+class RenewHookTest(RenewalHookTest):
+    """Tests for certbot.hooks.renew_hook"""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.hooks import renew_hook
+        return renew_hook(*args, **kwargs)
+
+    def setUp(self):
+        super(RenewHookTest, self).setUp()
+        self.config.renew_hook = "foo"
+
+        os.makedirs(self.config.renewal_deploy_hooks_dir)
+        self.dir_hook = os.path.join(self.config.renewal_deploy_hooks_dir,
+                                     "bar")
+        create_hook(self.dir_hook)
+
+    def test_disabled_dir_hooks(self):
+        self.config.directory_hooks = False
+        mock_execute = self._call_with_mock_execute(
+            self.config, ["example.org"], "/foo/bar")
+        mock_execute.assert_called_once_with(self.config.renew_hook)
+
+    @mock.patch("certbot.hooks.logger")
+    def test_dry_run(self, mock_logger):
+        self.config.dry_run = True
+        mock_execute = self._call_with_mock_execute(
+            self.config, ["example.org"], "/foo/bar")
+        self.assertFalse(mock_execute.called)
+        self.assertEqual(mock_logger.warning.call_count, 2)
+
+    def test_no_hooks(self):
+        self.config.renew_hook = None
+        os.remove(self.dir_hook)
+
+        with mock.patch("certbot.hooks.logger") as mock_logger:
+            mock_execute = self._call_with_mock_execute(
+                self.config, ["example.org"], "/foo/bar")
+        self.assertFalse(mock_execute.called)
+        self.assertFalse(mock_logger.info.called)
+
+    def test_overlap(self):
+        self.config.renew_hook = self.dir_hook
+        mock_execute = self._call_with_mock_execute(
+            self.config, ["example.net", "example.org"], "/foo/bar")
+        mock_execute.assert_called_once_with(self.dir_hook)
+
+    def test_no_overlap(self):
+        mock_execute = self._call_with_mock_execute(
+            self.config, ["example.org"], "/foo/bar")
+        mock_execute.assert_any_call(self.dir_hook)
+        mock_execute.assert_called_with(self.config.renew_hook)
+
+
+class ExecuteTest(unittest.TestCase):
+    """Tests for certbot.hooks.execute."""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.hooks import execute
+        return execute(*args, **kwargs)
+
+    def test_it(self):
+        for returncode in range(0, 2):
+            for stdout in ("", "Hello World!",):
+                for stderr in ("", "Goodbye Cruel World!"):
+                    self._test_common(returncode, stdout, stderr)
+
+    def _test_common(self, returncode, stdout, stderr):
+        given_command = "foo"
+        with mock.patch("certbot.hooks.Popen") as mock_popen:
+            mock_popen.return_value.communicate.return_value = (stdout, stderr)
+            mock_popen.return_value.returncode = returncode
+            with mock.patch("certbot.hooks.logger") as mock_logger:
+                self.assertEqual(self._call(given_command), (stderr, stdout))
+
+        executed_command = mock_popen.call_args[1].get(
+            "args", mock_popen.call_args[0][0])
+        self.assertEqual(executed_command, given_command)
+
+        if stdout:
+            self.assertTrue(mock_logger.info.called)
+        if stderr or returncode:
+            self.assertTrue(mock_logger.error.called)
+
+
+class ListHooksTest(util.TempDirTestCase):
+    """Tests for certbot.hooks.list_hooks."""
+
+    @classmethod
+    def _call(cls, *args, **kwargs):
+        from certbot.hooks import list_hooks
+        return list_hooks(*args, **kwargs)
+
+    def test_empty(self):
+        self.assertFalse(self._call(self.tempdir))
+
+    def test_multiple(self):
+        names = sorted(
+            os.path.join(self.tempdir, basename)
+            for basename in ("foo", "bar", "baz", "qux")
+        )
+        for name in names:
+            create_hook(name)
+
+        self.assertEqual(self._call(self.tempdir), names)
+
+    def test_single(self):
+        name = os.path.join(self.tempdir, "foo")
+        create_hook(name)
+
+        self.assertEqual(self._call(self.tempdir), [name])
+
+
+def create_hook(file_path):
+    """Creates an executable file at the specified path.
+
+    :param str file_path: path to create the file at
+
+    """
+    open(file_path, "w").close()
+    os.chmod(file_path, os.stat(file_path).st_mode | stat.S_IXUSR)
 
 
 if __name__ == '__main__':

certbot/tests/log_test.py

@@ -343,11 +343,17 @@ class PostArgParseExceptHookTest(unittest.TestCase):
     def _test_common(self, error_type, debug):
         """Returns the mocked logger and stderr output."""
         mock_err = six.StringIO()
+
+        def write_err(*args, **unused_kwargs):
+            """Write error to mock_err."""
+            mock_err.write(args[0])
+
         try:
             raise error_type(self.error_msg)
         except BaseException:
             exc_info = sys.exc_info()
             with mock.patch('certbot.log.logger') as mock_logger:
+                mock_logger.error.side_effect = write_err
                 with mock.patch('certbot.log.sys.stderr', mock_err):
                     try:
                         # pylint: disable=star-args

certbot/tests/main_test.py

@@ -164,9 +164,7 @@ class CertonlyTest(unittest.TestCase):
         self.assertTrue(mock_report_cert.call_count == 2)
 
         # error in _ask_user_to_confirm_new_names
-        util_mock = mock.Mock()
-        util_mock.yesno.return_value = False
-        self.mock_get_utility.return_value = util_mock
+        self.mock_get_utility().yesno.return_value = False
         self.assertRaises(errors.ConfigurationError, self._call,
             ('certonly --webroot -d example.com -d test.com --cert-name example.com').split())
 
@@ -265,8 +263,11 @@ class RevokeTest(test_util.TempDirTestCase):
         from certbot.main import revoke
         revoke(config, plugins)
 
+    @mock.patch('certbot.main._delete_if_appropriate')
     @mock.patch('certbot.main.client.acme_client')
-    def test_revoke_with_reason(self, mock_acme_client):
+    def test_revoke_with_reason(self, mock_acme_client,
+            mock_delete_if_appropriate):
+        mock_delete_if_appropriate.return_value = False
         mock_revoke = mock_acme_client.Client().revoke
         expected = []
         for reason, code in constants.REVOCATION_REASONS.items():
@@ -276,8 +277,10 @@ class RevokeTest(test_util.TempDirTestCase):
             expected.append(mock.call(mock.ANY, code))
         self.assertEqual(expected, mock_revoke.call_args_list)
 
-    def test_revocation_success(self):
+    @mock.patch('certbot.main._delete_if_appropriate')
+    def test_revocation_success(self, mock_delete_if_appropriate):
         self._call()
+        mock_delete_if_appropriate.return_value = False
         self.mock_success_revoke.assert_called_once_with(self.tmp_cert_path)
 
     def test_revocation_error(self):
@@ -286,6 +289,198 @@ class RevokeTest(test_util.TempDirTestCase):
         self.assertRaises(acme_errors.ClientError, self._call)
         self.mock_success_revoke.assert_not_called()
 
+    @mock.patch('certbot.main._delete_if_appropriate')
+    @mock.patch('certbot.cert_manager.delete')
+    @test_util.patch_get_utility()
+    def test_revocation_with_prompt(self, mock_get_utility,
+            mock_delete, mock_delete_if_appropriate):
+        mock_get_utility().yesno.return_value = False
+        mock_delete_if_appropriate.return_value = False
+        self._call()
+        self.assertFalse(mock_delete.called)
+
+class DeleteIfAppropriateTest(unittest.TestCase):
+    """Tests for certbot.main._delete_if_appropriate """
+
+    def setUp(self):
+        self.config = mock.Mock()
+        self.config.namespace = mock.Mock()
+        self.config.namespace.noninteractive_mode = False
+
+    def _call(self, mock_config):
+        from certbot.main import _delete_if_appropriate
+        _delete_if_appropriate(mock_config)
+
+    @mock.patch('certbot.cert_manager.delete')
+    @test_util.patch_get_utility()
+    def test_delete_opt_out(self, mock_get_utility, mock_delete):
+        util_mock = mock_get_utility()
+        util_mock.yesno.return_value = False
+        self._call(self.config)
+        mock_delete.assert_not_called()
+
+    # pylint: disable=too-many-arguments
+    @mock.patch('certbot.storage.renewal_file_for_certname')
+    @mock.patch('certbot.cert_manager.delete')
+    @mock.patch('certbot.cert_manager.match_and_check_overlaps')
+    @mock.patch('certbot.storage.full_archive_path')
+    @mock.patch('certbot.cert_manager.cert_path_to_lineage')
+    @test_util.patch_get_utility()
+    def test_overlapping_archive_dirs(self, mock_get_utility,
+            mock_cert_path_to_lineage, mock_archive,
+            mock_match_and_check_overlaps, mock_delete,
+            mock_renewal_file_for_certname):
+        # pylint: disable = unused-argument
+        config = self.config
+        config.cert_path = "/some/reasonable/path"
+        config.certname = ""
+        mock_cert_path_to_lineage.return_value = "example.com"
+        mock_match_and_check_overlaps.side_effect = errors.OverlappingMatchFound()
+        self._call(config)
+        mock_delete.assert_not_called()
+
+    # pylint: disable=too-many-arguments
+    @mock.patch('certbot.storage.renewal_file_for_certname')
+    @mock.patch('certbot.cert_manager.match_and_check_overlaps')
+    @mock.patch('certbot.storage.full_archive_path')
+    @mock.patch('certbot.cert_manager.delete')
+    @mock.patch('certbot.storage.cert_path_for_cert_name')
+    @test_util.patch_get_utility()
+    def test_cert_name_only(self, mock_get_utility,
+            mock_cert_path_for_cert_name, mock_delete, mock_archive,
+            mock_overlapping_archive_dirs, mock_renewal_file_for_certname):
+        # pylint: disable = unused-argument
+        config = self.config
+        config.certname = "example.com"
+        config.cert_path = ""
+        mock_cert_path_for_cert_name.return_value = "/some/reasonable/path"
+        mock_overlapping_archive_dirs.return_value = False
+        self._call(config)
+        mock_delete.assert_called_once()
+
+    # pylint: disable=too-many-arguments
+    @mock.patch('certbot.storage.renewal_file_for_certname')
+    @mock.patch('certbot.cert_manager.match_and_check_overlaps')
+    @mock.patch('certbot.storage.full_archive_path')
+    @mock.patch('certbot.cert_manager.delete')
+    @mock.patch('certbot.cert_manager.cert_path_to_lineage')
+    @test_util.patch_get_utility()
+    def test_cert_path_only(self, mock_get_utility,
+            mock_cert_path_to_lineage, mock_delete, mock_archive,
+            mock_overlapping_archive_dirs, mock_renewal_file_for_certname):
+        # pylint: disable = unused-argument
+        config = self.config
+        config.cert_path = "/some/reasonable/path"
+        config.certname = ""
+        mock_cert_path_to_lineage.return_value = "example.com"
+        mock_overlapping_archive_dirs.return_value = False
+        self._call(config)
+        mock_delete.assert_called_once()
+
+    # pylint: disable=too-many-arguments
+    @mock.patch('certbot.storage.renewal_file_for_certname')
+    @mock.patch('certbot.cert_manager.match_and_check_overlaps')
+    @mock.patch('certbot.storage.full_archive_path')
+    @mock.patch('certbot.cert_manager.cert_path_to_lineage')
+    @mock.patch('certbot.cert_manager.delete')
+    @test_util.patch_get_utility()
+    def test_noninteractive_deletion(self, mock_get_utility, mock_delete,
+            mock_cert_path_to_lineage, mock_full_archive_dir,
+            mock_match_and_check_overlaps, mock_renewal_file_for_certname):
+        # pylint: disable = unused-argument
+        config = self.config
+        config.namespace.noninteractive_mode = True
+        config.cert_path = "/some/reasonable/path"
+        config.certname = ""
+        mock_cert_path_to_lineage.return_value = "example.com"
+        mock_full_archive_dir.return_value = ""
+        mock_match_and_check_overlaps.return_value = ""
+        self._call(config)
+        mock_delete.assert_called_once()
+
+    # pylint: disable=too-many-arguments
+    @mock.patch('certbot.storage.renewal_file_for_certname')
+    @mock.patch('certbot.cert_manager.match_and_check_overlaps')
+    @mock.patch('certbot.storage.full_archive_path')
+    @mock.patch('certbot.cert_manager.delete')
+    @mock.patch('certbot.cert_manager.cert_path_to_lineage')
+    @test_util.patch_get_utility()
+    def test_certname_and_cert_path_match(self, mock_get_utility,
+            mock_cert_path_to_lineage, mock_delete, mock_archive,
+            mock_overlapping_archive_dirs, mock_renewal_file_for_certname):
+        # pylint: disable = unused-argument
+        config = self.config
+        config.certname = "example.com"
+        config.cert_path = "/some/reasonable/path"
+        mock_cert_path_to_lineage.return_value = config.certname
+        mock_overlapping_archive_dirs.return_value = False
+        self._call(config)
+        mock_delete.assert_called_once()
+
+    # pylint: disable=too-many-arguments
+    @mock.patch('certbot.cert_manager.match_and_check_overlaps')
+    @mock.patch('certbot.storage.full_archive_path')
+    @mock.patch('certbot.cert_manager.delete')
+    @mock.patch('certbot.cert_manager.human_readable_cert_info')
+    @mock.patch('certbot.storage.RenewableCert')
+    @mock.patch('certbot.storage.renewal_file_for_certname')
+    @mock.patch('certbot.cert_manager.cert_path_to_lineage')
+    @test_util.patch_get_utility()
+    def test_certname_and_cert_path_mismatch(self, mock_get_utility,
+            mock_cert_path_to_lineage, mock_renewal_file_for_certname,
+            mock_RenewableCert, mock_human_readable_cert_info,
+            mock_delete, mock_archive, mock_overlapping_archive_dirs):
+        # pylint: disable=unused-argument
+        config = self.config
+        config.certname = "example.com"
+        config.cert_path = "/some/reasonable/path"
+        mock_cert_path_to_lineage = "something else"
+        mock_RenewableCert.return_value = mock.Mock()
+        mock_human_readable_cert_info.return_value = ""
+        mock_overlapping_archive_dirs.return_value = False
+        from certbot.display import util as display_util
+        util_mock = mock_get_utility()
+        util_mock.menu.return_value = (display_util.OK, 0)
+        self._call(config)
+        mock_delete.assert_called_once()
+
+    # pylint: disable=too-many-arguments
+    @mock.patch('certbot.cert_manager.match_and_check_overlaps')
+    @mock.patch('certbot.storage.full_archive_path')
+    @mock.patch('certbot.cert_manager.delete')
+    @mock.patch('certbot.cert_manager.human_readable_cert_info')
+    @mock.patch('certbot.storage.RenewableCert')
+    @mock.patch('certbot.storage.renewal_file_for_certname')
+    @mock.patch('certbot.cert_manager.cert_path_to_lineage')
+    @test_util.patch_get_utility()
+    def test_noninteractive_certname_cert_path_mismatch(self, mock_get_utility,
+            mock_cert_path_to_lineage, mock_renewal_file_for_certname,
+            mock_RenewableCert, mock_human_readable_cert_info,
+            mock_delete, mock_archive, mock_overlapping_archive_dirs):
+        # pylint: disable=unused-argument
+        config = self.config
+        config.certname = "example.com"
+        config.cert_path = "/some/reasonable/path"
+        mock_cert_path_to_lineage.return_value = "some-reasonable-path.com"
+        mock_RenewableCert.return_value = mock.Mock()
+        mock_human_readable_cert_info.return_value = ""
+        mock_overlapping_archive_dirs.return_value = False
+        # Test for non-interactive mode
+        util_mock = mock_get_utility()
+        util_mock.menu.side_effect = errors.MissingCommandlineFlag("Oh no.")
+        self.assertRaises(errors.Error, self._call, config)
+        mock_delete.assert_not_called()
+
+    @mock.patch('certbot.cert_manager.delete')
+    @test_util.patch_get_utility()
+    def test_no_certname_or_cert_path(self, mock_get_utility, mock_delete):
+        # pylint: disable=unused-argument
+        config = self.config
+        config.certname = None
+        config.cert_path = None
+        self.assertRaises(errors.Error, self._call, config)
+        mock_delete.assert_not_called()
+
 
 class DetermineAccountTest(test_util.ConfigTestCase):
     """Tests for certbot.main._determine_account."""
@@ -541,7 +736,32 @@ class MainTest(test_util.ConfigTestCase):  # pylint: disable=too-many-public-met
         ifaces = []
         plugins = mock_disco.PluginsRegistry.find_all()
 
-        _, stdout, _, _ = self._call(['plugins'])
+        stdout = six.StringIO()
+        with test_util.patch_get_utility_with_stdout(stdout=stdout):
+            _, stdout, _, _ = self._call(['plugins'], stdout)
+
+        plugins.visible.assert_called_once_with()
+        plugins.visible().ifaces.assert_called_once_with(ifaces)
+        filtered = plugins.visible().ifaces()
+        self.assertEqual(stdout.getvalue().strip(), str(filtered))
+
+    @mock.patch('certbot.main.plugins_disco')
+    @mock.patch('certbot.main.cli.HelpfulArgumentParser.determine_help_topics')
+    def test_plugins_no_args_unprivileged(self, _det, mock_disco):
+        ifaces = []
+        plugins = mock_disco.PluginsRegistry.find_all()
+
+        def throw_error(directory, mode, uid, strict):
+            """Raises error.Error."""
+            _, _, _, _ = directory, mode, uid, strict
+            raise errors.Error()
+
+        stdout = six.StringIO()
+        with mock.patch('certbot.util.set_up_core_dir') as mock_set_up_core_dir:
+            with test_util.patch_get_utility_with_stdout(stdout=stdout):
+                mock_set_up_core_dir.side_effect = throw_error
+                _, stdout, _, _ = self._call(['plugins'], stdout)
+
         plugins.visible.assert_called_once_with()
         plugins.visible().ifaces.assert_called_once_with(ifaces)
         filtered = plugins.visible().ifaces()
@@ -553,7 +773,10 @@ class MainTest(test_util.ConfigTestCase):  # pylint: disable=too-many-public-met
         ifaces = []
         plugins = mock_disco.PluginsRegistry.find_all()
 
-        _, stdout, _, _ = self._call(['plugins', '--init'])
+        stdout = six.StringIO()
+        with test_util.patch_get_utility_with_stdout(stdout=stdout):
+            _, stdout, _, _ = self._call(['plugins', '--init'], stdout)
+
         plugins.visible.assert_called_once_with()
         plugins.visible().ifaces.assert_called_once_with(ifaces)
         filtered = plugins.visible().ifaces()
@@ -567,7 +790,11 @@ class MainTest(test_util.ConfigTestCase):  # pylint: disable=too-many-public-met
     def test_plugins_prepare(self, _det, mock_disco):
         ifaces = []
         plugins = mock_disco.PluginsRegistry.find_all()
-        _, stdout, _, _ = self._call(['plugins', '--init', '--prepare'])
+
+        stdout = six.StringIO()
+        with test_util.patch_get_utility_with_stdout(stdout=stdout):
+            _, stdout, _, _ = self._call(['plugins', '--init', '--prepare'], stdout)
+
         plugins.visible.assert_called_once_with()
         plugins.visible().ifaces.assert_called_once_with(ifaces)
         filtered = plugins.visible().ifaces()
@@ -698,7 +925,8 @@ class MainTest(test_util.ConfigTestCase):  # pylint: disable=too-many-public-met
                           self._certonly_new_request_common, mock_client)
 
     def _test_renewal_common(self, due_for_renewal, extra_args, log_out=None,
-                             args=None, should_renew=True, error_expected=False):
+                             args=None, should_renew=True, error_expected=False,
+                                 quiet_mode=False):
         # pylint: disable=too-many-locals,too-many-arguments
         cert_path = test_util.vector_path('cert_512.pem')
         chain_path = '/etc/letsencrypt/live/foo.bar/fullchain.pem'
@@ -710,15 +938,23 @@ class MainTest(test_util.ConfigTestCase):  # pylint: disable=too-many-public-met
         mock_certr = mock.MagicMock()
         mock_key = mock.MagicMock(pem='pem_key')
         mock_client = mock.MagicMock()
-        stdout = None
+        stdout = six.StringIO()
         mock_client.obtain_certificate.return_value = (mock_certr, 'chain',
                                                        mock_key, 'csr')
+
+        def write_msg(message, *args, **kwargs):
+            """Write message to stdout."""
+            _, _ = args, kwargs
+            stdout.write(message)
+
         try:
             with mock.patch('certbot.cert_manager.find_duplicative_certs') as mock_fdc:
                 mock_fdc.return_value = (mock_lineage, None)
                 with mock.patch('certbot.main._init_le_client') as mock_init:
                     mock_init.return_value = mock_client
                     with test_util.patch_get_utility() as mock_get_utility:
+                        if not quiet_mode:
+                            mock_get_utility().notification.side_effect = write_msg
                         with mock.patch('certbot.main.renewal.OpenSSL') as mock_ssl:
                             mock_latest = mock.MagicMock()
                             mock_latest.get_issuer.return_value = "Fake fake"
@@ -729,7 +965,7 @@ class MainTest(test_util.ConfigTestCase):  # pylint: disable=too-many-public-met
                                 if extra_args:
                                     args += extra_args
                                 try:
-                                    ret, stdout, _, _ = self._call(args)
+                                    ret, stdout, _, _ = self._call(args, stdout)
                                     if ret:
                                         print("Returned", ret)
                                         raise AssertionError(ret)
@@ -799,7 +1035,8 @@ class MainTest(test_util.ConfigTestCase):  # pylint: disable=too-many-public-met
         self.assertTrue("renew" in out)
 
         args = ["renew", "--dry-run", "-q"]
-        _, _, stdout = self._test_renewal_common(True, [], args=args, should_renew=True)
+        _, _, stdout = self._test_renewal_common(True, [], args=args,
+                                                 should_renew=True, quiet_mode=True)
         out = stdout.getvalue()
         self.assertEqual("", out)
 
@@ -992,8 +1229,11 @@ class MainTest(test_util.ConfigTestCase):  # pylint: disable=too-many-public-met
         self.assertTrue(
             'dry run' in mock_get_utility().add_message.call_args[0][0])
 
+    @mock.patch('certbot.main._delete_if_appropriate')
     @mock.patch('certbot.main.client.acme_client')
-    def test_revoke_with_key(self, mock_acme_client):
+    def test_revoke_with_key(self, mock_acme_client,
+            mock_delete_if_appropriate):
+        mock_delete_if_appropriate.return_value = False
         server = 'foo.bar'
         self._call_no_clientmock(['--cert-path', SS_CERT_PATH, '--key-path', RSA2048_KEY_PATH,
                                  '--server', server, 'revoke'])
@@ -1013,8 +1253,11 @@ class MainTest(test_util.ConfigTestCase):  # pylint: disable=too-many-public-met
             ['--cert-path', CERT, '--key-path', KEY,
                                  '--server', server, 'revoke'])
 
+    @mock.patch('certbot.main._delete_if_appropriate')
     @mock.patch('certbot.main._determine_account')
-    def test_revoke_without_key(self, mock_determine_account):
+    def test_revoke_without_key(self, mock_determine_account,
+            mock_delete_if_appropriate):
+        mock_delete_if_appropriate.return_value = False
         mock_determine_account.return_value = (mock.MagicMock(), None)
         _, _, _, client = self._call(['--cert-path', CERT, 'revoke'])
         with open(CERT) as f:
@@ -1115,7 +1358,7 @@ class UnregisterTest(unittest.TestCase):
     def test_abort_unregister(self):
         self.mocks['account'].AccountFileStorage.return_value = mock.Mock()
 
-        util_mock = self.mocks['get_utility'].return_value
+        util_mock = self.mocks['get_utility']()
         util_mock.yesno.return_value = False
 
         config = mock.Mock()
@@ -1161,5 +1404,27 @@ class UnregisterTest(unittest.TestCase):
         self.assertFalse(cb_client.acme.deactivate_registration.called)
 
 
+class MakeOrVerifyNeededDirs(test_util.ConfigTestCase):
+    """Tests for certbot.main.make_or_verify_needed_dirs."""
+
+    @mock.patch("certbot.main.util")
+    def test_it(self, mock_util):
+        main.make_or_verify_needed_dirs(self.config)
+        for core_dir in (self.config.config_dir, self.config.work_dir,):
+            mock_util.set_up_core_dir.assert_any_call(
+                core_dir, constants.CONFIG_DIRS_MODE,
+                os.geteuid(), self.config.strict_permissions
+            )
+
+        hook_dirs = (self.config.renewal_pre_hooks_dir,
+                     self.config.renewal_deploy_hooks_dir,
+                     self.config.renewal_post_hooks_dir,)
+        for hook_dir in hook_dirs:
+            # default mode of 755 is used
+            mock_util.make_or_verify_dir.assert_any_call(
+                hook_dir, uid=os.geteuid(),
+                strict=self.config.strict_permissions)
+
+
 if __name__ == '__main__':
     unittest.main()  # pragma: no cover

certbot/tests/storage_test.py

@@ -884,6 +884,25 @@ class DeleteFilesTest(BaseRenewableCertTest):
             self.config.live_dir, "example.org")))
         self.assertFalse(os.path.exists(archive_dir))
 
+class CertPathForCertNameTest(BaseRenewableCertTest):
+    """Test for certbot.storage.cert_path_for_cert_name"""
+    def setUp(self):
+        super(CertPathForCertNameTest, self).setUp()
+        self.config_file.write()
+        self._write_out_ex_kinds()
+        self.fullchain = os.path.join(self.config.config_dir, 'live', 'example.org',
+                'fullchain.pem')
+        self.config.cert_path = (self.fullchain, '')
+
+    def _call(self, cli_config, certname):
+        from certbot.storage import cert_path_for_cert_name
+        return cert_path_for_cert_name(cli_config, certname)
+
+    def test_simple_cert_name(self):
+        self.assertEqual(self._call(self.config, 'example.org'), (self.fullchain, 'fullchain'))
+
+    def test_no_such_cert_name(self):
+        self.assertRaises(errors.CertStorageError, self._call, self.config, 'fake-example.org')
 
 if __name__ == "__main__":
     unittest.main()  # pragma: no cover

certbot/tests/util.py

@@ -14,6 +14,7 @@ from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import serialization
 import mock
 import OpenSSL
+import six
 from six.moves import reload_module  # pylint: disable=import-error
 
 from acme import jose
@@ -169,12 +170,36 @@ def patch_get_utility(target='zope.component.getUtility'):
     return mock.patch(target, new_callable=_create_get_utility_mock)
 
 
+def patch_get_utility_with_stdout(target='zope.component.getUtility',
+                                  stdout=None):
+    """Patch zope.component.getUtility to use a special mock IDisplay.
+
+    The mock IDisplay works like a regular mock object, except it also
+    also asserts that methods are called with valid arguments.
+
+    The `message` argument passed to the IDisplay methods is passed to
+    stdout's write method.
+
+    :param str target: path to patch
+    :param object stdout: object to write standard output to; it is
+        expected to have a `write` method
+
+    :returns: mock zope.component.getUtility
+    :rtype: mock.MagicMock
+
+    """
+    stdout = stdout if stdout else six.StringIO()
+
+    freezable_mock = _create_get_utility_mock_with_stdout(stdout)
+    return mock.patch(target, new=freezable_mock)
+
+
 class FreezableMock(object):
     """Mock object with the ability to freeze attributes.
 
     This class works like a regular mock.MagicMock object, except
-    attributes and behavior can be set and frozen so they cannot be
-    changed during tests.
+    attributes and behavior set before the object is frozen cannot
+    be changed during tests.
 
     If a func argument is provided to the constructor, this function
     is called first when an instance of FreezableMock is called,
@@ -182,10 +207,12 @@ class FreezableMock(object):
     value of func is ignored.
 
     """
-    def __init__(self, frozen=False, func=None):
+    def __init__(self, frozen=False, func=None, return_value=mock.sentinel.DEFAULT):
         self._frozen_set = set() if frozen else set(('freeze',))
         self._func = func
         self._mock = mock.MagicMock()
+        if return_value != mock.sentinel.DEFAULT:
+            self.return_value = return_value
         self._frozen = frozen
 
     def freeze(self):
@@ -203,17 +230,38 @@ class FreezableMock(object):
                 return object.__getattribute__(self, name)
             except AttributeError:
                 return False
+        elif name in ('return_value', 'side_effect',):
+            return getattr(object.__getattribute__(self, '_mock'), name)
         elif name == '_frozen_set' or name in self._frozen_set:
             return object.__getattribute__(self, name)
         else:
             return getattr(object.__getattribute__(self, '_mock'), name)
 
     def __setattr__(self, name, value):
+        """ Before it is frozen, attributes are set on the FreezableMock
+        instance and added to the _frozen_set. Attributes in the _frozen_set
+        cannot be changed after the FreezableMock is frozen. In this case,
+        they are set on the underlying _mock.
+
+        In cases of return_value and side_effect, these attributes are always
+        passed through to the instance's _mock and added to the _frozen_set
+        before the object is frozen.
+
+        """
         if self._frozen:
-            return setattr(self._mock, name, value)
-        elif name != '_frozen_set':
+            if name in self._frozen_set:
+                raise AttributeError('Cannot change frozen attribute ' + name)
+            else:
+                return setattr(self._mock, name, value)
+
+        if name != '_frozen_set':
             self._frozen_set.add(name)
-        return object.__setattr__(self, name, value)
+
+        if name in ('return_value', 'side_effect'):
+            return setattr(self._mock, name, value)
+
+        else:
+            return object.__setattr__(self, name, value)
 
 
 def _create_get_utility_mock():
@@ -223,7 +271,37 @@ def _create_get_utility_mock():
             frozen_mock = FreezableMock(frozen=True, func=_assert_valid_call)
             setattr(display, name, frozen_mock)
     display.freeze()
-    return mock.MagicMock(return_value=display)
+    return FreezableMock(frozen=True, return_value=display)
+
+
+def _create_get_utility_mock_with_stdout(stdout):
+    def _write_msg(message, *unused_args, **unused_kwargs):
+        """Write to message to stdout.
+        """
+        if message:
+            stdout.write(message)
+
+    def mock_method(*args, **kwargs):
+        """
+        Mock function for IDisplay methods.
+        """
+        _assert_valid_call(args, kwargs)
+        _write_msg(*args, **kwargs)
+
+
+    display = FreezableMock()
+    for name in interfaces.IDisplay.names():  # pylint: disable=no-member
+        if name == 'notification':
+            frozen_mock = FreezableMock(frozen=True,
+                                        func=_write_msg)
+            setattr(display, name, frozen_mock)
+        else:
+            frozen_mock = FreezableMock(frozen=True,
+                                        func=mock_method)
+            setattr(display, name, frozen_mock)
+    display.freeze()
+
+    return FreezableMock(frozen=True, return_value=display)
 
 
 def _assert_valid_call(*args, **kwargs):
@@ -254,13 +332,16 @@ class ConfigTestCase(TempDirTestCase):
     def setUp(self):
         super(ConfigTestCase, self).setUp()
         self.config = configuration.NamespaceConfig(
-            mock.MagicMock(
-                config_dir=os.path.join(self.tempdir, 'config'),
-                work_dir=os.path.join(self.tempdir, 'work'),
-                logs_dir=os.path.join(self.tempdir, 'logs'),
-                server="example.com",
-            )
+            mock.MagicMock(**constants.CLI_DEFAULTS)
         )
+        self.config.verb = "certonly"
+        self.config.config_dir = os.path.join(self.tempdir, 'config')
+        self.config.work_dir = os.path.join(self.tempdir, 'work')
+        self.config.logs_dir = os.path.join(self.tempdir, 'logs')
+        self.config.cert_path = constants.CLI_DEFAULTS['auth_cert_path']
+        self.config.fullchain_path = constants.CLI_DEFAULTS['auth_chain_path']
+        self.config.chain_path = constants.CLI_DEFAULTS['auth_chain_path']
+        self.config.server = "example.com"
 
 def lock_and_call(func, lock_path):
     """Grab a lock for lock_path and call func.

certbot/tests/util_test.py

@@ -258,7 +258,7 @@ class UniqueLineageNameTest(test_util.TempDirTestCase):
         for _ in six.moves.range(10):
             f, name = self._call("wow")
         self.assertTrue(isinstance(f, file_type))
-        self.assertTrue(isinstance(name, str))
+        self.assertTrue(isinstance(name, six.string_types))
         self.assertTrue("wow-0009.conf" in name)
 
     @mock.patch("certbot.util.os.fdopen")

certbot/util.py

@@ -91,6 +91,18 @@ def run_script(params, log=logger.error):
     return stdout, stderr
 
 
+def is_exe(path):
+    """Is path an executable file?
+
+    :param str path: path to test
+
+    :returns: True iff path is an executable file
+    :rtype: bool
+
+    """
+    return os.path.isfile(path) and os.access(path, os.X_OK)
+
+
 def exe_exists(exe):
     """Determine whether path/name refers to an executable.
 
@@ -100,10 +112,6 @@ def exe_exists(exe):
     :rtype: bool
 
     """
-    def is_exe(path):
-        """Determine if path is an exe."""
-        return os.path.isfile(path) and os.access(path, os.X_OK)
-
     path, _ = os.path.split(exe)
     if path:
         return is_exe(exe)

docs/cli-help.txt

@@ -56,12 +56,24 @@ optional arguments:
   -d DOMAIN, --domains DOMAIN, --domain DOMAIN
                         Domain names to apply. For multiple domains you can
                         use multiple -d flags or enter a comma separated list
-                        of domains as a parameter. (default: Ask)
-  --cert-name CERTNAME  Certificate name to apply. Only one certificate name
-                        can be used per Certbot run. To see certificate names,
-                        run 'certbot certificates'. When creating a new
-                        certificate, specifies the new certificate's name.
-                        (default: None)
+                        of domains as a parameter. The first domain provided
+                        will be the subject CN of the certificate, and all
+                        domains will be Subject Alternative Names on the
+                        certificate. The first domain will also be used in
+                        some software user interfaces and as the file paths
+                        for the certificate and related material unless
+                        otherwise specified or you already have a certificate
+                        with the same name. In the case of a name collision it
+                        will append a number like 0001 to the file path name.
+                        (default: Ask)
+  --cert-name CERTNAME  Certificate name to apply. This name is used by
+                        Certbot for housekeeping and in file paths; it doesn't
+                        affect the content of the certificate itself. To see
+                        certificate names, run 'certbot certificates'. When
+                        creating a new certificate, specifies the new
+                        certificate's name. (default: the first provided
+                        domain or the name of an existing certificate on your
+                        system for the same domains)
   --dry-run             Perform a test run of the client, obtaining test
                         (invalid) certificates but not saving them to disk.
                         This can currently only be used with the 'certonly'
@@ -95,7 +107,7 @@ optional arguments:
                         case, and to know when to deprecate support for past
                         Python versions and flags. If you wish to hide this
                         information from the Let's Encrypt server, set this to
-                        "". (default: CertbotACMEClient/0.17.0 (certbot;
+                        "". (default: CertbotACMEClient/0.19.0 (certbot;
                         Ubuntu 16.04.3 LTS) Authenticator/XXX Installer/YYY
                         (SUBCOMMAND; flags: FLAGS) Py/2.7.12). The flags
                         encoded in the user agent are: --duplicate, --force-
@@ -170,8 +182,7 @@ security:
                         Ask)
   --hsts                Add the Strict-Transport-Security header to every HTTP
                         response. Forcing browser to always use SSL for the
-                        domain. Defends against SSL Stripping. (default:
-                        False)
+                        domain. Defends against SSL Stripping. (default: None)
   --uir                 Add the "Content-Security-Policy: upgrade-insecure-
                         requests" header to every HTTP response. Forcing the
                         browser to use https:// for every http:// resource.
@@ -305,6 +316,8 @@ renew:
                         rather simplistic and fails if you use more advanced
                         shell constructs, so you can use this switch to
                         disable it. (default: False)
+  --no-directory-hooks  Disable running executables found in Certbot's hook
+                        directories during renewal. (default: False)
 
 certificates:
   List certificates managed by Certbot
@@ -315,8 +328,9 @@ delete:
 revoke:
   Options for revocation of certificates
 
-  --reason {keycompromise,affiliationchanged,superseded,unspecified,cessationofoperation}
-                        Specify reason for revoking certificate. (default: 0)
+  --reason {unspecified,keycompromise,affiliationchanged,superseded,cessationofoperation}
+                        Specify reason for revoking certificate. (default:
+                        unspecified)
 
 register:
   Options for account registration & modification
@@ -438,7 +452,7 @@ apache:
                         Apache server root directory. (default: /etc/apache2)
   --apache-vhost-root APACHE_VHOST_ROOT
                         Apache server VirtualHost configuration root (default:
-                        /etc/apache2/sites-available)
+                        None)
   --apache-logs-root APACHE_LOGS_ROOT
                         Apache server logs directory (default:
                         /var/log/apache2)

docs/install.rst

@@ -80,7 +80,7 @@ For full command line help, you can type::
 Problems with Python virtual environment
 ----------------------------------------
 
-On a low memory system such as VPS with less than 512MB of RAM, the required dependencies of Certbot will failed to build.
+On a low memory system such as VPS with less than 512MB of RAM, the required dependencies of Certbot will fail to build.
 This can be identified if the pip outputs contains something like ``internal compiler error: Killed (program cc1)``.
 You can workaround this restriction by creating a temporary swapfile::
 

docs/using.rst

@@ -399,6 +399,8 @@ relevant files can be removed from the system with the ``delete`` subcommand::
 
 .. note:: If you don't use ``delete`` to remove the certificate completely, it will be renewed automatically at the next renewal event.
 
+.. note:: Revoking a certificate will have no effect on the rate limit imposed by the Let's Encrypt server.
+
 .. _renewal:
 
 Renewing certificates
@@ -484,6 +486,26 @@ apply appropriate file permissions.
            esac
    done
 
+You can also specify hooks by placing files in subdirectories of Certbot's
+configuration directory. Assuming your configuration directory is
+``/etc/letsencrypt``, any executable files found in
+``/etc/letsencrypt/renewal-hooks/pre``,
+``/etc/letsencrypt/renewal-hooks/deploy``, and
+``/etc/letsencrypt/renewal-hooks/post`` will be run as pre, deploy, and post
+hooks respectively when any certificate is renewed with the ``renew``
+subcommand. These hooks are run in alphabetical order and are not run for other
+subcommands. (The order the hooks are run is determined by the byte value of
+the characters in their filenames and is not dependent on your locale.)
+
+Hooks specified in the command line, :ref:`configuration file
+<config-file>`, or :ref:`renewal configuration files <renewal-config-file>` are
+run as usual after running all hooks in these directories. One minor exception
+to this is if a hook specified elsewhere is simply the path to an executable
+file in the hook directory of the same type (e.g. your pre-hook is the path to
+an executable in ``/etc/letsencrypt/renewal-hooks/pre``), the file is not run a
+second time. You can stop Certbot from automatically running executables found
+in these directories by including ``--no-directory-hooks`` on the command line.
+
 More information about hooks can be found by running
 ``certbot --help renew``.
 
@@ -540,6 +562,8 @@ commands into your individual environment.
   you will need to use the ``--post-hook`` since the exit status will be 0 both on successful renewal
   and when renewal is not necessary.
 
+.. _renewal-config-file:
+
 
 Modifying the Renewal Configuration File
 ----------------------------------------

letsencrypt-auto

@@ -23,12 +23,15 @@ fi
 if [ -z "$XDG_DATA_HOME" ]; then
   XDG_DATA_HOME=~/.local/share
 fi
-VENV_NAME="letsencrypt"
 if [ -z "$VENV_PATH" ]; then
-  VENV_PATH="$XDG_DATA_HOME/$VENV_NAME"
+  # We export these values so they are preserved properly if this script is
+  # rerun with sudo/su where $HOME/$XDG_DATA_HOME may have a different value.
+  export OLD_VENV_PATH="$XDG_DATA_HOME/letsencrypt"
+  export VENV_PATH="/opt/eff.org/certbot/venv"
 fi
 VENV_BIN="$VENV_PATH/bin"
-LE_AUTO_VERSION="0.17.0"
+BOOTSTRAP_VERSION_PATH="$VENV_PATH/certbot-auto-bootstrap-version.txt"
+LE_AUTO_VERSION="0.19.0"
 BASENAME=$(basename $0)
 USAGE="Usage: $BASENAME [OPTIONS]
 A self-updating wrapper script for the Certbot ACME client. When run, updates
@@ -49,6 +52,7 @@ Help for certbot itself cannot be provided until it is installed.
                                             implies --non-interactive
 
 All arguments are accepted and forwarded to the Certbot client when run."
+export CERTBOT_AUTO="$0"
 
 for arg in "$@" ; do
   case "$arg" in
@@ -77,7 +81,7 @@ for arg in "$@" ; do
           h)
             HELP=1;;
           n)
-            ASSUME_YES=1;;
+            NONINTERACTIVE=1;;
           q)
             QUIET=1;;
           v)
@@ -93,8 +97,8 @@ if [ $BASENAME = "letsencrypt-auto" ]; then
   HELP=0
 fi
 
-# Set ASSUME_YES to 1 if QUIET (i.e. --quiet implies --non-interactive)
-if [ "$QUIET" = 1 ]; then
+# Set ASSUME_YES to 1 if QUIET or NONINTERACTIVE
+if [ "$QUIET" = 1 -o "$NONINTERACTIVE" = 1 ]; then
   ASSUME_YES=1
 fi
 
@@ -119,16 +123,18 @@ else
   exit 1
 fi
 
-# certbot-auto needs root access to bootstrap OS dependencies, and
-# certbot itself needs root access for almost all modes of operation
-# The "normal" case is that sudo is used for the steps that need root, but
-# this script *can* be run as root (not recommended), or fall back to using
-# `su`. Auto-detection can be overridden by explicitly setting the
-# environment variable LE_AUTO_SUDO to 'sudo', 'sudo_su' or '' as used below.
+# Certbot itself needs root access for almost all modes of operation.
+# certbot-auto needs root access to bootstrap OS dependencies and install
+# Certbot at a protected path so it can be safely run as root. To accomplish
+# this, this script will attempt to run itself as root if it doesn't have the
+# necessary privileges by using `sudo` or falling back to `su` if it is not
+# available. The mechanism used to obtain root access can be set explicitly by
+# setting the environment variable LE_AUTO_SUDO to 'sudo', 'su', 'su_sudo',
+# 'SuSudo', or '' as used below.
 
 # Because the parameters in `su -c` has to be a string,
 # we need to properly escape it.
-su_sudo() {
+SuSudo() {
   args=""
   # This `while` loop iterates over all parameters given to this function.
   # For each parameter, all `'` will be replace by `'"'"'`, and the escaped string
@@ -147,37 +153,57 @@ su_sudo() {
   su root -c "$args"
 }
 
-SUDO_ENV=""
-export CERTBOT_AUTO="$0"
-if [ -n "${LE_AUTO_SUDO+x}" ]; then
-  case "$LE_AUTO_SUDO" in
-    su_sudo|su)
-      SUDO=su_sudo
-      ;;
-    sudo)
-      SUDO=sudo
-      SUDO_ENV="CERTBOT_AUTO=$0"
-      ;;
-    '') ;; # Nothing to do for plain root method.
-    *)
-      error "Error: unknown root authorization mechanism '$LE_AUTO_SUDO'."
-      exit 1
-  esac
-  say "Using preset root authorization mechanism '$LE_AUTO_SUDO'."
-else
-  if test "`id -u`" -ne "0" ; then
-    if $EXISTS sudo 1>/dev/null 2>&1; then
-      SUDO=sudo
-      SUDO_ENV="CERTBOT_AUTO=$0"
-    else
-      say \"sudo\" is not available, will use \"su\" for installation steps...
-      SUDO=su_sudo
-    fi
+# Sets the environment variable SUDO to be the name of the program or function
+# to call to get root access. If this script already has root privleges, SUDO
+# is set to an empty string. The value in SUDO should be run with the command
+# to called with root privileges as arguments.
+SetRootAuthMechanism() {
+  SUDO=""
+  if [ -n "${LE_AUTO_SUDO+x}" ]; then
+    case "$LE_AUTO_SUDO" in
+      SuSudo|su_sudo|su)
+        SUDO=SuSudo
+        ;;
+      sudo)
+        SUDO="sudo -E"
+        ;;
+      '') ;; # Nothing to do for plain root method.
+      *)
+        error "Error: unknown root authorization mechanism '$LE_AUTO_SUDO'."
+        exit 1
+    esac
+    say "Using preset root authorization mechanism '$LE_AUTO_SUDO'."
   else
-    SUDO=
+    if test "`id -u`" -ne "0" ; then
+      if $EXISTS sudo 1>/dev/null 2>&1; then
+        SUDO="sudo -E"
+      else
+        say \"sudo\" is not available, will use \"su\" for installation steps...
+        SUDO=SuSudo
+      fi
+    fi
+  fi
+}
+
+if [ "$1" = "--cb-auto-has-root" ]; then
+  shift 1
+else
+  SetRootAuthMechanism
+  if [ -n "$SUDO" ]; then
+    echo "Requesting to rerun $0 with root privileges..."
+    $SUDO "$0" --cb-auto-has-root "$@"
+    exit 0
   fi
 fi
 
+# Runs this script again with the given arguments. --cb-auto-has-root is added
+# to the command line arguments to ensure we don't try to acquire root a
+# second time. After the script is rerun, we exit the current script.
+RerunWithArgs() {
+    "$0" --cb-auto-has-root "$@"
+    exit 0
+}
+
 BootstrapMessage() {
   # Arguments: Platform name
   say "Bootstrapping dependencies for $1... (you can skip this with --no-bootstrap)"
@@ -238,6 +264,10 @@ DeterminePythonVersion() {
   fi
 }
 
+# If new packages are installed by BootstrapDebCommon below, this version
+# number must be increased.
+BOOTSTRAP_DEB_COMMON_VERSION=1
+
 BootstrapDebCommon() {
   # Current version tested with:
   #
@@ -261,7 +291,7 @@ BootstrapDebCommon() {
     QUIET_FLAG='-qq'
   fi
 
-  $SUDO apt-get $QUIET_FLAG update || error apt-get update hit problems but continuing anyway...
+  apt-get $QUIET_FLAG update || error apt-get update hit problems but continuing anyway...
 
   # virtualenv binary can be found in different packages depending on
   # distro version (#346)
@@ -311,13 +341,13 @@ BootstrapDebCommon() {
           esac
         fi
         if [ "$add_backports" = 1 ]; then
-          $SUDO sh -c "echo $BACKPORT_SOURCELINE >> /etc/apt/sources.list.d/$BACKPORT_NAME.list"
-          $SUDO apt-get $QUIET_FLAG update
+          sh -c "echo $BACKPORT_SOURCELINE >> /etc/apt/sources.list.d/$BACKPORT_NAME.list"
+          apt-get $QUIET_FLAG update
         fi
       fi
     fi
     if [ "$add_backports" != 0 ]; then
-      $SUDO apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends -t "$BACKPORT_NAME" $augeas_pkg
+      apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends -t "$BACKPORT_NAME" $augeas_pkg
       augeas_pkg=
     fi
   }
@@ -336,7 +366,7 @@ BootstrapDebCommon() {
     # XXX add a case for ubuntu PPAs
   fi
 
-  $SUDO apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends \
+  apt-get install $QUIET_FLAG $YES_FLAG --no-install-recommends \
     python \
     python-dev \
     $virtualenv \
@@ -354,6 +384,10 @@ BootstrapDebCommon() {
   fi
 }
 
+# If new packages are installed by BootstrapRpmCommon below, this version
+# number must be increased.
+BOOTSTRAP_RPM_COMMON_VERSION=1
+
 BootstrapRpmCommon() {
   # Tested with:
   #   - Fedora 20, 21, 22, 23 (x64)
@@ -380,9 +414,9 @@ BootstrapRpmCommon() {
     QUIET_FLAG='--quiet'
   fi
 
-  if ! $SUDO $tool list *virtualenv >/dev/null 2>&1; then
+  if ! $tool list *virtualenv >/dev/null 2>&1; then
     echo "To use Certbot, packages from the EPEL repository need to be installed."
-    if ! $SUDO $tool list epel-release >/dev/null 2>&1; then
+    if ! $tool list epel-release >/dev/null 2>&1; then
       error "Enable the EPEL repository and try running Certbot again."
       exit 1
     fi
@@ -394,7 +428,7 @@ BootstrapRpmCommon() {
       /bin/echo -e "\e[0K\rEnabling the EPEL repository in 1 seconds..."
       sleep 1s
     fi
-    if ! $SUDO $tool install $yes_flag $QUIET_FLAG epel-release; then
+    if ! $tool install $yes_flag $QUIET_FLAG epel-release; then
       error "Could not enable EPEL. Aborting bootstrap!"
       exit 1
     fi
@@ -410,9 +444,8 @@ BootstrapRpmCommon() {
     ca-certificates
   "
 
-  # Some distros and older versions of current distros use a "python27"
-  # instead of "python" naming convention. Try both conventions.
-  if $SUDO $tool list python >/dev/null 2>&1; then
+  # Most RPM distros use the "python" or "python-" naming convention.  Let's try that first.
+  if $tool list python >/dev/null 2>&1; then
     pkgs="$pkgs
       python
       python-devel
@@ -420,6 +453,20 @@ BootstrapRpmCommon() {
       python-tools
       python-pip
     "
+  # Fedora 26 starts to use the prefix python2 for python2 based packages.
+  # this elseif is theoretically for any Fedora over version 26:
+  elif $tool list python2 >/dev/null 2>&1; then
+    pkgs="$pkgs
+      python2
+      python2-libs
+      python2-setuptools
+      python2-devel
+      python2-virtualenv
+      python2-tools
+      python2-pip
+    "
+  # Some distros and older versions of current distros use a "python27"
+  # instead of the "python" or "python-" naming convention.
   else
     pkgs="$pkgs
       python27
@@ -430,18 +477,22 @@ BootstrapRpmCommon() {
     "
   fi
 
-  if $SUDO $tool list installed "httpd" >/dev/null 2>&1; then
+  if $tool list installed "httpd" >/dev/null 2>&1; then
     pkgs="$pkgs
       mod_ssl
     "
   fi
 
-  if ! $SUDO $tool install $yes_flag $QUIET_FLAG $pkgs; then
+  if ! $tool install $yes_flag $QUIET_FLAG $pkgs; then
     error "Could not install OS dependencies. Aborting bootstrap!"
     exit 1
   fi
 }
 
+# If new packages are installed by BootstrapSuseCommon below, this version
+# number must be increased.
+BOOTSTRAP_SUSE_COMMON_VERSION=1
+
 BootstrapSuseCommon() {
   # SLE12 don't have python-virtualenv
 
@@ -454,7 +505,7 @@ BootstrapSuseCommon() {
     QUIET_FLAG='-qq'
   fi
 
-  $SUDO zypper $QUIET_FLAG $zypper_flags in $install_flags \
+  zypper $QUIET_FLAG $zypper_flags in $install_flags \
     python \
     python-devel \
     python-virtualenv \
@@ -465,6 +516,10 @@ BootstrapSuseCommon() {
     ca-certificates
 }
 
+# If new packages are installed by BootstrapArchCommon below, this version
+# number must be increased.
+BOOTSTRAP_ARCH_COMMON_VERSION=1
+
 BootstrapArchCommon() {
   # Tested with:
   #   - ArchLinux (x86_64)
@@ -485,21 +540,25 @@ BootstrapArchCommon() {
   "
 
   # pacman -T exits with 127 if there are missing dependencies
-  missing=$($SUDO pacman -T $deps) || true
+  missing=$(pacman -T $deps) || true
 
   if [ "$ASSUME_YES" = 1 ]; then
     noconfirm="--noconfirm"
   fi
 
   if [ "$missing" ]; then
-    if [ "$QUIET" = 1]; then
-      $SUDO pacman -S --needed $missing $noconfirm > /dev/null
+    if [ "$QUIET" = 1 ]; then
+      pacman -S --needed $missing $noconfirm > /dev/null
     else
-      $SUDO pacman -S --needed $missing $noconfirm
+      pacman -S --needed $missing $noconfirm
     fi
   fi
 }
 
+# If new packages are installed by BootstrapGentooCommon below, this version
+# number must be increased.
+BOOTSTRAP_GENTOO_COMMON_VERSION=1
+
 BootstrapGentooCommon() {
   PACKAGES="
     dev-lang/python:2.7
@@ -517,29 +576,37 @@ BootstrapGentooCommon() {
 
   case "$PACKAGE_MANAGER" in
     (paludis)
-      $SUDO cave resolve --preserve-world --keep-targets if-possible $PACKAGES -x
+      cave resolve --preserve-world --keep-targets if-possible $PACKAGES -x
       ;;
     (pkgcore)
-      $SUDO pmerge --noreplace --oneshot $ASK_OPTION $PACKAGES
+      pmerge --noreplace --oneshot $ASK_OPTION $PACKAGES
       ;;
     (portage|*)
-      $SUDO emerge --noreplace --oneshot $ASK_OPTION $PACKAGES
+      emerge --noreplace --oneshot $ASK_OPTION $PACKAGES
       ;;
   esac
 }
 
+# If new packages are installed by BootstrapFreeBsd below, this version number
+# must be increased.
+BOOTSTRAP_FREEBSD_VERSION=1
+
 BootstrapFreeBsd() {
   if [ "$QUIET" = 1 ]; then
     QUIET_FLAG="--quiet"
   fi
 
-  $SUDO pkg install -Ay $QUIET_FLAG \
+  pkg install -Ay $QUIET_FLAG \
     python \
     py27-virtualenv \
     augeas \
     libffi
 }
 
+# If new packages are installed by BootstrapMac below, this version number must
+# be increased.
+BOOTSTRAP_MAC_VERSION=1
+
 BootstrapMac() {
   if hash brew 2>/dev/null; then
     say "Using Homebrew to install dependencies..."
@@ -548,7 +615,7 @@ BootstrapMac() {
   elif hash port 2>/dev/null; then
     say "Using MacPorts to install dependencies..."
     pkgman=port
-    pkgcmd="$SUDO port install"
+    pkgcmd="port install"
   else
     say "No Homebrew/MacPorts; installing Homebrew..."
     ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
@@ -568,8 +635,8 @@ BootstrapMac() {
   # Workaround for _dlopen not finding augeas on macOS
   if [ "$pkgman" = "port" ] && ! [ -e "/usr/local/lib/libaugeas.dylib" ] && [ -e "/opt/local/lib/libaugeas.dylib" ]; then
     say "Applying augeas workaround"
-    $SUDO mkdir -p /usr/local/lib/
-    $SUDO ln -s /opt/local/lib/libaugeas.dylib /usr/local/lib/
+    mkdir -p /usr/local/lib/
+    ln -s /opt/local/lib/libaugeas.dylib /usr/local/lib/
   fi
 
   if ! hash pip 2>/dev/null; then
@@ -585,17 +652,25 @@ BootstrapMac() {
   fi
 }
 
+# If new packages are installed by BootstrapSmartOS below, this version number
+# must be increased.
+BOOTSTRAP_SMARTOS_VERSION=1
+
 BootstrapSmartOS() {
   pkgin update
   pkgin -y install 'gcc49' 'py27-augeas' 'py27-virtualenv'
 }
 
+# If new packages are installed by BootstrapMageiaCommon below, this version
+# number must be increased.
+BOOTSTRAP_MAGEIA_COMMON_VERSION=1
+
 BootstrapMageiaCommon() {
   if [ "$QUIET" = 1 ]; then
     QUIET_FLAG='--quiet'
   fi
 
-  if ! $SUDO urpmi --force $QUIET_FLAG \
+  if ! urpmi --force $QUIET_FLAG \
       python \
       libpython-devel \
       python-virtualenv
@@ -604,7 +679,7 @@ BootstrapMageiaCommon() {
       exit 1
   fi
 
-  if ! $SUDO urpmi --force $QUIET_FLAG \
+  if ! urpmi --force $QUIET_FLAG \
       git \
       gcc \
       python-augeas \
@@ -618,23 +693,41 @@ BootstrapMageiaCommon() {
 }
 
 
-# Install required OS packages:
-Bootstrap() {
-  if [ "$NO_BOOTSTRAP" = 1 ]; then
-      return
-  elif [ -f /etc/debian_version ]; then
+# Set Bootstrap to the function that installs OS dependencies on this system
+# and BOOTSTRAP_VERSION to the unique identifier for the current version of
+# that function. If Bootstrap is set to a function that doesn't install any
+# packages (either because --no-bootstrap was included on the command line or
+# we don't know how to bootstrap on this system), BOOTSTRAP_VERSION is not set.
+if [ "$NO_BOOTSTRAP" = 1 ]; then
+  Bootstrap() {
+    :
+  }
+elif [ -f /etc/debian_version ]; then
+  Bootstrap() {
     BootstrapMessage "Debian-based OSes"
     BootstrapDebCommon
-  elif [ -f /etc/mageia-release ]; then
-    # Mageia has both /etc/mageia-release and /etc/redhat-release
+  }
+  BOOTSTRAP_VERSION="BootstrapDebCommon $BOOTSTRAP_DEB_COMMON_VERSION"
+elif [ -f /etc/mageia-release ]; then
+  # Mageia has both /etc/mageia-release and /etc/redhat-release
+  Bootstrap() {
     ExperimentalBootstrap "Mageia" BootstrapMageiaCommon
-  elif [ -f /etc/redhat-release ]; then
+  }
+  BOOTSTRAP_VERSION="BootstrapMageiaCommon $BOOTSTRAP_MAGEIA_COMMON_VERSION"
+elif [ -f /etc/redhat-release ]; then
+  Bootstrap() {
     BootstrapMessage "RedHat-based OSes"
     BootstrapRpmCommon
-  elif [ -f /etc/os-release ] && `grep -q openSUSE /etc/os-release` ; then
+  }
+  BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION"
+elif [ -f /etc/os-release ] && `grep -q openSUSE /etc/os-release` ; then
+  Bootstrap() {
     BootstrapMessage "openSUSE-based OSes"
     BootstrapSuseCommon
-  elif [ -f /etc/arch-release ]; then
+  }
+  BOOTSTRAP_VERSION="BootstrapSuseCommon $BOOTSTRAP_SUSE_COMMON_VERSION"
+elif [ -f /etc/arch-release ]; then
+  Bootstrap() {
     if [ "$DEBUG" = 1 ]; then
       BootstrapMessage "Archlinux"
       BootstrapArchCommon
@@ -646,25 +739,76 @@ Bootstrap() {
       error "--debug flag."
       exit 1
     fi
-  elif [ -f /etc/manjaro-release ]; then
+  }
+  BOOTSTRAP_VERSION="BootstrapArchCommon $BOOTSTRAP_ARCH_COMMON_VERSION"
+elif [ -f /etc/manjaro-release ]; then
+  Bootstrap() {
     ExperimentalBootstrap "Manjaro Linux" BootstrapArchCommon
-  elif [ -f /etc/gentoo-release ]; then
+  }
+  BOOTSTRAP_VERSION="BootstrapArchCommon $BOOTSTRAP_ARCH_COMMON_VERSION"
+elif [ -f /etc/gentoo-release ]; then
+  Bootstrap() {
     DeprecationBootstrap "Gentoo" BootstrapGentooCommon
-  elif uname | grep -iq FreeBSD ; then
+  }
+  BOOTSTRAP_VERSION="BootstrapGentooCommon $BOOTSTRAP_GENTOO_COMMON_VERSION"
+elif uname | grep -iq FreeBSD ; then
+  Bootstrap() {
     DeprecationBootstrap "FreeBSD" BootstrapFreeBsd
-  elif uname | grep -iq Darwin ; then
+  }
+  BOOTSTRAP_VERSION="BootstrapFreeBsd $BOOTSTRAP_FREEBSD_VERSION"
+elif uname | grep -iq Darwin ; then
+  Bootstrap() {
     DeprecationBootstrap "macOS" BootstrapMac
-  elif [ -f /etc/issue ] && grep -iq "Amazon Linux" /etc/issue ; then
+  }
+  BOOTSTRAP_VERSION="BootstrapMac $BOOTSTRAP_MAC_VERSION"
+elif [ -f /etc/issue ] && grep -iq "Amazon Linux" /etc/issue ; then
+  Bootstrap() {
     ExperimentalBootstrap "Amazon Linux" BootstrapRpmCommon
-  elif [ -f /etc/product ] && grep -q "Joyent Instance" /etc/product ; then
+  }
+  BOOTSTRAP_VERSION="BootstrapRpmCommon $BOOTSTRAP_RPM_COMMON_VERSION"
+elif [ -f /etc/product ] && grep -q "Joyent Instance" /etc/product ; then
+  Bootstrap() {
     ExperimentalBootstrap "Joyent SmartOS Zone" BootstrapSmartOS
-  else
+  }
+  BOOTSTRAP_VERSION="BootstrapSmartOS $BOOTSTRAP_SMARTOS_VERSION"
+else
+  Bootstrap() {
     error "Sorry, I don't know how to bootstrap Certbot on your operating system!"
     error
     error "You will need to install OS dependencies, configure virtualenv, and run pip install manually."
     error "Please see https://letsencrypt.readthedocs.org/en/latest/contributing.html#prerequisites"
     error "for more info."
     exit 1
+  }
+fi
+
+# Sets PREV_BOOTSTRAP_VERSION to the identifier for the bootstrap script used
+# to install OS dependencies on this system. PREV_BOOTSTRAP_VERSION isn't set
+# if it is unknown how OS dependencies were installed on this system.
+SetPrevBootstrapVersion() {
+  if [ -f $BOOTSTRAP_VERSION_PATH ]; then
+    PREV_BOOTSTRAP_VERSION=$(cat "$BOOTSTRAP_VERSION_PATH")
+  # The list below only contains bootstrap version strings that existed before
+  # we started writing them to disk.
+  #
+  # DO NOT MODIFY THIS LIST UNLESS YOU KNOW WHAT YOU'RE DOING!
+  elif grep -Fqx "$BOOTSTRAP_VERSION" << "UNLIKELY_EOF"
+BootstrapDebCommon 1
+BootstrapMageiaCommon 1
+BootstrapRpmCommon 1
+BootstrapSuseCommon 1
+BootstrapArchCommon 1
+BootstrapGentooCommon 1
+BootstrapFreeBsd 1
+BootstrapMac 1
+BootstrapSmartOS 1
+UNLIKELY_EOF
+  then
+    # If there's no bootstrap version saved to disk, but the currently selected
+    # bootstrap script is from before we started saving the version number,
+    # return the currently selected version to prevent us from rebootstrapping
+    # unnecessarily.
+    PREV_BOOTSTRAP_VERSION="$BOOTSTRAP_VERSION"
   fi
 }
 
@@ -678,18 +822,38 @@ if [ "$1" = "--le-auto-phase2" ]; then
   # Phase 2: Create venv, install LE, and run.
 
   shift 1  # the --le-auto-phase2 arg
-  if [ -f "$VENV_BIN/letsencrypt" ]; then
-    # --version output ran through grep due to python-cryptography DeprecationWarnings
-    # grep for both certbot and letsencrypt until certbot and shim packages have been released
-    INSTALLED_VERSION=$("$VENV_BIN/letsencrypt" --version 2>&1 | grep "^certbot\|^letsencrypt" | cut -d " " -f 2)
-    if [ -z "$INSTALLED_VERSION" ]; then
-        error "Error: couldn't get currently installed version for $VENV_BIN/letsencrypt: " 1>&2
-        "$VENV_BIN/letsencrypt" --version
-        exit 1
+  SetPrevBootstrapVersion
+
+  INSTALLED_VERSION="none"
+  if [ -d "$VENV_PATH" ]; then
+    # If the selected Bootstrap function isn't a noop and it differs from the
+    # previously used version
+    if [ -n "$BOOTSTRAP_VERSION" -a "$BOOTSTRAP_VERSION" != "$PREV_BOOTSTRAP_VERSION" ]; then
+      # if non-interactive mode or stdin and stdout are connected to a terminal
+      if [ \( "$NONINTERACTIVE" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then
+        rm -rf "$VENV_PATH"
+        RerunWithArgs "$@"
+      else
+        error "Skipping upgrade because new OS dependencies may need to be installed."
+        error
+        error "To upgrade to a newer version, please run this script again manually so you can"
+        error "approve changes or with --non-interactive on the command line to automatically"
+        error "install any required packages."
+        # Set INSTALLED_VERSION to be the same so we don't update the venv
+        INSTALLED_VERSION="$LE_AUTO_VERSION"
+      fi
+    elif [ -f "$VENV_BIN/letsencrypt" ]; then
+      # --version output ran through grep due to python-cryptography DeprecationWarnings
+      # grep for both certbot and letsencrypt until certbot and shim packages have been released
+      INSTALLED_VERSION=$("$VENV_BIN/letsencrypt" --version 2>&1 | grep "^certbot\|^letsencrypt" | cut -d " " -f 2)
+      if [ -z "$INSTALLED_VERSION" ]; then
+          error "Error: couldn't get currently installed version for $VENV_BIN/letsencrypt: " 1>&2
+          "$VENV_BIN/letsencrypt" --version
+          exit 1
+      fi
     fi
-  else
-    INSTALLED_VERSION="none"
   fi
+
   if [ "$LE_AUTO_VERSION" != "$INSTALLED_VERSION" ]; then
     say "Creating virtual environment..."
     DeterminePythonVersion
@@ -700,6 +864,12 @@ if [ "$1" = "--le-auto-phase2" ]; then
       virtualenv --no-site-packages --python "$LE_PYTHON" "$VENV_PATH" > /dev/null
     fi
 
+    if [ -n "$BOOTSTRAP_VERSION" ]; then
+      echo "$BOOTSTRAP_VERSION" > "$BOOTSTRAP_VERSION_PATH"
+    elif [ -n "$PREV_BOOTSTRAP_VERSION" ]; then
+      echo "$PREV_BOOTSTRAP_VERSION" > "$BOOTSTRAP_VERSION_PATH"
+    fi
+
     say "Installing Python packages..."
     TEMP_DIR=$(TempDir)
     trap 'rm -rf "$TEMP_DIR"' EXIT
@@ -766,8 +936,8 @@ cffi==1.10.0 \
     --hash=sha256:285ab352552f52f1398c912556d4d36d4ea9b8450e5c65d03809bf9886755533 \
     --hash=sha256:5576644b859197da7bbd8f8c7c2fb5dcc6cd505cadb42992d5f104c013f8a214 \
     --hash=sha256:b3b02911eb1f6ada203b0763ba924234629b51586f72a21faacc638269f4ced5
-ConfigArgParse==0.10.0 \
-    --hash=sha256:3b50a83dd58149dfcee98cb6565265d10b53e9c0a2bca7eeef7fb5f5524890a7
+ConfigArgParse==0.12.0 \
+    --hash=sha256:28cd7d67669651f2a4518367838c49539457504584a139709b2b8f6c208ef339
 configobj==5.0.6 \
     --hash=sha256:a2f5650770e1c87fb335af19a9b7eb73fc05ccf22144eb68db7d00cd2bcb0902
 cryptography==2.0.2 \
@@ -907,18 +1077,18 @@ letsencrypt==0.7.0 \
     --hash=sha256:105a5fb107e45bcd0722eb89696986dcf5f08a86a321d6aef25a0c7c63375ade \
     --hash=sha256:c36e532c486a7e92155ee09da54b436a3c420813ec1c590b98f635d924720de9
 
-certbot==0.17.0 \
-    --hash=sha256:64c25c7123357feffded6408660bc6f5c7d493dd635ae172081d21473075a86a \
-    --hash=sha256:43f5b26c3f314d14babf79a3bdf3522e4fc9eef867a0681c426f113c650a669c
-acme==0.17.0 \
-    --hash=sha256:501710171633af13fc52aa61d0277a6fe335f7477db5810e72239aaf4f3a09e7 \
-    --hash=sha256:3ccbe4aaeb98c77b98ee4093b4e4adb76a1a24cbdfec0130c489c206f1d9b66e
-certbot-apache==0.17.0 \
-    --hash=sha256:17a7e8d7526d838610e68b96cf052af17c4055655b76b06d1cbc74857d90a216 \
-    --hash=sha256:29b9e7bc5eaaff6dc4bce8398e35eeacdf346126aad68cac3d41bb87df20a6b9
-certbot-nginx==0.17.0 \
-    --hash=sha256:980c9a33a79ab839a089a0085ff0c5414f01f47b6db26ed342df25916658cec9 \
-    --hash=sha256:e573f8b4283172755c07b9cca8a8da7ef2d31b4df763881394b5339b2d42994a
+certbot==0.19.0 \
+    --hash=sha256:3207ee5319bfc37e855c25a43148275fcfb37869eefde9087405012049734a20 \
+    --hash=sha256:a7230791dff5d085738119fc22d88ad9d8a35d0b6a3d67806fe33990c7c79d53
+acme==0.19.0 \
+    --hash=sha256:c612eafe234d722d97bb5d3dbc49e5522f44be29611f7577954eb893e5c2d6de \
+    --hash=sha256:1fa23d64d494aaf001e6fe857c461fcfff10f75a1c2c35ec831447f641e1e822
+certbot-apache==0.19.0 \
+    --hash=sha256:fadb28b33bfabc85cdb962b5b149bef58b98f0606b78581db7895fe38323f37c \
+    --hash=sha256:70306ca2d5be7f542af68d46883c0ae39527cf202f17ef92cd256fb0bc3f1619
+certbot-nginx==0.19.0 \
+    --hash=sha256:4909cb3db49919fb35590793cac28e1c0b6dbd29cbedf887b9106e5fcef5362c \
+    --hash=sha256:cb5a224a3f277092555c25096d1678fc735306fd3a43447649ebe524c7ca79e1
 
 UNLIKELY_EOF
     # -------------------------------------------------------------------------
@@ -1131,20 +1301,15 @@ UNLIKELY_EOF
       rm -rf "$VENV_PATH"
       exit 1
     fi
+
+    if [ -d "$OLD_VENV_PATH" -a ! -L "$OLD_VENV_PATH" ]; then
+      rm -rf "$OLD_VENV_PATH"
+      ln -s "$VENV_PATH" "$OLD_VENV_PATH"
+    fi
+
     say "Installation succeeded."
   fi
-  if [ -n "$SUDO" ]; then
-    # SUDO is su wrapper or sudo
-    say "Requesting root privileges to run certbot..."
-    say "  $VENV_BIN/letsencrypt" "$@"
-  fi
-  if [ -z "$SUDO_ENV" ] ; then
-    # SUDO is su wrapper / noop
-    $SUDO "$VENV_BIN/letsencrypt" "$@"
-  else
-    # sudo
-    $SUDO "$SUDO_ENV" "$VENV_BIN/letsencrypt" "$@"
-  fi
+  "$VENV_BIN/letsencrypt" "$@"
 
 else
   # Phase 1: Upgrade certbot-auto if necessary, then self-invoke.
@@ -1155,12 +1320,14 @@ else
   # package). Phase 2 checks the version of the locally installed certbot.
 
   if [ ! -f "$VENV_BIN/letsencrypt" ]; then
-    if [ "$HELP" = 1 ]; then
-      echo "$USAGE"
-      exit 0
+    if [ -z "$OLD_VENV_PATH" -o ! -f "$OLD_VENV_PATH/bin/letsencrypt" ]; then
+      if [ "$HELP" = 1 ]; then
+        echo "$USAGE"
+        exit 0
+      fi
+      # If it looks like we've never bootstrapped before, bootstrap:
+      Bootstrap
     fi
-    # If it looks like we've never bootstrapped before, bootstrap:
-    Bootstrap
   fi
   if [ "$OS_PACKAGES_ONLY" = 1 ]; then
     say "OS packages installed."
@@ -1194,7 +1361,8 @@ from os.path import dirname, join
 import re
 from subprocess import check_call, CalledProcessError
 from sys import argv, exit
-from urllib2 import build_opener, HTTPHandler, HTTPSHandler, HTTPError
+from urllib2 import build_opener, HTTPHandler, HTTPSHandler
+from urllib2 import HTTPError, URLError
 
 PUBLIC_KEY = environ.get('LE_AUTO_PUBLIC_KEY', """-----BEGIN PUBLIC KEY-----
 MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6MR8W/galdxnpGqBsYbq
@@ -1230,7 +1398,9 @@ class HttpsGetter(object):
 
         """
         try:
-            return self._opener.open(url).read()
+            # socket module docs say default timeout is None: that is, no
+            # timeout
+            return self._opener.open(url, timeout=30).read()
         except (HTTPError, IOError) as exc:
             raise ExpectedError("Couldn't download %s." % url, exc)
 
@@ -1320,15 +1490,15 @@ UNLIKELY_EOF
       say "Replacing certbot-auto..."
       # Clone permissions with cp. chmod and chown don't have a --reference
       # option on macOS or BSD, and stat -c on Linux is stat -f on macOS and BSD:
-      $SUDO cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone"
-      $SUDO cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone"
+      cp -p "$0" "$TEMP_DIR/letsencrypt-auto.permission-clone"
+      cp "$TEMP_DIR/letsencrypt-auto" "$TEMP_DIR/letsencrypt-auto.permission-clone"
       # Using mv rather than cp leaves the old file descriptor pointing to the
       # original copy so the shell can continue to read it unmolested. mv across
       # filesystems is non-atomic, doing `rm dest, cp src dest, rm src`, but the
-      # cp is unlikely to fail (esp. under sudo) if the rm doesn't.
-      $SUDO mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0"
+      # cp is unlikely to fail if the rm doesn't.
+      mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0"
     fi  # A newer version is available.
   fi  # Self-upgrading is allowed.
 
-  "$0" --le-auto-phase2 "$@"
+  RerunWithArgs --le-auto-phase2 "$@"
 fi

letsencrypt-auto-source/Dockerfile.precise

@@ -10,6 +10,8 @@ RUN useradd --create-home --home-dir /home/lea --shell /bin/bash --groups sudo -
 RUN apt-get update && \
     apt-get -q -y install python-pip sudo openssl && \
     apt-get clean
+
+ENV PIP_INDEX_URL https://pypi.python.org/simple
 RUN pip install nose
 
 # Let that user sudo:

letsencrypt-auto-source/Dockerfile.wheezy

@@ -10,6 +10,8 @@ RUN useradd --create-home --home-dir /home/lea --shell /bin/bash --groups sudo -
 RUN apt-get update && \
     apt-get -q -y install python-pip sudo openssl && \
     apt-get clean
+
+ENV PIP_INDEX_URL https://pypi.python.org/simple
 RUN pip install nose
 
 # Let that user sudo:

letsencrypt-auto-source/certbot-auto.asc

@@ -1,11 +1,11 @@
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2
 
-iQEcBAABCAAGBQJZgRYdAAoJEE0XyZXNl3XyNskIAMh/M3tV8PTieSrMr3uzLua8
-R+tQJV31WlraoKGQAkZ9Ak+nEhJy0bOi3QAeOmEnS15sBM6ruD+UCfwUDrZxolfW
-5Fnue2ocym+MhfDNKoerQNAmaaHY8sutoR+RNTegFyfyr92zMDZVzPm/DFAAHbK+
-eJltSx2Jleaig4V/RcKpkCwHErjQxn6Tn4jHlafAdNL28tEIGXcExpRj4raw3X1L
-SoTq/yJiWe+M7t+1iBRVEMZHY1b47PbTo1ipKF/ZZ3Hrz5JKRhAKcA8diHlWp+1I
-ujAfU4uu0hR+C3wcpeJ1i2YdS4S9y6uMGyIWU5toJfYdolTSGRZ2lPB+x5Um9pw=
-=/7P7
+iQEcBAABCAAGBQJZ1TJAAAoJEE0XyZXNl3XyWjAIAKxR5v0qbSyOEwM1LrSoLqud
+V3KkyEUlMq7IPHxoPKXbqUrIi4eZuhpJz+84LtVJe4ZQ6HYP9lPogX+PtmWW7dyO
+YerxA2rUVGB9rFZofZYwTuJyvO5Nc0aDyp1FHHPg/5khWWhhhxKpWqqG3zT01+Vf
+W8Lvvn7vr7sjTvxBdqHQ3z3hlUY62P2IKui9C5un5ozlSQpDrWh3Thi9r6CxbASL
+/r1PQ6EfnNdPAizVrJWe5iUd0Nzj7VMkFwZ02A3OlOUvrHGVb1H6oj0S1lZ8LEpj
+awOTys8PVBQ3vW2qbAL3Zk7Lr+CGfVfmoWC9TQEKiSN1woYFrFD39S527vB1onc=
+=Meks
 -----END PGP SIGNATURE-----

letsencrypt-auto-source/letsencrypt-auto

@@ -31,7 +31,7 @@ if [ -z "$VENV_PATH" ]; then
 fi
 VENV_BIN="$VENV_PATH/bin"
 BOOTSTRAP_VERSION_PATH="$VENV_PATH/certbot-auto-bootstrap-version.txt"
-LE_AUTO_VERSION="0.18.0.dev0"
+LE_AUTO_VERSION="0.20.0.dev0"
 BASENAME=$(basename $0)
 USAGE="Usage: $BASENAME [OPTIONS]
 A self-updating wrapper script for the Certbot ACME client. When run, updates
@@ -187,8 +187,7 @@ SetRootAuthMechanism() {
 
 if [ "$1" = "--cb-auto-has-root" ]; then
   shift 1
-elif [ "$1" != "--le-auto-phase2" ]; then
-  # if $1 is --le-auto-phase2, we've executed this branch before
+else
   SetRootAuthMechanism
   if [ -n "$SUDO" ]; then
     echo "Requesting to rerun $0 with root privileges..."
@@ -197,6 +196,14 @@ elif [ "$1" != "--le-auto-phase2" ]; then
   fi
 fi
 
+# Runs this script again with the given arguments. --cb-auto-has-root is added
+# to the command line arguments to ensure we don't try to acquire root a
+# second time. After the script is rerun, we exit the current script.
+RerunWithArgs() {
+    "$0" --cb-auto-has-root "$@"
+    exit 0
+}
+
 BootstrapMessage() {
   # Arguments: Platform name
   say "Bootstrapping dependencies for $1... (you can skip this with --no-bootstrap)"
@@ -825,8 +832,7 @@ if [ "$1" = "--le-auto-phase2" ]; then
       # if non-interactive mode or stdin and stdout are connected to a terminal
       if [ \( "$NONINTERACTIVE" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then
         rm -rf "$VENV_PATH"
-        "$0" "$@"
-        exit 0
+        RerunWithArgs "$@"
       else
         error "Skipping upgrade because new OS dependencies may need to be installed."
         error
@@ -1071,18 +1077,18 @@ letsencrypt==0.7.0 \
     --hash=sha256:105a5fb107e45bcd0722eb89696986dcf5f08a86a321d6aef25a0c7c63375ade \
     --hash=sha256:c36e532c486a7e92155ee09da54b436a3c420813ec1c590b98f635d924720de9
 
-certbot==0.17.0 \
-    --hash=sha256:64c25c7123357feffded6408660bc6f5c7d493dd635ae172081d21473075a86a \
-    --hash=sha256:43f5b26c3f314d14babf79a3bdf3522e4fc9eef867a0681c426f113c650a669c
-acme==0.17.0 \
-    --hash=sha256:501710171633af13fc52aa61d0277a6fe335f7477db5810e72239aaf4f3a09e7 \
-    --hash=sha256:3ccbe4aaeb98c77b98ee4093b4e4adb76a1a24cbdfec0130c489c206f1d9b66e
-certbot-apache==0.17.0 \
-    --hash=sha256:17a7e8d7526d838610e68b96cf052af17c4055655b76b06d1cbc74857d90a216 \
-    --hash=sha256:29b9e7bc5eaaff6dc4bce8398e35eeacdf346126aad68cac3d41bb87df20a6b9
-certbot-nginx==0.17.0 \
-    --hash=sha256:980c9a33a79ab839a089a0085ff0c5414f01f47b6db26ed342df25916658cec9 \
-    --hash=sha256:e573f8b4283172755c07b9cca8a8da7ef2d31b4df763881394b5339b2d42994a
+certbot==0.19.0 \
+    --hash=sha256:3207ee5319bfc37e855c25a43148275fcfb37869eefde9087405012049734a20 \
+    --hash=sha256:a7230791dff5d085738119fc22d88ad9d8a35d0b6a3d67806fe33990c7c79d53
+acme==0.19.0 \
+    --hash=sha256:c612eafe234d722d97bb5d3dbc49e5522f44be29611f7577954eb893e5c2d6de \
+    --hash=sha256:1fa23d64d494aaf001e6fe857c461fcfff10f75a1c2c35ec831447f641e1e822
+certbot-apache==0.19.0 \
+    --hash=sha256:fadb28b33bfabc85cdb962b5b149bef58b98f0606b78581db7895fe38323f37c \
+    --hash=sha256:70306ca2d5be7f542af68d46883c0ae39527cf202f17ef92cd256fb0bc3f1619
+certbot-nginx==0.19.0 \
+    --hash=sha256:4909cb3db49919fb35590793cac28e1c0b6dbd29cbedf887b9106e5fcef5362c \
+    --hash=sha256:cb5a224a3f277092555c25096d1678fc735306fd3a43447649ebe524c7ca79e1
 
 UNLIKELY_EOF
     # -------------------------------------------------------------------------
@@ -1355,7 +1361,8 @@ from os.path import dirname, join
 import re
 from subprocess import check_call, CalledProcessError
 from sys import argv, exit
-from urllib2 import build_opener, HTTPHandler, HTTPSHandler, HTTPError
+from urllib2 import build_opener, HTTPHandler, HTTPSHandler
+from urllib2 import HTTPError, URLError
 
 PUBLIC_KEY = environ.get('LE_AUTO_PUBLIC_KEY', """-----BEGIN PUBLIC KEY-----
 MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6MR8W/galdxnpGqBsYbq
@@ -1391,7 +1398,9 @@ class HttpsGetter(object):
 
         """
         try:
-            return self._opener.open(url).read()
+            # socket module docs say default timeout is None: that is, no
+            # timeout
+            return self._opener.open(url, timeout=30).read()
         except (HTTPError, IOError) as exc:
             raise ExpectedError("Couldn't download %s." % url, exc)
 
@@ -1491,5 +1500,5 @@ UNLIKELY_EOF
     fi  # A newer version is available.
   fi  # Self-upgrading is allowed.
 
-  "$0" --le-auto-phase2 "$@"
+  RerunWithArgs --le-auto-phase2 "$@"
 fi

letsencrypt-auto-source/letsencrypt-auto.sig

@@ -1,2 +1,2 @@
-χ8
F^¦F6÷jùk‹“rÒØhÛ^~Œeq¤GJ<47>£©3Z	õ³š›)åÙaõÍ6ŒèÂqS|<7C>2 s'LÁÂ,6âò‚Ù²:Ç
{Mvw¡{ð‰¡Ál*l<6+f÷‹;ö^ýP7v®âº¥ø9ª tÃöžR/~<EFBFBD>½³<EFBFBD>!ñšžÕ’T²z³O_ðjƬ`‚³·ÅõTqYó~ç`Bgl~f™qy¥
-/^X-‰|£3ó-VM#¸ _>¬4Qb»¡Ì~Äê1qYÑ$Æ<>¯"¿*¦ å#(—»ˆtXG…{±‡™ëfSÎû‰®q¿	Ü®ø1ïÙ³m…¹{Ä
+è¾לHêÉ­mì³ÊÄ+ˆ²Ä~™¦ES«ëM„<0E>4ø»ò¡ÙK“íY”jLãŸÁèÚê7øñöZ½åÕ³ÿ°dŸ<64>dÝïI.:†ÓdZMOü|’<>±K¢Öí°¾âm|göÊ(–$bšl<C5A1>jÇÐ…’/ñAâ^Ãéÿ©¶`ra®^ª0˜Ôß÷xÜÐå’<C3A5>²ƒwæÈá9”¦ckâNÃù<C383>¬Å‘.[
?ë”
+hð¡/Ì<>8!÷ü\§º’Å!»ÎöØÿ¯U5ñ£9bÉR£Ÿlb±-•«1‰Âà‰±ü(›p>¹-û¢%Îu2ÁgnêÍ

letsencrypt-auto-source/letsencrypt-auto.template

@@ -187,8 +187,7 @@ SetRootAuthMechanism() {
 
 if [ "$1" = "--cb-auto-has-root" ]; then
   shift 1
-elif [ "$1" != "--le-auto-phase2" ]; then
-  # if $1 is --le-auto-phase2, we've executed this branch before
+else
   SetRootAuthMechanism
   if [ -n "$SUDO" ]; then
     echo "Requesting to rerun $0 with root privileges..."
@@ -197,6 +196,14 @@ elif [ "$1" != "--le-auto-phase2" ]; then
   fi
 fi
 
+# Runs this script again with the given arguments. --cb-auto-has-root is added
+# to the command line arguments to ensure we don't try to acquire root a
+# second time. After the script is rerun, we exit the current script.
+RerunWithArgs() {
+    "$0" --cb-auto-has-root "$@"
+    exit 0
+}
+
 BootstrapMessage() {
   # Arguments: Platform name
   say "Bootstrapping dependencies for $1... (you can skip this with --no-bootstrap)"
@@ -406,8 +413,7 @@ if [ "$1" = "--le-auto-phase2" ]; then
       # if non-interactive mode or stdin and stdout are connected to a terminal
       if [ \( "$NONINTERACTIVE" = 1 \) -o \( \( -t 0 \) -a \( -t 1 \) \) ]; then
         rm -rf "$VENV_PATH"
-        "$0" "$@"
-        exit 0
+        RerunWithArgs "$@"
       else
         error "Skipping upgrade because new OS dependencies may need to be installed."
         error
@@ -567,5 +573,5 @@ UNLIKELY_EOF
     fi  # A newer version is available.
   fi  # Self-upgrading is allowed.
 
-  "$0" --le-auto-phase2 "$@"
+  RerunWithArgs --le-auto-phase2 "$@"
 fi