Fix oldest tests when local dependencies are used (#7019)

Fixes #7014.

Using a --force-reinstall (only for oldest tests), dependencies are properly reinstalled. Since this action significantly increases the execution time of oldest tests, I split them into two parts to allow their parallel execution by Travis.

We will need to find a better way to solve this in the future.

An example of successful execution of oldest tests in the situation of a point release can be found here: https://travis-ci.org/adferrand/certbot/builds/527475532

* Fix for oldest requirements

* Split oldest tests

* Update a comment

.travis.yml

@@ -60,7 +60,12 @@ matrix:
       env: TOXENV=mypy
       <<: *not-on-master
     - python: "2.7"
-      env: TOXENV='py27-{acme,apache,certbot,dns,nginx,postfix}-oldest'
+      env: TOXENV='py27-{acme,apache,certbot,nginx,postfix}-oldest'
+      sudo: required
+      services: docker
+      <<: *not-on-master
+    - python: "2.7"
+      env: TOXENV='py27-dns-oldest'
       sudo: required
       services: docker
       <<: *not-on-master

tools/pip_install.py

@@ -99,8 +99,15 @@ def main(args):
         else:
             # Otherwise, we merge requirements to build the constraints and pin dependencies
             requirements = None
+            reinstall = False
             if os.environ.get('CERTBOT_OLDEST') == '1':
                 requirements = certbot_oldest_processing(tools_path, args, test_constraints)
+                # We need to --force-reinstall the tested distribution when using oldest
+                # requirements because of an error in these tests in particular situations
+                # described in https://github.com/certbot/certbot/issues/7014.
+                # However this slows down considerably the oldest tests (5 min -> 10 min),
+                # so we need to find a better mitigation in the future.
+                reinstall = True
             else:
                 certbot_normal_processing(tools_path, test_constraints)
 
@@ -109,8 +116,8 @@ def main(args):
                 pip_install_with_print('--constraint "{0}" --requirement "{1}"'
                                        .format(all_constraints, requirements))
 
-            pip_install_with_print('--constraint "{0}" {1}'
-                                   .format(all_constraints, ' '.join(args)))
+            pip_install_with_print('--constraint "{0}" {1} {2}'.format(
+                all_constraints, '--force-reinstall' if reinstall else '', ' '.join(args)))
     finally:
         if os.environ.get('TRAVIS'):
             print('travis_fold:end:install_certbot_deps')