Certificats Let's Encrypt
Find a file
Adrien Ferrand b10ceb7d90 Fix test sdists with atexit handlers (#6769)
So merging the study from @bmw and me, here is what happened.

Each invocation of `certbot.logger.post_arg_parse_setup` create a file handler on `letsencrypt.log`. This function also set an atexit handler invoking `logger.shutdown()`, that have the effect to close all logger file handler not already closed at this point. This method is supposed to be called when a python process is close to exit, because it makes all logger unable to write new logs on any handler.

Before #6667 and this PR, for tests, the atexit handle would be triggered only at the end of the pytest process. It means that each test that launches `certbot.logger.post_arg_parse_setup` add a new file handler. These tests were typically connecting the file handler on a `letsencrypt.log` located in a temporary directory, and this directory and content was wipped out at each test tearDown. As a consequence, the file handles, not cleared from the logger, were accumulating in the logger, with all of them connected to a deleted file log, except the last one that was just created by the current test. Considering the number of tests concerned, there were ~300 file handler at the end of pytest execution.

One can see that, on prior #6667, by calling `print(logger.getLogger().handlers` on the `tearDown` of these tests, and see the array growing at each test execution.

Even if this represent a memory leak, this situation was not really a problem on Linux: because a file can be deleted before it is closed, it was only meaning that a given invocation of `logger.debug` for instance, during the tests, was written in 300 log files. The overhead is negligeable. On Windows however, the file handlers were failing because you cannot delete a file before it is closed.

It was one of the reason for #6667, that added a call to `logging.shutdown()` at each test tearDown, with the consequence to close all file handlers. At this point, Linux is not happy anymore. Any call to `logger.warn` will generate an error for each closed file handler. As a file handler is added for each test, the number of errors grows on each test, following an arithmetical suite divergence.

On `test_sdists.py`, that is using the bare setuptools test suite without output capturing, we can see the damages. The total output takes 216000 lines, and 23000 errors are generated. A decent machine can support this load, but a not a small AWS instance, that is crashing during the execution. Even with pytest, the captured output and the memory leak become so large that segfaults are generated.

On the current PR, the problem is solved, by resetting the file handlers array on the logging system on each test tearDown. So each fileHandler is properly closed, and removed from the stack. They do not participate anymore in the logging system, and can be garbage collected. Then we stay on always one file handler opened at any time, and tests can succeed on AWS instances.

For the record, here is all the places where the logging system is called and fail if there is still file handlers closed but not cleaned (extracted from the original huge output before correction):

```
Logged from file account.py, line 116
Logged from file account.py, line 178
Logged from file client.py, line 166
Logged from file client.py, line 295
Logged from file client.py, line 415
Logged from file client.py, line 422
Logged from file client.py, line 480
Logged from file client.py, line 503
Logged from file client.py, line 540
Logged from file client.py, line 601
Logged from file client.py, line 622
Logged from file client.py, line 750
Logged from file cli.py, line 220
Logged from file cli.py, line 226
Logged from file crypto_util.py, line 101
Logged from file crypto_util.py, line 127
Logged from file crypto_util.py, line 147
Logged from file crypto_util.py, line 261
Logged from file crypto_util.py, line 283
Logged from file crypto_util.py, line 307
Logged from file crypto_util.py, line 336
Logged from file disco.py, line 116
Logged from file disco.py, line 124
Logged from file disco.py, line 134
Logged from file disco.py, line 138
Logged from file disco.py, line 141
Logged from file dns_common_lexicon.py, line 45
Logged from file dns_common_lexicon.py, line 61
Logged from file dns_common_lexicon.py, line 67
Logged from file dns_common.py, line 316
Logged from file dns_common.py, line 64
Logged from file eff.py, line 60
Logged from file eff.py, line 73
Logged from file error_handler.py, line 105
Logged from file error_handler.py, line 110
Logged from file error_handler.py, line 87
Logged from file hooks.py, line 248
Logged from file main.py, line 1071
Logged from file main.py, line 1075
Logged from file main.py, line 1189
Logged from file ops.py, line 122
Logged from file ops.py, line 325
Logged from file ops.py, line 338
Logged from file reporter.py, line 55
Logged from file selection.py, line 110
Logged from file selection.py, line 118
Logged from file selection.py, line 123
Logged from file selection.py, line 176
Logged from file selection.py, line 231
Logged from file selection.py, line 310
Logged from file selection.py, line 66
Logged from file standalone.py, line 101
Logged from file standalone.py, line 88
Logged from file standalone.py, line 97
Logged from file standalone.py, line 98
Logged from file storage.py, line 52
Logged from file storage.py, line 59
Logged from file storage.py, line 75
Logged from file util.py, line 56
Logged from file webroot.py, line 165
Logged from file webroot.py, line 186
Logged from file webroot.py, line 187
Logged from file webroot.py, line 204
Logged from file webroot.py, line 223
Logged from file webroot.py, line 234
Logged from file webroot.py, line 235
Logged from file webroot.py, line 237
Logged from file webroot.py, line 91
```

* Reapply #6667

* Make setuptools delegates tests execution to pytest, like in acme module.

* Clean handlers at each tearDown to avoid memory leaks.

* Update changelog
2019-02-21 16:55:08 -08:00
acme Fix test sdists with atexit handlers (#6769) 2019-02-21 16:55:08 -08:00
certbot Fix test sdists with atexit handlers (#6769) 2019-02-21 16:55:08 -08:00
certbot-apache Fix test sdists with atexit handlers (#6769) 2019-02-21 16:55:08 -08:00
certbot-compatibility-test Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-cloudflare Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-cloudxns Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-digitalocean Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-dnsimple Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-dnsmadeeasy Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-gehirn Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-google Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-linode Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-luadns Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-nsone Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-ovh Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-rfc2136 Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-route53 Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-dns-sakuracloud Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
certbot-nginx Fix test sdists with atexit handlers (#6769) 2019-02-21 16:55:08 -08:00
certbot-postfix Merge branch 'master' into warnings-are-errors 2018-11-17 02:29:20 +00:00
docs Release 0.31.0 2019-02-07 13:27:10 -08:00
examples Update instances of acme-staging url to acme-staging-v02 (#5734) 2018-03-16 15:24:55 -07:00
letsencrypt-auto-source Bump version to 0.32.0 2019-02-07 13:27:13 -08:00
letshelp-certbot Advertise our packages work on Python 3.7. (#6183) 2018-07-09 09:17:03 -07:00
tests Add failure message if test farm tests do not run the correct number of tests. (#6771) 2019-02-20 15:20:44 -08:00
tools [Windows] Working unit tests for certbot-nginx (#6782) 2019-02-20 16:20:16 -08:00
.coveragerc Switch from nose to pytest (#5282) 2017-12-01 10:59:55 -08:00
.dockerignore Update ignore files to remove shared tox.venv 2015-07-12 15:30:51 +00:00
.gitattributes Merge pull request #2136 from tboegi/gitattributes_eol_overrideses_auto 2016-06-16 14:29:39 -07:00
.gitignore git ignore pytest cache (#6340) 2018-09-05 18:05:48 -07:00
.pylintrc Add --disable=locally-enabled to .pylintrc. (#6159) 2018-06-28 15:06:52 -07:00
.travis.yml Test all on push events to tested non-master branches. (#6741) 2019-02-06 12:47:56 -08:00
appveyor.yml Reduce build log verbosity on Travis (#6597) 2019-01-08 20:45:16 -08:00
AUTHORS.md make a list of contributors (#4508) 2017-04-26 14:57:23 -07:00
certbot-auto Release 0.31.0 2019-02-07 13:27:10 -08:00
CHANGELOG.md Fix test sdists with atexit handlers (#6769) 2019-02-21 16:55:08 -08:00
CONTRIBUTING.md Update CONTRIBUTING.md to be more welcoming. (#3540) 2016-09-26 16:44:27 -07:00
docker-compose.yml Cleanup dockerfile-dev (#5435) 2018-02-16 09:51:27 -08:00
Dockerfile Upgrade to Alpine 3.9 (#6743) 2019-02-07 09:06:04 -08:00
Dockerfile-dev [Windows|Unix] Rewrite bash scripts for tests into python (#6435) 2018-11-07 17:16:16 -08:00
Dockerfile-old Add VIRTUALENV_NO_DOWNLOAD=1 to all calls to virtualenv (#6690) 2019-01-24 11:50:41 -08:00
ISSUE_TEMPLATE.md Suggest people try the community forum. (#5561) 2018-02-09 16:41:05 -08:00
letsencrypt-auto Release 0.31.0 2019-02-07 13:27:10 -08:00
LICENSE.txt More stray ncrypt reference cleanup 2016-04-14 17:04:23 -07:00
linter_plugin.py Rename misc files 2016-04-14 10:20:23 -07:00
local-oldest-requirements.txt Bump version to 0.30.0 2018-12-05 10:57:46 -08:00
MANIFEST.in Remove CHANGES.rst (#6162) 2018-09-12 16:40:10 -07:00
mypy.ini Get mypy passing with check_untyped_defs everywhere (#6021) 2018-05-21 20:23:21 -07:00
pull_request_template.md Tell people to update changed package list. (#6633) 2019-01-02 16:15:25 -08:00
pytest.ini Ignore color_scheme warning from IPython. (#6714) 2019-01-30 13:59:07 -08:00
README.rst Change build status links to travis-ci.com. (#6656) 2019-01-14 14:06:52 -08:00
readthedocs.org.requirements.txt RTD: install local deps for subpkgs (fixes #1086). 2015-10-23 19:01:13 +00:00
setup.cfg Switch from nose to pytest (#5282) 2017-12-01 10:59:55 -08:00
setup.py Fix test sdists with atexit handlers (#6769) 2019-02-21 16:55:08 -08:00
tox.cover.py [Windows] Working unit tests for certbot-nginx (#6782) 2019-02-20 16:20:16 -08:00
tox.ini Fix the pebble fetch script (#6765) 2019-02-14 10:43:27 -08:00

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

.. This file contains a series of comments that are used to include sections of this README in other files. Do not modify these comments unless you know what you are doing. tag:intro-begin

Certbot is part of EFFs effort to encrypt the entire Internet. Secure communication over the Web relies on HTTPS, which requires the use of a digital certificate that lets browsers verify the identity of web servers (e.g., is that really google.com?). Web servers obtain their certificates from trusted third parties called certificate authorities (CAs). Certbot is an easy-to-use client that fetches a certificate from Lets Encrypt—an open certificate authority launched by the EFF, Mozilla, and others—and deploys it to a web server.

Anyone who has gone through the trouble of setting up a secure website knows what a hassle getting and maintaining a certificate is. Certbot and Lets Encrypt can automate away the pain and let you turn on and manage HTTPS with simple commands. Using Certbot and Let's Encrypt is free, so theres no need to arrange payment.

How you use Certbot depends on the configuration of your web server. The best way to get started is to use our `interactive guide <https://certbot.eff.org>`_. It generates instructions based on your configuration settings. In most cases, youll need `root or administrator access <https://certbot.eff.org/faq/#does-certbot-require-root-administrator-privileges>`_ to your web server to run Certbot.

Certbot is meant to be run directly on your web server, not on your personal computer. If youre using a hosted service and dont have direct access to your web server, you might not be able to use Certbot. Check with your hosting provider for documentation about uploading certificates or using certificates issued by Lets Encrypt.

Certbot is a fully-featured, extensible client for the Let's
Encrypt CA (or any other CA that speaks the `ACME
<https://github.com/ietf-wg-acme/acme/blob/master/draft-ietf-acme-acme.md>`_
protocol) that can automate the tasks of obtaining certificates and
configuring webservers to use them. This client runs on Unix-based operating
systems.

To see the changes made to Certbot between versions please refer to our
`changelog <https://github.com/certbot/certbot/blob/master/CHANGELOG.md>`_.

Until May 2016, Certbot was named simply ``letsencrypt`` or ``letsencrypt-auto``,
depending on install method. Instructions on the Internet, and some pieces of the
software, may still refer to this older name.

Contributing
------------

If you'd like to contribute to this project please read `Developer Guide
<https://certbot.eff.org/docs/contributing.html>`_.

.. _installation:

Installation
------------

The easiest way to install Certbot is by visiting `certbot.eff.org`_, where you can
find the correct installation instructions for many web server and OS combinations.
For more information, see `Get Certbot <https://certbot.eff.org/docs/install.html>`_.

.. _certbot.eff.org: https://certbot.eff.org/

How to run the client
---------------------

In many cases, you can just run ``certbot-auto`` or ``certbot``, and the
client will guide you through the process of obtaining and installing certs
interactively.

For full command line help, you can type::

  ./certbot-auto --help all


You can also tell it exactly what you want it to do from the command line.
For instance, if you want to obtain a cert for ``example.com``,
``www.example.com``, and ``other.example.net``, using the Apache plugin to both
obtain and install the certs, you could do this::

  ./certbot-auto --apache -d example.com -d www.example.com -d other.example.net

(The first time you run the command, it will make an account, and ask for an
email and agreement to the Let's Encrypt Subscriber Agreement; you can
automate those with ``--email`` and ``--agree-tos``)

If you want to use a webserver that doesn't have full plugin support yet, you
can still use "standalone" or "webroot" plugins to obtain a certificate::

  ./certbot-auto certonly --standalone --email admin@example.com -d example.com -d www.example.com -d other.example.net


Understanding the client in more depth
--------------------------------------

To understand what the client is doing in detail, it's important to
understand the way it uses plugins.  Please see the `explanation of
plugins <https://certbot.eff.org/docs/using.html#plugins>`_ in
the User Guide.

Links
=====

.. Do not modify this comment unless you know what you're doing. tag:links-begin

Documentation: https://certbot.eff.org/docs

Software project: https://github.com/certbot/certbot

Notes for developers: https://certbot.eff.org/docs/contributing.html

Main Website: https://certbot.eff.org

Let's Encrypt Website: https://letsencrypt.org

Community: https://community.letsencrypt.org

ACME spec: http://ietf-wg-acme.github.io/acme/

ACME working area in github: https://github.com/ietf-wg-acme/acme

|build-status| |coverage| |docs| |container|

.. |build-status| image:: https://travis-ci.com/certbot/certbot.svg?branch=master
   :target: https://travis-ci.com/certbot/certbot
   :alt: Travis CI status

.. |coverage| image:: https://codecov.io/gh/certbot/certbot/branch/master/graph/badge.svg
   :target: https://codecov.io/gh/certbot/certbot
   :alt: Coverage status

.. |docs| image:: https://readthedocs.org/projects/letsencrypt/badge/
   :target: https://readthedocs.org/projects/letsencrypt/
   :alt: Documentation status

.. |container| image:: https://quay.io/repository/letsencrypt/letsencrypt/status
   :target: https://quay.io/repository/letsencrypt/letsencrypt
   :alt: Docker Repository on Quay.io

.. Do not modify this comment unless you know what you're doing. tag:links-end

System Requirements
===================

See https://certbot.eff.org/docs/install.html#system-requirements.

.. Do not modify this comment unless you know what you're doing. tag:intro-end

.. Do not modify this comment unless you know what you're doing. tag:features-begin

Current Features
=====================

* Supports multiple web servers:

  - apache/2.x
  - nginx/0.8.48+
  - webroot (adds files to webroot directories in order to prove control of
    domains and obtain certs)
  - standalone (runs its own simple webserver to prove you control a domain)
  - other server software via `third party plugins <https://certbot.eff.org/docs/using.html#third-party-plugins>`_

* The private key is generated locally on your system.
* Can talk to the Let's Encrypt CA or optionally to other ACME
  compliant services.
* Can get domain-validated (DV) certificates.
* Can revoke certificates.
* Adjustable RSA key bit-length (2048 (default), 4096, ...).
* Can optionally install a http -> https redirect, so your site effectively
  runs https only (Apache only)
* Fully automated.
* Configuration changes are logged and can be reverted.
* Supports an interactive text UI, or can be driven entirely from the
  command line.
* Free and Open Source Software, made with Python.

.. Do not modify this comment unless you know what you're doing. tag:features-end

For extensive documentation on using and contributing to Certbot, go to https://certbot.eff.org/docs. If you would like to contribute to the project or run the latest code from git, you should read our `developer guide <https://certbot.eff.org/docs/contributing.html>`_.