diff --git a/bin/tests/system/README.md b/bin/tests/system/README.md index 96048c15a5..596f2d31e3 100644 --- a/bin/tests/system/README.md +++ b/bin/tests/system/README.md @@ -16,24 +16,28 @@ information regarding copyright ownership. This directory holds test environments for running bind9 system tests involving multiple name servers. -Each system test directory holds a set of scripts and configuration files to -test different parts of BIND. The directories are named for the aspect of BIND -they test, for example: +Each system test directory holds a set of test scripts and configuration files +to test different parts of BIND. The directories are named for the aspect of +BIND they test, for example: - dnssec/ DNSSEC tests - forward/ Forwarding tests - glue/ Glue handling tests +``` +dnssec/ DNSSEC tests +forward/ Forwarding tests +glue/ Glue handling tests +``` etc. -A system test directory must start with an alphabetic character and may not -contain any special characters. Only hyphen may be used as a word separator. +A system test directory name must start with an alphabetic character and may +only contain alphanumeric characters and underscores. Use underscore as the +word separator; hyphens are not allowed — they are reserved for the temporary +directories and symlinks the test runner creates. Typically each set of tests sets up 2-5 name servers and then performs one or more tests against them. Within the test subdirectory, each name server has a separate subdirectory containing its configuration data. These subdirectories -are named "nsN" or "ansN" (where N is a number between 1 and 8, e.g. ns1, ans2 -etc.) +are named "nsN" or "ansN" (where N is a number between 1 and 11, e.g. ns1, +ans2 etc.) The tests are completely self-contained and do not require access to the real DNS. Generally, one of the test servers (usually ns1) is set up as a root @@ -42,15 +46,37 @@ nameserver and is listed in the hints file of the others. ## Running the Tests +### Building BIND + +The system tests run the binaries from the build directory, so BIND must be +built first, along with a few test-only binaries and plugins: + +```sh +meson setup build +meson compile -C build +meson compile -C build system-test-dependencies +``` + +When BIND is configured with `-Ddeveloper=enabled`, the test dependencies are +built by default and the last step is not needed. + +Every `meson compile` invocation re-registers its build directory with the +source tree, so the tests always use the binaries from the most recently +compiled build directory. + ### Prerequisites To run system tests, make sure you have the following dependencies installed: - python3 (3.10 and newer) -- perl +- pytest (7.0 and newer) +- pytest-xdist +- perl (still needed by the test runner internals; some legacy tests + additionally need the Net::DNS module and are skipped when it is missing) -List of required python packages and their versions can be found in -requirements.txt (can be installed with `pip3 install -r requirements.txt`). +The full list of required and optional python packages can be found in +[requirements.txt](requirements.txt) (it can be installed with +`pip3 install -r requirements.txt`). ### Network Setup @@ -59,11 +85,15 @@ IP addresses on the loopback interface. ns1 runs on 10.53.0.1, ns2 on 10.53.0.2, etc. Before running any tests, you must set up these addresses by running the command - sh ifconfig.sh up +```sh +sh ifconfig.sh up +``` as root. The interfaces can be removed by executing the command: - sh ifconfig.sh down +```sh +sh ifconfig.sh down +``` ... also as root. @@ -77,45 +107,65 @@ If you wish to make the interfaces survive across reboots, copy org.isc.bind.system and org.isc.bind.system.plist to /Library/LaunchDaemons then run - launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist +```sh +launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist +``` ... as root. -### Running a Single Test - -The recommended way is to use pytest and its test selection facilities: - - pytest -k - -Using `-k` to specify a pattern allows to run a single pytest test case within -a system test. E.g. you can use `-k test_sslyze_dot` to execute just the -`test_sslyze_dot()` function from `doth/tests_sslyze.py`. - -However, using the `-k` pattern might pick up more tests than intended. You can -use the `--collect-only` option to check the list of tests which match you `-k` -pattern. If you just want to execute all system tests within a single test -directory, you can also use the utility script: - - ./run.sh system_test_dir_name - ### Running All the System Tests -Issuing plain `pytest` command without any argument will execute all tests -sequentially. To execute them in parallel, ensure you have pytest-xdist -installed and run: +Issue a plain `pytest` command in this directory to execute all tests +sequentially. To execute them in parallel instead, run: - pytest [-n ] +```sh +pytest -n +``` -Alternately, using the make command is also supported: +Parallel execution requires pytest-xdist; `-n auto` uses one worker per CPU. - make [-j numproc] test +### Running a Single Test + +To run all test modules in a single system test directory, pass the directory +name to pytest: + +```sh +pytest dns64 +``` + +The utility script `./run.sh dns64` does the same thing. + +To narrow the run down further, prefer pytest node IDs over `-k` matching — +they are exact: + +```sh +pytest dnssec_py/tests_mixed_ds.py +pytest doth/tests_sslyze.py::test_sslyze_dot +``` + +Parametrized tests have the parameter ID in brackets, so a single case of a +parametrized test can be selected as: + +```sh +pytest "dnssec_py/tests_nsec3_answer.py::test_nodata[ns2]" +``` + +The `-k` option selects tests by pattern matching: + +```sh +pytest -k +``` + +Beware that a `-k` pattern might pick up more tests than intended. Use the +`--collect-only` option to check the list of tests which match your `-k` +pattern. ### rr When running system tests, named can be run under the rr tool. rr records a trace to the $system_test/nsX/named-Y/ directory, which can be later used to -replay named. To enable this, execute start.pl with the USE_RR environment -variable set. +replay named. To enable this, run pytest with the USE_RR environment variable +set. ### Test Artifacts @@ -147,22 +197,8 @@ system test directories may contain the following standard files: - `tests_*.py`: These python files are picked up by pytest as modules. If they contain any test functions, they're added to the test suite. -- `*.j2`: These jinja2 templates can be used for configuration files or any - other files which require certain variables filled in, e.g. ports from the - environment variables. During test setup, the pytest runner will automatically - fill those in and strip the filename extension .j2, e.g. `ns1/named.conf.j2` - becomes `ns1/named.conf`. When using advanced templating to conditionally - include/omit entire sections or when filling in custom variables used for the - test, ensure the templates always include the defaults. If you don't need the - file to be auto-templated during test setup, use `.j2.manual` instead and then - no defaults are needed. - -- `setup.sh`: This sets up the preconditions for the tests. - -- `tests.sh`: Any shell-based tests are located within this file. Runs the - actual tests. - -- `tests_sh_*.py`: A glue file for the pytest runner for executing shell tests. +- `*.j2`: Jinja2 templates, rendered automatically during test setup (see + [Templates](#templates) below). - `ns`: These subdirectories contain test name servers that can be queried or can interact with each other. The value of N indicates the address the @@ -171,10 +207,23 @@ system test directories may contain the following standard files: run as root. These servers log at the highest debug level and the log is captured in the file "named.run". -- `ans`: Like ns[X], but these are simple mock name servers implemented in - Perl or Python. They are generally programmed to misbehave in ways named - would not so as to exercise named's ability to interoperate with badly - behaved name servers. +- `ans`: Like ns, but these are mock name servers implemented in python + (`ans.py`), usually with the `isctest.asyncserver` module. They are + generally programmed to misbehave in ways named would not, so as to exercise + named's ability to interoperate with badly behaved name servers. A few + legacy mock servers are still implemented in perl (`ans.pl`); don't write + new ones. + +The following files appear in test directories that have not yet been fully +ported to python; do not add them to new tests: + +- `tests.sh`: Legacy shell-based tests, run via a `tests_sh_*.py` glue module. + +- `setup.sh`: Legacy shell test setup. New tests use templates and a + `bootstrap()` function instead. + +- `prereq.sh`: Legacy prerequisite check; when it exits non-zero, the test is + skipped. New tests use pytest marks (see `isctest/mark.py`). ### Module Scope @@ -196,15 +245,132 @@ In order for the tests to run in parallel, each test requires a unique set of ports. This is ensured by the pytest runner, which assigns a unique set of ports to each test module. -Inside the python tests, it is possible to use the `ports` fixture to get the -assigned port numbers. They're also set as environment variables. These include: +Inside the python tests, it is possible to use fixtures like `named_port` to +get the assigned port numbers. They're also set as environment variables. +These include: - `PORT`: used as the basic dns port - `TLSPORT`: used as the port for DNS-over-TLS -- `HTTPPORT`, `HTTPSPORT`: used as the ports for DNS-over-HTTP +- `HTTPPORT`, `HTTPSPORT`: used as the ports for DNS-over-HTTP(S) - `CONTROLPORT`: used as the RNDC control port - `EXTRAPORT1` through `EXTRAPORT8`: additional ports that can be used as needed +### Templates + +Configuration files which need values that are only known at test run time — +ports, default crypto algorithms, conditional sections — are written as jinja2 +templates with a `.j2` extension. During test setup, the pytest runner +renders every `*.j2` file in the test directory and strips the extension: +`ns1/named.conf.j2` becomes `ns1/named.conf`. + +Inside a template, all the runner's environment variables are available with +`@...@` delimiters, e.g.: + +```jinja +options { + port @PORT@; + listen-on { 10.53.0.1; }; +}; +key rndc_key { + secret "1234abcd8765"; + algorithm @DEFAULT_HMAC@; +}; +controls { + inet 10.53.0.1 port @CONTROLPORT@ allow { any; } keys { rndc_key; }; +}; +``` + +Standard jinja2 block syntax (`{% if %}`, `{% for %}`, …) can be used for +conditional or repeated sections. + +Custom template variables come from an optional module-level `bootstrap()` +function. When a test module defines one, the runner calls it before +rendering the templates and passes the returned dict to the template engine: + +```python +def bootstrap(): + return {"valid": True} +``` + +Templates using custom variables must always provide defaults, so that the +file also renders when no value is supplied (e.g. when another module in the +same directory has no `bootstrap()`): + +```jinja +{% set valid = valid | default(False) %} +``` + +`bootstrap()` is also the place where a module generates test data that has +to exist before the servers start — typically zone files and DNSSEC keys. + +Templates can also be re-rendered while the test is running, using the +`templates` fixture, e.g. to change a server's config before reloading it: + +```python +def test_reload(ns1, templates): + templates.render("ns1/named.conf", {"valid": True}) + ns1.reconfigure() +``` + +If you don't need the file to be auto-templated during test setup, use the +extension `.j2.manual` instead; such templates are only rendered when the test +calls `templates.render()` explicitly, and no defaults are needed. + +### Fixtures and Helpers + +Fixtures defined in `conftest.py` provide the test context: + +- `servers` is a dictionary of all started `isctest.instance.NamedInstance` + servers, keyed by directory name; the shortcut fixtures `ns1` through `ns11` + return the corresponding instance directly. A `NamedInstance` is the + interface for driving a server: `ns1.rndc("...")`, `ns1.reconfigure()`, + `ns1.nsupdate(...)`, `ns1.watch_log_from_here()`, `ns1.ip`, ... +- `templates` renders jinja2 templates at runtime (see above). +- `named_port`, `named_tlsport`, `control_port`, ... return the assigned port + numbers. +- `system_test_dir` is the temporary directory the module runs in. + +The `isctest` package provides the helper library; the modules most tests +need are `isctest.query` (send DNS queries), `isctest.check` (assert on +responses), `isctest.zone` (zone and key setup), `isctest.kasp` (DNSSEC +key state checks), `isctest.asyncserver` (mock servers) and `isctest.log` +(logging and log watchers). + +Pytest marks control test collection and setup: + +- `@pytest.mark.extra_artifacts([...])` declares the files (globs) the test is + expected to leave behind in addition to the common ones; undeclared + leftovers fail the run. +- `@pytest.mark.requires_zones_loaded("ns1", ...)` delays the test until the + listed servers have loaded all zones. +- `isctest.mark` has skip-unless conditions for environment prerequisites, + e.g. `isctest.mark.with_dnstap`, `isctest.mark.softhsm2_environment`, + `isctest.mark.live_internet_test`. + +### Parametrization + +Use `pytest.mark.parametrize` to run one test function over several inputs +instead of copy-pasting test cases or looping inside one test function — each +case is reported (and can be re-run) individually: + +```python +@pytest.mark.parametrize( + "qname,rdtype", + [ + ("exists.example.", "A"), + ("exists.example.", "TXT"), + ("other.example.", "A"), + ], +) +def test_answers(qname, rdtype, ns1): + msg = isctest.query.create(qname, rdtype) + response = isctest.query.udp(msg, ns1.ip) + isctest.check.noerror(response) +``` + +This creates the test cases `test_answers[exists.example.-A]` etc., which can +be passed to pytest as node IDs. + ### Logging Each module has a separate log which will be saved as pytest.log.txt in the @@ -212,56 +378,27 @@ temporary directory in which the test is executed. This log includes messages for this module setup/teardown as well as any logging from the tests. Logging level DEBUG and above will be present in this log. -In general, any log messages using INFO or above will also be printed out -during pytest execution. In CI, the pytest output is also saved to -pytest.out.txt in the bin/tests/system directory. +Use `isctest.log` for test output (`isctest.log.info("...")` etc.); in +general, any log messages using INFO or above will also be printed out during +pytest execution. In CI, the pytest output is also saved to pytest.out.txt in +the bin/tests/system directory. ### Adding a Test to the System Test Suite Once a test has been created it will be automatically picked up by the pytest -runner if it upholds the convention expected by pytest (especially when it comes -to naming files and test functions). - -However, if a new system test directory is created, it also needs to be added to -`TESTS` in `Makefile.am`, in order to work with `make check`. - - -## Test Files - -### setup.sh - -This script is responsible for setting up the configuration files used in the -test. It is used by both the python and shell tests. It is interpreted just -before the servers are started up for each test module. - -### tests_*.py - -These are test modules containing tests written in python. Every test is a -function which begins with the name `test_` (according to pytest convention). It -is possible to pass fixtures to the test function by specifying their name as -function arguments. Fixtures are used to provide context to the tests, e.g.: - -- `ports` is a dictionary with assigned port numbers - -### tests_sh_*.py - -These are glue files that are required to execute shell based tests (see below). -These modules shouldn't contain any python tests (use a separate file instead). - -### tests.sh - -This is the test file for shell based tests. +runner if it upholds the convention expected by pytest (especially when it +comes to naming files and test functions). New system test directories are +discovered automatically; no registration in the build system is needed. ## Nameservers As noted earlier, a system test will involve a number of nameservers. These -will be either instances of named, or special servers written in a language -such as Perl or Python. +will be either instances of named, or mock servers, typically written in +Python. -For the former, the version of "named" being run is that in the "bin/named" -directory in the tree holding the tests (i.e. if "make test" is being run -immediately after "make", the version of "named" used is that just built). The +For the former, the version of "named" being run is the one from the build +directory registered by the most recent `meson compile` invocation. The configuration files, zone files etc. for these servers are located in subdirectories of the test directory named "nsN", where N is a small integer. The latter are special nameservers, mostly used for generating deliberately bad @@ -281,66 +418,71 @@ starts those servers. By default, `named` server is started with the following options: - -c named.conf Specifies the configuration file to use (so by implication, - each "nsN" nameserver's configuration file must be called - named.conf). +``` +-c named.conf Specifies the configuration file to use (so by implication, + each "nsN" nameserver's configuration file must be called + named.conf). - -d 99 Sets the maximum debugging level. +-d 99 Sets the maximum debugging level. - -D The "-D" option sets a string used to identify the - nameserver in a process listing. In this case, the string - is the name of the subdirectory. +-D The "-D" option sets a string used to identify the + nameserver in a process listing. In this case, the string + is the name of the subdirectory. - -g Runs the server in the foreground and logs everything to - stderr. +-g Runs the server in the foreground and logs everything to + stderr. - -m record - Turns on these memory usage debugging flags. +-m record + Turns on these memory usage debugging flags. +``` All output is sent to a file called `named.run` in the nameserver directory. The options used to start named can be altered. There are a couple ways of -doing this. `start.pl` checks the methods in a specific order: if a check -succeeds, the options are set and any other specification is ignored. In order, -these are: +doing this. The runner checks the methods in a specific order: if a check +succeeds, the options are set and any other specification is ignored. In +order, these are: -1. Specifying options to `start.pl` or `start_server` shell utility function - after the name of the test directory, e.g. - - start_server --noclean --restart --port ${PORT} ns1 -- "-D xfer-ns1 -T transferinsecs -T transferslowly" - -2. Including a file called "named.args" in the "nsN" directory. If present, +1. Including a file called "named.args" in the "nsN" directory. If present, the contents of the first non-commented, non-blank line of the file are used as the named command-line arguments. The rest of the file is ignored. -3. Tweaking the default command line arguments with "-T" options. This flag is +2. Tweaking the default command line arguments with "-T" options. This flag is used to alter the behavior of BIND for testing and is not documented in the -ARM. The presence of certain files in the "nsN" directory adds flags to -the default command line (the content of the files is irrelevant - it -is only the presence that counts): +ARM. The presence of a file called `named.` in the "nsN" directory adds +`-T ` to the default command line (the content of the file is irrelevant +- it is only the presence that counts). The recognized flags are: - named.noaa Appends "-T noaa" to the command line, which causes - "named" to never set the AA bit in an answer. +``` +dropedns Recognise EDNS options in messages, but drop messages + containing them. - named.dropedns Adds "-T dropedns" to the command line, which causes - "named" to recognise EDNS options in messages, but drop - messages containing them. +ednsformerr, ednsnotimp, ednsrefused + Answer EDNS queries with the given rcode, pretending to + be an old server that doesn't understand EDNS. - named.maxudp1460 Adds "-T maxudp1460" to the command line, setting the - maximum UDP size handled by named to 1460. +cookiealwaysvalid + Accept any DNS cookie presented by a client. - named.maxudp512 Adds "-T maxudp512" to the command line, setting the - maximum UDP size handled by named to 512. +noaa Never set the AA bit in an answer. - named.noedns Appends "-T noedns" to the command line, which disables - recognition of EDNS options in messages. +noedns Disable recognition of EDNS options in messages. - named.notcp Adds "-T notcp", which disables TCP in "named". +nonearest Omit the closest-encloser NSEC3 proof from negative + responses (except for DS queries). - named.soa Appends "-T nosoa" to the command line, which disables - the addition of SOA records to negative responses (or to - the additional section if the response is triggered by RPZ - rewriting). +nosoa Disable the addition of SOA records to negative + responses (or to the additional section if the response + is triggered by RPZ rewriting). + +maxudp512, maxudp1460 + Set the maximum UDP size handled by named to 512/1460. + +tat=1, tat=3 Send trust-anchor-telemetry queries every N seconds. + +notcp Disable TCP in "named". Unlike the other flags, this + one is also applied when "named.args" is used. +``` ### Running Nameservers Interactively @@ -349,15 +491,17 @@ setup and interact with the servers before the test starts, or even at specific points during the test, using the `--trace` option to drop you into pdb debugger which pauses the execution of the tests, while keeping the server state intact: - pytest -k dns64 --trace +```sh +pytest -k dns64 --trace +``` ## Developer Notes ### Test discovery and collection -There are two distinct types of system tests. The first is a shell script -tests.sh containing individual test cases executed sequentially and the +There are two distinct types of system tests. The first is a legacy shell +script tests.sh containing individual test cases executed sequentially and the success/failure is determined by return code. The second type is a regular pytest file which contains test functions. @@ -374,52 +518,53 @@ complex solution. ### Compatibility with older pytest version -Keep in mind that the pytest runner must work with ancient versions of pytest. -When implementing new features, it is advisable to check feature support in -pytest and pytest-xdist in older distributions first. - -As a general rule, any changes to the pytest runner need to keep working on all -platforms in CI that use the pytest runner. As of 2023-11-14, the oldest -supported version is whatever is available in EL8. - -We may need to add more compat code eventually to handle breaking upstream -changes. For example, using request.fspath attribute is already deprecated in -latest pytest. +The minimum supported versions of python and the required python packages are +declared in [requirements.txt](requirements.txt) and in the +`pytest_configure()` check in `conftest.py`. When implementing new runner +features, check feature support in the pytest and pytest-xdist versions +available in the oldest distributions covered by CI first; we may need to add +compat code to handle breaking upstream changes in either direction. ### Format of Shell Test Output -Shell-based tests have the following format of output: +Legacy shell-based tests have the following format of output: - :: [()] +``` +:: [()] +``` e.g. - I:catz:checking that dom1.example is not served by primary (1) +``` +I:catz:checking that dom1.example is not served by primary (1) +``` The meanings of the fields are as follows: This indicates the type of message. This is one of: - S Start of the test - A Start of test (retained for backwards compatibility) - T Start of test (retained for backwards compatibility) - E End of the test - I Information. A test will typically output many of these messages - during its run, indicating test progress. Note that such a message may - be of the form "I:testname:failed", indicating that a sub-test has - failed. - R Result. Each test will result in one such message, which is of the - form: +``` +S Start of the test +A Start of test (retained for backwards compatibility) +T Start of test (retained for backwards compatibility) +E End of the test +I Information. A test will typically output many of these messages + during its run, indicating test progress. Note that such a message may + be of the form "I:testname:failed", indicating that a sub-test has + failed. +R Result. Each test will result in one such message, which is of the + form: - R:: + R:: - where is one of: + where is one of: - PASS The test passed - FAIL The test failed - SKIPPED The test was not run, usually because some - prerequisites required to run the test are missing. + PASS The test passed + FAIL The test failed + SKIPPED The test was not run, usually because some + prerequisites required to run the test are missing. +``` This is the name of the temporary test directory from which the message @@ -433,8 +578,9 @@ If present, this will correlate with a file created by the test. The tests execute commands and route the output of each command to a file. The name of this file depends on the command and the test, but will usually be of the form: - .out. +``` +.out. +``` e.g. nsupdate.out.test28, dig.out.q3. This aids diagnosis of problems by allowing the output that caused the problem message to be identified. - diff --git a/bin/tests/system/requirements.txt b/bin/tests/system/requirements.txt index 0bd1e00fc8..df85e9bcdf 100644 --- a/bin/tests/system/requirements.txt +++ b/bin/tests/system/requirements.txt @@ -7,8 +7,8 @@ h2 hypothesis>=4.41.2 jinja2 pytest>=7.0.0 +pytest-xdist requests -### Utility packages for executing the tests +### Optional packages flaky -pytest-xdist