upstream/bind9
1
0
Fork 0
mirror of https://github.com/isc-projects/bind9.git synced 2026-06-13 20:30:00 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

chg: nil: Update the system test README for the pytest-native workflow

Browse source 
The README predated the meson migration and most of the pytest runner
features. Document building the test dependencies with meson and drop
the make-based instructions, the Makefile.am registration step, and the
stale -T flag list. Describe the jinja2 templating, bootstrap(), the
conftest fixtures, the pytest marks, and recommend node IDs and
parametrization over -k matching. Fix the directory naming rule, which
switched from hyphens to underscores.

Also declare pytest and pytest-xdist as required dependencies: the
runner's pytest.ini uses --dist=loadscope unconditionally, so pytest
without pytest-xdist cannot even start.

Related #3810

Assisted-by: Claude:claude-fable-5

Merge branch 'nicki/systest-readme-refresh' into 'main'

See merge request isc-projects/bind9!12232
This commit is contained in:
Nicki Křížek 2026-06-11 15:22:49 +02:00
commit eec23bbe29
2 changed files with 327 additions and 181 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

504
bin/tests/system/README.md
View file

@ -16,24 +16,28 @@ information regarding copyright ownership.
This directory holds test environments for running bind9 system tests involving
multiple name servers.
Each system test directory holds a set of scripts and configuration files to
test different parts of BIND. The directories are named for the aspect of BIND
they test, for example:
Each system test directory holds a set of test scripts and configuration files
to test different parts of BIND. The directories are named for the aspect of
BIND they test, for example:
dnssec/ DNSSEC tests
forward/ Forwarding tests
glue/ Glue handling tests
```
dnssec/ DNSSEC tests
forward/ Forwarding tests
glue/ Glue handling tests
```
etc.
A system test directory must start with an alphabetic character and may not
contain any special characters. Only hyphen may be used as a word separator.
A system test directory name must start with an alphabetic character and may
only contain alphanumeric characters and underscores. Use underscore as the
word separator; hyphens are not allowed — they are reserved for the temporary
directories and symlinks the test runner creates.
Typically each set of tests sets up 2-5 name servers and then performs one or
more tests against them. Within the test subdirectory, each name server has a
separate subdirectory containing its configuration data. These subdirectories
are named "nsN" or "ansN" (where N is a number between 1 and 8, e.g. ns1, ans2
etc.)
are named "nsN" or "ansN" (where N is a number between 1 and 11, e.g. ns1,
ans2 etc.)
The tests are completely self-contained and do not require access to the real
DNS. Generally, one of the test servers (usually ns1) is set up as a root
@ -42,15 +46,37 @@ nameserver and is listed in the hints file of the others.
## Running the Tests
### Building BIND
The system tests run the binaries from the build directory, so BIND must be
built first, along with a few test-only binaries and plugins:
```sh
meson setup build
meson compile -C build
meson compile -C build system-test-dependencies
```
When BIND is configured with `-Ddeveloper=enabled`, the test dependencies are
built by default and the last step is not needed.
Every `meson compile` invocation re-registers its build directory with the
source tree, so the tests always use the binaries from the most recently
compiled build directory.
### Prerequisites
To run system tests, make sure you have the following dependencies installed:
- python3 (3.10 and newer)
- perl
- pytest (7.0 and newer)
- pytest-xdist
- perl (still needed by the test runner internals; some legacy tests
additionally need the Net::DNS module and are skipped when it is missing)
List of required python packages and their versions can be found in
requirements.txt (can be installed with `pip3 install -r requirements.txt`).
The full list of required and optional python packages can be found in
[requirements.txt](requirements.txt) (it can be installed with
`pip3 install -r requirements.txt`).
### Network Setup
@ -59,11 +85,15 @@ IP addresses on the loopback interface. ns1 runs on 10.53.0.1, ns2 on
10.53.0.2, etc. Before running any tests, you must set up these addresses by
running the command
sh ifconfig.sh up
```sh
sh ifconfig.sh up
```
as root. The interfaces can be removed by executing the command:
sh ifconfig.sh down
```sh
sh ifconfig.sh down
```
... also as root.
@ -77,45 +107,65 @@ If you wish to make the interfaces survive across reboots, copy
org.isc.bind.system and org.isc.bind.system.plist to /Library/LaunchDaemons
then run
launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist
```sh
launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist
```
... as root.
### Running a Single Test
The recommended way is to use pytest and its test selection facilities:
pytest -k <test-name-or-pattern>
Using `-k` to specify a pattern allows to run a single pytest test case within
a system test. E.g. you can use `-k test_sslyze_dot` to execute just the
`test_sslyze_dot()` function from `doth/tests_sslyze.py`.
However, using the `-k` pattern might pick up more tests than intended. You can
use the `--collect-only` option to check the list of tests which match you `-k`
pattern. If you just want to execute all system tests within a single test
directory, you can also use the utility script:
./run.sh system_test_dir_name
### Running All the System Tests
Issuing plain `pytest` command without any argument will execute all tests
sequentially. To execute them in parallel, ensure you have pytest-xdist
installed and run:
Issue a plain `pytest` command in this directory to execute all tests
sequentially. To execute them in parallel instead, run:
pytest [-n <number-of-workers>]
```sh
pytest -n <number-of-workers>
```
Alternately, using the make command is also supported:
Parallel execution requires pytest-xdist; `-n auto` uses one worker per CPU.
make [-j numproc] test
### Running a Single Test
To run all test modules in a single system test directory, pass the directory
name to pytest:
```sh
pytest dns64
```
The utility script `./run.sh dns64` does the same thing.
To narrow the run down further, prefer pytest node IDs over `-k` matching —
they are exact:
```sh
pytest dnssec_py/tests_mixed_ds.py
pytest doth/tests_sslyze.py::test_sslyze_dot
```
Parametrized tests have the parameter ID in brackets, so a single case of a
parametrized test can be selected as:
```sh
pytest "dnssec_py/tests_nsec3_answer.py::test_nodata[ns2]"
```
The `-k` option selects tests by pattern matching:
```sh
pytest -k <test-name-or-pattern>
```
Beware that a `-k` pattern might pick up more tests than intended. Use the
`--collect-only` option to check the list of tests which match your `-k`
pattern.
### rr
When running system tests, named can be run under the rr tool. rr records a
trace to the $system_test/nsX/named-Y/ directory, which can be later used to
replay named. To enable this, execute start.pl with the USE_RR environment
variable set.
replay named. To enable this, run pytest with the USE_RR environment variable
set.
### Test Artifacts
@ -147,22 +197,8 @@ system test directories may contain the following standard files:
- `tests_*.py`: These python files are picked up by pytest as modules. If they
contain any test functions, they're added to the test suite.
- `*.j2`: These jinja2 templates can be used for configuration files or any
other files which require certain variables filled in, e.g. ports from the
environment variables. During test setup, the pytest runner will automatically
fill those in and strip the filename extension .j2, e.g. `ns1/named.conf.j2`
becomes `ns1/named.conf`. When using advanced templating to conditionally
include/omit entire sections or when filling in custom variables used for the
test, ensure the templates always include the defaults. If you don't need the
file to be auto-templated during test setup, use `.j2.manual` instead and then
no defaults are needed.
- `setup.sh`: This sets up the preconditions for the tests.
- `tests.sh`: Any shell-based tests are located within this file. Runs the
actual tests.
- `tests_sh_*.py`: A glue file for the pytest runner for executing shell tests.
- `*.j2`: Jinja2 templates, rendered automatically during test setup (see
[Templates](#templates) below).
- `ns<N>`: These subdirectories contain test name servers that can be queried
or can interact with each other. The value of N indicates the address the
@ -171,10 +207,23 @@ system test directories may contain the following standard files:
run as root. These servers log at the highest debug level and the log is
captured in the file "named.run".
- `ans<N>`: Like ns[X], but these are simple mock name servers implemented in
Perl or Python. They are generally programmed to misbehave in ways named
would not so as to exercise named's ability to interoperate with badly
behaved name servers.
- `ans<N>`: Like ns<N>, but these are mock name servers implemented in python
(`ans.py`), usually with the `isctest.asyncserver` module. They are
generally programmed to misbehave in ways named would not, so as to exercise
named's ability to interoperate with badly behaved name servers. A few
legacy mock servers are still implemented in perl (`ans.pl`); don't write
new ones.
The following files appear in test directories that have not yet been fully
ported to python; do not add them to new tests:
- `tests.sh`: Legacy shell-based tests, run via a `tests_sh_*.py` glue module.
- `setup.sh`: Legacy shell test setup. New tests use templates and a
`bootstrap()` function instead.
- `prereq.sh`: Legacy prerequisite check; when it exits non-zero, the test is
skipped. New tests use pytest marks (see `isctest/mark.py`).
### Module Scope
@ -196,15 +245,132 @@ In order for the tests to run in parallel, each test requires a unique set of
ports. This is ensured by the pytest runner, which assigns a unique set of
ports to each test module.
Inside the python tests, it is possible to use the `ports` fixture to get the
assigned port numbers. They're also set as environment variables. These include:
Inside the python tests, it is possible to use fixtures like `named_port` to
get the assigned port numbers. They're also set as environment variables.
These include:
- `PORT`: used as the basic dns port
- `TLSPORT`: used as the port for DNS-over-TLS
- `HTTPPORT`, `HTTPSPORT`: used as the ports for DNS-over-HTTP
- `HTTPPORT`, `HTTPSPORT`: used as the ports for DNS-over-HTTP(S)
- `CONTROLPORT`: used as the RNDC control port
- `EXTRAPORT1` through `EXTRAPORT8`: additional ports that can be used as needed
### Templates
Configuration files which need values that are only known at test run time —
ports, default crypto algorithms, conditional sections — are written as jinja2
templates with a `.j2` extension. During test setup, the pytest runner
renders every `*.j2` file in the test directory and strips the extension:
`ns1/named.conf.j2` becomes `ns1/named.conf`.
Inside a template, all the runner's environment variables are available with
`@...@` delimiters, e.g.:
```jinja
options {
port @PORT@;
listen-on { 10.53.0.1; };
};
key rndc_key {
secret "1234abcd8765";
algorithm @DEFAULT_HMAC@;
};
controls {
inet 10.53.0.1 port @CONTROLPORT@ allow { any; } keys { rndc_key; };
};
```
Standard jinja2 block syntax (`{% if %}`, `{% for %}`, …) can be used for
conditional or repeated sections.
Custom template variables come from an optional module-level `bootstrap()`
function. When a test module defines one, the runner calls it before
rendering the templates and passes the returned dict to the template engine:
```python
def bootstrap():
return {"valid": True}
```
Templates using custom variables must always provide defaults, so that the
file also renders when no value is supplied (e.g. when another module in the
same directory has no `bootstrap()`):
```jinja
{% set valid = valid | default(False) %}
```
`bootstrap()` is also the place where a module generates test data that has
to exist before the servers start — typically zone files and DNSSEC keys.
Templates can also be re-rendered while the test is running, using the
`templates` fixture, e.g. to change a server's config before reloading it:
```python
def test_reload(ns1, templates):
templates.render("ns1/named.conf", {"valid": True})
ns1.reconfigure()
```
If you don't need the file to be auto-templated during test setup, use the
extension `.j2.manual` instead; such templates are only rendered when the test
calls `templates.render()` explicitly, and no defaults are needed.
### Fixtures and Helpers
Fixtures defined in `conftest.py` provide the test context:
- `servers` is a dictionary of all started `isctest.instance.NamedInstance`
servers, keyed by directory name; the shortcut fixtures `ns1` through `ns11`
return the corresponding instance directly. A `NamedInstance` is the
interface for driving a server: `ns1.rndc("...")`, `ns1.reconfigure()`,
`ns1.nsupdate(...)`, `ns1.watch_log_from_here()`, `ns1.ip`, ...
- `templates` renders jinja2 templates at runtime (see above).
- `named_port`, `named_tlsport`, `control_port`, ... return the assigned port
numbers.
- `system_test_dir` is the temporary directory the module runs in.
The `isctest` package provides the helper library; the modules most tests
need are `isctest.query` (send DNS queries), `isctest.check` (assert on
responses), `isctest.zone` (zone and key setup), `isctest.kasp` (DNSSEC
key state checks), `isctest.asyncserver` (mock servers) and `isctest.log`
(logging and log watchers).
Pytest marks control test collection and setup:
- `@pytest.mark.extra_artifacts([...])` declares the files (globs) the test is
expected to leave behind in addition to the common ones; undeclared
leftovers fail the run.
- `@pytest.mark.requires_zones_loaded("ns1", ...)` delays the test until the
listed servers have loaded all zones.
- `isctest.mark` has skip-unless conditions for environment prerequisites,
e.g. `isctest.mark.with_dnstap`, `isctest.mark.softhsm2_environment`,
`isctest.mark.live_internet_test`.
### Parametrization
Use `pytest.mark.parametrize` to run one test function over several inputs
instead of copy-pasting test cases or looping inside one test function — each
case is reported (and can be re-run) individually:
```python
@pytest.mark.parametrize(
"qname,rdtype",
[
("exists.example.", "A"),
("exists.example.", "TXT"),
("other.example.", "A"),
],
)
def test_answers(qname, rdtype, ns1):
msg = isctest.query.create(qname, rdtype)
response = isctest.query.udp(msg, ns1.ip)
isctest.check.noerror(response)
```
This creates the test cases `test_answers[exists.example.-A]` etc., which can
be passed to pytest as node IDs.
### Logging
Each module has a separate log which will be saved as pytest.log.txt in the
@ -212,56 +378,27 @@ temporary directory in which the test is executed. This log includes messages
for this module setup/teardown as well as any logging from the tests. Logging
level DEBUG and above will be present in this log.
In general, any log messages using INFO or above will also be printed out
during pytest execution. In CI, the pytest output is also saved to
pytest.out.txt in the bin/tests/system directory.
Use `isctest.log` for test output (`isctest.log.info("...")` etc.); in
general, any log messages using INFO or above will also be printed out during
pytest execution. In CI, the pytest output is also saved to pytest.out.txt in
the bin/tests/system directory.
### Adding a Test to the System Test Suite
Once a test has been created it will be automatically picked up by the pytest
runner if it upholds the convention expected by pytest (especially when it comes
to naming files and test functions).
However, if a new system test directory is created, it also needs to be added to
`TESTS` in `Makefile.am`, in order to work with `make check`.
## Test Files
### setup.sh
This script is responsible for setting up the configuration files used in the
test. It is used by both the python and shell tests. It is interpreted just
before the servers are started up for each test module.
### tests_*.py
These are test modules containing tests written in python. Every test is a
function which begins with the name `test_` (according to pytest convention). It
is possible to pass fixtures to the test function by specifying their name as
function arguments. Fixtures are used to provide context to the tests, e.g.:
- `ports` is a dictionary with assigned port numbers
### tests_sh_*.py
These are glue files that are required to execute shell based tests (see below).
These modules shouldn't contain any python tests (use a separate file instead).
### tests.sh
This is the test file for shell based tests.
runner if it upholds the convention expected by pytest (especially when it
comes to naming files and test functions). New system test directories are
discovered automatically; no registration in the build system is needed.
## Nameservers
As noted earlier, a system test will involve a number of nameservers. These
will be either instances of named, or special servers written in a language
such as Perl or Python.
will be either instances of named, or mock servers, typically written in
Python.
For the former, the version of "named" being run is that in the "bin/named"
directory in the tree holding the tests (i.e. if "make test" is being run
immediately after "make", the version of "named" used is that just built). The
For the former, the version of "named" being run is the one from the build
directory registered by the most recent `meson compile` invocation. The
configuration files, zone files etc. for these servers are located in
subdirectories of the test directory named "nsN", where N is a small integer.
The latter are special nameservers, mostly used for generating deliberately bad
@ -281,66 +418,71 @@ starts those servers.
By default, `named` server is started with the following options:
-c named.conf Specifies the configuration file to use (so by implication,
each "nsN" nameserver's configuration file must be called
named.conf).
```
-c named.conf Specifies the configuration file to use (so by implication,
each "nsN" nameserver's configuration file must be called
named.conf).
-d 99 Sets the maximum debugging level.
-d 99 Sets the maximum debugging level.
-D <name> The "-D" option sets a string used to identify the
nameserver in a process listing. In this case, the string
is the name of the subdirectory.
-D <name> The "-D" option sets a string used to identify the
nameserver in a process listing. In this case, the string
is the name of the subdirectory.
-g Runs the server in the foreground and logs everything to
stderr.
-g Runs the server in the foreground and logs everything to
stderr.
-m record
Turns on these memory usage debugging flags.
-m record
Turns on these memory usage debugging flags.
```
All output is sent to a file called `named.run` in the nameserver directory.
The options used to start named can be altered. There are a couple ways of
doing this. `start.pl` checks the methods in a specific order: if a check
succeeds, the options are set and any other specification is ignored. In order,
these are:
doing this. The runner checks the methods in a specific order: if a check
succeeds, the options are set and any other specification is ignored. In
order, these are:
1. Specifying options to `start.pl` or `start_server` shell utility function
after the name of the test directory, e.g.
start_server --noclean --restart --port ${PORT} ns1 -- "-D xfer-ns1 -T transferinsecs -T transferslowly"
2. Including a file called "named.args" in the "nsN" directory. If present,
1. Including a file called "named.args" in the "nsN" directory. If present,
the contents of the first non-commented, non-blank line of the file are used as
the named command-line arguments. The rest of the file is ignored.
3. Tweaking the default command line arguments with "-T" options. This flag is
2. Tweaking the default command line arguments with "-T" options. This flag is
used to alter the behavior of BIND for testing and is not documented in the
ARM. The presence of certain files in the "nsN" directory adds flags to
the default command line (the content of the files is irrelevant - it
is only the presence that counts):
ARM. The presence of a file called `named.<flag>` in the "nsN" directory adds
`-T <flag>` to the default command line (the content of the file is irrelevant
- it is only the presence that counts). The recognized flags are:
named.noaa Appends "-T noaa" to the command line, which causes
"named" to never set the AA bit in an answer.
```
dropedns Recognise EDNS options in messages, but drop messages
containing them.
named.dropedns Adds "-T dropedns" to the command line, which causes
"named" to recognise EDNS options in messages, but drop
messages containing them.
ednsformerr, ednsnotimp, ednsrefused
Answer EDNS queries with the given rcode, pretending to
be an old server that doesn't understand EDNS.
named.maxudp1460 Adds "-T maxudp1460" to the command line, setting the
maximum UDP size handled by named to 1460.
cookiealwaysvalid
Accept any DNS cookie presented by a client.
named.maxudp512 Adds "-T maxudp512" to the command line, setting the
maximum UDP size handled by named to 512.
noaa Never set the AA bit in an answer.
named.noedns Appends "-T noedns" to the command line, which disables
recognition of EDNS options in messages.
noedns Disable recognition of EDNS options in messages.
named.notcp Adds "-T notcp", which disables TCP in "named".
nonearest Omit the closest-encloser NSEC3 proof from negative
responses (except for DS queries).
named.soa Appends "-T nosoa" to the command line, which disables
the addition of SOA records to negative responses (or to
the additional section if the response is triggered by RPZ
rewriting).
nosoa Disable the addition of SOA records to negative
responses (or to the additional section if the response
is triggered by RPZ rewriting).
maxudp512, maxudp1460
Set the maximum UDP size handled by named to 512/1460.
tat=1, tat=3 Send trust-anchor-telemetry queries every N seconds.
notcp Disable TCP in "named". Unlike the other flags, this
one is also applied when "named.args" is used.
```
### Running Nameservers Interactively
@ -349,15 +491,17 @@ setup and interact with the servers before the test starts, or even at specific
points during the test, using the `--trace` option to drop you into pdb debugger
which pauses the execution of the tests, while keeping the server state intact:
pytest -k dns64 --trace
```sh
pytest -k dns64 --trace
```
## Developer Notes
### Test discovery and collection
There are two distinct types of system tests. The first is a shell script
tests.sh containing individual test cases executed sequentially and the
There are two distinct types of system tests. The first is a legacy shell
script tests.sh containing individual test cases executed sequentially and the
success/failure is determined by return code. The second type is a regular
pytest file which contains test functions.
@ -374,52 +518,53 @@ complex solution.
### Compatibility with older pytest version
Keep in mind that the pytest runner must work with ancient versions of pytest.
When implementing new features, it is advisable to check feature support in
pytest and pytest-xdist in older distributions first.
As a general rule, any changes to the pytest runner need to keep working on all
platforms in CI that use the pytest runner. As of 2023-11-14, the oldest
supported version is whatever is available in EL8.
We may need to add more compat code eventually to handle breaking upstream
changes. For example, using request.fspath attribute is already deprecated in
latest pytest.
The minimum supported versions of python and the required python packages are
declared in [requirements.txt](requirements.txt) and in the
`pytest_configure()` check in `conftest.py`. When implementing new runner
features, check feature support in the pytest and pytest-xdist versions
available in the oldest distributions covered by CI first; we may need to add
compat code to handle breaking upstream changes in either direction.
### Format of Shell Test Output
Shell-based tests have the following format of output:
Legacy shell-based tests have the following format of output:
<letter>:<test-name>:<message> [(<number>)]
```
<letter>:<test-name>:<message> [(<number>)]
```
e.g.
I:catz:checking that dom1.example is not served by primary (1)
```
I:catz:checking that dom1.example is not served by primary (1)
```
The meanings of the fields are as follows:
<letter>
This indicates the type of message. This is one of:
S Start of the test
A Start of test (retained for backwards compatibility)
T Start of test (retained for backwards compatibility)
E End of the test
I Information. A test will typically output many of these messages
during its run, indicating test progress. Note that such a message may
be of the form "I:testname:failed", indicating that a sub-test has
failed.
R Result. Each test will result in one such message, which is of the
form:
```
S Start of the test
A Start of test (retained for backwards compatibility)
T Start of test (retained for backwards compatibility)
E End of the test
I Information. A test will typically output many of these messages
during its run, indicating test progress. Note that such a message may
be of the form "I:testname:failed", indicating that a sub-test has
failed.
R Result. Each test will result in one such message, which is of the
form:
R:<test-tmpdir>:<result>
R:<test-tmpdir>:<result>
where <result> is one of:
where <result> is one of:
PASS The test passed
FAIL The test failed
SKIPPED The test was not run, usually because some
prerequisites required to run the test are missing.
PASS The test passed
FAIL The test failed
SKIPPED The test was not run, usually because some
prerequisites required to run the test are missing.
```
<test-tmpdir>
This is the name of the temporary test directory from which the message
@ -433,8 +578,9 @@ If present, this will correlate with a file created by the test. The tests
execute commands and route the output of each command to a file. The name of
this file depends on the command and the test, but will usually be of the form:
<command>.out.<suffix><number>
```
<command>.out.<suffix><number>
```
e.g. nsupdate.out.test28, dig.out.q3. This aids diagnosis of problems by
allowing the output that caused the problem message to be identified.

4
bin/tests/system/requirements.txt
View file

@ -7,8 +7,8 @@ h2
hypothesis>=4.41.2
jinja2
pytest>=7.0.0
pytest-xdist
requests
### Utility packages for executing the tests
### Optional packages
flaky
pytest-xdist