bind9/bin/tests/system/README.md

BIND9 System Test Framework

This directory holds test environments for running bind9 system tests involving multiple name servers.

Each system test directory holds a set of test scripts and configuration files to test different parts of BIND. The directories are named for the aspect of BIND they test, for example:

dnssec/       DNSSEC tests
forward/      Forwarding tests
glue/         Glue handling tests

etc.

A system test directory name must start with an alphabetic character and may only contain alphanumeric characters and underscores. Use underscore as the word separator; hyphens are not allowed — they are reserved for the temporary directories and symlinks the test runner creates.

Typically each set of tests sets up 2-5 name servers and then performs one or more tests against them. Within the test subdirectory, each name server has a separate subdirectory containing its configuration data. These subdirectories are named "nsN" or "ansN" (where N is a number between 1 and 11, e.g. ns1, ans2 etc.)

The tests are completely self-contained and do not require access to the real DNS. Generally, one of the test servers (usually ns1) is set up as a root nameserver and is listed in the hints file of the others.

For task-oriented recipes (adding a new test directory, writing a regression reproducer, mocking a misbehaving server, setting up zones), see the cookbook.

Running the Tests

Building BIND

The system tests run the binaries from the build directory, so BIND must be built first, along with a few test-only binaries and plugins:

meson setup build
meson compile -C build
meson compile -C build system-test-dependencies

When BIND is configured with -Ddeveloper=enabled, the test dependencies are built by default and the last step is not needed.

Every meson compile invocation re-registers its build directory with the source tree, so the tests always use the binaries from the most recently compiled build directory.

Prerequisites

To run system tests, make sure you have the following dependencies installed:

• python3 (3.10 and newer)
• pytest (7.0 and newer)
• pytest-xdist
• perl (still needed by the test runner internals; some legacy tests additionally need the Net::DNS module and are skipped when it is missing)

The full list of required and optional python packages can be found in requirements.txt (it can be installed with pip3 install -r requirements.txt).

Network Setup

To enable all servers to run on the same machine, they bind to separate virtual IP addresses on the loopback interface. ns1 runs on 10.53.0.1, ns2 on 10.53.0.2, etc. Before running any tests, you must set up these addresses by running the command

sh ifconfig.sh up

as root. The interfaces can be removed by executing the command:

sh ifconfig.sh down

... also as root.

The servers use unprivileged ports (above 1024) instead of the usual port 53, so they can be run without root privileges once the interfaces have been set up.

Note for MacOS Users

If you wish to make the interfaces survive across reboots, copy org.isc.bind.system and org.isc.bind.system.plist to /Library/LaunchDaemons then run

launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist

... as root.

Running All the System Tests

Issue a plain pytest command in this directory to execute all tests sequentially. To execute them in parallel instead, run:

pytest -n <number-of-workers>

Parallel execution requires pytest-xdist; -n auto uses one worker per CPU.

Running a Single Test

To run all test modules in a single system test directory, pass the directory name to pytest:

pytest dns64

The utility script ./run.sh dns64 does the same thing.

To narrow the run down further, prefer pytest node IDs over -k matching — they are exact:

pytest dnssec_py/tests_mixed_ds.py
pytest doth/tests_sslyze.py::test_sslyze_dot

Parametrized tests have the parameter ID in brackets, so a single case of a parametrized test can be selected as:

pytest "dnssec_py/tests_nsec3_answer.py::test_nodata[ns2]"

The -k option selects tests by pattern matching:

pytest -k <test-name-or-pattern>

Beware that a -k pattern might pick up more tests than intended. Use the --collect-only option to check the list of tests which match your -k pattern.

rr

When running system tests, named can be run under the rr tool. rr records a trace to the $system_test/nsX/named-Y/ directory, which can be later used to replay named. To enable this, run pytest with the USE_RR environment variable set.

Test Artifacts

Each test module is executed inside a unique temporary directory which contains all the artifacts from the test run. If the tests succeed, they are deleted by default. To override this behaviour, pass --noclean to pytest.

The directory name starts with the system test name, followed by -tmp-XXXXXX, i.e. dns64-tmp-r07vei9s for dns64 test run. Since this name changes each run, a convenience symlink that has a stable name is also created. It points to the latest test artifacts directory and has a form of dns64-sh_dns64 (depending on the particular test module).

The following test artifacts are typically available:

• pytest.log.txt: main log file with test output
• files generated by the test itself, e.g. output from "dig" and "rndc"
• files produced by named, other tools or helper scripts

Writing System Tests

File Overview

Tests are organized into system test directories which may hold one or more test modules (python files). Each module may have multiple test cases. The system test directories may contain the following standard files:

• tests_*.py: These python files are picked up by pytest as modules. If they contain any test functions, they're added to the test suite.

• *.j2: Jinja2 templates, rendered automatically during test setup (see Templates below).

• ns<N>: These subdirectories contain test name servers that can be queried or can interact with each other. The value of N indicates the address the server listens on: for example, ns2 listens on 10.53.0.2, and ns4 on 10.53.0.4. All test servers use an unprivileged port, so they don't need to run as root. These servers log at the highest debug level and the log is captured in the file "named.run".

• ans<N>: Like ns, but these are mock name servers implemented in python (ans.py), usually with the isctest.asyncserver module. They are generally programmed to misbehave in ways named would not, so as to exercise named's ability to interoperate with badly behaved name servers. A few legacy mock servers are still implemented in perl (ans.pl); don't write new ones.

The following files appear in test directories that have not yet been fully ported to python; do not add them to new tests:

• tests.sh: Legacy shell-based tests, run via a tests_sh_*.py glue module.

• setup.sh: Legacy shell test setup. New tests use templates and a bootstrap() function instead.

• prereq.sh: Legacy prerequisite check; when it exits non-zero, the test is skipped. New tests use pytest marks (see isctest/mark.py).

Module Scope

A module is a python file which contains test functions. Every system test directory may contain multiple modules (i.e. tests_*.py files).

The server setup/teardown is performed for each module. Bundling test cases together inside a single module may save some resources. However, test cases inside a single module can't be executed in parallel.

It is possible to execute different modules defined within a single system test directory in parallel. This is possible thanks to executing the tests inside a temporary directory and proper port assignment to ensure there won't be any conflicts.

Port Usage

In order for the tests to run in parallel, each test requires a unique set of ports. This is ensured by the pytest runner, which assigns a unique set of ports to each test module.

Inside the python tests, it is possible to use fixtures like named_port to get the assigned port numbers. They're also set as environment variables. These include:

• PORT: used as the basic dns port
• TLSPORT: used as the port for DNS-over-TLS
• HTTPPORT, HTTPSPORT: used as the ports for DNS-over-HTTP(S)
• CONTROLPORT: used as the RNDC control port
• EXTRAPORT1 through EXTRAPORT8: additional ports that can be used as needed

Templates

Configuration files which need values that are only known at test run time — ports, default crypto algorithms, conditional sections — are written as jinja2 templates with a .j2 extension. During test setup, the pytest runner renders every *.j2 file in the test directory and strips the extension: ns1/named.conf.j2 becomes ns1/named.conf.

Inside a template, all the runner's environment variables are available with @...@ delimiters, e.g.:

options {
    port @PORT@;
    listen-on { 10.53.0.1; };
};
key rndc_key {
    secret "1234abcd8765";
    algorithm @DEFAULT_HMAC@;
};
controls {
    inet 10.53.0.1 port @CONTROLPORT@ allow { any; } keys { rndc_key; };
};

Standard jinja2 block syntax ({% if %}, {% for %}, …) can be used for conditional or repeated sections.

Custom template variables come from an optional module-level bootstrap() function. When a test module defines one, the runner calls it before rendering the templates and passes the returned dict to the template engine:

def bootstrap():
    return {"valid": True}

Templates using custom variables must always provide defaults, so that the file also renders when no value is supplied (e.g. when another module in the same directory has no bootstrap()):

{% set valid = valid | default(False) %}

bootstrap() is also the place where a module generates test data that has to exist before the servers start — typically zone files and DNSSEC keys (see the cookbook for a complete example).

Templates can also be re-rendered while the test is running, using the templates fixture, e.g. to change a server's config before reloading it:

def test_reload(ns1, templates):
    templates.render("ns1/named.conf", {"valid": True})
    ns1.reconfigure()

If you don't need the file to be auto-templated during test setup, use the extension .j2.manual instead; such templates are only rendered when the test calls templates.render() explicitly, and no defaults are needed.

Fixtures and Helpers

Fixtures defined in conftest.py provide the test context:

• servers is a dictionary of all started isctest.instance.NamedInstance servers, keyed by directory name; the shortcut fixtures ns1 through ns11 return the corresponding instance directly. A NamedInstance is the interface for driving a server: ns1.rndc("..."), ns1.reconfigure(), ns1.nsupdate(...), ns1.watch_log_from_here(), ns1.ip, ...
• templates renders jinja2 templates at runtime (see above).
• named_port, named_tlsport, control_port, ... return the assigned port numbers.
• system_test_dir is the temporary directory the module runs in.

The isctest package provides the helper library; the modules most tests need are isctest.query (send DNS queries), isctest.check (assert on responses), isctest.zone (zone and key setup), isctest.kasp (DNSSEC key state checks), isctest.asyncserver (mock servers) and isctest.log (logging and log watchers).

Pytest marks control test collection and setup:

• @pytest.mark.extra_artifacts([...]) declares the files (globs) the test is expected to leave behind in addition to the common ones; undeclared leftovers fail the run.
• @pytest.mark.requires_zones_loaded("ns1", ...) delays the test until the listed servers have loaded all zones.
• isctest.mark has skip-unless conditions for environment prerequisites, e.g. isctest.mark.with_dnstap, isctest.mark.softhsm2_environment, isctest.mark.live_internet_test.

Parametrization

Use pytest.mark.parametrize to run one test function over several inputs instead of copy-pasting test cases or looping inside one test function — each case is reported (and can be re-run) individually:

@pytest.mark.parametrize(
    "qname,rdtype",
    [
        ("exists.example.", "A"),
        ("exists.example.", "TXT"),
        ("other.example.", "A"),
    ],
)
def test_answers(qname, rdtype, ns1):
    msg = isctest.query.create(qname, rdtype)
    response = isctest.query.udp(msg, ns1.ip)
    isctest.check.noerror(response)

This creates the test cases test_answers[exists.example.-A] etc., which can be passed to pytest as node IDs.

Logging

Each module has a separate log which will be saved as pytest.log.txt in the temporary directory in which the test is executed. This log includes messages for this module setup/teardown as well as any logging from the tests. Logging level DEBUG and above will be present in this log.

Use isctest.log for test output (isctest.log.info("...") etc.); in general, any log messages using INFO or above will also be printed out during pytest execution. In CI, the pytest output is also saved to pytest.out.txt in the bin/tests/system directory.

Adding a Test to the System Test Suite

Once a test has been created it will be automatically picked up by the pytest runner if it upholds the convention expected by pytest (especially when it comes to naming files and test functions). New system test directories are discovered automatically; no registration in the build system is needed.

Writing a regression reproducer

The goal: turn "issue #NNNN" into a failing test with minimal ceremony.

1. Decide the server topology. Most reproducers need one of:

   • a single authoritative named (answer content bugs) — the skeleton recipe above;
   • a resolver plus a mock server that misbehaves (resolver bugs) — see the mock server recipe below;
   • signed zones and a validating resolver (DNSSEC bugs) — see the zone setup recipe below.

2. Find the closest existing test and copy its shape. Good exemplars: cyclic_glue (resolver + python mock server), dnssec_py (signed zones, validator, multiple modules sharing one server set), nsec3 (multi-module family), kasp/rollover_* (key management state machines).

3. Decide where the test lives. If an existing directory already has the server set you need, add a new tests_*.py module there; otherwise create a new directory. Each module gets its own temporary directory, port range, and parallel slot, so you are not entangled with the other modules. Test functions within a module, however, run in file order against the same live servers: a new test inherits whatever state the tests above it left behind (cache contents, dynamic updates) and can disturb the tests below it.

4. Write the test to fail first. Run it against an unfixed build and make sure it fails for the reason the issue describes — ns*/named.run in the kept temporary directory is the place to verify that. Then apply the fix and watch it pass.

Mock a misbehaving server

When a test needs a server that answers in ways named never would (bogus glue, truncation, dropped queries, malformed records), add an ansN subdirectory containing an ans.py script based on isctest.asyncserver. The runner starts it automatically on 10.53.0.N, logging to ans.run.

Implementing a custom ansN server happens in two phases:

• define all static DNS data that the server needs to serve (if any) in *.db files, like you would for a regular named instance,

• implement any non-standard behavior (modifying zone-based responses or generating responses from scratch) by defining a response handler class, scoping it to the QNAMEs/QTYPEs/domains it owns, and installing it into an AsyncDnsServer.

Most importantly, avoid the temptation to define all DNS responses that a given ansN server needs to serve using just dnspython APIs; zone files are much easier to follow for static DNS data. Splitting up static DNS data and custom behavior also makes it easier to follow the idea behind each test.

The most commonly subclassed handler classes are (ordered by descending specificity):

• QnameQtypeHandler
• QnameHandler
• DomainHandler

These handler classes require certain properties (e.g. qnames, qtypes, domains) to be defined by their subclasses. These properties define the set of queries that a given handler should be used for. Please see isctest/asyncserver.py for up-to-date information on available handler classes and existing ans.py files for how they can be used in practice. Consult the log files (ans.run) in case a query is not matched by its intended handler.

NOTE: For readability (of both code and logs), defining separate handler classes for distinct queries is strongly preferred over using a single handler containing an if/elif/else chain.

NOTE: If you find yourself implementing an __init__() method in your handler subclass, it often indicates that you're approaching the problem at hand from the wrong side; contact QA for guidance in such a case.

When a query is matched to a handler, the latter is expected to yield a response action through its get_responses() method, an async generator that inspects the query context and decides how the server should react:

from collections.abc import AsyncGenerator

import dns.flags

from isctest.asyncserver import (
    AsyncDnsServer,
    DnsResponseSend,
    DomainHandler,
    QueryContext,
    ResponseAction,
)


class TruncateHandler(DomainHandler):
    """Answer everything under broken.example. with TC=1."""

    domains = ["broken.example."]

    async def get_responses(
        self, qctx: QueryContext
    ) -> AsyncGenerator[ResponseAction, None]:
        qctx.response.flags |= dns.flags.TC
        yield DnsResponseSend(qctx.response)


def main() -> None:
    server = AsyncDnsServer()
    server.install_response_handler(TruncateHandler())
    server.run()


if __name__ == "__main__":
    main()

The available response actions are DnsResponseSend (optionally with a delay), ResponseDrop (don't answer at all), BytesResponseSend (raw bytes, for malformed packets) and CloseConnection (TCP). Queries that no handler matches are answered from zone data — AsyncDnsServer loads every *.db zone file found in the ansN directory at startup — or with the server's default rcode (REFUSED unless configured otherwise).

NOTE: For returning static responses, subclassing StaticResponseHandler is strongly recommended instead of implementing the get_responses() generator manually; see resolver/ans3/ans.py for practical examples.

NOTE: Calling yield does NOT make get_responses() return! This is by design: get_responses() can yield multiple DNS messages in response to a single query, so that it can also handle AXFR/IXFR queries, among others. Be careful not to unintentionally cause multiple DNS messages to be returned for a single query. If your handler's get_responses() method contains multiple yield statements, it might be a sign that it needs to be refactored into multiple separate handlers.

If multiple ansN instances used in a given system test need to share common logic, extract that logic into a <test-name>_ans.py module in the system test directory. See the qmin system test for a practical example.

If multiple system tests would benefit from sharing some common logic, consider submitting a merge request adding that logic to isctest/asyncserver.py itself.

To the extent possible, try to keep each ans.py file limited in length and scope. Look at existing ans.py files to see what is meant by that. If the response generation logic required for reproducing a given bug is particularly complex, consider dedicating the entire ans.py file just to that logic instead of appending it to an existing one; ansN instances are cheap to spawn and run compared to regular named instances. If the number of ansN instances used in a given system test is becoming unwieldy, it usually indicates the need to start adding/moving code to a new system test directory.

In some rare cases, it may be useful to reuse a common set of nsN server instances to reproduce a whole class of related issues, triggering which relies on some non-standard behavior and therefore needs a custom ansN server to be implemented. If the logic necessary for reproducing each of these issues is complex and the amount of those issues makes it impractical to add a separate ansN server for each issue (as recommended in the previous paragraph), it is acceptable to split up the test logic for each issue into separate ans_*.py modules inside a single ansN directory and reduce ans.py itself to a loader that imports and installs handlers defined in those separate modules:

from mytest.ans1 import ans_some_bug, ans_some_other_bug
from isctest.asyncserver import AsyncDnsServer


def main() -> None:
    server = AsyncDnsServer()
    server.install_response_handler(ans_some_bug.SomeBugHandler())
    server.install_response_handler(ans_some_other_bug.SomeOtherBugHandler())
    server.run()


if __name__ == "__main__":
    main()

However, in such a case it is particularly important to ensure consistency between the names of all the Python files related to a given issue - otherwise, chaos ensues. Furthermore, avoid using cryptic file names (e.g. numeric bug identifiers). The recommended naming scheme is:

mytest/
├── ans1
│   ├── ans.py
│   ├── ans_some_bug.py
│   └── ans_some_other_bug.py
├── ns2
│   └── ...
├── tests_some_bug.py
└── tests_some_other_bug.py

To point a resolver at the mock, delegate to it from the test's root zone (served by ns1) or list it as a forwarder; cyclic_glue shows the delegation pattern end to end.

The existing mock servers are the best reference. To find them, grep for what you're about to use: git grep -l isctest.asyncserver -- '*/ans*/ans.py' lists every python mock, and a grep for the base class (DomainHandler, QnameHandler, ConnectionHandler) or the response action (ResponseDrop, BytesResponseSend, ...) you need usually turns up a test already doing something similar. The full toolbox lives in isctest/asyncserver.py (query matching, TCP connection handling, TSIG keyrings).

Nameservers

As noted earlier, a system test will involve a number of nameservers. These will be either instances of named, or mock servers, typically written in Python.

For the former, the version of "named" being run is the one from the build directory registered by the most recent meson compile invocation. The configuration files, zone files etc. for these servers are located in subdirectories of the test directory named "nsN", where N is a small integer. The latter are special nameservers, mostly used for generating deliberately bad responses, located in subdirectories named "ansN" (again, N is an integer). In addition to configuration files, these directories should hold the appropriate script files as well.

Note that the "N" for a particular test forms a single number space, e.g. if there is an "ns2" directory, there cannot be an "ans2" directory as well. Ideally, the directory numbers should start at 1 and work upwards.

When tests are executed, pytest takes care of the test setup and teardown. It looks for any nsN and ansN directories in the system test directory and starts those servers.

named Command-Line Options

By default, named server is started with the following options:

-c named.conf   Specifies the configuration file to use (so by implication,
                each "nsN" nameserver's configuration file must be called
                named.conf).

-d 99           Sets the maximum debugging level.

-D <name>       The "-D" option sets a string used to identify the
                nameserver in a process listing.  In this case, the string
                is the name of the subdirectory.

-g              Runs the server in the foreground and logs everything to
                stderr.

-m record
                Turns on these memory usage debugging flags.

All output is sent to a file called named.run in the nameserver directory.

The options used to start named can be altered. There are a couple ways of doing this. The runner checks the methods in a specific order: if a check succeeds, the options are set and any other specification is ignored. In order, these are:

1. Including a file called "named.args" in the "nsN" directory. If present, the contents of the first non-commented, non-blank line of the file are used as the named command-line arguments. The rest of the file is ignored.

2. Tweaking the default command line arguments with "-T" options. This flag is used to alter the behavior of BIND for testing and is not documented in the ARM. The presence of a file called named.<flag> in the "nsN" directory adds -T <flag> to the default command line (the content of the file is irrelevant

• it is only the presence that counts). The recognized flags are:

dropedns         Recognise EDNS options in messages, but drop messages
                 containing them.

ednsformerr, ednsnotimp, ednsrefused
                 Answer EDNS queries with the given rcode, pretending to
                 be an old server that doesn't understand EDNS.

cookiealwaysvalid
                 Accept any DNS cookie presented by a client.

noaa             Never set the AA bit in an answer.

noedns           Disable recognition of EDNS options in messages.

nonearest        Omit the closest-encloser NSEC3 proof from negative
                 responses (except for DS queries).

nosoa            Disable the addition of SOA records to negative
                 responses (or to the additional section if the response
                 is triggered by RPZ rewriting).

maxudp512, maxudp1460
                 Set the maximum UDP size handled by named to 512/1460.

tat=1, tat=3     Send trust-anchor-telemetry queries every N seconds.

notcp            Disable TCP in "named".  Unlike the other flags, this
                 one is also applied when "named.args" is used.

Running Nameservers Interactively

In order to debug the nameservers, you can let pytest perform the nameserver setup and interact with the servers before the test starts, or even at specific points during the test, using the --trace option to drop you into pdb debugger which pauses the execution of the tests, while keeping the server state intact:

pytest -k dns64 --trace

Developer Notes

Test discovery and collection

There are two distinct types of system tests. The first is a legacy shell script tests.sh containing individual test cases executed sequentially and the success/failure is determined by return code. The second type is a regular pytest file which contains test functions.

Dealing with the regular pytest files doesn't require any special consideration as long as the naming conventions are met. Discovering the tests.sh tests is more complicated.

The chosen solution is to add a bit of glue for each system test. For every tests.sh, there is an accompanying tests_sh_*.py file that contains a test function which utilizes a custom run_tests_sh fixture to call the tests.sh script. Other solutions were tried and eventually rejected. While this introduces a bit of extra glue, it is the most portable, compatible and least complex solution.

Compatibility with older pytest version

The minimum supported versions of python and the required python packages are declared in requirements.txt and in the pytest_configure() check in conftest.py. When implementing new runner features, check feature support in the pytest and pytest-xdist versions available in the oldest distributions covered by CI first; we may need to add compat code to handle breaking upstream changes in either direction.

Format of Shell Test Output

Legacy shell-based tests have the following format of output:

<letter>:<test-name>:<message> [(<number>)]

e.g.

I:catz:checking that dom1.example is not served by primary (1)

The meanings of the fields are as follows:

This indicates the type of message. This is one of:

S   Start of the test
A   Start of test (retained for backwards compatibility)
T   Start of test (retained for backwards compatibility)
E   End of the test
I   Information.  A test will typically output many of these messages
    during its run, indicating test progress.  Note that such a message may
    be of the form "I:testname:failed", indicating that a sub-test has
    failed.
R   Result.  Each test will result in one such message, which is of the
    form:

            R:<test-tmpdir>:<result>

    where <result> is one of:

        PASS        The test passed
        FAIL        The test failed
        SKIPPED     The test was not run, usually because some
                    prerequisites required to run the test are missing.

This is the name of the temporary test directory from which the message emanated, which is also the name of the subdirectory holding the test files. This is text output by the test during its execution.

() If present, this will correlate with a file created by the test. The tests execute commands and route the output of each command to a file. The name of this file depends on the command and the test, but will usually be of the form:

<command>.out.<suffix><number>

e.g. nsupdate.out.test28, dig.out.q3. This aids diagnosis of problems by allowing the output that caused the problem message to be identified.