System tests that set up zones — especially DNSSEC tests — require a
chain of common operations: rendering zone files from templates,
generating keys, signing, and propagating DS records to parent zones.
Implement these as methods on isctest.zone.Zone so individual tests
don't need to repeat the logic in shell or ad-hoc Python.
isctest.zone.Zone is a plain class that holds the zone's data and
accumulated state (delegations, keys) alongside the methods that operate
on it. It is intentionally separate from isctest.template.Zone, which
remains a dumb data container for jinja2 template rendering.
Key design points:
- zone.Zone.name is the text form without trailing dot ("." for root);
zone.Zone.dname holds the dns.name.Name for DNS-level operations;
zone.Zone.basename is the filesystem-safe name ("root" for ".").
- filepath_unsigned / filepath_signed are both always available.
filepath returns the appropriate one based on zone.Zone.signed.
- The zones/ subdirectory is the default (subdir="zones"); old-style
tests that place zone files directly in the ns workdir can pass
subdir=None.
- Signing is opt-in via signed=True; configure() auto-detects whether to
generate keys and sign based on this flag, so the same method handles
both signed and unsigned zones.
- delegations and keys are mutable list attributes; callers append to
them before calling configure() rather than threading them through
every call.
Also:
- Add isctest.template.zones() as a bridge from a list of zone.Zone to a
{name: template.Zone} dict suitable for use as the ``zones`` template
variable. template.zones() resolves filepath to the actual zone file
so templates don't need to know whether a zone is signed.
Assisted-by: Claude:claude-opus-4-8
LoadScopeScheduling._split_scope() uses rsplit("::", 1) to
extract the test file scope from a node ID. When parametrized
test values contain "::" (IPv6 addresses like "cafe:cafe::cafe"
or "::1"), the split lands inside the parameter instead of at
the .py:: boundary. This creates spurious scopes that get
assigned to different workers, each triggering a full fixture
setup (starting named instances).
Override _split_scope() in conftest.py to split on ".py::"
which is unambiguous.
Six tests in synthrecord/tests_synthrecord.py are affected.
A verification script is included in util/.
Assisted-by: Claude:claude-opus-4-7
Update PRIORITY_TESTS with the 10 longest-running test
scopes measured from CI (job 7468217). These get scheduled
first so that with --dist=loadscope they land on separate
workers instead of piling up at the end.
Also fix "serve-stale/" to "serve_stale/" to match the
actual directory name, and add a startup check that fails
if any PRIORITY_TESTS entry does not match an existing
directory.
Assisted-by: Claude:claude-opus-4-7
Extend the Nameserver to generate the default IPv4/IPv6 values, add NSX
values for the predefined nameservers (there are 11 of them, as per
bin/tests/system/ifconfig.sh.in max value). Add the missing ns11
fixture.
Extend the Zone to derive the zone filename by default, unless
specified.
Adjust the existing uses of these classes to utilize the simplified
defaults.
Change the convention for system test directory names to always use an
underscore rather than a hyphen. Names using underscore are valid python
package names and can be used with standard `import` facilities in
python, which allows easier code reuse.
The temporary directories for test execution and their convenience
symlinks have been switched to using hyphens rather than underscores to
keep the pytest collection, filtering and .gitignore working as
expected.
Importing pytest fixture trips up static analysis tools, so move
default_algorithm to conftest.py and use it instead of os.environ
accesses in various system tests.
For use outside test function, use Algorithm.default().
If the system_test_dir contains a symlink, then it might cause issues
further down when using relative_to(), unless it is resolved first. This
has been observed on FreeBSD13 in CI where /home is a symlink to
/usr/home.
Drop support of EoL python versions for running system tests. The
maintenance cost of supporting end of life ecosystem, especially Python
3.6 on EL8 and the related outdated packages (pytest, dnspython, ...),
has become unreasonable.
Use collection_path rather than the deprecated path argument for
pytest_ignore_collect() hook.
The collection_path argument was added in pytest 7.0.0, which is the
minimum supported pytest version from now on.
This allows rendering multiple named*.conf files using the jinja2
template engine at test start and then simply copying the required
config to named.conf as needed.
It's a fairly common pattern to use regular expression in our tests.
Instead of using the fairly verbose re.compile(), import that function
as Re() instead to allow for more brevity in the test syntax.
During the system test execution, allow use of module-specific
bootstrap() function in addition to the setup.sh script which this
function should ultimately replace.
The purpose of bootstrap() is two-fold. First, it can execute any
commands needed to create the initial conditions for the test, such as
creating key materials, manipulating files etc. Second, it should return
any test-specific template values as a dictionary. Those will be used to
render the jinja2 templates.
Many of our test cases only use a single NamedInstance from the
`servers` fixture. Introduce `nsX` helper fixtures to simplify these
tests and reduce boilterplate code further.
Specifically, the test no longer has to either define its own variable
to extract a single server from the list, or use the longer
servers["nsX"] syntax. While this may seem minor, the amount of times it
is repeated across the tests justifies the change. It also promotes
using more explicit server identification, i.e. `nsX`, rather than
generic `server`. This also improves the clarity of the tests and may be
helpful in traceback during debugging as well.
When compiling with meson, it may be easy to forget to compile system
test dependencies before running the tests. In that case, the test
results would be quite incosistent and unpredictable, with some tests
ending up with ERROR, some with FAILURE and others PASS, without a clear
indication that something is off before running the entire machinery.
Add a check to fail early on if the FEATURETEST binary isn't available,
indicating that system test dependencies were most likely not compiled.
The extra messages are typically traceback from assertion failures.
Previously, they'd be printed only after all individual test case
results have been printed. That made it difficult to pair the traceback
to the failing test in some cases, as the node information (aka test
name) might not always be present.
Instead, log any extra messages related to a particular test failure
directly after reporting its result, making the failure details more
readily available and easy to connect with a particular test case.
The messages obtained from test results may contain stuff like detailed
failure/error information, tracebacks etc. In many cases, the message
will be empty, in which case it doesn't need to be logged.
For an example, run test with many test cases, e.g.
verify/test_verify.py, and inspect the tail of the pytest.log.txt before
and after this commit.
There is an ongoing debate about the usefulness of the extra artifacts
check. While it might be useful to detect unexpected behaviour in some
tests, it feels extraneous in many cases. This change provides a middle
ground by making the artifact checking optional. This might be
especially useful for writing new tests, since the author gets to decide
whether the check is useful -- and can utilize it, or can skip it for
sake of brevity.
Meson is a modern build system that has seen a rise in adoption and some
version of it is available in almost every platform supported.
Compared to automake, meson has the following advantages:
* Meson provides a significant boost to the build and configuration time
by better exploiting parallelism.
* Meson is subjectively considered to be better in readability.
These merits alone justify experimenting with meson as a way of
improving development time and ergonomics. However, there are some
compromises to ensure the transition goes relatively smooth:
* The system tests currently rely on various files within the source
directory. Changing this requirement is a non-trivial task that can't
be currently justified. Currently the last compiled build directory
writes into the source tree which is in turn used by pytest.
* The minimum version supported has been fixed at 0.61. Increasing this
value will require choosing a baseline of distributions that can
package with meson. On the contrary, there will likely be an attempt
to decrease this value to ensure almost universal support for building
BIND 9 with meson.
Some versions of the Hypothesis Python library - notably the one
included in stock OS repositories for Ubuntu 20.04 Focal Fossa - cause a
.hypothesis file to be created in a Python script's working directory
when the hypothesis module is present in its import chain. Ignore such
files by adding them to the list of expected test artifacts to prevent
pytest teardown checks from failing due to these files appearing in the
file system after running system tests.
Previously, these functions have been provided as fixtures. This was
limiting re-use, because it wasn't possible to call these outside of
tests / other fixtures without passing these utility functions around.
Move them into isctest.run package instead.
The servers are setup and torn down once per each test module. All the
logs and server state persists between individual tests within the same
module. The servers fixture representing these servers should be
module-wide as well.
When a test is re-run by the flaky plugin, the TestReport outcomes
collected in the pytest_runtest_makereport() hook should be overriden.
Each of the setup/call/teardown phases is reported again and since we
care about the overall outcome, their respective results should be
overriden so that only the outcome from the final test (re)run gets
reported.
Prior to this change, it lead to a situation where an extra_artifact
generated during the test might be ignored. This was caused because the
check was skipped, since the test was incorrectly considered as "failed"
in the case where the test would fail on the first run, but pass on a
subsequent flaky rerun.
Prior to introducing the pytest runner, clean.sh files were used as a
list of files that the test is expected to leave around as artifacts and
check that no extra files were created.
With the pytest runner, those scripts are no longer used, but the
ability to detect extraneous files is still useful. Add a new
"extra_artifacts" mark which can be used for the same purpose.
Configuration files in system tests which require some variables (e.g.
port numbers) filled in during test setup, can now use jinja2 templates
when `jinja2` python package is available.
Any `*.j2` file found within the system test directory will be
automatically rendered with the environment variables into a file
without the `.j2` extension by the pytest runner. E.g.
`ns1/named.conf.j2` will become `ns1/named.conf` during test setup. To
avoid automatic rendering, use `.j2.manual` extension and render the
files manually at test time.
New `templates` pytest fixture has been added. Its `render()` function
can be used to render a template with custom test variables. This can be
useful to fill in different config options during the test. With
advanced jinja2 template syntax, it can also be used to include/omit
entire sections of the config file rather than using `named1.conf.in`,
`named2.conf.in` etc.
The pytest collection mechanism has been overhauled in pytest 8.0.0,
resulting in a different node tree when collecting the tests. Ensure the
paths / names we're using that are derived from the node tree are
consistent across different pytest versions.
Particularly, this has affected the convenience symlink name (which is
supposed to be in the form of e.g. dns64_sh_dns64 for the dns64 module
and tests_sh_dns64.py module) and the test name that's logged at the
start of the test, which is supposed to include the system test
directory relative to the root system test directory as well as the
module name (e.g. dns64/tests_sh_dns64.py).
Related https://github.com/pytest-dev/pytest/issues/7777
Enforcing pylint standards and default for our test code seems
counter-productive. Since most of the newly added code are tests or is
test-related, encountering these checks rarely make us refactor the code
in other ways and we just disable these checks individually. Code that
is too complex or convoluted will be pointed out in reviews anyways.
Instead of invoking get_algorithms.py script repeatedly (which may yield
different results), move the algorithm configuration to an isctest
module. This ensures the variables are consistent across the entire test
run.
Make sure all initialization takes place in isctest.vars.__init__ and
export the initial env vars there. Remove the no longer needed env
fixture and use os.environ instead.
Provide a single point of access to all the variables used by tests. Use
a custom dict-like structure to access the underlying data without
making a copy. This allows the individual modules to update the contents
at runtime, which is used for some variables.
Remove conf.sh.in and move the environment variables into isctest/vars
python package. This enabled the removal of an ugly pytest hack which
loaded and parsed these variables from the environment.
Initializing the conftest logging upon importing the isctest package
isn't practical when there are standalone pieces which can be used
outside of the testing framework, such as the asyncdnsserver module.
Unify the different loggers (conftest, module, test) into a single
interface. Remove the need to select the proper logger by automatically
selecting the most-specific logger currently available.
This also removes the need to use the logger/mlogger fixtures manually
and pass these around. This was especially annoying and unwieldy when
splitting the test cases into functions, because logger had to always be
passed around. Instead, it is now possible to use the
isctest.log.(debug,info,warning,error) functions.
