Some dns message modifications like TSIG happen only after .to_wire() is
called on the message. To ensure there isn't a discrepancy between what
has been logged and what has been sent, log the query after
dns.query.udp() is executed (which calls .to_wire() on the message).
Co-Authored-By: Štěpán Balážik <stepan@isc.org>
Add a pylint plugin that enforces:
- There is no bare `import dns` statement.
- All `dns.<module>` used are explicitly imported.
- There are no unused `dns.<module>` imports.
Fix all the imports to conform with this check.
Importing pytest fixture trips up static analysis tools, so move
default_algorithm to conftest.py and use it instead of os.environ
accesses in various system tests.
For use outside test function, use Algorithm.default().
Add a new response handler, ForwarderHandler, which enables forwarding
all queries to another DNS server. To simplify implementation, always
forward queries to the target server via UDP, even if they are
originally received using a different transport protocol.
Extend AsyncDnsServer._log_query() and AsyncDnsServer._log_response() so
that they also log the <address, port> tuple for the socket on which a
given query was received on. Minimize the signatures of those methods
by taking advantage of all the information contained in the QueryContext
instances passed to them.
Extend the QueryContext class with a field holding the <address, port>
tuple for the socket on which a given query was received. This will
enable query handlers to act upon that information in arbitrary ways.
Previously, the issue when the kasp.test_kasp_case[secondary.kasp] fails
due to a timeout has been only ocassionally observed on FreeBSD 13
in our CI. It seems to have come back on FreeBSD 15.
Store the most specific matching domain in DomainHandler and
expose it through the `matched_domain` property for subclasses
to use in their implementations of `get_responses`.
The minimum required hypothesis version has been set in requirements.txt
and no longer needs to be checked at runtime.
Since the hypothesis package is now a mandatory prerequisite, include it
in isctest as the other subpackages.
To provide feature parity with `bin/tests/system/ans.pl` add a control
command to allow easy switching between different sequences of
ResponseHandlers.
It saves an indent and brackets on the call sites.
Also sort the handlers alphabetically where their order doesn't matter
and split the fallback handlers into a separate call to signify that
their position in the end matters.
The action can be used to close the connection even after some response
was sent, depending on the ordering of actions in the handler that uses
it. Rename it to CloseConnection to use a more fitting name.
If at all possible, all the responses should be created by
AsyncDnsServer's internal methods. To ensure this, mark them with a
magic attribute and check it on send and crash the server if a manually
created response is detected.
Fix the qmin test server which uses `make_response`.
In 6e684d44 I mistakenly set the default for `default_aa` for
`AsyncDnsServer()` to `True` and then explicitly set it to True in
cases where all the `ResponseHandlers` said
`yield DnsResponseSend(..., authoritative=True)` as if the default was
`False`.
Also the rest of `AsyncDnsServer` code (namely `_prepare_responses`)
reads like `default_aa` is `False` by default.
This accidentally changed the behavior of servers which don't set the
`default_aa` and where AA is not set from the zone data
(e.g. `dispatch/ans3`).
Due to the way various asyncio-related objects (tasks, streams,
transports, selectors) are referencing each other, pausing reads for a
TCP transport (which in practice means removing the client socket from
the set of descriptors monitored by a selector) can cause the client
task (AsyncDnsServer._handle_tcp()) to be prematurely garbage-collected,
causing asyncio code to raise a "Task was destroyed but it is pending!"
exception. Who knew that solutions as elegant as the one introduced by
e407888507 could cause unexpected trouble?
Fix by making a horrible hack even more horrible, specifically by
keeping a reference to each incoming TCP connection to protect its
related asyncio objects from getting garbage-collected. This prevents
AsyncDnsServer from closing any of the ignored TCP connections
indefinitely, which is obviously a pretty brain-dead idea for a
production-grade DNS server, but AsyncDnsServer was never meant to be
one and this hack reliably solves the problem at hand.
Only apply this change for the IgnoreAllConnections handler as the
ConnectionReset handler triggers a connection reset immediately after
pausing reads for an incoming TCP connection.
As pointed out in e407888507, the proper
solution would require implementing a custom asyncio transport from
scratch and that is still deemed to be too much work for the purpose at
hand. Let's see how much longer we can limp along with the existing
approach.
Calling asyncio.Future.set_exception() or asyncio.Future.set_result()
more than once for a given Future object raises an
asyncio.InvalidStateError exception.
In the case of AsyncServer:
- it is enough to capture the first exception raised by higher-level
logic as no exceptions at all are expected to be raised in the first
place,
- no distinction is made between SIGINT and SIGTERM; the only purpose
of the signal handler is to make the server exit cleanly.
Given the above, make both AsyncServer._handle_exception() and
AsyncServer._signal_done() idempotent by ignoring
asyncio.InvalidStateError exceptions raised by the relevant
asyncio.Future.set_*() calls.
Introduce rollover/setup.py for all setup related test code.
Introduce rollover/ns1 and rollover/ns2 to create a chain of trust to
all rollover related test zones. The tld zones in rollover/ns2 contain
a DSYNC record that at a later time will be used for testing Generalized
DNS Notifications.
Write a python version of private_type_record so we can put such
records in the zone via jinja2 templating.
Previously, this was only possible by making a new response by calling
make_response on qctx.query. This however ignored the `default_aa` and
`default_rcode` parameters of AsyncDnsServer.
Add prepare_new_response and save_initialized_response methods to
QueryContext.
Previously, ResponseHandlers had to reparse the queries themselves if
they wanted to use TSIG. This led to `default_aa` and `default_rcode`
information being lost from the newly created messages.
Add support for TSIG keyrings to the AsyncDnsServer class directly.
Previously, the server relied on the modules being imported by the
isctest.asyncserver module. This is fragile and confuses tooling.
Clean up stray imports in the process.
Previously, all responses had to be set as authoritative explicitly
using DnsResponseSend(..., authoritative=True). After using this,
it became obvious that this is obnoxious.
Add an optional keyword-only parameter to AsyncDnsServer that sets the
default value of the AA bit on outgoing responses.
Make all the other parameters keyword-only as well.
When this class was introduced, the constructor of its base class had no
parameters. This was changed in the meantime and these parameters were
not accessible by users of the subclass.
Don't override the constructor.
Move command setup to methods.
Move subclass-specific storage to cached properties.
Take instances of Command instead of the classes themselves for
symmetry with install_response_handler.
The purpose of these variables is to be able to detect feature support
without calling feature-test. This becomes useful when detecting feature
support in jinja2 templates.
To unify the command handling, utilize EnvCmd() to handle rndc commands:
1. Remove isctest.rndc abstractions. They were intended for an upcoming
python-only implementation. A couple of years later, it doesn't seem
to be coming any time soon, so let's stick with the interface that
makes sense today, i.e. use the same command handling interface
everywhere.
2. Remove the specialized rndc.log in favor of the generic logging
already implemented by isctest.run.cmd(). I believe the cause of the
many rndc(log=False) invocations was that nobody wanted this extra
file. Yet, logging everything by default makes sense for debugging,
unless there's a good reason not to. In almost all cases, logging was
switched to the default (enabled).
3. With the NamedInstance.rndc() call now returning CmdResult rather
than combined stdout+stderr string, adjust all the invocations to use
`.out` or `.err` as necessary.
4. Replace some manual rndc invocation and its base argument
construction with the standardized nsX.rndc() call.
5. In cases where rndc is expected to fail, utilize
raise_on_exception=False and check the `.rc` from the result, rather
than handling an exception.
6. In addzone/tests_rndc_deadlock.py, refactor the test slightly to
avoid using EnvCmd() entirely to avoid spamming the logs. This test
calls rndc in a loop from multiple threads and such test case is an
exception which doesn't warrant changing the `isctest.run.cmd()`
implementation.
A generic helper that calls the environment-specified binaries in a
developer-friendly manner, i.e. passing arguments as strings rather than
having to split them first.
The isctest.run.cmd() remains as the basis which provides a clean and
robust interface, while the isctest.run.EnvCmd() can be used as a
convenient wrapper for tests, or when there are some shared default
parameters.
The isctest.run.Dig() is superseded with the isctest.run.EnvCmd(). In
the future, we might revisit adding Dig() or command-specific helpers
again, but it probably only makes sense if they offer command-aware
attributes / methods, rather than just being shortcuts to
isctest.run.EnvCmd().
Add a new Grep-like interface which can be used for searching for
regular expressions in files. Replace the prior LogFile used for named
logs with the new TextFile interface.
Add a new module for working with text and keep the isctest.log.watchlog
module focused on its purpose. Move LogFile and LineReader into the new
module. Add compile_pattern() helper which will be useful in subsequent
commits.
