The pthread-based implementation of the isc_rwlock_tryupgrade()
function always returns ISC_R_LOCKBUSY. Fix the test by adding
conditional checks.
(cherry picked from commit c27659bc62)
`python-jinja2` is now required to run system tests.
Related #4938
Backport of MR !9588
Merge branch 'backport-nicki/replace-setup-sh-files-with-jinja2-templates-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10396
Many of the system tests now use jinja2 template engine. Adding jinja2
as a hard dependency is preferable than potentially silently skipping
many system tests.
(cherry picked from commit 543ba8da5a)
These setup.sh scripts only do templating and copying files. Both of
these can be replaced with either jinja templates, or using plain files.
Since each test invocation creates its own temporary directory, copying
files to ensure a "clean" state is no longer necessary.
In cases where named writes some content to the files, a jinja template
can be used instead of a plain file to avoid an artifact check which
would detect a change to a git-tracked file.
(cherry picked from commit e0f0c557a0)
All these setup files only use copy_setports function which can be done
with jinja2 templates instead -- simply by renaming the .in files to
.j2, without any other changes. The pytest runner will render these
templates during test setup without any need for an additional script.
(cherry picked from commit a9f1b8d6a6)
This change allows the client to identify the server that returns the
BADVERS and to provide a DNS SERVER COOKIE to be included in the
resend of the request.
Closes#5235
Backport of MR !10334
Merge branch 'backport-5235-return-the-server-cookie-when-returning-badvers-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10392
DNS COOKIE and NSID should also be being processed when returning
BADVERS. Check that this has actually occured by looking for the
cookie and nsid in the response.
(cherry picked from commit f69b4bc5e0)
A BADVERS response can also include EDNS SERVER COOKIE. Extract
that from the OPT record and use it when resending the request.
(cherry picked from commit 44140cad3b)
This will help identify the broken server if we happen to break
EDNS version negotiation. It will also help protect the client
from spoofed BADVERSION responses.
(cherry picked from commit 0d9cab1555)
The original check_pid() always returned 0 on FreeBSD, even if the
process was still running. This makes the "verifying that named checks
for conflicting named processes" check fail on FreeBSD with TSAN.
Backport of MR !10373
Merge branch 'backport-mnowak/fix-runtime-pid-check-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10386
The original check_pid() always returned 0 on FreeBSD, even if the
process was still running. This makes the "verifying that named checks
for conflicting named processes" check fail on FreeBSD with TSAN.
(cherry picked from commit 6acaca963d)
Replace the custom DNS servers used in the "forward" system test with new
code based on the isctest.asyncserver module.
For ans6, instead of configuring the responses to send at runtime, set
them up when the server is started. Make sure the server supports
toggling response sending at runtime to enable simulating forwarder
timeouts as required by one of the checks.
For ans11, put most of the responses to be provided by that server into
a zone file, only retaining code modifying zone-based answers in the
form of a response handler, to improve code readability. Use explicit
domain names instead of variables as that server only handles a single
domain and fixed strings improve readability in this case. Make sure
the server supports toggling response sending at runtime to enable
simulating forwarder timeouts as required by one of the checks.
Migrate sendcmd() and its uses to the new way of sending control queries
to custom servers used in system tests.
Depends on !10339
Backport of MR !10340
Merge branch 'backport-michal/forward-asyncserver-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10384
Replace the custom DNS servers used in the "forward" system test with
new code based on the isctest.asyncserver module.
For ans6, instead of configuring the responses to send at runtime, set
them up when the server is started. Make sure the server supports
toggling response sending at runtime to enable simulating forwarder
timeouts as required by one of the checks.
For ans11, put most of the responses to be provided by that server into
a zone file, only retaining code modifying zone-based answers in the
form of a response handler, to improve code readability. Use explicit
domain names instead of variables as that server only handles a single
domain and fixed strings improve readability in this case. Make sure
the server supports toggling response sending at runtime to enable
simulating forwarder timeouts as required by one of the checks.
Migrate sendcmd() and its uses to the new way of sending control queries
to custom servers used in system tests.
(cherry picked from commit 23bc8430d9)
Some BIND 9 system tests need to dynamically change custom server
behavior at runtime. Existing custom servers typically use a separate
TCP socket for listening to control commands, which mimics what `named`
does, but adds extra complexity to the custom server's networking code
for no gain (given the purpose at hand). There is also no common way of
performing typical runtime actions (like toggling response dropping)
across all custom servers.
Instead of listening on a separate TCP socket in `asyncserver.py`, make
it detect DNS queries to a "magic" domain (`_control.`) on the same port
as the one it uses for receiving "production" DNS traffic. This enables
query/response logging code to be reused for control traffic, clearly
denotes behavior changes in packet captures, facilitates implementing
commonly used features as reusable chunks of code (by making them "own"
distinct subdomains of the control domain), voids the need for separate
tools sending control commands, and enables using DNS facilities for
returning information to the user (e.g. RCODE for status codes, TXT
records for additional information, etc.).
Backport of MR !10339
Merge branch 'backport-michal/asyncserver-control-commands-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10383
Implement a reusable control command that makes it possible to
dynamically disable/enable sending responses to clients. This is a
typical use case for custom DNS servers employed in various BIND 9
system tests.
(cherry picked from commit 92b39f8352)
Some BIND 9 system tests need to dynamically change custom server
behavior at runtime. Existing custom servers typically use a separate
TCP socket for listening to control commands, which mimics what named
does, but adds extra complexity to the custom server's networking code
for no gain (given the purpose at hand). There is also no common way of
performing typical runtime actions (like toggling response dropping)
across all custom servers.
Instead of listening on a separate TCP socket in asyncserver.py, make it
detect DNS queries to a "magic" domain ("_control.") on the same port as
the one it uses for receiving "production" DNS traffic. This enables
query/response logging code to be reused for control traffic, clearly
denotes behavior changes in packet captures, facilitates implementing
commonly used features as reusable chunks of code (by making them "own"
distinct subdomains of the control domain), voids the need for separate
tools sending control commands, and enables using DNS facilities for
returning information to the user (e.g. RCODE for status codes, TXT
records for additional information, etc.).
(cherry picked from commit a7e1de716b)
With multiple and/or dynamically managed response handlers at play, it
becomes useful for debugging purposes to know which handler (if any) was
used for preparing each response sent by the server. Add debug logs
providing that information. Make class name the default string
representation of each response handler to prettify logs.
(cherry picked from commit 5e71fd081e)
Extend AsyncDnsServer.install_response_handler() so that the provided
response handler can be inserted at the beginning of the handler list.
This enables installing a response handler that takes priority over all
previously installed handlers.
Add a new method, AsyncDnsServer.uninstall_response_handler(), which
enables removing a previously installed response handler.
Together, these two methods provide full control over the response
handler list at runtime.
(cherry picked from commit 92b072bff4)
Add a main() function to all custom servers based on isctest.asyncserver
and move server startup code there. This prevents redefining variables
from outer scope in custom server code as it evolves.
(cherry picked from commit 8cb51d4c2b)
Prevent custom servers based on asyncserver.py from exiting prematurely
due to unhandled exceptions raised as a result of attempting to parse
invalid queries sent by clients.
(cherry picked from commit fd0290c919)
The StreamWriter.wait_closed() method was introduced in Python 3.7, so
attempting to use it with Python 3.6 raises an exception. This has not
been noticed before because awaiting StreamWriter.wait_closed() is the
last action taken for each TCP connection and unhandled exceptions were
not causing the scripts based on AsyncServer to exit prematurely until
the previous commit.
As per Python documentation [1], awaiting StreamWriter.wait_closed()
after calling StreamWriter.close() is recommended, but not mandatory, so
try to use it if it is available, without taking any fallback action in
case it isn't.
[1] https://docs.python.org/3.13/library/asyncio-stream.html#asyncio.StreamWriter.close
(cherry picked from commit 715bd1b667)
Uncaught exceptions raised by tasks running on event loops are not
handled by Python's default exception handler, so they do not cause
scripts to die immediately with a non-zero exit code. Set up an
exception handler for AsyncServer code that makes any uncaught exception
the result of the Future that the top-level coroutine awaits. This
ensures that any uncaught exceptions cause scripts based on AsyncServer
to immediately exit with an error, enabling the system test framework to
fail tests in which custom servers encounter unforeseen problems.
(cherry picked from commit ec4c92d9d5)
Move test code that can be reused for the kasp pytest-based system test.
Backport of MR !10252
Merge branch 'backport-matthijs-pytest-rewrite-kasp-system-test-1-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10382
In the kasp system test there are cases that the SyncPublish is not
set, nor it is required to do so. Update the _check_dnskeys function
accordingly.
(cherry picked from commit 0a6cc42914)
For the kasp tests we need a new utility that can retrieve a list of
Keys from a given directory, belonging to a specific zone. This is
'keydir_to_keylist' and is the replacement of 'kasp.sh:get_keyids()'.
'next_key_event_eqauls' is a method to check when the next key event is
scheduled, needed for the rollover tests, and is the equivalent of shell
script 'check_next_key_event'.
(cherry picked from commit 12e57eb222)
This commit introduces replacements for the 'check_keys' and
'check_keytimes' from the shell test library. 'check_keys' is renamed
to 'verify_keys' because it does not assert.
For that, we introduce more functions for the class Key. The
'match_properties' function is used in 'verify_keys' to see if a set of
KeyProperties match the Key. This speficially ignores timing metadata.
The function resembles what is in 'kasp.sh:check_key()'.
The 'match_timingmetadata' function is used in 'check_keytimes' to see
if the timing metadata of a set of KeyProperties match the Key. The
values are checked in all three key files (except if the private key is
not available (set with properties["private"]), or if it is a legacy key
(set with properties["legacy"]).
An additional check function is added, to check if the key relationships
are set correctly. It follows a similar pattern as 'check_keytimes'. If
"Predecessor" and/or "Successor" are expected to be set in the state
file, this function checks so, and also verifies that they are not set
if they should not be.
(cherry picked from commit 44ff63a50d)
Because we want to check the metadata in all three files, a new
value in the Key class is added: 'privatefile'. The 'get_metadata'
function is adapted so that we can also check metadata in other files.
Introduce methods to easily retrieve the TTL and public DNSKEY record
from the keyfile.
When checking if the CDS is equal to the expected value, use the DNSKEY
TTL instead of hardcoded 3600.
(cherry picked from commit 97f6b7ad11)
In isctest.kasp, introduce a new class 'KeyProperties' that can be used
to check if a Key matches expected properties. Properties are for the
time being divided in three parts: 'properties' that contain some
attributes of the expected properties (such as are we dealing with a
legacy key, is the private key available, and other things that do not
fit the metadata exactly), 'metadata' that contains expected metadata
(such as 'Algorithm', 'Lifetime', 'Length'), and 'timing', which is
metadata of the class KeyTimingMetadata.
The 'default()' method fills in the expected properties for the default
DNSSEC policy.
The 'set_expected_times()' sets the expected timing metadata, derived
from when the key was created. This method can take an offset to push
the expected timing metadata a duration in the future or back into the
past. If 'pregenerated=True', derive the expected timing metadata from
the 'Publish' metadata derived from the keyfile, rather than from the
'Created' metadata.
The calculations in the 'Ipub', 'IpubC' and 'Iret' methods are derived
from RFC 7583 DNSSEC Key Rollover Timing Considerations.
(cherry picked from commit 0b9fbca18e)
This is the first step of converting the kasp system test to pytest.
Well, perhaps not the first, because earlier the ksr system test was
already converted to pytest and then the `isctest/kasp.py` library
was already introduced. Lots of this code can be reused for the kasp
pytest code.
First of all, 'check_file_contents_equal' is moved out of the ksr test
and into the 'check' library. This feels the most appropriate place
for this function to be reused in other tests. Then, 'keystr_to_keylist'
is moved to the 'kasp' library.
Introduce two new methods that are unused in this point of time, but
we are going to need them for the kasp system test. 'zone_contains'
will be used to check if a signature exists in the zonefile. This way
we can tell whether the signature has been reused or refreshed.
'file_contents_contain' will be used to check if the comment and public
DNSKEY record in the keyfile is correct.
(cherry picked from commit ee8e9f1ded)
In PRIVATEOID keys, the key data begins with a length byte followed
by an ASN.1 object identifier that indicates the cryptographic algorithm
to use. Previously, the length byte was not accounted for when
checking the contents of keys and signatures, which could have led
to interoperability problems with any zones signed using PRIVATEOID.
This has been fixed.
Closes#5270
Backport of MR !10372
Merge branch 'backport-5270-fix-check-private-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10376
We were failing to account for the length byte before the OID.
See RFC 4034.
Algorithm number 254 is reserved for private use and will never be
assigned to a specific algorithm. The public key area in the DNSKEY
RR and the signature area in the RRSIG RR begin with an unsigned
length byte followed by a BER encoded Object Identifier (ISO OID) of
that length. The OID indicates the private algorithm in use, and the
remainder of the area is whatever is required by that algorithm.
Entities should only use OIDs they control to designate their private
algorithms.
(cherry picked from commit ca7355b7d0)
Reformat the section to be more consistent with the rest of the rndc
documentation and avoid using :program: directive which would needlessly
break rst links.
(cherry picked from commit 22bd41e308)
Caching prevents server upgrades being detected in a timely manner
and it can also prevent DNSSEC responses being requested.
(cherry picked from commit 90b2f94d9b)
When multiple nested DNS validations were destroyed out of order,
the EDE context could be freed before all EDE codes were copied,
which could cause an assertion failure. This has been fixed.
Closes#5213
Backport of MR !10365
Merge branch 'backport-5213-use-dns_ede_copy-in-dns_validator-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10366
If the nested DNS validator ends up in the same fetch because of the
loops, the code could be copying the EDE codes from the same source EDE
context as the destination EDE context. Skip copying the EDE codes if
the source and the destination is the same.
(cherry picked from commit 2988ebae21)
Pass NULL as edectx for the fetch_and_forget() fetches as nobody
is reading the EDE contexts and it can mess the main client buffer.
(cherry picked from commit fe48290140)
Instead of passing the edectx from the fetchctx into all subvalidators,
make the ede context ownership explict for dns_resolver_createfetch()
callers, and copy the ede result codes from the children validators to
the parent when finishing the validation process.
(cherry picked from commit d7593196a1)
TSAN reports a lock-order-inversion (potential deadlock) issue in
`add_trace_entry()`.
While it is true that in one case a lock in the `isc_mem_t` structure is
locked first, and then a lock in the `FILE` structure is locked second,
and in the the second case it is the other way around, this isn't an
issue, because those are `FILE` structures for totally different files,
used in different parts of the code.
Closes#5266
Backport of MR !10355
Merge branch 'backport-5266-freebsd-suppress-tsan-lock-order-inversion-false-positive-9.20' into 'bind-9.20'
See merge request isc-projects/bind9!10362
