The <sys/endian.h> header has existed in macOS since around ~26. This
causes the `htobeNN`/`htoleNN` macros to be redefined in <isc/endian.h>
in terms of <libkern/OSByteOrder.h> when other system headers include
<sys/endian.h>.
Fix this issue by using checking for the existence of <sys/endian.h> in
meson and including it according to the probe result.
Busy resolvers will now gradually fill the configured :any:max-cache-size
before entries start being evicted. Previously, expired records were
proactively removed based on their TTL, which kept memory usage below the
configured limit but added overhead. Cache eviction now relies solely on the
SIEVE-LRU mechanism, which has matured to the point where TTL-based cleaning
is no longer necessary.
Setting :any:max-cache-size to unlimited or 0 is no longer supported
and falls back to the default (90% of physical memory).
Merge branch 'ondrej/no-ttl-based-cleaning' into 'main'
See merge request isc-projects/bind9!11459
Now that TTL-based cleaning has been removed, the dns_expire_ttl enum
value, its switch case in expireheader(), and the deletettl stats counter
(text, XML, JSON) are all dead code. Remove them so the stats channel
no longer reports a permanently-zero counter.
Instead of doing a full sweep of all names and entries before dumping,
expire stale entries lazily as they are encountered during the dump
iteration. This aligns with the QPcache approach of avoiding separate
TTL-based cleaning passes.
dns_adb_flush() retains its explicit full sweep since it needs to
force-expire everything.
Lower the hard floor for max-cache-size from 2 MB to 8 MB to support
resource-constrained environments (e.g. CPE devices) while remaining
safe for LRU-only eviction.
Extract the inline max-cache-size logic from configure_view() into
reusable helpers: configure_max_cache_size(), default_max_cache_size(),
max_cache_size_as_percent(), and sanitized_max_cache_size().
Move DNS_CACHE_MINSIZE and DNS_ADB_MINADBSIZE to public headers and
remove the SIZE_AS_PERCENT sentinel.
Since TTL-based cache cleaning has been removed, an unlimited
max-cache-size would eventually exhaust system memory.
Both 'max-cache-size unlimited;' and 'max-cache-size 0;' now fall
back to the default value (90% of physical memory for recursive
views).
The heaps have been removed, so the separate heap memory context
(hmctx) is no longer needed. Remove it from both dns_cache and
dns_qpcache, along with the HeapMemInUse statistics.
The experiments show that the SIEVE-LRU based mechanism is good enough
as the only mechanism for cleaning up the expired entries from the
cache.
This simplifies the internal logic and memory usage of the cache.
The disadvantage is that the cache use will organically grow until it
hits the overmem cleaning mechanism.
The advantage is that the measurements show that BIND 9 is well behaved
even with 512 MB cache under heavy load.
dns_rdataslab_fromrdataset() set .expire to rdataset->ttl, but the
only consumer (qpcache_addrdataset) immediately overwrote it with
now + rdataset->ttl. Remove the redundant initialization and set the
expire time only once.
The `named` resolver now uses a separate "delegation database" to store zone referral data instead of the DNS cache. This new database holds the NS RRset on the parent side of a zone cut, as well as necessary glue records that were included in the referral. The NS RRset from the child side is cached in the DNS cache and is not used for name resolution.
This will be a step toward simplifying resolver logic and also supporting DELEG referrals.
Closes#3311
Merge branch 'colin/deleg-resolver' into 'main'
See merge request isc-projects/bind9!11621
The default max-cache-size on the system test being enforced to 2MB
(from the -T switch). It means the main cache size is 6/8 of it (1/8
being reserved for ADB and 1/8 for the delegation database).
However, the minimal cache size of the main cache is 2MB, which means
that the main cache size will actually be 2MB anyway when running the system tests.
This was breaking some resolver tests were failing because when the
cache was shared and named reload, the previous cache size was, indeed,
2MB whereas the newly requested size only 6/8 of it (that happened
before the size is bumped to the minimal value).
Set the ns1 of resolver tests max cache size of 3MB, so even 6/8 of the
max cache size is more than the minimum 2MB of the main cache size,
which enable to keep the shared cached between views for this test
between reloads.
The delegation DB now uses the same amount of memory than ADB, which is
1/8 of the `max-cache-size`.
The main cache database, instead of using `max-cache-size`, now use the
"remaining" part of it, after the delegation DB and ADB took their part,
so 6/8.
This avoid blowing up the host memory, typically when specifying
`max-cache-size 95%`, as the global cache usage would go way ahead 100%.
Add a simple case (i.e. not relying on zone being generated during the
test) ensuring that RPZ NSDNAME rules are correctly applied using the
parent-centric resolver.
When a resolver+auth server has a delegation on a local zone and has a
glue, the glue can only be for in-domain NS.
In this case, when the resolver is looking at the zonecut,
`dns_view_bestzonecut()` synthesizes a delegset from an NS rdataset
found in the local zone (the delegation inside auth zone), and ignores
the glues if any.
As a result, the delegset will contain a single delegation of type
DNS_DELEGTYPE_NS_NAMES, which leads to an ADB fetch. But it's actually an
in-memory fetch, because in this case, the fetch will immediately find
the A/AAAA glues from the local zone.
An alternative approach (not chosen here) would be to make
`dns_view_bestzonecut()`, when converting an NS rdataset into a
`dns_deleg_t`, check for glues for the delegation in the auth zone, and
add those in the `dns_deleg_t`. The delegation would be of type
DNS_DELEGTYPE_NS_GLUES which would avoid the ADB name lookup.
However, that's extra code, extra logic and complexities, for a lookup
that will be done in memory anyway, just a bit later. So for now, this
is not implemented that way.
The test is added, however, to confirm that there is no attempt from the
resolver to get the NS fron the child zone.
because the cache no longer stores delegation (parent-side) NS rrsets,
and authoritative (child-side) NS rrsets don't affect recursion,
it no longer makes sense for qpcache_find() to look for NS rrsets
and return DNS_R_DELEGATION. that code has been removed.
the cache still does search for covering DNAME records. the
check_zonecut() function has been renamed to check_dname() for clarity.
related changes:
- one test case has been removed from the mirror system test, because it
tested the behavior of a cached delegation.
- query_checkrrl() and rpz_rrset_find() have been updated so they no
longer expect cache responses to have DNS_R_DELEGATION response codes.
When dumping the cache, include the contents of the delegation
database. Add a new 'rndc dumpdb -deleg' option, which dumps
the delegation database exclusively.
While the delegation database dumping format mimic the zone file format,
the API does not use the `dns_master_style_t` configuration (i.e. to
specify how many spaces/tab are used between each RR fields), because
the generic API handling this relies on databse using `dns_rdataset_t`
as internal storage format. This can be improved later.
The get_nameservers path in rctx_nextserver() is only reachable from
rctx_referral(), which already detaches fctx->delegset. Assert that
it is NULL rather than redundantly detaching it, since
dns_view_bestzonecut() requires *delegsetp == NULL.
If both dns_view_bestzonecut() and dns_deleg_fromrdataset() fail,
delegset stays NULL. Passing it to ns_query_recurse() would crash
on the REQUIRE(DNS_DELEGSET_VALID(delegset)) in createfetch().
Return ISC_R_NOTFOUND instead, which lets the caller handle the
failure gracefully.
When resume_dslookup() receives ISC_R_SHUTTINGDOWN or ISC_R_CANCELED,
frdataset (fctx->nsrrset) was not disassociated. While fctx__destroy()
eventually cleans it up, leaving it associated keeps the underlying DB
node referenced longer than necessary.
When a referral lookup is triggered by a QMIN query, it should be
exempt from the fetches-per-zone limit just as the QMIN query itself
is.
Also restart the test server between the fetches-per-server and
fetches-per-zone tests so that leftover statistics from the former
do not pollute the latter.
Another fix is because zone spills and general query drops are no longer
in a strict >= relation (on a parent-centric resolver), so check that
both counters are non-zero instead.
The resolver now uses glue addresses from the parent side of a
zonecut without triggering an additional address lookup. Update the
test involving a nameserver target name below a DNAME so that the
delegation does not use glue.
Stop storing the NS referral into the main cache when processing a
negative response. These records are already cached in the delegation
database and are not needed elsewhere.
Update dnssec tests that relied on parent-side NS RRsets being
returned in recursive query responses.
The ADB flushtree test was failing because the test zone
(flushtest.example.) uses an in-domain nameserver with parent glue,
so the ADB cache was never populated.
Add a new zone with an out-of-domain nameserver to force an ADB
lookup and ensure the flushtree test exercises the intended code
path.
When `query.c` finds a zonecut in the main cache (e.g. from stale NS
records), it must still use the correct delegation for recursion. Look
up the delegation DB via `dns_view_bestzonecut()` first; fall back to
`dns_deleg_fromrdataset()` only if no delegation is found.
This might also be done inside `query_lookup()` instead, with the `qctx`
holding a delegset property, but that approach needs further work to
avoid breakage and it is not clear so far if there would be other use
case of it. Current approach is simpler for now.
Remove the dynamic NS update loop from the delegation expiry test.
With the delegation DB, it is sufficient to wait for the delegation
to expire (after 5 seconds) and verify that names below the removed
zone return NXDOMAIN.
In 'additional', pre-cache the A RRset for ns1.rt.example so the
additional-data handling in the cache can be tested; previously this
was cached as part of resolution, but now must be queried explicitly.
In 'cookie', pre-cache an NS to prevent a QMIN query from distorting
log results and causing a test failure.
In 'resolver', increase the expected query count in the timeout test.
Remove expected queries for ns2 addresses (and ns.b.stale.) from the
qmin system test. The parent-centric resolver no longer attempts to
get the child-side NS of the delegation, so these queries do not
occur.
As named is now parent-centric, the global query count can be lower (in
particular for queries which has a long delegation chain), as the
resolver doesn't proactively resolve an NS name when a glue is provided
by the parent-side of a zonecut.
Update camp system test to lower the max-query-count of ns9, to ensure
the max-query-count limit enforcement is still working.
The resolver now uses glue addresses from `dns_deleg_t` objects stored
in the delegation database. The main cache is still used for ADB A/AAAA
lookups when no glue is available for a nameserver name.
The resolver's `fctx_getaddresses()` is refactored to, for each
delegation of the delegation set, try to get the address-based finds,
then nameserver name lookups. (Later, the logic to handle DELEG
`include-delegparm=` will be hooked there too.)
Add a new ADB API function that creates a find from a list of addresses
rather than by looking up nameserver names. This enables the resolver
to handle address-based delegations (NS-based with glues or DELEG with
addresses) and name-based delegations uniformly (i.e. the list of finds
from ADB is handled the same way no matter the type of the delegation).
Add the 'minimalresponses' system test in order to cover the multiple
combinations of server 'minimal-responses' configurations and contexts
(resolver, auth only, etc.) when filling the ANSWER, AUTHORITY and
ADDITIONAL sections.
Previously, when answering from the cache, and when minimal-responses
was not set, we added the best known zone cut to the authority section
of the response message, using dns_db_findzonecut() to look it up in
the DNS cache. Since the DNS cache will no longer be used to store
parent-side NS RRsets, it will now be possible for an ancestor node
to be used as the zone cut, leading to the wrong NS record being
included.
There are various ways we could correct this:
1. Use dns_deleg_lookup() instead of dns_db_findzonecut() to find the
zone cut. But currently, the deleg database stores only the server
addresses for the delegation, not the full NS RRset; this would need
to be changed.
2. Look up <name>/NS whenever we cache a referral; that way we'll get
the child-side NS RRset and cache that, and we can retrieve it when
building the response.
But the solution chosen here is simply not to look up the NS record
when answering from the cache, effectively making "minimal-responses
yes;" mandatory for queries answered from the cache.
System tests have been updated as needed, so they no longer expect
NS RRsets in the authority section of recursive responses.
Function `dns_view_bestzonecut()` now uses the delegation DB instead of
the main cache when looking up at the cache.
As a result, replace `dns_rdataset_t` (representing an NS RRset) with
`dns_delegset_t` in `dns_view_bestzonecut()` and
`dns_resolver_createfetch()` APIs. The resolver and query processing now
use the delegation DB instead of the cache for zonecut lookups.
In the case of the delegation lives in the local database, the locally
found `rdataset` is internally converted into a `dns_delegset_t` object.
From caller POV, it doesn't change anything: a delegation set is a
read-only object which can be used as long as needed and must be
detached one it's done with it.
Create an instance of the delegation database in the view created by
delv in resolver mode. As the default `named` configuration is not in
use, also specifically configure the `max-delegation-servers` setting
using the same default value than `named`.
The resolver now caches NS records and their A/AAAA glues from referral
answers into the delegation database.
A new `cache_delegns()` function extracts NS names and associated glue
addresses from the authority/additional sections of a referral answer
and use those informations to build a delegation set, which is then
inserted into the delegation database.
The created delegation set contains a delegation per NS RR. If the NS RR
has matching A/AAAA RR, the delegation only store the addresses and not
the name. (Note this is technically possible to group all NS RR which
doesn't have glues into a single delegation, and the implementation can
be changed in that way in the future).
Each view has its own instance of the delegation database (they are
never shared between views), but a server restart/reload preserve the
delegation database state.
Add `dns_delegdb_t`, a qpmulti-based database enabling to lookup a
delegation set (`dns_delegset_t`) from a zonecut name (`dns_name_t`). A
delegation set object essentially contains an expiration time and a list
of delegation (`dns_deleg_t`). Finally, a delegation can be either:
- A list of IP addresses (`isc_netaddrlist_t`), for NS-based delegation
providing glues or DELEG-based delegation using `server-ipv4=` or
`server-ipv6=`;
- Or a list of nameserver names, for NS-based delegation without glues,
or DELEG-based delegation using `server-name=`;
- Or a list of nameserver names, for DELEG-based delegation using
`include-delegparam=`.
The delegation database API provides lookup by closest zonecut,
delegation and delegation set builders as well as insertion of those
newly built delegation set, dumping to a `FILE *`, conversion from an NS
rdataset to a delegation set, deletion of a specific zonecut or all the
sub-tree of a given zonecut.
A memory context is internally used inside the delegation database and
can be constraint to a maximum size. Once it gets close to its maximum
size and a new delegation set is inserted into the database, a
reclamation flow is run internally removing the least recently used
entries.
The delegation set and delegation objects are, once they been inserted
into the database, read-only object. Thus, the caller can use them
without concurrency or locking concerns, and must detached them once its
done with it.
Add an `isc_netaddrlink_t` type wrapping an `isc_netaddr_t` and an
`ISC_LINK`. This enable to build list of `isc_netaddr_t` without
increasing the memory footprint of existing usages of `isc_netaddr_t`
(which doesn't require to be linked).
Flushing the name when NTA expires causes problems for the ongoing
resolving process. Do not flush the name from the cache. Instead,
the resolver should do the flushing (this is planned to be implemented
later).
Merge branch 'aram/revert-nta-flush-on-expire' into 'main'
See merge request isc-projects/bind9!11765
Flushing the name when NTA expires causes problems for the ongoing
resolving process. Do not flush the name from the cache. Instead,
the resolver should do the flushing (this is planned to be merged
next).
After KeyTrap, the temporal DNSSEC were originally hard errors that
caused validation failures even if the records had another valid
signature. This has been changed and the RRSIGs outside of the
inception and expiration time are not counted as hard errors. However,
these errors are not even counted as validation attempts, so excessive
number of expired RRSIGs would cause some non-cryptograhic extra work
for the validator. This has been fixed and the temporal errors are
correctly counted as validation attempts.
Closes#5760
Merge branch '5760-count-DNSSEC-temporal-errors-as-validation-attempts' into 'main'
See merge request isc-projects/bind9!11589
After KeyTrap, the temporal DNSSEC were originally hard errors that
caused validation failures even if the records had another valid
signature. This has been changed and the RRSIGs outside of the
inception and expiration time are not counted as hard errors. However,
these errors are not even counted as validation attempts, so excessive
number of expired RRSIGs would cause some non-cryptograhic extra work
for the validator. This has been fixed and the temporal errors are
correctly counted as validation attempts.
