Move the per-Link-value processing into a new cache_extract_link_hints()
helper. cache_extract_hints() continues to walk the cached row's HDR
blocks and now invokes the helper once per Link header it finds.
No behaviour change. This prepares the helper for reuse by an upcoming
hints-only storage path that extracts Link headers directly from the
live response HTX rather than from a cached row.
Add four regtests to validate the behavior of the HTTP cache when early
hints are enabled.
- early_hints.vtc: covers 103 emission on stripped entries and that the
"no-early-hints" rule keyword suppresses it.
- early_hints_ratio.vtc: covers cap-driven eviction of hints entries when
the ratio is reached.
- early_hints_parsing.vtc: pins down the (deliberately strict) Link
header parsing behaviour of link_is_hint() so future relaxations
cannot silently change hint detection.
- early_hints_multiblock.vtc: checks that hints are still extracted when
the relevant Link headers span more than one cache block.
It is now possible to give a "ratio" parameter to the "early-hints"
option to indicate what percentage of blocks to reserve for hints-only
entries. Defaults to 25%.
A "cache-use" directive can now specify the "no-early-hints" keyword
to suppress 103 Early Hints emission for matching requests, when used
with a cache that has the "early-hints" setting enabled.
Add a cache_hint_hits counter to the per-proxy HTTP stats, incremented
each time the cache emits a 103 Early Hints response. It is reported
in the CSV/typed outputs, on the stats page next to the cache hits,
and exported to prometheus as http_cache_hint_hits_total. Since the
new field changes the shm stats file mapping, its minor version is
bumped along with the object size assertion.
The HTTP cache now supports sending 103 Early Hints responses if
enabled, without the need for explicit per-rule configuration.
Two paths feed the 103 response depending on the matched entry. A
stripped entry (CACHE_EF_STRIPPED) carries its Link records
pre-extracted alongside the cache entry header and is copied out of
the row as-is. A complete entry only carries the cached HTX; the
Link headers are walked on the fly at request time, deliberately
avoiding the extra storage cost of duplicating them alongside the
body.
The on-the-fly path has a cost: a workload where many hits land on
expired complete entries re-parses the cached response on each hit.
To bound this, the first such hit also strips the entry in place
when conditions allow (the hints pool is below its cap, and the row
is not currently being read by another request); subsequent hits for
the same URL then take the cheap pre-extracted path.
To let the cache-use lookup reach that logic, get_entry() now returns
expired entries to callers passing delete_expired=0; such callers are
responsible for checking entry->expire themselves.
Pull the 103 emission code out of http_action_early_hint() and split
it into http_early_hint_start() (open the response and return the htx)
and http_early_hint_end() (close with EOH and forward). The split
shape lets callers add their own Link headers in between, which is
what the HTTP cache will need when it gets support for sending 103
Early Hints from cached responses.
We now use the make_room callback from shctx to customize the eviction
procedure. We attempt to balance eviction between hints and full-blown
entries by capping the number of blocks used by hints entries to 25% of
the size of the cache. Upon eviction of a full entry, we first try to
strip it to a hints-only entry if possible.
A Link header's value (which may carry multiple comma-separated links)
is limited to 1024 bytes for hint extraction. Longer values are skipped
entirely and contribute no hint records.
Introduce two LRU lists in struct cache (full_lru and hints_lru) and an
"lru" link in struct cache_entry, plus a CACHE_EF_STRIPPED flag to
mark hints-only entries. A new cache_row_reattach() helper places each
entry in the appropriate LRU when a row is returned to the cold pool.
No behaviour change yet: the lists are populated but no code consumes
them. The next commit uses them to drive early hints-aware eviction.
This introduces a make_room() callback that consumers can provide in
order to implement more specific eviction strategies. To be used in the
HTTP cache code. This goes in hand with a new shctx_row_truncate()
function that allows consumers to truncate a row, which can be used in
the make_room() callback.
Add a flags field to struct cache_entry and replace the complete
field with a CACHE_EF_COMPLETE flag.
Likewise, add a flags field to struct cache and replace the
vary_processing_enabled configuration boolean with a
CACHE_CF_VARY_PROCESSING flag.
Add http_next_hdr_value() to iterate over the comma-separated values
that some headers contain. Add http_get_hdr_param() to iterate over the
';'-separated parameters that may follow a value.
Currently unused but will be used for parsing Link headers in the cache
to support 103 Early Hints, and could also clean up Cache-Control
parsing among others.
In http_action_req_cache_use() with Vary, the row of the secondary entry
matching the request is detached from the avail list for reading. Since
get_secondary_entry() only rejects expired entries, that secondary may
be incomplete (still being written by another stream), and in that case
the request was forwarded without reattaching the row, leaking its
blocks out of the avail list (refcount and nbav never restored). Over a
Vary workload this eventually prevents the cache from reserving any row.
Reattach the row before returning, as the other paths that give up on
the cache entry do.
Present since the Vary support was added in 2.4 (1785f3dd9); to be
backported as far as 2.6.
Fields 114 to 117 (reqbin, reqbout, resbin, resbout) were added to the
CSV output but never listed in the stats description of management.txt.
This should be backported up to 3.3.
The length clamp capped the request to the whole row length but ignored
<offset>, so a read starting past the beginning of the row could run
beyond the stored data and copy stale bytes from the reserved tail of
its blocks. No current caller reads past first->len so this was never
triggered, but the helper is public and its contract does not forbid it.
Clamp against (first->len - offset) instead, returning early when the
offset is already at or past the stored data.
This may be backported along with the shctx_row_data_get() offset fix.
The loop advanced the block pointer only after the copy, but skipped
blocks preceding <offset> with a continue, which bypassed that advance.
As a result any read starting past the first block copied nothing.
This notably affects conditional requests: should_send_notmodified_response()
reads the stored ETag at entry->etag_offset, so a cached response whose
ETag header sits beyond the first block fails revalidation and is served
in full (200) instead of 304.
Rewrite the loop as a for() so the block pointer always advances,
regardless of whether the current block is copied or skipped.
This should be backported as far as 2.6.
It was reported by Red Hat in partnership with AISLE Research that
proxies updated via a stats page in "stats admin" mode were not subject
to the filtering of "stats scope".
It is totally true and was never meant to be when "stats scope" was
added in 1.2 in 2006 while "stats admin" arrived 5 years later in 1.4,
nor was it even implied by the doc, but it is easy to see how someone
who would combine the two could expect them to work together:
- "stats scope" only limits which proxies are shown, the purpose being
to avoid polluting the page with a long list of auxiliary proxies that
take ages to render (e.g. those dedictated to stick-tables etc). A
common use case is to only show known frontends so that numerous
backends are not listed, it's most often used on public access.
- "stats admin" is used to permit to change a proxy or a server's
state and is protected using HTTP Basic auth, cleartext passwords
by default, and is mostly meant to be used from trusted sources,
hence private access.
The two are almost never seen together (no single occurrence found
in github issues for 92 occurrences of "stats admin" and 5 of
"stats scope"), but admittedly, if someone were to combine the two,
seeing buttons on the listed proxies only, they would surely expect
to only be able to act on those.
So in order to clear this void, this commit does the following upon
a POST:
- when a scope is defined, the requested proxy must exist and match
the list of permitted ones, otherwise a DENY is returned. This
ensures that the POST cannot be abused either to infer the existence
of a given proxy name.
- when no scope is defined, a non-existing proxy continues to return
an "unknown proxy" error.
The doc was updated to explain this new behavior. This commit should
be backported to stable versions, along with the previous patch it
depends on:
MINOR: stats: factor the proxy vs scope check into its own function
The stats code was split between 2.9 and 3.0, so the changes must all
be performed in stats.c before 3.0. Also, ctx->http_px does not exist
earlier, but http_px is simply the calling stream's backend (s->be at
some places).
Thanks to Red Hat and AISLE Research for reporting this.
We'll need to check for matching between a proxy and a stats scope at
a few more places, so let's first move the code out of its current
function (stats_dump_proxy_to_buffer) into its own function
(stats_proxy_in_scope). At this point it doesn't change anything.
Note that this will have to be backported as this will be used with
a subsequent fix.
When a applet is waiting for a connection, like Socket applet in lua, reads
are blocked. When the connection is finally available , a read activity is
reported and reads are reenabled (flag SE_FL_APPLET_NEED_CONN is removed).
However, most of time (always ?), this operation is performed in the context
of the connection and the applet is woken up. It means the task expiration
date is not updated where the read activity is reported. It is an issue for
small timeouts because the task may be queued in past, triggering a BUG_ON
in sc_notify(). In fact, a read activity must never be reported in another
context of the endpoint itself or the upper stream, specifically to properly
set the task expiration date.
To fix the issue, in sc_chk_rcv(), the applet is only woken up in that case
and the reads are re-enabled when the applet is executed. For applets using
the new API, it is performed in sc_appet_sync_recv(). For legacy applets, it
is directly performed in task_run_applet().
This patch is related to #3442. It must be backported as far as 2.8. On the
2.8, only task_run_applet must be fixed because sc_applet_sync_recv() does
not exist.
In lua, the timeout that can be set on an Socket applet was applied on
client side and server side. In fact, it must only be applied on server
side, to apply a timeout on the connection. The client side is the applet.
This patch should fix the issue #3442. It must be backported to all
supported versions.
clang (rightfully) complains that when neither HA_CAS_IS_8B nor
HA_HAVE_CAS_DW is defined, there is a mismatch when we're doing a CAS,
as we're mixing unsigned int and int.
To fix that, just turn old_state to an unsigned int, as it should be.
This should fix github issue #3441
gcc 16 whines that in human_number(), the calls to snprintf() could lead
to a truncated output because the buffer is not big enough. This is not
really true, because we always have a limited number of digits, but gcc
can't figure that out, so just bump the buffer size from 5 bytes to 8
bytes to make gcc happy.
Released version 3.5-dev2 with the following main changes :
- MINOR: proxy: permit to report version info for option deprecation
- MAJOR: proxy: remove support for "dispatch" and "transparent" proxy keywords
- MEDIUM: cli/show-fd: no longer accept filtering for dispatch mode
- CLEANUP: connection: remove some checks for objt_proxy(conn->target)
- CLEANUP: backend: drop checks for OBJ_TYPE_PROXY in connect() code
- CLEANUP: trace: remove backend retrieval attempt from conn->target
- MAJOR: ot: remove deprecated OpenTracing support
- BUG/MEDIUM: h3: fix trace crash on frontend response headers
- CI: github: remove OpenTracing leftovers
- BUG/MEDIUM: mux_quic: fix memory leak of rx app_buf on stream free
- MINOR: ssl: export ssl_sock_init_srv()
- MEDIUM: httpclient: initialize the httpclient with default SSL values
- BUG/MINOR: hq-interop: fix transcoding of wrapping response buffer
- BUG/MINOR: hq-interop: support transcoding of absolute URI
- BUG/MINOR: h3: adjust HTTP headers traces
- MINOR: mux_quic: add minimal traces for QUIC MUX init/release
- MINOR: hq-interop: add request start-line traces
- MINOR: hq-interop: trace transcoding of response status line
- MINOR: hq-interop: trace HTX headers
- BUG/MEDIUM: server: initialise agent.health in srv_settings_init()
- BUG/MINOR: sample: set SMP_F_CONST on srv_name fetch
- MINOR: server: distinguish name references with new SRV_F_NAME_REFD flag
- MEDIUM: server: add 'set server name' CLI command for runtime server renaming
- REGTESTS: server: add test for 'set server name' CLI command
- DOC: server: document 'set server name' CLI command
- BUG/MEDIUM: servers: Use a refcount for port_range and free it properly
- MINOR: hbuf: new lightweight hbuf API
- MINOR: init: add no listener mode
- MINOR: trace: add definitions for haload streams
- MINOR: hldstream: add definition of hldstream struct objects
- MINOR: obj_type: add OBJ_TYPE_HALOAD for haload stream objects
- MINOR: stconn: add sc_hastream() and __sc_hastream() helpers
- MINOR: stconn: export sc_new()
- MINOR: server: export functions used during server initialization
- MINOR: haload: import source code and documentation
- MINOR: log: add app_log_raw() and send_log_raw() for binary-safe logging
- BUG/MINOR: init: fix default global settings being overwritten by -G
- BUG/MINOR: tools: fix invalid character detection in strl2ic()
- BUG/MAJOR: htx: Don't swap buffers for empty HTX message with an error
- CLEANUP: applet/http-client: Don't needlessly copy HTX flags after htx_xfer()
- BUG/MINOR: mux-quic: Fix handling EOM after in qcs_http_rcv_buf()
- BUG/MINOR: http-htx: Don't by-pass HTX API when merging cookie values
- BUG/MEDIUM: h3: fix parser desync on error with multiple frames
- BUG/MINOR: mux_quic: prevent multiple STOP_SENDING emission per stream
- BUILD: quic: workaround a gcc bug saying "maybe used uninitialized" when USE_TRACE=0
- CLEANUP: traces: get rid of a few rare empty args in TRACE calls
- MINOR: compiler: add a macro to ignore all arguments
- MINOR: trace: always pretend to use args when disabled
- BUILD: ssl: avoid a wrong null deref warning in ssl_sock_handshake
- CLEANUP: haproxy: remove -dt parsing and help when !USE_TRACE
- CLEANUP: mux-h2/traces: remove unused trace code when building without USE_TRACE
- CLEANUP: debug/trace: remote "debug dev trace" when USE_TRACE is not set
- BUG/MINOR: trace/quic_frame: use buf, not trace_buf in chunk_frm_appendf()
- CLEANUP: trace/h3: allow to disable traces in H3
- CLEANUP: trace/config: do not register section "traces" with USE_TRACE=0
- CLEANUP: trace/tree-wide: drop trace decoding/definition when USE_TRACE=0
- BUILD: makefile: only build trace.c and ssl_trace.c when USE_TRACE is set
- BUILD: makefile: add macros enable_opts and disable_opts
- BUILD: makefile: add an option to enable or disable HTTP/2 (USE_H2)
- BUILD: makefile: add an option to enable or disable FCGI (USE_FCGI)
- BUILD: makefile: add an option to enable or disable SPOE (USE_SPOE)
- BUILD: makefile: add a new generic target "tiny"
- BUG/MEDIUM: mux_quic: do not free QCS if STOP_SENDING to sent
- MINOR: mux_quic: use separate error code for STOP_SENDING
- MINOR: mux_quic: adjust shut stream callback
- BUG/MEDIUM: mux_quic: complete stream shutdown for read channel
- BUG/MINOR: quic: ignore STREAM after MUX closure on BE side
- BUG/MEDIUM: fd: Fix a deadlock when closing other tgroups fds
- MEDIUM: quic: remove deprecated keywords
- DEV: patchbot: keep the review start in sync with the radios on reload
- DEV: patchbot: only display the first 8 chars of the commit id
- DEV: patchbot: pass the branch version to the generated page
- DEV: patchbot: let the page fetch the shared review state
- DEV: patchbot: let the page save the review edits to the server
- DEV: patchbot: let the page edit and delete whole notes
- DEV: patchbot: gray the save button when there is nothing to save
- DEV: patchbot: don't pretend a save succeeded when the server ignored it
- DEV: patchbot: tolerate polluted save responses and show server warnings
- DEV: patchbot: repeat the syncing buttons at the bottom of the page
- DEV: patchbot: update: add an awk backend to persist review edits
- DEV: patchbot: update: return the stored overlay as JSON on GET
- DEV: patchbot: update: support replacing a whole note blob (setnotes)
- DEV: patchbot: update: report git commit failures in the save response
- DEV: patchbot: update: report the exact git error to the user
- DEV: patchbot: update: never write to stderr, thttpd sends it first
- DEV: patchbot: update: name the touched commits in the storage messages
- DEV: patchbot: document the shared review persistence
- BUG/MINOR: haload: fix spurious task wakeup in hld_strm_task()
- BUG/MINOR: hbuf: treat unexpected escape sequences as literals
- BUG/MEDIUM: tcpcheck: Add proxy used for healthcheck sections in proxies list
- BUG/MINOR: sample: Fix a possible underflow on be2hex for large chunk size
- MINOR: chunks: Add function to get a large/regular chunk depending on a buffer
- BUG/MEDIUM: chunk: Review chunks usage to not retrieve a large buffer by error
- MINOR: htx: Add a field to save the headers data size
- MEDIUM: htx: Be sure size of headers never exceed regular buffer on update
- BUG/MINOR: stream: Fix custom timeouts initialization when setting backend
- REGTESTS: Improve script testing the set-timeout action
- BUG/MINOR: stream: Fix custom max-retries initialization when setting backend
- MINOR: stream: Add be_max_retries/cur_max_retries sample fetch functions
- BUG/MINOR: http-conv: Make url-dec failed if no space for trailing null byte
- MAJOR: mworker: remove deprecated "master-worker" global keyword
- DOC: haterm: add a missing 'haterm' build target on an example
- DOC: readme: add a pointer to haterm/haload docs
- MINOR: ocsp: Do not see ocsp loading failures as fatal anymore
- REGTESTS: Remove unused `add_range_to_test_list` function from `scripts/run-regtests.sh`
- REGTESTS: Remove unused `_version` function from `scripts/run-regtests.sh`
- REGTESTS: Migrate REQUIRE_OPTION to `haproxy -cc`
- REGTESTS: Remove support for `REQUIRE_OPTION` from scripts/run-regtests.sh
- REGTESTS: Migrate `REQUIRE_SERVICE=prometheus-exporter` to a `feature(PROMEX)` check
- REGTESTS: Remove support for `REQUIRE_SERVICE` from scripts/run-regtests.sh
- CI: github: update vmactions/freebsd-vm to 14.4
- MINOR: haload: move statistics header printing to mtask_cb
- MINOR: haterm: add note about QUIC usage on SSL port
- MINOR: haload: allow "0" shortcut for IPv4 bind address
The address parser was failing when using "0" as a shortcut for INADDR_ANY.
This patch intercepts the "0" prefix and normalizes it to "0.0.0.0"
before the address is parsed.
Move the header printing logic from hld_init to mtask_cb using a static flag.
This ensures the header is displayed only when the main task starts
processing, avoiding issues where the header might be printed before other
haproxy initialization messages or unexpected early exits.
Until now, if the call to 'ssl_sock_load_ocsp' raised an error, because
of a missing issuer certificate for instance, we would send a fatal
error code and the init would fail.
With this patch we only consider this as a non-critical error and we
will send a warning instead. In such a case the OCSP response will not
be loaded and stapling or auto update features will not work for the
certificate.
People landing on the page seeking for help on how to build haterm/haload
need to have a direct entry on this page, so let's add a line pointing to
both docs.
This keyword, when alone, was already deprecated in 3.3 and marked
for removal in 3.5. Now using it results in an error message inviting
to start haproxy with -W or -Ws instead.
In addition, "master-worker exit-on-failure", which used to automatically
enable the master-worker mode, now doesn't set it and complains if used
without. -W/-Ws. A special case was made for config checks though (-c).
Also worth noting that "daemon" applies to master as well as max-reloads,
and will soon need to be adjusted.
for url-dec converter, a trailing null byte is added at the end of the input
sample because it is requested by url_decode() function. However, when the
buffer was full, the last byte was crushed by the trailing null byte. In
that case, the last character was lost and not decoded.
Now, the converter just fails by returning 0.
This patch could be backported to all supported versions but it is a very
minor issue.
These new samples can be used to get, resepectively, the backend value for
the maximum connection retries and the current value applied to the stream.
They could be used to know if a "set-retries" action was performed.
A dedicated reg-test was also added.
It is possible to overwrite the configured max-retries value with the
"set-retries" action. However there is an issue with the listeners. When the
backend is set, this value is reset with the backend value. However, when
the backend is unchanged, the custom value must be preserved.
To fix the issue, when the stream is created, we set the max-retries to the
listener value. It remains 0 for pure-frontend. And the backend value is
used only if it is not the same proxy.
The documentation of set-retries action was improved.
This patch should be backported as far as 3.2.
First, instead of using syslog to test the result, http-after-response rules
are now used. It is a bit simple. In addition, all custom timeouts are
tested. Some testcase were useless and were thus removed.
In addition, the script was moved in "reg-tests/http-rules/" directory.
Timeouts may be overwritten via the action "set-timeout". Client timeout is
supported on frontends, connect/queue/tunnel/server timeouts are supported
on backends and tarpit timeout is supported on both.
However, there is an issue with the listener. In that case, the action is
always executed in the frontend context and the backend-side timeouts are
then reset, when the backend is selected. If it is another backend, it is
logical. But most of time, the backend is unchanged. And in that case, these
timeouts must be preserved.
Documentation of set-timeout action was improved.
This patch should be backported to all supported versions. Be carefull, all
timeouts are not supported in 3.3 and lower (3.3->3.0: server/tunnel/client
- 2.8->2.6: server/tunnel)
When an HTX header or an HTX start-line is added or updated, a test is now
performed to be sure the whole headers size, including the start-line never
exceeds the regular buffer size. It is mandatory because it is now possible
to use larger buffers. However, it remains impossible to send HEADERS frame
in H2 and QUIC larger than a regular buffer. So we must be sure the HTTP
analysis will never produce too big headers because they will be rejected
later, at the forwarding stage. The purpose of large buffers is to be able
to store large payload, larger than regular buffers. Headers must remain
quite small.
This commit depends on the following one:
MINOR: htx: Add a field to save the headers data size
Both should be backported to 3.4 to avoid any trouble with large buffers. It
is not strictly speaking a bug, but it will avoid hazardous behavior
depending on the payload size.
The size of headers present in an HTX message are now counted. A dedicated
field was added in the HTX structure to do so. This patch is quite simple
but it will be mandatory to be able to perform some tests on headers. One of
them is to be sure headers never exceed the regular buffer size, even when a
large buffer is used.
All calls to get_trash_chunk_sz() and alloc_trash_chunk_sz() were reviewed
to be sure we never retrieve a large chunk when it is not expected. Some
calls were not upgrade, but several calls now rely on get_best_trash_chunk()
and alloc_best_trash_chunk() functions.
The idea of the fix is to get a large buffer only when the original buffer
is already a large buffer and the expected size of data exceeds the size of
a regular buffer. This should prevent unexpected memory usage.
This commit relies on the previous one:
MINOR: chunk: Add function to get a large/regular chunk depending on a buffer
Both must be backported to 3.4.
get_best_trash_chunk() function was added to be able to get a large or a
regular chunk depending on a given size but never larget than a given
buffer. It will be usefull in a futur fix, to prevent unexpected large chunk
usage.
alloc_best_trash_chunk() function is similar but instead of returning one of
the static chunks, it allocate it from the corresponding pool, large or
regular.
For a chunk size exactly equal to bufsize/2, the "max_size" variable could
underflow if a separator is provided. Indeed, "max_size" is first set to
(trash->size - 2 * chunk_size), so to 0. And on the first iteration, the
separator length is removed, making it to underflow.
On 3.3 and lower it is especially an issue with samples larger than
bufsize/2 because data could be written ouside of the buffer. On 3.4 and
3.5, it is not an issue because we fail to retrieve a chunk.
This should be backported to all supported versions.
