Added the span context injection and extraction layer that bridges the
OTel C wrapper's propagation API with HAProxy's HTTP headers and text map
carriers.
The new otelc.c module implements four public functions that wrap the
OTel C wrapper's context propagation methods: flt_otel_inject_text_map()
and flt_otel_inject_http_headers() serialize a span's context into
a text map or HTTP headers carrier for outbound propagation, while
flt_otel_extract_text_map() and flt_otel_extract_http_headers()
deserialize an inbound carrier into an otelc_span_context for parent
linking.
Each direction uses a pair of callbacks registered on the carrier
structure. The injection writers (flt_otel_text_map_writer_set_cb and
flt_otel_http_headers_writer_set_cb) store key-value pairs emitted by the
SDK into the carrier's text map via OTELC_TEXT_MAP_ADD(). The extraction
readers (flt_otel_text_map_reader_foreach_key_cb and
flt_otel_http_headers_reader_foreach_key_cb) iterate the carrier's text
map entries and pass each pair to the SDK's handler callback.
The scope context initialization in flt_otel_scope_context_init() now
calls flt_otel_extract_http_headers() to extract the span context from the
provided text map carrier and stores it in the scope context structure,
making extracted contexts available for parent linking in subsequent span
creation.
Implemented the scope execution engine that creates OTel spans, evaluates
sample expressions to collect telemetry data, and manages span lifecycle
during request and response processing.
The scope runner flt_otel_scope_run() was expanded from a stub into a
complete implementation that evaluates ACL conditions on the scope,
extracts span contexts from HTTP headers when configured, iterates over
the scope's span definitions calling flt_otel_scope_run_span() for each,
marks and finishes completed spans, and cleans up unused runtime
resources.
The span runner flt_otel_scope_run_span() creates OTel spans via the
tracer with optional parent references (from other spans or extracted
contexts), collects telemetry by calling flt_otel_sample_add() for each
configured attribute, event, baggage and status entry, then applies the
collected data to the span (attributes, events with their own key-value
arrays, baggage items, and status code with description) and injects the
span context into HTTP headers when configured.
The sample evaluation layer converts HAProxy sample expressions into OTel
telemetry data. flt_otel_sample_add() evaluates each sample expression
against the stream, converts the result via flt_otel_sample_to_value()
which preserves native types (booleans as OTELC_VALUE_BOOL, integers as
OTELC_VALUE_INT64, all others as strings), and routes the key-value pair
to the appropriate collector based on the sample type (attribute, event,
baggage, or status). The key-value arrays grow dynamically using the
FLT_OTEL_ATTR_INIT_SIZE and FLT_OTEL_ATTR_INC_SIZE constants.
Span finishing is handled in two phases: flt_otel_scope_finish_mark()
marks spans and contexts for completion using exact name matching or
wildcards ("*" for all, "*req*" for request-direction, "*res*" for
response-direction), and flt_otel_scope_finish_marked() ends all marked
spans with a common monotonic timestamp and destroys their contexts.
Connected the OpenTelemetry C wrapper library to the filter lifecycle by
implementing the library initialization, tracer creation, memory and
thread callbacks, shutdown sequence, and span completion.
The flt_otel_lib_init() function now verifies the C wrapper library
version against the compiled headers, calls otelc_init() with the absolute
configuration file path, and creates the tracer via otelc_tracer_create().
On success, it registers HAProxy pool-based memory callbacks
(flt_otel_mem_malloc, flt_otel_mem_free) and a thread ID callback
(flt_otel_thread_id) through otelc_ext_init(), so the C++ SDK allocates
span and context objects from pool_head_otel_span_context. A custom log
handler (flt_otel_log_handler_cb) is registered via otelc_log_set_handler()
to count OTel SDK internal diagnostic messages in the flt_otel_drop_cnt
counter.
The per-thread init callback now starts the tracer thread via
OTELC_OPS(tracer, start) instead of unconditionally returning success.
The deinit callback saves the tracer handle before freeing the
configuration, then shuts down the library via otelc_deinit() after the
pool is destroyed, ensuring the ext callbacks remain valid while the
configuration structures are still being freed. In debug builds, it logs
wrapper statistics, attach counters, and per-event HTX usage counters
before shutdown.
The runtime context cleanup in flt_otel_runtime_context_free() now ends
all active spans with a common monotonic timestamp via
OTELC_OPSR(span, end_with_options) before freeing them. The scope context
cleanup in flt_otel_scope_context_free() now destroys the underlying OTel
span context via OTELC_OPSR(context, destroy).
The parser gained static storage for the debug memory tracker
(OTELC_DBG_MEM) and its initialization in the parse entry point, used when
compiled with the OTELC_DBG_MEM flag.
Replaced the stub filter callbacks with full implementations that dispatch
OTel events through the scope execution engine, and added the supporting
debug, error handling and utility infrastructure.
The filter lifecycle callbacks (init, deinit, init_per_thread) now
initialize the OpenTelemetry C wrapper library, create the tracer from the
instrumentation configuration file, enable HTX stream filtering, and clean
up the configuration and memory pools on shutdown.
The stream callbacks (attach, stream_start, stream_set_backend,
stream_stop, detach, check_timeouts) create the per-stream runtime context
on attach with rate-limit based sampling, fire the corresponding OTel
events (on-stream-start, on-backend-set, on-stream-stop), manage the
idle timeout timer with reschedule logic in detach, and free the runtime
context in check_timeouts. The attach callback also registers the
required pre and post channel analyzers from the instrumentation
configuration.
The channel callbacks (start_analyze, pre_analyze, post_analyze,
end_analyze) register per-channel analyzers, map analyzer bits to event
indices via flt_otel_get_event(), and dispatch the matching events.
The end_analyze callback also fires the on-server-unavailable event
when response analyzers were configured but never executed.
The HTTP callbacks (http_headers, http_end, http_reply, and the debug-only
http_payload and http_reset) dispatch their respective request/response
events based on the channel direction.
The event dispatcher flt_otel_event_run() in event.c iterates over all
scopes matching a given event index and calls flt_otel_scope_run() for
each, sharing a common monotonic and wall-clock timestamp across all spans
within a single event.
Error handling is centralized in flt_otel_return_int() and
flt_otel_return_void(), which implement the hard-error/soft-error policy:
hard errors disable the filter for the stream, soft errors are silently
cleared.
The new debug.h header provides conditional debug macros
(FLT_OTEL_DBG_ARGS, FLT_OTEL_DBG_BUF) and the FLT_OTEL_LOG macro for
structured logging through the instrumentation's log server list. The
utility layer gained debug-only label functions for channel direction,
proxy mode, stream position, filter type, and analyzer bit name lookups.
Added the memory pool management and the runtime scope layer that track
per-stream OTel spans and contexts during request processing.
The pool layer in pool.c manages HAProxy memory pools for the runtime
structures used by the filter: scope spans, scope contexts, runtime
contexts, and span contexts. Each pool is conditionally compiled via
USE_POOL_OTEL_* macros defined in config.h and registered with
REGISTER_POOL(). The allocation functions (flt_otel_pool_alloc,
flt_otel_pool_strndup, flt_otel_pool_free) transparently fall back to
heap allocation when the corresponding pool is not enabled. Trash buffer
helpers (flt_otel_trash_alloc, flt_otel_trash_free) provide scratch space
using either HAProxy's trash chunk pool or direct heap allocation.
The scope layer in scope.c implements the per-stream runtime state. The
flt_otel_runtime_context structure is allocated when a stream starts and
holds the stream and filter references, hard-error/disabled/logging flags
copied from the instrumentation configuration, idle timeout state, a
generated UUID, and lists of active scope spans and extracted scope
contexts. Scope spans (flt_otel_scope_span) carry the operation name,
fetch direction, the OTel span handle, and optional parent references
resolved from other spans or extracted contexts. Scope contexts
(flt_otel_scope_context) hold an extracted span context obtained from
a carrier text map via the tracer. The scope data structures
(flt_otel_scope_data) aggregate growable key-value arrays for attributes
and baggage, a linked list of named events with their own attribute
arrays, and a span status code with description, representing the
telemetry collected during a single event execution.
Implemented the flt_otel_ops_check() callback that validates the parsed
OTel filter configuration after all HAProxy configuration sections have
been processed.
The check callback performs the following validations: resolves deferred
sample fetch arguments under full frontend and backend capabilities,
verifies uniqueness of filter IDs across all proxies, ensures the
instrumentation section and its configuration file are present, checks
for duplicate group and scope section names, verifies that groups are not
empty, resolves group-to-scope and instrumentation-to-group/scope
cross-references by linking placeholder entries to their definitions,
detects unused scopes, counts root spans and warns when the count differs
from one, and accumulates the required channel analyzer bits from all used
scopes into the instrumentation configuration.
The commit also added the flt_otel_counters structure to track per-event
diagnostic counters in debug builds, the FLT_OTEL_ALERT macro for
filter-scoped error messages, and the FLT_OTEL_DBG_LIST macro for
iterating and dumping named configuration lists.
Added the full configuration parser that reads the OTel filter's external
configuration file and the event model that maps filter events to HAProxy
channel analyzers.
The event model in event.h defines an X-macro table
(FLT_OTEL_EVENT_DEFINES) that maps each filter event to its HAProxy
channel analyzer bit, sample fetch direction, and event name. Events
cover stream lifecycle (start, stop, backend-set, idle-timeout), client
and server sessions, request analyzers (frontend and backend TCP and
HTTP inspection, switching rules, sticking rules, RDP cookie), response
analyzers (TCP inspection, HTTP response processing), and HTTP headers,
end, and reply callbacks. The event names are partially compatible with
the SPOE filter. The flt_otel_event_data[] table in event.c is generated
from the same X-macro and provides per-event metadata at runtime.
The parser in parser.c implements section parsers for the three OTel
configuration blocks: otel-instrumentation (tracer identity, log server,
config file path, groups, scopes, ACLs, rate-limit, options for
disabled/hard-errors/nolognorm, and debug-level), otel-group (group
identity and scope list), and otel-scope (scope identity, span definitions
with optional root/parent modifiers, attributes, events, baggages, status
codes, inject/extract context operations, finish lists, idle-timeout,
ACLs, and otel-event binding with optional if/unless ACL conditions).
Each section has a post-parse callback that validates the parsed state.
The top-level flt_otel_parse_cfg() temporarily registers these section
parsers, loads the external configuration file via parse_cfg(), and
handles deferred resolution of sample fetch arguments by saving them in
conf->smp_args for later resolution in flt_otel_check() when full frontend
and backend capabilities are available. The main flt_otel_parse() entry
point was extended to parse the filter ID and config file keywords, verify
that insecure-fork-wanted is enabled, and wire the parsed configuration
into the flt_conf structure.
The utility layer gained flt_otel_strtod() and flt_otel_strtoll() for
validated string-to-number conversion used by rate-limit and debug-level
parsing.
Added the configuration structures that model the OTel filter's
instrumentation hierarchy and the utility functions that support the
configuration parser.
The configuration is organized as a tree rooted at flt_otel_conf, which
holds the proxy reference, filter identity, and lists of groups and
scopes. Below it, flt_otel_conf_instr carries the instrumentation
settings: tracer handle, rate limiting, hard-error mode, logging state,
channel analyzers, and placeholder references to groups and scopes.
Groups (flt_otel_conf_group) aggregate scopes by name. Scopes
(flt_otel_conf_scope) bind an event to its ACL condition, span context
declarations, span definitions and a list of spans scheduled for
finishing. Spans (flt_otel_conf_span) carry attributes, events,
baggages and status entries, each represented as flt_otel_conf_sample
structures that pair a key with concatenated sample-expression arguments.
All configuration types share a common header macro (FLT_OTEL_CONF_HDR)
that embeds an identifier string, its length, a configuration line number,
and a list link. Their init and free functions are generated by the
FLT_OTEL_CONF_FUNC_INIT and FLT_OTEL_CONF_FUNC_FREE macros in
conf_funcs.h, with per-type custom initialization and cleanup bodies.
The utility layer in util.c provides argument counting and concatenation
for the configuration parser, sample data to string conversion covering
boolean, integer, IPv4, IPv6, string and HTTP method types, and debug
helpers for dumping argument arrays and linked list state.
The OpenTelemetry (OTel) filter enables distributed tracing of requests
across service boundaries, export of metrics such as request rates,
latencies and error counts, and structured logging tied to trace context,
giving operators a unified view of HAProxy traffic through any
OpenTelemetry-compatible backend.
The OTel filter is implemented using the standard HAProxy stream filter
API. Stream filters attach to proxies and intercept traffic at each stage
of processing: they receive callbacks on stream creation and destruction,
channel analyzer events, HTTP header and payload processing, and TCP data
forwarding. This allows the filter to collect telemetry data at every
stage of the request/response lifecycle without modifying the core proxy
logic.
This commit added the minimum set of files required for the filter to
compile: the addon Makefile with pkg-config-based detection of the
opentelemetry-c-wrapper library, header files with configuration
constants, utility macros and type definitions, and the source files
containing stub filter operation callbacks registered through
flt_otel_ops and the "opentelemetry" keyword parser entry point.
The filter uses the opentelemetry-c-wrapper library from HAProxy
Technologies, which provides a C interface to the OpenTelemetry C++ SDK.
This wrapper allows HAProxy, a C codebase, to leverage the full
OpenTelemetry observability pipeline without direct C++ dependencies
in the HAProxy source tree.
https://github.com/haproxytech/opentelemetry-c-wrapperhttps://github.com/open-telemetry/opentelemetry-cpp
Build options:
USE_OTEL - enable the OpenTelemetry filter
OTEL_DEBUG - compile the filter in debug mode
OTEL_INC - force the include path to the C wrapper
OTEL_LIB - force the library path to the C wrapper
OTEL_RUNPATH - add the C wrapper RUNPATH to the executable
Example build with OTel and debug enabled:
make -j8 USE_OTEL=1 OTEL_DEBUG=1 TARGET=linux-glibc
conn_recv_qstrm() may be called several times per connection if the read
data is too short and a truncated record is received.
Previously, record length was parsed every time the function is invoked.
However, this must only be performed if record length varint is
incomplete. Once read and parsed, data are removed from the buffer via
b_quic_dec_int(). Thus, next conn_recv_qstrm() run will reread an
invalid record length this time.
This patch fixes this by only parsing record length if <rxrlen> member
is null. Prior to it, parsing of QMux transport parameters would fail in
case of a first truncated read, which would prevent the connection
initialization.
No need to backport.
A QCC connection may be flagged with QC_CF_ERRL to trigger a
CONNECTION_CLOSE emission. However, for now error reporting is not
functional with QMux, as it relies on quic_conn layer access.
To prevent a crash in qcc_io_send() when using QMux, add a
conn_is_quic() check when QC_CF_ERRL is set to ensure no access will be
performed on quic_conn layer. In the future, this should be extended so
that QMux is also able to emit CONNECTION_CLOSE for connection closure.
No need to backport.
First, we must not emit any warning if splicing is not configured and the
global maxpipes value is 0. Then we must not remove GTUNE_USE_SPLICE flag
when we fail to allocate the haterm master pipe. Instead, we test it when we
negociate with the opposite side, to properly exclude the splicing if it is
not usable.
No backport needed.
This reverts commit 8056117e98.
Moving haterm init from haproxy is not the right way to fix the issue
because it should be possible to use a haterm configuration in haproxy.
So let's revert the commit above.
Add QUIC BLOCKED frames in the list of supported types in
qstrm_parse_frm(). Nothing is really implemented for them as for QUIC,
but this prevents a crash when receiving one of them via QMux.
No need to backport.
QMux draft 01 support is mostly achieved thanks to the recent
implementation of the Record layer. This patch thus updates the link in
the documentation to the validated draft version.
This patch implements emission of the new Record layer for QMux frames.
This handles mux-quic and xprt_qstrm layers as this is performed
similarly in both cases.
Currently, the simplest approach has been prefered : each frame is
encoded in its own record. This is not the most efficient in size but it
is extremely simple to implement for a first interop testing.
This patch implements the new QMux record layer parsing for xprt_qstrm.
This is mostly similar to the MUX code from the previous patch.
Along with this change, a new xprt_qstrm layer accessor exposes the
possible remaining record length after Transport parameters parsing.
This can only occur when xprt_qstrm Rx buffer is not completely emptied
due to other following frames. If stored in the same record, MUX layer
has to know the remaining record length.
Thus, xprt_qstrm_rxrlen() is now used in qmux_init() to preinitialize
<rx.rlen> QCC field.
This is the first patch of a serie which aims to support the new Record
layer defined by the draft 01 of QMux protocol.
https://www.ietf.org/archive/id/draft-ietf-quic-qmux-01.html#name-qmux-records
This patch deals with QMux reception at the MUX layer. The function
qcc_qstrm_recv() is adapted to read record headers before frame parsing.
This requires to keep the last record length read in a new QCC field
named <rx.rlen>.
Frames are only parsed once a full record is received. One of the
advantage of the record layer is that it can only contains whole frame
without truncation.
This patch implements proper connection error handling for xprt_qstrm
layer. Basically, processing is interrupted if CO_FL_ERROR is
encountered after either rcv_buf or snd_buf operations. Connectionn
error is set to the newly defined value CO_ER_QSTRM.
This commit adds buffering on transmission for xprt_qstrm layer. This is
necessary in the rare case where send syscall only emits partial data.
A new <txbuf> member is defined in xprt_qstrm context. On first send
invokation, buffer is allocated and then the QMux transport parameters
frame is encoded. Then emission is performed via snd_buf and each time
the send function is invoked.
Layer xprt_qstrm is responsible to read the initial QMux transport
parameters frame. However, it could receive more data if some other
frames follow it. This extra content can only be handled by the MUX
layer once initialized.
Theorically, it could have been implemented via MSG_PEEK. However, this
flag is currently ignored by SSL layer. Besides, it is tedious to
implement safely. A new approach has been prefered where the MUX layer
is responsible to retrieve remaining data via xprt_qstrm_rxbuf()
accessor function during its initialization.
Thus, qmux_init() now may retrieve the buffer from xprt_qstrm layer.
This is performed via b_xfer() which will result in a zero copy
transfer. If this happens, tasklet is immediately scheduled to start
demuxing.
Implement buffering for reception on xprt_qstrm layer. This is necessary
to handle reception of a truncated QMux transport parameters frame.
This is performed via a new dedicated <rxbuf> member in xprt_qstrm
context. Read is performed by reusing the buffer until a whole frame can
be read.
QUIC frame parsers functions take a <pos> pointer as input argument for
the data to be parsed. If parsing is successful, <pos> must be
incremented to point to the next data.
Increment was not performed when parsing QMux transport parameters
frame. This commit fixes this. Note that for now there is no real issue
as xprt_qstrm does not check the QMux frame length.
No need to backport.
In qcc_release(), <conn> may be NULL. Thus every access on it must be
tested.
With recent QMux introduction, a call to conn_is_quic() has been added
prior to registration of the stream rejection callback. It could lead to
NULL deref as <conn> is not tested there. Fix this by adding an extra
check on the pointer validity.
No need to backport.
hlua_applet_http_status() stored the result of luaL_optlstring()
directly in http_ctx->reason. The pointer references Lua-managed
string storage which is only guaranteed valid until the C function
returns to Lua. If the GC runs between applet:set_status(200, str)
and applet:start_response(), the pointer dangles.
hlua_applet_http_send_response() then calls ist(http_ctx->reason)
which does strlen() on freed memory, followed by memcpy into the
HTX status line. The freed-and-reallocated chunk contents are sent
verbatim to the HTTP client.
Trigger:
applet:set_status(200, table.concat({"Reason ", str:rep(50)}))
collectgarbage("collect"); collectgarbage("collect")
applet:start_response()
With heap grooming, adjacent allocation contents (session data, TLS
material from the same thread) leak into the response status line.
Anchor the Lua string in the registry keyed by the http_ctx field
address so it survives until the applet is done with it. The
registry entry is overwritten on each call (handles repeated
set_status) and naturally cleaned up when the lua_State is closed.
This patch should be backported to all stable versions.
FCGI content_length is a 16-bit field but fcgi_set_record_size()
is called with size_t/uint32_t arguments. With tune.bufsize >= 65544
(legal; cfgparse-global.c only enforces <= INT_MAX-16), a single
HTX DATA block or accumulated outbuf can exceed 65535 bytes. The
implicit conversion to uint16_t silently truncates the length field
while b_add(mbuf, outbuf.data) writes the full body.
A client posting ~99000 bytes can craft the body so that bytes
after the truncated length are parsed by PHP-FPM as fresh FCGI
records on the connection: a smuggled BEGIN_REQUEST + PARAMS with
arbitrary SCRIPT_FILENAME / PHP_VALUE bypasses all haproxy ACLs.
Fix the zero-copy path by refusing it when the block exceeds 65535
bytes (falls through to copy). Fix the copy path by capping
outbuf.size to 65535 + header so the data-fill loop naturally
stops at the FCGI maximum and emits the rest in a subsequent record.
The PARAMS path at line 2084 is similarly affected but harder to
trigger (requires combined header+param size > 65535) and is
covered by the same outbuf.size cap pattern if applied there.
This patch must be backported to all stable versions.
exp_replace() returns int and returns -1 when the back-reference
expansion overflows the output buffer (regex.c:51). output->data is
size_t, so -1 becomes SIZE_MAX. There was no error check.
The subsequent comparisons interpret SIZE_MAX as a huge length:
"output->data > b_room(trash)" tries to grow trash, then
"max > output->data" is false so max stays at trash->size, and
memcpy(trash, output->area, trash->size) copies the full chunk.
output->area is a pool_alloc()'d chunk that is NOT zeroed; the
bytes after the partial exp_replace output are stale data from a
prior pool user (request headers, response bodies from the same
worker thread).
Trigger with a backreference whose expansion exceeds bufsize:
http-request set-header X %[req.hdr(In),regsub('(.+)','\1\1')]
and a request with In: of ~9000 bytes. The X header sent to the
backend then contains ~9KB of stale heap data.
With tune.bufsize.large set, get_larger_trash_chunk() upgrades trash
and the memcpy reads up to ~50KB past the (smaller) output->area
allocation.
http_ana.c:2728 and http_act.c:551 already check exp_replace() for
-1; this call site was missed when backreferences were added.
This patch must be backported to all stable versions.
Samples of type SMP_T_METH were not properly handled in smp_dup(),
smp_is_safe() and smp_is_rw(). For "other" methods, for instance PATCH, a
fallback was performed on the SMP_T_STR type. Only the buffer considered
changed. "smp->data.u.meth.str" should be used for the SMP_T_METH samples
while smp->data.u.str should be used for SMP_T_STR samples. However, in
smp_dup(), the result was stored in wrong buffer, the string one instead of
the method one. In smp_is_safe() and smp_is_rw(), the method buffer was not
used at all.
We now take care to use the right buffer.
This patch must be backported to all stable versions.
When "Expect" header was found in request headers, "HTTP/1.1 100-continue"
was returned instead of "HTTP/1.1 100 continue". Let's fix it.
No backport needed.
http_action_set_headers_bin() decodes varint name and value lengths
from a binary sample but never validates that the decoded length
fits in the remaining sample data before constructing the ist.
If the value's varint decodes to a large number with only a few
bytes following, v.len exceeds the buffer and http_add_header()
memcpys past the sample, copying adjacent heap data into a header
sent to the backend (or client, with http-response).
The intended source for this action is the hdrs_bin sample fetch
which produces well-formed output, but nothing prevents an admin
from feeding it req.body or another untrusted source. With:
http-request set-var(txn.h) req.body
http-request add-headers-bin var(txn.h)
a POST body of [05]"X-Foo"[c8]"AB" produces v = {ptr="AB", len=200}
and 198 bytes of adjacent heap data go into X-Foo.
http_action_del_headers_bin() was fixed too.
Compare spoe_decode_buffer() which has the equivalent check.
Validate both name and value lengths against remaining data.
No backport needed.
decode_varint() has no iteration cap and accepts varints decoding to
any uint64_t value. When sz is large enough that p + sz wraps modulo
2^64, the check "p + sz > end" passes, *buf is set to the wrapped
pointer, and the caller's parsing loop continues from an arbitrary
relative offset before the demux buffer.
A malicious SPOE agent sending an AGENT_HELLO frame with a key-name
length varint of 0xfffffffffffff000 causes spop_conn_handle_hello()
to dereference memory ~64KB before the dbuf allocation, resulting in
SIGSEGV (DoS) or, if the read lands on live heap data, parser
confusion. The relative offset is fully attacker-controlled and
ASLR-independent.
Compare against the remaining length instead of computing p + sz.
Since p <= end is guaranteed after a successful decode_varint(),
end - p is non-negative.
This patch must be backport to all stable versions.
Commit c84c15d393 ("BUG/MINOR: resolvers: Apply dns-accept-family
setting on additional records") converted a switch statement to an
if/else chain but left the break; in the AAAA branch. In the new
form, break exits the surrounding for loop instead of a switch case.
For every AAAA additional record in an SRV response:
- answer_record allocated at line 1460 is never freed and never
inserted into answer_tree -> ~580 bytes leaked per response
- all subsequent additional records in the response are silently
discarded
A DNS server controlling SRV responses for haproxy service discovery
can leak memory at MB/min rates given default resolution intervals.
Also breaks IPv6 SRV target resolution outright since the AAAA record
is leaked rather than attached to its SRV entry.
Ensure that all Lua regression tests exercise the restricted library
mode by setting "tune.lua.openlibs none" in their global section.
Only txn_get_priv-thread.vtc requires "string,table"
HAProxy has always called luaL_openlibs() unconditionally, which opens
all standard Lua libraries including io, os, package and debug. This
makes it impossible to prevent Lua scripts from executing binaries
(os.execute, io.popen), loading native C modules (package/require), or
bypassing any Lua-level sandbox via the debug library.
Add a new global directive tune.lua.openlibs that accepts a comma-separated
list of library names to load:
tune.lua.openlibs none # only base + coroutine
tune.lua.openlibs string,math,table,utf8 # safe libs only
tune.lua.openlibs all # default, same as before
The base and coroutine libraries are always loaded regardless: base provides
core Lua functions that HAProxy relies on, and coroutine is required because
HAProxy overrides coroutine.create() with its own safe implementation.
When all libraries are enabled (the default), the fast path still calls
luaL_openlibs() directly with no overhead. A parse error is returned if
the directive appears after lua-load or lua-load-per-thread (the Lua state
is already initialised at that point), or if 'none' is combined with other
library names. Note that fork() and new thread creation are already blocked
by default regardless of this setting (see "insecure-fork-wanted").
Literals are sent in two ways:
- in EOB state, unencoded and prefixed with their length
- in FIXED state, huffman-encoded
And references are only sent in FIXED state.
The API promises that the amount of data will not grow by more than
5 bytes every 65535 input bytes (the comment was adjusted to remind
this last point). This is guaranteed by the literal encoding in EOB
state (BT, LEN, NLEN + bytes), which is supposed to be the worst
case by design.
However, as reported by Greg KH, this is currently not true: the test
that decides whether or not to switch to FIXED state to send references
doesn't properly account for the number of bytes needed to roll back
to the *exact* same state in EOB, which means sending EOB, BT,
alignment, LEN and NLEN in addition to the referenced bytes, versus
sending the encoding for the reference. By not taking into account the
cost of returning to the initial state (BT+LEN+NLEN), it was possible
to stay too long in the FIXED state and to consume the extra bytes that
are needed to return to the EOB state, resulting in producing much more
data in case of multiple switchovers (up to 6.25% increase was measured
in tests, or 1/16, which matches worst case estimates based on the code).
And this check is only valid when starting from EOB (in order to restore
the same state that offers this guarantee). When already in FIXED state,
the encoded reference is always smaller than or same size as the data.
The smallest match length we support is 4 bytes, and when encoded this
is no more than 28 bits, so it is safe to stay in FIXED state as long
as needed while checking the possibility of switching back to EOB.
This very slightly reduces the compression ratio (-0.17% on a linux
kernel source) but makes sure we respect the API promise of no more
than 5 extra bytes per 65535 of input. A side effect of the slightly
simpler check is an ~7.5% performance increase in compression speed.
Many thanks to Greg for the detailed report allowing to reproduce
the issue.
This is libslz upstream commit 002e838935bf298d967f670036efa95822b6c84e.
Note: in haproxy's default configuration (tune.bufsize 16384,
tune.maxrewrite 1024), this problem cannot be triggered, because the
reserve limits input to 15360 bytes, and the overflow is maximum
960 bytes resulting in 16320 bytes total, which still fits into the
buffer. However, reducing tune.maxrewrite below 964, or tune.bufsize
above 17408 can result in overflows for specially crafted patterns.
A workaround for larger buffers consists in always setting tune.bufsize
to at least 1/16 of tune.bufsize.
Reported-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Link: https://www.mail-archive.com/haproxy@formilux.org/msg46837.html
Storing the protocol directly into the check was not a good idea,
because the protocol may not be determined until after a DNS resolution
on the server, and may even change at runtime, if the DNS changes.
What we can, however, figure out at start up, is the net_addr_type,
which will contain all that we need to find out which protocol to use
later.
Also revert the changes made by commit 07edaed191
that would not reuse the server xprt if a different alpn is set for
checks. The alpn is just a string, and should not influence the choice
of the xprt.
We'll now make sure to use the server xprt, unless an address is
provided, in which case we'll use whatever xprt matches that address, or
a port, in which case we'll assume we want TCP, and use check_ssl to
know whetver we want the SSL xprt or not.
Now that the check contains all that is needed to know which protocol to
look up, always just use that when creating a new check connection if it
is the default check connection, and for now, always use TCP when a
tcp-check or http-check connect rule is used (which means those can't be
used for QUIC so far).
This should hopefully fix github issue #3324.
Commit 1b0dfff552 attempted to make it so
the mux would expect a QUIC-like protocol or not, however it only made
that we would not instantiate a non-QUIC mux on a QUIC protocol, but not
that we tried to instance a QUIC mux on a non-QUIC protocol, so fix
that.
The vtest binary does not seem to be cached correctly by actions/cache,
the cause of the problem seems to be the binary is installed outside the
github workspace. This patch installs the binary in ~/vtest/ to fix the
issue.
This reverts commit a03120e228.
A WIP version of the patch was applied before the actual patch by
accident. The correct patch is 2db801c ("BUG/MINOR: hlua: fix stack
overflow in httpclient headers conversion")
github complains about cache@v4:
Node.js 20 actions are deprecated. The following actions are running on
Node.js 20 and may not work as expected: actions/cache@v4. Actions will
be forced to run with Node.js 24 by default starting June 2nd, 2026.
Node.js 20 will be removed from the runner on September 16th, 2026.
Please check if updated versions of these actions are available that
support Node.js 24. To opt into Node.js 24 now, set the
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the
runner or in your workflow file. Once Node.js 24 becomes the default,
you can temporarily opt out by setting
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see:
https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
When the app_ops were removed, direct calls to the SC wake callback function
were replaced by tasklet wakeups. However, in conn_create_mux(), it was
replaced by a direct call to sc_conn_process(). However, sc_conn_process()
is only usable when the SC is attach to a stream. A backend mux can be
created for a healcheck. In this context, sc_conn_process() cannot be
called.
Because of this bug, crashes can be experienced when an error is triggered
during a SSL connection attempt from a healthcheck.
To fix the issue, the call to sc_conn_process() was replaced by a tasklet
wakeup.
This patch should fix the issue #3326. No backport needed.
The VTest2 tarball URL at code.vinyl-cache.org/vtest/VTest2/archive/main.tar.gz
no longer works. Switch scripts/build-vtest.sh to use a git clone of the
repository instead.
Add a cache step in the setup-vtest CI action so VTest is only rebuilt
when its HEAD commit changes, keyed on the runner OS and the VTest2 HEAD
SHA.
When a peer sends a dictionary entry update with a value (the else
branch at line 2109), the entry id decoded from the wire was never
validated against dc->max_entries before being used as an array index
into dc->rx[].
A malicious peer can send id=N where N > 128 (PEER_STKT_CACHE_MAX_ENTRIES)
to:
- dc->rx[id-1].de at line 2123: OOB read followed by atomic decrement
and potential free of an attacker-controlled pointer via
dict_entry_unref()
- dc->rx[id-1].de = de at line 2124: OOB write of a heap pointer at
an attacker-controlled offset (16-byte stride, ~64 GiB range)
The bounds check was added to the key-only branch in commit f9e51beec
("BUG/MINOR: peers: Do not ignore a protocol error for dictionary
entries.") but was never added to the with-value branch. The bug has
been present since dictionary support was introduced in commit
8d78fa7def ("MINOR: peers: Make peers protocol support new
"server_name" data type.").
Reachable from any TCP client that knows the configured peer name
(no cryptographic authentication on the peers protocol). Requires a
stick-table with "store server_key" in the configuration.
Fix by hoisting the bounds check above the branch so it covers both
paths.
Must be backported as far as 2.6.
When the input chunk is already the large buffer (chk->size ==
large_trash_size), the <= comparison still matched and returned
another large buffer of the same size. Callers that retry on a
non-NULL return value (sample.c:4567 in json_query) loop forever.
The json_query infinite loop is trivially triggered: mjson_unescape()
returns -1 not only when the output buffer is too small but also for
any \uXXYY escape where XX != "00" (mjson.c:305) and for invalid
escapes like \q. The retry loop assumes -1 always means "grow the
buffer", so a 14-byte JSON body of {"k":"\u0100"} hangs the worker
thread permanently. Send N such requests to exhaust all worker
threads.
Use < instead of <= so a chunk that is already large yields NULL.
This also fixes the json converter overflow at sample.c:2869 where
no recheck happens after the "growth" returned a same-size buffer.
Introduced in commit ce912271db ("MEDIUM: chunk: Add support for
large chunks"). No backport needed.
A copy-paste error in alloc_trash_buffers_per_thread() passes
global.tune.bufsize_large to alloc_small_trash_buffers() instead of
global.tune.bufsize_small. This sets small_trash_size = bufsize_large.
When tune.bufsize.large is configured, get_larger_trash_chunk() then
incorrectly matches a large buffer against small_trash_size at line
169 and "grows" it to a regular (smaller) buffer. b_xfer() at line
179 attempts to copy the large buffer's contents into the smaller one:
- Default builds (DEBUG_STRICT=1): BUG_ON in __b_putblk() aborts
the process -> remote DoS
- DEBUG_STRICT=0 builds: BUG_ON becomes ASSUME() and the compiler
elides the check -> heap overflow with attacker-controlled bytes
Reachable via the json converter (sample.c:2862) when escaping
~bufsize_large/6 control characters in attacker-supplied data such
as a request header or body.
Introduced in commit 92a24a4e87 ("MEDIUM: chunk: Add support for
small chunks"). No backport needed.
hlua_error() is a printf-family function (calls vsnprintf), but
hlua_patref_set, hlua_patref_add, and _hlua_patref_add_bulk pass
errmsg directly as the format string. errmsg is built by pattern.c
helpers that embed the user-supplied key or value verbatim, e.g.
pat_ref_set_elt() generates "unable to parse '<value>'".
A Lua script calling:
ref:set("key", "%p.%p.%p.%p.%p.%p.%p.%p")
against a map with an integer output type (where the parse fails)
gets stack/register contents formatted into the (nil, err) return
value -> ASLR/canary leak. With %n and no _FORTIFY_SOURCE this
becomes an arbitrary write primitive.
This must be backported as far as the Patref Lua API exists.
hlua_httpclient_table_to_hdrs() declares a VLA of size
global.tune.max_http_hdr (default 101) on the stack but never checks
hdr_num against that bound. A Lua script that supplies a header table
with more than 101 values writes struct http_hdr entries (two ist =
two heap pointers + two lengths) past the end of the VLA, smashing
the stack frame.
Trigger from any Lua action/task/service:
local hc = core.httpclient()
local v = {}
for i = 1, 300 do v[i] = "x" end
hc:get{ url = "http://127.0.0.1/", headers = { ["X"] = v } }
Each out-of-bounds entry writes a heap pointer (controllable
allocation contents via istdup) plus an attacker-chosen length onto
the stack, overwriting the saved return address.
[wla: this is only reachable if the Lua script passes more than
max_http_hdr header values, which requires access to the script itself]
This must be backported as far as the httpclient Lua API exists.
Signed-off-by: William Lallemand <wlallemand@haproxy.com>
hlua_httpclient_table_to_hdrs() declares a VLA of size
global.tune.max_http_hdr (default 101) on the stack but never checks
hdr_num against that bound. A Lua script that supplies a header table
with more than 101 values writes struct http_hdr entries (two ist =
two heap pointers + two lengths) past the end of the VLA, smashing
the stack frame.
Trigger from any Lua action/task/service:
local hc = core.httpclient()
local v = {}
for i = 1, 300 do v[i] = "x" end
hc:get{ url = "http://127.0.0.1/", headers = { ["X"] = v } }
Each out-of-bounds entry writes a heap pointer (controllable
allocation contents via istdup) plus an attacker-chosen length onto
the stack, overwriting the saved return address. With no stack
canary, this is direct RCE; with a canary, it requires a leak first.
Reachable from any deployment that loads Lua scripts. While Lua
scripts are nominally trusted, this turns "can edit Lua" into "can
execute arbitrary native code", which is a meaningful boundary in
many setups (Lua sandbox escape).
This must be backported as far as the httpclient Lua API exists.
