This reduces the total code size by 6-10% and speeds up the build a
bit. It can be further reduced by disabling the trace decoding code
inside certain subsystems like muxes. But at least like this it will
help users on small systems to reduce the footprint when not needed
by explicitly passing USE_TRACE=0 (they remain enabled by default).
When we store the negociated server parameters, such as the ALPN, also
store the calculated hash with the connection. If it is different, as
can happen because the IP address is different because set-dst was used,
we certainly do not want to reuse the information in the cache,
otherwise we could end up using the wrong ALPN and mux.
That means we already have to calculate the hash in connect_server()
now, while before we would not do it for Websockets, if we could not do
connection reuse, as that's all the hash was used for.
This should fix Github issue #3386
This should be backported as far as 3.2.
Add a new QCC member <ms_bidi_rel>. This represents the number of
concurrent streams advertised similarly to ms_bidi, but as a relative
value.
This patch does not introduce any functional change. For now,
<ms_bidi_rel> will be equal to <ms_bidi_init>. However, with the
implementation of stream elasticity and dynamic adjustment for
concurrent max-streams-bidi, the former will be required to keep the
last advertised value.
When server fleets are constantly updated, using a stable distribution
across a bunch of load balancers can be convenient. The addr and port
already provide a bit of this but for situations were addresses might
differ between sites or change dynamically this does not work. The guid
is perfect for this because by definition it's supposed to designate a
single server and be unique. So when two servers anywhere have the same,
the tool that provisionned them promises that they are the same server.
So here we introduce "hash-key guid" which performs a 32-bit hash on
the GUID value. When no guid is provided, a fallback is performed on
ID, as is done for other keys.
The "id" hash-key scales the ID by a factor of 16 that tries to leave
room between the nodes on the 32-bit space to permit smooth weight
variations (e.g. during slowstart). However this does not deal well
with overlaps between server IDs. For example, assigning IDs that are
only multiples of 256 million to 16 servers yields traffic only on
one since in practice they all have the same 28 lower bits.
The new "id32" hash key bridges this gap by using the full 32-bit ID
of the server as the key. On the other hand, the user must be careful
not to switch the hash function to "none" when using incremental IDs
because in this case they might be very poorly distributed. But this
can be convenient for automated provisionning systems which assign
IDs themselves, as the full 32 bits are used now.
The first part of this patch defines a new mux_proto_list field named
<xprt_init>. This allows to define an extra XPRT layer which should be
activated first prior to the MUX creation both on frontend and backend
sides.
This is immediately used for QMux mux_proto_list to require XPRT_QMUX
handshake. With this change, activation of QMux connection flags in
session_accept_fd() and connect_server() are adjusted to take into
account <init_xprt> field. This approach is much more evolutive than
relying on the previous MUX name.
Change in connect_server() will also be necessary to support QMux
activation on a TCP server with h3 ALPN without explicit "proto qmux".
This guarantees that MUX initialization is delayed after QMux handshake.
This patch is similar to the previous one but this time for backend
connections. The MUX selection code is directly extracted from
conn_install_mux_chk() and conn_install_mux_be().
Define a new function conn_select_mux_fe().
The objective is to have a preliminary function to determine the MUX
which will be used without initializing it. This will be useful for MUX
which relies on a specific XPRT handshake prior to its startup, which is
the case for QMux protocol.
The code of conn_select_mux_fe() is identical to the beginning of
conn_install_mux_fe() with a similar MUX selection logic. However,
connection MUX initialization is not performed in this case. In a future
patch, both functions should be merged together to reduce code
duplication.
In conn_get_best_mux() and conn_get_best_mux_entry(), the mux name was
provided sometimes based on the "proto" directive, sometimes based on
the ALPN, but in any case, it was compared again the mux_proto_list
mux_proto field. This is not correct, as ALPN can be different from the
internal mux_proto. So enhance those functions so that they wll accept
an ALPN as well. If a mux_proto is provided, that will be used, if not,
and if an ALPN is provided, then that will be used, and compared against
the ALPN provided by the mux, if any.
In struct mux_proto_list, rename the "token" field to "mux_proto". That
field should only be used to match the name provided in the "proto"
directive, and it will be soon.
This should be a no-op.
During the H1 message parsing, the Upgrade header values are checked to
detect "h2c" and "h2" tokens and skip them. To do so, we rely on
H1_MF_UPG_H2C flag, set during the parsing. And during the request
post-parsing, if this flag was set, all Upgrade headers are removed.
This was fixed by the commit 7b89aa5b1 ("BUG/MINOR: h1: do not forward h2c
upgrade header token").
However, there are two issues here and the commit above must be refined.
First, the flag is reset for each new Upgrade header. So "h2c" or "h2"
tokens will be properly detected if all tokens are set on the same Upgrade
header. But if splitted on several headers, previously detected tokens will
be hidden by a next ones.
Concretly, the following will be properly caught
Connection: upgrade
Upgrade: foo, h2c, bar
But then following not:
Connection: upgrade:
Upgrade: foo, h2c
Upgrade: bar
Then, when a "h2c" or "h2" token is finally reported, all Upgrade headers
are removed, regardless other tokens.
So, to fix the both issues, everything is now handled during the message
parsing by skipping "h2c" and "h2" tokens, rebuilding the Upgrade header
value without then offending tokens. The same was already performed for the
Connection header, to skip "keep-alive" and "close" value. So it is not a so
fancy change.
Thanks to this change, it is no longer necessary to handle H1_MF_UPG_H2C
during the request post-parsing. And in fact, this flag is no longer
necessary. So let's remove it too.
Thanks to Vincent55 for finding and reporting this.
This patch must be backported as far as 2.4.
Unlike stated in the configuration manual, the server 'init-state' parameter
was not evaluated during haproxy startup/reload. After a review, it appeared
there were also issues if combined with the 'track' parameter. In addtition,
this parameter was only evaluated when health-checks were enabled for the
server, leading to unexpected behavior if the serve settings are dynamically
changed via the CLI.
To fix those issues, behavior of the 'init-state' parameter was slightly
adapted. It is always evaluated, even when there is no running health-checks
for the server. An error is reported if the 'track' parameter is also
defined. Both cannot work together.
In addition, the "none" state was introduced to be able to restore the
default behavior. It will be especially useful when the parameter is
inherited from a 'default-server' directive.
This patch should fix the issue #3298. It must be backported as far as 3.2.
The function cli_find_frontend was declared twice identically at lines 98-99
of include/haproxy/proxy.h. The second declaration should have been for
cli_find_backend, which is defined in src/proxy.c and used in several places
but was missing from the header's exported symbols.
This is a simple copy-paste mistake where line 99 duplicated line 98 verbatim
instead of declaring cli_find_backend.
Support of pseudo-headers was removed as unused, but mention of it
in the description remains and disrupt the numbering in comment, which
can be confusing.
Add a new "alias" member in trace_source structure. Its purpose is to be
an alternative to the member "name". This will be used in the next patch
to allow renaming of QUIC mux traces while preserving compatibility.
This new member is only used in trace_find_source() which is the helper
used to retrieve a trace source from its name.
This is a follow-up on the QUIC MUX renaming process.
The current patch performs renaming in xprt_qmux layer. Older "qstrm"
identifier is replaced by the new name "qmux". Every remaining functions
and structures in xprt_qmux are changed. Outside effects are only
present in QUIC MUX which directly uses some of these functions.
This is a follow-up on the QUIC MUX renaming process.
The current patch performs renaming of "qstrm" to "qmux" in connection
flags. These flags are only used in linked with the xprt_qmux layer.
This has an impact on every files which manipulates these flags, namely
backend, session and ssl_sock sources.
Also, internal xprt identifier is renamed from XPRT_QSTRM to XPRT_QMUX,
This is a follow-up on the QUIC MUX renaming process.
The current patch replaces "qstrm" naming with "qmux" in QUIC MUX source
file. Some members are impacted in qcc and qcs structures, as well as
some internal functions used for QMux receive/send. Internal mux_ops is
also rename to qmux_ops. This is not a breaking change as its externally
visible name was already set to "qmux" originally.
This is a follow-up on the QUIC MUX renaming process. Now most of "qmux"
generic usages as been replaced in favor of "qcm" naming. The next part
of the renaming is to replace "qstrm" naming with "qmux" for stuffs
related to the new QMux protocol specifically.
This is first applied on filenames. As with the previous renaming,
Makefile and include statements are updated as well to prevent
compilation issues.
This is a follow-up on the QUIC MUX renaming process.
This patch renames several definitions in QUIC MUX traces source code.
This is only an internal change. Trace source name is kept to "qmux" for
now, but will be changed in a dedicated patch.
This is a follow-up on the QUIC MUX renaming process.
A previous patch already renames MUX ops callbacks. The current patch
proceed to a similar operation for the rest of the MUX functions.
This patch is the first one of the renaming serie, affecting the QUIC
MUX module. The objective is to remove older "qmux" naming which was
used as a generic identifier. Now it should be restricted to the QMux
experimental protocol. A new "qcm" naming will replace the generic
usage.
The current patch renames the files themselves. Token "qmux" is replaced
by the new "qcm" identifier. Makefile and include statements are
adjusted as required.
This patch adds the missing include of xprt_qstrm header into its
companion source file. This helped to detect an incoherence in the
xprt_qstrm_xfer_rxbuf() prototype which is now fixed.
Header files is also updated with mandatory include statements and
forward declaration.
No backport needed.
Some outdated comments, as well as typos were fixed in the following files:
dgram.h protocol.h queue-t.h cpu_topo.c debug.c dict.c
protocol.c queue.c raw_sock.c trace.c wdt.c
Dmitry Sivachenko reported a build warning on FreeBSD -dev, where
__nonstring is apparently already defined. Let's guard our own
definition to avoid such issues. It could make sense to backport
this to recent stable versions which may soon be exposed to modern
compilers.
In task_schedule(), it is not enough to get the TASK_RUNNING flag before
setting the expire field, we also have to keep it while queueing the
taks, otherwise the task may run in the meanwhile and set expire to 0,
triggering the BUG_ON() in __task_queue() again. So now, only drop the
running flag once it's done.
This should be backported up to 2.8.
In comp_prepare_compress_request(), the compression rate limit and CPU
usage checks were duplicated. The first set runs before selecting the
algorithm, and the second set runs after. That's definitely a copy-paste
issue or a patch being applied twice. Let's just drop one.
This fixes typos and spelling mistakes in the following files:
channel-t.h channel.h filters-t.h http_htx.h htx-t.h tools.h
cfgcond.c channel.c flt_http_comp.c http_ana.c htx.c mqtt.c
mux_h1.c regex.c stats-proxy.c
This lets lb_ops specify the conditions necessary to bind to this set of
ops. The condition is expressed as a list of mask and match fields on
the algorithm flags. This is then used in proxy_finalize() to locate the
lb_ops corresponding to the current configuration, by iterating over
the list of lb_ops structures. This list is implemented using the same
mechanisms used for configuration keywords: an INITCALL1 macro to a
registration function.
This also moves the lookup and property flags into the lb_ops structure
that were previously applied manually on a case by case basis.
This adds a new tune.streams-elasticity parameter. This parameter
indicates, as a percentage, the average number of streams per connection
at full load. It is used to calculate limits of the number of streams to
advertise on new connections. 0 means that no such limit is set.
When a limit is set, the new function conn_calc_max_streams() determines
the optimal number of streams to allow on a connection. It will assign at
least the ratio of streams left to connections left, and at least a fair
share of what's left times the number of desired streams. It will always
ensure that each connection gets at least 1 stream, and everything beyond
this will be evenly distributed. For now the function is not used.
In order to be able to enforce global streams limitations, we'll first
have to be able to account how many streams we promised to serve via
frontend muxes. We'll always need to support at least one stream, which
is why here we're only counting extra streams beyond the first one. It
also has the benefit of leaving H1 out of this, and save it from updating
a variable. Also in order to avoid an important update cost, we're storing
this value per thread group. For now only H2 is implemented, but QUIC
should follow shortly and should only count bidirectional streams.
This implements configuration for custom mac alg in EAB.
I don't think there are any reasons to allow that TBH,
but it is something that exists in the spec.
Depends on the EAB impl.
No backport needed
Patch introduces ACME EAB support.
Configuring EAB requires two parts: Key ID and MAC Key.
Key ID is an ASCII string that specifies the name of the record CA
should look up. MAC Key is a base64url encoded key that is used
for the sake of JWS signing, using HS256 or other algorithms.
They are the credentials so must be stored securely.
A thing about EAB is that it is required only during account creation
so it is unexpectedly complex to think about.
Some CAs provide EAB credential pair that is reused between
multiple account order requests, for example ZeroSSL, but others like
Google Trusted Services require an unique EAB credential for each new
account creation request.
There are a lot of ways config could be implemented, I decided to make
so that Key ID and MAC Key are stored in separate files on disk,
that decision was made because of the security concerns.
File based approach in particular works well with systemd credentials,
works well with systems that have config world readable, or immutable,
and is compatible with existing setups that specify credentials in a
file.
EAB is configured through options like this in an acme section:
eab-mac-alg HS512
eab-mac-key pebble.eab.mac-key
eab-key-id pebble.eab.key-id
I decided to not error out on empty files, but issue a log msg instead,
so that credentials can be removed without changing the haproxy config.
Used read_line_to_trash function from tools.c for reading files,
that is something that could be replaced by a dedicated function too.
No backport needed
New jws_b64_hmac_signature() duplicates the same functionality as
jws_b64_signature(), but for the use case of HMAC signing.
Intended to be used for ACME EAB.
OpenSSL allows to use EVP_PKEY for HMAC functionality, so
jws_b64_signature() could be reused, but the problem is that although
isn't deprecated it was removed in BoringSSL, and was removed
(due to BoringSSL roots) but then readded back in AWS-LC, because of
"legacy clients" (citing them), for that reason alone I say that having
a dedicated function for hmac is better, HMAC() macro seems to be widely
supported unlike other ways of doing same thing. Another alternative
would be to use EVP_MD API, but it was introduced in OpenSSL 3.0,
so not as widely supported.
In order for the code behind the "commit ssl cert" logic to be usable
outside of the CLI context, some new "ckch_store_update_" functions are
created. They allow to perform all the operations on ckch_stores to be
performed without needing an appctx.
The first function being called is ckch_store_update_init which mainly
takes the ckch_store lock and checks that there is an ongoing
transaction with the proper path (which was already done in
cli_parse_commit_cert).
The main one is ckch_store_update_process which replicates the logic
that could be found in the cli_io_handler_commit_cert function. We
iterate over the ckch instances of an existing ckch store and duplicate
them in the new ckch store which is still detached from the tree, before
replacing the old store with the new one. This whole operation could
take some time so we were yielding every 10 instances or when
applet_putstr calls would fail. The actual ckch_store operations and the
applet related calls are now decorrelated in order to stop having to
have an appctx during the ckch store/instances processing.
The ckch_store_update_process will now update a "msg" buffer and a
"state" that allow to send processing messages to the caller as well as
keep the state of the processing "state machine".
When the ckch_store_update_process loop is over,
ckch_store_update_cleanup can be called to release the lock and free
some now useless structures.
The ckch instances for a given ckch_store have to be rebuilt when a
certificate is updated during runtime (via cli or lua). The code was
duplicated in lua so factorizing the actual loop avoids future errors
if the code changes. The new 'ckch_store_rebuild_instances' will have a
dedicated 0 return code if it needs to be called again (because of the
yielding logic since ckch instance rebuild might take some time).
Muxes are responsible to release connections once they are inactive and
won't be reusable. In QUIC mux, such connections are detected via
qcc_is_dead(). The first precondition is that there is no more upper
streams attached. This was accounted via QCC <nb_sc> counter.
A special characteristic of QCS instances is that they can be in
detached state : upper stream has been removed but there is still data
to emit. Such QCS were not taken into account in qcc_is_dead(), so a
connection could be freed with some remaining data not yet emitted.
It is also not possible for QUIC MUX to simply look at the QCS tree to
determine if the connection is inactive. Indeed, some streams are opened
for protocol internal usage. This is the case for example with HTTP/3
unidirectional control stream or QPACK encoder/decoder streams. These
streams are never closed. In the end, only requests streams should be
taken into account for the connection activity.
This patch improves the situation by reworking <nb_hreq> QCC counter.
Previously, it served for http-request timeout implementation. However,
this timeout only relies on <opening_list> now. Thus, <nb_hreq> scope is
changed : it is now incremented via qcs_wait_http_req(), used by app
protocol layer once a request stream is identified. Decrement is
performed on qcs_free(), so this guarantees that a connection cannot be
freed anymore if request streams still exists, unless if inactivity
timeout fires. As such, <nb_hreq> now supersedes <nb_sc> entirely, so
the qcc_is_dead() can now relies on the former.
Along with this change, qcc_timeout_task() must be updated. Call to
qcc_is_dead() was unnecessary prior to this patch as timeout handling
was only active when no upper streams were attached. When tested, both
<nb_sc> and QCC <task> were already null, so a connection was always
released on timeout, as expected. With qcc_is_dead() now checking
<nb_hreq> instead, this is not always the case anymore. In fact, this
check is unnecessary as inactivity timeout serves precisely to free a
stucked connection with remaining data to emit.
This patch also has some impact on http-keep-alive timeout. Previously,
this timeout could be armed if only detached streams remained. Now, it
is only applicable if all QCS request instances are closed and freed.
Thus, qcc_reset_idle_start() is now closed directly on qcs_free().
Ideally this should be backported up to 2.6, or at least 2.8 as QUIC
experimental status was removed there.
MUX implements a timeout for HTTP keep-alive which monitors the delay
between two HTTP requests. This is only applicable for frontend
connections, as on the backend side idle connections can be kept in the
server pool. In QUIC mux, this timeout relies on QCC <idle_start> which
is refresh when the last request is terminated.
This patch modifies the refresh operation so that it is only performed
for frontend connections. This is not strictly necessary but the timeout
timeout management is now clearer and it eliminates an unnecessary
operation for backend connections.
Similarly, http-request timeout is also only applicable for frontend
connections. This relies on qcs_wait_http_req() function. A request QCS
is inserted in <opening_list> until the headers are received. This is
unnecessary on the backend side so this is excluded as well.
req_body field is no longer used, except in trace messages. And in fact, it
is not necessarily true if some data are received with the request headers.
So no reason to still use it.
Function h2_phdr_to_list() was missing the decoding for the :protocol
header and would emit :UNKNOWN in this case. It's only used in traces
so it's not important. The fix can be backported in all versions.
