Commit graph

6054 commits

Author SHA1 Message Date
Frédéric Lécaille
b44cbc68a6 MINOR: quic: Do not retransmit frames from coalesced packets
Add QUIC_FL_TX_PACKET_COALESCED flag to mark a TX packet as coalesced with others
to build a datagram.
Ensure we do not directly retransmit frames from such coalesced packets. They must
be retransmitted from their packet number spaces to avoid duplications.
2022-04-28 16:22:40 +02:00
Frédéric Lécaille
b917191817 MINOR: quic: Prepare quic_frame struct duplication
We want to track the frames which have been duplicated during retransmissions so
that to avoid uselessly retransmitting frames which would already have been
acknowledged. ->origin new member is there to store the frame from which a copy
was done, ->reflist is a list to store the frames which are copies.
Also ensure all the frames are zeroed and that their ->reflist list member is
initialized.
Add QUIC_FL_TX_FRAME_ACKED flag definition to mark a TX frame as acknowledged.
2022-04-28 16:22:40 +02:00
Amaury Denoyelle
47447af1ef MINOR: mux-quic: add a app-layer context in qcs
Define 2 new callback for qcc_app_ops : attach and detach. They are
called when a qcs instance is respectively allocated and freed. If
implemented, they can allocate a custom context stored in the new
abstract field ctx of qcs.

For now, h3 and hq-interop does not use these new callbacks. They will
be soon implemented by the h3 layer to allocate a context used for
stateful demuxing.

This change is required to support the demuxing of H3 frames bigger than
a buffer.
2022-04-28 15:44:19 +02:00
Amaury Denoyelle
3df8ca0a4d MINOR: mux-quic: partially copy Rx frame if almost full buf
Improve the reception for STREAM frames. In qcc_recv(), if the frame is
bigger than the remaining space in rx buffer, do not reject it wholly.
Instead, copy as much data as possible. The rest of the data is
buffered.

This is necessary to handle H3 frames bigger than a buffer. The H3 code
does not demux until the frame is complete or the buffer is full.
Without this, the transfer on payload larger than the Rx buffer can
rapidly freeze.
2022-04-28 15:42:21 +02:00
Amaury Denoyelle
44d0912f7b MINOR: mux-quic: count local flow-control stream limit on reception
Add new qcs fields to count the sum of bytes received for each stream.
This is necessary to enforce flow-control for reception on the peer.

For the moment, the implementation is partial. No MAX_STREAM_DATA or
FLOW_CONTROL_ERROR are emitted. BUG_ON statements are here as a
remainder.

This means that for the moment we do not support POST payloads greater
that the initial max-stream-data announced (256k currently).

At least, we now ensure that we never buffer a frame which overflows the
flow-control limit : this ensures that the memory consumption per stream
should stay under control.
2022-04-28 14:56:14 +02:00
Amaury Denoyelle
408d226aa1 MINOR: mux-quic: remove unused bogus qcc_get_stream()
qcc_get_stream() was used when qcs and qc_stream_desc shared the same
node-tree. This is not the case anymore since
  e4301da5ed
  MINOR: quic-stream: use distinct tree nodes for quic stream and qcs

Now this function is broken as the qcc tree only contains qcs.
Thankfully it is unused so it can be removed without impact.
2022-04-28 14:47:53 +02:00
Willy Tarreau
226866e1bb CLEANUP: deinit: release the config postparsers
These ones were not released either, it just requires to export the list
("postparsers") and it makes valgrind happy.
2022-04-27 18:07:24 +02:00
Thomas Prckl
10243938db MINOR: ssl: add a new global option "tune.ssl.hard-maxrecord"
Low footprint client machines may not have enough memory to download a
complete 16KB TLS record at once. With the new option the maximum
record size can be defined on the server side.

Note: Before limiting the the record size on the server side, a client should
consider using the TLS Maximum Fragment Length Negotiation Extension defined
in RFC6066.

This patch fixes GitHub issue #1679.
2022-04-27 16:53:43 +02:00
Remi Tricot-Le Breton
4d7fdc65d4 MINOR: connection: Add way to disable active connection closing during soft-stop
If the "close-spread-time" option is set to "infinite", active
connection closing during a soft-stop can be disabled. The 'connection:
close' header or the GOAWAY frame will not be added anymore to the
server's response and active connections will only be closed once the
clients disconnect. Idle connections will not be closed all at once when
the soft-stop starts anymore, and each idle connection will follow its
own timeout based on the multiple timeouts set in the configuration (as
is the case during regular execution).

This feature request was described in GitHub issue #1614.
This patch should be backported to 2.5. It depends on 'MEDIUM: global:
Add a "close-spread-time" option to spread soft-stop on time window'.
2022-04-26 19:56:47 +02:00
Willy Tarreau
65d9f83794 BUILD: compiler: properly distinguish weak and global symbols
While weak symbols were finally fixed with commit fb1b6f5bc ("BUILD:
compiler: use a more portable set of asm(".weak") statements"), it
was an error to think that initcall symbols were also weak. They must
not be and they're only global. The reason is that any externally
linked code loaded as a .so would drop its weak symbols when being
loaded, hence its initcalls that may contain various function
registration calls.

The ambiguity came from the fact that we initially reused the initcall's
HA_GLOBL macro for OSX then generalized it, then turned it to a choice
between .globl and .weak based on the OS, while in fact we needed a
macro to define weak symbols.

Let's rename the macro to HA_WEAK() to make it clear it's only for weak
symbols, and redefine HA_GLOBL() that initcall needs.

This will need to be backported wherever the commit above is backported
(at least 2.5 for now).
2022-04-26 19:49:33 +02:00
Willy Tarreau
a80e4a3546 MINOR: fd: add functions to set O_NONBLOCK and FD_CLOEXEC
Instead of seeing each location manipulate the fcntl() themselves and
often forget to check previous flags, let's centralize the functions to
do this. It also allows to drop fcntl.h from most call places and will
ease the adoption of different OS-specific mechanisms if needed. Note
that the fd_set_nonblock() function purposely doesn't check the previous
flags as it's meant to be used on new FDs only.
2022-04-26 10:59:48 +02:00
Willy Tarreau
197715ae21 CLEANUP: compression: move the default setting of maxzlibmem to defaults
__comp_fetch_init() only presets the maxzlibmem, and only when both
USE_ZLIB and DEFAULT_MAXZLIBMEM are set. The intent is to preset a
default value to protect the system against excessive memory usage
when no setting is set by the user.

Nowadays the entry in the global struct is always there so there's no
point anymore in passing via a constructor to possibly set this value.
Let's go the cleaner way by always presetting DEFAULT_MAXZLIBMEM to 0
in defaults.h unless these conditions are met, and always assigning it
instead of pre-setting the entry to zero. This is more straightforward
and removes some ifdefs and the last constructor. In addition, now the
setting has a chance of being found.
2022-04-25 19:42:43 +02:00
Willy Tarreau
2df1fbf816 MINOR: init: add global setting "fd-hard-limit" to bound system limits
On some systems, the hard limit for ulimit -n may be huge, in the order
of 1 billion, and using this to automatically compute maxconn doesn't
work as it requires way too much memory. Users tend to hard-code maxconn
but that's not convenient to manage deployments on heterogenous systems,
nor when porting configs to developers' machines. The ulimit-n parameter
doesn't work either because it forces the limit. What most users seem to
want (and it makes sense) is to respect the system imposed limits up to
a certain value and cap this value. This is exactly what fd-hard-limit
does.

This addresses github issue #1622.
2022-04-25 18:04:49 +02:00
Willy Tarreau
7c9a0fe2a6 MEDIUM: backend: add new "balance hash <expr>" algorithm
Almost all of our hash-based LB algorithms are implemented as special
cases of something that can now be achieved using sample expressions,
and some of them have adopted some options to adapt their behavior in
ways that could also be achieved using converters.

There are users who want to hash other parameters that are combined
into variables, and who set headers from these values and use
"balance hdr(name)" for this.

Instead of constantly implementing specific options and having users
hack around when they want a real hash, let's implement a native hash
mode that applies to a standard sample expression. This way, any
fetchable element (including variables) may be used to construct the
hash, even modified by any converter if desired.
2022-04-25 16:09:26 +02:00
Willy Tarreau
a4e39890f3 MINOR: task: add a new task_instant_wakeup() function
This function's purpose is to wake up either a local or remote task,
bypassing the tree-based run queue. It is meant for fast wakeups that
are supposed to be equivalent to those used with tasklets, i.e. a task
had to pause some processing and can complete (typically a resource
becomes available again). In all cases, it's important to keep in mind
that the task must have gone through the regular scheduling path before
being blocked, otherwise the task priorities would be ignored.

The reason for this is that some wakeups are massively inter-thread
(e.g. server queues), that these inter-thread wakeups cause a huge
contention on the shared runqueue lock. A user reported 47% CPU spent
in process_runnable_tasks with only 32 threads and 80k requests in
queues. With this mechanism, purely one-to-one wakeups can avoid
taking the lock thanks to the mt_list used for the shared tasklet
queue.

Right now the shared tasklet queue moves everything to the TL_URGENT
queue. It's not dramatic but it would seem better to have a new shared
list dedicated to tasks, and that would deliver into TL_NORMAL, for an
even better fairness. This could be improved in the future.
2022-04-22 19:11:59 +02:00
William Lallemand
b53eb8790e MINOR: init: add the pre-check callback
This adds a call to function <fct> to the list of functions to be called at
the step just before the configuration validity checks. This is useful when you
need to create things like it would have been done during the configuration
parsing and where the initialization should continue in the configuration
check.
It could be used for example to generate a proxy with multiple servers using
the configuration parser itself. At this step the trash buffers are allocated.
Threads are not yet started so no protection is required. The function is
expected to return non-zero on success, or zero on failure. A failure will make
the process emit a succinct error message and immediately exit.
2022-04-22 15:45:47 +02:00
Christopher Faulet
eb50c01fef MINOR: conn-stream: Make cs_detach_* private and use cs_destroy() from outside
A conn-stream is never detached from an endpoint or an application alone,
except on a reset. Thus, to avoid any error, these functions are now
private. And cs_destroy() function is added to destroy a conn-stream. This
function is called when a stream is released, on the front and back
conn-streams, and when a health-check is finished.
2022-04-22 14:32:30 +02:00
Christopher Faulet
ca6c9bba82 CLEANUP: conn-stream: Rename cs_applet_release()
This function does not release the applet but only call the applet release
callback. It is equivalent to cs_conn_shut() but for applets. Thus the
function is renamed cs_applet_shut().
2022-04-22 14:14:27 +02:00
Christopher Faulet
ff022a2b8c CLEANUP: conn-stream: Rename cs_conn_close() and cs_conn_drain_and_close()
These functions don't close the connection but only perform shutdown for
reads and writes at the mux level. It is a bit ambiguous. Thus,
cs_conn_close() is renamed cs_conn_shut() and cs_conn_drain_and_close() is
renamed cs_conn_drain_and_shut(). These both functions rely on
cs_conn_shutw() and cs_conn_shutr().
2022-04-22 14:14:27 +02:00
Remi Tricot-Le Breton
f87c67e5e4 MINOR: ssl: Add 'show ssl providers' cli command and providers list in -vv option
Starting from OpenSSLv3, providers are at the core of cryptography
functions. Depending on the provider used, the way the SSL
functionalities work could change. This new 'show ssl providers' CLI
command allows to show what providers were loaded by the SSL library.
This is required because the provider configuration is exclusively done
in the OpenSSL configuration file (/usr/local/ssl/openssl.cnf for
instance).
A new line is also added to the 'haproxy -vv' output containing the same
information.
2022-04-21 14:54:45 +02:00
Amaury Denoyelle
97e84c6c69 MINOR: cfg-quic: define tune.quic.conn-buf-limit
Add a new global configuration option to set the limit of buffers per
QUIC connection. By default, this value is set to 30.
2022-04-21 12:04:04 +02:00
Amaury Denoyelle
1b2dba531d MINOR: mux-quic: implement immediate send retry
Complete qc_send function. After having processed each qcs emission, it
will now retry send on qcs where transfer can continue. This is useful
when qc_stream_desc buffer is full and there is still data present in
qcs buf.

To implement this, each eligible qcs is inserted in a new list
<qcc.send_retry_list>. This is done on send notification from the
transport layer through qcc_streams_sent_done(). Retry emission until
send_retry_list is empty or the transport layer cannot proceed more
data.

Several send operations are now called on two different places. Thus a
new _qc_send_qcs() function is defined to factorize the code.

This change should maximize the throughput during QUIC transfers.
2022-04-21 12:04:04 +02:00
Amaury Denoyelle
d2f80a2e63 MINOR: quic: limit total stream buffers per connection
MUX streams can now allocate multiple buffers for sending. quic-conn is
responsible to limit the total count of allowed allocated buffers. A
counter is stored in the new field <stream_buf_count>.

For the moment, the value is hardcoded to 30.

On stream buffer allocation failure, the qcc MUX is flagged with
QC_CF_CONN_FULL. The MUX is then woken up as soon as a buffer is freed,
most notably on ACK reception.
2022-04-21 12:04:04 +02:00
Amaury Denoyelle
1b81dda3e0 MINOR: quic-stream: refactor ack management
Acknowledge of STREAM has been complexified with the introduction of
stream multi buffers. Two functions are executing roughly the same set
of instructions in xprt_quic.c.

To simplify this, move the code complexity in a new function
qc_stream_desc_ack(). It will handle offset calculation, removal of
data, freeing oldest buffer and freeing stream instance if required.
The qc_stream_desc API is cleaner as qc_stream_desc_free_buf() ambiguous
function has been removed.
2022-04-21 12:04:04 +02:00
Amaury Denoyelle
a456920491 MEDIUM: quic: implement multi-buffered Tx streams
Complete the qc_stream_desc type to support multiple buffers on
emission. The main objective is to increase the transfer throughput.
The MUX is now able to transfer more data without having to wait ACKs.

To implement this feature, a new type qc_stream_buf is declared. it
encapsulates a buffer with a list element. New functions are defined to
retrieve the current buffer, release it or allocate a new one. Each
buffer is kept in the qc_stream_desc list until all of its data is
acknowledged.

On the MUX side, a qcs uses the current stream buffer to transfer data.
Once the buffer is full, it is released and a new one will be allocated
on a future qc_send() invocation.
2022-04-21 12:03:20 +02:00
Amaury Denoyelle
b22c0460d6 MINOR: quic-stream: add qc field
Add a new member <qc> in qc_stream_desc structure. This change is
possible since previous patch which add quic-conn argument to
qc_stream_desc_new().

The purpose of this change is to simplify the future evolution of
qc-stream-desc API. This will avoid to repeat qc as argument in various
functions which already used a qc_stream_desc.
2022-04-21 11:55:29 +02:00
Amaury Denoyelle
e4301da5ed MINOR: quic-stream: use distinct tree nodes for quic stream and qcs
Simplify the model qcs/qc_stream_desc. Each types has now its own tree
node, stored respectively in qcc and quic-conn trees. It is still
necessary to mark the stream as detached by the MUX once all data is
transfered to the lower layer.

This might improve slightly the performance on ACK management as now
only the lookup in quic-conn is necessary. On the other hand, memory
size of qcs structure is increased.
2022-04-21 11:05:58 +02:00
Amaury Denoyelle
0cc02a345b REORG: quic: use a dedicated module for qc_stream_desc
Regroup all type definitions and functions related to qc_stream_desc in
the source file src/quic_stream.c.

qc_stream_desc complexity will be increased with the development of Tx
multi-buffers. Having a dedicated module is useful to mix it with
pure transport/quic-conn code.
2022-04-21 11:05:27 +02:00
Remi Tricot-Le Breton
1d6338ea96 MEDIUM: ssl: Disable DHE ciphers by default
DHE ciphers do not present a security risk if the key is big enough but
they are slow and mostly obsoleted by ECDHE. This patch removes any
default DH parameters. This will effectively disable all DHE ciphers
unless a global ssl-dh-param-file is defined, or
tune.ssl.default-dh-param is set, or a frontend has DH parameters
included in its PEM certificate. In this latter case, only the frontends
that have DH parameters will have DHE ciphers enabled.
Adding explicitely a DHE ciphers in a "bind" line will not be enough to
actually enable DHE. We would still need to know which DH parameters to
use so one of the three conditions described above must be met.

This request was described in GitHub issue #1604.
2022-04-20 17:30:55 +02:00
David CARLIER
7747d465d5 MINOR: tcp_sample: extend support for get_tcp_info to macOs.
MacOS can feed fc_rtt, fc_rttvar, fc_sacked, fc_lost and fc_retrans
so let's expose them on this platform.

Note that at the tcp(7) level, the API is slightly different, as
struct tcp_info is called tcp_connection_info and TCP_INFO is
called TCP_CONNECTION_INFO, so for convenience these ones were
defined to point to their equivalent. However there is a small
difference now in that tcpi_rtt is called tcpi_rttcur on this
platform, which forces us to make a special case for it before
other platforms.
2022-04-15 17:51:09 +02:00
Willy Tarreau
fb1b6f5bc0 BUILD: compiler: use a more portable set of asm(".weak") statements
The two recent patches b12966af1 ("BUILD: debug: mark the
__start_mem_stats/__stop_mem_stats symbols as weak") and 2a06e248f
("BUILD: initcall: mark the __start_i_* symbols as weak, not global")
aimed at fixing a build warning and resulted in a build breakage on
MacOS which doesn't have a ".weak" asm statement.

We've already had MacOS-specific asm() statements for section names, so
this patch continues on this trend by moving HA_GLOBL() to compiler.h
and using ".globl" on MacOS since apparently nobody complains there.

It is debatable whether to expose this only when !USE_OBSOLETE_LINKER
or all the time, but since these are just macroes it's no big deal to
let them be available when needed and let the caller decide on the
build conditions.

If any of the patches above is backported, this one will need to as
well.
2022-04-14 16:57:12 +02:00
Willy Tarreau
e1efd2a2d7 BUILD: sched: workaround crazy and dangerous warning in Clang 14
Ilya reported in issue #1638 that Clang 14 has invented a new warning
that encourages to modify the code in a way that is not always
equivalent, by turning "|" to "||" between some logical operators,
except that the first one guarantees that all members of the expression
will always be evaluated while the latter will stop at the first one
which is true!

This warning triggers in thread_has_tasks(), which is not sensitive to
such change of behavior but which is built this way because it results
in branchless code for something that most often evaluates to false for
all terms. As such it was out of question to turn this to less efficient
compare-and-jump that needlessly pollute the branch predictor, so the
workaround consists in casting each expression to (int). It was verified
that the code is the same.

Yet another example of how-to-introduce-bugs-by-fixing-valid-code
through warnings invented around a beer without thinking longer!

This may need to be backported to a few older branches in case this
compiler lands in recent distros or if gcc finds it wise to imitate it.
2022-04-14 15:11:12 +02:00
Willy Tarreau
2a06e248f5 BUILD: initcall: mark the __start_i_* symbols as weak, not global
Just like for previous fix, these symbols are marked ".globl" during
their declaration, but their later mention uses __attribute__((weak)),
so it's better to only use ".weak" during the declaration so that the
symbol's class does not change.

No need to backport this unless someone reports build issues.
2022-04-13 19:13:49 +02:00
Willy Tarreau
b12966af10 BUILD: debug: mark the __start_mem_stats/__stop_mem_stats symbols as weak
Building with clang and DEBUG_MEM_STATS shows the following warnings:

  warning: __start_mem_stats changed binding to STB_WEAK [-Wsource-mgr]
  warning: __stop_mem_stats changed binding to STB_WEAK [-Wsource-mgr]

The reason is that the symbols are declared using ".globl" while they
are also referenced as __attribute__((weak)) elsewhere. It turns out
that a weak symbol is implicitly a global one and that the two classes
are exclusive, thus it may confuse the linker. Better fix this.

This may be backported where the patch applies.
2022-04-13 19:13:49 +02:00
Christopher Faulet
1336ccffab CLEANUP: conn-stream: rename cs_register_applet() to cs_applet_create()
cs_register_applet() was not a good name because it suggests it happens
during startup, just like any other registration mechanisms..
2022-04-13 15:10:16 +02:00
Christopher Faulet
a97ccedf6f CLEANUP: muxes: Remove MX_FL_CLEAN_ABRT flag
This flag is unused. Thus, it may be removed. No reason to still set it. It
also cleans up "haproxy -vv" output.
2022-04-13 15:10:16 +02:00
Christopher Faulet
9ed7742673 DOC: conn-stream: Add comments on functions of the new CS api
With the conn-stream refactoring, new functions were added. This patch adds
missing comments to help devs to use them.
2022-04-13 15:10:16 +02:00
Christopher Faulet
265e165d82 CLEANUP: conn-stream: Don't export internal functions
cs_new() and cs_attach_app() are only used internally. Thus, there is no
reason to export them.
2022-04-13 15:10:16 +02:00
Christopher Faulet
6b0a0fb2f9 CLEANUP: tree-wide: Remove any ref to stream-interfaces
Stream-interfaces are gone. Corresponding files can be safely be removed. In
addition, comments are updated accordingly.
2022-04-13 15:10:16 +02:00
Christopher Faulet
582a226a2c MINOR: conn-stream: Remove the stream-interface from the conn-stream
The stream-interface API is no longer used. Thus, it is removed from the
conn-stream. From now, stream-interfaces are now longer used !
2022-04-13 15:10:16 +02:00
Christopher Faulet
0eb32c0dd1 MINOR: stream: Use conn-stream to report server error
the stream's srv_error callback function now manipulates a conn-stream
instead of a stream-interface.
2022-04-13 15:10:16 +02:00
Christopher Faulet
1d03e6e3a1 REORG: stream-int/conn-stream: Move remaining functions to conn-stream
functions to get or set blocking flags on a conn-stream are moved to
conn_stream.h.
2022-04-13 15:10:15 +02:00
Christopher Faulet
5e29b76ea6 MEDIUM: stream-int/conn-stream: Move I/O functions to conn-stream
cs_conn_io_cb(), cs_conn_sync_recv() and cs_conn_sync_send() are moved in
conn_stream.c. Associated functions are moved too (cs_notify, cs_conn_read0,
cs_conn_recv, cs_conn_send and cs_conn_process).
2022-04-13 15:10:15 +02:00
Christopher Faulet
a0bdec350f MEDIUM: stream-int/conn-stream: Move blocking flags from SI to CS
Remaining flags and associated functions are move in the conn-stream
scope. These flags are added on the endpoint and not the conn-stream
itself. This way it will be possible to get them from the mux or the
applet. The functions to get or set these flags are renamed accordingly with
the "cs_" prefix and updated to manipualte a conn-stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
ba92787d13 CLEANUP: stream-int: Remove unused SI functions
Some stream-interface's functions are now unused and can safely be removed.
2022-04-13 15:10:15 +02:00
Christopher Faulet
8f45eec016 MINOR: stream-int/conn-stream: Move si_alloc_ibuf() in the conn-stream scope
si_alloc_ibuf() is renamed as c_alloc_ibuf() and update to manipulate a
conn-stream instead of a stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
158f33615d MINOR: stream-int/conn-stream Move si_is_conn_error() in the conn-stream scope
si_is_conn_error() is renamed as cs_is_conn_erro() and updated to manipulate
a conn-stream instead of a stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
000ba3e613 MINOR: conn-stream: Move si_conn_cb in the conn-stream scope
si_conn_cb variable is renamed cs_data_conn_cb. In addtion, its associated
functions are also renamed. si_cs_recv(), si_cs_send() and si_cs_process() are
renamed cs_conn_recv(), cs_conn_send and cs_conn_process(). These functions are
updated to manipulate conn-streams instead of stream-interfaces.
2022-04-13 15:10:15 +02:00
Christopher Faulet
431ce2e3c1 MINOR: stream-int/conn-stream: Move si_sync_recv/send() in conn-stream scope
si_sync_recv() and si_sync_send() are renamesd cs_conn_recv() and
cs_conn_send() and updated to manipulate conn-streams instead of
stream-interfaces.
2022-04-13 15:10:15 +02:00
Christopher Faulet
4a7764ae9d MINOR: stream-int/conn-stream: Move si_cs_io_cb() in the conn-stream scope
si_cs_io_cb() is renamed cs_conn_io_cb(). In addition, the context of the
tasklet used to wake-up the conn-stream is now a conn-stream.
2022-04-13 15:10:15 +02:00
Christopher Faulet
6059ba4acc MEDIUM: conn-stream/applet: Add a data callback for applets
data callbacks were only used for streams attached to a connection and
for health-checks. However there is a callback used by task_run_applet. So,
si_applet_wake_cb() is first renamed to cs_applet_process() and it is
defined as the data callback for streams attached to an applet. This way,
this part now manipulates a conn-stream instead of a stream-interface. In
addition, applets are no longer handled as an exception for this part.
2022-04-13 15:10:15 +02:00
Christopher Faulet
ef285c18f2 MINOR: stream-int/stream: Move si_update_both in stream scope
si_update_both() is renamed stream_update_both_cs() and moved in stream.c.
The function is slightly changed to manipulate the stream instead the front
and back conn-streams.
2022-04-13 15:10:15 +02:00
Christopher Faulet
13045f0eae MINOR: stream-int-conn-stream: Move si_update_* in conn-stream scope
si_update_rx(), si_update_tx() and si_update() are renamed cs_update_rx(),
cs_upate_tx() and cs_update() and updated to manipulate a conn-stream
instead of a stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
9ffddd5ca5 REORG: conn-stream: Move cs_app_ops in conn_stream.c
Callback functions to perform shutdown for reads and writes and to trigger
I/O calls are now moved in conn_stream.c.
2022-04-13 15:10:15 +02:00
Christopher Faulet
19bd728642 REORG: conn-stream: Move cs_shut* and cs_chk* in cs_utils
cs_shutr(), cs_shutw(), cs_chk_rcv() and cs_chk_snd() are moved in
cs_utils.h
2022-04-13 15:10:15 +02:00
Christopher Faulet
dde33046bd REORG: stream-int: Move si_is_conn_error() in the header file
To ease next changes, this function is moved in the header file. It is a
transient commit.
2022-04-13 15:10:15 +02:00
Christopher Faulet
9b7a9b400d REORG: stream-int: Export si_cs_recv(), si_cs_send() and si_cs_process()
It is a transient commit. It should ease next changes about the conn-stream
refactoring. At the end these functions will be moved in the conn-stream
scope.
2022-04-13 15:10:15 +02:00
Christopher Faulet
aa91d6292b MINOR: stream-int/connection: Move conn_si_send_proxy() in the connection scope
conn_si_send_proxy() function is renamed conn_send_proxy() and moved in
connection.c
2022-04-13 15:10:15 +02:00
Christopher Faulet
64b8d33577 MINOR: connection: unconst mux's get_fist_cs() callback function
This change is mandatory for next commits.
2022-04-13 15:10:15 +02:00
Christopher Faulet
3704663e5f MINOR: applet: Use the CS to register and release applets instead of SI
si_register_applet() and si_applet_release() are renamed
cs_register_applet() and cs_applet_release() and now manipulate a
conn-stream instead of a stream-inteface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
0c6a64cd5f MEDIUM: stream-int/conn-stream: Move si_ops in the conn-stream scope
The si_ops structure is renamed to cs_app_ops and the callback functions are
changed to manipulate a conn-stream instead of a stream-interface..
2022-04-13 15:10:15 +02:00
Christopher Faulet
da098e6c17 MINOR: stream-int/conn-stream: Move si_shut* and si_chk* in conn-stream scope
si_shutr(), si_shutw(), si_chk_rcv() and si_chk_snd() are moved in the
conn-stream scope and renamed, respectively, cs_shutr(), cs_shutw(),
cs_chk_rcv(), cs_chk_snd() and manipulate a conn-stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
69ef6c9ef4 MINOR: conn-stream: Rename CS functions dedicated to connections
Some conn-stream functions are only used when there is a connection. Thus,
they was renamed with "cs_conn_" prefix. In addition, we expect to have a
connection, so a BUG_ON is added to be sure the functions are never called
in another context.
2022-04-13 15:10:15 +02:00
Christopher Faulet
2f35e7b6ab MEDIUM: stream-int/conn-stream: Handle I/O subscriptions in the conn-stream
wait_event structure is moved in the conn-stream. The tasklet is only
created if the conn-stream is attached to a mux and released when the mux is
detached. This implies a subtle change. In stream_int_chk_rcv() function,
the wakeup of the tasklet was removed because there is no longer tasklet at
this stage (stream_int_chk_rcv() is a callback function of si_embedded_ops).
2022-04-13 15:10:15 +02:00
Christopher Faulet
070b91bc11 MEDIUM: conn-stream: Be prepared to fail to attach a cs to a mux
To be able to move wait_event from the stream-interface to the conn-stream,
we must be prepare to handle errors when a mux is attached to a conn-stream.
Indeed, the wait_event's tasklet will be allocated when both a mux and a
stream will be both attached to a stream. So, we must be prepared to handle
allocation errors.
2022-04-13 15:10:15 +02:00
Christopher Faulet
0797656ead MINOR: conn-stream/connection: Move SHR/SHW modes in the connection scope
These flags only concerns the connection part. In addition, it is required
for a next commit, to avoid circular deps. Thus CS_SHR_* and CS_SHW_* were
renamed with the "CO_" prefix.
2022-04-13 15:10:15 +02:00
Christopher Faulet
e39a4dfdf0 MINOR: stream-int/conn-stream: Move si_conn_ready() in the conn-stream scope
si_conn_ready() is renamed cs_conn_ready() and handle a conn-stream insted
of a stream-interface. The function is now in cs_utils.h.
2022-04-13 15:10:15 +02:00
Christopher Faulet
0a4dcb65ff MINOR: stream-int/backend: Move si_connect() in the backend scope
si_connect() is moved in backend.c and renamed as do_connect_server(). In
addition, the function now manipulate a stream instead of a
stream-interface.
2022-04-13 15:10:15 +02:00
Christopher Faulet
9125f3cc77 MINOR: stream-int/stream: Move si_retnclose() in the stream scope
si_retnclose() is used to send a reply to a client before closing. There is
no use on the server side, in spite of the function is generic. Thus, it is
renamed stream_retnclose() and moved into the stream scope. The function now
handle a stream and explicitly send a message to the client.
2022-04-13 15:10:15 +02:00
Christopher Faulet
62e757470a MEDIUM: stream-int/conn-stream: Move stream-interface state in the conn-stream
The stream-interface state (SI_ST_*) is now in the conn-stream. It is a
mechanical replacement for now. Nothing special. SI_ST_* and SI_SB_* were
renamed accordingly. Utils functions to manipulate these infos were moved
under the conn-stream scope.

But it could be good to keep in mind that this part should be
reworked. Indeed, at the CS level, we only need to know if it is ready to
receive or to send. The state of conn-stream from INI to EST is only used on
the server side. The client CS is immediately set to EST. Thus current
SI_ST_* states should probably be moved to the stream to reflect the server
connection state during the establishment stage.
2022-04-13 15:10:15 +02:00
Christopher Faulet
50264b41c8 MEDIUM: stream-int: Move SI err_type in the stream
Only the server side is concerned by the stream-interface error type. It is
useless to have an err_type field on the client side. So, it is now move to
the stream. SI_ET_* are renames STRM_ET_* and moved in stream-t.h header
file.
2022-04-13 15:10:14 +02:00
Christopher Faulet
a70a3548bc MINOR: stream: Only save previous connection state for the server side
The previous connection state on the client side was only used for debugging
purpose to report client close. But this may be handled when the client
stream-interface is switched from SI_ST_DIS to SI_ST_CLO.

So, there only remains the previous connection state on the server side that
is used by the stream, in process_stream(), to be able to set the correct
termination flags. Thus, instead of keeping this info in the
stream-interface for only one side, the info is now stored in the stream
itself.
2022-04-13 15:10:14 +02:00
Christopher Faulet
78ed7f247b CLEANUP: stream-int: Remove unused SI_FL_CLEAN_ABRT flag
This flag is unused. So remove it to be able to remove the stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
d139138bbc MINOR: stream-int: Remove SI_FL_SRC_ADDR to rely on stream flags instead
Flag to get the source ip/port with getsockname is now handled at the stream
level. Thus SI_FL_SRC_ADDR stream-int flag is replaced by SF_SRC_ADDR stream
flag.
2022-04-13 15:10:14 +02:00
Christopher Faulet
a728518c15 MINOR: stream-int: Remove SI_FL_INDEP_STR to rely on CS flags instead
Flag to consider a stream as indepenent is now handled at the conn-stream
level. Thus SI_FL_INDEP_STR stream-int flag is replaced by CS_FL_INDEP_STR
conn-stream flags.
2022-04-13 15:10:14 +02:00
Christopher Faulet
974da9f8a4 MINOR: stream-int: Remove SI_FL_DONT_WAKE to rely on CS flags instead
Flag to not wake the stream up on I/O is now handled at the conn-stream
level. Thus SI_FL_DONT_WAKE stream-int flag is replaced by CS_FL_DONT_WAKE
conn-stream flags.
2022-04-13 15:10:14 +02:00
Christopher Faulet
8abe712749 MINOR: stream-int: Remove SI_FL_NOLINGER/NOHALF to rely on CS flags instead
Flags to disable lingering and half-close are now handled at the conn-stream
level. Thus SI_FL_NOLINGER and SI_FL_NOHALF stream-int flags are replaced by
CS_FL_NOLINGER and CS_FL_NOHALF conn-stream flags.
2022-04-13 15:10:14 +02:00
Christopher Faulet
9a52123800 MINOR: stream-int: Remove SI_FL_KILL_CON to rely on conn-stream endpoint only
Instead of setting a stream-interface flag to then set the corresponding
conn-stream endpoint flag, we now only rely the conn-stream endoint. Thus
SI_FL_KILL_CON is replaced by CS_EP_KILL_CONN.

In addition si_must_kill_conn() is replaced by cs_must_kill_conn().
2022-04-13 15:10:14 +02:00
Christopher Faulet
7b5ca8f457 MINOR: channel: Use conn-streams as channel producer and consumer
chn_prod() and chn_cons() now return a conn-stream instead of a
stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
6cd56d5a69 MEDIUM: conn-stream: Use endpoint error instead of conn-stream error
Instead of relying on the conn-stream error, via CS_FL_ERR flags, we now
directly use the error at the endpoint level with the flag CS_EP_ERROR. It
should be safe to do so. But we must be careful because it is still possible
that an error is processed too early. Anyway, a conn-stream has always a
valid endpoint, maybe detached from any endpoint, but valid.
2022-04-13 15:10:14 +02:00
Christopher Faulet
af642df3b8 MINOR: stream-int/conn-stream: Report error to the CS instead of the SI
SI_FL_ERR is removed and replaced by CS_FL_ERROR. It is a transient patch
because the idea is to rely on the endpoint to handle errors at this
level. But if for any reason it is not possible, the stream-interface flags
will still be replaced.
2022-04-13 15:10:14 +02:00
Christopher Faulet
ae024ced03 MEDIUM: stream-int/stream: Use connect expiration instead of SI expiration
The expiration date in the stream-interface was only used on the server side
to set the connect, queue or turn-around timeout. It was checked on the
frontend stream-interface, but never used concretely. So it was removed and
replaced by a connect expiration date in the stream itself. Thus, SI_FL_EXP
flag in stream-interfaces is replaced by a stream flag, SF_CONN_EXP.
2022-04-13 15:10:14 +02:00
Christopher Faulet
1d9877700e MINOR: stream-int/conn-stream: Move half-close timeout in the conn-stream
The half-close timeout (hcto) is now part of the conn-stream. It is a step
closer to the stream-interface removal.
2022-04-13 15:10:14 +02:00
Christopher Faulet
8da67aae3e MEDIUM: stream-int/conn-stream: Move src/dst addresses in the conn-stream
The source and destination addresses at the applicative layer are moved from
the stream-interface to the conn-stream. This simplifies a bit the code and
it is a logicial step to remove the stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
731c8e6cf9 MINOR: stream: Simplify retries counter calculation
The conn_retries counter was set to the max value and decremented at each
connection retry. Thus the counter reflected the number of retries left and
not the real number of retries. All calculations of redispatch or reporting
of number of retries experienced were made using subtracts from the
configured retries, which was complicated and didn't bring any benefit.

Now, this counter is set to 0 and incremented at each retry. We know we've
reached the maximum allowed connection retries by comparing it to the
configured value. In all other cases, we directly use the counter.

This patch should address the feature request #1608.
2022-04-13 15:10:14 +02:00
Christopher Faulet
909f318259 MINOR: stream-int/stream: Move conn_retries counter in the stream
The conn_retries counter may be moved into the stream structure. It only
concerns the connection establishment. The frontend stream-interface does not
use it. So it is a logical change.
2022-04-13 15:10:14 +02:00
Christopher Faulet
e05bf9e413 MINOR: stream-int/txn: Move buffer for L7 retries in the HTTP transaction
The L7 retries only concerns the stream when a server connection is
established. Thus instead of storing the L7 buffer into the
stream-interface, it may be moved to the stream. And because it is only
available for HTTP streams, it may be moved in the HTTP transaction.

Associated flags are also moved into the HTTP transaction.
2022-04-13 15:10:14 +02:00
Christopher Faulet
908628c4c0 MEDIUM: tree-wide: Use CS util functions instead of SI ones
At many places, we now use the new CS functions to get a stream or a channel
from a conn-stream instead of using the stream-interface API. It is the
first step to reduce the scope of the stream-interfaces. The main change
here is about the applet I/O callback functions. Before the refactoring, the
stream-interface was the appctx owner. Thus, it was heavily used. Now, as
far as possible,the conn-stream is used. Of course, it remains many calls to
the stream-interface API.
2022-04-13 15:10:14 +02:00
Christopher Faulet
1cf9ef25a9 MINOR: conn-stream: Add header file with util functions related to conn-streams
cs_utils.h header file will contain all util functions related to the
conn_streams. For now, few functions were added, all are equivalent to SI
functions. Idea is to progressively replace SI functions by CS ones.
2022-04-13 15:10:14 +02:00
Christopher Faulet
3099511571 MINOR: conn-stream: Add ISBACK conn-stream flag
CS_FL_ISBACK is a new flag, set on backend conn-streams. We must just be
careful to preserve this flag when the endpoint is detached from the
conn-stream.
2022-04-13 15:10:14 +02:00
Christopher Faulet
b041b23ae4 MEDIUM: conn-stream: Move remaning flags from CS to endpoint
All old flags CS_FL_* are now moved in the endpoint scope and renamed
CS_EP_* accordingly. It is a systematic replacement. There is no true change
except for the health-check and the endpoint reset. Here it is a bit special
because the same conn-stream is reused. Thus, we must handle endpoint
allocation errors. To do so, cs_reset_endp() has been adapted.

Thanks to this last change, it will now be possible to simplify the
multiplexer and probably the applets too. A review must also be performed to
remove some flags in the channel or the stream-interface. The HTX will
probably be simplified too. Finally, there is now some place in the
conn-stream to move info from the stream-interface.
2022-04-13 15:10:14 +02:00
Christopher Faulet
9ec2f4dc7c MAJOR: conn-stream: Share endpoint struct between the CS and the mux/applet
The conn-stream endpoint is now shared between the conn-stream and the
applet or the multiplexer. If the mux or the applet is created first, it is
responsible to also create the endpoint and share it with the conn-stream.
If the conn-stream is created first, it is the opposite.

When the endpoint is only owned by an applet or a mux, it is called an
orphan endpoint (there is no conn-stream). When it is only owned by a
conn-stream, it is called a detached endpoint (there is no mux/applet).

The last entity that owns an endpoint is responsible to release it. When a
mux or an applet is detached from a conn-stream, the conn-stream
relinquishes the endpoint to recreate a new one. This way, the endpoint
state is never lost for the mux or the applet.
2022-04-13 15:10:14 +02:00
Christopher Faulet
cb2fa368e9 REORG: applet: Uninline appctx_new function
appctx_new() is moved in the C file and appctx_init() is now private.
2022-04-13 15:10:14 +02:00
Christopher Faulet
a9e8b3979d MEDIUM: conn-stream: Pre-allocate endpoint to create CS from muxes and applets
It is a transient commit to prepare next changes. Now, when a conn-stream is
created from an applet or a multiplexer, an endpoint is always provided. In
addition, the API to create a conn-stream was specialized to have one
function per type.

The next step will be to share the endpoint structure.
2022-04-13 15:10:14 +02:00
Christopher Faulet
b669d684c0 MEDIUM: conn-stream: Be able to pass endpoint to create a conn-stream
It is a transient commit to prepare next changes. It is possible to pass a
pre-allocated endpoint to create a new conn-stream. If it is NULL, a new
endpoint is created, otherwise the existing one is used. There no more
change at the conn-stream level.

In the applets, all conn-stream are created with no pre-allocated
endpoint. But for multiplexers, an endpoint is systematically created before
creating the conn-stream.
2022-04-13 15:10:14 +02:00
Christopher Faulet
e9e4820288 MINOR: conn-stream: Move some CS flags to the endpoint
Some CS flags, only related to the endpoint, are moved into the endpoint
struct. More will probably moved later. Those ones are not critical. So it
is pretty safe to move them now and this will ease next changes.
2022-04-13 15:10:14 +02:00
Christopher Faulet
db90f2aa9f MEDIUM: conn-stream: Add an endpoint structure in the conn-stream
Group the endpoint target of a conn-stream, its context and the associated
flags in a dedicated structure in the conn-stream. It is not inlined in the
conn-stream structure. There is a dedicated pool.

For now, there is no complexity. It is just an indirection to get the
endpoint or its context. But the purpose of this structure is to be able to
share a refcounted context between the mux and the conn-stream. This way, it
will be possible to preserve it when the mux is detached from the
conn-stream.
2022-04-13 15:10:14 +02:00
Christopher Faulet
bb772d09f5 REORG: Initialize the conn-stream by hand in cs_init()
The function cs_init() is only called by cs_new(). The conn-stream
initialization will be reviewed. It is easier to do it in cs_new() instead
of using a dedicated function. cs_new() is pretty simple, there is no reason
to split the code in this case.
2022-04-13 15:10:14 +02:00
Christopher Faulet
9388204db1 MAJOR: conn-stream: Invert conn-stream endpoint and its context
This change is only significant for the multiplexer part. For the applets,
the context and the endpoint are the same. Thus, there is no much change. For
the multiplexer part, the connection was used to set the conn-stream
endpoint and the mux's stream was the context. But it is a bit strange
because once a mux is installed, it takes over the connection. In a
wonderful world, the connection should be totally hidden behind the mux. The
stream-interface and, in a lesser extent, the stream, still access the
connection because that was inherited from the pre-multiplexer era.

Now, the conn-stream endpoint is the mux's stream (an opaque entity for the
conn-stream) and the connection is the context. Dedicated functions have
been added to attached an applet or a mux to a conn-stream.
2022-04-13 15:10:14 +02:00
Christopher Faulet
2479e5f775 MEDIUM: applet: Set the appctx owner during allocation
The appctx owner is now always a conn-stream. Thus, it can be set during the
appctx allocation. But, to do so, the conn-stream must be created first. It
is not a problem on the server side because the conn-stream is created with
the stream. On the client side, we must take care to create the conn-stream
first.

This change should ease other changes about the applets bootstrapping.
2022-04-13 15:10:13 +02:00