upstream/haproxy
1
0
Fork 0
mirror of https://github.com/haproxy/haproxy.git synced 2026-04-27 00:57:03 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
haproxy/src/h1.c

1274 lines
38 KiB
C
Raw Normal View History

REORG: http: move some very http1-specific parts to h1.{c,h} Certain types and enums are very specific to the HTTP/1 parser, and we'll need to share them with the HTTP/2 to HTTP/1 translation code. Let's move them to h1.c/h1.h. Those with very few occurrences or only used locally were renamed to explicitly mention the relevant HTTP version : enum ht_state -> h1_state. http_msg_state_str -> h1_msg_state_str HTTP_FLG_* -> H1_FLG_* http_char_classes -> h1_char_classes Others like HTTP_IS_*, HTTP_MSG_* are left to be done later.
2017-09-21 03:30:46 -04:00
/*
* HTTP/1 protocol analyzer
*
* Copyright 2000-2017 Willy Tarreau <w@1wt.eu>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version
* 2 of the License, or (at your option) any later version.
*
*/
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
#include <ctype.h>
MEDIUM: h1: generate WebSocket key on response if needed Add the Sec-Websocket-Accept header on a websocket handshake response. This header may be missing if a h2 server is used with a h1 client. The response key is calculated following the rfc6455. For this, the handshake request key must be stored in the h1 session, as a new field name ws_key. Note that this is only done if the message has been prealably identified as a Websocket handshake request.
2020-12-11 11:53:03 -05:00
#include <import/sha1.h>
REORG: include: update all files to use haproxy/api.h or api-t.h if needed All files that were including one of the following include files have been updated to only include haproxy/api.h or haproxy/api-t.h once instead: - common/config.h - common/compat.h - common/compiler.h - common/defaults.h - common/initcall.h - common/tools.h The choice is simple: if the file only requires type definitions, it includes api-t.h, otherwise it includes the full api.h. In addition, in these files, explicit includes for inttypes.h and limits.h were dropped since these are now covered by api.h and api-t.h. No other change was performed, given that this patch is large and affects 201 files. At least one (tools.h) was already freestanding and didn't get the new one added.
2020-05-27 06:58:42 -04:00
#include <haproxy/api.h>
MEDIUM: h1: generate WebSocket key on response if needed Add the Sec-Websocket-Accept header on a websocket handshake response. This header may be missing if a h2 server is used with a h1 client. The response key is calculated following the rfc6455. For this, the handshake request key must be stored in the h1 session, as a new field name ws_key. Note that this is only done if the message has been prealably identified as a Websocket handshake request.
2020-12-11 11:53:03 -05:00
#include <haproxy/base64.h>
MEDIUM: h1: allow to preserve keep-alive on T-E + C-L In 2.5-dev9, commit 631c7e866 ("MEDIUM: h1: Force close mode for invalid uses of T-E header") enforced a recently arrived new security rule in the HTTP specification aiming at preventing a class of content-smuggling attacks involving HTTP/1.0 agents. It consists in handling the very rare T-E + C-L requests or responses in close mode. It happens it does have an impact of a rare few and very old clients (probably running insecure TLS stacks by the way) that continue to send both with their POST requests. The impact is that for each and every request they'll have to reconnect, possibly negotiating a full TLS handshake that becomes harmful to the machine in terms of CPU computation. This commit adds a new option "h1-do-not-close-on-insecure-transfer-encoding" that does exactly what it says, it just asks not to close on such messages, even though the message continues to be sanitized and C-L dropped. It means that the risk is only between the sender and haproxy, which is limited, and might be the only acceptable solution for such environments having to deal with broken implementations. The cases are so rare that it should not need to be backported, or in the worst case, to the latest LTS if there is any demand.
2024-07-26 09:14:58 -04:00
#include <haproxy/cfgparse.h>
REORG: include: move common/h1.h to haproxy/h1.h The file was moved as-is. There was a wrong dependency on dynbuf.h instead of buf.h which was addressed. There was no benefit to splitting this between types and functions.
2020-06-02 13:33:08 -04:00
#include <haproxy/h1.h>
REORG: include: split common/http-hdr.h into haproxy/http-hdr{,-t}.h There's only one struct and 2 inline functions. It could have been merged into http.h but that would have added a massive dependency on the hpack parts for nothing, so better keep it this way since hpack is already freestanding and portable.
2020-06-02 13:25:28 -04:00
#include <haproxy/http-hdr.h>
MEDIUM: h1: add a WebSocket key on handshake if needed Add the header Sec-Websocket-Key when generating a h1 handshake websocket without this header. This is the case when doing h2-h1 conversion. The key is randomly generated and base64 encoded. It is stored on the session side to be able to verify response key and reject it if not valid.
2020-12-11 11:53:07 -05:00
#include <haproxy/tools.h>
REORG: http: move some very http1-specific parts to h1.{c,h} Certain types and enums are very specific to the HTTP/1 parser, and we'll need to share them with the HTTP/2 to HTTP/1 translation code. Let's move them to h1.c/h1.h. Those with very few occurrences or only used locally were renamed to explicitly mention the relevant HTTP version : enum ht_state -> h1_state. http_msg_state_str -> h1_msg_state_str HTTP_FLG_* -> H1_FLG_* http_char_classes -> h1_char_classes Others like HTTP_IS_*, HTTP_MSG_* are left to be done later.
2017-09-21 03:30:46 -04:00
MEDIUM: h1: allow to preserve keep-alive on T-E + C-L In 2.5-dev9, commit 631c7e866 ("MEDIUM: h1: Force close mode for invalid uses of T-E header") enforced a recently arrived new security rule in the HTTP specification aiming at preventing a class of content-smuggling attacks involving HTTP/1.0 agents. It consists in handling the very rare T-E + C-L requests or responses in close mode. It happens it does have an impact of a rare few and very old clients (probably running insecure TLS stacks by the way) that continue to send both with their POST requests. The impact is that for each and every request they'll have to reconnect, possibly negotiating a full TLS handshake that becomes harmful to the machine in terms of CPU computation. This commit adds a new option "h1-do-not-close-on-insecure-transfer-encoding" that does exactly what it says, it just asks not to close on such messages, even though the message continues to be sanitized and C-L dropped. It means that the risk is only between the sender and haproxy, which is limited, and might be the only acceptable solution for such environments having to deal with broken implementations. The cases are so rare that it should not need to be backported, or in the worst case, to the latest LTS if there is any demand.
2024-07-26 09:14:58 -04:00
/* by default, RFC9112#6.1 applies, t-e combined with c-l represents a risk of
* smuggling if it crosses another 1.0 agent so we must close at the end of the
* transaction. But it may cause difficulties to some very old broken devices.
*/
int h1_do_not_close_on_insecure_t_e = 0;
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
/* Parse the Transfer-Encoding: header field of an HTTP/1 request, looking for
MINOR: h1: Change T-E header parsing to fail if chunked encoding is found twice According to the RFC7230, "chunked" encoding must not be applied more than once to a message body. To handle this case, h1_parse_xfer_enc_header() is now responsible to fail when a parsing error is found. It also fails if the "chunked" encoding is not the last one for a request. To help the parsing, two H1 parser flags have been added: H1_MF_TE_CHUNKED and H1_MF_TE_OTHER. These flags are set, respectively, when "chunked" encoding and any other encoding are found. H1_MF_CHNK flag is used when "chunked" encoding is the last one.
2021-09-28 03:36:25 -04:00
* "chunked" encoding to perform some checks (it must be the last encoding for
* the request and must not be performed twice for any message). The
* H1_MF_TE_CHUNKED is set if a valid "chunked" encoding is found. The
* H1_MF_TE_OTHER flag is set if any other encoding is found. The H1_MF_XFER_ENC
* flag is always set. The H1_MF_CHNK is set when "chunked" encoding is the last
* one. Note that transfer codings are case-insensitive (cf RFC7230#4). This
MEDIUM: h1: Accept invalid T-E values with accept-invalid-http-response option Since the 2.6, A parsing error is reported when the chunked encoding is found twice. As stated in RFC9112, A sender must not apply the chunked transfer coding more than once to a message body. It means only one chunked coding must be found. In addition, empty values are also rejected becaues it is forbidden by RFC9110. However, in both cases, it may be useful to relax the rules for trusted legacy servers when accept-invalid-http-response option is set. Especially because it was accepted on 2.4 and older. In addition, T-E header is now sanitized before sending it. It is not a problem Because it is a hop-by-hop header Note that it remains invalid on client side because there is no good reason to relax the parsing on this side. We can argue a server is trusted so we can decide to support some legacy behavior. It is not true on client side and it is highly suspicious if a client is sending an invalid T-E header. Note also we continue to reject unsupported T-E values (so all codings except "chunked"). Because the "TE" header is sanitized and cannot contain other value than "Trailers", there is absolutely no reason for a server to use something else. This patch should fix the issue #2677. It could probably be backported as far as 2.6 if necessary.
2024-09-11 09:02:07 -04:00
* function returns -2 for a fatal error, -1 for an error that may be hiidden by
* config, 0 if the whole header can be dropped (not used yet), or >0 if the
* value can be indexed.
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
*/
MINOR: h1: Change T-E header parsing to fail if chunked encoding is found twice According to the RFC7230, "chunked" encoding must not be applied more than once to a message body. To handle this case, h1_parse_xfer_enc_header() is now responsible to fail when a parsing error is found. It also fails if the "chunked" encoding is not the last one for a request. To help the parsing, two H1 parser flags have been added: H1_MF_TE_CHUNKED and H1_MF_TE_OTHER. These flags are set, respectively, when "chunked" encoding and any other encoding are found. H1_MF_CHNK flag is used when "chunked" encoding is the last one.
2021-09-28 03:36:25 -04:00
int h1_parse_xfer_enc_header(struct h1m *h1m, struct ist value)
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
{
char *e, *n;
struct ist word;
MEDIUM: h1: Accept invalid T-E values with accept-invalid-http-response option Since the 2.6, A parsing error is reported when the chunked encoding is found twice. As stated in RFC9112, A sender must not apply the chunked transfer coding more than once to a message body. It means only one chunked coding must be found. In addition, empty values are also rejected becaues it is forbidden by RFC9110. However, in both cases, it may be useful to relax the rules for trusted legacy servers when accept-invalid-http-response option is set. Especially because it was accepted on 2.4 and older. In addition, T-E header is now sanitized before sending it. It is not a problem Because it is a hop-by-hop header Note that it remains invalid on client side because there is no good reason to relax the parsing on this side. We can argue a server is trusted so we can decide to support some legacy behavior. It is not true on client side and it is highly suspicious if a client is sending an invalid T-E header. Note also we continue to reject unsupported T-E values (so all codings except "chunked"). Because the "TE" header is sanitized and cannot contain other value than "Trailers", there is absolutely no reason for a server to use something else. This patch should fix the issue #2677. It could probably be backported as far as 2.6 if necessary.
2024-09-11 09:02:07 -04:00
int ret = 1;
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
BUG/MEDIUM: h1: Reject empty Transfer-encoding header The Transfer-Encoding headers list the transfer coding that have been applied to the content in order to form the message body. It is a list of tokens. And as specified by RFC 9110, a token cannot be empty. When several coding names are specify as a comma-separated value, this case is properly handled and an error is triggered. However, an empty header value will just be skipped and no error is triggered. This could be an issue with some buggy servers. Now, empty Transfer-Encoding header are rejected too. This patch must be backported as far as 2.6.
2024-07-09 01:55:58 -04:00
/* Reject empty header */
MEDIUM: h1: Accept invalid T-E values with accept-invalid-http-response option Since the 2.6, A parsing error is reported when the chunked encoding is found twice. As stated in RFC9112, A sender must not apply the chunked transfer coding more than once to a message body. It means only one chunked coding must be found. In addition, empty values are also rejected becaues it is forbidden by RFC9110. However, in both cases, it may be useful to relax the rules for trusted legacy servers when accept-invalid-http-response option is set. Especially because it was accepted on 2.4 and older. In addition, T-E header is now sanitized before sending it. It is not a problem Because it is a hop-by-hop header Note that it remains invalid on client side because there is no good reason to relax the parsing on this side. We can argue a server is trusted so we can decide to support some legacy behavior. It is not true on client side and it is highly suspicious if a client is sending an invalid T-E header. Note also we continue to reject unsupported T-E values (so all codings except "chunked"). Because the "TE" header is sanitized and cannot contain other value than "Trailers", there is absolutely no reason for a server to use something else. This patch should fix the issue #2677. It could probably be backported as far as 2.6 if necessary.
2024-09-11 09:02:07 -04:00
if (istptr(value) == istend(value)) {
ret = -1;
goto end;
}
BUG/MEDIUM: h1: Reject empty Transfer-encoding header The Transfer-Encoding headers list the transfer coding that have been applied to the content in order to form the message body. It is a list of tokens. And as specified by RFC 9110, a token cannot be empty. When several coding names are specify as a comma-separated value, this case is properly handled and an error is triggered. However, an empty header value will just be skipped and no error is triggered. This could be an issue with some buggy servers. Now, empty Transfer-Encoding header are rejected too. This patch must be backported as far as 2.6.
2024-07-09 01:55:58 -04:00
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
h1m->flags |= H1_MF_XFER_ENC;
word.ptr = value.ptr - 1; // -1 for next loop's pre-increment
CLEANUP: Apply ist.cocci Make use of the new rules to use `istend()`.
2021-11-06 10:14:44 -04:00
e = istend(value);
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
while (++word.ptr < e) {
CLEANUP: assorted typo fixes in the code and comments This is 10th iteration of typo fixes
2020-06-21 12:42:57 -04:00
/* skip leading delimiter and blanks */
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
if (HTTP_IS_LWS(*word.ptr))
continue;
n = http_find_hdr_value_end(word.ptr, e); // next comma or end of line
BUG/MINOR: h1: Reject empty coding name as last transfer-encoding value The following Transfer-Encoding header is now rejected with a 400-bad-request: Transfer-Encoding: chunked,\r\n This case was not properly handled and the last empty value was just ignored. This patch must be backported as far as 2.6.
2024-07-09 02:57:53 -04:00
/* a comma at the end means the last value is empty */
if (n+1 == e)
MEDIUM: h1: Accept invalid T-E values with accept-invalid-http-response option Since the 2.6, A parsing error is reported when the chunked encoding is found twice. As stated in RFC9112, A sender must not apply the chunked transfer coding more than once to a message body. It means only one chunked coding must be found. In addition, empty values are also rejected becaues it is forbidden by RFC9110. However, in both cases, it may be useful to relax the rules for trusted legacy servers when accept-invalid-http-response option is set. Especially because it was accepted on 2.4 and older. In addition, T-E header is now sanitized before sending it. It is not a problem Because it is a hop-by-hop header Note that it remains invalid on client side because there is no good reason to relax the parsing on this side. We can argue a server is trusted so we can decide to support some legacy behavior. It is not true on client side and it is highly suspicious if a client is sending an invalid T-E header. Note also we continue to reject unsupported T-E values (so all codings except "chunked"). Because the "TE" header is sanitized and cannot contain other value than "Trailers", there is absolutely no reason for a server to use something else. This patch should fix the issue #2677. It could probably be backported as far as 2.6 if necessary.
2024-09-11 09:02:07 -04:00
ret = -1;
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
word.len = n - word.ptr;
/* trim trailing blanks */
while (word.len && HTTP_IS_LWS(word.ptr[word.len-1]))
word.len--;
h1m->flags &= ~H1_MF_CHNK;
BUG/MINOR: h1: Fail to parse empty transfer coding names Empty transfer coding names, inside a comma-separated list, are already rejected. But it is only by chance. Today, it is detected as an unknown coding names (not "chunked" concretly). Then, it is handled by the H1 multiplexer as an error and a 422-Unprocessable-Content response is returned. So, the error is properly detected in this case, but it is not accurate. A 400-bad-request response must be returned instead. Then, it is better to catch the error during the header parsing. It is the purpose of this patch. This patch should be backported as far as 2.6.
2024-07-09 02:15:14 -04:00
/* empty values are forbidden */
if (!word.len)
MEDIUM: h1: Accept invalid T-E values with accept-invalid-http-response option Since the 2.6, A parsing error is reported when the chunked encoding is found twice. As stated in RFC9112, A sender must not apply the chunked transfer coding more than once to a message body. It means only one chunked coding must be found. In addition, empty values are also rejected becaues it is forbidden by RFC9110. However, in both cases, it may be useful to relax the rules for trusted legacy servers when accept-invalid-http-response option is set. Especially because it was accepted on 2.4 and older. In addition, T-E header is now sanitized before sending it. It is not a problem Because it is a hop-by-hop header Note that it remains invalid on client side because there is no good reason to relax the parsing on this side. We can argue a server is trusted so we can decide to support some legacy behavior. It is not true on client side and it is highly suspicious if a client is sending an invalid T-E header. Note also we continue to reject unsupported T-E values (so all codings except "chunked"). Because the "TE" header is sanitized and cannot contain other value than "Trailers", there is absolutely no reason for a server to use something else. This patch should fix the issue #2677. It could probably be backported as far as 2.6 if necessary.
2024-09-11 09:02:07 -04:00
ret = -1;
BUG/MINOR: h1: Fail to parse empty transfer coding names Empty transfer coding names, inside a comma-separated list, are already rejected. But it is only by chance. Today, it is detected as an unknown coding names (not "chunked" concretly). Then, it is handled by the H1 multiplexer as an error and a 422-Unprocessable-Content response is returned. So, the error is properly detected in this case, but it is not accurate. A 400-bad-request response must be returned instead. Then, it is better to catch the error during the header parsing. It is the purpose of this patch. This patch should be backported as far as 2.6.
2024-07-09 02:15:14 -04:00
else if (isteqi(word, ist("chunked"))) {
MINOR: h1: Change T-E header parsing to fail if chunked encoding is found twice According to the RFC7230, "chunked" encoding must not be applied more than once to a message body. To handle this case, h1_parse_xfer_enc_header() is now responsible to fail when a parsing error is found. It also fails if the "chunked" encoding is not the last one for a request. To help the parsing, two H1 parser flags have been added: H1_MF_TE_CHUNKED and H1_MF_TE_OTHER. These flags are set, respectively, when "chunked" encoding and any other encoding are found. H1_MF_CHNK flag is used when "chunked" encoding is the last one.
2021-09-28 03:36:25 -04:00
if (h1m->flags & H1_MF_TE_CHUNKED) {
/* cf RFC7230#3.3.1 : A sender MUST NOT apply
* chunked more than once to a message body
* (i.e., chunking an already chunked message is
* not allowed)
*/
MEDIUM: h1: Accept invalid T-E values with accept-invalid-http-response option Since the 2.6, A parsing error is reported when the chunked encoding is found twice. As stated in RFC9112, A sender must not apply the chunked transfer coding more than once to a message body. It means only one chunked coding must be found. In addition, empty values are also rejected becaues it is forbidden by RFC9110. However, in both cases, it may be useful to relax the rules for trusted legacy servers when accept-invalid-http-response option is set. Especially because it was accepted on 2.4 and older. In addition, T-E header is now sanitized before sending it. It is not a problem Because it is a hop-by-hop header Note that it remains invalid on client side because there is no good reason to relax the parsing on this side. We can argue a server is trusted so we can decide to support some legacy behavior. It is not true on client side and it is highly suspicious if a client is sending an invalid T-E header. Note also we continue to reject unsupported T-E values (so all codings except "chunked"). Because the "TE" header is sanitized and cannot contain other value than "Trailers", there is absolutely no reason for a server to use something else. This patch should fix the issue #2677. It could probably be backported as far as 2.6 if necessary.
2024-09-11 09:02:07 -04:00
ret = -1;
MINOR: h1: Change T-E header parsing to fail if chunked encoding is found twice According to the RFC7230, "chunked" encoding must not be applied more than once to a message body. To handle this case, h1_parse_xfer_enc_header() is now responsible to fail when a parsing error is found. It also fails if the "chunked" encoding is not the last one for a request. To help the parsing, two H1 parser flags have been added: H1_MF_TE_CHUNKED and H1_MF_TE_OTHER. These flags are set, respectively, when "chunked" encoding and any other encoding are found. H1_MF_CHNK flag is used when "chunked" encoding is the last one.
2021-09-28 03:36:25 -04:00
}
h1m->flags |= (H1_MF_TE_CHUNKED|H1_MF_CHNK);
}
else {
if ((h1m->flags & (H1_MF_RESP|H1_MF_TE_CHUNKED)) == H1_MF_TE_CHUNKED) {
/* cf RFC7230#3.3.1 : If any transfer coding
* other than chunked is applied to a request
* payload body, the sender MUST apply chunked
* as the final transfer coding to ensure that
* the message is properly framed.
*/
MEDIUM: h1: Accept invalid T-E values with accept-invalid-http-response option Since the 2.6, A parsing error is reported when the chunked encoding is found twice. As stated in RFC9112, A sender must not apply the chunked transfer coding more than once to a message body. It means only one chunked coding must be found. In addition, empty values are also rejected becaues it is forbidden by RFC9110. However, in both cases, it may be useful to relax the rules for trusted legacy servers when accept-invalid-http-response option is set. Especially because it was accepted on 2.4 and older. In addition, T-E header is now sanitized before sending it. It is not a problem Because it is a hop-by-hop header Note that it remains invalid on client side because there is no good reason to relax the parsing on this side. We can argue a server is trusted so we can decide to support some legacy behavior. It is not true on client side and it is highly suspicious if a client is sending an invalid T-E header. Note also we continue to reject unsupported T-E values (so all codings except "chunked"). Because the "TE" header is sanitized and cannot contain other value than "Trailers", there is absolutely no reason for a server to use something else. This patch should fix the issue #2677. It could probably be backported as far as 2.6 if necessary.
2024-09-11 09:02:07 -04:00
ret = -2;
goto end;
MINOR: h1: Change T-E header parsing to fail if chunked encoding is found twice According to the RFC7230, "chunked" encoding must not be applied more than once to a message body. To handle this case, h1_parse_xfer_enc_header() is now responsible to fail when a parsing error is found. It also fails if the "chunked" encoding is not the last one for a request. To help the parsing, two H1 parser flags have been added: H1_MF_TE_CHUNKED and H1_MF_TE_OTHER. These flags are set, respectively, when "chunked" encoding and any other encoding are found. H1_MF_CHNK flag is used when "chunked" encoding is the last one.
2021-09-28 03:36:25 -04:00
}
h1m->flags |= H1_MF_TE_OTHER;
}
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
word.ptr = n;
}
MINOR: h1: Change T-E header parsing to fail if chunked encoding is found twice According to the RFC7230, "chunked" encoding must not be applied more than once to a message body. To handle this case, h1_parse_xfer_enc_header() is now responsible to fail when a parsing error is found. It also fails if the "chunked" encoding is not the last one for a request. To help the parsing, two H1 parser flags have been added: H1_MF_TE_CHUNKED and H1_MF_TE_OTHER. These flags are set, respectively, when "chunked" encoding and any other encoding are found. H1_MF_CHNK flag is used when "chunked" encoding is the last one.
2021-09-28 03:36:25 -04:00
MEDIUM: h1: Accept invalid T-E values with accept-invalid-http-response option Since the 2.6, A parsing error is reported when the chunked encoding is found twice. As stated in RFC9112, A sender must not apply the chunked transfer coding more than once to a message body. It means only one chunked coding must be found. In addition, empty values are also rejected becaues it is forbidden by RFC9110. However, in both cases, it may be useful to relax the rules for trusted legacy servers when accept-invalid-http-response option is set. Especially because it was accepted on 2.4 and older. In addition, T-E header is now sanitized before sending it. It is not a problem Because it is a hop-by-hop header Note that it remains invalid on client side because there is no good reason to relax the parsing on this side. We can argue a server is trusted so we can decide to support some legacy behavior. It is not true on client side and it is highly suspicious if a client is sending an invalid T-E header. Note also we continue to reject unsupported T-E values (so all codings except "chunked"). Because the "TE" header is sanitized and cannot contain other value than "Trailers", there is absolutely no reason for a server to use something else. This patch should fix the issue #2677. It could probably be backported as far as 2.6 if necessary.
2024-09-11 09:02:07 -04:00
end:
return ret;
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
}
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
/* Validate the authority and the host header value for CONNECT method. If there
* is hast header, its value is normalized. 0 is returned on success, -1 if the
* authority is invalid and -2 if the host is invalid.
*/
BUG/MEDIUM: h1: Reject CONNECT request if the target has a scheme The target of a CONNECT request must not have scheme. However, this was not checked during the message parsing. It is now rejected. This patch may be backported as far as 2.4.
2024-05-14 09:06:48 -04:00
static int h1_validate_connect_authority(struct ist scheme, struct ist authority, struct ist *host_hdr)
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
{
struct ist uri_host, uri_port, host, host_port;
BUG/MEDIUM: h1: Reject CONNECT request if the target has a scheme The target of a CONNECT request must not have scheme. However, this was not checked during the message parsing. It is now rejected. This patch may be backported as far as 2.4.
2024-05-14 09:06:48 -04:00
if (isttest(scheme) || !isttest(authority))
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
goto invalid_authority;
uri_host = authority;
uri_port = http_get_host_port(authority);
MINOR: h1: Consider empty port as invalid in authority for CONNECT For now, this change is useless because http_get_host_port() returns IST_NULL when the port is empty. But this will change. For other methods, empty ports are valid. But not for CONNECT method. To still return a 400-Bad-Request if a CONNECT is performed with an empty port, istlen() is used to test the port, instead of isttest().
2022-11-22 04:27:54 -05:00
if (!istlen(uri_port))
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
goto invalid_authority;
uri_host.len -= (istlen(uri_port) + 1);
if (!host_hdr || !isttest(*host_hdr))
goto end;
/* Get the port of the host header value, if any */
host = *host_hdr;
host_port = http_get_host_port(*host_hdr);
MINOR: h1: Consider empty port as invalid in authority for CONNECT For now, this change is useless because http_get_host_port() returns IST_NULL when the port is empty. But this will change. For other methods, empty ports are valid. But not for CONNECT method. To still return a 400-Bad-Request if a CONNECT is performed with an empty port, istlen() is used to test the port, instead of isttest().
2022-11-22 04:27:54 -05:00
if (isttest(host_port))
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
host.len -= (istlen(host_port) + 1);
MINOR: h1: Consider empty port as invalid in authority for CONNECT For now, this change is useless because http_get_host_port() returns IST_NULL when the port is empty. But this will change. For other methods, empty ports are valid. But not for CONNECT method. To still return a 400-Bad-Request if a CONNECT is performed with an empty port, istlen() is used to test the port, instead of isttest().
2022-11-22 04:27:54 -05:00
if (istlen(host_port)) {
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
if (!isteqi(host, uri_host) || !isteq(host_port, uri_port))
goto invalid_host;
if (http_is_default_port(IST_NULL, uri_port))
*host_hdr = host; /* normalize */
}
else {
if (!http_is_default_port(IST_NULL, uri_port) || !isteqi(host, uri_host))
goto invalid_host;
}
end:
return 0;
invalid_authority:
return -1;
invalid_host:
return -2;
}
BUG/MINOR: h1: Replace authority validation to conform RFC3986 Except for CONNECT method, where a normalization is performed, we expected to have an exact match between the authority and the host header value. However it was too strict. Indeed, default port must be handled and the matching must respect the RFC3986. There is already a scheme based normalizeation performed on the URI later, on the HTX message. And we cannot normalize the URI during H1 parsing to be able to report proper errors on the original raw buffer. And a systematic read-only normalization to validate the authority will consume CPU for only few requests. At the end, we decided to perform extra-checks when the exact match fails. Now, following authority/host are considered as equivalent: http: domain.com <=> domain.com:80 <=> domain.com: https: domain.com <=> domain.com:443 <=> domain.com: This patch depends on: * MINOR: h1: Consider empty port as invalid in authority for CONNECT * MINOR: http: Considere empty ports as valid default ports It is a bug regarding the RFC3986. Technically, I may be backported as far as 2.4. However, this must be discussed first. If it is backported, the commits above must be backported too.
2022-11-22 04:04:16 -05:00
/* Validate the authority and the host header value for non-CONNECT method, when
* an absolute-URI is detected but when it does not exactly match the host
* value. The idea is to detect default port (http or https). authority and host
* are defined here. 0 is returned on success, -1 if the host is does not match
* the authority.
*/
static int h1_validate_mismatch_authority(struct ist scheme, struct ist authority, struct ist host_hdr)
{
struct ist uri_host, uri_port, host, host_port;
if (!isttest(scheme))
goto mismatch;
uri_host = authority;
uri_port = http_get_host_port(authority);
if (isttest(uri_port))
uri_host.len -= (istlen(uri_port) + 1);
host = host_hdr;
host_port = http_get_host_port(host_hdr);
if (isttest(host_port))
host.len -= (istlen(host_port) + 1);
if (!isttest(uri_port) && !isttest(host_port)) {
/* No port on both: we already know the authority does not match
* the host value
*/
goto mismatch;
}
else if (isttest(uri_port) && !http_is_default_port(scheme, uri_port)) {
/* here there is no port for the host value and the port for the
* authority is not the default one
*/
goto mismatch;
}
else if (isttest(host_port) && !http_is_default_port(scheme, host_port)) {
/* here there is no port for the authority and the port for the
* host value is not the default one
*/
goto mismatch;
}
else {
/* the authority or the host value contain a default port and
* there is no port on the other value
*/
if (!isteqi(uri_host, host))
goto mismatch;
}
return 0;
mismatch:
return -1;
}
MINOR: h1: parse the Connection header field The new function h1_parse_connection_header() is called when facing a connection header in the generic parser, and it will set up to 3 bits in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade" tokens was seen.
2018-09-13 08:15:58 -04:00
/* Parse the Connection: header of an HTTP/1 request, looking for "close",
* "keep-alive", and "upgrade" values, and updating h1m->flags according to
* what was found there. Note that flags are only added, not removed, so the
* function is safe for being called multiple times if multiple occurrences
MEDIUM: h1: Add an option to sanitize connection headers during parsing The flag H1_MF_CLEAN_CONN_HDR has been added to let the H1 parser sanitize connection headers. It means it will remove all "close" and "keep-alive" values during the parsing. One noticeable effect is that connection headers may be unfolded. In practice, this is not a problem because it is not frequent to have multiple values for the connection headers. If this flag is set, during the parsing The function h1_parse_next_connection_header() is called in a loop instead of h1_parse_conection_header(). No need to backport this patch
2019-03-29 10:03:13 -04:00
* are found. If the flag H1_MF_CLEAN_CONN_HDR, the header value is cleaned
* up from "keep-alive" and "close" values. To do so, the header value is
* rewritten in place and its length is updated.
MINOR: h1: parse the Connection header field The new function h1_parse_connection_header() is called when facing a connection header in the generic parser, and it will set up to 3 bits in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade" tokens was seen.
2018-09-13 08:15:58 -04:00
*/
MEDIUM: h1: Add an option to sanitize connection headers during parsing The flag H1_MF_CLEAN_CONN_HDR has been added to let the H1 parser sanitize connection headers. It means it will remove all "close" and "keep-alive" values during the parsing. One noticeable effect is that connection headers may be unfolded. In practice, this is not a problem because it is not frequent to have multiple values for the connection headers. If this flag is set, during the parsing The function h1_parse_next_connection_header() is called in a loop instead of h1_parse_conection_header(). No need to backport this patch
2019-03-29 10:03:13 -04:00
void h1_parse_connection_header(struct h1m *h1m, struct ist *value)
MINOR: h1: parse the Connection header field The new function h1_parse_connection_header() is called when facing a connection header in the generic parser, and it will set up to 3 bits in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade" tokens was seen.
2018-09-13 08:15:58 -04:00
{
MEDIUM: h1: Add an option to sanitize connection headers during parsing The flag H1_MF_CLEAN_CONN_HDR has been added to let the H1 parser sanitize connection headers. It means it will remove all "close" and "keep-alive" values during the parsing. One noticeable effect is that connection headers may be unfolded. In practice, this is not a problem because it is not frequent to have multiple values for the connection headers. If this flag is set, during the parsing The function h1_parse_next_connection_header() is called in a loop instead of h1_parse_conection_header(). No need to backport this patch
2019-03-29 10:03:13 -04:00
char *e, *n, *p;
MINOR: h1: parse the Connection header field The new function h1_parse_connection_header() is called when facing a connection header in the generic parser, and it will set up to 3 bits in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade" tokens was seen.
2018-09-13 08:15:58 -04:00
struct ist word;
MEDIUM: h1: Add an option to sanitize connection headers during parsing The flag H1_MF_CLEAN_CONN_HDR has been added to let the H1 parser sanitize connection headers. It means it will remove all "close" and "keep-alive" values during the parsing. One noticeable effect is that connection headers may be unfolded. In practice, this is not a problem because it is not frequent to have multiple values for the connection headers. If this flag is set, during the parsing The function h1_parse_next_connection_header() is called in a loop instead of h1_parse_conection_header(). No need to backport this patch
2019-03-29 10:03:13 -04:00
word.ptr = value->ptr - 1; // -1 for next loop's pre-increment
p = value->ptr;
e = value->ptr + value->len;
if (h1m->flags & H1_MF_CLEAN_CONN_HDR)
value->len = 0;
MINOR: h1: parse the Connection header field The new function h1_parse_connection_header() is called when facing a connection header in the generic parser, and it will set up to 3 bits in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade" tokens was seen.
2018-09-13 08:15:58 -04:00
while (++word.ptr < e) {
CLEANUP: assorted typo fixes in the code and comments This is 10th iteration of typo fixes
2020-06-21 12:42:57 -04:00
/* skip leading delimiter and blanks */
MINOR: h1: parse the Connection header field The new function h1_parse_connection_header() is called when facing a connection header in the generic parser, and it will set up to 3 bits in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade" tokens was seen.
2018-09-13 08:15:58 -04:00
if (HTTP_IS_LWS(*word.ptr))
continue;
n = http_find_hdr_value_end(word.ptr, e); // next comma or end of line
word.len = n - word.ptr;
/* trim trailing blanks */
while (word.len && HTTP_IS_LWS(word.ptr[word.len-1]))
word.len--;
MEDIUM: h1: Add an option to sanitize connection headers during parsing The flag H1_MF_CLEAN_CONN_HDR has been added to let the H1 parser sanitize connection headers. It means it will remove all "close" and "keep-alive" values during the parsing. One noticeable effect is that connection headers may be unfolded. In practice, this is not a problem because it is not frequent to have multiple values for the connection headers. If this flag is set, during the parsing The function h1_parse_next_connection_header() is called in a loop instead of h1_parse_conection_header(). No need to backport this patch
2019-03-29 10:03:13 -04:00
if (isteqi(word, ist("keep-alive"))) {
MINOR: h1: parse the Connection header field The new function h1_parse_connection_header() is called when facing a connection header in the generic parser, and it will set up to 3 bits in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade" tokens was seen.
2018-09-13 08:15:58 -04:00
h1m->flags |= H1_MF_CONN_KAL;
MEDIUM: h1: Add an option to sanitize connection headers during parsing The flag H1_MF_CLEAN_CONN_HDR has been added to let the H1 parser sanitize connection headers. It means it will remove all "close" and "keep-alive" values during the parsing. One noticeable effect is that connection headers may be unfolded. In practice, this is not a problem because it is not frequent to have multiple values for the connection headers. If this flag is set, during the parsing The function h1_parse_next_connection_header() is called in a loop instead of h1_parse_conection_header(). No need to backport this patch
2019-03-29 10:03:13 -04:00
if (h1m->flags & H1_MF_CLEAN_CONN_HDR)
goto skip_val;
}
else if (isteqi(word, ist("close"))) {
MINOR: h1: parse the Connection header field The new function h1_parse_connection_header() is called when facing a connection header in the generic parser, and it will set up to 3 bits in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade" tokens was seen.
2018-09-13 08:15:58 -04:00
h1m->flags |= H1_MF_CONN_CLO;
MEDIUM: h1: Add an option to sanitize connection headers during parsing The flag H1_MF_CLEAN_CONN_HDR has been added to let the H1 parser sanitize connection headers. It means it will remove all "close" and "keep-alive" values during the parsing. One noticeable effect is that connection headers may be unfolded. In practice, this is not a problem because it is not frequent to have multiple values for the connection headers. If this flag is set, during the parsing The function h1_parse_next_connection_header() is called in a loop instead of h1_parse_conection_header(). No need to backport this patch
2019-03-29 10:03:13 -04:00
if (h1m->flags & H1_MF_CLEAN_CONN_HDR)
goto skip_val;
}
MINOR: h1: parse the Connection header field The new function h1_parse_connection_header() is called when facing a connection header in the generic parser, and it will set up to 3 bits in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade" tokens was seen.
2018-09-13 08:15:58 -04:00
else if (isteqi(word, ist("upgrade")))
h1m->flags |= H1_MF_CONN_UPG;
MEDIUM: h1: Add an option to sanitize connection headers during parsing The flag H1_MF_CLEAN_CONN_HDR has been added to let the H1 parser sanitize connection headers. It means it will remove all "close" and "keep-alive" values during the parsing. One noticeable effect is that connection headers may be unfolded. In practice, this is not a problem because it is not frequent to have multiple values for the connection headers. If this flag is set, during the parsing The function h1_parse_next_connection_header() is called in a loop instead of h1_parse_conection_header(). No need to backport this patch
2019-03-29 10:03:13 -04:00
if (h1m->flags & H1_MF_CLEAN_CONN_HDR) {
if (value->ptr + value->len == p) {
/* no rewrite done till now */
value->len = n - value->ptr;
}
else {
if (value->len)
value->ptr[value->len++] = ',';
istcat(value, word, e - value->ptr);
}
}
skip_val:
word.ptr = p = n;
MINOR: h1: parse the Connection header field The new function h1_parse_connection_header() is called when facing a connection header in the generic parser, and it will set up to 3 bits in h1m->flags indicating if at least one "close", "keep-alive" or "upgrade" tokens was seen.
2018-09-13 08:15:58 -04:00
}
}
MINOR: h1: reject websocket handshake if missing key If a request is identified as a WebSocket handshake, it must contains a websocket key header or else it can be reject, following the rfc6455. A new flag H1_MF_UPG_WEBSOCKET is set on such messages. For the request te be identified as a WebSocket handshake, it must contains the headers: Connection: upgrade Upgrade: websocket This commit is a compagnon of "MEDIUM: h1: generate WebSocket key on response if needed" and "MEDIUM: h1: add a WebSocket key on handshake if needed". Indeed, it ensures that a WebSocket key is added only from a http/2 side and not for a http/1 bogus peer.
2020-12-11 11:53:02 -05:00
/* Parse the Upgrade: header of an HTTP/1 request.
* If "websocket" is found, set H1_MF_UPG_WEBSOCKET flag
BUG/MINOR: h1: do not forward h2c upgrade header token haproxy supports tunnel establishment through HTTP Upgrade mechanism. Since the following commit, extended CONNECT is also supported for HTTP/2 both on frontend and backend side. commit 9bf957335e2c385b74901481f7a89c9565dfce53 MEDIUM: mux_h2: generate Extended CONNECT from htx upgrade As specified by HTTP/2 rfc, "h2c" can be used by an HTTP/1.1 client to request an upgrade to HTTP/2. In haproxy, this is not supported so it silently ignores this. However, Connection and Upgrade headers are forwarded as-is on the backend side. If using HTTP/1 on the backend side and the server supports this upgrade mechanism, haproxy won't be able to parse the HTTP response. If using HTTP/2, mux backend tries to incorrectly convert the request to an Extended CONNECT with h2c protocol, which may also prevent the response to be transmitted. To fix this, flag HTTP/1 request with "h2c" or "h2" token in an upgrade header. On converting the header list to HTX, the upgrade header is skipped if any of this token is present and the H1_MF_CONN_UPG flag is removed. This issue can easily be reproduced using curl --http2 argument to connect to an HTTP/1 frontend. This must be backported up to 2.4 after a period of observation.
2024-08-01 09:35:06 -04:00
* If "h2c" or "h2" found, set H1_MF_UPG_H2C flag.
MINOR: h1: reject websocket handshake if missing key If a request is identified as a WebSocket handshake, it must contains a websocket key header or else it can be reject, following the rfc6455. A new flag H1_MF_UPG_WEBSOCKET is set on such messages. For the request te be identified as a WebSocket handshake, it must contains the headers: Connection: upgrade Upgrade: websocket This commit is a compagnon of "MEDIUM: h1: generate WebSocket key on response if needed" and "MEDIUM: h1: add a WebSocket key on handshake if needed". Indeed, it ensures that a WebSocket key is added only from a http/2 side and not for a http/1 bogus peer.
2020-12-11 11:53:02 -05:00
*/
void h1_parse_upgrade_header(struct h1m *h1m, struct ist value)
{
char *e, *n;
struct ist word;
BUG/MINOR: h1: do not forward h2c upgrade header token haproxy supports tunnel establishment through HTTP Upgrade mechanism. Since the following commit, extended CONNECT is also supported for HTTP/2 both on frontend and backend side. commit 9bf957335e2c385b74901481f7a89c9565dfce53 MEDIUM: mux_h2: generate Extended CONNECT from htx upgrade As specified by HTTP/2 rfc, "h2c" can be used by an HTTP/1.1 client to request an upgrade to HTTP/2. In haproxy, this is not supported so it silently ignores this. However, Connection and Upgrade headers are forwarded as-is on the backend side. If using HTTP/1 on the backend side and the server supports this upgrade mechanism, haproxy won't be able to parse the HTTP response. If using HTTP/2, mux backend tries to incorrectly convert the request to an Extended CONNECT with h2c protocol, which may also prevent the response to be transmitted. To fix this, flag HTTP/1 request with "h2c" or "h2" token in an upgrade header. On converting the header list to HTX, the upgrade header is skipped if any of this token is present and the H1_MF_CONN_UPG flag is removed. This issue can easily be reproduced using curl --http2 argument to connect to an HTTP/1 frontend. This must be backported up to 2.4 after a period of observation.
2024-08-01 09:35:06 -04:00
h1m->flags &= ~(H1_MF_UPG_WEBSOCKET|H1_MF_UPG_H2C);
MINOR: h1: reject websocket handshake if missing key If a request is identified as a WebSocket handshake, it must contains a websocket key header or else it can be reject, following the rfc6455. A new flag H1_MF_UPG_WEBSOCKET is set on such messages. For the request te be identified as a WebSocket handshake, it must contains the headers: Connection: upgrade Upgrade: websocket This commit is a compagnon of "MEDIUM: h1: generate WebSocket key on response if needed" and "MEDIUM: h1: add a WebSocket key on handshake if needed". Indeed, it ensures that a WebSocket key is added only from a http/2 side and not for a http/1 bogus peer.
2020-12-11 11:53:02 -05:00
word.ptr = value.ptr - 1; // -1 for next loop's pre-increment
CLEANUP: Apply ist.cocci Make use of the new rules to use `istend()`.
2021-11-06 10:14:44 -04:00
e = istend(value);
MINOR: h1: reject websocket handshake if missing key If a request is identified as a WebSocket handshake, it must contains a websocket key header or else it can be reject, following the rfc6455. A new flag H1_MF_UPG_WEBSOCKET is set on such messages. For the request te be identified as a WebSocket handshake, it must contains the headers: Connection: upgrade Upgrade: websocket This commit is a compagnon of "MEDIUM: h1: generate WebSocket key on response if needed" and "MEDIUM: h1: add a WebSocket key on handshake if needed". Indeed, it ensures that a WebSocket key is added only from a http/2 side and not for a http/1 bogus peer.
2020-12-11 11:53:02 -05:00
while (++word.ptr < e) {
/* skip leading delimiter and blanks */
if (HTTP_IS_LWS(*word.ptr))
continue;
n = http_find_hdr_value_end(word.ptr, e); // next comma or end of line
word.len = n - word.ptr;
/* trim trailing blanks */
while (word.len && HTTP_IS_LWS(word.ptr[word.len-1]))
word.len--;
if (isteqi(word, ist("websocket")))
h1m->flags |= H1_MF_UPG_WEBSOCKET;
BUG/MINOR: h1: do not forward h2c upgrade header token haproxy supports tunnel establishment through HTTP Upgrade mechanism. Since the following commit, extended CONNECT is also supported for HTTP/2 both on frontend and backend side. commit 9bf957335e2c385b74901481f7a89c9565dfce53 MEDIUM: mux_h2: generate Extended CONNECT from htx upgrade As specified by HTTP/2 rfc, "h2c" can be used by an HTTP/1.1 client to request an upgrade to HTTP/2. In haproxy, this is not supported so it silently ignores this. However, Connection and Upgrade headers are forwarded as-is on the backend side. If using HTTP/1 on the backend side and the server supports this upgrade mechanism, haproxy won't be able to parse the HTTP response. If using HTTP/2, mux backend tries to incorrectly convert the request to an Extended CONNECT with h2c protocol, which may also prevent the response to be transmitted. To fix this, flag HTTP/1 request with "h2c" or "h2" token in an upgrade header. On converting the header list to HTX, the upgrade header is skipped if any of this token is present and the H1_MF_CONN_UPG flag is removed. This issue can easily be reproduced using curl --http2 argument to connect to an HTTP/1 frontend. This must be backported up to 2.4 after a period of observation.
2024-08-01 09:35:06 -04:00
else if (isteqi(word, ist("h2c")) || isteqi(word, ist("h2")))
h1m->flags |= H1_MF_UPG_H2C;
MINOR: h1: reject websocket handshake if missing key If a request is identified as a WebSocket handshake, it must contains a websocket key header or else it can be reject, following the rfc6455. A new flag H1_MF_UPG_WEBSOCKET is set on such messages. For the request te be identified as a WebSocket handshake, it must contains the headers: Connection: upgrade Upgrade: websocket This commit is a compagnon of "MEDIUM: h1: generate WebSocket key on response if needed" and "MEDIUM: h1: add a WebSocket key on handshake if needed". Indeed, it ensures that a WebSocket key is added only from a http/2 side and not for a http/1 bogus peer.
2020-12-11 11:53:02 -05:00
word.ptr = n;
}
}
REORG: h1: move legacy http functions to http_msg.c Now that h1 and legacy HTTP are two distinct things, there's no need to keep the legacy HTTP parsers in h1.c since they're only used by the legacy code in proto_http.c, and h1.h doesn't need to include hdr_idx anymore. This concerns the following functions : - http_parse_reqline(); - http_parse_stsline(); - http_msg_analyzer(); - http_forward_trailers(); All of these were moved to http_msg.c.
2018-12-11 04:59:20 -05:00
/* Macros used in the HTTP/1 parser, to check for the expected presence of
* certain bytes (ef: LF) or to skip to next byte and yield in case of failure.
*/
/* Expects to find an LF at <ptr>. If not, set <state> to <where> and jump to
* <bad>.
*/
#define EXPECT_LF_HERE(ptr, bad, state, where) \
do { \
if (unlikely(*(ptr) != '\n')) { \
state = (where); \
goto bad; \
} \
} while (0)
/* Increments pointer <ptr>, continues to label <more> if it's still below
* pointer <end>, or goes to <stop> and sets <state> to <where> if the end
* of buffer was reached.
*/
#define EAT_AND_JUMP_OR_RETURN(ptr, end, more, stop, state, where) \
do { \
if (likely(++(ptr) < (end))) \
goto more; \
else { \
state = (where); \
goto stop; \
} \
} while (0)
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
/* This function parses a contiguous HTTP/1 headers block starting at <start>
* and ending before <stop>, at once, and converts it a list of (name,value)
* pairs representing header fields into the array <hdr> of size <hdr_num>,
* whose last entry will have an empty name and an empty value. If <hdr_num> is
MEDIUM: h1: let the caller pass the initial parser's state This way the caller controls if it's the request or response which has to be used, and it will allow to restart after an incomplete parsing.
2018-09-11 09:33:32 -04:00
* too small to represent the whole message, an error is returned. Some
* protocol elements such as content-length and transfer-encoding will be
MINOR: h1: make the message parser support a null <hdr> argument This will allow some iterative calls to be made on incomplete messages without having to store all the headers.
2018-09-11 10:04:48 -04:00
* parsed and stored into h1m as well. <hdr> may be null, in which case only
* the parsing state will be updated. This may be used to restart the parsing
* where it stopped for example.
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
*
* For now it's limited to the response. If the header block is incomplete,
* 0 is returned, waiting to be called again with more data to try it again.
MEDIUM: h1: let the caller pass the initial parser's state This way the caller controls if it's the request or response which has to be used, and it will allow to restart after an incomplete parsing.
2018-09-11 09:33:32 -04:00
* The caller is responsible for initializing h1m->state to H1_MSG_RPBEFORE,
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
* and h1m->next to zero on the first call, the parser will do the rest. If
* an incomplete message is seen, the caller only needs to present h1m->state
* and h1m->next again, with an empty header list so that the parser can start
* again. In this case, it will detect that it interrupted a previous session
* and will first look for the end of the message before reparsing it again and
* indexing it at the same time. This ensures that incomplete messages fed 1
* character at a time are never processed entirely more than exactly twice,
* and that there is no need to store all the internal state and pre-parsed
* headers or start line between calls.
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
*
MEDIUM: h1: make the parser support a pointer to a start line This will allow the parser to fill some extra fields like the method or status without having to store them permanently in the HTTP message. At this point however the parser cannot restart from an interrupted read.
2018-09-11 09:34:50 -04:00
* A pointer to a start line descriptor may be passed in <slp>, in which case
* the parser will fill it with whatever it found.
*
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
* The code derived from the main HTTP/1 parser above but was simplified and
* optimized to process responses produced or forwarded by haproxy. The caller
* is responsible for ensuring that the message doesn't wrap, and should ensure
* it is complete to avoid having to retry the operation after a failed
* attempt. The message is not supposed to be invalid, which is why a few
* properties such as the character set used in the header field names are not
* checked. In case of an unparsable response message, a negative value will be
* returned with h1m->err_pos and h1m->err_state matching the location and
* state where the error was met. Leading blank likes are tolerated but not
MINOR: h1: make the H1 headers block parser able to parse headers only Currently the H1 headers parser works for either a request or a response because it starts from the start line. It is also able to resume its processing when it was interrupted, but in this case it doesn't update the list. Make it support a new flag, H1_MF_HDRS_ONLY so that the caller can indicate it's only interested in the headers list and not the start line. This will be convenient to parse H1 trailers.
2019-01-04 04:48:03 -05:00
* recommended. If flag H1_MF_HDRS_ONLY is set in h1m->flags, only headers are
* parsed and the start line is skipped. It is not required to set h1m->state
* nor h1m->next in this case.
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
*
* This function returns :
* -1 in case of error. In this case, h1m->err_state is filled (if h1m is
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
* set) with the state the error occurred in and h1m->err_pos with the
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
* the position relative to <start>
* -2 if the output is full (hdr_num reached). err_state and err_pos also
* indicate where it failed.
* 0 in case of missing data.
* > 0 on success, it then corresponds to the number of bytes read since
* <start> so that the caller can go on with the payload.
*/
int h1_headers_to_hdr_list(char *start, const char *stop,
struct http_hdr *hdr, unsigned int hdr_num,
MEDIUM: h1: make the parser support a pointer to a start line This will allow the parser to fill some extra fields like the method or status without having to store them permanently in the HTTP message. At this point however the parser cannot restart from an interrupted read.
2018-09-11 09:34:50 -04:00
struct h1m *h1m, union h1_sl *slp)
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
{
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
enum h1m_state state;
register char *ptr;
register const char *end;
unsigned int hdr_count;
unsigned int skip; /* number of bytes skipped at the beginning */
unsigned int sol; /* start of line */
unsigned int col; /* position of the colon */
unsigned int eol; /* end of line */
unsigned int sov; /* start of value */
MEDIUM: h1: make the parser support a pointer to a start line This will allow the parser to fill some extra fields like the method or status without having to store them permanently in the HTTP message. At this point however the parser cannot restart from an interrupted read.
2018-09-11 09:34:50 -04:00
union h1_sl sl;
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
int skip_update;
int restarting;
MINOR: h1: Reject requests with different occurrences of the header host There is no reason for a client to send several headers host. It even may be considered as a bug. However, it is totally invalid to have different values for those. So now, in such case, an error is triggered during the request parsing. In addition, when several headers host are found with the same value, only the first instance is kept and others are skipped.
2019-10-11 03:01:44 -04:00
int host_idx;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
struct ist n, v; /* header name and value during parsing */
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
skip = 0; // do it only once to keep track of the leading CRLF.
try_again:
hdr_count = sol = col = eol = sov = 0;
MEDIUM: h1: make the parser support a pointer to a start line This will allow the parser to fill some extra fields like the method or status without having to store them permanently in the HTTP message. At this point however the parser cannot restart from an interrupted read.
2018-09-11 09:34:50 -04:00
sl.st.status = 0;
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
skip_update = restarting = 0;
MINOR: h1: Reject requests with different occurrences of the header host There is no reason for a client to send several headers host. It even may be considered as a bug. However, it is totally invalid to have different values for those. So now, in such case, an error is triggered during the request parsing. In addition, when several headers host are found with the same value, only the first instance is kept and others are skipped.
2019-10-11 03:01:44 -04:00
host_idx = -1;
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
MINOR: h1: make the H1 headers block parser able to parse headers only Currently the H1 headers parser works for either a request or a response because it starts from the start line. It is also able to resume its processing when it was interrupted, but in this case it doesn't update the list. Make it support a new flag, H1_MF_HDRS_ONLY so that the caller can indicate it's only interested in the headers list and not the start line. This will be convenient to parse H1 trailers.
2019-01-04 04:48:03 -05:00
if (h1m->flags & H1_MF_HDRS_ONLY) {
state = H1_MSG_HDR_FIRST;
h1m->next = 0;
}
BUG/MEDIUM: h1: Get the h1m state when restarting the headers parsing Since the commit 0f8fb6b7f ("MINOR: h1: make the H1 headers block parser able to parse headers only"), when headers are not received in one time, a parsing error is returned because the local state in the function h1_headers_to_hdr_list() was not initialized with the previous one (in fact, it was not initialized at all). So now, we start the parsing of headers with the state H1_MSG_HDR_FIRST when the flag H1_MF_HDRS_ONLY is set. Otherwise, we always get it from the h1m. This patch must be backported to 1.9.
2019-01-04 10:06:48 -05:00
else {
MINOR: h1: make the H1 headers block parser able to parse headers only Currently the H1 headers parser works for either a request or a response because it starts from the start line. It is also able to resume its processing when it was interrupted, but in this case it doesn't update the list. Make it support a new flag, H1_MF_HDRS_ONLY so that the caller can indicate it's only interested in the headers list and not the start line. This will be convenient to parse H1 trailers.
2019-01-04 04:48:03 -05:00
state = h1m->state;
BUG/MEDIUM: h1: Get the h1m state when restarting the headers parsing Since the commit 0f8fb6b7f ("MINOR: h1: make the H1 headers block parser able to parse headers only"), when headers are not received in one time, a parsing error is returned because the local state in the function h1_headers_to_hdr_list() was not initialized with the previous one (in fact, it was not initialized at all). So now, we start the parsing of headers with the state H1_MSG_HDR_FIRST when the flag H1_MF_HDRS_ONLY is set. Otherwise, we always get it from the h1m. This patch must be backported to 1.9.
2019-01-04 10:06:48 -05:00
if (h1m->state != H1_MSG_RQBEFORE && h1m->state != H1_MSG_RPBEFORE)
restarting = 1;
}
MINOR: h1: make the H1 headers block parser able to parse headers only Currently the H1 headers parser works for either a request or a response because it starts from the start line. It is also able to resume its processing when it was interrupted, but in this case it doesn't update the list. Make it support a new flag, H1_MF_HDRS_ONLY so that the caller can indicate it's only interested in the headers list and not the start line. This will be convenient to parse H1 trailers.
2019-01-04 04:48:03 -05:00
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
ptr = start + h1m->next;
end = stop;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
if (unlikely(ptr >= end))
goto http_msg_ood;
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
/* don't update output if hdr is NULL or if we're restarting */
if (!hdr || restarting)
MINOR: h1: make the message parser support a null <hdr> argument This will allow some iterative calls to be made on incomplete messages without having to store all the headers.
2018-09-11 10:04:48 -04:00
skip_update = 1;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
switch (state) {
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
case H1_MSG_RQBEFORE:
http_msg_rqbefore:
if (likely(HTTP_IS_TOKEN(*ptr))) {
/* we have a start of message, we may have skipped some
* heading CRLF. Skip them now.
*/
skip += ptr - start;
start = ptr;
sol = 0;
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.rq.m.ptr = ptr;
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
hdr_count = 0;
state = H1_MSG_RQMETH;
goto http_msg_rqmeth;
}
if (unlikely(!HTTP_IS_CRLF(*ptr))) {
state = H1_MSG_RQBEFORE;
goto http_msg_invalid;
}
if (unlikely(*ptr == '\n'))
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rqbefore, http_msg_ood, state, H1_MSG_RQBEFORE);
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rqbefore_cr, http_msg_ood, state, H1_MSG_RQBEFORE_CR);
/* stop here */
case H1_MSG_RQBEFORE_CR:
http_msg_rqbefore_cr:
EXPECT_LF_HERE(ptr, http_msg_invalid, state, H1_MSG_RQBEFORE_CR);
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rqbefore, http_msg_ood, state, H1_MSG_RQBEFORE);
/* stop here */
case H1_MSG_RQMETH:
http_msg_rqmeth:
if (likely(HTTP_IS_TOKEN(*ptr)))
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rqmeth, http_msg_ood, state, H1_MSG_RQMETH);
if (likely(HTTP_IS_SPHT(*ptr))) {
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.rq.m.len = ptr - sl.rq.m.ptr;
sl.rq.meth = find_http_meth(start, sl.rq.m.len);
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rqmeth_sp, http_msg_ood, state, H1_MSG_RQMETH_SP);
}
if (likely(HTTP_IS_CRLF(*ptr))) {
/* HTTP 0.9 request */
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.rq.m.len = ptr - sl.rq.m.ptr;
sl.rq.meth = find_http_meth(sl.rq.m.ptr, sl.rq.m.len);
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
http_msg_req09_uri:
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.rq.u.ptr = ptr;
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
http_msg_req09_uri_e:
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.rq.u.len = ptr - sl.rq.u.ptr;
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
http_msg_req09_ver:
CLEANUP: Reapply ist.cocci with `--include-headers-for-types --recursive-includes` Previous uses of `ist.cocci` did not add `--include-headers-for-types` and `--recursive-includes` preventing Coccinelle seeing `struct ist` members of other structs. Reapply the patch with proper flags to further clean up the use of the ist API. The command used was: spatch -sp_file dev/coccinelle/ist.cocci -in_place --include-headers --include-headers-for-types --recursive-includes --dir src/
2022-03-15 08:11:06 -04:00
sl.rq.v = ist2(ptr, 0);
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
goto http_msg_rqline_eol;
}
state = H1_MSG_RQMETH;
goto http_msg_invalid;
case H1_MSG_RQMETH_SP:
http_msg_rqmeth_sp:
if (likely(!HTTP_IS_LWS(*ptr))) {
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.rq.u.ptr = ptr;
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
goto http_msg_rquri;
}
if (likely(HTTP_IS_SPHT(*ptr)))
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rqmeth_sp, http_msg_ood, state, H1_MSG_RQMETH_SP);
/* so it's a CR/LF, meaning an HTTP 0.9 request */
goto http_msg_req09_uri;
case H1_MSG_RQURI:
http_msg_rquri:
CLEANUP: http/h1: rely on HA_UNALIGNED_LE instead of checking for CPU families Now that we have flags indicating the CPU's capabilities, better use them instead of missing some updates for new CPU families (ARMv8 was missing there).
2020-02-21 10:31:22 -05:00
#ifdef HA_UNALIGNED_LE
BUG/MINOR: h1: do not accept '#' as part of the URI component Seth Manesse and Paul Plasil reported that the "path" sample fetch function incorrectly accepts '#' as part of the path component. This can in some cases lead to misrouted requests for rules that would apply on the suffix: use_backend static if { path_end .png .jpg .gif .css .js } Note that this behavior can be selectively configured using "normalize-uri fragment-encode" and "normalize-uri fragment-strip". The problem is that while the RFC says that this '#' must never be emitted, as often it doesn't suggest how servers should handle it. A diminishing number of servers still do accept it and trim it silently, while others are rejecting it, as indicated in the conversation below with other implementers: https://lists.w3.org/Archives/Public/ietf-http-wg/2023JulSep/0070.html Looking at logs from publicly exposed servers, such requests appear at a rate of roughly 1 per million and only come from attacks or poorly written web crawlers incorrectly following links found on various pages. Thus it looks like the best solution to this problem is to simply reject such ambiguous requests by default, and include this in the list of controls that can be disabled using "option accept-invalid-http-request". We're already rejecting URIs containing any control char anyway, so we should also reject '#'. In the H1 parser for the H1_MSG_RQURI state, there is an accelerated parser for bytes 0x21..0x7e that has been tightened to 0x24..0x7e (it should not impact perf since 0x21..0x23 are not supposed to appear in a URI anyway). This way '#' falls through the fine-grained filter and we can add the special case for it also conditionned by a check on the proxy's option "accept-invalid-http-request", with no overhead for the vast majority of valid URIs. Here this information is available through h1m->err_pos that's set to -2 when the option is here (so we don't need to change the API to expose the proxy). Example with a trivial GET through netcat: [08/Aug/2023:16:16:52.651] frontend layer1 (#2): invalid request backend <NONE> (#-1), server <NONE> (#-1), event #0, src 127.0.0.1:50812 buffer starts at 0 (including 0 out), 16361 free, len 23, wraps at 16336, error at position 7 H1 connection flags 0x00000000, H1 stream flags 0x00000810 H1 msg state MSG_RQURI(4), H1 msg flags 0x00001400 H1 chunk len 0 bytes, H1 body len 0 bytes : 00000 GET /aa#bb HTTP/1.0\r\n 00021 \r\n This should be progressively backported to all stable versions along with the following patch: REGTESTS: http-rules: add accept-invalid-http-request for normalize-uri tests Similar fixes for h2 and h3 will come in followup patches. Thanks to Seth Manesse and Paul Plasil for reporting this problem with detailed explanations.
2023-08-08 10:17:22 -04:00
/* speedup: skip bytes not between 0x24 and 0x7e inclusive */
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
while (ptr <= end - sizeof(int)) {
CLEANUP: h1: make use of the multi-byte matching functions Instead of leaving the hard-coded non-trivial operations in the H1 parsing code, let's just rely on the new intops functions that do the same and that are less prone to being accidentally touched. It was verified that the resulting code is exactly the same.
2024-04-24 09:57:15 -04:00
if (is_char4_outside(*(uint *)ptr, 0x24, 0x7e))
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
break;
ptr += sizeof(int);
}
#endif
if (ptr >= end) {
state = H1_MSG_RQURI;
goto http_msg_ood;
}
http_msg_rquri2:
BUG/MINOR: h1: do not accept '#' as part of the URI component Seth Manesse and Paul Plasil reported that the "path" sample fetch function incorrectly accepts '#' as part of the path component. This can in some cases lead to misrouted requests for rules that would apply on the suffix: use_backend static if { path_end .png .jpg .gif .css .js } Note that this behavior can be selectively configured using "normalize-uri fragment-encode" and "normalize-uri fragment-strip". The problem is that while the RFC says that this '#' must never be emitted, as often it doesn't suggest how servers should handle it. A diminishing number of servers still do accept it and trim it silently, while others are rejecting it, as indicated in the conversation below with other implementers: https://lists.w3.org/Archives/Public/ietf-http-wg/2023JulSep/0070.html Looking at logs from publicly exposed servers, such requests appear at a rate of roughly 1 per million and only come from attacks or poorly written web crawlers incorrectly following links found on various pages. Thus it looks like the best solution to this problem is to simply reject such ambiguous requests by default, and include this in the list of controls that can be disabled using "option accept-invalid-http-request". We're already rejecting URIs containing any control char anyway, so we should also reject '#'. In the H1 parser for the H1_MSG_RQURI state, there is an accelerated parser for bytes 0x21..0x7e that has been tightened to 0x24..0x7e (it should not impact perf since 0x21..0x23 are not supposed to appear in a URI anyway). This way '#' falls through the fine-grained filter and we can add the special case for it also conditionned by a check on the proxy's option "accept-invalid-http-request", with no overhead for the vast majority of valid URIs. Here this information is available through h1m->err_pos that's set to -2 when the option is here (so we don't need to change the API to expose the proxy). Example with a trivial GET through netcat: [08/Aug/2023:16:16:52.651] frontend layer1 (#2): invalid request backend <NONE> (#-1), server <NONE> (#-1), event #0, src 127.0.0.1:50812 buffer starts at 0 (including 0 out), 16361 free, len 23, wraps at 16336, error at position 7 H1 connection flags 0x00000000, H1 stream flags 0x00000810 H1 msg state MSG_RQURI(4), H1 msg flags 0x00001400 H1 chunk len 0 bytes, H1 body len 0 bytes : 00000 GET /aa#bb HTTP/1.0\r\n 00021 \r\n This should be progressively backported to all stable versions along with the following patch: REGTESTS: http-rules: add accept-invalid-http-request for normalize-uri tests Similar fixes for h2 and h3 will come in followup patches. Thanks to Seth Manesse and Paul Plasil for reporting this problem with detailed explanations.
2023-08-08 10:17:22 -04:00
if (likely((unsigned char)(*ptr - 33) <= 93)) { /* 33 to 126 included */
if (*ptr == '#') {
if (h1m->err_pos < -1) /* PR_O2_REQBUG_OK not set */
goto invalid_char;
if (h1m->err_pos == -1) /* PR_O2_REQBUG_OK set: just log */
h1m->err_pos = ptr - start + skip;
}
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rquri2, http_msg_ood, state, H1_MSG_RQURI);
BUG/MINOR: h1: do not accept '#' as part of the URI component Seth Manesse and Paul Plasil reported that the "path" sample fetch function incorrectly accepts '#' as part of the path component. This can in some cases lead to misrouted requests for rules that would apply on the suffix: use_backend static if { path_end .png .jpg .gif .css .js } Note that this behavior can be selectively configured using "normalize-uri fragment-encode" and "normalize-uri fragment-strip". The problem is that while the RFC says that this '#' must never be emitted, as often it doesn't suggest how servers should handle it. A diminishing number of servers still do accept it and trim it silently, while others are rejecting it, as indicated in the conversation below with other implementers: https://lists.w3.org/Archives/Public/ietf-http-wg/2023JulSep/0070.html Looking at logs from publicly exposed servers, such requests appear at a rate of roughly 1 per million and only come from attacks or poorly written web crawlers incorrectly following links found on various pages. Thus it looks like the best solution to this problem is to simply reject such ambiguous requests by default, and include this in the list of controls that can be disabled using "option accept-invalid-http-request". We're already rejecting URIs containing any control char anyway, so we should also reject '#'. In the H1 parser for the H1_MSG_RQURI state, there is an accelerated parser for bytes 0x21..0x7e that has been tightened to 0x24..0x7e (it should not impact perf since 0x21..0x23 are not supposed to appear in a URI anyway). This way '#' falls through the fine-grained filter and we can add the special case for it also conditionned by a check on the proxy's option "accept-invalid-http-request", with no overhead for the vast majority of valid URIs. Here this information is available through h1m->err_pos that's set to -2 when the option is here (so we don't need to change the API to expose the proxy). Example with a trivial GET through netcat: [08/Aug/2023:16:16:52.651] frontend layer1 (#2): invalid request backend <NONE> (#-1), server <NONE> (#-1), event #0, src 127.0.0.1:50812 buffer starts at 0 (including 0 out), 16361 free, len 23, wraps at 16336, error at position 7 H1 connection flags 0x00000000, H1 stream flags 0x00000810 H1 msg state MSG_RQURI(4), H1 msg flags 0x00001400 H1 chunk len 0 bytes, H1 body len 0 bytes : 00000 GET /aa#bb HTTP/1.0\r\n 00021 \r\n This should be progressively backported to all stable versions along with the following patch: REGTESTS: http-rules: add accept-invalid-http-request for normalize-uri tests Similar fixes for h2 and h3 will come in followup patches. Thanks to Seth Manesse and Paul Plasil for reporting this problem with detailed explanations.
2023-08-08 10:17:22 -04:00
}
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
if (likely(HTTP_IS_SPHT(*ptr))) {
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.rq.u.len = ptr - sl.rq.u.ptr;
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rquri_sp, http_msg_ood, state, H1_MSG_RQURI_SP);
}
if (likely((unsigned char)*ptr >= 128)) {
/* non-ASCII chars are forbidden unless option
MINOR: proxy: Rename accept-invalid-http-* options With these options, it is possible to accept some invalid messages that may considered as unsafe and may result as vulnerabilities. The naming is not explicit enough on this point. These option must really be considered as dangerous and only used as a temporary workaround. Unfortunately, when used, it is probably because there are some legacy and unsupported applications in place. Nevermind. The documentation warns about the use of these options. Now the name of the options itself is a warning. So now, "accept-invalid-http-request" and "accept-invalid-http-response" options are deprecated and replaced by "accept-unsafe-violations-in-http-request" and "accept-unsafe-violations-in-http-response" options.
2024-09-12 03:33:32 -04:00
* accept-unsafe-violations-in-http-request is enabled in the frontend.
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
* In any case, we capture the faulty char.
*/
if (h1m->err_pos < -1)
goto invalid_char;
if (h1m->err_pos == -1)
h1m->err_pos = ptr - start + skip;
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rquri, http_msg_ood, state, H1_MSG_RQURI);
}
if (likely(HTTP_IS_CRLF(*ptr))) {
/* so it's a CR/LF, meaning an HTTP 0.9 request */
goto http_msg_req09_uri_e;
}
/* OK forbidden chars, 0..31 or 127 */
invalid_char:
state = H1_MSG_RQURI;
goto http_msg_invalid;
case H1_MSG_RQURI_SP:
http_msg_rquri_sp:
if (likely(!HTTP_IS_LWS(*ptr))) {
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.rq.v.ptr = ptr;
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
goto http_msg_rqver;
}
if (likely(HTTP_IS_SPHT(*ptr)))
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rquri_sp, http_msg_ood, state, H1_MSG_RQURI_SP);
/* so it's a CR/LF, meaning an HTTP 0.9 request */
goto http_msg_req09_ver;
case H1_MSG_RQVER:
http_msg_rqver:
if (likely(HTTP_IS_VER_TOKEN(*ptr)))
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rqver, http_msg_ood, state, H1_MSG_RQVER);
if (likely(HTTP_IS_CRLF(*ptr))) {
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.rq.v.len = ptr - sl.rq.v.ptr;
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
http_msg_rqline_eol:
/* We have seen the end of line. Note that we do not
* necessarily have the \n yet, but at least we know that we
* have EITHER \r OR \n, otherwise the request would not be
* complete. We can then record the request length and return
* to the caller which will be able to register it.
*/
if (likely(!skip_update)) {
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
if ((sl.rq.v.len == 8) &&
(*(sl.rq.v.ptr + 5) > '1' ||
(*(sl.rq.v.ptr + 5) == '1' && *(sl.rq.v.ptr + 7) >= '1')))
MINOR: h1: report in the h1m struct if the HTTP version is 1.1 or above This will be needed for the mux to know how to process the Connection header, and will save it from having to re-parse the request line since it's captured on the fly.
2018-09-13 05:32:51 -04:00
h1m->flags |= H1_MF_VER_11;
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
if (unlikely(hdr_count >= hdr_num)) {
state = H1_MSG_RQVER;
goto http_output_full;
}
MINOR: h1: Add the flag H1_MF_NO_PHDR to not add pseudo-headers during parsing Some pseudo-headers are added during the headers parsing, mainly for the mux H2. With this flag, it is possible to not add them. This avoid some boring filtering in the mux H1.
2018-10-08 09:50:15 -04:00
if (!(h1m->flags & H1_MF_NO_PHDR))
http_set_hdr(&hdr[hdr_count++], ist(":method"), sl.rq.m);
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
if (unlikely(hdr_count >= hdr_num)) {
state = H1_MSG_RQVER;
goto http_output_full;
}
MINOR: h1: Add the flag H1_MF_NO_PHDR to not add pseudo-headers during parsing Some pseudo-headers are added during the headers parsing, mainly for the mux H2. With this flag, it is possible to not add them. This avoid some boring filtering in the mux H1.
2018-10-08 09:50:15 -04:00
if (!(h1m->flags & H1_MF_NO_PHDR))
http_set_hdr(&hdr[hdr_count++], ist(":path"), sl.rq.u);
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
}
sol = ptr - start;
if (likely(*ptr == '\r'))
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rqline_end, http_msg_ood, state, H1_MSG_RQLINE_END);
goto http_msg_rqline_end;
}
/* neither an HTTP_VER token nor a CRLF */
state = H1_MSG_RQVER;
goto http_msg_invalid;
case H1_MSG_RQLINE_END:
http_msg_rqline_end:
/* check for HTTP/0.9 request : no version information
* available. sol must point to the first of CR or LF. However
* since we don't save these elements between calls, if we come
* here from a restart, we don't necessarily know. Thus in this
* case we simply start over.
*/
if (restarting)
goto restart;
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
if (unlikely(sl.rq.v.len == 0))
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
goto http_msg_last_lf;
EXPECT_LF_HERE(ptr, http_msg_invalid, state, H1_MSG_RQLINE_END);
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_first, http_msg_ood, state, H1_MSG_HDR_FIRST);
/* stop here */
/*
* Common states below
*/
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_RPBEFORE:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_rpbefore:
if (likely(HTTP_IS_TOKEN(*ptr))) {
/* we have a start of message, we may have skipped some
* heading CRLF. Skip them now.
*/
skip += ptr - start;
start = ptr;
sol = 0;
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.st.v.ptr = ptr;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
hdr_count = 0;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
state = H1_MSG_RPVER;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_rpver;
}
if (unlikely(!HTTP_IS_CRLF(*ptr))) {
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
state = H1_MSG_RPBEFORE;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_invalid;
}
if (unlikely(*ptr == '\n'))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpbefore, http_msg_ood, state, H1_MSG_RPBEFORE);
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpbefore_cr, http_msg_ood, state, H1_MSG_RPBEFORE_CR);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
/* stop here */
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_RPBEFORE_CR:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_rpbefore_cr:
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EXPECT_LF_HERE(ptr, http_msg_invalid, state, H1_MSG_RPBEFORE_CR);
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpbefore, http_msg_ood, state, H1_MSG_RPBEFORE);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
/* stop here */
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_RPVER:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_rpver:
if (likely(HTTP_IS_VER_TOKEN(*ptr)))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpver, http_msg_ood, state, H1_MSG_RPVER);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
if (likely(HTTP_IS_SPHT(*ptr))) {
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.st.v.len = ptr - sl.st.v.ptr;
MINOR: h1: report in the h1m struct if the HTTP version is 1.1 or above This will be needed for the mux to know how to process the Connection header, and will save it from having to re-parse the request line since it's captured on the fly.
2018-09-13 05:32:51 -04:00
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
if ((sl.st.v.len == 8) &&
(*(sl.st.v.ptr + 5) > '1' ||
(*(sl.st.v.ptr + 5) == '1' && *(sl.st.v.ptr + 7) >= '1')))
MINOR: h1: report in the h1m struct if the HTTP version is 1.1 or above This will be needed for the mux to know how to process the Connection header, and will save it from having to re-parse the request line since it's captured on the fly.
2018-09-13 05:32:51 -04:00
h1m->flags |= H1_MF_VER_11;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpver_sp, http_msg_ood, state, H1_MSG_RPVER_SP);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
}
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
state = H1_MSG_RPVER;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_invalid;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_RPVER_SP:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_rpver_sp:
if (likely(!HTTP_IS_LWS(*ptr))) {
MEDIUM: h1: make the parser support a pointer to a start line This will allow the parser to fill some extra fields like the method or status without having to store them permanently in the HTTP message. At this point however the parser cannot restart from an interrupted read.
2018-09-11 09:34:50 -04:00
sl.st.status = 0;
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.st.c.ptr = ptr;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_rpcode;
}
if (likely(HTTP_IS_SPHT(*ptr)))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpver_sp, http_msg_ood, state, H1_MSG_RPVER_SP);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
/* so it's a CR/LF, this is invalid */
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
state = H1_MSG_RPVER_SP;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_invalid;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_RPCODE:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_rpcode:
BUG/MINOR: h1: the HTTP/1 make status code parser check for digits The H1 parser used by the H2 gateway was a bit lax and could validate non-numbers in the status code. Since it computes the code on the fly it's problematic, as "30:" is read as status code 310. Let's properly check that it's a number now. No backport needed.
2017-11-09 05:15:45 -05:00
if (likely(HTTP_IS_DIGIT(*ptr))) {
MEDIUM: h1: make the parser support a pointer to a start line This will allow the parser to fill some extra fields like the method or status without having to store them permanently in the HTTP message. At this point however the parser cannot restart from an interrupted read.
2018-09-11 09:34:50 -04:00
sl.st.status = sl.st.status * 10 + *ptr - '0';
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpcode, http_msg_ood, state, H1_MSG_RPCODE);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
}
BUG/MINOR: h1: the HTTP/1 make status code parser check for digits The H1 parser used by the H2 gateway was a bit lax and could validate non-numbers in the status code. Since it computes the code on the fly it's problematic, as "30:" is read as status code 310. Let's properly check that it's a number now. No backport needed.
2017-11-09 05:15:45 -05:00
if (unlikely(!HTTP_IS_LWS(*ptr))) {
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
state = H1_MSG_RPCODE;
BUG/MINOR: h1: the HTTP/1 make status code parser check for digits The H1 parser used by the H2 gateway was a bit lax and could validate non-numbers in the status code. Since it computes the code on the fly it's problematic, as "30:" is read as status code 310. Let's properly check that it's a number now. No backport needed.
2017-11-09 05:15:45 -05:00
goto http_msg_invalid;
}
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
if (likely(HTTP_IS_SPHT(*ptr))) {
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.st.c.len = ptr - sl.st.c.ptr;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpcode_sp, http_msg_ood, state, H1_MSG_RPCODE_SP);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
}
/* so it's a CR/LF, so there is no reason phrase */
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.st.c.len = ptr - sl.st.c.ptr;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_rsp_reason:
CLEANUP: Reapply ist.cocci with `--include-headers-for-types --recursive-includes` Previous uses of `ist.cocci` did not add `--include-headers-for-types` and `--recursive-includes` preventing Coccinelle seeing `struct ist` members of other structs. Reapply the patch with proper flags to further clean up the use of the ist API. The command used was: spatch -sp_file dev/coccinelle/ist.cocci -in_place --include-headers --include-headers-for-types --recursive-includes --dir src/
2022-03-15 08:11:06 -04:00
sl.st.r = ist2(ptr, 0);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_rpline_eol;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_RPCODE_SP:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_rpcode_sp:
if (likely(!HTTP_IS_LWS(*ptr))) {
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.st.r.ptr = ptr;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_rpreason;
}
if (likely(HTTP_IS_SPHT(*ptr)))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpcode_sp, http_msg_ood, state, H1_MSG_RPCODE_SP);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
/* so it's a CR/LF, so there is no reason phrase */
goto http_msg_rsp_reason;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_RPREASON:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_rpreason:
if (likely(!HTTP_IS_CRLF(*ptr)))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpreason, http_msg_ood, state, H1_MSG_RPREASON);
MINOR: h1: Change the union h1_sl to use indirect strings to store infos Instead of using offsets relating to the parsed buffer to store start line infos, we now use indirect strings. So now, these infos remain valid only if the origin buffer remains untouched. But it's not a real problem because this union is used during the parsing and never stored to a later use.
2018-10-08 09:34:02 -04:00
sl.st.r.len = ptr - sl.st.r.ptr;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_rpline_eol:
/* We have seen the end of line. Note that we do not
* necessarily have the \n yet, but at least we know that we
* have EITHER \r OR \n, otherwise the response would not be
* complete. We can then record the response length and return
* to the caller which will be able to register it.
*/
MINOR: h1: make the message parser support a null <hdr> argument This will allow some iterative calls to be made on incomplete messages without having to store all the headers.
2018-09-11 10:04:48 -04:00
if (likely(!skip_update)) {
if (unlikely(hdr_count >= hdr_num)) {
state = H1_MSG_RPREASON;
goto http_output_full;
}
MINOR: h1: Add the flag H1_MF_NO_PHDR to not add pseudo-headers during parsing Some pseudo-headers are added during the headers parsing, mainly for the mux H2. With this flag, it is possible to not add them. This avoid some boring filtering in the mux H1.
2018-10-08 09:50:15 -04:00
if (!(h1m->flags & H1_MF_NO_PHDR))
http_set_hdr(&hdr[hdr_count++], ist(":status"), sl.st.c);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
}
sol = ptr - start;
if (likely(*ptr == '\r'))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_rpline_end, http_msg_ood, state, H1_MSG_RPLINE_END);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_rpline_end;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_RPLINE_END:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_rpline_end:
/* sol must point to the first of CR or LF. */
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EXPECT_LF_HERE(ptr, http_msg_invalid, state, H1_MSG_RPLINE_END);
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_first, http_msg_ood, state, H1_MSG_HDR_FIRST);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
/* stop here */
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_HDR_FIRST:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_hdr_first:
sol = ptr - start;
if (likely(!HTTP_IS_CRLF(*ptr))) {
goto http_msg_hdr_name;
}
if (likely(*ptr == '\r'))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_last_lf, http_msg_ood, state, H1_MSG_LAST_LF);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_last_lf;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_HDR_NAME:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_hdr_name:
/* assumes sol points to the first char */
if (likely(HTTP_IS_TOKEN(*ptr))) {
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
if (!skip_update) {
/* turn it to lower case if needed */
if (isupper((unsigned char)*ptr) && h1m->flags & H1_MF_TOLOWER)
BUILD: tree-wide: cast arguments to tolower/toupper to unsigned char NetBSD apparently uses macros for tolower/toupper and complains about the use of char for array subscripts. Let's properly cast all of them to unsigned char where they are used. This is needed to fix issue #729.
2020-07-05 15:46:32 -04:00
*ptr = tolower((unsigned char)*ptr);
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
}
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_name, http_msg_ood, state, H1_MSG_HDR_NAME);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
}
if (likely(*ptr == ':')) {
col = ptr - start;
BUG/CRITICAL: http: properly reject empty http header field names The HTTP header parsers surprizingly accepts empty header field names, and this is a leftover from the original code that was agnostic to this. When muxes were introduced, for H2 first, the HPACK decompressor needed to feed headers lists, and since empty header names were strictly forbidden by the protocol, the lists of headers were purposely designed to be terminated by an empty header field name (a principle that is similar to H1's empty line termination). This principle was preserved and generalized to other protocols migrated to muxes (H1/FCGI/H3 etc) without anyone ever noticing that the H1 parser was still able to deliver empty header field names to this list. In addition to this it turns out that the HPACK decompressor, despite a comment in the code, may successfully decompress an empty header field name, and this mistake was propagated to the QPACK decompressor as well. The impact is that an empty header field name may be used to truncate the list of headers and thus make some headers disappear. While for H2/H3 the impact is limited as haproxy sees a request with missing headers, and headers are not used to delimit messages, in the case of HTTP/1, the impact is significant because the presence (and sometimes contents) of certain sensitive headers is detected during the parsing. Thus, some of these headers may be seen, marked as present, their value extracted, but never delivered to upper layers and obviously not forwarded to the other side either. This can have for consequence that certain important header fields such as Connection, Upgrade, Host, Content-length, Transfer-Encoding etc are possibly seen as different between what haproxy uses to parse/forward/route and what is observed in http-request rules and of course, forwarded. One direct consequence is that it is possible to exploit this property in HTTP/1 to make affected versions of haproxy forward more data than is advertised on the other side, and bypass some access controls or routing rules by crafting extraneous requests. Note, however, that responses to such requests will normally not be passed back to the client, but this can still cause some harm. This specific risk can be mostly worked around in configuration using the following rule that will rely on the bug's impact to precisely detect the inconsistency between the known body size and the one expected to be advertised to the server (the rule works from 2.0 to 2.8-dev): http-request deny if { fc_http_major 1 } !{ req.body_size 0 } !{ req.hdr(content-length) -m found } !{ req.hdr(transfer-encoding) -m found } !{ method CONNECT } This will exclusively block such carefully crafted requests delivered over HTTP/1. HTTP/2 and HTTP/3 do not need content-length, and a body that arrives without being announced with a content-length will be forwarded using transfer-encoding, hence will not cause discrepancies. In HAProxy 2.0 in legacy mode ("no option http-use-htx"), this rule will simply have no effect but will not cause trouble either. A clean solution would consist in modifying the loops iterating over these headers lists to check the header name's pointer instead of its length (since both are zero at the end of the list), but this requires to touch tens of places and it's very easy to miss one. Functions such as htx_add_header(), htx_add_trailer(), htx_add_all_headers() would be good starting points for such a possible future change. Instead the current fix focuses on blocking empty headers where they are first inserted, hence in the H1/HPACK/QPACK decoders. One benefit of the current solution (for H1) is that it allows "show errors" to report a precise diagnostic when facing such invalid HTTP/1 requests, with the exact location of the problem and the originating address: $ printf "GET / HTTP/1.1\r\nHost: localhost\r\n:empty header\r\n\r\n" | nc 0 8001 HTTP/1.1 400 Bad request Content-length: 90 Cache-Control: no-cache Connection: close Content-Type: text/html <html><body><h1>400 Bad request</h1> Your browser sent an invalid request. </body></html> $ socat /var/run/haproxy.stat <<< "show errors" Total events captured on [10/Feb/2023:16:29:37.530] : 1 [10/Feb/2023:16:29:34.155] frontend decrypt (#2): invalid request backend <NONE> (#-1), server <NONE> (#-1), event #0, src 127.0.0.1:31092 buffer starts at 0 (including 0 out), 16334 free, len 50, wraps at 16336, error at position 33 H1 connection flags 0x00000000, H1 stream flags 0x00000810 H1 msg state MSG_HDR_NAME(17), H1 msg flags 0x00001410 H1 chunk len 0 bytes, H1 body len 0 bytes : 00000 GET / HTTP/1.1\r\n 00016 Host: localhost\r\n 00033 :empty header\r\n 00048 \r\n I want to address sincere and warm thanks for their great work to the team composed of the following security researchers who found the issue together and reported it: Bahruz Jabiyev, Anthony Gavazzi, and Engin Kirda from Northeastern University, Kaan Onarlioglu from Akamai Technologies, Adi Peleg and Harvey Tuch from Google. And kudos to Amaury Denoyelle from HAProxy Technologies for spotting that the HPACK and QPACK decoders would let this pass despite the comment explicitly saying otherwise. This fix must be backported as far as 2.0. The QPACK changes can be dropped before 2.6. In 2.0 there is also the equivalent code for legacy mode, which doesn't suffer from the list truncation, but it would better be fixed regardless. CVE-2023-25725 was assigned to this issue.
2023-02-09 15:36:54 -05:00
if (col <= sol) {
state = H1_MSG_HDR_NAME;
goto http_msg_invalid;
}
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_l1_sp, http_msg_ood, state, H1_MSG_HDR_L1_SP);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
}
MEDIUM: h1: consider err_pos before deciding to accept a header name or not Till now the H1 parser made for H2 used to be lenient on invalid header field names because they were supposed to be produced by haproxy. Now instead we'll rely on err_pos to know how to act (ie: -2 == must block).
2018-09-12 03:20:40 -04:00
if (likely(h1m->err_pos < -1) || *ptr == '\n') {
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
state = H1_MSG_HDR_NAME;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_invalid;
}
MEDIUM: h1: consider err_pos before deciding to accept a header name or not Till now the H1 parser made for H2 used to be lenient on invalid header field names because they were supposed to be produced by haproxy. Now instead we'll rely on err_pos to know how to act (ie: -2 == must block).
2018-09-12 03:20:40 -04:00
if (h1m->err_pos == -1) /* capture the error pointer */
h1m->err_pos = ptr - start + skip; /* >= 0 now */
/* and we still accept this non-token character */
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_name, http_msg_ood, state, H1_MSG_HDR_NAME);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_HDR_L1_SP:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_hdr_l1_sp:
/* assumes sol points to the first char */
if (likely(HTTP_IS_SPHT(*ptr)))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_l1_sp, http_msg_ood, state, H1_MSG_HDR_L1_SP);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
/* header value can be basically anything except CR/LF */
sov = ptr - start;
if (likely(!HTTP_IS_CRLF(*ptr))) {
goto http_msg_hdr_val;
}
if (likely(*ptr == '\r'))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_l1_lf, http_msg_ood, state, H1_MSG_HDR_L1_LF);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_hdr_l1_lf;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_HDR_L1_LF:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_hdr_l1_lf:
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EXPECT_LF_HERE(ptr, http_msg_invalid, state, H1_MSG_HDR_L1_LF);
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_l1_lws, http_msg_ood, state, H1_MSG_HDR_L1_LWS);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_HDR_L1_LWS:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_hdr_l1_lws:
if (likely(HTTP_IS_SPHT(*ptr))) {
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
if (!skip_update) {
/* replace HT,CR,LF with spaces */
for (; start + sov < ptr; sov++)
start[sov] = ' ';
}
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_hdr_l1_sp;
}
/* we had a header consisting only in spaces ! */
eol = sov;
goto http_msg_complete_header;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_HDR_VAL:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_hdr_val:
/* assumes sol points to the first char, and sov
* points to the first character of the value.
*/
/* speedup: we'll skip packs of 4 or 8 bytes not containing bytes 0x0D
* and lower. In fact since most of the time is spent in the loop, we
* also remove the sign bit test so that bytes 0x8e..0x0d break the
* loop, but we don't care since they're very rare in header values.
*/
CLEANUP: http/h1: rely on HA_UNALIGNED_LE instead of checking for CPU families Now that we have flags indicating the CPU's capabilities, better use them instead of missing some updates for new CPU families (ARMv8 was missing there).
2020-02-21 10:31:22 -05:00
#ifdef HA_UNALIGNED_LE64
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
while (ptr <= end - sizeof(long)) {
CLEANUP: h1: make use of the multi-byte matching functions Instead of leaving the hard-coded non-trivial operations in the H1 parsing code, let's just rely on the new intops functions that do the same and that are less prone to being accidentally touched. It was verified that the resulting code is exactly the same.
2024-04-24 09:57:15 -04:00
if (is_char8_below_opt(*(ulong *)ptr, 0x0e))
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_hdr_val2;
ptr += sizeof(long);
}
#endif
CLEANUP: http/h1: rely on HA_UNALIGNED_LE instead of checking for CPU families Now that we have flags indicating the CPU's capabilities, better use them instead of missing some updates for new CPU families (ARMv8 was missing there).
2020-02-21 10:31:22 -05:00
#ifdef HA_UNALIGNED_LE
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
while (ptr <= end - sizeof(int)) {
CLEANUP: h1: make use of the multi-byte matching functions Instead of leaving the hard-coded non-trivial operations in the H1 parsing code, let's just rely on the new intops functions that do the same and that are less prone to being accidentally touched. It was verified that the resulting code is exactly the same.
2024-04-24 09:57:15 -04:00
if (is_char4_below_opt(*(uint *)ptr, 0x0e))
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_hdr_val2;
ptr += sizeof(int);
}
#endif
if (ptr >= end) {
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
state = H1_MSG_HDR_VAL;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_ood;
}
http_msg_hdr_val2:
BUG/MEDIUM: h1: always reject the NUL character in header values Ben Kallus kindly reported that we still hadn't blocked the NUL character from header values as clarified in RFC9110 and that, even though there's no known issure related to this, it may one day be used to construct an attack involving another component. Actually, both Christopher and I sincerely believed we had done it prior to releasing 2.9, shame on us for missing that one and thanks to Ben for the reminder! The change was applied, it was confirmed to properly reject this NUL byte from both header and trailer values, and it's still possible to force it to continue to be supported using the usual pair of unsafe "option accept-invalid-http-{request|response}" for those who would like to keep it for whatever reason that wouldn't make sense. This was tagged medium so that distros also remember to apply it as a preventive measure. It should progressively be backported to all versions down to 2.0.
2024-01-31 09:10:39 -05:00
if (likely(!*ptr)) {
/* RFC9110 clarified that NUL is explicitly forbidden in header values
* (like CR and LF).
*/
if (h1m->err_pos < -1) { /* PR_O2_REQBUG_OK not set */
state = H1_MSG_HDR_VAL;
goto http_msg_invalid;
}
if (h1m->err_pos == -1) /* PR_O2_REQBUG_OK set: just log */
h1m->err_pos = ptr - start + skip;
}
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
if (likely(!HTTP_IS_CRLF(*ptr)))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_val2, http_msg_ood, state, H1_MSG_HDR_VAL);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
eol = ptr - start;
/* Note: we could also copy eol into ->eoh so that we have the
* real header end in case it ends with lots of LWS, but is this
* really needed ?
*/
if (likely(*ptr == '\r'))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_l2_lf, http_msg_ood, state, H1_MSG_HDR_L2_LF);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_hdr_l2_lf;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_HDR_L2_LF:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_hdr_l2_lf:
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EXPECT_LF_HERE(ptr, http_msg_invalid, state, H1_MSG_HDR_L2_LF);
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_hdr_l2_lws, http_msg_ood, state, H1_MSG_HDR_L2_LWS);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_HDR_L2_LWS:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_hdr_l2_lws:
if (unlikely(HTTP_IS_SPHT(*ptr))) {
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
if (!skip_update) {
/* LWS: replace HT,CR,LF with spaces */
for (; start + eol < ptr; eol++)
start[eol] = ' ';
}
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_hdr_val;
}
http_msg_complete_header:
/*
* It was a new header, so the last one is finished. Assumes
* <sol> points to the first char of the name, <col> to the
* colon, <sov> points to the first character of the value and
* <eol> to the first CR or LF so we know how the line ends. We
* will trim spaces around the value. It's possible to do it by
* adjusting <eol> and <sov> which are no more used after this.
* We can add the header field to the list.
*/
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
if (likely(!skip_update)) {
while (sov < eol && HTTP_IS_LWS(start[sov]))
sov++;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
while (eol - 1 > sov && HTTP_IS_LWS(start[eol - 1]))
eol--;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
n = ist2(start + sol, col - sol);
v = ist2(start + sov, eol - sov);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
do {
int ret;
MINOR: h1: make the message parser support a null <hdr> argument This will allow some iterative calls to be made on incomplete messages without having to store all the headers.
2018-09-11 10:04:48 -04:00
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
if (unlikely(hdr_count >= hdr_num)) {
MEDIUM: h1: deduplicate the content-length header Just like we used to do in proto_http, we now check that each and every occurrence of the content-length header field and each of its values are exactly identical, and we normalize the header to return the last value of the first header with spaces trimmed.
2018-09-14 11:11:33 -04:00
state = H1_MSG_HDR_L2_LWS;
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
goto http_output_full;
MEDIUM: h1: deduplicate the content-length header Just like we used to do in proto_http, we now check that each and every occurrence of the content-length header field and each of its values are exactly identical, and we normalize the header to return the last value of the first header with spaces trimmed.
2018-09-14 11:11:33 -04:00
}
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
if (isteqi(n, ist("transfer-encoding"))) {
MINOR: h1: Change T-E header parsing to fail if chunked encoding is found twice According to the RFC7230, "chunked" encoding must not be applied more than once to a message body. To handle this case, h1_parse_xfer_enc_header() is now responsible to fail when a parsing error is found. It also fails if the "chunked" encoding is not the last one for a request. To help the parsing, two H1 parser flags have been added: H1_MF_TE_CHUNKED and H1_MF_TE_OTHER. These flags are set, respectively, when "chunked" encoding and any other encoding are found. H1_MF_CHNK flag is used when "chunked" encoding is the last one.
2021-09-28 03:36:25 -04:00
ret = h1_parse_xfer_enc_header(h1m, v);
if (ret < 0) {
MEDIUM: h1: Accept invalid T-E values with accept-invalid-http-response option Since the 2.6, A parsing error is reported when the chunked encoding is found twice. As stated in RFC9112, A sender must not apply the chunked transfer coding more than once to a message body. It means only one chunked coding must be found. In addition, empty values are also rejected becaues it is forbidden by RFC9110. However, in both cases, it may be useful to relax the rules for trusted legacy servers when accept-invalid-http-response option is set. Especially because it was accepted on 2.4 and older. In addition, T-E header is now sanitized before sending it. It is not a problem Because it is a hop-by-hop header Note that it remains invalid on client side because there is no good reason to relax the parsing on this side. We can argue a server is trusted so we can decide to support some legacy behavior. It is not true on client side and it is highly suspicious if a client is sending an invalid T-E header. Note also we continue to reject unsupported T-E values (so all codings except "chunked"). Because the "TE" header is sanitized and cannot contain other value than "Trailers", there is absolutely no reason for a server to use something else. This patch should fix the issue #2677. It could probably be backported as far as 2.6 if necessary.
2024-09-11 09:02:07 -04:00
/* For the response only, don't report error if PR_O2_RSPBUG_OK is set
* and the error can be hidden */
if (ret == -2 || !(h1m->flags & H1_MF_RESP) || (h1m->err_pos < -1)) {
state = H1_MSG_HDR_L2_LWS;
ptr = v.ptr; /* Set ptr on the error */
goto http_msg_invalid;
}
if (h1m->err_pos == -1)
h1m->err_pos = ptr - start + skip;
MINOR: h1: Change T-E header parsing to fail if chunked encoding is found twice According to the RFC7230, "chunked" encoding must not be applied more than once to a message body. To handle this case, h1_parse_xfer_enc_header() is now responsible to fail when a parsing error is found. It also fails if the "chunked" encoding is not the last one for a request. To help the parsing, two H1 parser flags have been added: H1_MF_TE_CHUNKED and H1_MF_TE_OTHER. These flags are set, respectively, when "chunked" encoding and any other encoding are found. H1_MF_CHNK flag is used when "chunked" encoding is the last one.
2021-09-28 03:36:25 -04:00
}
else if (ret == 0) {
/* skip it */
break;
}
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
}
else if (isteqi(n, ist("content-length"))) {
MINOR: hlua/h1: Use http_parse_cont_len_header() to parse content-length value Till now, h1_parse_cont_len_header() was used during the H1 message parsing and by the lua HTTP applets to parse the content-length header value. But a more generic function was added some years ago doing exactly the same operations. So let's use it instead.
2025-04-15 13:14:31 -04:00
unsigned long long body_len = h1m->body_len;
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
MINOR: hlua/h1: Use http_parse_cont_len_header() to parse content-length value Till now, h1_parse_cont_len_header() was used during the H1 message parsing and by the lua HTTP applets to parse the content-length header value. But a more generic function was added some years ago doing exactly the same operations. So let's use it instead.
2025-04-15 13:14:31 -04:00
ret = http_parse_cont_len_header(&v, &body_len, (h1m->flags & H1_MF_CLEN));
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
if (ret < 0) {
state = H1_MSG_HDR_L2_LWS;
BUG/MINOR: h1: Report the right error position when a header value is invalid During H1 messages parsing, when the parser has finished to parse a full header line, some tests are performed on its value, depending on its name, to be sure it is valid. The content-length is checked and converted in integer and the host header is also checked. If an error occurred during this step, the error position must point on the header value. But from the parser point of view, we are already on the start of the next header. Thus the effective reported position in the error capture is the beginning of the unparsed header line. It is a bit confusing when we try to figure out why a message is rejected. Now, the parser state is updated to point on the invalid value. This way, the error position really points on the right position. This patch must be backported as far as 1.9.
2020-01-06 07:41:01 -05:00
ptr = v.ptr; /* Set ptr on the error */
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
goto http_msg_invalid;
}
else if (ret == 0) {
/* skip it */
break;
}
MINOR: hlua/h1: Use http_parse_cont_len_header() to parse content-length value Till now, h1_parse_cont_len_header() was used during the H1 message parsing and by the lua HTTP applets to parse the content-length header value. But a more generic function was added some years ago doing exactly the same operations. So let's use it instead.
2025-04-15 13:14:31 -04:00
h1m->flags |= H1_MF_CLEN;
h1m->curr_len = h1m->body_len = body_len;
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
}
else if (isteqi(n, ist("connection"))) {
MEDIUM: h1: Add an option to sanitize connection headers during parsing The flag H1_MF_CLEAN_CONN_HDR has been added to let the H1 parser sanitize connection headers. It means it will remove all "close" and "keep-alive" values during the parsing. One noticeable effect is that connection headers may be unfolded. In practice, this is not a problem because it is not frequent to have multiple values for the connection headers. If this flag is set, during the parsing The function h1_parse_next_connection_header() is called in a loop instead of h1_parse_conection_header(). No need to backport this patch
2019-03-29 10:03:13 -04:00
h1_parse_connection_header(h1m, &v);
if (!v.len) {
/* skip it */
break;
MINOR: h1: Reject requests with different occurrences of the header host There is no reason for a client to send several headers host. It even may be considered as a bug. However, it is totally invalid to have different values for those. So now, in such case, an error is triggered during the request parsing. In addition, when several headers host are found with the same value, only the first instance is kept and others are skipped.
2019-10-11 03:01:44 -04:00
}
}
MINOR: h1: reject websocket handshake if missing key If a request is identified as a WebSocket handshake, it must contains a websocket key header or else it can be reject, following the rfc6455. A new flag H1_MF_UPG_WEBSOCKET is set on such messages. For the request te be identified as a WebSocket handshake, it must contains the headers: Connection: upgrade Upgrade: websocket This commit is a compagnon of "MEDIUM: h1: generate WebSocket key on response if needed" and "MEDIUM: h1: add a WebSocket key on handshake if needed". Indeed, it ensures that a WebSocket key is added only from a http/2 side and not for a http/1 bogus peer.
2020-12-11 11:53:02 -05:00
else if (isteqi(n, ist("upgrade"))) {
h1_parse_upgrade_header(h1m, v);
}
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
else if (!(h1m->flags & H1_MF_RESP) && isteqi(n, ist("host"))) {
BUG/MEDIUM: h1/h2/h3: reject forbidden chars in the Host header field In continuation with 9a05c1f574 ("BUG/MEDIUM: h2/h3: reject some forbidden chars in :authority before reassembly") and the discussion in issue #2941, @DemiMarie rightfully suggested that Host should also be sanitized, because it is sometimes used in concatenation, such as this: http-request set-url https://%[req.hdr(host)]%[pathq] which was proposed as a workaround for h2 upstream servers that require :authority here: https://www.mail-archive.com/haproxy@formilux.org/msg43261.html The current patch then adds the same check for forbidden chars in the Host header, using the same function as for the patch above, since in both cases we validate the host:port part of the authority. This way we won't reconstruct ambiguous URIs by concatenating Host and path. Just like the patch above, this can be backported afer a period of observation.
2025-05-16 08:58:52 -04:00
if (host_idx == -1) {
MINOR: h1: Reject requests with different occurrences of the header host There is no reason for a client to send several headers host. It even may be considered as a bug. However, it is totally invalid to have different values for those. So now, in such case, an error is triggered during the request parsing. In addition, when several headers host are found with the same value, only the first instance is kept and others are skipped.
2019-10-11 03:01:44 -04:00
host_idx = hdr_count;
BUG/MEDIUM: h1/h2/h3: reject forbidden chars in the Host header field In continuation with 9a05c1f574 ("BUG/MEDIUM: h2/h3: reject some forbidden chars in :authority before reassembly") and the discussion in issue #2941, @DemiMarie rightfully suggested that Host should also be sanitized, because it is sometimes used in concatenation, such as this: http-request set-url https://%[req.hdr(host)]%[pathq] which was proposed as a workaround for h2 upstream servers that require :authority here: https://www.mail-archive.com/haproxy@formilux.org/msg43261.html The current patch then adds the same check for forbidden chars in the Host header, using the same function as for the patch above, since in both cases we validate the host:port part of the authority. This way we won't reconstruct ambiguous URIs by concatenating Host and path. Just like the patch above, this can be backported afer a period of observation.
2025-05-16 08:58:52 -04:00
if (http_authority_has_forbidden_char(v)) {
state = H1_MSG_HDR_L2_LWS;
ptr = v.ptr; /* Set ptr on the error */
goto http_msg_invalid;
}
}
MINOR: h1: Reject requests with different occurrences of the header host There is no reason for a client to send several headers host. It even may be considered as a bug. However, it is totally invalid to have different values for those. So now, in such case, an error is triggered during the request parsing. In addition, when several headers host are found with the same value, only the first instance is kept and others are skipped.
2019-10-11 03:01:44 -04:00
else {
if (!isteqi(v, hdr[host_idx].v)) {
state = H1_MSG_HDR_L2_LWS;
BUG/MINOR: h1: Report the right error position when a header value is invalid During H1 messages parsing, when the parser has finished to parse a full header line, some tests are performed on its value, depending on its name, to be sure it is valid. The content-length is checked and converted in integer and the host header is also checked. If an error occurred during this step, the error position must point on the header value. But from the parser point of view, we are already on the start of the next header. Thus the effective reported position in the error capture is the beginning of the unparsed header line. It is a bit confusing when we try to figure out why a message is rejected. Now, the parser state is updated to point on the invalid value. This way, the error position really points on the right position. This patch must be backported as far as 1.9.
2020-01-06 07:41:01 -05:00
ptr = v.ptr; /* Set ptr on the error */
MINOR: h1: Reject requests with different occurrences of the header host There is no reason for a client to send several headers host. It even may be considered as a bug. However, it is totally invalid to have different values for those. So now, in such case, an error is triggered during the request parsing. In addition, when several headers host are found with the same value, only the first instance is kept and others are skipped.
2019-10-11 03:01:44 -04:00
goto http_msg_invalid;
}
/* if the same host, skip it */
break;
MEDIUM: h1: Add an option to sanitize connection headers during parsing The flag H1_MF_CLEAN_CONN_HDR has been added to let the H1 parser sanitize connection headers. It means it will remove all "close" and "keep-alive" values during the parsing. One noticeable effect is that connection headers may be unfolded. In practice, this is not a problem because it is not frequent to have multiple values for the connection headers. If this flag is set, during the parsing The function h1_parse_next_connection_header() is called in a loop instead of h1_parse_conection_header(). No need to backport this patch
2019-03-29 10:03:13 -04:00
}
MEDIUM: h1: deduplicate the content-length header Just like we used to do in proto_http, we now check that each and every occurrence of the content-length header field and each of its values are exactly identical, and we normalize the header to return the last value of the first header with spaces trimmed.
2018-09-14 11:11:33 -04:00
}
MINOR: h1: add headers to the list after controls, not before This will ease removal/skipping of duplicates such as content-length.
2018-09-14 10:28:15 -04:00
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
http_set_hdr(&hdr[hdr_count++], n, v);
} while (0);
}
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
sol = ptr - start;
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are parsed In h1_headers_to_hdr_list, when an incomplete message is parsed, all updates must be skipped until the end of the message is found. Then the parsing is restarted from the beginning. But not all updates were skipped, leading to invalid rewritting or segfault. No backport is needed.
2018-09-19 08:01:04 -04:00
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
if (likely(!HTTP_IS_CRLF(*ptr)))
goto http_msg_hdr_name;
if (likely(*ptr == '\r'))
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EAT_AND_JUMP_OR_RETURN(ptr, end, http_msg_last_lf, http_msg_ood, state, H1_MSG_LAST_LF);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
goto http_msg_last_lf;
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
case H1_MSG_LAST_LF:
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_last_lf:
REORG: h1: create a new h1m_state This is the *parsing* state of an HTTP/1 message. Currently the h1_state is composite as it's made both of parsing and control (100SENT, BODY, DONE, TUNNEL, ENDING etc). The purpose here is to have a purely H1 state that can be used by H1 parsers. For now it's equivalent to h1_state.
2018-09-11 05:45:04 -04:00
EXPECT_LF_HERE(ptr, http_msg_invalid, state, H1_MSG_LAST_LF);
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
ptr++;
/* <ptr> now points to the first byte of payload. If needed sol
* still points to the first of either CR or LF of the empty
* line ending the headers block.
*/
MINOR: h1: make the message parser support a null <hdr> argument This will allow some iterative calls to be made on incomplete messages without having to store all the headers.
2018-09-11 10:04:48 -04:00
if (likely(!skip_update)) {
if (unlikely(hdr_count >= hdr_num)) {
state = H1_MSG_LAST_LF;
goto http_output_full;
}
MINOR: h1: Add EOH marker during headers parsing When headers parsing ends, a pseudo header with an empty name and an empty value is added to the array of parsed headers to mark its end. It is convenient to loop on this array, but not really useful if we want remove the last header or add a new one, because we don't really know where is the last CRLF (the empty line ending the headers block). So now, instead the name of this pseudo header points on this last CRLF. Its length is still 0 and its value is still empty, so loops on the array remains unchanged.
2018-09-25 07:59:46 -04:00
http_set_hdr(&hdr[hdr_count++], ist2(start+sol, 0), ist(""));
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
}
MEDIUM: h1: remove the useless H1_MSG_BODY state This state was only a delimiter between headers and body but it now causes more harm than good because it requires someone to change it. Since the H1 parser knows if we're in DATA or CHUNK_SIZE, simply let it set the right next state so that h1m->state constantly matches what is expected afterwards.
2018-09-12 11:25:32 -04:00
/* reaching here we've parsed the whole message. We may detect
* that we were already continuing an interrupted parsing pass
* so we were silently looking for the end of message not
* updating anything before deciding to parse it fully at once.
* It's guaranteed that we won't match this test twice in a row
* since restarting will turn zero.
*/
if (restarting)
goto restart;
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
if (!(h1m->flags & (H1_MF_HDRS_ONLY|H1_MF_RESP))) {
struct http_uri_parser parser = http_uri_parser_init(sl.rq.u);
BUG/MINOR: h1: Check authority for non-CONNECT methods only if a scheme is found When a non-CONNECT H1 request is parsed, the authority is compared to the host header value, to validate that they are the same. However there is an issue here when a relative path is used (not begining with a '/'). In this case, the path is considered as the authority and will be erroneously compared to the host header value. It is observable with this kind of request: GET admin HTTP/1.1 Host: www.mysite.com In this case "admin" is parsed as an authority while it is in fact a path. At this step, it is not a big deal because it just happens on the very first checks on the message during the parsing. However, the same happens when the authority is updated. This will be fixed in another commit Note this kind of request is invalid because the path does not start with a '/'. But, till now, HAProxy does not reject it. This patch is related to issue #2565. It must be backported as far as 2.4.
2024-05-14 05:42:21 -04:00
struct ist scheme, authority = IST_NULL;
BUG/MINOR: h1: Replace authority validation to conform RFC3986 Except for CONNECT method, where a normalization is performed, we expected to have an exact match between the authority and the host header value. However it was too strict. Indeed, default port must be handled and the matching must respect the RFC3986. There is already a scheme based normalizeation performed on the URI later, on the HTX message. And we cannot normalize the URI during H1 parsing to be able to report proper errors on the original raw buffer. And a systematic read-only normalization to validate the authority will consume CPU for only few requests. At the end, we decided to perform extra-checks when the exact match fails. Now, following authority/host are considered as equivalent: http: domain.com <=> domain.com:80 <=> domain.com: https: domain.com <=> domain.com:443 <=> domain.com: This patch depends on: * MINOR: h1: Consider empty port as invalid in authority for CONNECT * MINOR: http: Considere empty ports as valid default ports It is a bug regarding the RFC3986. Technically, I may be backported as far as 2.4. However, this must be discussed first. If it is backported, the commits above must be backported too.
2022-11-22 04:04:16 -05:00
int ret;
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
BUG/MAJOR: h1: Be stricter on request target validation during message parsing As stated in issue #2565, checks on the request target during H1 message parsing are not good enough. Invalid paths, not starting by a slash are in fact parsed as authorities. The same error is repeated at the sample fetch level. This last point is annoying because routing rules may be fooled. It is also an issue when the URI or the Host header are updated. Because the error is repeated at different places, it must be fixed. We cannot be lax by arguing it is the server's job to accept or reject invalid request targets. With this patch, we strengthen the checks performed on the request target during H1 parsing. Idea is to reject invalid requests at this step to be sure it is safe to manipulate the path or the authority at other places. So now, the asterisk-form is only allowed for OPTIONS and OTHER methods. This last point was added to not reject the H2 preface. In addition, we take care to have only one asterisk and nothing more. For the CONNECT method, we take care to have a valid authority-form. All other form are rejected. The authority-form is now only supported for CONNECT method. No specific check is performed on the origin-form (except for the CONNECT method). For the absolute-form, we take care to have a scheme and a valid authority. These checks are not perfect but should be good enough to properly identify each part of the request target for a relative small cost. But, it is a breaking change. Some requests are now be rejected while they was not on older versions. However, nowadays, it is most probably not an issue. If it turns out it's really an issue for legitimate use-cases, an option would be to supports these kinds of requests when the "accept-invalid-http-request" option is set, with the consequence of seeing some sample fetches having an unexpected behavior. This patch should fix the issue #2665. It MUST NOT be backported. First because it is a breaking change. And then because by avoiding backporting it, it remains possible to relax the parsing with the "accept-invalid-http-request" option.
2024-05-15 10:53:50 -04:00
/* WT: gcc seems to see a path where sl.rq.u.ptr was used
* uninitialized, but it doesn't know that the function is
* called with initial states making this impossible.
*/
ALREADY_CHECKED(sl.rq.u.ptr);
switch (parser.format) {
case URI_PARSER_FORMAT_ASTERISK:
/* We must take care "PRI * HTTP/2.0" is supported here. check for OTHER methods here is enough */
if ((sl.rq.meth != HTTP_METH_OTHER && sl.rq.meth != HTTP_METH_OPTIONS) || istlen(sl.rq.u) != 1) {
ptr = sl.rq.u.ptr; /* Set ptr on the error */
goto http_msg_invalid;
}
break;
BUG/MINOR: h1: Check authority for non-CONNECT methods only if a scheme is found When a non-CONNECT H1 request is parsed, the authority is compared to the host header value, to validate that they are the same. However there is an issue here when a relative path is used (not begining with a '/'). In this case, the path is considered as the authority and will be erroneously compared to the host header value. It is observable with this kind of request: GET admin HTTP/1.1 Host: www.mysite.com In this case "admin" is parsed as an authority while it is in fact a path. At this step, it is not a big deal because it just happens on the very first checks on the message during the parsing. However, the same happens when the authority is updated. This will be fixed in another commit Note this kind of request is invalid because the path does not start with a '/'. But, till now, HAProxy does not reject it. This patch is related to issue #2565. It must be backported as far as 2.4.
2024-05-14 05:42:21 -04:00
BUG/MAJOR: h1: Be stricter on request target validation during message parsing As stated in issue #2565, checks on the request target during H1 message parsing are not good enough. Invalid paths, not starting by a slash are in fact parsed as authorities. The same error is repeated at the sample fetch level. This last point is annoying because routing rules may be fooled. It is also an issue when the URI or the Host header are updated. Because the error is repeated at different places, it must be fixed. We cannot be lax by arguing it is the server's job to accept or reject invalid request targets. With this patch, we strengthen the checks performed on the request target during H1 parsing. Idea is to reject invalid requests at this step to be sure it is safe to manipulate the path or the authority at other places. So now, the asterisk-form is only allowed for OPTIONS and OTHER methods. This last point was added to not reject the H2 preface. In addition, we take care to have only one asterisk and nothing more. For the CONNECT method, we take care to have a valid authority-form. All other form are rejected. The authority-form is now only supported for CONNECT method. No specific check is performed on the origin-form (except for the CONNECT method). For the absolute-form, we take care to have a scheme and a valid authority. These checks are not perfect but should be good enough to properly identify each part of the request target for a relative small cost. But, it is a breaking change. Some requests are now be rejected while they was not on older versions. However, nowadays, it is most probably not an issue. If it turns out it's really an issue for legitimate use-cases, an option would be to supports these kinds of requests when the "accept-invalid-http-request" option is set, with the consequence of seeing some sample fetches having an unexpected behavior. This patch should fix the issue #2665. It MUST NOT be backported. First because it is a breaking change. And then because by avoiding backporting it, it remains possible to relax the parsing with the "accept-invalid-http-request" option.
2024-05-15 10:53:50 -04:00
case URI_PARSER_FORMAT_ABSPATH:
if (sl.rq.meth == HTTP_METH_CONNECT) {
ptr = sl.rq.u.ptr; /* Set ptr on the error */
goto http_msg_invalid;
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
}
BUG/MAJOR: h1: Be stricter on request target validation during message parsing As stated in issue #2565, checks on the request target during H1 message parsing are not good enough. Invalid paths, not starting by a slash are in fact parsed as authorities. The same error is repeated at the sample fetch level. This last point is annoying because routing rules may be fooled. It is also an issue when the URI or the Host header are updated. Because the error is repeated at different places, it must be fixed. We cannot be lax by arguing it is the server's job to accept or reject invalid request targets. With this patch, we strengthen the checks performed on the request target during H1 parsing. Idea is to reject invalid requests at this step to be sure it is safe to manipulate the path or the authority at other places. So now, the asterisk-form is only allowed for OPTIONS and OTHER methods. This last point was added to not reject the H2 preface. In addition, we take care to have only one asterisk and nothing more. For the CONNECT method, we take care to have a valid authority-form. All other form are rejected. The authority-form is now only supported for CONNECT method. No specific check is performed on the origin-form (except for the CONNECT method). For the absolute-form, we take care to have a scheme and a valid authority. These checks are not perfect but should be good enough to properly identify each part of the request target for a relative small cost. But, it is a breaking change. Some requests are now be rejected while they was not on older versions. However, nowadays, it is most probably not an issue. If it turns out it's really an issue for legitimate use-cases, an option would be to supports these kinds of requests when the "accept-invalid-http-request" option is set, with the consequence of seeing some sample fetches having an unexpected behavior. This patch should fix the issue #2665. It MUST NOT be backported. First because it is a breaking change. And then because by avoiding backporting it, it remains possible to relax the parsing with the "accept-invalid-http-request" option.
2024-05-15 10:53:50 -04:00
break;
case URI_PARSER_FORMAT_ABSURI_OR_AUTHORITY:
scheme = http_parse_scheme(&parser);
if (!isttest(scheme)) { /* scheme not found: MUST be an authority */
struct ist *host = NULL;
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
BUG/MAJOR: h1: Be stricter on request target validation during message parsing As stated in issue #2565, checks on the request target during H1 message parsing are not good enough. Invalid paths, not starting by a slash are in fact parsed as authorities. The same error is repeated at the sample fetch level. This last point is annoying because routing rules may be fooled. It is also an issue when the URI or the Host header are updated. Because the error is repeated at different places, it must be fixed. We cannot be lax by arguing it is the server's job to accept or reject invalid request targets. With this patch, we strengthen the checks performed on the request target during H1 parsing. Idea is to reject invalid requests at this step to be sure it is safe to manipulate the path or the authority at other places. So now, the asterisk-form is only allowed for OPTIONS and OTHER methods. This last point was added to not reject the H2 preface. In addition, we take care to have only one asterisk and nothing more. For the CONNECT method, we take care to have a valid authority-form. All other form are rejected. The authority-form is now only supported for CONNECT method. No specific check is performed on the origin-form (except for the CONNECT method). For the absolute-form, we take care to have a scheme and a valid authority. These checks are not perfect but should be good enough to properly identify each part of the request target for a relative small cost. But, it is a breaking change. Some requests are now be rejected while they was not on older versions. However, nowadays, it is most probably not an issue. If it turns out it's really an issue for legitimate use-cases, an option would be to supports these kinds of requests when the "accept-invalid-http-request" option is set, with the consequence of seeing some sample fetches having an unexpected behavior. This patch should fix the issue #2665. It MUST NOT be backported. First because it is a breaking change. And then because by avoiding backporting it, it remains possible to relax the parsing with the "accept-invalid-http-request" option.
2024-05-15 10:53:50 -04:00
if (sl.rq.meth != HTTP_METH_CONNECT) {
ptr = sl.rq.u.ptr; /* Set ptr on the error */
goto http_msg_invalid;
}
if (host_idx != -1)
host = &hdr[host_idx].v;
authority = http_parse_authority(&parser, 1);
ret = h1_validate_connect_authority(scheme, authority, host);
BUG/MINOR: h1: Replace authority validation to conform RFC3986 Except for CONNECT method, where a normalization is performed, we expected to have an exact match between the authority and the host header value. However it was too strict. Indeed, default port must be handled and the matching must respect the RFC3986. There is already a scheme based normalizeation performed on the URI later, on the HTX message. And we cannot normalize the URI during H1 parsing to be able to report proper errors on the original raw buffer. And a systematic read-only normalization to validate the authority will consume CPU for only few requests. At the end, we decided to perform extra-checks when the exact match fails. Now, following authority/host are considered as equivalent: http: domain.com <=> domain.com:80 <=> domain.com: https: domain.com <=> domain.com:443 <=> domain.com: This patch depends on: * MINOR: h1: Consider empty port as invalid in authority for CONNECT * MINOR: http: Considere empty ports as valid default ports It is a bug regarding the RFC3986. Technically, I may be backported as far as 2.4. However, this must be discussed first. If it is backported, the commits above must be backported too.
2022-11-22 04:04:16 -05:00
if (ret < 0) {
if (h1m->err_pos < -1) {
state = H1_MSG_LAST_LF;
BUG/MAJOR: h1: Be stricter on request target validation during message parsing As stated in issue #2565, checks on the request target during H1 message parsing are not good enough. Invalid paths, not starting by a slash are in fact parsed as authorities. The same error is repeated at the sample fetch level. This last point is annoying because routing rules may be fooled. It is also an issue when the URI or the Host header are updated. Because the error is repeated at different places, it must be fixed. We cannot be lax by arguing it is the server's job to accept or reject invalid request targets. With this patch, we strengthen the checks performed on the request target during H1 parsing. Idea is to reject invalid requests at this step to be sure it is safe to manipulate the path or the authority at other places. So now, the asterisk-form is only allowed for OPTIONS and OTHER methods. This last point was added to not reject the H2 preface. In addition, we take care to have only one asterisk and nothing more. For the CONNECT method, we take care to have a valid authority-form. All other form are rejected. The authority-form is now only supported for CONNECT method. No specific check is performed on the origin-form (except for the CONNECT method). For the absolute-form, we take care to have a scheme and a valid authority. These checks are not perfect but should be good enough to properly identify each part of the request target for a relative small cost. But, it is a breaking change. Some requests are now be rejected while they was not on older versions. However, nowadays, it is most probably not an issue. If it turns out it's really an issue for legitimate use-cases, an option would be to supports these kinds of requests when the "accept-invalid-http-request" option is set, with the consequence of seeing some sample fetches having an unexpected behavior. This patch should fix the issue #2665. It MUST NOT be backported. First because it is a breaking change. And then because by avoiding backporting it, it remains possible to relax the parsing with the "accept-invalid-http-request" option.
2024-05-15 10:53:50 -04:00
/* WT: gcc seems to see a path where sl.rq.u.ptr was used
* uninitialized, but it doesn't know that the function is
* called with initial states making this impossible.
*/
ALREADY_CHECKED(sl.rq.u.ptr);
ptr = ((ret == -1) ? sl.rq.u.ptr : host->ptr); /* Set ptr on the error */
BUG/MINOR: h1: Replace authority validation to conform RFC3986 Except for CONNECT method, where a normalization is performed, we expected to have an exact match between the authority and the host header value. However it was too strict. Indeed, default port must be handled and the matching must respect the RFC3986. There is already a scheme based normalizeation performed on the URI later, on the HTX message. And we cannot normalize the URI during H1 parsing to be able to report proper errors on the original raw buffer. And a systematic read-only normalization to validate the authority will consume CPU for only few requests. At the end, we decided to perform extra-checks when the exact match fails. Now, following authority/host are considered as equivalent: http: domain.com <=> domain.com:80 <=> domain.com: https: domain.com <=> domain.com:443 <=> domain.com: This patch depends on: * MINOR: h1: Consider empty port as invalid in authority for CONNECT * MINOR: http: Considere empty ports as valid default ports It is a bug regarding the RFC3986. Technically, I may be backported as far as 2.4. However, this must be discussed first. If it is backported, the commits above must be backported too.
2022-11-22 04:04:16 -05:00
goto http_msg_invalid;
}
if (h1m->err_pos == -1) /* capture the error pointer */
BUG/MAJOR: h1: Be stricter on request target validation during message parsing As stated in issue #2565, checks on the request target during H1 message parsing are not good enough. Invalid paths, not starting by a slash are in fact parsed as authorities. The same error is repeated at the sample fetch level. This last point is annoying because routing rules may be fooled. It is also an issue when the URI or the Host header are updated. Because the error is repeated at different places, it must be fixed. We cannot be lax by arguing it is the server's job to accept or reject invalid request targets. With this patch, we strengthen the checks performed on the request target during H1 parsing. Idea is to reject invalid requests at this step to be sure it is safe to manipulate the path or the authority at other places. So now, the asterisk-form is only allowed for OPTIONS and OTHER methods. This last point was added to not reject the H2 preface. In addition, we take care to have only one asterisk and nothing more. For the CONNECT method, we take care to have a valid authority-form. All other form are rejected. The authority-form is now only supported for CONNECT method. No specific check is performed on the origin-form (except for the CONNECT method). For the absolute-form, we take care to have a scheme and a valid authority. These checks are not perfect but should be good enough to properly identify each part of the request target for a relative small cost. But, it is a breaking change. Some requests are now be rejected while they was not on older versions. However, nowadays, it is most probably not an issue. If it turns out it's really an issue for legitimate use-cases, an option would be to supports these kinds of requests when the "accept-invalid-http-request" option is set, with the consequence of seeing some sample fetches having an unexpected behavior. This patch should fix the issue #2665. It MUST NOT be backported. First because it is a breaking change. And then because by avoiding backporting it, it remains possible to relax the parsing with the "accept-invalid-http-request" option.
2024-05-15 10:53:50 -04:00
h1m->err_pos = ((ret == -1) ? sl.rq.u.ptr : host->ptr) - start + skip; /* >= 0 now */
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
}
}
BUG/MAJOR: h1: Be stricter on request target validation during message parsing As stated in issue #2565, checks on the request target during H1 message parsing are not good enough. Invalid paths, not starting by a slash are in fact parsed as authorities. The same error is repeated at the sample fetch level. This last point is annoying because routing rules may be fooled. It is also an issue when the URI or the Host header are updated. Because the error is repeated at different places, it must be fixed. We cannot be lax by arguing it is the server's job to accept or reject invalid request targets. With this patch, we strengthen the checks performed on the request target during H1 parsing. Idea is to reject invalid requests at this step to be sure it is safe to manipulate the path or the authority at other places. So now, the asterisk-form is only allowed for OPTIONS and OTHER methods. This last point was added to not reject the H2 preface. In addition, we take care to have only one asterisk and nothing more. For the CONNECT method, we take care to have a valid authority-form. All other form are rejected. The authority-form is now only supported for CONNECT method. No specific check is performed on the origin-form (except for the CONNECT method). For the absolute-form, we take care to have a scheme and a valid authority. These checks are not perfect but should be good enough to properly identify each part of the request target for a relative small cost. But, it is a breaking change. Some requests are now be rejected while they was not on older versions. However, nowadays, it is most probably not an issue. If it turns out it's really an issue for legitimate use-cases, an option would be to supports these kinds of requests when the "accept-invalid-http-request" option is set, with the consequence of seeing some sample fetches having an unexpected behavior. This patch should fix the issue #2665. It MUST NOT be backported. First because it is a breaking change. And then because by avoiding backporting it, it remains possible to relax the parsing with the "accept-invalid-http-request" option.
2024-05-15 10:53:50 -04:00
else { /* Scheme found: MUST be an absolute-URI */
struct ist host = IST_NULL;
if (sl.rq.meth == HTTP_METH_CONNECT) {
ptr = sl.rq.u.ptr; /* Set ptr on the error */
goto http_msg_invalid;
}
if (host_idx != -1)
host = hdr[host_idx].v;
authority = http_parse_authority(&parser, 1);
/* For non-CONNECT method, the authority must match the host header value */
if (isttest(host) && !isteqi(authority, host)) {
ret = h1_validate_mismatch_authority(scheme, authority, host);
if (ret < 0) {
if (h1m->err_pos < -1) {
state = H1_MSG_LAST_LF;
ptr = host.ptr; /* Set ptr on the error */
goto http_msg_invalid;
}
if (h1m->err_pos == -1) /* capture the error pointer */
h1m->err_pos = v.ptr - start + skip; /* >= 0 now */
}
}
}
break;
default:
ptr = sl.rq.u.ptr; /* Set ptr on the error */
goto http_msg_invalid;
BUG/MEDIUM: h1: Improve authority validation for CONNCET request From time to time, users complain to get 400-Bad-request responses for totally valid CONNECT requests. After analysis, it is due to the H1 parser performs an exact match between the authority and the host header value. For non-CONNECT requests, it is valid. But for CONNECT requests the authority must contain a port while it is often omitted from the host header value (for default ports). So, to be sure to not reject valid CONNECT requests, a basic authority validation is now performed during the message parsing. In addition, the host header value is normalized. It means the default port is removed if possible. This patch should solve the issue #1761. It must be backported to 2.6 and probably as far as 2.4.
2022-07-05 08:50:17 -04:00
}
}
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
state = H1_MSG_DATA;
if (h1m->flags & H1_MF_XFER_ENC) {
if (h1m->flags & H1_MF_CLEN) {
MEDIUM: h1: Force close mode for invalid uses of T-E header Transfer-Encoding header is not supported in HTTP/1.0. However, softwares dealing with HTTP/1.0 and HTTP/1.1 messages may accept it and transfer it. When a Content-Length header is also provided, it must be ignored. Unfortunately, this may lead to vulnerabilities (request smuggling or response splitting) if an intermediary is only implementing HTTP/1.0. Because it may ignore Transfer-Encoding header and only handle Content-Length one. To avoid any security issues, when Transfer-Encoding and Content-Length headers are found in a message, the close mode is forced. The same is performed for HTTP/1.0 message with a Transfer-Encoding header only. This change is conform to what it is described in the last HTTP/1.1 draft. See also httpwg/http-core#879. Note that Content-Length header is also removed from any incoming messages if a Transfer-Encoding header is found. However it is not true (not yet) for responses generated by HAProxy.
2021-09-27 03:47:03 -04:00
/* T-E + C-L: force close and remove C-L */
MEDIUM: h1: allow to preserve keep-alive on T-E + C-L In 2.5-dev9, commit 631c7e866 ("MEDIUM: h1: Force close mode for invalid uses of T-E header") enforced a recently arrived new security rule in the HTTP specification aiming at preventing a class of content-smuggling attacks involving HTTP/1.0 agents. It consists in handling the very rare T-E + C-L requests or responses in close mode. It happens it does have an impact of a rare few and very old clients (probably running insecure TLS stacks by the way) that continue to send both with their POST requests. The impact is that for each and every request they'll have to reconnect, possibly negotiating a full TLS handshake that becomes harmful to the machine in terms of CPU computation. This commit adds a new option "h1-do-not-close-on-insecure-transfer-encoding" that does exactly what it says, it just asks not to close on such messages, even though the message continues to be sanitized and C-L dropped. It means that the risk is only between the sender and haproxy, which is limited, and might be the only acceptable solution for such environments having to deal with broken implementations. The cases are so rare that it should not need to be backported, or in the worst case, to the latest LTS if there is any demand.
2024-07-26 09:14:58 -04:00
if (!h1_do_not_close_on_insecure_t_e)
h1m->flags |= H1_MF_CONN_CLO;
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
h1m->flags &= ~H1_MF_CLEN;
BUG/MEDIUM: h1: Ignore C-L value in the H1 parser if T-E is also set In fact, during the parsing there is already a test to remove the Content-Length header if a Transfer-Encoding one is found. However, in the parser, the content-length value was still used to set the body length (the final one and the remaining one). This value is thus also used to set the extra field in the HTX message and is then used during the sending stage to announce the chunk size. So, Content-Length header value must be ignored by the H1 parser to properly reformat the message when it is sent. This patch must be backported as far as 2.6. Lower versions don"t handle this case.
2023-09-27 09:21:28 -04:00
h1m->curr_len = h1m->body_len = 0;
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
hdr_count = http_del_hdr(hdr, ist("content-length"));
}
MEDIUM: h1: Force close mode for invalid uses of T-E header Transfer-Encoding header is not supported in HTTP/1.0. However, softwares dealing with HTTP/1.0 and HTTP/1.1 messages may accept it and transfer it. When a Content-Length header is also provided, it must be ignored. Unfortunately, this may lead to vulnerabilities (request smuggling or response splitting) if an intermediary is only implementing HTTP/1.0. Because it may ignore Transfer-Encoding header and only handle Content-Length one. To avoid any security issues, when Transfer-Encoding and Content-Length headers are found in a message, the close mode is forced. The same is performed for HTTP/1.0 message with a Transfer-Encoding header only. This change is conform to what it is described in the last HTTP/1.1 draft. See also httpwg/http-core#879. Note that Content-Length header is also removed from any incoming messages if a Transfer-Encoding header is found. However it is not true (not yet) for responses generated by HAProxy.
2021-09-27 03:47:03 -04:00
else if (!(h1m->flags & H1_MF_VER_11)) {
/* T-E + HTTP/1.0: force close */
h1m->flags |= H1_MF_CONN_CLO;
}
MEDIUM: h1: better handle transfer-encoding vs content-length The transfer-encoding header processing was a bit lenient in this part because it was made to read messages already validated by haproxy. We absolutely need to reinstate the strict processing defined in RFC7230 as is currently being done in proto_http.c. That is, transfer-encoding presence alone is enough to cancel content-length, and must be terminated by the "chunked" token, except in the response where we can fall back to the close mode if it's not last. For this we now use a specific parsing function which updates the flags and we introduce a new flag H1_MF_XFER_ENC indicating that the transfer-encoding header is present. Last, if such a header is found, we delete all content-length header fields found in the message.
2018-09-14 10:34:47 -04:00
if (h1m->flags & H1_MF_CHNK)
state = H1_MSG_CHUNK_SIZE;
else if (!(h1m->flags & H1_MF_RESP)) {
/* cf RFC7230#3.3.3 : transfer-encoding in
* request without chunked encoding is invalid.
*/
goto http_msg_invalid;
}
}
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
break;
default:
/* impossible states */
goto http_msg_invalid;
}
MEDIUM: h1: remove the useless H1_MSG_BODY state This state was only a delimiter between headers and body but it now causes more harm than good because it requires someone to change it. Since the H1 parser knows if we're in DATA or CHUNK_SIZE, simply let it set the right next state so that h1m->state constantly matches what is expected afterwards.
2018-09-12 11:25:32 -04:00
/* Now we've left the headers state and are either in H1_MSG_DATA or
* H1_MSG_CHUNK_SIZE.
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
*/
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
MINOR: h1: make the message parser support a null <hdr> argument This will allow some iterative calls to be made on incomplete messages without having to store all the headers.
2018-09-11 10:04:48 -04:00
if (slp && !skip_update)
MEDIUM: h1: make the parser support a pointer to a start line This will allow the parser to fill some extra fields like the method or status without having to store them permanently in the HTTP message. At this point however the parser cannot restart from an interrupted read.
2018-09-11 09:34:50 -04:00
*slp = sl;
MEDIUM: h1: let the caller pass the initial parser's state This way the caller controls if it's the request or response which has to be used, and it will allow to restart after an incomplete parsing.
2018-09-11 09:33:32 -04:00
h1m->state = state;
h1m->next = ptr - start + skip;
return h1m->next;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
http_msg_ood:
/* out of data at <ptr> during state <state> */
MINOR: h1: make the message parser support a null <hdr> argument This will allow some iterative calls to be made on incomplete messages without having to store all the headers.
2018-09-11 10:04:48 -04:00
if (slp && !skip_update)
MEDIUM: h1: make the parser support a pointer to a start line This will allow the parser to fill some extra fields like the method or status without having to store them permanently in the HTTP message. At this point however the parser cannot restart from an interrupted read.
2018-09-11 09:34:50 -04:00
*slp = sl;
MEDIUM: h1: let the caller pass the initial parser's state This way the caller controls if it's the request or response which has to be used, and it will allow to restart after an incomplete parsing.
2018-09-11 09:33:32 -04:00
h1m->state = state;
h1m->next = ptr - start + skip;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
return 0;
http_msg_invalid:
/* invalid message, error at <ptr> */
MINOR: h1: make the message parser support a null <hdr> argument This will allow some iterative calls to be made on incomplete messages without having to store all the headers.
2018-09-11 10:04:48 -04:00
if (slp && !skip_update)
MEDIUM: h1: make the parser support a pointer to a start line This will allow the parser to fill some extra fields like the method or status without having to store them permanently in the HTTP message. At this point however the parser cannot restart from an interrupted read.
2018-09-11 09:34:50 -04:00
*slp = sl;
MEDIUM: h1: let the caller pass the initial parser's state This way the caller controls if it's the request or response which has to be used, and it will allow to restart after an incomplete parsing.
2018-09-11 09:33:32 -04:00
h1m->err_state = h1m->state = state;
h1m->err_pos = h1m->next = ptr - start + skip;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
return -1;
http_output_full:
/* no more room to store the current header, error at <ptr> */
MINOR: h1: make the message parser support a null <hdr> argument This will allow some iterative calls to be made on incomplete messages without having to store all the headers.
2018-09-11 10:04:48 -04:00
if (slp && !skip_update)
MEDIUM: h1: make the parser support a pointer to a start line This will allow the parser to fill some extra fields like the method or status without having to store them permanently in the HTTP message. At this point however the parser cannot restart from an interrupted read.
2018-09-11 09:34:50 -04:00
*slp = sl;
MEDIUM: h1: let the caller pass the initial parser's state This way the caller controls if it's the request or response which has to be used, and it will allow to restart after an incomplete parsing.
2018-09-11 09:33:32 -04:00
h1m->err_state = h1m->state = state;
h1m->err_pos = h1m->next = ptr - start + skip;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
return -2;
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
restart:
BUG/MEDIUM: h1: Properly reset h1m flags when headers parsing is restarted If H1 headers are not fully received at once, the parsing is restarted a last time when all headers are finally received. When this happens, the h1m flags are sanitized to remove all value set during parsing. But some flags where erroneously preserved. Among others, H1_MF_TE_CHUNKED flag was not removed, what could lead to parsing error. To fix the bug and make things easy, a mask has been added with all flags that must be preserved. It will be more stable. This mask is used to sanitize h1m flags. This patch should fix the issue #1469. It must be backported to 2.5.
2021-12-01 12:01:48 -05:00
h1m->flags &= H1_MF_RESTART_MASK;
BUG/MINOR: h1: Properly reset h1m when parsing is restarted Otherwise some processing may be performed twice. For instance, if the header "Content-Length" is parsed on the first pass, when the parsing is restarted, we skip it because we think another header with the same value was already seen. In fact, it is currently the only existing bug that can be encountered. But it is safer to reset all the h1m on restart to avoid any future bugs. This patch must be backported to 2.0 and 1.9
2019-09-03 10:05:31 -04:00
h1m->curr_len = h1m->body_len = h1m->next = 0;
MEDIUM: h1: implement the request parser as well The original H1 request parsing code was reintroduced into the generic H1 parser so that it can be used regardless of the direction. If the parser is interrupted and restarts, it makes use of the H1_MF_RESP flag to decide whether to re-parse a request or a response. While parsing the request, the method is decoded and set into the start line structure.
2018-09-11 11:57:05 -04:00
if (h1m->flags & H1_MF_RESP)
h1m->state = H1_MSG_RPBEFORE;
else
h1m->state = H1_MSG_RQBEFORE;
MEDIUM: h1: support partial message parsing While it was not needed in the H2 mux which was reading full H1 messages from the channel, it is mandatory for the H1 mux reading contents from outside to be able to restart on a message. The problem is that the headers are indexed on the fly, and it's not fun to have to store everything between calls. The solution here is to complete the first pass doing a partial restart, and only once the end of message was found, to start over it again at once, filling entries. This way there is a bounded number of passes on the contents and no need to store an intermediary result anymore. Later this principle could even be used to decide to completely drop an output buffer to save memory.
2018-09-11 10:20:30 -04:00
goto try_again;
MEDIUM: h1: reimplement the http/1 response parser for the gateway The HTTP/2->HTTP/1 gateway will need to process HTTP/1 responses. We cannot sanely rely on the HTTP/1 txn to parse a response because : 1) responses generated by haproxy such as error messages, redirects, stats or Lua are neither parsed nor indexed ; this could be addressed over the long term but will take time. 2) the http txn is useless to parse the body : the states present there are only meaningful to received bytes (ie next bytes to parse) and not at all to sent bytes. Thus chunks cannot be followed at all. Even when implementing this later, it's unsure whether it will be possible when dealing with compression. So using the HTTP txn is now out of the equation and the only remaining solution is to call an HTTP/1 message parser. We already have one, it was slightly modified to avoid keeping states by benefitting from the fact that the response was produced by haproxy and this is entirely available. It assumes the following rules are true, or that incuring an extra cost to work around them is acceptable : - the response buffer is read-write and supports modifications in place - headers sent through / by haproxy are not folded. Folding is still implemented by replacing CR/LF/tabs/spaces with spaces if encountered - HTTP/0.9 responses are never sent by haproxy and have never been supported at all - haproxy will not send partial responses, the whole headers block will be sent at once ; this means that we don't need to keep expensive states and can afford to restart the parsing from the beginning when facing a partial response ; - response is contiguous (does not wrap). This was already the case with the original parser and ensures we can safely dereference all fields with (ptr,len) The parser replaces all of the http_msg fields that were necessary with local variables. The parser is not called on an http_msg but on a string with a start and an end. The HTTP/1 states were reused for ease of use, though the request-specific ones have not been implemented for now. The error position and error state are supported and optional ; these ones may be used later for bug hunting. The parser issues the list of all the headers into a caller-allocated array of struct ist. The content-length/transfer-encoding header are checked and the relevant info fed the h1 message state (flags + body_len).
2017-07-26 03:07:47 -04:00
}
MEDIUM: h1: add a WebSocket key on handshake if needed Add the header Sec-Websocket-Key when generating a h1 handshake websocket without this header. This is the case when doing h2-h1 conversion. The key is randomly generated and base64 encoded. It is stored on the session side to be able to verify response key and reject it if not valid.
2020-12-11 11:53:07 -05:00
/* Generate a random key for a WebSocket Handshake in respect with rfc6455
* The key is 128-bits long encoded as a base64 string in <key_out> parameter
* (25 bytes long).
*/
void h1_generate_random_ws_input_key(char key_out[25])
{
/* generate a random websocket key */
const uint64_t rand1 = ha_random64(), rand2 = ha_random64();
char key[16];
memcpy(key, &rand1, 8);
memcpy(&key[8], &rand2, 8);
a2base64(key, 16, key_out, 25);
}
MEDIUM: h1: generate WebSocket key on response if needed Add the Sec-Websocket-Accept header on a websocket handshake response. This header may be missing if a h2 server is used with a h1 client. The response key is calculated following the rfc6455. For this, the handshake request key must be stored in the h1 session, as a new field name ws_key. Note that this is only done if the message has been prealably identified as a Websocket handshake request.
2020-12-11 11:53:03 -05:00
#define H1_WS_KEY_SUFFIX_GUID "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
/*
* Calculate the WebSocket handshake response key from <key_in>. Following the
* rfc6455, <key_in> must be 24 bytes longs. The result is stored in <key_out>
* as a 29 bytes long string.
*/
void h1_calculate_ws_output_key(const char *key, char *result)
{
blk_SHA_CTX sha1_ctx;
char hash_in[60], hash_out[20];
/* concatenate the key with a fixed suffix */
memcpy(hash_in, key, 24);
memcpy(&hash_in[24], H1_WS_KEY_SUFFIX_GUID, 36);
/* sha1 the result */
blk_SHA1_Init(&sha1_ctx);
blk_SHA1_Update(&sha1_ctx, hash_in, 60);
blk_SHA1_Final((unsigned char *)hash_out, &sha1_ctx);
/* encode in base64 the hash */
a2base64(hash_out, 20, result, 29);
}
MEDIUM: h1: allow to preserve keep-alive on T-E + C-L In 2.5-dev9, commit 631c7e866 ("MEDIUM: h1: Force close mode for invalid uses of T-E header") enforced a recently arrived new security rule in the HTTP specification aiming at preventing a class of content-smuggling attacks involving HTTP/1.0 agents. It consists in handling the very rare T-E + C-L requests or responses in close mode. It happens it does have an impact of a rare few and very old clients (probably running insecure TLS stacks by the way) that continue to send both with their POST requests. The impact is that for each and every request they'll have to reconnect, possibly negotiating a full TLS handshake that becomes harmful to the machine in terms of CPU computation. This commit adds a new option "h1-do-not-close-on-insecure-transfer-encoding" that does exactly what it says, it just asks not to close on such messages, even though the message continues to be sanitized and C-L dropped. It means that the risk is only between the sender and haproxy, which is limited, and might be the only acceptable solution for such environments having to deal with broken implementations. The cases are so rare that it should not need to be backported, or in the worst case, to the latest LTS if there is any demand.
2024-07-26 09:14:58 -04:00
/* config parser for global "h1-do-not-close-on-insecure-transfer-encoding" */
static int cfg_parse_h1_do_not_close_insecure_t_e(char **args, int section_type, struct proxy *curpx,
const struct proxy *defpx, const char *file, int line,
char **err)
{
if (too_many_args(0, args, err, NULL))
return -1;
h1_do_not_close_on_insecure_t_e = 1;
return 0;
}
/* config keyword parsers */
static struct cfg_kw_list cfg_kws = {{ }, {
{ CFG_GLOBAL, "h1-do-not-close-on-insecure-transfer-encoding", cfg_parse_h1_do_not_close_insecure_t_e },
{ 0, NULL, NULL },
}};
INITCALL1(STG_REGISTER, cfg_register_keywords, &cfg_kws);
Reference in a new issue Copy permalink