From 3f68eea1fd10ae84c54dcd5e55b9d0a363255205 Mon Sep 17 00:00:00 2001 From: Yonas Habteab Date: Tue, 24 Mar 2026 12:07:42 +0100 Subject: [PATCH] Reduce default `flush_threshold` to `16MiB` So that it doesn't cause `request body too large` errors when used with the default OpenTelemetry Collector config that has `max_request_body_size` set to `20MiB`. --- doc/09-object-types.md | 11 ++++++++++- etc/icinga2/features-available/otlpmetrics.conf | 2 +- lib/perfdata/otlpmetricswriter.ti | 2 +- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/doc/09-object-types.md b/doc/09-object-types.md index 4cc45d6c3..02d4cf9a2 100644 --- a/doc/09-object-types.md +++ b/doc/09-object-types.md @@ -1901,7 +1901,7 @@ There are more configuration options available as described in the table below. | host\_resource\_attributes | Dictionary | **Optional.** Additional resource attributes to be included with host metrics. Defaults to none. | | service\_resource\_attributes | Dictionary | **Optional.** Additional resource attributes to be included with service metrics. Defaults to none. | | flush\_interval | Duration | **Optional.** How long to buffer data points before transferring to the OTLP backend. Defaults to `15s`. | -| flush\_threshold | Number | **Optional.** How many bytes to buffer before forcing a transfer to the OTLP backend. Defaults to `32MiB`. | +| flush\_threshold | Number | **Optional.** How many bytes to buffer before forcing a transfer to the OTLP backend. Defaults to `16MiB`. | | enable\_ha | Boolean | **Optional.** Enable the high availability functionality. Has no effect in non-cluster setups. Defaults to `false`. | | enable\_send\_thresholds | Boolean | **Optional.** Whether to stream warning, critical, minimum & maximum as separate metrics to the OTLP backend. Defaults to `false`. | | diconnect\_timeout | Duration | **Optional.** Timeout to wait for any outstanding data to be flushed to the OTLP backend before disconnecting. Defaults to `10s`. | @@ -1911,6 +1911,15 @@ There are more configuration options available as described in the table below. | tls\_cert\_file | String | **Optional.** Path to the client certificate to present to the OTLP backend for mutual verification. | | tls\_key\_file | String | **Optional.** Path to the client certificate key. | +!!! tip + + The `flush_threshold` is a byte size threshold, not a metric count threshold. By default, the writer will flush all + buffered metrics to the OTLP backend once the total size of buffered metrics exceeds 16 MiB. This number is chosen + based on the default `max_request_body_size` of the OpenTelemetry Collector, and you must adjust it according to the + `max_request_body_size` of your OTLP backend to avoid metrics being dropped due to exceeding the maximum request body + size. Furthermore, the writer may not flush at the exact byte size threshold due to the internal structure of OTLP + messages, so make sure that the threshold is lower than the configured `max_request_body_size` of your OTLP backend. + ### PerfdataWriter Writes check result performance data to a defined path using macro diff --git a/etc/icinga2/features-available/otlpmetrics.conf b/etc/icinga2/features-available/otlpmetrics.conf index a4ccf6031..bd5af67a2 100644 --- a/etc/icinga2/features-available/otlpmetrics.conf +++ b/etc/icinga2/features-available/otlpmetrics.conf @@ -34,7 +34,7 @@ object OTLPMetricsWriter "otlp-metrics" { # Please refer to the documentation for more details on each option. // enable_ha = false // flush_interval = 15s - // flush_threshold = 32*1024*1024 + // flush_threshold = 16*1024*1024 # When stopping Icinga 2, this timeout defines how long to wait for any pending OTel # metrics to be sent before disconnecting and discarding them. // disconnect_timeout = 10s diff --git a/lib/perfdata/otlpmetricswriter.ti b/lib/perfdata/otlpmetricswriter.ti index 214b12baa..7e3ba9ea1 100644 --- a/lib/perfdata/otlpmetricswriter.ti +++ b/lib/perfdata/otlpmetricswriter.ti @@ -35,7 +35,7 @@ class OTLPMetricsWriter : ConfigObject default {{{ return 15; }}} }; [config] int64_t flush_threshold { - default {{{ return 32 * 1024 * 1024; }}} + default {{{ return 16 * 1024 * 1024; }}} }; [config] bool enable_ha { default {{{ return false; }}}