diff --git a/addons/otel/README.md b/addons/otel/README.md new file mode 100644 index 000000000..07ea7180b --- /dev/null +++ b/addons/otel/README.md @@ -0,0 +1,288 @@ +## HAProxy OpenTelemetry Filter (OTel) + +The OTel filter enables HAProxy to emit telemetry data -- traces, metrics and +logs -- to any OpenTelemetry-compatible backend via the OpenTelemetry protocol +(OTLP). + +It is the successor to the OpenTracing (OT) filter, built on the OpenTelemetry +standard which unifies distributed tracing, metrics and logging into a single +observability framework. + +### Features + +- **Distributed tracing** -- spans with parent-child relationships, context + propagation via HTTP headers or HAProxy variables, links, baggage and status. +- **Metrics** -- counter, histogram, up-down counter and gauge instruments with + configurable aggregation and bucket boundaries. +- **Logging** -- log records with severity levels, optional span correlation and + runtime-evaluated attributes. +- **Rate limiting** -- percentage-based sampling (0.0--100.0) for controlling + overhead. +- **ACL integration** -- fine-grained conditional execution at instrumentation, + scope and event levels. +- **CLI management** -- runtime enable/disable, rate adjustment, error mode + switching and status inspection. +- **Context propagation** -- inject/extract span contexts between cascaded + HAProxy instances or external services. + +### Dependencies + +The filter requires the +[OpenTelemetry C Wrapper](https://github.com/haproxytech/opentelemetry-c-wrapper) +library, which wraps the OpenTelemetry C++ SDK. + +### Building + +The OTel filter is compiled together with HAProxy by adding `USE_OTEL=1` to the +make command. + +#### Using pkg-config + +``` +PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 TARGET=linux-glibc +``` + +#### Explicit paths + +``` +make -j8 USE_OTEL=1 OTEL_INC=/opt/include OTEL_LIB=/opt/lib TARGET=linux-glibc +``` + +#### Build options + +| Variable | Description | +|-----------------|-----------------------------------------------------| +| `USE_OTEL` | Enable the OpenTelemetry filter | +| `OTEL_DEBUG` | Compile in debug mode | +| `OTEL_INC` | Force path to opentelemetry-c-wrapper include files | +| `OTEL_LIB` | Force path to opentelemetry-c-wrapper library | +| `OTEL_RUNPATH` | Add opentelemetry-c-wrapper RUNPATH to executable | +| `OTEL_USE_VARS` | Enable context propagation via HAProxy variables | + +#### Debug mode + +``` +PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 OTEL_DEBUG=1 TARGET=linux-glibc +``` + +#### Variable-based context propagation + +``` +PKG_CONFIG_PATH=/opt/lib/pkgconfig make -j8 USE_OTEL=1 OTEL_USE_VARS=1 TARGET=linux-glibc +``` + +#### Verifying the build + +``` +./haproxy -vv | grep -i opentelemetry +``` + +If the filter is built in, the output contains: + +``` +Built with OpenTelemetry support (C++ version 1.26.0, C Wrapper version 1.0.0-842). + [OTEL] opentelemetry +``` + +#### Library path at runtime + +When pkg-config is not used, the executable may not find the library at startup. +Use `LD_LIBRARY_PATH` or build with `OTEL_RUNPATH=1`: + +``` +LD_LIBRARY_PATH=/opt/lib ./haproxy ... +``` + +``` +make -j8 USE_OTEL=1 OTEL_RUNPATH=1 OTEL_INC=/opt/include OTEL_LIB=/opt/lib TARGET=linux-glibc +``` + +### Configuration + +The filter uses a two-file configuration model: + +1. **OTel configuration file** (`.cfg`) -- defines the telemetry model: + instrumentation settings, scopes and groups. +2. **YAML configuration file** (`.yml`) -- defines the OpenTelemetry SDK + pipeline: exporters, samplers, processors, providers and signal routing. + +#### Activating the filter + +The OTel filter requires the `insecure-fork-wanted` keyword in the HAProxy +`global` section. This is necessary because the OpenTelemetry C++ SDK creates +background threads for data export and batch processing. HAProxy will refuse +to load the configuration if this keyword is missing. + +``` +global + insecure-fork-wanted + ... +``` + +Add the filter to a HAProxy proxy section (frontend/listen/backend): + +``` +frontend my-frontend + ... + filter opentelemetry [id ] config + ... +``` + +If no filter id is specified, `otel-filter` is used as default. + +#### OTel configuration file structure + +The OTel configuration file contains three section types: + +- `otel-instrumentation` -- mandatory; references the YAML file, sets rate + limits, error modes, logging and declares groups and scopes. +- `otel-scope` -- defines actions (spans, attributes, metrics, logs) triggered + by stream events or from groups. +- `otel-group` -- a named collection of scopes triggered from HAProxy TCP/HTTP + rules. + +#### Minimal YAML configuration + +```yaml +exporters: + my_exporter: + type: otlp_http + endpoint: "http://localhost:4318/v1/traces" + +samplers: + my_sampler: + type: always_on + +processors: + my_processor: + type: batch + +providers: + my_provider: + resources: + - service.name: "haproxy" + +signals: + traces: + scope_name: "HAProxy OTel" + exporters: my_exporter + samplers: my_sampler + processors: my_processor + providers: my_provider +``` + +#### Supported YAML exporters + +| Type | Description | +|-----------------|---------------------------------------| +| `otlp_grpc` | OTLP over gRPC | +| `otlp_http` | OTLP over HTTP (JSON or Protobuf) | +| `otlp_file` | Local files in OTLP format | +| `zipkin` | Zipkin-compatible backends | +| `elasticsearch` | Elasticsearch | +| `ostream` | Text output to a file (for debugging) | +| `memory` | In-memory buffer (for testing) | + +### Scope keywords + +| Keyword | Description | +|----------------|---------------------------------------------------------| +| `span` | Create or reference a span | +| `attribute` | Set key-value span attributes | +| `event` | Add timestamped span events | +| `baggage` | Set context propagation data | +| `status` | Set span status (ok/error/ignore/unset) | +| `link` | Add span links to related spans | +| `inject` | Inject context into headers or variables | +| `extract` | Extract context from headers or variables | +| `finish` | Close spans (supports wildcards: `*`, `*req*`, `*res*`) | +| `instrument` | Create or update metric instruments | +| `log-record` | Emit a log record with severity | +| `otel-event` | Bind scope to a filter event with optional ACL | +| `idle-timeout` | Set periodic event interval for idle streams | + +### CLI commands + +Available via the HAProxy CLI socket (prefix: `flt-otel`): + +| Command | Description | +|----------------------------|------------------------------------| +| `flt-otel status` | Show filter status | +| `flt-otel enable` | Enable the filter | +| `flt-otel disable` | Disable the filter | +| `flt-otel hard-errors` | Enable hard-errors mode | +| `flt-otel soft-errors` | Disable hard-errors mode | +| `flt-otel logging [state]` | Set logging state | +| `flt-otel rate [value]` | Set or show the rate limit | +| `flt-otel debug [level]` | Set debug level (debug build only) | + +When invoked without arguments, `rate`, `logging` and `debug` display the +current value. + +### Performance + +Benchmark results from the standalone (`sa`) configuration, which exercises all +events (worst-case scenario): + +| Rate limit | Req/s | Avg latency | Overhead | +|------------|--------|-------------|----------| +| 100.0% | 38,202 | 213.08 us | 21.6% | +| 50.0% | 42,777 | 190.49 us | 12.2% | +| 25.0% | 45,302 | 180.46 us | 7.0% | +| 10.0% | 46,879 | 174.69 us | 3.7% | +| 2.5% | 47,993 | 170.58 us | 1.4% | +| disabled | 48,788 | 167.74 us | ~0 | +| off | 48,697 | 168.00 us | baseline | + +With a rate limit of 10% or less, the performance impact is negligible. +Detailed methodology and additional results are in the `test/` directory. + +### Test configurations + +The `test/` directory contains ready-to-run example configurations: + +- **sa** -- standalone; the most comprehensive example, demonstrating spans, + attributes, events, links, baggage, status, metrics, log records, ACL + conditions and idle-timeout events. +- **fe/be** -- distributed tracing across two cascaded HAProxy instances using + HTTP header-based context propagation. +- **ctx** -- context propagation via HAProxy variables using the inject/extract + mechanism. +- **cmp** -- minimal configuration for benchmarking comparison. +- **empty** -- filter initialized with no active telemetry. + +#### Quick start with Jaeger + +Start a Jaeger all-in-one container: + +``` +docker run -d --name jaeger -p 4317:4317 -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one:latest +``` + +Run one of the test configurations: + +``` +./test/run-sa.sh +``` + +Open the Jaeger UI at `http://localhost:16686` to view traces. + +### Documentation + +Detailed documentation is available in the following files: + +- [README](README) -- complete reference documentation +- [README-configuration](README-configuration) -- configuration guide +- [README-conf](README-conf) -- configuration details +- [README-design](README-design) -- cross-cutting design patterns +- [README-implementation](README-implementation) -- component architecture +- [README-func](README-func) -- function reference +- [README-misc](README-misc) -- miscellaneous notes + +### Copyright + +Copyright 2026 HAProxy Technologies + +### Author + +Miroslav Zagorac