Add a new optional histogram_format query parameter to /api/v1/query
and /api/v1/query_range. When set to native, native histograms in the
JSON response are emitted in a schema-aware shape.
For exponential schemas:
{
"count": "<count>",
"sum": "<sum>",
"schema": <int>,
"zero_threshold": "<threshold>",
"zero_count": "<count>", // when non-zero
"negative_buckets": [ [ <index>, "<count>" ], ... ], // when non-empty
"buckets": [ [ <index>, "<count>" ], ... ] // when non-empty
}
zero_threshold is always emitted because, together with schema, it
fully describes the bucket layout. The negative/positive bucket
arrays carry the schema-defined bucket index and count, so empty
buckets that fall between non-empty ones are not transmitted.
For custom-bucket histograms (schema -53):
{
"count": "<count>",
"sum": "<sum>",
"schema": -53,
"boundaries": [ "<upper_bound>", ... ],
"buckets": [ [ <index>, "<count>" ], ... ]
}
boundaries lists the bucket upper bounds; each bucket entry's index
is the 0-based index into boundaries.
This lets clients that already understand the schema (e.g. Grafana
rendering NHCB heatmaps) reconstruct every bucket including empty
custom-bucket slots, and is also substantially more compact than the
boundary-based representation for histograms with many buckets.
The existing boundary-based representation is unchanged when
histogram_format is absent or set to any other value.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Replace the catch-all default branch in encodeToSnapshotRecord and
decodeSeriesFromChunkSnapshot with an explicit EncFloatHistogram case and a
default that panics (encode) or returns an error (decode), making unknown
encodings immediately visible rather than silently mishandling them.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
EncXOR2 is a float encoding and must be treated like EncXOR in all
places that enumerate chunk types:
- compact.go: NumFloatSamples was not incremented for EncXOR2 chunks
during compaction, leading to under-reported block stats.
- head_wal.go: encodeToSnapshotRecord fell through to the default
(FloatHistogram) branch for EncXOR2 head chunks, which would corrupt
chunk snapshots; the decode path already handled EncXOR2 correctly.
- ooo_head.go: update stale comment to mention EncXOR2.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* web/api: reject 0 for limit and batch_size in search endpoints
Treat 0 as invalid for limit and batch_size query parameters; clients
must supply a positive integer or omit the parameter to use the server
default of 100. Update OpenAPI descriptions accordingly.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* Update web/api/v1/openapi.go
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: Julien <291750+roidelapluie@users.noreply.github.com>
---------
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Signed-off-by: Julien <291750+roidelapluie@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Add t.Parallel() to the 13 top-level test functions in marathon_test.go.
Tests have no shared mutable state — each test creates its own registry,
metrics, and config — so parallelisation is safe and speeds up the suite.
Refs #15185
Signed-off-by: Ogulcan Aydogan <ogulcanaydogan@hotmail.com>
Each test creates its own httptest.Server and prometheus.Registry so
there is no shared global state between them. Adding t.Parallel() to
all 13 top-level test functions and the subtests in TestUnmarshalConfig
allows the Go test runner to overlap them, cutting wall-clock time.
Refs: #15185
Signed-off-by: Ogulcan Aydogan <ogulcanaydogan@hotmail.com>
Signed-off-by: Kai Tanaka <275430420+quyentonndbs@users.noreply.github.com>
Co-authored-by: Kai Tanaka <275430420+quyentonndbs@users.noreply.github.com>
- Fix copy-paste bug: smoothed branch in bare MatrixSelector eval now
says "smoothed modifier is not supported with histograms" instead of
"anchored".
- Change annosFromInterpolationError to take *annotations.Annotations,
consistent with all other histogram helper functions.
- Rename delta to result in interpolateHistograms to avoid the misleading
name (the variable holds the interpolated result, not a delta).
- Fix extendedRate doc comment: interpolates at boundaries, not
extrapolates.
- Fix stale comment in correctForCounterResetsHistogram that referenced
the nonexistent variable leftInterpolationHandledReset.
- Add comment explaining why right.Copy() is not redundant in
extendedHistogramRate.
- Hoist s.Labels().Get(labels.MetricName) out of the per-step loop in
smoothSeries; was called up to three times per step.
- Reset CounterResetHint to UnknownCounterReset on carry-forward
histograms in smoothSeries; a Reset hint at a new timestamp is
semantically misleading.
- Add TestInterpolateHistograms unit test covering exact-match, midpoint,
counter, reset, and quarter-point cases.
- Add rate(hist_counter[1m] anchored) test.
- Add delta(hist_counter[1m] anchored/smoothed) tests with
NativeHistogramNotGaugeWarning assertions.
- Add carry-forward test for smoothed vector selector.
- Add range eval test for rate(hist_counter[1m] smoothed).
- Add right-boundary reset interpolation test (right_boundary_reset).
- Fix stale test comment about "prev initialPrev path".
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Implement least(a, b) and greatest(a, b) as scalar-returning PromQL
functions backed by math.Min and math.Max respectively.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
The previous split required callers to compute the inner slice before
calling correctForCounterResetsHistogram, which led to a bug where
h[lastSampleIndex] could be double-counted when it carries a reset hint
that equals the right boundary value.
Merge the boundary logic into correctForCounterResetsHistogram so the
function owns its own loop range, always excludes lastSampleIndex from
the inner loop, and initialises prev to h[firstSampleIndex+1] when the
left-boundary interpolation already spanned the first reset.
Add regression tests covering: reset at the last sample, two-sample
anchored windows, double-reset in smoothed mode, and the two-sample
smoothed window where both boundaries are interpolated from the same
reset segment.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Each test function creates its own mock AWS client and operates on
independent data stores with no shared global state between them.
Adding t.Parallel() to the 33 top-level test functions across the
six test files (aws, ec2, ecs, elasticache, msk, rds) allows the
Go test runner to overlap their execution, cutting wall-clock time.
TestLoadRegion is excluded because its subtests use t.Setenv, which
panics when a parallel ancestor is detected (Go 1.25+).
Refs: #15185
Signed-off-by: Ogulcan Aydogan <ogulcanaydogan@hotmail.com>
The oversize-chunk trigger introduced in #18692 was implemented as a
closure defined inside the per-sample loop in
populateChunksFromIterable and invoked once at the `if` condition.
Replace it with a plain conditional and hoist
`len(currentChunk.Bytes())` out of the switch so the two encoding
cases don't repeat the same expression. The new shape preserves the
original `||` short-circuit: the size check is only evaluated when
neither the encoding nor the start-timestamp capability forces a new
chunk, which also keeps `currentChunk` non-nil at the point of read.
`gcflags=-m=2` reports the closure body inlined and the symbol table
shows no separate `func1` symbol, yet benchstat shows a measurable
speedup. The most likely explanation: the closure body inlines, but
the `funcval` struct (capturing `currentChunk` and `currentValueType`)
is still stack-constructed each iteration — invisible to escape
analysis, but a real per-iteration cost in a hot loop.
Benchmark, `go test -count=6 -benchmem -bench=BenchmarkQuerierSelectWithOutOfOrder -benchtime=5s -run=^$ ./tsdb/`,
Intel Xeon Platinum 8280 @ 2.70 GHz (linux/amd64), 1M-series head,
query selectivity varies:
│ main │ optimized │
│ sec/op │ sec/op vs base │
Head/1of1000000-16 301.5m ± 4% 257.0m ± 4% -14.74% (p=0.002 n=6)
Head/10of1000000-16 305.6m ± 3% 260.4m ± 2% -14.80% (p=0.002 n=6)
Head/100of1000000-16 303.9m ± 2% 259.7m ± 2% -14.54% (p=0.002 n=6)
Head/1000of1000000-16 303.8m ± 2% 267.0m ± 2% -12.13% (p=0.002 n=6)
Head/10000of1000000-16 318.1m ± 1% 278.9m ± 8% -12.33% (p=0.002 n=6)
Head/100000of1000000-16 364.1m ± 7% 352.8m ± 4% ~ (p=0.065 n=6)
Head/1000000of1000000-16 1.115 ± 2% 1.089 ± 26% ~ (p=0.394 n=6)
geomean 377.8m 337.3m -10.71%
allocs/op and B/op unchanged. The two largest-selectivity cases trend
faster but are dominated by the per-sample append cost so the relative
delta is smaller and lost in variance.
`TestChunkQuerier_OverlappingInOrderAndOOOChunks` continues to
exercise the overflow path.
```release-notes
NONE
```
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
* fix(tsdb): chunk overflow on ooo query
Protect against and fix overflow of chunks with more than 2^16-1 samples
in case we're recoding chunks due to for example in-order and ooo samples
overlap during compaction or query.
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
* tsdb: extract stripeSeries.refStripe helper
Extract the repeated ref-to-stripe-index calculation into a method on
stripeSeries, replacing five inline copies that used two different
casting styles (int and uint64). The helper computes with uint64
internally so it is correct on 32-bit architectures.
* tsdb: skip entire stripes in mmapHeadChunks via per-stripe ready count
Add a per-stripe mmapReady counter to stripeSeries that tracks how many
series in each stripe have headChunkCount >= 2 (i.e., are ready for
mmapping). mmapHeadChunks skips stripes where the counter is zero,
avoiding the RLock and map iteration entirely.
---------
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
