This is a followup to our previous commit where we made convert_classic_histograms_to_nhcb
configurable at discovery time. We add the other native histogram labels
too here.
Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com>
promql: Add metric and stats to track total samples read per query
[FEATURE] PromQL: Expose per-query `samplesRead` (and `samplesReadPerStep` with `stats=all` and the `promql-per-step-stats` feature flag) in the query stats response, and add the `prometheus_engine_query_samples_read_total` engine counter. `samplesRead` reflects storage I/O distinct from `totalQueryableSamples`, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows).
[BUGFIX] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result.
[BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an `@` modifier (e.g. `predict_linear(metric[60s] @ T, X)`) silently under-counted `totalQueryableSamples` for steps after step 0.
Replace the deprecated SHA-1 hash used to generate rule group pagination
tokens in getRuleGroupNextToken with SHA-256. The token is opaque and
compared server-side only, so this does not change the API contract. Add
a unit test covering the token format and determinism.
Fixes#18841
Signed-off-by: Yash Anil <yashanil98@gmail.com>
Adding this as a meta label makes it possible to dynamically configure
this setting through discovery labels.
This helps in usecases like K8S where we could enable this with a pod
annotation.
Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com>
Add a per-query samples-read counter that distinguishes storage I/O
from the existing TotalSamples count. TotalSamples reports samples
loaded into the evaluator and so over-counts when a sample is reused
across multiple range-vector windows; samplesRead reflects what the
storage layer actually had to read.
For range-vector functions in range queries, samplesRead applies
delta semantics: step 0 counts the full window, later steps count
only the points not already present in the previous step's window.
For other query shapes samplesRead equals TotalSamples.
Expose the new counter:
- as `samplesRead` and `samplesReadPerStep` in /api/v1/query{,_range}
stats responses (the per-step variant only with stats=all and the
per-step stats feature flag enabled);
- as the `prometheus_engine_query_samples_read_total` Prometheus
counter, reporting the cumulative samples-read across all queries;
- in the OpenAPI schemas.
Also update the per-step stats feature flag and querying API
documentation, and add unit tests covering instant queries, range
queries, range-vector functions, and subqueries.
Signed-off-by: Dan Cech <dcech@grafana.com>
* web/api: reject 0 for limit and batch_size in search endpoints
Treat 0 as invalid for limit and batch_size query parameters; clients
must supply a positive integer or omit the parameter to use the server
default of 100. Update OpenAPI descriptions accordingly.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* Update web/api/v1/openapi.go
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: Julien <291750+roidelapluie@users.noreply.github.com>
---------
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Signed-off-by: Julien <291750+roidelapluie@users.noreply.github.com>
Co-authored-by: Arve Knudsen <arve.knudsen@gmail.com>
Rename the `min()` and `max()` duration expression functions to
`least()` and `greatest()` to avoid conflicts with the existing
PromQL aggregate functions `min` and `max`.
Update documentation and tests accordingly.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Without the return, a successful respond() call would always follow
the error response, writing a second body to an already-written
ResponseWriter and producing a malformed HTTP response.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
This adds a /api/v1/status/self_metrics endpoint that allows the frontend to
fetch metrics about the server itself, making it easier to construct frontend
pages that show the current server state. This is needed because fetching
metrics from its own /metrics endpoint would be both hard to parse and also
require CORS permissions on that endpoint (for cases where the frontend
dashboard is not the same origin, at least).
Signed-off-by: Julius Volz <julius.volz@gmail.com>
Parser configuration is now per-engine/API/loader and no longer uses package-level flags, so behavior is consistent and tests don't rely on save/restore of global variables.
Signed-off-by: Martin Valiente Ainz <64830185+tinitiuset@users.noreply.github.com>
Replace gopkg.in/yaml.v2 and gopkg.in/yaml.v3 imports with
go.yaml.in/yaml/v2 and go.yaml.in/yaml/v3 respectively.
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
* Add OpenAPI 3.2 specification generation for Prometheus HTTP API
This commit introduces an OpenAPI specification for the Prometheus API.
After testing multiple code-generation servers with built-in APIs, this
implementation uses an independent spec file outside of the critical path.
This spec file is tested with a framework present in this pull request.
The specification helps clients know which parameters they can use and is
served at /api/v1/openapi.yaml. The spec file will evolve with the
Prometheus API and has the same version number.
Downstream projects can tune the APIs presented in the spec file with
configuration options using the IncludePaths setting for path filtering.
In the future, there is room to generate a server from this spec file
(e.g. with interfaces), but this is out of scope for this pull request.
Architecture:
- Core OpenAPI infrastructure (openapi.go): Dynamic spec building,
caching, and thread-safe spec generation
- Schema definitions (openapi_schemas.go): Complete type definitions
for all API request and response types
- Path specifications (openapi_paths.go): Endpoint definitions with
parameters, request bodies, and response schemas
- Examples (openapi_examples.go): Realistic request/response examples
- Helper functions (openapi_helpers.go): Reusable builders for common
OpenAPI structures
Testing:
- Comprehensive test suite with golden file validation
- Test helpers package for API testing infrastructure
- OpenAPI compliance validation utilities
The golden file captures the complete specification for snapshot testing.
Update with: go test -run TestOpenAPIGolden -update-openapi-spec
REVIEWERS: The most important thing to check would be the OpenAPI golden
file (web/api/v1/testdata/openapi_golden.yaml). Test scenarios are important
as they test the actual OpenAPI spec validity.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* Add OpenAPI 3.1 support with version selection
Add support for both OpenAPI 3.1 and 3.2 specifications with version
selection via openapi_version query parameter. Defaults to 3.1 for
broader compatibility
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* Enhance OpenAPI examples and add helper functions
- Add timestampExamples helper for consistent time formatting
- Add exampleMap helper to simplify example creation
- Improve example summaries with query details
- Add matrix result example for range vector queries
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* web/api: Add AtST method to test helper iterators
Implement the AtST() method required by chunkenc.Iterator interface
for FakeSeriesIterator and FakeHistogramSeriesIterator test helpers.
The method returns 0 as these test helpers don't use start timestamps
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* OpenAPI: Add minimum coverage test
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* OpenAPI: Improve examples handling
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
---------
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Add a maximum limit of 10,000 to the TSDB status endpoint to prevent
resource exhaustion from excessively large limit values, as we preallocate
[]Stat for up to the limit: `make([]Stat, 0, length)`.
Note that the endpoint acquires a cardinality mutex during
stats calculation, so this can not be run in parallel.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Partially fixes https://github.com/prometheus/prometheus/issues/17416 by
renaming all CT* names to ST* in the whole codebase except RW2 (this is
done in separate
[PR](https://github.com/prometheus/prometheus/pull/17411)) and
PrometheusProto exposition proto.
```
CreatedTimestamp -> StartTimestamp
CreatedTimeStamp -> StartTimestamp
created_timestamp -> start_timestamp
CT -> ST
ct -> st
```
Signed-off-by: bwplotka <bwplotka@gmail.com>
OTLP Receiver: Only update metadata to WAL when metadata-wal-records feature is enabled.
---------
Signed-off-by: pipiland2612 <nguyen.t.dang.minh@gmail.com>
