* feat(promtool): Add flag --remote-write.path=/api/v1/write
* docs(promtool): Generate cli docs for new flag
Signed-off-by: Steffen Baarsgaard <steff.bpoulsen@gmail.com>
Co-authored-by: Varsha U N <varshaun58@gmail.com>
Remove the Experimental flag from first_over_time and move it into the
stable *_over_time function lists in the docs and UI function reference.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
* Add histogram chunk encoding with Start Timestamp support
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update documentation
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Add more tests
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix for appendHistogram
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Linting
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix start timestamp for histogram samples in TestCompactingChunkSeriesMerger
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Refactor and reduce code duplication
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Move counter reset hint bits into sample count field
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix bug with counter reset header
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Clean up docs
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Address feedback
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Lint
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Refactor to use generic appenders
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Update to include changes to histogram appenders
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Address feedback
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* tsdb: add histograms-st-encoding feature flag (#18912)
* tsdb: add histograms-st-encoding feature flag
Decouple the ST-capable histogram chunk encodings (EncHistogramST,
EncFloatHistogramST) from the xor2-encoding feature flag and gate them
behind a new histograms-st-encoding flag instead. Previously
--enable-feature=xor2-encoding turned on both the float XOR2 encoding
and the histogram ST encodings via a single useXOR2 boolean; that
conflated two independent on-disk formats and made it impossible to opt
into one without the other.
ValueType.ChunkEncoding and NewChunk now take separate useXOR2 and
useHistogramST flags. Dispatch only consumes the relevant flag per
value type; data-driven callers that gate on actual ST presence in the
samples (storage/series.go, tsdb/querier.go, tsdb/chunks/chunks.go)
pass hasST for both. tsdb.Options gains EnableHistogramSTEncoding;
tsdb.HeadOptions gains a matching atomic.Bool. The new flag is
propagated through chunkOpts/headAppenderBase and read at every site
that currently reads EnableXOR2Encoding (head append v1/v2, head_wal
WAL/WBL replay, ooo_head_read).
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
* Address feedback
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix histogram tests
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fix histogram ST PromQL tests
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
* Fixes for promQL tests
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
---------
Signed-off-by: Carrie Edwards <edwrdscarrie@gmail.com>
Signed-off-by: György Krajcsovits <gyorgy.krajcsovits@grafana.com>
Signed-off-by: George Krajcsovits <krajorama@users.noreply.github.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
Duration expressions no longer require the promql-duration-expr feature
flag, which is now a no-op in both prometheus and promtool. The parser
option (ExperimentalDurationExpr) is kept and enabled by default in the
binaries, so downstream consumers of the parser can still keep it off.
Move the documentation from the feature flags page into the PromQL
querying basics.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
The SD only read the primary VNIC's IPv4 PrivateIp/PublicIp, so dual-stack
instances could not be discovered over IPv6 and the VNIC hostname label was
not available for relabeling.
Expose two new meta labels taken straight from the OCI VNIC:
__meta_oci_ipv6_addresses comma-surrounded list of the primary VNIC's
IPv6 addresses, sorted for deterministic output
__meta_oci_hostname_label the primary VNIC's hostname label
OCI's API names the IPv4 fields privateIp/publicIp (no v4 qualifier) and the
v6 field ipv6Addresses, so the existing private_ip/public_ip labels are kept
as-is rather than renamed.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Lowercase the validate()/NewDiscovery errors with an oci_sd: prefix and resolve only the primary VNIC so multi-VNIC instances no longer fan out a GetVnic per attachment, dropping the secondary_* labels. Lower the rate cap back to 10 rps to stay under OCI's 20/s IAM limit, and drop the unneeded read tag-namespaces policy from the docs.
Signed-off-by: Amaan Ul Haq Siddiqui <amaanulhaq.s@outlook.com>
Revert rate_limit_rps from a YAML field back to an internal constant
(500 RPS, burst 50). No other cloud SD exposes an API rate knob and
the SDK's retry policy already handles per-tenancy quota variance.
Signed-off-by: Amaan Ul Haq Siddiqui <amaanulhaq.s@outlook.com>
Drop the filter knob, document rate_limit_rps, and note that the
secondary VNIC IP labels are sorted and scalar defined-tag values are
stringified.
Signed-off-by: Amaan Ul Haq Siddiqui <amaanulhaq.s@outlook.com>
Compaction creates a new block before deleting the source blocks, so on-disk size can briefly exceed `storage.tsdb.retention.size` until the next retention cleanup removes the source blocks. The current docs describe the setting as a hard limit and the right-sizing recommendation as a heuristic, without explaining the mechanism.
Document the temporary overhead in the Compaction section and point to it from the Right-Sizing Retention Size section, so the 80-85% recommendation has a clear basis.
Verified against tsdb/compact.go and tsdb/db.go: write() creates the new block in a tmp directory and atomically renames it; source blocks are marked Deletable=true rather than deleted; BeyondSizeRetention sums Head().Size() plus block.Size() for every persistent block.
Signed-off-by: Lohit Kolluri <lohitkolluri@gmail.com>
Document __meta_oci_vnic_id and the secondary VNIC IP list labels,
describe key_passphrase_file, and replace the tag_filter section with
a pointer to relabel_configs. Note that skipped compartments are now
logged rather than silently dropped.
Signed-off-by: Amaan Ul Haq Siddiqui <amaanulhaq.s@outlook.com>
Documents the embedded HTTP options, drops rate_limit_rps, and notes case-preserving tag sanitization.
Signed-off-by: Amaan Ul Haq Siddiqui <amaanulhaq.s@outlook.com>
This is a followup to our previous commit where we made convert_classic_histograms_to_nhcb
configurable at discovery time. We add the other native histogram labels
too here.
Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com>
promql: Add metric and stats to track total samples read per query
[FEATURE] PromQL: Expose per-query `samplesRead` (and `samplesReadPerStep` with `stats=all` and the `promql-per-step-stats` feature flag) in the query stats response, and add the `prometheus_engine_query_samples_read_total` engine counter. `samplesRead` reflects storage I/O distinct from `totalQueryableSamples`, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows).
[BUGFIX] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result.
[BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an `@` modifier (e.g. `predict_linear(metric[60s] @ T, X)`) silently under-counted `totalQueryableSamples` for steps after step 0.
- Rename chunkenc.Compatible to CompatibleValues and document that it
concerns sample-value encoding only; ST compatibility is the caller's
responsibility. Update callers and tests.
- Use audience-neutral wording in validateOpts errors (the tsdb package is
also consumed as a library) instead of Prometheus-only config-key names.
- Correct docs/feature_flags.md: with st-storage active and a resolved XOR
encoding, Prometheus fails configuration validation rather than warning.
- Drop the unrelated LICENSE-check change from scripts/sync_repo_files.sh.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
--enable-feature=xor2-encoding sets tsdb.Options.FloatChunkEncoding to
EncXOR2 (a concrete encoding value, not a feature-flag bool). ApplyConfig
uses this as the fallback when chunk_encoding.floats is absent. No global
mutable state is introduced in the config package.
Signed-off-by: Julien Pivotto <julien.pivotto@grafana.com>
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Adding this as a meta label makes it possible to dynamically configure
this setting through discovery labels.
This helps in usecases like K8S where we could enable this with a pod
annotation.
Signed-off-by: Michael Hoffmann <mhoffmann@cloudflare.com>
Add a per-query samples-read counter that distinguishes storage I/O
from the existing TotalSamples count. TotalSamples reports samples
loaded into the evaluator and so over-counts when a sample is reused
across multiple range-vector windows; samplesRead reflects what the
storage layer actually had to read.
For range-vector functions in range queries, samplesRead applies
delta semantics: step 0 counts the full window, later steps count
only the points not already present in the previous step's window.
For other query shapes samplesRead equals TotalSamples.
Expose the new counter:
- as `samplesRead` and `samplesReadPerStep` in /api/v1/query{,_range}
stats responses (the per-step variant only with stats=all and the
per-step stats feature flag enabled);
- as the `prometheus_engine_query_samples_read_total` Prometheus
counter, reporting the cumulative samples-read across all queries;
- in the OpenAPI schemas.
Also update the per-step stats feature flag and querying API
documentation, and add unit tests covering instant queries, range
queries, range-vector functions, and subqueries.
Signed-off-by: Dan Cech <dcech@grafana.com>
Signed-off-by: Kai Tanaka <275430420+quyentonndbs@users.noreply.github.com>
Co-authored-by: Kai Tanaka <275430420+quyentonndbs@users.noreply.github.com>
Implement least(a, b) and greatest(a, b) as scalar-returning PromQL
functions backed by math.Min and math.Max respectively.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
- Clear stCache state in scrape loop when append fails for existing series.
- Copy float histogram before storing in cache to avoid mutation.
- Add test for state mutation on OOO failure.
- Update docs to reflect behavior on failure.
Signed-off-by: Ridwan Sharif <ridwanmsharif@google.com>
Rename the `min()` and `max()` duration expression functions to
`least()` and `greatest()` to avoid conflicts with the existing
PromQL aggregate functions `min` and `max`.
Update documentation and tests accordingly.
Signed-off-by: Julien Pivotto <291750+roidelapluie@users.noreply.github.com>
Adds CheckpointFromInMemorySeries option for agent.Options to enable a faster checkpoint implementation that skips segment re-read and just uses in-memory data instead.
* feat: impl agent-specific checkpoint dir
* feat: impl ActiveSeries interface
* feat: use new checkpoint impl
* feat: hide new checkpoint impl behind a feature flag
* feat: add benchmark
* feat: add benchstat case
* feat: use feature flag in bench
* feat: use same labels for persisted state and append
* feat: set WAL segment size
* feat: add checkpoint size metric and bump series size
* feat: wal replay test
* feat: expose new checkpoint opts in cmd flags
* feat: update cli doc
* add ActiveSeries and DeletedSeries doc
Signed-off-by: x1unix <9203548+x1unix@users.noreply.github.com>
Signed-off-by: Denys Sedchenko <9203548+x1unix@users.noreply.github.com>
Co-authored-by: George Krajcsovits <krajorama@users.noreply.github.com>
