From f5971bf292aa40547e0ba1a718212493a76382b4 Mon Sep 17 00:00:00 2001 From: Antoine Pultier <45740+fungiboletus@users.noreply.github.com> Date: Wed, 4 Sep 2024 14:57:30 +0200 Subject: [PATCH 1/3] tsdb documenation: More details about chunks Signed-off-by: Antoine Pultier <45740+fungiboletus@users.noreply.github.com> --- tsdb/docs/format/chunks.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/tsdb/docs/format/chunks.md b/tsdb/docs/format/chunks.md index 8318e0a540..8a35721358 100644 --- a/tsdb/docs/format/chunks.md +++ b/tsdb/docs/format/chunks.md @@ -29,15 +29,16 @@ in-file offset (lower 4 bytes) and segment sequence number (upper 4 bytes). # Chunk ``` -┌───────────────┬───────────────────┬──────────────┬────────────────┐ -│ len │ encoding <1 byte> │ data │ CRC32 <4 byte> │ -└───────────────┴───────────────────┴──────────────┴────────────────┘ +┌───────────────┬───────────────────┬──────────────┬─────────────────┐ +│ len │ encoding <1 byte> │ data │ CRC32C <4 byte> │ +└───────────────┴───────────────────┴──────────────┴─────────────────┘ ``` Notes: -* `` has 1 to 10 bytes. -* `encoding`: Currently either `XOR` or `histogram`. +* `` has 1 to 10 bytes, from [`encoding/binary`](https://go.dev/src/encoding/binary/varint.go). +* `encoding`: Currently either `XOR` (1), `histogram` (2), or `float histogram` (3). * `data`: See below for each encoding. +* `CRC32C`: CRC32 Castagnoli of `encoding` and `data`, serialised as an unsigned 32 bits big endian. ## XOR chunk data From 60467699411fa6e15e6f1b8e0eb91f5cba6933d2 Mon Sep 17 00:00:00 2001 From: Antoine Pultier <45740+fungiboletus@users.noreply.github.com> Date: Tue, 3 Dec 2024 14:24:50 +0100 Subject: [PATCH 2/3] tsdb documenation: Improve Chunk documentation Signed-off-by: Antoine Pultier <45740+fungiboletus@users.noreply.github.com> Signed-off-by: Antoine Pultier <45740+fungiboletus@users.noreply.github.com> --- tsdb/docs/format/chunks.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/tsdb/docs/format/chunks.md b/tsdb/docs/format/chunks.md index 8a35721358..d4f78dfd3d 100644 --- a/tsdb/docs/format/chunks.md +++ b/tsdb/docs/format/chunks.md @@ -29,16 +29,16 @@ in-file offset (lower 4 bytes) and segment sequence number (upper 4 bytes). # Chunk ``` -┌───────────────┬───────────────────┬──────────────┬─────────────────┐ -│ len │ encoding <1 byte> │ data │ CRC32C <4 byte> │ -└───────────────┴───────────────────┴──────────────┴─────────────────┘ +┌───────────────┬───────────────────┬──────────────┬───────────────────┐ +│ len │ encoding <1 byte> │ data │ checksum <4 byte> │ +└───────────────┴───────────────────┴──────────────┴───────────────────┘ ``` Notes: -* `` has 1 to 10 bytes, from [`encoding/binary`](https://go.dev/src/encoding/binary/varint.go). +* `len`: Chunk size in bytes. Encoded using 1 to 10 bytes, using the [`` encoding](https://go.dev/src/encoding/binary/varint.go). * `encoding`: Currently either `XOR` (1), `histogram` (2), or `float histogram` (3). * `data`: See below for each encoding. -* `CRC32C`: CRC32 Castagnoli of `encoding` and `data`, serialised as an unsigned 32 bits big endian. +* `checksum`: Checksum of `encoding` and `data`. It's a [cyclic redudancy check](https://en.wikipedia.org/wiki/Cyclic_redundancy_check) with the Castagnoli polynomial, serialised as an unsigned 32 bits big endian number. Can be refered as a `CRC-32C`. ## XOR chunk data From f1340bac64f415df757583026476848efdda931f Mon Sep 17 00:00:00 2001 From: Antoine Pultier Date: Tue, 3 Dec 2024 14:36:56 +0100 Subject: [PATCH 3/3] documentation: put back trailing punctuation. markdownlint wasn't happy about the trailing punctuation in the headings. Signed-off-by: Antoine Pultier --- tsdb/docs/format/chunks.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/tsdb/docs/format/chunks.md b/tsdb/docs/format/chunks.md index 837db020e3..d6e3a3ae6c 100644 --- a/tsdb/docs/format/chunks.md +++ b/tsdb/docs/format/chunks.md @@ -48,7 +48,7 @@ Notes: └──────────────────────┴───────────────┴───────────────┴──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴─────┴──────────────────────┴──────────────────────┴──────────────────┘ ``` -### Notes +### Notes: * `ts` is the timestamp, `v` is the value. * `...` means to repeat the previous two fields as needed, with `n` starting at 2 and going up to `num_samples` – 1. @@ -73,7 +73,7 @@ Notes: └──────────────────────┴──────────────────────────┴───────────────────────────────┴─────────────────────┴──────────────────┴──────────────────┴──────────────────────┴────────────────┴──────────────────┘ ``` -### Positive and negative spans data +### Positive and negative spans data: ``` ┌─────────────────────────┬────────────────────────┬───────────────────────┬────────────────────────┬───────────────────────┬─────┬────────────────────────┬───────────────────────┐ @@ -81,7 +81,7 @@ Notes: └─────────────────────────┴────────────────────────┴───────────────────────┴────────────────────────┴───────────────────────┴─────┴────────────────────────┴───────────────────────┘ ``` -### Custom values data +### Custom values data: The `custom_values` data is currently only used for schema -53 (custom bucket boundaries). For other schemas, it is empty (length of zero). @@ -91,7 +91,7 @@ The `custom_values` data is currently only used for schema -53 (custom bucket bo └──────────────────────────┴─────────────────────────────────────┴─────┴──────────────────┘ ``` -### Samples data +### Samples data: ``` ┌──────────────────────────┐ @@ -107,7 +107,7 @@ The `custom_values` data is currently only used for schema -53 (custom bucket bo └──────────────────────────┘ ``` -#### Sample 0 data +#### Sample 0 data: ``` ┌─────────────────┬─────────────────────┬──────────────────────────┬───────────────┬───────────────────────────┬─────┬───────────────────────────┬───────────────────────────┬─────┬───────────────────────────┐ @@ -115,7 +115,7 @@ The `custom_values` data is currently only used for schema -53 (custom bucket bo └─────────────────┴─────────────────────┴──────────────────────────┴───────────────┴───────────────────────────┴─────┴───────────────────────────┴───────────────────────────┴─────┴───────────────────────────┘ ``` -#### Sample 1 data +#### Sample 1 data: ``` ┌───────────────────────┬──────────────────────────┬───────────────────────────────┬──────────────────────┬─────────────────────────────────┬─────┬─────────────────────────────────┬─────────────────────────────────┬─────┬─────────────────────────────────┐ @@ -123,7 +123,7 @@ The `custom_values` data is currently only used for schema -53 (custom bucket bo └───────────────────────┴──────────────────────────┴───────────────────────────────┴──────────────────────┴─────────────────────────────────┴─────┴─────────────────────────────────┴─────────────────────────────────┴─────┴─────────────────────────────────┘ ``` -#### Sample 2 data and following +#### Sample 2 data and following: ``` ┌─────────────────────┬────────────────────────┬─────────────────────────────┬──────────────────────┬───────────────────────────────┬─────┬───────────────────────────────┬───────────────────────────────┬─────┬───────────────────────────────┐ @@ -131,7 +131,7 @@ The `custom_values` data is currently only used for schema -53 (custom bucket bo └─────────────────────┴────────────────────────┴─────────────────────────────┴──────────────────────┴───────────────────────────────┴─────┴───────────────────────────────┴───────────────────────────────┴─────┴───────────────────────────────┘ ``` -### Notes +### Notes: * `histogram_flags` is a byte of which currently only the first two bits are used: * `10`: Counter reset between the previous chunk and this one. @@ -181,7 +181,7 @@ bound of 33554430 is picked so that the varbit encoded value will take at most Float histograms have the same layout as histograms apart from the encoding of samples. -### Samples data +### Samples data: ``` ┌──────────────────────────┐ @@ -197,7 +197,7 @@ Float histograms have the same layout as histograms apart from the encoding of s └──────────────────────────┘ ``` -#### Sample 0 data +#### Sample 0 data: ``` ┌─────────────────┬─────────────────┬──────────────────────┬───────────────┬────────────────────────┬─────┬────────────────────────┬────────────────────────┬─────┬────────────────────────┐ @@ -205,7 +205,7 @@ Float histograms have the same layout as histograms apart from the encoding of s └─────────────────┴─────────────────┴──────────────────────┴───────────────┴────────────────────────┴─────┴────────────────────────┴────────────────────────┴─────┴────────────────────────┘ ``` -#### Sample 1 data +#### Sample 1 data: ``` ┌───────────────────────┬────────────────────────┬─────────────────────────────┬──────────────────────┬───────────────────────────────┬─────┬───────────────────────────────┬───────────────────────────────┬─────┬───────────────────────────────┐ @@ -213,7 +213,7 @@ Float histograms have the same layout as histograms apart from the encoding of s └───────────────────────┴────────────────────────┴─────────────────────────────┴──────────────────────┴───────────────────────────────┴─────┴───────────────────────────────┴───────────────────────────────┴─────┴───────────────────────────────┘ ``` -#### Sample 2 data and following +#### Sample 2 data and following: ``` ┌─────────────────────┬────────────────────────┬─────────────────────────────┬──────────────────────┬───────────────────────────────┬─────┬───────────────────────────────┬───────────────────────────────┬─────┬───────────────────────────────┐