Ensure backward compatibility and consistent behavior across different
architectures by explicitly setting the default value.
Fixes#15175
Co-authored-by: ofiryanai <ofiryanai1@gmail.com>
* Limit VADD REDUCE dim to not exceed original dim
Enforce VADD key [REDUCE dim] to reject dim that is bigger than the HNSW original dim, as dimension reduction makes no sense for reduce_dim > original_dim.
This also avoids OOM and possible heap overflow on later allocations using reduce_dim.
This should be backported to Redis version 8.0, 8.2 and 8.4.
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> **Low Risk**
> Low risk Makefile change that only alters build flags for the
RediSearch module; primary risk is build/compatibility issues on some
toolchains when LTO is enabled.
>
> **Overview**
> **RediSearch module builds now default to link-time optimization.**
The `modules/redisearch/Makefile` introduces `LTO ?= 1` and exports it
so the upstream RediSearch build can pick it up, with an escape hatch to
disable via `LTO=0`.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
de33dc7033. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Update RediSearch module version to 8.8 RC1 (v8.7.90)
Made with [Cursor](https://cursor.com)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Low Risk**
> Low risk: a single version bump that changes which RediSearch git tag
is cloned/built; main risk is build/runtime incompatibility from the
upstream RC update.
>
> **Overview**
> Updates the RediSearch module build configuration to fetch and build
upstream `redisearch` tag `v8.7.90` (8.8 RC1) instead of `v8.5.90`.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
21e121c738. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
`VSIM_RedisCommand` in `vset.c` had two memory leak bugs related to the
compiled FILTER expression (`exprstate` allocated by `exprCompile`):
1. Duplicate FILTER: When two FILTER options are provided in a single
VSIM command, the second `exprCompile` overwrites `filter_expr` without
freeing the first. Only the last one is freed in `VSIM_execute cleanup`.
Fix: call `exprFree` on the existing `filter_expr` before reassigning.
2. Error path leaks: When FILTER is parsed successfully but a later
option fails validation (invalid COUNT/EF/EPSILON/FILTER-EF or unknown
option), the error return frees `vec` but not `filter_expr`.
Fix: add `exprFree(filter_expr)` to all five error return paths.
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> **Low Risk**
> Build-script-only change that updates pinned checksums and a
verification flag; main risk is breaking Rust toolchain installation if
the new hashes are incorrect.
>
> **Overview**
> Updates the `modules/Makefile` Rust toolchain bootstrap to use new
SHA256 checksums for the Rust 1.93.1 standalone installers across
`x86_64`/`aarch64` and `musl`/`gnu` targets.
>
> Also changes the checksum verification command to use `sha256sum -c
--status` (instead of `--quiet`) while keeping the same failure
behavior, reducing noisy output during successful installs.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
30ab5a3ba6. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> **Low Risk**
> Simple version bumps in build configuration; main risk is upstream
module behavior/compatibility changes when building `v8.6.0`.
>
> **Overview**
> Bumps the pinned build versions of the Redis data type modules
`redisbloom`, `redisjson`, and `redistimeseries` from `v8.5.90` to
`v8.6.0` via their Makefiles.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
f1319fafa2. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
This PR adds SIMD vectorization for binary quantization distance
calculation, similar to PR #14474.
---------
Co-authored-by: debing.sun <debing.sun@redis.com>
This PR replaces the manual `popcount64()` implementation with
`__builtin_popcountll()` for computing Hamming distance in binary
vectors, when the underlying hardware supports the `POPCNT` instruction.
The built-in version simplifies the code and enables the compiler to
emit a single `POPCNT` instruction on supported CPUs, which is
significantly faster than the manual bitwise method. You can verify the
difference here:
[https://godbolt.org/z/TxWMcE8M3](https://godbolt.org/z/TxWMcE8M3) — the
manual version generates a long sequence of instructions (approximately
34 on modern HW) vs 1 instruction (popcnt) when using
__builtin_popcountll()
## Portability across platforms
This change maintains full portability across platforms and compilers.
The use of `__builtin_popcountll()` is guarded by the `HAVE_POPCNT`
macro, which is defined only when the compiler supports the
target("popcnt") attribute. At runtime, we also check
`__builtin_cpu_supports("popcnt")` to ensure the hardware provides
support for the instruction. If not available, the implementation safely
falls back to the original manual `popcount64()` logic.
---------
Co-authored-by: debing.sun <debing.sun@redis.com>
This pull request vectorizes the 8-bit quantization vector-search path
in a similar was as the non-quantization path.
The assembly intrinsics are a bit more complicated than in the
non-quantization path, since we are operating on 8-bit integers and we
need to worry about preventing overflow. Thus, after loading the 8-bit
integers, they are extended into 16-bits before multiplying and
accumulating into 32-bit integers.
---------
Co-authored-by: debing.sun <debing.sun@redis.com>
(https://github.com/RediSearch/RediSearch/pull/7076,
https://github.com/RediSearch/RediSearch/pull/6857) - Introducing
`FT.HYBRID`, a new command that enables hybrid queries combining both
text and vector search, with support for **RRF** and **LINEAR** result
combination. This update enhances performance and reliability through a
more efficient networking layer, smoother query execution, and improved
overall stability.
(https://github.com/RediSearch/RediSearch/pull/7065) - Add
`search-default-scorer` configuration to set the default text scorer
across queries (by default it is BM25).
https://github.com/RediSearch/RediSearch/pull/7022 - Handle Atomic Slot
Migration events upon moving slots from one node to another in a cluster
mode.
(https://github.com/RediSearch/RediSearch/pull/6769,
https://github.com/RediSearch/RediSearch/pull/6828,
https://github.com/RediSearch/RediSearch/pull/6877,
https://github.com/RediSearch/RediSearch/pull/6921) - Introduce query
time memory guardrails by adding a new `search-on-oom` configuration
which defines the query engine behavior when OOM (Out Of Memory) is
reached. OOM checks are applied to `FT.SEARCH`, `FT.AGGREGATE`, and
`FT.HYBRID` commands. The behavior on OOM can be configured to one of
three modes: `IGNORE`, `FAIL`, or `RETURN`.
`IGNORE` - The default behavior, run queries anyway (not recommended for
heavy queries returning a large result set).
`FAIL` - Fail query execution immediately if any of the nodes are in OOM
state when query execution starts.
`RETURN` - A best effort appraoch to return partial results when OOM is
detected in only some of the nodes in a cluster mode.
Conducted tests on IceLake server with LAION 512 1M vectors dataset. We
can see in the perf profile, that function ‘vector_distance_float’ is
consuming majority of the computation time (this profile is captured
during upload). Vectorizing this function, leads to about 2.38X speedup
in upload time and 1.45X speedup in RPS during search.
---------
Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com>
This is basically the Vector Set iteration primitive.
It exploits the underlying radix tree implementation.
The usage pattern is strongly reminiscent of other Redis commands doing
similar things.
The command usage is straightforward:
```
> VRANGE word_embeddings_int8 [Redis + 10
1) "Redis"
2) "Rediscover"
3) "Rediscover_Ashland"
4) "Rediscover_Northern_Ireland"
5) "Rediscovered"
6) "Rediscovered_Bookshop"
7) "Rediscovering"
8) "Rediscovering_God"
9) "Rediscovering_Lost"
10) "Rediscovers"
```
The above command returns 10 (or less, if less are available in the
specified range) elements from "Redis" (inclusive) to the maximum
possible element. The comparison is performed byte by byte, as
`memcmp()` would do, in this way the elements have a total order. The
start and end range can be either a string, prefixed
by `[` or `(` (the prefix is mandatory) to tell the command if the range
is inclusive or exclusive, or can be the special symbols `-` and `+`
that means the maximum and minimum element.
More info can be found in the implementation itself and in the README
file change.
---------
Co-authored-by: debing.sun <debing.sun@redis.com>
This PR aims to avoid the situation of a potential crash when efSearch
is too large (and therefore the memory allocated could lead to a server
crash or an integer overflow (where less memory is allocated than
expected).
- Limit the accepted EF in the request o 100_000 as in VADD
- Limit the ef search to the number of nodes in the HNSW graph
This PR adds a `--ollama-url` option to `cli.py`, the lightweight
redis-cli-like tool that expands !"text" arguments into embeddings via
Ollama.
Previously, the embedding call was hardcoded to
http://localhost:11434/api/embeddings. With this change, users can
specify a custom Ollama server URL when starting the tool.
If no URL is provided, the tool defaults to what it was before.
This PR fixes a bug in the `hnsw_cursor_free` function where the prev
pointer was never updated during cursor list traversal. As a result, if
the cursor being freed was not the head of the list, it would not be
correctly unlinked, potentially causing memory leaks or corruption of
the cursor list.
Note that since `hnsw_cursor_free()` is never used for now, this PR does
not actually fix any bug.
Hi, this PR implements the following changes:
1. The EPSILON option of VSIM is now documented.
2. The EPSILON behavior was fixed: the score was incorrectly divided by
two in the meaning, with a 0-2 interval provided by the underlying
cosine similarity, instead of the 0-1 interval. So an EPSILON of 0.2
only returned elements with a distance between 1 and 0.9 instead of 1
and 0.8. This is a *breaking change* but the command was not documented
so far, and it is a fix, as the user sees the similarity score so was a
total mismatch. I believe this fix should definitely be back ported as
soon as possible.
3. There are now tests.
Thanks for checking,
Salvatore
