redis/scripts/build.sh
Tal Bar Yakar 4dd58caa7c
Some checks are pending
CI / test-ubuntu-latest (push) Waiting to run
CI / test-sanitizer-address (push) Waiting to run
CI / build-debian-old (push) Waiting to run
CI / build-macos-latest (push) Waiting to run
CI / build-32bit (push) Waiting to run
CI / build-libc-malloc (push) Waiting to run
CI / build-centos-jemalloc (push) Waiting to run
CI / build-old-chain-jemalloc (push) Waiting to run
Codecov / code-coverage (push) Waiting to run
External Server Tests / test-external-standalone (push) Waiting to run
External Server Tests / test-external-cluster (push) Waiting to run
External Server Tests / test-external-nodebug (push) Waiting to run
Spellcheck / Spellcheck (push) Waiting to run
RED-191626 , RED-191854 - modules modernization (#15253)
## Summary

Redis core and the external modules (Search, JSON, Bloom, TimeSeries)
are currently built and released through separate flows, forcing anyone
assembling a full distribution to juggle multiple repos, refs, build
commands, and config files.

This PR collapses that into one Redis-owned workflow:

```sh
make modules-update    # clone/refresh modules from modules/modules.yaml
make bootstrap         # install per-module build/test deps
make build             # build Redis core + selected modules
make run               # start redis-server with modules loaded
make test all          # run module test suites
make tarball TAG=...   # build a reproducible source bundle
```

## What This Adds

**1. Manifest-driven modules** — `modules/modules.yaml` is the single
source of truth for bundled modules. Each entry pins `repo`/`ref` and
defines its build artifact and runtime `loadmodule` path. Refs resolve
dynamically (tag → branch → SHA, else fail).

Current pins: RedisBloom `v8.9.81`, RediSearch `v8.9.82`, RedisJSON
`v8.9.81`, RedisTimeSeries `v8.9.82`.

**2. Central manifest parser** — `scripts/lib/manifest.sh` is the only
YAML reader, usable both as a shell library and a CLI helper (via
`modules/manifest.mk`).

**3. Per-module stub Makefiles removed** — The `modules/redis*/Makefile`
stubs are deleted; their data moves to `modules.yaml` and shared
behavior to `modules/common.mk`.

## Make Workflow

The root `Makefile` is now a thin dispatch layer over scripts in
`scripts/`:

| Target | Script | Purpose |
|---|---|---|
| `make build [args]` | `scripts/build.sh` | Build Redis core plus
selected modules |
| `make bootstrap [args]` | `scripts/bootstrap.sh` | Install per-module
build/test prerequisites |
| `make setup [args]` | `scripts/setup.sh` | Run module update plus
bootstrap |
| `make run [args]` | `scripts/run.sh` | Start Redis with selected
modules loaded |
| `make test [args]` | `scripts/test.sh` | Run Redis or module tests |
| `make clean [args]` | `scripts/clean.sh` | Clean Redis and/or module
outputs |
| `make deploy [args]` | `scripts/deploy.sh` | Install Redis and module
artifacts |
| `make modules-update [args]` | `scripts/modules-update.sh` |
Clone/refresh module sources from the manifest |
| `make modules-shallow [args]` | `scripts/modules-shallow.sh` |
Re-clone selected modules with shallow history |
| `make sync-redis-conf [args]` | `scripts/sync-redis-conf.sh` |
Regenerate managed module config |
| `make tarball TAG=...` | `scripts/tarball.sh` | Create a reproducible
Redis + modules source tarball |

All accept module selectors:

```sh
make build all              # core + all modules
make build redisjson        # core + one module
make run none               # core only, no modules
make test redisearch TEST='some:test:name'
```

## Runtime Configuration

Removes the hand-maintained `redis-full.conf` (which could drift from
upstream). Config is now generated: `sync-redis-conf.sh` produces a
managed Modules block appended to verbatim `redis.conf`, with
`loadmodule` lines (active when built or `ASSUME_BUILT=1`) plus each
module's `module.conf`. The generated file is git-ignored.

## Docker & Release

- **`docker/Dockerfile.noble`** — Ubuntu 24.04 build env for core +
modules (amd64/arm64).
- **`scripts/tarball.sh`** — reproducible source bundle: archives core
at `TAG`, clones modules at pinned refs, strips `.git`/build artifacts,
generates config, and produces a deterministic gzip.
`utils/releasetools/01_create_tarball.sh` now delegates to it.
- **`post-release-automation.yml`** — builds the bundle on published
releases, uploads it as artifact + release asset, and reports SHA256.

## Docs

`modules/MODULES.md` (operator guide), `modules/README.md`, and
`README.md` cover the manifest schema, Make targets, tarball flow,
Docker env, and troubleshooting.

## Compatibility Notes

- `BUILD_WITH_MODULES=yes` is replaced by `make build
[redis|core|none|all|<module>...]`.
- `redis-full.conf` is now generated and git-ignored.
- Module clones live under `modules/<name>/src/` (git-ignored).
- `vector-sets` is omitted — it already lives in-tree.
- RediSearch builds with `LTO=1 REDISEARCH_GENERATE_HEADERS=0
INLINE_LSE_ATOMICS=0`.

---------

Co-authored-by: debing.sun <debing.sun@redis.com>
2026-07-08 22:55:45 +08:00

94 lines
2.9 KiB
Bash
Executable file

#!/bin/sh
# build.sh — build Redis (src/) + selected modules.
#
# Usage: scripts/build.sh [<name> ...|all|.|'*'|redis|core|none]
#
# Tokens:
# (no args) | all | . | '*' build Redis + every cloned module
# redis | core | none build Redis only (no modules)
# <name> [<name> ...] build Redis + the listed modules
#
# Each selected module is built via `make -C modules/<name>` using its
# upstream defaults — RM_INCLUDE_DIR / RS_INCLUDE_DIR are intentionally
# NOT overridden here, so a bundled build resolves `redismodule.h` the
# same way the module would when built standalone in its own repo.
# REDIS_SERVER is overridden so module test harnesses run against the
# Redis we just built.
# Failures are collected and reported at the end.
#
# This script backs the default `make` goal, so it must stay POSIX-sh
# clean: Redis builds on systems whose only shell is busybox ash or dash
# (e.g. the alpine:latest Daily CI containers install no bash).
set -eu
SCRIPT_DIR="$(cd -- "$(dirname -- "$0")" && pwd)" || exit 1
. "$SCRIPT_DIR/lib/manifest.sh"
cd "$REPO_ROOT"
MAKE_BIN="${MAKE:-make}"
cloned="$(cloned_modules)"
modules="$(resolve_modules "$*" "$cloned" "redis core none")"
echo "==> Building main Redis (src/)"
if ! "$MAKE_BIN" -C src all; then
echo
echo "ERROR: Redis core build failed (make -C src all)." >&2
echo " Fix the failure above before module builds will run." >&2
exit 1
fi
failed=""
if [ -z "$modules" ]; then
echo
if [ -z "$cloned" ]; then
echo "==> No cloned modules under modules/*/src, Redis-only build"
else
echo "==> Module builds skipped by request"
fi
else
echo
echo "==> Building modules (REDIS_SERVER=$REPO_ROOT/src/redis-server):"
echo " $modules"
for name in $modules; do
echo
echo "==> [module] $name (modules/$name)"
mkdir -p "modules/$name"
if ! "$MAKE_BIN" -C "modules/$name" -f "$REPO_ROOT/modules/common.mk" \
REDIS_SERVER="$REPO_ROOT/src/redis-server"; then
failed="$failed $name"
echo "==> [module] $name: FAILED (continuing with remaining modules)"
fi
done
if [ -n "$failed" ]; then
echo
echo "==> WARNING: The following module(s) failed to build:$failed"
fi
fi
echo
echo "==> Build complete."
echo " redis-server: $PWD/src/redis-server"
if [ -n "$modules" ]; then
echo " Module artifacts:"
for name in $modules; do
sos="$(find "modules/$name" -type f -name '*.so' \
-not -path '*/venv/*' \
-not -path '*/.cargo/*' \
-not -path '*/site-packages/*' \
-not -path '*/deps/*' 2>/dev/null || true)"
if [ -n "$sos" ]; then
printf " %-20s " "$name:"; echo "$sos" | head -1
echo "$sos" | tail -n +2 | sed 's|^| |'
else
printf " %-20s (no .so found)\n" "$name:"
fi
done
fi
if [ -n "$failed" ]; then
echo
echo "ERROR: make build finished with module failure(s):$failed"
exit 1
fi