mirror of
https://github.com/redis/redis.git
synced 2026-07-11 11:05:39 -04:00
Some checks are pending
CI / test-ubuntu-latest (push) Waiting to run
CI / test-sanitizer-address (push) Waiting to run
CI / build-debian-old (push) Waiting to run
CI / build-macos-latest (push) Waiting to run
CI / build-32bit (push) Waiting to run
CI / build-libc-malloc (push) Waiting to run
CI / build-centos-jemalloc (push) Waiting to run
CI / build-old-chain-jemalloc (push) Waiting to run
Codecov / code-coverage (push) Waiting to run
External Server Tests / test-external-standalone (push) Waiting to run
External Server Tests / test-external-cluster (push) Waiting to run
External Server Tests / test-external-nodebug (push) Waiting to run
Spellcheck / Spellcheck (push) Waiting to run
## Summary Redis core and the external modules (Search, JSON, Bloom, TimeSeries) are currently built and released through separate flows, forcing anyone assembling a full distribution to juggle multiple repos, refs, build commands, and config files. This PR collapses that into one Redis-owned workflow: ```sh make modules-update # clone/refresh modules from modules/modules.yaml make bootstrap # install per-module build/test deps make build # build Redis core + selected modules make run # start redis-server with modules loaded make test all # run module test suites make tarball TAG=... # build a reproducible source bundle ``` ## What This Adds **1. Manifest-driven modules** — `modules/modules.yaml` is the single source of truth for bundled modules. Each entry pins `repo`/`ref` and defines its build artifact and runtime `loadmodule` path. Refs resolve dynamically (tag → branch → SHA, else fail). Current pins: RedisBloom `v8.9.81`, RediSearch `v8.9.82`, RedisJSON `v8.9.81`, RedisTimeSeries `v8.9.82`. **2. Central manifest parser** — `scripts/lib/manifest.sh` is the only YAML reader, usable both as a shell library and a CLI helper (via `modules/manifest.mk`). **3. Per-module stub Makefiles removed** — The `modules/redis*/Makefile` stubs are deleted; their data moves to `modules.yaml` and shared behavior to `modules/common.mk`. ## Make Workflow The root `Makefile` is now a thin dispatch layer over scripts in `scripts/`: | Target | Script | Purpose | |---|---|---| | `make build [args]` | `scripts/build.sh` | Build Redis core plus selected modules | | `make bootstrap [args]` | `scripts/bootstrap.sh` | Install per-module build/test prerequisites | | `make setup [args]` | `scripts/setup.sh` | Run module update plus bootstrap | | `make run [args]` | `scripts/run.sh` | Start Redis with selected modules loaded | | `make test [args]` | `scripts/test.sh` | Run Redis or module tests | | `make clean [args]` | `scripts/clean.sh` | Clean Redis and/or module outputs | | `make deploy [args]` | `scripts/deploy.sh` | Install Redis and module artifacts | | `make modules-update [args]` | `scripts/modules-update.sh` | Clone/refresh module sources from the manifest | | `make modules-shallow [args]` | `scripts/modules-shallow.sh` | Re-clone selected modules with shallow history | | `make sync-redis-conf [args]` | `scripts/sync-redis-conf.sh` | Regenerate managed module config | | `make tarball TAG=...` | `scripts/tarball.sh` | Create a reproducible Redis + modules source tarball | All accept module selectors: ```sh make build all # core + all modules make build redisjson # core + one module make run none # core only, no modules make test redisearch TEST='some:test:name' ``` ## Runtime Configuration Removes the hand-maintained `redis-full.conf` (which could drift from upstream). Config is now generated: `sync-redis-conf.sh` produces a managed Modules block appended to verbatim `redis.conf`, with `loadmodule` lines (active when built or `ASSUME_BUILT=1`) plus each module's `module.conf`. The generated file is git-ignored. ## Docker & Release - **`docker/Dockerfile.noble`** — Ubuntu 24.04 build env for core + modules (amd64/arm64). - **`scripts/tarball.sh`** — reproducible source bundle: archives core at `TAG`, clones modules at pinned refs, strips `.git`/build artifacts, generates config, and produces a deterministic gzip. `utils/releasetools/01_create_tarball.sh` now delegates to it. - **`post-release-automation.yml`** — builds the bundle on published releases, uploads it as artifact + release asset, and reports SHA256. ## Docs `modules/MODULES.md` (operator guide), `modules/README.md`, and `README.md` cover the manifest schema, Make targets, tarball flow, Docker env, and troubleshooting. ## Compatibility Notes - `BUILD_WITH_MODULES=yes` is replaced by `make build [redis|core|none|all|<module>...]`. - `redis-full.conf` is now generated and git-ignored. - Module clones live under `modules/<name>/src/` (git-ignored). - `vector-sets` is omitted — it already lives in-tree. - RediSearch builds with `LTO=1 REDISEARCH_GENERATE_HEADERS=0 INLINE_LSE_ATOMICS=0`. --------- Co-authored-by: debing.sun <debing.sun@redis.com>
197 lines
8 KiB
Bash
Executable file
197 lines
8 KiB
Bash
Executable file
#!/bin/sh
|
|
# deploy.sh — build (via scripts/build.sh) then copy artifacts to $(PREFIX).
|
|
#
|
|
# Usage: scripts/deploy.sh [<name> ...|all|.|'*'|redis|none]
|
|
# Env: PREFIX install root (default /usr/local). Files land in:
|
|
# $PREFIX/bin/ - redis-server, -cli, -benchmark
|
|
# $PREFIX/lib/redis/modules/ - per-module .so files
|
|
# DESTDIR optional staging root prepended to PREFIX (for packaging).
|
|
# MAKE make binary (defaults to `make`); only used when shelling
|
|
# into build.sh, which itself respects it.
|
|
#
|
|
# Tokens:
|
|
# (no args) | all | . | '*' build + install Redis + every cloned module
|
|
# redis | none Redis only
|
|
# <name> [<name> ...] Redis + the listed modules
|
|
#
|
|
# Flow:
|
|
# 1. Delegate build to scripts/build.sh — same orchestration the user gets
|
|
# from `make` (failure collection, redis-full.conf refresh, etc.).
|
|
# 2. After build, copy artifacts into place ourselves (NOT via each
|
|
# module's `install` target). This keeps the install step a pure copy
|
|
# and avoids depending on per-module Makefile install recipes.
|
|
#
|
|
# Failures during build are surfaced from build.sh; failures during copy are
|
|
# collected and reported at the end of this script.
|
|
|
|
set -eu
|
|
|
|
SCRIPT_DIR="$(cd -- "$(dirname -- "$0")" && pwd)" || exit 1
|
|
. "$SCRIPT_DIR/lib/manifest.sh"
|
|
cd "$REPO_ROOT"
|
|
|
|
PREFIX="${PREFIX:-/usr/local}"
|
|
DESTDIR="${DESTDIR:-}"
|
|
|
|
cloned="$(cloned_modules)"
|
|
modules="$(resolve_modules "$*" "$cloned" "redis none")"
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Phase 1: build via the shared orchestrator. Pass the resolved module list
|
|
# verbatim so build.sh sees the same selection we intend to install.
|
|
# ---------------------------------------------------------------------------
|
|
echo "==> Building before deploy (delegating to scripts/build.sh)"
|
|
echo
|
|
# Build artifacts always go to the dev tree — PREFIX is irrelevant to the
|
|
# build phase. Two scrubs needed before handing off to build.sh:
|
|
# 1. PREFIX env shadow — so anything reading $PREFIX in build.sh / scripts
|
|
# it spawns sees the empty value.
|
|
# 2. MAKEFLAGS — when the user runs `make deploy PREFIX=…`, the parent
|
|
# make smuggles PREFIX through MAKEFLAGS, and any recursive `make` we
|
|
# spawn would re-apply it as a command-line variable (which beats env).
|
|
# Strip just the PREFIX=… token from MAKEFLAGS so the rest (e.g. -j8)
|
|
# still propagates normally.
|
|
# Our local $PREFIX (in this script's scope) is untouched and reused for
|
|
# the copy + apply-redis-conf phases below.
|
|
build_makeflags="$(printf '%s' " ${MAKEFLAGS:-} " | sed -E 's/ PREFIX=[^ ]*/ /g; s/^ *//; s/ *$//')"
|
|
|
|
# If $modules is empty (`none` token) we still want Redis core built — feed
|
|
# the literal "redis" token in that case so build.sh builds core only.
|
|
# Tolerate a non-zero exit from build.sh: it returns 1 when any one module
|
|
# fails to build, but other modules may have succeeded (or have artifacts left
|
|
# over from prior successful builds). The per-module existence check in
|
|
# phase 2 below filters out modules that genuinely don't have a .so to copy,
|
|
# so we'd rather install what's available than bail out wholesale.
|
|
build_rc=0
|
|
if [ -z "$modules" ]; then
|
|
MAKEFLAGS="$build_makeflags" PREFIX="" "$SCRIPT_DIR/build.sh" redis || build_rc=$?
|
|
else
|
|
MAKEFLAGS="$build_makeflags" PREFIX="" "$SCRIPT_DIR/build.sh" $modules || build_rc=$?
|
|
fi
|
|
if [ "$build_rc" != 0 ]; then
|
|
echo
|
|
echo "==> WARNING: build.sh reported failures (exit $build_rc) — continuing to install what's available"
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Phase 2: copy artifacts to $DESTDIR$PREFIX. Pure file ops, no recursive make.
|
|
# ---------------------------------------------------------------------------
|
|
INSTALL_BIN_DIR="$DESTDIR$PREFIX/bin"
|
|
INSTALL_MOD_DIR="$DESTDIR$PREFIX/lib/redis/modules"
|
|
|
|
echo
|
|
echo "==> Deploying to PREFIX=$PREFIX${DESTDIR:+ (DESTDIR=$DESTDIR)}"
|
|
echo
|
|
|
|
# Redis core binaries + the three symlinks that point to redis-server.
|
|
echo "==> Installing Redis core binaries to $INSTALL_BIN_DIR"
|
|
install -d "$INSTALL_BIN_DIR"
|
|
install -m 0755 src/redis-server "$INSTALL_BIN_DIR/redis-server"
|
|
install -m 0755 src/redis-cli "$INSTALL_BIN_DIR/redis-cli"
|
|
install -m 0755 src/redis-benchmark "$INSTALL_BIN_DIR/redis-benchmark"
|
|
ln -sf redis-server "$INSTALL_BIN_DIR/redis-check-rdb"
|
|
ln -sf redis-server "$INSTALL_BIN_DIR/redis-check-aof"
|
|
ln -sf redis-server "$INSTALL_BIN_DIR/redis-sentinel"
|
|
|
|
# Per-module `.so` files. After scripts/build.sh, each cloned module has its
|
|
# .so copied to modules/<name>/<basename> by common.mk (the `cp $(TARGET_MODULE) ./`
|
|
# step in $(TARGET_MODULE)'s recipe). Read the manifest's `target_module:`
|
|
# field to know the exact basename for each module (handles redisjson →
|
|
# rejson.so) and copy from there.
|
|
failed=""
|
|
if [ -n "$modules" ]; then
|
|
echo
|
|
echo "==> Installing modules to $INSTALL_MOD_DIR"
|
|
install -d "$INSTALL_MOD_DIR"
|
|
for name in $modules; do
|
|
target="$(manifest_field "$name" target_module)"
|
|
if [ -z "$target" ]; then
|
|
echo "==> [module] $name: no 'target_module' in modules.yaml — skipping"
|
|
failed="$failed $name"
|
|
continue
|
|
fi
|
|
so_basename="$(basename "$target")"
|
|
so_src="modules/$name/$so_basename"
|
|
if [ ! -f "$so_src" ]; then
|
|
echo "==> [module] $name: built artifact not found at $so_src — skipping"
|
|
failed="$failed $name"
|
|
continue
|
|
fi
|
|
install -m 0755 "$so_src" "$INSTALL_MOD_DIR/$so_basename"
|
|
echo " [module] $name -> $INSTALL_MOD_DIR/$so_basename"
|
|
done
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Phase 3: rewrite the loadmodule paths in redis-full.conf and redis.conf
|
|
# in-place. Only the LOADMODULE_BEGIN/END block is replaced — the rest of
|
|
# each file is untouched. Runs even when some modules failed to copy so that
|
|
# the successfully-installed ones get correct absolute paths.
|
|
# ---------------------------------------------------------------------------
|
|
REDIS_FULL_CONF="${REDIS_GEN_CONF:-redis-full.conf}"
|
|
REDIS_CONF="${REDIS_CONF:-redis.conf}"
|
|
LOADMODULE_BEGIN="# >>> BEGIN: loadmodule paths (replaced by make deploy) <<<"
|
|
LOADMODULE_END="# <<< END: loadmodule paths <<<"
|
|
|
|
# Build the list of successfully-installed modules (i.e. those NOT in $failed).
|
|
installed_modules=""
|
|
for name in $modules; do
|
|
case " $failed " in *" $name "*) continue ;; esac
|
|
installed_modules="$installed_modules $name"
|
|
done
|
|
installed_modules="${installed_modules# }"
|
|
|
|
if [ -n "$installed_modules" ]; then
|
|
new_lines=""
|
|
for name in $installed_modules; do
|
|
target="$(manifest_field "$name" target_module)"
|
|
[ -z "$target" ] && continue
|
|
so_basename="$(basename "$target")"
|
|
new_lines="${new_lines}loadmodule $INSTALL_MOD_DIR/$so_basename
|
|
"
|
|
done
|
|
|
|
# Write new_lines to a temp file so awk can read it with getline — BSD awk
|
|
# (macOS /usr/bin/awk) rejects embedded newlines in -v values.
|
|
new_lines_file="$(mktemp)"
|
|
printf '%s' "$new_lines" > "$new_lines_file"
|
|
|
|
_patch_conf() {
|
|
local conf="$1"
|
|
[ -f "$conf" ] || return 0
|
|
grep -qF "$LOADMODULE_BEGIN" "$conf" 2>/dev/null || return 0
|
|
local tmp
|
|
tmp="$(mktemp "${conf}.deploy.XXXXXX")"
|
|
trap 'rm -f "$tmp" "$new_lines_file"' EXIT
|
|
awk -v begin="$LOADMODULE_BEGIN" -v end="$LOADMODULE_END" -v newfile="$new_lines_file" '
|
|
$0 == begin {
|
|
print
|
|
while ((getline line < newfile) > 0) print line
|
|
close(newfile)
|
|
skip=1; next
|
|
}
|
|
$0 == end { skip=0 }
|
|
!skip { print }
|
|
' "$conf" > "$tmp"
|
|
mv "$tmp" "$conf"
|
|
trap - EXIT
|
|
echo "==> Updated loadmodule paths in $conf"
|
|
}
|
|
|
|
_patch_conf "$REDIS_FULL_CONF"
|
|
_patch_conf "$REDIS_CONF"
|
|
rm -f "$new_lines_file"
|
|
fi
|
|
|
|
echo
|
|
echo "==> Deploy complete."
|
|
echo " redis-server: $INSTALL_BIN_DIR/redis-server"
|
|
if [ -n "$modules" ]; then
|
|
echo " Module .so directory: $INSTALL_MOD_DIR/"
|
|
fi
|
|
|
|
if [ -n "$failed" ]; then
|
|
echo
|
|
echo "ERROR: deploy finished with module copy failure(s):$failed" >&2
|
|
exit 1
|
|
fi
|