Automatically create external caches for MULTI_SITE deployments

Closes #32129

Signed-off-by: Pedro Ruivo <1492066+pruivo@users.noreply.github.com>
Co-authored-by: Pedro Ruivo <1492066+pruivo@users.noreply.github.com>

docs/documentation/release_notes/topics/26_4_0.adoc

@@ -122,3 +122,21 @@ The Operator now provisions a `ServiceMonitor` for the management endpoint if me
 specification of the `ServiceMonitor` takes into account the various management endpoint configurations, to ensure that
 metrics can be scraped without any additional configuration. If you do not want a `ServiceMonitor` to be created, you can disable
 this by setting `spec.serviceMonitor.enabled: false`. For more details, see the link:{operatorguide_link}[{operatorguide_name}].
+
+== {project_name} automatically creates the necessary caches on the first startup if they do not exist
+
+You no longer need to manually create caches in your external Infinispan cluster.
+
+When using the `multi-site` or `clusterless` features, {project_name} now automatically creates the necessary caches during startup if they do not already exist on the Infinispan server.
+
+Any existing caches, manually created before {project_name} startup, will be preserved, and their configuration will not be modified.
+
+For high availability, you can now easily configure cross-site replication.
+Simply set the backup site name (e.g., availability zone) using the following option:
+
+[source,bash]
+----
+--cache-remote-backup-sites=<name>
+----
+
+When this option is set, Infinispan will automatically replicate the cache data to the specified location.

docs/guides/high-availability/examples/generated/keycloak-ispn.yaml

@@ -486,8 +486,10 @@ spec:
       secret:
         name: remote-store-secret
         key: password
+    - name: cache-remote-backup-sites
+      value: "site-2" # <5>
+    # end::keycloak-ispn[]
     - name: db-driver
-      # end::keycloak-ispn[]
 
       value: software.amazon.jdbc.Driver
   http:

docs/guides/high-availability/multi-cluster/deploy-infinispan-kubernetes-crossdc.adoc

@@ -172,7 +172,7 @@ include::../examples/generated/ispn-site-a.yaml[tag=infinispan-crossdc]
 <8> The secret key (filename) of the Keystore as defined in the previous step.
 <9> The secret name where the Truststore exists as defined in the previous step.
 <10> The Truststore key (filename) of the Keystore as defined in the previous step.
-<11> The remote site's name, in this case  `{site-b}`.
+<11> The remote site's name, in this case  `{site-b}`. Use this value in {project_name} option `cache-remote-backup-sites`.
 <12> The namespace of the {jdgserver_name} cluster from the remote site.
 <13> The {ocp} API URL for the remote site.
 <14> The secret with the access token to authenticate into the remote site.
@@ -189,75 +189,9 @@ include::../examples/generated/ispn-site-b.yaml[tag=infinispan-crossdc]
 
 . Creating the caches for {project_name}.
 +
-{project_name} requires the following caches to be present: `actionTokens`, `authenticationSessions`, `loginFailures`, and `work`.
-+
-The {jdgserver_name} {infinispan-operator-docs}#creating-caches[Cache CR] allows deploying the caches in the {jdgserver_name} cluster.
-Cross-site needs to be enabled per cache as documented by {infinispan-xsite-docs}[Cross Site Documentation].
-The documentation contains more details about the options used by this {section}.
-The following example shows the `Cache` CR for `{site-a}`.
-+
---
-. In `{site-a}` create a `Cache` CR for each of the caches mentioned above with the following content.
-+
-.Cache `actionTokens`
-[source,yaml]
-----
-include::../examples/generated/ispn-site-a.yaml[tag=infinispan-cache-actionTokens]
-----
-+
-.Cache `authenticationSessions`
-[source,yaml]
-----
-include::../examples/generated/ispn-site-a.yaml[tag=infinispan-cache-authenticationSessions]
-----
-+
-.Cache `loginFailures`
-[source,yaml]
-----
-include::../examples/generated/ispn-site-a.yaml[tag=infinispan-cache-loginFailures]
-----
-+
-.Cache `work`
-[source,yaml]
-----
-include::../examples/generated/ispn-site-a.yaml[tag=infinispan-cache-work]
-----
-<1> The transaction mode.
-<2> The locking mode used by the transaction.
-<3> The remote site name.
-<4> The cross-site communication strategy, in this case, `SYNC`.
-<5> The cross-site replication timeout.
-<6> The cross-site replication failure policy.
---
-+
-The example above is the recommended configuration to achieve the best data consistency.
-+
-====
-*Background information*
-
-Deadlocks may occur in an active-active setup as entries are modified concurrently in both sites.
-
-The `transaction.mode: NON_DURABLE_XA` ensures that the transaction is rolled back keeping the data consistent if this occurs.
-The setting `backup.failurePolicy: FAIL` is required in this case.
-It will throw an error that allows the transaction to be safely rolled back.
-When this occurs, {project_name} will attempt a retry.
-
-The `transaction.locking: PESSIMISTIC` is the only supported locking mode; `OPTIMISTIC` is not recommended due to its network costs.
-The same settings also prevent that one site is updated while the other site is unreachable.
-
-The `backup.strategy: SYNC` ensures the data is visible and stored in the other site when the {project_name} request is completed.
-
-NOTE: The `locking.acquireTimeout` can be reduced to fail fast in a deadlock scenario.
-The `backup.timeout` must always be higher than the `locking.acquireTimeout`.
-====
-+
-For `{site-b}`, the `Cache` CR is similar, except for the `backups.<name>` outlined in point 3 of the above diagram.
-+
-.Example for `actionTokens` cache in `{site-b}`
-[source,yaml]
-----
-include::../examples/generated/ispn-site-b.yaml[tag=infinispan-cache-actionTokens]
-----
+{project_name} automatically creates the necessary caches on the first startup if they do not exist.
+To customize any default configuration, use the {infinispan-operator-docs}#creating-caches_creating-caches[`Cache` CR] to declare the full cache configuration.
+To take effect, the `Cache` CR must be deployed before any {project_name} Pod starts.
 
 == Verifying the deployment
 
@@ -301,9 +235,10 @@ include::../examples/generated/keycloak-ispn.yaml[tag=keycloak-ispn]
 ----
 <1> The hostname of the remote {jdgserver_name} cluster.
 <2> The port of the remote {jdgserver_name} cluster.
-This is optional and it defaults to `11222`.
+This is optional, and it defaults to `11222`.
 <3> The Secret `name` and `key` with the {jdgserver_name} username credential.
 <4> The Secret `name` and `key` with the {jdgserver_name} password credential.
+<5> The name of the remote site.
 
 === Architecture
 

model/infinispan/src/main/java/org/keycloak/spi/infinispan/impl/embedded/CacheConfigurator.java

@@ -24,6 +24,8 @@ import java.util.stream.Stream;
 
 import org.infinispan.commons.dataconversion.MediaType;
 import org.infinispan.configuration.cache.AbstractStoreConfiguration;
+import org.infinispan.configuration.cache.BackupConfiguration;
+import org.infinispan.configuration.cache.BackupFailurePolicy;
 import org.infinispan.configuration.cache.CacheMode;
 import org.infinispan.configuration.cache.ConfigurationBuilder;
 import org.infinispan.configuration.cache.HashConfiguration;
@@ -36,6 +38,11 @@ import org.infinispan.transaction.lookup.EmbeddedTransactionManagerLookup;
 import org.jboss.logging.Logger;
 import org.keycloak.Config;
 import org.keycloak.config.CachingOptions;
+import org.keycloak.marshalling.Marshalling;
+import org.keycloak.models.sessions.infinispan.entities.LoginFailureEntity;
+import org.keycloak.models.sessions.infinispan.entities.RemoteAuthenticatedClientSessionEntity;
+import org.keycloak.models.sessions.infinispan.entities.RemoteUserSessionEntity;
+import org.keycloak.models.sessions.infinispan.entities.RootAuthenticationSessionEntity;
 
 import static org.keycloak.connections.infinispan.InfinispanConnectionProvider.ACTION_TOKEN_CACHE;
 import static org.keycloak.connections.infinispan.InfinispanConnectionProvider.ALL_CACHES_NAME;
@@ -78,6 +85,8 @@ public final class CacheConfigurator {
 
     private static final String MAX_COUNT_SUFFIX = "MaxCount";
     private static final String OWNER_SUFFIX = "Owners";
+    private static final int STATE_TRANSFER_CHUNK_SIZE = 16;
+    private static final int MIN_NUM_OWNERS_REMOTE_CACHE = 2;
 
     private CacheConfigurator() {
     }
@@ -319,8 +328,67 @@ public final class CacheConfigurator {
         }
     }
 
+    /**
+     * Creates a {@link ConfigurationBuilder} for a cache in a remote Infinispan cluster.
+     * <p>
+     * The returned builder is a template based on the provider's default configuration, which can be freely modified by
+     * the caller before use.
+     *
+     * @param cacheName The name of the cache for which to create the configuration.
+     * @param config    The provider's base configuration scope, which may contain cache-specific customizations.
+     * @param sites     An array of remote site names for cross-site replication backups. If null or empty, cross-site
+     *                  replication will be disabled.
+     * @return A {@link ConfigurationBuilder} for the specified cache, or {@code null} if no configuration exists for
+     * the given {@code cacheName}.
+     */
+    public static ConfigurationBuilder getRemoteCacheConfiguration(String cacheName, Config.Scope config, String[] sites) {
+        return switch (cacheName) {
+            case CLIENT_SESSION_CACHE_NAME, OFFLINE_CLIENT_SESSION_CACHE_NAME ->
+                    remoteCacheConfigurationBuilder(cacheName, config, sites, RemoteAuthenticatedClientSessionEntity.class);
+            case USER_SESSION_CACHE_NAME, OFFLINE_USER_SESSION_CACHE_NAME ->
+                    remoteCacheConfigurationBuilder(cacheName, config, sites, RemoteUserSessionEntity.class);
+            case AUTHENTICATION_SESSIONS_CACHE_NAME ->
+                    remoteCacheConfigurationBuilder(cacheName, config, sites, RootAuthenticationSessionEntity.class);
+            case LOGIN_FAILURE_CACHE_NAME ->
+                    remoteCacheConfigurationBuilder(cacheName, config, sites, LoginFailureEntity.class);
+            case ACTION_TOKEN_CACHE, WORK_CACHE_NAME -> remoteCacheConfigurationBuilder(cacheName, config, sites, null);
+            default -> null;
+        };
+    }
+
     // private methods below
 
+    private static ConfigurationBuilder remoteCacheConfigurationBuilder(String name, Config.Scope config, String[] sites, Class<?> indexedEntity) {
+        var builder = new ConfigurationBuilder();
+        builder.clustering().cacheMode(CacheMode.DIST_SYNC);
+        builder.clustering().hash().numOwners(Math.max(MIN_NUM_OWNERS_REMOTE_CACHE, config.getInt(numOwnerConfigKey(name), MIN_NUM_OWNERS_REMOTE_CACHE)));
+        builder.clustering().stateTransfer().chunkSize(STATE_TRANSFER_CHUNK_SIZE);
+        builder.encoding().mediaType(MediaType.APPLICATION_PROTOSTREAM);
+        builder.statistics().enable();
+
+        if (indexedEntity != null) {
+            builder.indexing().enable().addIndexedEntities(Marshalling.protoEntity(indexedEntity));
+        }
+
+        if (sites == null || sites.length == 0) {
+            return builder;
+        }
+
+        // we need transactions for cross-site to detect deadlock and rollback any changes.
+        builder.transaction()
+                .transactionMode(TransactionMode.TRANSACTIONAL)
+                .useSynchronization(false)
+                .lockingMode(LockingMode.PESSIMISTIC);
+        for (var site : sites) {
+            builder.sites().addBackup()
+                    .site(site)
+                    .strategy(BackupConfiguration.BackupStrategy.SYNC)
+                    .backupFailurePolicy(BackupFailurePolicy.FAIL)
+                    .stateTransfer().chunkSize(STATE_TRANSFER_CHUNK_SIZE);
+        }
+        return builder;
+    }
+
     private static void configureRevisionCache(ConfigurationBuilderHolder holder, String baseCache, String revisionCache, long defaultMaxEntries) {
         var baseBuilder = holder.getNamedConfigurationBuilders().get(baseCache);
         if (baseBuilder == null) {

model/infinispan/src/main/java/org/keycloak/spi/infinispan/impl/remote/DefaultCacheRemoteConfigProviderFactory.java

@@ -34,8 +34,6 @@ import org.infinispan.client.hotrod.configuration.Configuration;
 import org.infinispan.client.hotrod.configuration.ConfigurationBuilder;
 import org.infinispan.client.hotrod.configuration.ExhaustedAction;
 import org.infinispan.client.hotrod.impl.ConfigurationProperties;
-import org.infinispan.commons.dataconversion.MediaType;
-import org.infinispan.configuration.cache.CacheMode;
 import org.jboss.logging.Logger;
 import org.keycloak.Config;
 import org.keycloak.config.CachingOptions;
@@ -46,8 +44,10 @@ import org.keycloak.models.KeycloakSessionFactory;
 import org.keycloak.provider.EnvironmentDependentProviderFactory;
 import org.keycloak.provider.ProviderConfigProperty;
 import org.keycloak.provider.ProviderConfigurationBuilder;
+import org.keycloak.spi.infinispan.CacheEmbeddedConfigProviderSpi;
 import org.keycloak.spi.infinispan.CacheRemoteConfigProvider;
 import org.keycloak.spi.infinispan.CacheRemoteConfigProviderFactory;
+import org.keycloak.spi.infinispan.impl.embedded.CacheConfigurator;
 
 import javax.net.ssl.SSLContext;
 
@@ -79,6 +79,7 @@ public class DefaultCacheRemoteConfigProviderFactory implements CacheRemoteConfi
     private static final String CONNECTION_POOL_EXHAUSTED_ACTION = "connectionPoolExhaustedAction";
     private static final String AUTH_REALM = "authRealm";
     private static final String SASL_MECHANISM = "saslMechanism";
+    private static final String BACKUP_SITES = "backupSites";
 
     // configuration defaults
     private static final String CLIENT_INTELLIGENCE_DEFAULT = ClientIntelligence.getDefault().name();
@@ -135,6 +136,7 @@ public class DefaultCacheRemoteConfigProviderFactory implements CacheRemoteConfi
         addConnectionPoolConfig(builder);
         addTlsConfig(builder);
         addAuthenticationConfig(builder);
+        addCreateRemoteCachesConfig(builder);
         return builder.build();
     }
 
@@ -259,29 +261,19 @@ public class DefaultCacheRemoteConfigProviderFactory implements CacheRemoteConfi
         }
     }
 
-    private static boolean shouldCreateRemoteCaches() {
-        // TODO convert to SPI option when we want to support this feature
-        // http://github.com/keycloak/keycloak/issues/32129
-        return Boolean.getBoolean("kc.cache-remote-create-caches");
-    }
-
-    private static void configureRemoteCaches(ConfigurationBuilder builder) {
-        if (!shouldCreateRemoteCaches()) {
-            return;
-        }
-        // fall back for distributed caches if not defined
-        logger.warn("Creating remote cache in external Infinispan server. It should not be used in production!");
-        var baseConfig = defaultRemoteCacheBuilder();
+    private void configureRemoteCaches(ConfigurationBuilder builder) {
+        var sites = keycloakConfiguration.getArray(BACKUP_SITES);
 
+        // hijack the embedded cache configuration :)
+        var embeddedKeycloakConfig = Config.scope(CacheEmbeddedConfigProviderSpi.SPI_NAME, DefaultCacheRemoteConfigProviderFactory.PROVIDER_ID);
         skipSessionsCacheIfRequired(Arrays.stream(CLUSTERED_CACHE_NAMES))
-                .forEach(name -> builder.remoteCache(name).configuration(baseConfig.toStringConfiguration(name)));
-    }
-
-    private static org.infinispan.configuration.cache.Configuration defaultRemoteCacheBuilder() {
-        var builder = new org.infinispan.configuration.cache.ConfigurationBuilder();
-        builder.clustering().cacheMode(CacheMode.DIST_SYNC);
-        builder.encoding().mediaType(MediaType.APPLICATION_PROTOSTREAM);
-        return builder.build();
+                .forEach(name -> {
+                    var cacheConfig = CacheConfigurator.getRemoteCacheConfiguration(name, embeddedKeycloakConfig, sites);
+                    if (cacheConfig == null) {
+                        return;
+                    }
+                    builder.remoteCache(name).configuration(cacheConfig.build().toStringConfiguration(name));
+                });
     }
 
     // configuration option below
@@ -357,4 +349,8 @@ public class DefaultCacheRemoteConfigProviderFactory implements CacheRemoteConfi
                 .type(ProviderConfigProperty.STRING_TYPE)
                 .add();
     }
+
+    private static void addCreateRemoteCachesConfig(ProviderConfigurationBuilder builder) {
+        copyFromOption(builder, BACKUP_SITES, "sites", ProviderConfigProperty.LIST_TYPE, CachingOptions.CACHE_REMOTE_BACKUP_SITES, false);
+    }
 }

quarkus/config-api/src/main/java/org/keycloak/config/CachingOptions.java

@@ -1,6 +1,7 @@
 package org.keycloak.config;
 
 import java.io.File;
+import java.util.List;
 
 import com.google.common.base.CaseFormat;
 
@@ -28,6 +29,7 @@ public class CachingOptions {
     public static final String CACHE_REMOTE_USERNAME_PROPERTY = CACHE_REMOTE_PREFIX + "-username";
     public static final String CACHE_REMOTE_PASSWORD_PROPERTY = CACHE_REMOTE_PREFIX + "-password";
     public static final String CACHE_REMOTE_TLS_ENABLED_PROPERTY = CACHE_REMOTE_PREFIX + "-tls-enabled";
+    public static final String CACHE_REMOTE_BACKUP_SITES_PROPERTY = CACHE_REMOTE_PREFIX + "-backup-sites";
 
     private static final String CACHE_METRICS_PREFIX = "cache-metrics";
     public static final String CACHE_METRICS_HISTOGRAMS_ENABLED_PROPERTY = CACHE_METRICS_PREFIX + "-histograms-enabled";
@@ -180,6 +182,11 @@ public class CachingOptions {
             .defaultValue(Boolean.TRUE)
             .build();
 
+    public static final Option<List<String>> CACHE_REMOTE_BACKUP_SITES = OptionBuilder.listOptionBuilder(CACHE_REMOTE_BACKUP_SITES_PROPERTY, String.class)
+            .category(OptionCategory.CACHE)
+            .description("Configures a list of backup sites names to where the external Infinispan cluster backups the Keycloak data.")
+            .build();
+
     public static Option<Integer> maxCountOption(String cache) {
         return new OptionBuilder<>(cacheMaxCountProperty(cache), Integer.class)
               .category(OptionCategory.CACHE)

quarkus/runtime/src/main/java/org/keycloak/quarkus/runtime/configuration/mappers/CachingPropertyMappers.java

@@ -152,6 +152,11 @@ final class CachingPropertyMappers implements PropertyMapperGrouping {
                 fromOption(CachingOptions.CACHE_METRICS_HISTOGRAMS_ENABLED)
                         .isEnabled(MetricsPropertyMappers::metricsEnabled, MetricsPropertyMappers.METRICS_ENABLED_MSG)
                         .to("kc.spi-cache-embedded--default--metrics-histograms-enabled")
+                        .build(),
+                fromOption(CachingOptions.CACHE_REMOTE_BACKUP_SITES)
+                        .isEnabled(CachingPropertyMappers::remoteHostSet, CachingPropertyMappers.REMOTE_HOST_SET)
+                        .to("kc.spi-cache-remote--default--backup-sites")
+                        .paramLabel("sites")
                         .build()
         );
 

quarkus/tests/integration/src/test/java/org/keycloak/it/storage/database/ExternalInfinispanTest.java

@@ -46,7 +46,6 @@ public class ExternalInfinispanTest {
             "--cache-remote-tls-enabled=false",
             "--spi-cache-embedded-default-site-name=ISPN",
             "--spi-load-balancer-check-remote-poll-interval=500",
-            "-Dkc.cache-remote-create-caches=true",
             "--verbose"
     })
     void testLoadBalancerCheckFailureWithMultiSite() {
@@ -64,7 +63,6 @@ public class ExternalInfinispanTest {
             "--cache-remote-tls-enabled=false",
             "--spi-cache-embedded-default-site-name=ISPN",
             "--spi-load-balancer-check-remote-poll-interval=500",
-            "-Dkc.cache-remote-create-caches=true",
             "--verbose"
     })
     void testLoadBalancerCheckFailureWithRemoteOnlyCaches() {

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartDevHelpAll.approved.txt

@@ -101,6 +101,9 @@ Cache:
 --cache-metrics-histograms-enabled <true|false>
                      Enable histograms for metrics for the embedded caches. Default: false.
                        Available only when metrics are enabled.
+--cache-remote-backup-sites <sites>
+                     Configures a list of backup sites names to where the external Infinispan
+                       cluster backups the Keycloak data. Available only when remote host is set.
 --cache-remote-host <hostname>
                      The hostname of the external Infinispan cluster. Available only when feature
                        'multi-site' or 'clusterless' is set. Required when feature 'multi-site' or
@@ -715,4 +718,4 @@ Bootstrap Admin:
 Do NOT start the server using this command when deploying to production.
 
 Use 'kc.sh start-dev --help-all' to list all available options, including build
-options.
+options.

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartHelpAll.approved.txt

@@ -102,6 +102,9 @@ Cache:
 --cache-metrics-histograms-enabled <true|false>
                      Enable histograms for metrics for the embedded caches. Default: false.
                        Available only when metrics are enabled.
+--cache-remote-backup-sites <sites>
+                     Configures a list of backup sites names to where the external Infinispan
+                       cluster backups the Keycloak data. Available only when remote host is set.
 --cache-remote-host <hostname>
                      The hostname of the external Infinispan cluster. Available only when feature
                        'multi-site' or 'clusterless' is set. Required when feature 'multi-site' or
@@ -720,4 +723,4 @@ By default, this command tries to update the server configuration by running a
       $ kc.sh start '--optimized'
 
 By doing that, the server should start faster based on any previous
-configuration you have set when manually running the 'build' command.
+configuration you have set when manually running the 'build' command.

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testStartOptimizedHelpAll.approved.txt

@@ -102,6 +102,9 @@ Cache:
 --cache-metrics-histograms-enabled <true|false>
                      Enable histograms for metrics for the embedded caches. Default: false.
                        Available only when metrics are enabled.
+--cache-remote-backup-sites <sites>
+                     Configures a list of backup sites names to where the external Infinispan
+                       cluster backups the Keycloak data. Available only when remote host is set.
 --cache-remote-host <hostname>
                      The hostname of the external Infinispan cluster. Available only when feature
                        'multi-site' or 'clusterless' is set. Required when feature 'multi-site' or
@@ -627,4 +630,4 @@ By default, this command tries to update the server configuration by running a
       $ kc.sh start '--optimized'
 
 By doing that, the server should start faster based on any previous
-configuration you have set when manually running the 'build' command.
+configuration you have set when manually running the 'build' command.

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testUpdateCompatibilityCheckHelpAll.approved.txt

@@ -101,6 +101,9 @@ Cache:
 --cache-metrics-histograms-enabled <true|false>
                      Enable histograms for metrics for the embedded caches. Default: false.
                        Available only when metrics are enabled.
+--cache-remote-backup-sites <sites>
+                     Configures a list of backup sites names to where the external Infinispan
+                       cluster backups the Keycloak data. Available only when remote host is set.
 --cache-remote-host <hostname>
                      The hostname of the external Infinispan cluster. Available only when feature
                        'multi-site' or 'clusterless' is set. Required when feature 'multi-site' or
@@ -710,4 +713,4 @@ Bootstrap Admin:
 --bootstrap-admin-username <username>
                      Temporary bootstrap admin username. Used only when the master realm is
                        created. Available only when bootstrap admin password is set. Default:
-                       temp-admin.
+                       temp-admin.

quarkus/tests/integration/src/test/resources/org/keycloak/it/cli/dist/approvals/cli/help/HelpCommandDistTest.testUpdateCompatibilityMetadataHelpAll.approved.txt

@@ -99,6 +99,9 @@ Cache:
 --cache-metrics-histograms-enabled <true|false>
                      Enable histograms for metrics for the embedded caches. Default: false.
                        Available only when metrics are enabled.
+--cache-remote-backup-sites <sites>
+                     Configures a list of backup sites names to where the external Infinispan
+                       cluster backups the Keycloak data. Available only when remote host is set.
 --cache-remote-host <hostname>
                      The hostname of the external Infinispan cluster. Available only when feature
                        'multi-site' or 'clusterless' is set. Required when feature 'multi-site' or
@@ -708,4 +711,4 @@ Bootstrap Admin:
 --bootstrap-admin-username <username>
                      Temporary bootstrap admin username. Used only when the master realm is
                        created. Available only when bootstrap admin password is set. Default:
-                       temp-admin.
+                       temp-admin.

test-framework/core/src/main/java/org/keycloak/testframework/infinispan/InfinispanExternalServer.java

@@ -35,8 +35,7 @@ public class InfinispanExternalServer extends InfinispanContainer implements Inf
                 "cache-remote-tls-enabled", "false",
                 "spi-cache-embedded-default-site-name", "ispn",
                 "spi-load-balancer-check-remote-poll-interval", "500",
-                "spi-cache-remote-default-client-intelligence", "BASIC",
-                "-Dkc.cache-remote-create-caches", "true"
+                "spi-cache-remote-default-client-intelligence", "BASIC"
         );
     }
 }

testsuite/integration-arquillian/tests/base/src/main/java/org/keycloak/testsuite/arquillian/containers/AbstractQuarkusDeployableContainer.java

@@ -233,8 +233,6 @@ public abstract class AbstractQuarkusDeployableContainer implements DeployableCo
             commands.add("--cache-remote-password=Password1!");
             commands.add("--cache-remote-tls-enabled=false");
             commands.add("--spi-cache-embedded-default-site-name=test");
-            configuration.appendJavaOpts("-Dkc.cache-remote-create-caches=true");
-            System.setProperty("kc.cache-remote-create-caches", "true");
         }
 
         if (!suiteContext.get().isAuthServerMigrationEnabled()) {