diff --git a/website/content/docs/upgrading/deduplication/acl-policy-templates.mdx b/website/content/docs/upgrading/deduplication/acl-policy-templates.mdx new file mode 100644 index 0000000000..0d588ff374 --- /dev/null +++ b/website/content/docs/upgrading/deduplication/acl-policy-templates.mdx @@ -0,0 +1,364 @@ +--- +layout: docs +page_title: Resolve ACL policy templates +description: >- + Resolve templated ACL behavior for deduplicated entities and groups. +--- + +# Resolve deduplication impact on ACL policy templates + +Fix templated ACL policy behavior for entities and groups renamed during +identity deduplication. + + + +- You are running Vault 1.19 or later. +- You have [deduplication **renaming** targets in your system logs](/vault/docs/upgrading/identity-deduplication). +- You have admin permission on the relevant Vault server or cluster. + + + + +## How renaming affect ACL policies + +Operators use +[policy templating](/vault/docs/concepts/policies#templated-policies) to specify +permissions that dynamically resolve based on specific properties of an +authenticated entity. During deduplication, Vault renames entities or groups +with duplicate identities by appending a universally unique identifier (UUID) to +all but one of the entities to avoid future duplication. + +If a templated ACL policy relies on an entity or group name, renaming duplicate +identities could inadvertently change the meaning of the policy. In practice, +identity deduplication is unlikely to grant unintended access to an existing +resource since the renamed identity contains a UUID that should +not collide with other resource paths. But the rename might **remove** access to +existing resources after deduplication if the associated resource policy +references the old entity or group name. + +For example, assume you have the following templated ACL policy that grants +users access to a `kv` plugin: + +```hcl +path "kv/users/{{identity.entity.name}}/*" { + capabilities = ["read", "create", "update"] +} +``` + +The policy grants a user entity named `janine` permission to read and write +key-value pairs to the `kv` plugin on the path `kv/users/janine/`. If there are +duplicate entities named `janine`, Vault renames all but one of entities +to `janine-` during deduplication. + +The ACL policy now gives the renamed entity permission to read and write +While the renamed entity can still access the `kv` plugin, the change in +no longer grants access to path `kv/users/janine/`. As a result, the user can no +longer access their data stored under `kv/users/janine/`. + + +## Find affected ACL policies + +Deduplication has the potential to break templated ACL policies that rely on +entity or group name keys: + +Template key | Description +------------------------------------------------------------- | ---------------- +`identity.entity.name` | Entity name +`identity.entity.aliases..name` | Entity alias name for the given mount +`identity.groups.ids..name` | Group name for the given group ID +`identity.groups.names..id` | Group ID for the given group name +`identity.groups.names..metadata.` | Metadata for the given group name and key + +You can use the following `bash` script to check for templated policies using +entity or group names: + + + + + +```bash +policy_list=$(vault policy list) + +for policy in ${policy_list} ; do + + # Get the policy details + policy_details=$(vault policy read ${policy}) + + # Check for a name based template key + if echo "${policy_details}" | grep -q "\.name" ; then + echo "" + echo "Policy name: ${policy}" + fi + echo "${policy_details}" | + tr -s ' ' '\n' | + grep "\.name" --label=" template key" -H +done +``` + + + + + +```bash +policy_list=$( + curl \ + --silent \ + --request GET \ + --header "X-Vault-Token: ${VAULT_TOKEN}" \ + ${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/sys/policy \ + | jq .data.policies | jq -c '.[]' | tr -d '"' +) + +for policy in ${policy_list} ; do + + # Get the policy details + policy_details=$( + curl \ + --silent \ + --request GET \ + --header "X-Vault-Token: ${VAULT_TOKEN}" \ + ${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/sys/policy/${policy} \ + | jq '.data.rules' | sed 's/\\"/"/g' | sed 's/\\n/ /g' + ) + + # Check for a name based template key + if echo "${policy_details}" | grep -q "\.name" ; then + echo "" + echo "Policy name: ${policy}" + fi + echo "${policy_details}" | + tr -s ' ' '\n' | + grep "\.name" --label=" template key" -H +done +``` + + + + + +Compare the returned list of policies and template keys against the list of +deduplication targets identified in your system logs to determine if the policy +will break after duplication. + +If you identify policies that may break during deduplication, consult with +the relevant teams to determine if the policies are still used and if you should +address the renaming problem before or after deduplication. + +If you find policies that need remediation, there are two ways to fix user +access: + +1. Update relevant templated ACL policies. +1. Move the affected resource. + + +## Solution 1: Update relevant ACL policies + +The easiest way to deal with broken ACL policy templates due to deduplication is +to define new policies or add rules to existing policies that explicitly grants +access to previous resources for the affected entities. + +If you have a large number of duplicate identities, it may be easier to +pre-populate a custom metadata field (e.g., `prev_name`) for the renamed +entities. Then create a single policy that uses the custom metadata field +instead of the entity name and attach that policy to the renamed entities. + + + + + +Assume you have a file called `rename-targets.txt` with a list of entity names +currently attached to a templated ACL policy, which will no longer work after +deduplication. + +1. Use `vault write` and the + [`{namespace}/identity/entity/name/{name}`](/vault/api-docs/secret/identity/entity#create-update-entity-by-name) + endpoint to update your target entries with a custom metadata field called + `old_name` set to the **current** entity name: + + ```bash + while read entity_name; do + + if [[ "" = "${entity_name}" ]] ; then continue ; fi + + # Create a payload file with the new metadata field + echo -n '{"metadata": { "old_name": "'${entity_name}'"}}' > ./metadata.json + + # Save the metadata to the entity + vault write /identity/entity/name/${entity_name} @metadata.json + + done < rename-targets.txt + ``` + +1. Create a new templatized ACL policy file that replaces the entity name + reference with a reference to the metadata field. For example: + + ```hcl + path "kv/users/{{identity.entity.metadata.old_name}}/*" { + capabilities = ["read", "create", "update"] + } + ``` + +1. Use `vault policy write` to create a new ACL policy with the policy + definition file: + + ```shell-session + $ vault policy write + ``` + + For example: + + + + ```shell-session + $ vault policy write "kv-access-preservation" ./dedupe-policy.hcl + ``` + + + +1. Use `vault read` with the + [`{namespace}/identity/entity/name/{name}`](/vault/api-docs/secret/identity/entity#create-update-entity-by-name) + path to read in the existing policy assignments and add the new policy to the + target entities. For example: + + ```bash + policy_name="" + while read entity_name; do + + if [[ "" = "${entity_name}" ]] ; then continue ; fi + + # Create a payload file with new policy added to any existing policy + # assignments + vault read \ + -field policies \ + -format json \ + /identity/entity/name/${entity_name} \ + | jq ". + [\"${policy_name}\"] | {policies: .}" > policy_update.json + + # Update the policy assignment for the entity + vault write /identity/entity/name/${entity_name} @policy_update.json + + done < rename-targets.txt + ``` + + + + + +Assume you have a file called `rename-targets.txt` with a list of entity names +currently attached to a templated ACL policy, which will no longer work after +deduplication. + +1. Use the + [`{namespace}/identity/entity/name/{name}`](/vault/api-docs/secret/identity/entity#create-update-entity-by-name) + endpoint to update your target entries with a custom metadata field called + `old_name` set to the **current** entity name: + + ```bash + while read entity_name; do + + if [[ "" = "${entity_name}" ]] ; then continue ; fi + + # Create a payload file with the new metadata + echo -n '{"metadata": { "old_name": "'${entity_name}'"}}' > ./metadata.json + + # Save the metadata to the entity + curl \ + --request POST \ + --header "X-Vault-Token: ${VAULT_TOKEN}" \ + --data @./metadata.json \ + ${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/identity/entity/name/${entity_name} + + done < rename-targets.txt + ``` + +1. Create a new templatized ACL policy file that replaces the entity name + reference with a reference to the metadata field. For example: + + ```hcl + path "kv/users/{{identity.entity.metadata.old_name}}/*" { + capabilities = ["read", "create", "update"] + } + ``` + +1. Escape your policy file and make a `POST` call to the + [`{namespace}/sys/policy/{policy_name}`](/vault/api-docs/system/policy#create-update-policy) + with your policy details: + + ```shell-session + $ jq -Rs '{ "policy": . | gsub("[\\r\\n\\t]"; "") }' | + curl \ + --request POST \ + --header "X-Vault-Token: ${VAULT_TOKEN}" \ + "$( + ``` + + For example: + + + + ```shell-session + $ jq -Rs '{ "policy": . | gsub("[\\r\\n\\t]"; "") }' ./dedupe-policy.hcl | + curl \ + --request POST \ + --header "X-Vault-Token: ${VAULT_TOKEN}" \ + --data "$( + + `/sys/mounts/{plugin_mount_path}` does not return data on success. + + +1. Make a `GET` call to the + [`{namespace}/identity/entity/name/{name}`](/vault/api-docs/secret/identity/entity#create-update-entity-by-name) + endpoint to read in the existing policy assignments and add the new policy to the + target entities. For example: + + ```bash + policy_name="" + while read entity_name; do + + if [[ "" = "${entity_name}" ]] ; then continue ; fi + + # Create a payload file with new policy added to any existing policy assignments + curl \ + --request GET \ + --header "X-Vault-Token: ${VAULT_TOKEN}" \ + ${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/identity/entity/name/${entity_name} \ + | jq ".data.policies + [\"${policy_name}\"] | {policies: .}" > policy_update.json + + # Update the policy assignment for the entity + curl \ + --request POST \ + --header "X-Vault-Token: ${VAULT_TOKEN}" \ + --data @policy_update.json \ + ${VAULT_ADDR}/v1/${VAULT_NAMESPACE}/identity/entity/name/${entity_name} + + done < rename-targets.txt + ``` + + + + + + +## Solution 2: Move the affected resource + +Some resources are immovable and others are difficult to move. If you decide you +can, and want, to move resources affected by the renaming: + +1. Inventory the relevant resources and the associated mount paths. +1. Determine which resources you will migrate before activating the + deduplication flag. + - Resources migrating **before** deduplication will be unavailable on the + old and new paths until **deduplication** completes. + - Resources migrating **after** deduplication will be unavailable until + **migration** completes. +1. Move any resources flagged for migration before deduplication and alert the + relevant users that the paths will remain unavailable until deduplication + completes. +1. [Enable the deduplication option for Vault](/vault/docs/upgrading/deduplication#dedupe-flag) +1. Move any resources flagged for migration after deduplication and alert the + relevant users that the paths will remain unavailable until migration + completes and the changes fully propagate. diff --git a/website/content/docs/upgrading/deduplication/different-case.mdx b/website/content/docs/upgrading/deduplication/different-case.mdx new file mode 100644 index 0000000000..50b3a3d23b --- /dev/null +++ b/website/content/docs/upgrading/deduplication/different-case.mdx @@ -0,0 +1,221 @@ +--- +layout: docs +page_title: Resolve different-case entity alias duplicates +description: >- + Fix duplicate identities for entity aliases due to case differences. +--- + +# Fix different-case entity alias duplicates + +Fix duplicate identities for entity aliases due to case differences. + +**You must review different-case entity alias duplicates before enabling forced +identity deduplication due to the potential security risk of incorrectly merging +distinct aliases**. + + + + +- You are running Vault 1.19 or later. +- You have [deduplication **merge** targets in your system logs](/vault/docs/upgrading/identity-deduplication). +- You have admin permission on the relevant Vault server or cluster. + + + + +## Why duplicates happen + +Historical bugs in Vault allowed authN mounts to create duplicate entries of the +same entity alias if the aliases used difference cases. For example, if an +external LDAP system returns unnormalized usernames or changes the normalization +scheme over time. By default, Vault uses case-insensitive matching for the +usernames and considers `username` and `uSeRnAmE` to be the same alias. But, +past bugs have left some users with different-case duplicates in storage. + +To avoid the security impact of incorrectly merging identities that do not +represent the same user, Vault **tries** to honor past behavior by switching the +identity engine to use case-sensitive matching mode. But case-sensitive matching +is poorly supported, deviates from the foundational security assumptions Vault +makes, and can lead to unexpected behavior. + +## Example server log + +The Vault system log provides warnings about duplicate aliases, including: + +- the alias string +- the associated mount path (`mount accessor`) +- the alias ID (`id`) +- the ID for the entity linked to the alias (`canonical_id`) + + + +```text +[WARN] identity: 2 different-case entity alias duplicates found (potential security risk) +[WARN] identity: entity-alias "alias-case" with mount accessor "auth_userpass_34aca7ec" duplicates 1 others: id=df3568a4-3b65-4104-9481-1129ecbed72f canonical_id=5f013d99-a6c7-9a00-6ad5-4ad724b14f60 force_deduplication="would merge into entity 7da76b0d-fe9b-a125-3362-2a8ff055dcf8" +[WARN] identity: entity-alias "alias-cAsE" with mount accessor "auth_userpass_34aca7ec" duplicates 1 others: id=2992253b-1e99-4e47-b6f9-afb0c7cedf7a canonical_id=7da76b0d-fe9b-a125-3362-2a8ff055dcf8 force_deduplication="would merge others into this entity" +[WARN] identity: entity-alias "alias-case" with mount accessor "auth_userpass_a555989a" duplicates 1 others: id=0e4bd46e-a868-4dd4-a34a-cb73f097e3a5 canonical_id=1ce06951-e2fd-0923-8ae0-a0e2c6c2378b force_deduplication="would merge into entity 37f41fcf-5e15-f13f-248d-9fa8405b1ecd" +[WARN] identity: entity-alias "alias-cAsE" with mount accessor "auth_userpass_a555989a" duplicates 1 others: id=6cafe546-665c-4bd0-a4fd-5e699ba413c1 canonical_id=37f41fcf-5e15-f13f-248d-9fa8405b1ecd force_deduplication="would merge others into this entity" +[WARN] identity: end of different-case entity-alias duplicates +``` + + + +When reviewing system logs for duplicate resolution, always consider the authN +mount and namespace when grouping potential duplicates. For example, despite +having identical, case-insensitive names in the example logs, the four aliases +represent two distinct identities because they come from different authN mounts: + +- `alias-case` and `alias-cAsE` are duplicates in the mount `auth_userpass_34aca7ec` + and will merge under the alias with canonical ID `7da76b0d-fe9b-a125-3362-2a8ff055dcf8` + during deduplication. +- `alias-case` and `alias-cAsE` are duplicates in the mount `auth_userpass_a555989a` + and will merge with the alias with canonical ID `37f41fcf-5e15-f13f-248d-9fa8405b1ecd` + during deduplication. + + +To resolve different-case duplicates, you must determine if the case difference +is a **mergeable duplicate** or **unmergeable duplicate**: + +- For mergeable duplicates, the case difference is **unintentional** and the + duplicate entries represent the same logical entity. You can ignore mergeable + duplicates and let Vault force-merge them for you to establish and enforce + default identity matching behavior. + +- For unmergeable duplicates, the case difference is **deliberate** and the + source identity provider **intended** to distinguish between different + entities using case. For example, `Alice` and `alice` are different entities + that correspond to different humans with distinct permission sets. **You must + resolve unmergeable duplicates before activating the deduplication flag.** + + + +## Case 1: Resolving mergeable duplicates + +Vault will force-merge duplicates once you enable forced identity deduplication +with the `force-identity-deduplication` flag. Vault also merges any policies +attached to the duplicates so the final entry has the union of all permissions +granted to the duplicates. + +We recommend confirming the expected merge behavior **before** enabling +deduplication by reviewing the `force_deduplication` label in the log line to: + +1. confirm the identified duplicates are all mergeable. +1. confirm permissions granted by the unified policy are appropriate for the + identity. + +After deduplication, users can continue to log in using any case combination of +their username, but those logins will map to a single entity gated by the same +policies per the expected, default Vault behavior. + + +## Case 2: Resolving unmergeable duplicates + + + +We strongly recommend resolving duplicates and moving away from case-sensitive +usernames as soon as possible to restore the security model supported by Vault. + +Vault does not officially support case-sensitive names because it carries +significant security risks. And unresolved duplicates could merge at any time +due to unrelated changes in storage such as when deleting related entities or +groups. + + + +When different-case variations of an alias name actually represent different +logical users or entities in an external system, investigate each alias before +resolving duplicates. You **must** resolve the duplicates on each mount before +enabling forced identity deduplication. + +**If you confirm you need to keep the duplicated aliases**, you will need to +reconfigure the authN mount or external service to stop creating case-sensitive +aliases. For example, if the authN plugin allows it, you could update the plugin +configuration to append unique IDs to alias names to differentiate between +similar usernames. + +Regardless of how you choose to address unmergeable duplicates, be mindful of +how the modified behavior may disrupt users as you roll out the change. + + + +We are not aware of specific integrations where changing mount or external +service behaviors is necessary and expect those situations to be rare. If you +determine your deployment requires changes to authN mounts or external services, +we encourage Vault Enterprise customers to work with HashiCorp support staff to +determine the best strategy and implementation. + + + +**If you can confirm you do not need the duplicated alias, or the entire auth +mount**, delete all but one of the entity aliases on each mount using the ID +noted in the log line with `id=`. + + + + + +Use `vault delete` with the `/identity/entity-alias/id/{id}` path to delete the +duplicate identity: + +```shell-session +$ vault delete /identity/entity-alias/id/ +``` + +For example, if one of the duplicate entity log entries includes +`id=df3568a4-3b65-4104-9481-1129ecbed72f`: + + + +```shell-session +$ vault delete /identity/entity-alias/id/df3568a4-3b65-4104-9481-1129ecbed72f + +Success! Data deleted (if it existed) at: identity/entity-alias/id/df3568a4-3b65-4104-9481-1129ecbed72f +``` + + + + + + + +Call the [`/identity/entity-alias/id/{id}`](/vault/api-docs/secret/identity/entity-alias#delete-entity-alias-by-id) +endpoint to delete the duplicate entry: + +```shell-session +$ curl \ + --request DELETE \ + --header "X-Vault-Token: ${VAULT_TOKEN}" \ + ${VAULT_ADDR}/v1/identity/entity-alias/id/ +``` + +For example, if one of the duplicate entity log entries includes +`id=df3568a4-3b65-4104-9481-1129ecbed72f`: + + + +```shell-session +$ curl \ + --request DELETE \ + --header "X-Vault-Token: ${VAULT_TOKEN}" \ + ${VAULT_ADDR}/v1/identity/entity-alias/id/df3568a4-3b65-4104-9481-1129ecbed72f | jq +``` + + + + + + + +After you resolve the duplicates by deleting them or reconfiguring the relevant +services: + +1. If the node is a standby, restart the node manually. + +1. Recheck the system logs for the current node to confirm the duplicates no + longer appear in the log during unseal. + + +## Next steps + +Once you confirm any remaining duplicates can be force-merged safely, you can +[enable forced identity deduplication](/vault/docs/upgrading/deduplication#dedupe-flag). diff --git a/website/content/docs/upgrading/deduplication/entity-group.mdx b/website/content/docs/upgrading/deduplication/entity-group.mdx new file mode 100644 index 0000000000..008cdadc41 --- /dev/null +++ b/website/content/docs/upgrading/deduplication/entity-group.mdx @@ -0,0 +1,136 @@ +--- +layout: docs +page_title: Fix entity and group duplicates +description: >- + Fix duplicate identities for Vault entities and groups +--- + +# Fix entity and group duplicates + +Fix duplicate identities for entities and groups targeted for renaming. + + + +- You are running Vault 1.19 or later. +- You have [deduplication **renaming** targets in your system logs](/vault/docs/upgrading/identity-deduplication). +- You have admin permission on the relevant Vault server or cluster. + + + + +## Why duplicates happen + +Historical bugs in Vault allowed authN mounts to create duplicate entries of the +same entity or group name if the names used difference cases. By default, Vault +uses case-insensitive matching for names and considers `bob` and `BOB` to be the +same name. But, past bugs have allowed for identical or different-case +duplicates in entity and group names. + +Duplicate entity and group names can cause unexpected behavior and make working +with Vault more difficult: + +- Fetching entities/groups by name returns the first identity found. Depending + on your Vault version, the identity returned can vary depending on the server + handling the request and when the last seal/unseal event occurred. +- Deleting entities/groups by name is not guaranteed to delete all duplicates. +- All lookup requests **must** use ID to find the right identity. + individual identity. + +## Example server log + +The Vault system log provides warnings about duplicate entities and groups, +including: + +- the entity/group name +- the associated namespace (`namespace ID`) +- the entity/group ID (`id`) +- the expected deduplication action (`force_deduplication`) + + + +```text +2025-01-28T13:15:13.641-0800 [WARN] identity: 2 entity duplicates found +2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-cAsE" with namespace ID "root" duplicates 2 others: id=2562b42d-f603-ac6b-2591-8a20dd050897 force_deduplication="would not rename" +2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-case" with namespace ID "root" duplicates 2 others: id=290a643d-6043-da5e-943f-3a3d09e4ecbd force_deduplication="would rename to entity-case-290a643d-6043-da5e-943f-3a3d09e4ecbd" +2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-case" with namespace ID "root" duplicates 2 others: id=b0141be5-3f03-a1c7-a57b-02f045d04426 force_deduplication="would rename to entity-case-b0141be5-3f03-a1c7-a57b-02f045d04426" +2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-cAsE" with namespace ID "sYMXY" duplicates 2 others: id=95f0743b-a1d5-26da-b4ef-a50490da0787 force_deduplication="would not rename" +2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-case" with namespace ID "sYMXY" duplicates 2 others: id=9d3be96f-490a-9625-3118-896bd2b3a5f3 force_deduplication="would rename to entity-case-9d3be96f-490a-9625-3118-896bd2b3a5f3" +2025-01-28T13:15:13.641-0800 [WARN] identity: entity "entity-case" with namespace ID "sYMXY" duplicates 2 others: id=d82231d2-3716-6b9c-e80d-7d09c0409739 force_deduplication="would rename to entity-case-d82231d2-3716-6b9c-e80d-7d09c0409739" +2025-01-28T13:15:13.641-0800 [WARN] identity: end of entity duplicates +2025-01-28T13:15:13.641-0800 [WARN] identity: 2 group duplicates found +2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-case" with namespace ID "root" duplicates 2 others: id=8ad26e0c-8cf6-5b67-7c77-6571fa374f34 force_deduplication="would not rename" +2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-cAsE" with namespace ID "root" duplicates 2 others: id=9fe86ea0-f80c-1131-5be1-1d6e3b70237f force_deduplication="would rename to group-cAsE-9fe86ea0-f80c-1131-5be1-1d6e3b70237f" +2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-case" with namespace ID "root" duplicates 2 others: id=32dd070c-c1f8-c796-9a71-15887014b813 force_deduplication="would rename to group-case-32dd070c-c1f8-c796-9a71-15887014b813" +2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-case" with namespace ID "sYMXY" duplicates 2 others: id=8aaeff7e-7343-c883-1e0c-c5c9968f75a5 force_deduplication="would not rename" +2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-case" with namespace ID "sYMXY" duplicates 2 others: id=f11277b3-d985-4d72-d2e9-9c8c6c0db02c force_deduplication="would rename to group-case-f11277b3-d985-4d72-d2e9-9c8c6c0db02c" +2025-01-28T13:15:13.641-0800 [WARN] identity: group "group-cAsE" with namespace ID "sYMXY" duplicates 2 others: id=7c753d07-b0d9-e13b-6184-48247b8f7504 force_deduplication="would rename to group-cAsE-07c753d07-b0d9-e13b-6184-48247b8f7504" +2025-01-28T13:15:13.641-0800 [WARN] identity: end of group duplicates +``` + + + +Duplicate entities and groups might be exact matches or differ in case (for +example, `Admin` and `admin`). But when reviewing system logs for duplicate +resolution, always consider the associated namespace when grouping potential +duplicates. For example, despite having identical, case-insensitive names in the +example logs, Vault only targets 8 for renaming because the names belong to +different namespaces: + +Identity type | Old name | New name +------------- | ------------------- | -------- +Entity | `root/entity-cAsE` | Unchanged +Entity | `root/entity-case` | `root/entity-case-290a643d-6043-da5e-943f-3a3d09e4ecbd` +Entity | `root/entity-case` | `root/entity-case-b0141be5-3f03-a1c7-a57b-02f045d04426` +Entity | `sYMXY/entity-cAsE` | Unchanged +Entity | `sYMXY/entity-case` | `sYMXY/entity-case-9d3be96f-490a-9625-3118-896bd2b3a5f3` +Entity | `sYMXY/entity-case` | `sYMXY/entity-case-d82231d2-3716-6b9c-e80d-7d09c0409739` +Group | `root/group-case` | Unchanged +Group | `root/group-cAsE` | `group-cAsE-9fe86ea0-f80c-1131-5be1-1d6e3b70237f` +Group | `root/group-case` | `group-case-32dd070c-c1f8-c796-9a71-15887014b813` +Group | `sYMXY/group-case` | Unchanged +Group | `sYMXY/group-case` | `group-case-f11277b3-d985-4d72-d2e9-9c8c6c0db02c` +Group | `sYMXY/group-cAsE` | `group-cAsE-07c753d07-b0d9-e13b-6184-48247b8f7504` + + +The automatic renaming process is security safe and preserves existing data. +Renaming **does not** grant new permissions to existing tokens and **does not** +delete data, so you can manually roll back the change if necessary. + + +## Resolving entity and group duplicates + +In most cases, renaming preserves resource access such that logins and +permissions remain unaffected. But, there are two edge cases you may need to +address before deduplication: **templated policies** and **external references**. + +- If you use templated ACL policies that reference an entity or group name as + part of the resource path, deduplication may affect access to those resources. + +- If you use Terraform to manage Vault resources or have other systems outside + Vault that reference entity or group names, those references will break when the + deduplication process renames those entities/groups. + + +For each duplicate identified by the logs, you have 3 options: + +1. **If you know you do not need or use the renaming target**, you can use + the + [`/identity/entity/id/{id}`](/vault/api-docs/secret/identity/entity#delete-entity-by-id) and + [`/identity/group/id/{id}`](/vault/api-docs/secret/identity/group#delete-group-by-id) + endpoints to delete the identities by ID. + +1. **If you know the risk to templated policies or external references is low or + nonexistent**, you can opt to ignore the duplicates and address issues + if/when they occur. + + +1. **If you cannot confirm the risk to templated policies or external references**, + review the guidance in the following troubleshooting guide: + - [Resolve deduplication impact on ACL policy templates](/vault/docs/upgrading/deduplication/acl-policy-templates) + - [Resolve deduplication impact on Terraform resource references](/vault/docs/upgrading/deduplication/external-refs) + +## Next steps + +Once you are comfortable that all the entity and group duplicates are properly +addressed, you can +[enable forced identity deduplication](/vault/docs/upgrading/deduplication#dedupe-flag). diff --git a/website/content/docs/upgrading/deduplication/index.mdx b/website/content/docs/upgrading/deduplication/index.mdx new file mode 100644 index 0000000000..023c420915 --- /dev/null +++ b/website/content/docs/upgrading/deduplication/index.mdx @@ -0,0 +1,225 @@ +--- +layout: docs +page_title: Resolve duplicate identities +description: >- + Find duplicate identities in your Vault cluster and safely resolve them. +--- + +# Find and resolve duplicate Vault identities + +Bugs in Vault versions before 1.19 might lead to duplicate +[identities](/vault/docs/concepts/identity) for entities, aliases, and groups. +Duplicate identities in the persistent storage of a Vault cluster can cause +unexpected behavior as duplicate identities are outside typical expectations and +test scenarios for Vault. + +Vault server logs include information to help you identify when duplicate +identities exist in a given cluster. We strongly recommend identifying +and resolving duplicates as soon as possible to return your Vault clusters to +normal, supported behavior. + +## Before you start + +- **You must have Vault 1.19 or later running on all clusters**. Vault only logs + deduplication details once you upgrade to 1.19+. +- **You must have admin permissions for the Vault cluster**. + +## Step 1: Look for duplicates + +To identify duplicates in server logs: + +1. Look for `core: post-unseal setup starting` to find the last unseal operation + in the system logs of the active node. For example: + + ```text + [INFO] core: post-unseal setup starting + ... ... + [INFO] core: post-unseal setup complete + ``` + +1. Check for `DUPLICATES DETECTED` between the `setup starting` and + `setup complete` entries in the system log. For example: + + ```text + [WARN] identity: DUPLICATES DETECTED, see following logs for details [...] + ``` + +If you do not see entries for `DUPLICATES DETECTED`, the current cluster is +clean and you can move to the next cluster. If you confirm none of the logs +have duplicates, you can jump to [Step #4](#dedupe-flag). + + + +If you use replication, repeat the duplication check on all primary clusters and +all **performance replication** (PR) secondary clusters. PR secondary clusters +handle client requests and may have additional, **local** duplicates. + +Disaster recovery (DR) secondaries do not handle client requests, so you do not +need to check DR secondary clusters separately. + + + +## Step 2: Find deduplication targets + +If you find `DUPLICATES DETECTED` in the system logs, create a list of same-case +aliases and renaming targets from the system log to make working through +duplicate resolution easier. If you have a large number of deduplication targets, +you can use the following script to pull out the relevant items and create +four files: + +- **`unseal-process.log`** - section of the system log specific to the unseal + process. +- **`merge-details.txt`** - same-case identities Vault will auto-merge during + deduplication. +- **`rename-targets.txt`** - entities and group names in the form + `namespace/name` Vault will rename during deduplication. +- **`rename-details.txt`** - full rename details for rename targets. + +```bash + +declare -A identity_key=( + ['entity_alias']="identity: entity-alias" + ['entity_rename']="identity: entity" + ['group_rename']="identity: group" +) + +declare -A match_str=( + ['has_rename']="would rename to" + ['has_merge']="would merge" + ['rename']="s/.*would rename to \(.*\)\".*/\1/" + ['group_name']="s/.*identity: group \"\(.*\)\" with.*/\1/" + ['entity_name']="s/.*identity: entity \"\(.*\)\" with.*/\1/" + ['namespace']="s/.*namespace ID \"\(.*\)\" duplicates.*/\1/" +) + +declare -A rename_targets + +# Grab the unseal portion of the log +log_start="core: post-unseal setup starting" +log_stop="core: post-unseal setup complete" +sed "/${log_start}/,/${log_stop}/!d;/${log_stop}/q" ../short.log > unseal-process.log + +while read line; +do + + # The log line relates to a renamed identity + if [[ "${line}" == *"${match_str['has_rename']}"* ]] ; then + + if [[ "${line}" == *"${identity_key['entity_rename']}"* ]] ; then + type="${identity_key['entity_rename']}" + name_match="${match_str['entity_name']}" + fi + if [[ "${line}" == *"${identity_key['group_rename']}"* ]] ; then + type="${identity_key['group_rename']}" + name_match="${match_str['group_name']}" + fi + + new=$(echo $line | sed -e "${match_str['rename']}") + old=$(echo $line | sed -e "${name_match}") + space=$(echo $line | sed -e "${match_str['namespace']}") + rename_targets["${space}/${old}"]="[${type}] ${space}/${old} --> ${space}/${new}" + + fi + + # The line indicates a same-case merge operation + if [[ "${line}" == *"${match_str['has_merge']}"* ]] ; then + merge_details=$(echo $line | sed -e "${identity_key['entity_alias']}") + echo ${merge_details} >> merge-details.txt + fi + +done < unseal-process.log + +# Save the rename target information to files for further processing +root_ns="root/" +for name in "${!rename_targets[@]}" +do + echo ${name//"${root_ns}"/""} >> rename-targets.txt + echo ${rename_targets["${name}"]} >> rename-details.txt +done +``` + + +## Step 3: Resolve any duplicates + +If you **did not** find duplicates in the log, you can jump to +[Step #5](#dedupe-flag). + +If you **did** find duplicates in the logs: + +1. Identify the duplication type for each entry in your server logs: + - **In PR/DR deployments**: identify duplication types on your primary + clusters. + - **In PR deployments**: identify duplication types for **local aliases** on + your performance secondaries. Local duplicates exist independently on the + secondary cluster and require explicit resolution but follow the same + process as resolving non-local different-case entity alias duplicates. + +1. Follow the appropriate de-duplication steps based on the duplication type: + - [Fix different-case entity alias duplicates](/vault/docs/upgrading/deduplication/different-case) + - [Fix entity and group duplicates](/vault/docs/upgrading/deduplication/entity-group) + + +## Step 4: Prepare for (possible) latency impacts + +When you activate deduplication for a cluster, the cluster reloads the in-memory +cache of all entities, aliases and groups. When deduplication replicates to +performance replication secondaries and performance standby nodes, those nodes +also pause and reload their identity system caches. + +Vault remains unsealed during deduplication, but nodes may pause processing +other requests until deduplication completes. On **moderately large** clusters, +deduplication may take long enough to impact request latencies. On **large** +clusters, deduplication may take longer than 30 seconds, which could cause some +requests to timeout. + +In general, deduplication should take less time than sealing and unsealing Vault +and be less disruptive than a regular failover. And clusters that had duplicates +should see their future `unseal` times reduced. + + +## Step 5: Enforce identity de-duplication ((#dedupe-flag)) + +Once you resolve existing duplicates, you can enable the +`force-identity-deduplication` activation flag on the primary cluster using the +[activation flag API path](/vault/api-docs/system/activation-flags). + +Activating feature flags is a one-time, one-way action. Once you activate a +feature gated with a feature flag, you cannot un-activate the feature. + + + + + +```shell-session +vault write -f sys/activation-flags/force-identity-deduplication/activate +``` + + + + + +```shell-session +$ curl \ + --request PUT \ + --header "X-Vault-Token: ${VAULT_TOKEN}" \ + ${VAULT_ADDR}/v1/sys/activation-flags/force-identity-deduplication/activate +``` + + + + + +You can track the start and end of deduplication on each node in the system +by looking for `force-identity-deduplication activated` entries in the logs: + +``` +INFO core: force-identity-deduplication activated, reloading identity store +... +INFO core: force-identity-deduplication activated, reloading identity store complete +``` + +Going forward, Vault requires unique identities and re-runs the deduplication +check as a part of the unseal process. You can review the impact on unseal time +by reviewing system log timestamps. The difference between the log lines +`core: post-unseal setup starting` and `core: post-unseal setup complete` +indicates the total unseal time required for the cluster. \ No newline at end of file diff --git a/website/content/docs/upgrading/deduplication/terraform-refs.mdx b/website/content/docs/upgrading/deduplication/terraform-refs.mdx new file mode 100644 index 0000000000..b50488283b --- /dev/null +++ b/website/content/docs/upgrading/deduplication/terraform-refs.mdx @@ -0,0 +1,160 @@ +--- +layout: docs +page_title: Resolve Terraform config +description: >- + Fix external reference behavior for deduplicated entities and groups in + Terraform config files. +--- + +# Resolve deduplication impact on Terraform resource references + +Fix external reference behavior in Terraform configuration files for entities +and groups renamed during identity deduplication. + + + +- You are running Vault 1.19 or later. +- You have [deduplication **renaming** targets in your system logs](/vault/docs/upgrading/identity-deduplication). +- You have admin permission on the relevant Vault server or cluster. + + + + +## How renaming affects external references + +Renaming entities and groups can break references in Terraform (and other +external services) when those reference refer directly to the entity or group by +name. For example, assume you have a Terraform configuration file with named +identity resources like the following: + + + +```hcl +terraform { + required_providers { + vault = { + source = "hashicorp/vault" + } + } +} + +provider "vault" {} + +resource "vault_identity_entity" "BOB" { + name = "BOB" + policies = ["TEST"] +} + +resource "vault_identity_entity" "bob" { + name = "bob" + policies = ["test"] +} +``` + + + +By default, Vault ignore case when matching identities, treats `BOB` and `bob` +as the same name, and rejects the second resource as a duplicate. However, if +your Vault cluster is running in a mode that allows both resource names due to +historical issues, the resources might exist as separate entities. + +If Vault identifies `bob` and `BOB` as duplicates during deduplication, it +renames one of the identities `-`. After deduplication, Terraform +tries to reapply the previous name for the related resource, but the in-place +update fails because the existing resource now violates the case-insensitive +name constraint on the Vault side. + +For example: + + + +``` +➜ tf_dupe_testing terraform apply +vault_identity_entity.bob: Refreshing state... [id=e8c5e633-fe37-5a49-4a29-32e2643d03bd] +vault_identity_entity.BOB: Refreshing state... [id=2577bc3f-67ab-dab7-93dc-e86f78194ff0] + +Terraform used the selected providers to generate the following execution plan. Resource +actions are indicated with the following symbols: + ~ update in-place + +Terraform will perform the following actions: + + # vault_identity_entity.bob will be updated in-place + ~ resource "vault_identity_entity" "bob" { + + external_policies = false + id = "e8c5e633-fe37-5a49-4a29-32e2643d03bd" + ~ name = "bob-e8c5e633-fe37-5a49-4a29-32e2643d03bd" -> "bob" + # (3 unchanged attributes hidden) + } + +... + +vault_identity_entity.bob: Modifying... [id=e8c5e633-fe37-5a49-4a29-32e2643d03bd] +╷ +│ Error: error updating IdentityEntity "e8c5e633-fe37-5a49-4a29-32e2643d03bd": Error making API request. +│ +│ URL: PUT https://127.0.0.1:8200/v1/identity/entity/id/e8c5e633-fe37-5a49-4a29-32e2643d03bd +│ Code: 400. Errors: +│ +│ * entity name is already in use +│ +│ with vault_identity_entity.bob, +│ on main.tf line 17, in resource "vault_identity_entity" "bob": +│ 17: resource "vault_identity_entity" "bob" { +``` + + + + +## Solution + +The easiest way to deal with renamed entities and groups is to manually update the +the associated resource in your Terraform configuration with the updated name +before forcing deduplication. + + + +Use the same process to identify and update target names for other external +systems that reference an entity or group by name. + + + +For example, if your system logs include lines like the following: + + + +```text +2025-01-28T13:15:13.641-0800 [WARN] identity: entity "bob" with namespace ID "admin" duplicates 1 others: id=8ad26e0c-8cf6-5b67-7c77-6571fa374881 force_deduplication="would not rename" +2025-01-28T13:15:13.641-0800 [WARN] identity: entity "BOB" with namespace ID "admin" duplicates 1 others: id=9fe86ea0-f80c-1199-5ad1-1d01ab70237f force_deduplication="would rename to BOB-9fe86ea0-f80c-1199-5ad1-1d01ab70237f" +``` + + + +You would update any resources associated with `BOB` in your Terraform +configuration files. For example: + + + +```hcl +terraform { + required_providers { + vault = { + source = "hashicorp/vault" + } + } +} + +provider "vault" {} + +resource "vault_identity_entity" "BOB-9fe86ea0-f80c-1199-5ad1-1d01ab70237f" { + name = "BOB-9fe86ea0-f80c-1199-5ad1-1d01ab70237f" + policies = ["TEST"] +} + +resource "vault_identity_entity" "bob" { + name = "bob" + policies = ["test"] +} +``` + + \ No newline at end of file diff --git a/website/content/docs/upgrading/upgrade-to-1.19.x.mdx b/website/content/docs/upgrading/upgrade-to-1.19.x.mdx index 294a00a0a5..960d6f3431 100644 --- a/website/content/docs/upgrading/upgrade-to-1.19.x.mdx +++ b/website/content/docs/upgrading/upgrade-to-1.19.x.mdx @@ -42,6 +42,24 @@ based on the table below. | CE | true | any value other than sha2-512 | An error is returned | Pure Ed25519 | | CE | true | sha2-512 | An error is returned (not supported on CE) | Pure Ed25519 | +### Identity system duplicate cleanup + +**Users should review their server logs after upgrading to see if identity +duplicates are reported.** + +Vault 1.19.0 enables users impacted by historical identity duplicate bugs to +manually trigger a one-time de-duplication process. This process restores the +cluster to supported default behavior by resolving the duplicates. + +Vault 1.19.0 also includes improved reporting in server logs to help diagnose +whether this de-duplication is required. No behavior will change until the +operator activates the relevant flag via the API. + +To understand if you need to take action, consult +[Resolve duplicate identities](/vault/docs/upgrading/deduplication) which +demonstrates the log lines to watch for and how to ensure your resolve +duplicates safely. + ### LDAP user DN search with `upndomain` The github.com/hashicorp/cap/ldap dependency has been upgraded to include a security improvement diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index 327a6d2afd..165659fc8a 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -2551,6 +2551,31 @@ "title": "Upgrade to Raft WAL", "path": "upgrading/raft-wal" }, + { + "title": "Resolve duplicate identities", + "routes": [ + { + "title": "Process overview", + "path": "upgrading/deduplication" + }, + { + "title": "Fix different-case entity alias duplicates", + "path": "upgrading/deduplication/different-case" + }, + { + "title": "Fix entity and group duplicates", + "path": "upgrading/deduplication/entity-group" + }, + { + "title": "Resolve ACL policy templates", + "path": "upgrading/deduplication/acl-policy-templates" + }, + { + "title": "Resolve Terraform config", + "path": "upgrading/deduplication/terraform-refs" + } + ] + }, { "title": "Upgrade to 1.19.x", "path": "upgrading/upgrade-to-1.19.x"