From e3546a52916c0edd71ff5936023bc415583b9e65 Mon Sep 17 00:00:00 2001 From: Loann Le <84412881+taoism4504@users.noreply.github.com> Date: Sun, 27 Mar 2022 15:19:42 -0700 Subject: [PATCH] Vault documentation: added new vault lambda extension doc (#14717) * added new lambda extension doc * Suggestion to split the commands into individual code block (#14719) * Suggestion to split the commands into individual code block * Update lambda-extension-cache.mdx * Update lambda-extension-cache.mdx Co-authored-by: Yoko Hyakuna --- .../docs/platform/{aws-mp => aws}/index.mdx | 0 .../platform/aws/lambda-extension-cache.mdx | 304 ++++++++++++++++++ .../docs/platform/{aws-mp => aws}/run.mdx | 0 website/data/docs-nav-data.json | 10 +- 4 files changed, 311 insertions(+), 3 deletions(-) rename website/content/docs/platform/{aws-mp => aws}/index.mdx (100%) create mode 100644 website/content/docs/platform/aws/lambda-extension-cache.mdx rename website/content/docs/platform/{aws-mp => aws}/run.mdx (100%) diff --git a/website/content/docs/platform/aws-mp/index.mdx b/website/content/docs/platform/aws/index.mdx similarity index 100% rename from website/content/docs/platform/aws-mp/index.mdx rename to website/content/docs/platform/aws/index.mdx diff --git a/website/content/docs/platform/aws/lambda-extension-cache.mdx b/website/content/docs/platform/aws/lambda-extension-cache.mdx new file mode 100644 index 0000000000..9ff465f817 --- /dev/null +++ b/website/content/docs/platform/aws/lambda-extension-cache.mdx @@ -0,0 +1,304 @@ +--- +layout: docs +page_title: Vault Lambda Extension Caching +description: >- + Supports caching to the local proxy server for the Vault Lambda Extension. +--- + +# Vault Lambda Extension + +AWS Lambda lets you run code without provisioning and managing servers. +You can use the [quick-start](https://github.com/hashicorp/vault-lambda-extension/tree/0af1a648bfa4b9f37a04dd4311d8355f5c3902c3/quick-start) directory which has an end-to-end example if you would like to try out the extension from scratch. + +~> **Note**: If you decide to create one from scratch, be aware that this will create real infrastructure with an associated cost as per AWS' pricing. + +## Usage + +To use the extension, include the following ARN as a layer in your Lambda function: + +```text +arn:aws:lambda:us-east-1:634166935893:layer:vault-lambda-extension:11 +``` + +Where region may be any of `af-south-1`, `ap-east-1`, `ap-northeast-1`, +`ap-northeast-2`, `ap-northeast-3`, `ap-south-1`, `ap-southeast-1`, +`ap-southeast-2`, `ca-central-1`, `eu-central-1`, `eu-north-1`, `eu-south-1`, +`eu-west-1`, `eu-west-2`, `eu-west-3`, `me-south-1`, `sa-east-1`, `us-east-1`, +`us-east-2`, `us-west-1`, `us-west-2`. + +The extension authenticates with Vault using [AWS IAM auth](/docs/auth/aws), +and all configuration is supplied via environment variables. There are two methods +to read secrets, which can both be used side-by-side: + +- **Recommended**: Make unauthenticated requests to the extension's local proxy + server at `http://127.0.0.1:8200`, which will add an authentication header and + proxy to the configured `VAULT_ADDR`. Responses from Vault are returned without + modification. +- Configure environment variables such as `VAULT_SECRET_PATH` for the extension + to read a secret and write it to disk. + +### Adding the extension to your existing Lambda and Vault infrastructure + +#### Requirements + +- ARN of the role your Lambda runs as +- An instance of Vault accessible from AWS Lambda +- An authenticated `vault` client +- A secret in Vault that you want your Lambda to access, and a policy giving read access to it +- Your Lambda function must use one of the [supported runtimes][lambda-supported-runtimes] for extensions + +#### Step 1. Configure Vault + +Enable the aws auth method. + +```shell-session +$ vault auth enable aws +``` + +Configure the AWS client to use the default options. + +```shell-session +$ vault write -force auth/aws/config/client +``` + +Create a role prefixed with the AWS environment name. + +```shell-session +$ vault write auth/aws/role/vault-lambda-role \ + auth_type=iam \ + bound_iam_principal_arn="${YOUR_ARN}" \ + policies="${YOUR_POLICY}" \ + ttl=1h +``` + +#### Step 2. Option a) Install the extension for Lambda functions packaged in zip archives + +If you deploy your Lambda function as a zip file, you can add the extension +to your Lambda layers using the console or [cli](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-using): + +```text +arn:aws:lambda::634166935893:layer:vault-lambda-extension:11 +``` + +#### Step 2. Option b) Install the extension for Lambda functions packaged in container images + +Alternatively, if you deploy your Lambda function as a container image, simply +place the built binary in the `/opt/extensions` directory of your image. + +Fetch the binary from +[releases.hashicorp.com](https://releases.hashicorp.com/vault-lambda-extension/). +The following command requires cURL. + +```shell-session +$ curl --silent https://releases.hashicorp.com/vault-lambda-extension/0.5.0/vault-lambda-extension_0.5.0_linux_amd64.zip \ + --output vault-lambda-extension.zip +``` + +Unzip the donwloaded binary. + +```shell-session +$ unzip vault-lambda-extension.zip +``` + +Optionally, you can verify the integrity of the downloaded zip using the release +archive checksum verification instructions +[here](https://www.hashicorp.com/security). + +Or to build the binary from source. This requires Golang installed. Run from the root of this repository. + +```shell-session +$ GOOS=linux GOARCH=amd64 go build -o vault-lambda-extension main.go +``` + +#### Step 3. Configure vault-lambda-extension + +Configure the extension using [Lambda environment +variables](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html): + +Set the Vault API address. + +```shell-session +$ VAULT_ADDR=http://vault.example.com:8200 +``` + +Set the AWS IAM auth mount point (i.e. the path segment after `auth/` from above). + +```shell-session +$ VAULT_AUTH_PROVIDER=aws +``` + +Set the Vault role to authenticate as. Must be configured for the ARN of your +Lambda's role. + +```shell-session +$ VAULT_AUTH_ROLE=vault-lambda-role +``` + +The path to a secret in Vault. Can be static or dynamic. Unless +VAULT_SECRET_FILE is specified, JSON response will be written to +`/tmp/vault/secret.json`. + +```shell-session +$ VAULT_SECRET_PATH=secret/lambda-app/token +``` + +If everything is correctly set up, your Lambda function can then read secret +material from `/tmp/vault/secret.json`. The exact contents of the JSON object +will depend on the secret read, but its schema is the [Secret struct](https://github.com/hashicorp/vault/blob/api/v1.0.4/api/secret.go#L15) +from the Vault API module. + +Alternatively, you can send normal Vault API requests over HTTP to the local +proxy at `http://127.0.0.1:8200`, and the extension will add authentication +before forwarding the request. Vault responses will be returned unmodified. +Although local communication is over plain HTTP, the proxy server will use TLS +to communicate with Vault if configured to do so as detailed below. + +## Configuration + +The extension is configured via [Lambda environment variables](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html). +Most of the [Vault CLI client's environment variables](/docs/commands#environment-variables) are available, +as well as some additional variables to configure auth, which secret(s) to read and +where to write secrets. + +| Environment variable | Description | Required | Example value | +| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | --------------------------- | +| `VLE_VAULT_ADDR` | Vault address to connect to. Takes precedence over `VAULT_ADDR` so that clients of the proxy server can be configured using the standard `VAULT_ADDR` | No | `https://x.x.x.x:8200` | +| `VAULT_ADDR` | Vault address to connect to if `VLE_VAULT_ADDR` is not set. Required if `VLE_VAULT_ADDR` is not set | No | `https://x.x.x.x:8200` | +| `VAULT_AUTH_PROVIDER` | Name of the configured AWS IAM auth route on Vault | Yes | `aws` | +| `VAULT_AUTH_ROLE` | Vault role to authenticate as | Yes | `lambda-app` | +| `VAULT_IAM_SERVER_ID` | Value to pass to the Vault server via the [`X-Vault-AWS-IAM-Server-ID` HTTP Header for AWS Authentication](/api-docs/auth/aws#iam_server_id_header_value) | No | `vault.example.com` | +| `VAULT_SECRET_PATH` | Secret path to read, written to `/tmp/vault/secret.json` unless `VAULT_SECRET_FILE` is specified | No | `database/creds/lambda-app` | +| `VAULT_SECRET_FILE` | Path to write the JSON response for `VAULT_SECRET_PATH` | No | `/tmp/db.json` | +| `VAULT_SECRET_PATH_FOO` | Additional secret path to read, where FOO can be any name, as long as a matching `VAULT_SECRET_FILE_FOO` is specified | No | `secret/lambda-app/token` | +| `VAULT_SECRET_FILE_FOO` | Must exist for any correspondingly named `VAULT_SECRET_PATH_FOO`. Name has no further effect beyond matching to the correct path variable | No | `/tmp/token` | +| `VAULT_TOKEN_EXPIRY_GRACE_PERIOD` | Period at the end of the proxy server's auth token TTL where it will consider the token expired and attempt to re-authenticate to Vault. Must have a unit and be parseable by `time.Duration`. Defaults to 10s. | No | `1m` | +| `VAULT_STS_ENDPOINT_REGION` | The region of the STS regional endpoint to authenticate with. If the AWS IAM auth mount specified uses a regional STS endpoint, then this needs to match the region of that endpoint. Defaults to using the global endpoint, or the region the Lambda resides in if `AWS_STS_REGIONAL_ENDPOINTS` is set to `regional` | No | `eu-west-1` | + +The remaining environment variables are not required, and function exactly as +described in the [Vault Commands (CLI)](/docs/commands#environment-variables) documentation. However, +note that `VAULT_CLIENT_TIMEOUT` cannot extend the timeout beyond the 10s +initialization timeout imposed by the Extensions API when writing files to disk. + +| Environment variable | Description | Required | Example value | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------- | +| `VAULT_CACERT` | Path to a PEM-encoded CA certificate _file_ on the local disk | No | `/tmp/ca.crt` | +| `VAULT_CAPATH` | Path to a _directory_ of PEM-encoded CA certificate files on the local disk | No | `/tmp/certs` | +| `VAULT_CLIENT_CERT` | Path to a PEM-encoded client certificate on the local disk | No | `/tmp/client.crt` | +| `VAULT_CLIENT_KEY` | Path to an unencrypted, PEM-encoded private key on disk which corresponds to the matching client certificate | No | `/tmp/client.key` | +| `VAULT_CLIENT_TIMEOUT` | Timeout for Vault requests. Default value is 60s. Ignored by proxy server. **Any value over 10s will exceed the Extensions API timeout and therefore have no effect** | No | `5s` | +| `VAULT_MAX_RETRIES` | Maximum number of retries on `5xx` error codes. Defaults to 2. Ignored by proxy server | No | `2` | +| `VAULT_SKIP_VERIFY` | Do not verify Vault's presented certificate before communicating with it. Setting this variable is not recommended and voids Vault's [security model](/docs/internals/security) | No | `true` | +| `VAULT_TLS_SERVER_NAME` | Name to use as the SNI host when connecting via TLS | No | `vault.example.com` | +| `VAULT_RATE_LIMIT` | Only applies to a single invocation of the extension. See [Vault Commands (CLI)](/docs/commands#environment-variables) documentation for details. Ignored by proxy server | No | `10` | +| `VAULT_NAMESPACE` | The namespace to use for pre-configured secrets. Ignored by proxy server | No | `education` | +| `VAULT_DEFAULT_CACHE_TTL` | The time to live configuration (aka, TTL) of the cache used by proxy server. Must have a unit and be parsable as a time.Duration. Required for caching to be enabled. | No | `15m` | +| `VAULT_DEFAULT_CACHE_ENABLED` | Enable caching for all requests, without needing to set the X-Vault-Cache-Control header for each request. Must be set to a boolean value. | No | `true` | + +### AWS STS client configuration + +In addition to Vault configuration, you can configure certain aspects of the STS +client the extension uses through the usual AWS environment variables. For example, +if your Vault instance's IAM auth is configured to use regional STS endpoints: + +```shell-session +$ vault write auth/aws/config/client \ + sts_endpoint="https://sts.eu-west-1.amazonaws.com" \ + sts_region="eu-west-1" +``` + +Then you may need to configure the extension's STS client to also use the regional +STS endpoint by setting `AWS_STS_REGIONAL_ENDPOINTS=regional`, because both the AWS Golang +SDK and Vault IAM auth method default to using the global endpoint in many regions. +See documentation on [`sts_regional_endpoints`](https://docs.aws.amazon.com/credref/latest/refdocs/setting-global-sts_regional_endpoints.html) for more information. + +### Caching + +Caching can be configured for the extension's local proxy server so that it does +not forward every HTTP request to Vault. The main consideration behind caching +design is to make caching an explicit opt-in at the request level, so that it is +only enabled for scenarios where caching makes sense without negative impact in +others. To turn on caching, set the environment variable +`VAULT_DEFAULT_CACHE_TTL` to a valid value that is parsable as a time.Duration +in Go, for example, "15m", "1h", "2m3s" or "1h2m3s", depending on application +needs. An invalid or negative value will be treated the same as a missing value, +in which case, caching will not be set up and enabled. + +Then requests with HTTP method of "GET", and the HTTP header +`X-Vault-Cache-Control: cache` will be returned directly from the cache if +there's a cache hit. On a cache miss the request will be forwarded to Vault and +the response returned and cached. If the header is set to +`X-Vault-Cache-Control: recache`, the cache lookup will be skipped, and the +request will be forwarded to Vault and the response returned and cached. +Currently, the cache key is a hash of the request URL path, headers, body, and +token. + +Caching may also be enabled for all requests by setting the environment variable +`VAULT_DEFAULT_CACHE_ENABLE` to `true`. Then all requests will be fetched and/or +cached as though the header `X-Vault-Cache-Control: cache` was present. Setting +the header to `nocache` on a request will opt-out of caching entirely in this +configuration. Setting the header to `recache` will skip the cache lookup and +return and cache the response from Vault as described previously. + +## Limitations + +Secrets written to disk or returned from the proxy server will not be automatically +refreshed when they expire. This is particularly important if you configure the +extension to write secrets to disk, because the extension will only write to disk +once per execution environment, rather than once per function invocation. If you +use [provisioned concurrency](https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html#configuration-concurrency-provisioned) or if your Lambda +is invoked often enough that execution contexts live beyond the lifetime of the +secret, then secrets on disk are likely to become invalid. + +In line with [Lambda best practices](https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html), we recommend avoiding +writing secrets to disk where possible, and exclusively consuming secrets via +the proxy server. However, the proxy server will still not perform any additional +processing with returned secrets such as automatic lease renewal. The proxy server's +own Vault auth token is the only thing that gets automatically refreshed. It will +synchronously refresh its own token before proxying requests if the token is +expired (including a grace window), and it will attempt to renew its token if the +token is nearly expired but renewable. + +## Performance impact + +AWS Lambda pricing is based on [number of invocations, time of execution and memory +used](https://aws.amazon.com/lambda/pricing/). The following table details some approximate performance +related statistics to help assess the cost impact of this extension. Note that AWS +Lambda allocates [CPU power in proportion to memory](https://docs.aws.amazon.com/lambda/latest/dg/configuration-memory.html) so results +will vary widely. These benchmarks were run with the minimum 128MB of memory allocated +so aim to give an approximate baseline. + +| Metric | Value | Description | Derivation | +| -------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | +| Layer size | 8.5MB | The size of the unpacked extension binary | `ls -la` | +| Init latency | 8.5ms (standard deviation 2.4ms) + one network round trip to authenticate to Vault | Extension initialization time in a new execution environment. Authentication round trip time will be highly deployment-dependent | Instrumented in code | +| Invoke latency | <1ms | The base processing time for each function invocation, assuming no calls to the proxy server | Instrumented in code | +| Memory impact | 12MB | The marginal impact on "Max Memory Used" when running the extension | As reported by Lambda when running Hello World function with and without extension | + +## Uploading to your own AWS account and region + +If you would like to upload the extension as a Lambda layer in your own AWS +account and region, you can do the following: + +```shell-session +$ curl --silent https://releases.hashicorp.com/vault-lambda-extension/0.5.0/vault-lambda-extension_0.5.0_linux_amd64.zip \ + --output vault-lambda-extension.zip +``` + +Set your target AWS region. + +```shell-session +$ export REGION="YOUR REGION HERE" +``` + +Upload the extension as a Lambda layer. + +```shell-session +$ aws lambda publish-layer-version \ + --layer-name vault-lambda-extension \ + --zip-file "fileb://vault-lambda-extension.zip" \ + --region "${REGION}" +``` + +## Learn + +For step-by-step instructions, refer to the [Vault AWS Lambda Extension](https://learn.hashicorp.com/tutorials/vault/aws-lambda) tutorial for details on how to create an AWS Lambda function and use the Vault Lambda Extension to authenticate with Vault. diff --git a/website/content/docs/platform/aws-mp/run.mdx b/website/content/docs/platform/aws/run.mdx similarity index 100% rename from website/content/docs/platform/aws-mp/run.mdx rename to website/content/docs/platform/aws/run.mdx diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index 73562d1e9b..1f9e3193a4 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -1470,15 +1470,19 @@ ] }, { - "title": "AWS Marketplace", + "title": "AWS", "routes": [ { "title": "Overview", - "path": "platform/aws-mp" + "path": "platform/aws" + }, + { + "title": "Vault Lambda Extension Cache", + "path": "platform/aws/lambda-extension-cache" }, { "title": "Running Vault", - "path": "platform/aws-mp/run" + "path": "platform/aws/run" } ] },