* make router-lookup helper
* add policyPaths arg to flyout and update route cache to map
* update kv flyouts and test coverage
* round out test coverage, rename method from get to lookup
* alphabetize PATH_MAP
* support other change events for inputSearch to allow copy/pasting items
* update overview requests and improve ux for limited permissions
* request each key permissions
* add flyout to pki page header
* update changelog
Co-authored-by: claire bontempo <68122737+hellobontempo@users.noreply.github.com>
* Added census metric names and new file build doesn't complete
* Added metrics. Working on tests
* Tests passing
* Added tests
* Added a nil check
* Try to fix race condition
* Fix some nits
* Some more nits
* Added a go doc
* Fix some data races
* Remove billing storage lock all together
* Revert "Remove billing storage lock all together"
This reverts commit cdf4a518c343d8fd0fbb5de13fd860999ca2d131.
* Fix failing test
* Remove billing storage lock all together
* Removed schema json file
* Revert "Remove billing storage lock all together"
This reverts commit 76970c7d4f7fe98acb993a737aa7410216caf2a8.
Co-authored-by: divyaac <divya.chandrasekaran@hashicorp.com>
* adds playwright
* adds playwright auth setup and kv tests
* removes generated gh action for playwright
* removes testem ignore paths
* consolidates kv e2e workflows into single test
* adds missing ids to key shares and threshold inputs
* updates ariaLabel arg to attribute in enabled and disabled plugin card components
* adds script to start vault with config for playwright tests
* updates playwright setup to initialize and unseal vault and create user for testing rather than using root token
* adds policies for e2e tests
* updates e2e init setup to use web repl for creating token
* moves kv e2e test under superuser directory
* updates playwright config to create projects for multiple user types
Co-authored-by: Jordan Reimer <zofskeez@gmail.com>
Vault currently logs a warning when a duplicate alias is detected. This
happens on upsert (during loading and creation-time). Since MemDB
enforces uniqueness on the primary key (entity/alias ID), any lookup
resulting in an ID match would by definition be an update. Consistently
check for ID equality across entities, groups, and aliases to make sure
we don't error for updates.
Updated TestDuplicateRenameResolver to use a global counter for unique
entity IDs across namespaces. This trues up the tests to enforce realistic
duplicate scenarios.
Addresses: VAULT-35109
Co-authored-by: Mike Palmiotto <mpalmi@ibm.com>
Co-authored-by: GitHub Copilot <noreply@github.com>
* make router-lookup helper
* add policyPaths arg to flyout and update route cache to map
* update kv flyouts and test coverage
* round out test coverage, rename method from get to lookup
Co-authored-by: claire bontempo <68122737+hellobontempo@users.noreply.github.com>
* Make VAULT_EXPERIMENTS work as feature flags.
Make method IsFlagEnabled treat experiments as feature flags so that they
are accessible to plugins.
* Add experiment kmip.client_api.alpha1.
This experiment enables the KMIP client and template API endpoints.
* Use IsExperimentEnabled rather than ValidExperiments.
* Document TestCore_IsFlagEnabled.
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
Reset the PKI certificae counts after enabled additional PKI mounts so that the
certificates for root and intermediates used by those mounts are excluded from
the counts the test is making assertions on.
Co-authored-by: Victor Rodriguez <vrizo@hashicorp.com>
* Change PkiCertificateCountManager.GetCounts() to return a CertCount.
* Add PkiDurationAdjustedCerts field to CertCount.
Add a new field to CertCount to keep track of "duration adjusted" issued
certificates.
Add an x509.Certificate argument to CertCountIncrementer.AddIssuedCertificate.
In the implementation, use the certificate's NotBefore and NotAfter fields to
calculate the validity duration for the certificate, and use that to compute the
duration adjusted units.
* Add the issued certificate to calls to AddIssuedCertificate.
* Add PkiDurationAdjustedCerts when forwarding counts.
Add pki_duration_adjusted_certificate_count to IncrementPkiCount proto.
Update replicationServiceHandler.IncrementPkiCertCountRequest to take into
account the new field.
* Run make proto.
* Update testingPkiCertificateCounter to make assertions on time adjusted counts.
* PR review: Don't use NotAfter.Sub(NotBefore), since time.Duration is max 290 years.
* PR review: Move DurationAdjustedCertificateCount to logical.pki/test_helpers.
Add Bob generated unit tests for logical.durationAdjustedCertificateCount.
* Run make fmt.
Co-authored-by: Victor Rodriguez Rizo <vrizo@hashicorp.com>
* rotation-manager: enable RM to send rotation information to plugins on registration/rotation operations (#11810)
* initial commit for sending NVR to plugins
* add changelog
* add NVR to plugin fields, add RotationInfo to GRPC request handler
* fix tests
* ensure consistent formats on times and ttls
* add translation to allow grpc data transfer
* fix tests and rename fields
* fix missed field renames in tests
* make all methods net-new for backwards compatibility
* update mock plugin and add oss stub back
* remove method with no usages
* Address wrapper comments
* Rebuild proto
* Nil check around SetRotationInfo, return n/a for no last_vault_rotation
* Fix error to match other instances
* Update fields.go
* Return nil if unset for next/last vault rotation times
---------
Co-authored-by: robmonte <17119716+robmonte@users.noreply.github.com>
* Fix return type in stub method
---------
Co-authored-by: vinay-gopalan <86625824+vinay-gopalan@users.noreply.github.com>
Co-authored-by: robmonte <17119716+robmonte@users.noreply.github.com>
* define the proto, define the RPC
* add external unit tests
* change type of request from int to any
* fix things around after conflict merge
* protobuf lint
* protobuf lint issue fix
* implement cadence based testing of the standby worker
* expand the RPC to include transform call counts, update data returned from endpoint to include transform, remove all todo comments, add tests, fix test helper methods
* remove unnecessary locks from util testing methods, remove unused method
* Update vault/consumption_billing.go
* feedback: remove a check on whether system view is initialized
* re-add the check on initialized state of system view
* access the standby state field directly instead of using a method that uses a lock causing deadlock
* feedback: use require.Eventually instead of manual looping to wait for primary to update the count
* feedback: simplify logic to identify initialized consumption billing worker, remove unused methods
* Some edits to test
Added some functions to delete old month data
* Fixed some tests
* fix some merge conflicts
* Delete older month data
* Removed unused functions
* Some merge conflicts
* Adds some more tests
* Edit billing ent test
* temp
* Revert "temp"
This reverts commit f817b9813959fc2ab478669b67c7d67c381ceff6.
* Edited billing bp
* Reverted head
* Edited billing test
* Slight cleaning up
* Moved some things
* Revert changes on pb file
* Edited flaky test
* Edited some nits
* edited a comment
---------
Co-authored-by: divyaac <divya.chandrasekaran@hashicorp.com>
Co-authored-by: Amir Aslamov <amir.aslamov@hashicorp.com>
* PKI: Allow the HTTP post body to be reread for binary paths
* Use a TeeReader to simplify the solution
* Add test and cl
* Apply suggestions from code review
* PR feedback
---------
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
Co-authored-by: Nick Cabatoff <ncabatoff@hashicorp.com>
* improve dismissal logic, use AutomationSnippet component, use wizard service for tracking dismissal
* use class helper to check for multiple nodes when rendering tree chart, add test coverage
* update comments
* add modal for namespace intro and improve reusability
* style updates and general reusability updates
* make intro pages more generic and rename welcome to intro
* update tests
* update styles, use service to track intro visibility, rename components
* Update arg docs
Co-authored-by: lane-wetmore <lane.wetmore@hashicorp.com>
* define the proto, define the RPC
* create new standby and active node workers
* fix bugs: set up consumption billing worker on standby nodes
* add external unit tests
* change type of request from int to any
* remove active node worker - the existing logic already updates and flushes data to storage for active node inside updateBillingMetrics
* fix things around after conflict merge
* make fmt
* protobuf lint
* protobuf lint issue fix
* implement cadence based testing of the standby worker
* protobuf formatting
* expand the RPC to include transform call counts, update data returned from endpoint to include transform, remove all todo comments, add tests, fix test helper methods
* create oss no-op stub for the standby worker
* remove unnecessary locks from util testing methods, remove unused method
* Update vault/consumption_billing.go
* remove unused method
* feedback: remove a check on whether system view is initialized
* re-add the check on initialized state of system view
* access the standby state field directly instead of using a method that uses a lock causing deadlock
* feedback: use require.Eventually instead of manual looping to wait for primary to update the count
* feedback: simplify logic to identify initialized consumption billing worker, remove unused methods
* feedback: use log.Error instead of log.Warn
* Update vault/consumption_billing_util_ent.go
* feedback: increase the timeout for the rpc call to 30 secs
* feedback: fix comments in tests to clarify active node and standby node definitions
---------
Co-authored-by: Amir Aslamov <amir.aslamov@hashicorp.com>
Co-authored-by: divyaac <divya.chandrasekaran@hashicorp.com>
* Copy https://github.com/hashicorp/vault/pull/31733 into main
* fix(transit): prevent panic on restore with missing policy
* test: add unit test for RestorePolicy nil policy validation
* changelog: add entry for transit restore panic fix
* Update changelog/31733.txt
---------
Co-authored-by: Abhishek Dadwal <dadwalabhishek10@gmail.com>
Co-authored-by: Abhishek Dadwal <73817744+Abhishek00810@users.noreply.github.com>
Co-authored-by: Steven Clark <steven@sclark.me>
* WIP
* WIP...
* Update page headers and move logic back
* remove unused stuff
* Fix failing tests
* Use hasNavPermission
* Update raft storage locatin and add namespace link on top
* Update access sidebar link to Access control
* Update order of API_PATHS
* Remove namespace link
Co-authored-by: Kianna <30884335+kiannaquach@users.noreply.github.com>
* [VAULT-33083] support mount external engine
* add "Plugin type" and "Plugin version" fields to the enable mount page
* add changelog
* address copilot review comments
* address PR comments, code cleanup
* fix test failures
* Add support for external plugins registered without a plugin version
* external plugin should be enabled for enterprise only, plugin version should be mandatory for external plugins
* fix tests
* address copilot feedback
* fix failing tests, add unit test coverage
* address PR comments
* address PR comments
* remove dead code
* move no external versions alert
* Only show un-versioned plugin message if there are un-versioned plugins in the catalog.
* address PR comments
* use ApiService instead of custom PluginPinsService; fix failing tests
* revert changes to forms/mount.ts and forms/auth/method.ts
Co-authored-by: Shannon Roberts (Beagin) <beagins@users.noreply.github.com>
* VAULT-42143 Update role counting to official only
* KV thing
* remove old func
* update name
Co-authored-by: Violet Hynes <violet.hynes@hashicorp.com>
* Fix test race in TestPkiCertificateCountManager_IncrementAndConsume (#12275)
* Fix test race in TestPkiCertificateCountManager_IncrementAndConsume.
* Fix signature of stub function forwardPkiCertCounts.
* Fix call to forwardPkiCertCounts.
---------
Co-authored-by: Victor Rodriguez Rizo <vrizo@hashicorp.com>
* fix flacky test by checking for nil billing storage view
* simplify nil checks by checking it inside the worker itself and inside GetBillingSubView to prevent direct method calls in external tests; this we we have single point checks instead of checking it everywhere
* remove redundant nil checks
* add locks and a check against sealed core
* removed a check of sealed core
* re-add individual nil checks
* make getBillingSubView return a bool to to more explicitely indicate possibility of nil storage
Co-authored-by: Amir Aslamov <amir.aslamov@hashicorp.com>
* removes require-input-label ember-template-lint override
* adds aria-label to textarea in MaskedInput component
* more a11y fixes for missing labels
* adds aria-label back to lookup-input
* adds aria-label back to select component
* ensures aria-label has value in select component
* removes duplicate onchange handler from lookup-input component
Co-authored-by: Jordan Reimer <zofskeez@gmail.com>
* Refactor CertificateCounter.IncrementeCount to use a param object.
In preparation to start collecting more information, refactor the
CertificateCounter to take a parameter object which can be later gain more
fields.
* Rework CertificateCounter to use a fluent interface.
Rename method IncrementCount to AddCount.
Remove method AddIssuedCertificate.
Add method Incrementer, which returns an implementation of the new
CertCountIncrementer.
* Add method CertCountIncrement.Add.
* Refactor PkiCertificateCountConsumer to take a CertCountIncrement.
* Fix TestPkiCertificateCountManager_IncrementAndConsume.
* Rename type CertCountIncrement to CertCount.
* Refactor ReadStoredCounts to return a CertCount value.
Co-authored-by: Victor Rodriguez Rizo <vrizo@hashicorp.com>
* improve dismissal logic, use AutomationSnippet component, use wizard service for tracking dismissal
* use class helper to check for multiple nodes when rendering tree chart, add test coverage
* update comments
* add wizard service unit test coverage
* move item filtering into helper func
Co-authored-by: lane-wetmore <lane.wetmore@hashicorp.com>
* adding ibm tests for ent files
* adding debug commands
* adding code changes
* adding reload tests
* remove settings.json
* remove ryboe q
* changing isHashicorpLicense to isIBMLicense and moving DiagnoseCheckLicenseGeneration to core_util_common.go
* fix test
* reverting non-license related tests
* reverting non-license related tests
* removing hashicorp license test
* modify reload server_ent_test.go
* change ibm-license paths
* adding census reload server test
* moving LicensingEntitlementSelectionConfig to core_util_common.go
* add EntReloadLicenseAndConfig to stubs
* fix operator diagnose bug
* move bug fix into ce and ent files
* add more ibm test cases
* Update command/command_testonly/server_testonly_ent_test.go
* address comments
* make fmt
---------
Co-authored-by: akshya96 <87045294+akshya96@users.noreply.github.com>
Co-authored-by: Jenny Deng <jenny.deng@hashicorp.com>
* rough draft
* add some stuff for dynamic secrets
* add some more helpers and sample tests
* new helpers, new tests, refactoring
* Add Basic Smoke SDK Scenario (#11678)
* Add simple test for stepdown election
* Add a smoke_sdk scenario
* add script to run tests locally
* fix up a few things
* VAULT-39746 - Add Tests to Smoke SDK and Cloud Scenarios (#11795)
* Add some go verification steps in enos sdk test run script
* formatting
* Add a smoke_sdk scenario userpass secret engine create test (#11808)
* Add a smoke_sdk scenario userpass secret engine create test
* Add the some additional tests
* Add Smoke tests to Cloud Scenario (#11876)
* Add a smoke_sdk scenario userpass secret engine create test
* Add the some additional tests
* Add smoke testing to cloud
* Add test results to output and test filtering
* comment
* fix test
* fix the smoke scenario
* Address some various feedback
* missed cleanup
* remove node count dependency in the tests
* Fix test perms
* Adjust the testing and clean them up a bit
* formatting
* fmt
* fmt2
* more fmt
* formatting
* tryagain
* remove the docker/hcp divide
* use the SHA as ID
* adjust perms
* Add transit test
* skip blackbox testing in test-go
* copywrite
* Apply suggestion from @brewgator
* Add godoc
* grep cleanup
---------
Co-authored-by: Josh Black <raskchanky@gmail.com>
Co-authored-by: Luis (LT) Carbonell <lt.carbonell@hashicorp.com>
* Update kubernetes and kmip breadcrumbs
* Add breadcrumbs for non ember engines
* LDAP breadcrumbs..
* PKI breadcrumbs..
* Mostly working kv tests
* kv data persona tests
* Fix remaining kv-workflow test
* Route to vault.cluster instead of dashboard
* VAULT-41964 Update client count breadcrumb + test
* VAULT-41143 Remove user dropdown a11y override
* Add test for nav links on community
* Ensure we check for enterprise
* Address feedback
* Use selector instead
Co-authored-by: Kianna <30884335+kiannaquach@users.noreply.github.com>
* PKI: SCEP make GetCACaps response dynamic based on configuration
* add cl
* Use strings.Builder for more efficient response building
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* Update CHANGELOG.md for versions 1.21.3, 1.20.8, 1.19.14, and 1.16.30
Added release notes for version 1.21.3, 1.20.8, 1.19.14, and 1.16.30 Enterprise, including security updates, changes, features, improvements, and bug fixes.
* Update CHANGELOG for version 1.19.14
Added new security, changes, improvements, and bug fixes for version 1.19.14.
* Fix formatting issues in CHANGELOG.md
Removed extra newlines in CHANGELOG.md to improve formatting.
Co-authored-by: Tony Wittinger <anwittin@users.noreply.github.com>
We've already deployed some changed file detection in the CI pipeline. It uses the Github API to fetch a list of all changed files on a PR and then run it through a simple groups categorization pass. It's been a useful strategy in the context of a Pull Request because it does not depend on the local state of the Git repo.
This commit introduces a local git-based file change detection and validation system for the pipeline tool, enabling developers to identify and validate changed files before pushing code. We intend to use the new tool in two primary ways:
- As a Git pre-push hook when pushing new or updated branches. (Implemented here)
- As part of the scheduled automated repository synchronization. (Up next, and it will use the same `git.CheckChangedFilesReq{}` implementation.
This will allow us to guard all pushes to `hashicorp/vault` and `ce/*` branches in `hashicorp/vault-enterprise`, whether run locally on a developer machine or in CI by our service user.
We introduce two new `pipeline` CLI commands:
- `pipeline git list changed-files`
- `pipeline git check changed-files`
Both support specifying what method of git inspection we want to use for the changed files list:
- **`--branch <branch>`**: Lists all files added in the entire history of a specific branch. We use this when pushing a _new_ branch.
- **`--range <range>`**: Lists all changed files within a commit range (e.g., `HEAD~5..HEAD`). We use this when updating an existing branch.
- **`--commit <sha>`**: Lists all changed files in a specific commit (using `git show`). This isn't actually used at all in the pre-push hook but it useful if you wish to inspect a single commit on your branch.
The behavior when passing the `range` and `commit` is similar. We inspect the changed file list either for one or many commits (but with slightly different implementations for efficiency and accuracy. The `branch` option is a bit different. We use it to inspect the branches entire history of changed files for enterprise files before pushing a new branch. We do this to ensure that our branch doesn't accidentally add and then subsequently remove enterprise files, leaving the contents in the history but nothing obvious in the diff.
Each command supports several different output formats. The default is the human readable text table, though `--format json` will write all of the details as valid JSON to STDOUT. When given the `--github-output` command each will write a more concise version of the JSON output to `$GITHUB_OUTPUT`. It differs from our standard JSON output as it has been formatted to be easier to use in Github Actions contexts without requiring complex filtering.
When run, changed files are automatically categorized into logical groups based on their file name, just like our existing changed file detection. A follow-up to this PR will introduce a configuration based system for classifying file groups. This will allow us to create generic support for changed file detection so that many repositories can adopt this pattern.
The major difference in behavior between the two new commands is that the `list` command will always list the changed files for the given method/target, while the `check` command requires one-or-more changed file groups that we want to disallow to be included via the `-g` flag. If any changed files match the given group(s) then the command will fail. That allows us to specify the `enterprise` group and disallow the command to succeed if any of the changed files match the group.
The pre-push git hook now uses this system to prevent accidental pushes, however, it requires the local machine to have the `pipeline` tool in the `$PATH`. This ought not be much of a requirement as a working Go toolchain is required for any Vault developer. When it is not present we explain in our error messages how to resolve the problem and direct them to our slack channel if they need further assistance.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* add a new method to identify whether KMIP is enabled
* add a new prefix for the new metric
* add new methods to store and update the metric
* update the kmip usage in billing
* move the method to ent file since kmip is ent only feature
* add unit tests at the core metrics level
* add new unit tests to test the billing methods for the new metric
* add persistence to test cases
* add external tests for primary and secondary
* account for DR secondaries, add clarifying comments, fix tests
* fmt
* move call of update into update local hwm metric method
* feedback: simplify update method by removing operation to get stored value
* feedback: optimize kmip usage detection by adding atomic tracker to detect usage once kmip mount is enabled
* fmt
* feedback: remove check on DR secondary inside update method but leave it at Get method for now
* feedback: change kmip prefix to a more flexible structure with sub item
* feedback: rename atomic tracker for kmip usage
* feedback: simplify the kmip identifier method
* revert back on kmip path prefix changes
* feedback: move the atomic bool into consumption billing struct
* feedback: remove DR check in Get method since dr needs to have billing data replicated
* add another external test to test local mount detection in perf secondary
* add a no-op oss stub for kmip enabled method
Co-authored-by: Amir Aslamov <amir.aslamov@hashicorp.com>
* PKI: Address failures with SCEP handlers on standby nodes
* add cl
* Add extra safety to protect against nil panics
* PR feedback
* PR feedback 2
* Ignore DR states
Co-authored-by: Steven Clark <steven.clark@hashicorp.com>
* VAULT-41682 - update header to add title tag and centre align with icon
* Fixed linting error
Co-authored-by: mohit-hashicorp <mohit.ojha@hashicorp.com>
