We've already deployed some changed file detection in the CI pipeline. It uses the Github API to fetch a list of all changed files on a PR and then run it through a simple groups categorization pass. It's been a useful strategy in the context of a Pull Request because it does not depend on the local state of the Git repo.
This commit introduces a local git-based file change detection and validation system for the pipeline tool, enabling developers to identify and validate changed files before pushing code. We intend to use the new tool in two primary ways:
- As a Git pre-push hook when pushing new or updated branches. (Implemented here)
- As part of the scheduled automated repository synchronization. (Up next, and it will use the same `git.CheckChangedFilesReq{}` implementation.
This will allow us to guard all pushes to `hashicorp/vault` and `ce/*` branches in `hashicorp/vault-enterprise`, whether run locally on a developer machine or in CI by our service user.
We introduce two new `pipeline` CLI commands:
- `pipeline git list changed-files`
- `pipeline git check changed-files`
Both support specifying what method of git inspection we want to use for the changed files list:
- **`--branch <branch>`**: Lists all files added in the entire history of a specific branch. We use this when pushing a _new_ branch.
- **`--range <range>`**: Lists all changed files within a commit range (e.g., `HEAD~5..HEAD`). We use this when updating an existing branch.
- **`--commit <sha>`**: Lists all changed files in a specific commit (using `git show`). This isn't actually used at all in the pre-push hook but it useful if you wish to inspect a single commit on your branch.
The behavior when passing the `range` and `commit` is similar. We inspect the changed file list either for one or many commits (but with slightly different implementations for efficiency and accuracy. The `branch` option is a bit different. We use it to inspect the branches entire history of changed files for enterprise files before pushing a new branch. We do this to ensure that our branch doesn't accidentally add and then subsequently remove enterprise files, leaving the contents in the history but nothing obvious in the diff.
Each command supports several different output formats. The default is the human readable text table, though `--format json` will write all of the details as valid JSON to STDOUT. When given the `--github-output` command each will write a more concise version of the JSON output to `$GITHUB_OUTPUT`. It differs from our standard JSON output as it has been formatted to be easier to use in Github Actions contexts without requiring complex filtering.
When run, changed files are automatically categorized into logical groups based on their file name, just like our existing changed file detection. A follow-up to this PR will introduce a configuration based system for classifying file groups. This will allow us to create generic support for changed file detection so that many repositories can adopt this pattern.
The major difference in behavior between the two new commands is that the `list` command will always list the changed files for the given method/target, while the `check` command requires one-or-more changed file groups that we want to disallow to be included via the `-g` flag. If any changed files match the given group(s) then the command will fail. That allows us to specify the `enterprise` group and disallow the command to succeed if any of the changed files match the group.
The pre-push git hook now uses this system to prevent accidental pushes, however, it requires the local machine to have the `pipeline` tool in the `$PATH`. This ought not be much of a requirement as a working Go toolchain is required for any Vault developer. When it is not present we explain in our error messages how to resolve the problem and direct them to our slack channel if they need further assistance.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-41857] pipeline(find-artifact): add support for finding artifacts from branches (#11799)
Add support for finding matching workflow artifacts from branches rather than PRs. This allows us to trigger custom HCP image builds from a branch rather than an PR. It also enables us to build and test the HCP image on a scheduled nightly cadence, which we've also enabled.
As part of these changes I also added support for specifying which environment you want to test and threaded it through the cloud scenario now that there are multiple variants. We also make the testing workflow workflow_dispatch-able so that we can trigger HVD testing for any custom image in any environment without building a new image.
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* license: update headers to IBM Corp.
* `make proto`
* update offset because source file changed
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-39424] pipeline(close-origin-pr): add support for closing the origin of copied PRs
When we copy a community contributed Pull Request to Enterprise the
source PR is effectively orphaned, leaving the original PR still
opened, the author unsure of what state the copied PR is in, and any
issues associated with it open.
When the copied PR is closed we ought to close the origin PR if it's
still open, and any other issues that might be associated with either
the origin PR or the copied PR.
We can also add comments to both PRs that include links to each other
and the squash commit to make discovery of the work visible to those
with access to both repos. Unfortunately there is no way to know what
the SHA will be when it's synced so we have to rely on the
'Co-Authored-By:' trailers in commit message.
There are some challenges to this:
- The automation should only execute when copied PRs are closed
- How to determine the origin PR from only the copied PR
- How to determine the PR's linked issues (which the v3 REST API does not expose)
We solved them by:
- Requiring the PR HEAD ref to start with `copy/`
- Encoding the origin PR information in the PR HEAD ref.
e.g. `copy/hashicorp/vault/31580/ryan/VAULT-39424-test-ce`
- Using the V4 GraphQL API to determine "closed issue references"
The result is a new `pipeline` CLI command that can close the origin PR,
all of the issues, and write status comments on each PR with links to
everything to establish omnidirectional linking in the Github UI.
```bash
pipeline github close origin-pull-request 9903
```
* fix feedback
---------
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* pipeline(copy-pr): cherry-pick commits instead of merging
* fix staticcheck for docs in pkg/github
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* [VAULT-39159]: pipeline: add support for querying HCP image service
In order to facilitate testing Vault Enterprise directly in HCP we need
tools to both request an image be built from a candidate build and to
also wait for the image to be available in order to execute test
scenarios with it. This PR adds a few new `pipeline` sub-commands that
can will be used for this purpose.
`pipeline github find workflow-artifact` can be used to find the path of
an artifact that matches the given filter criteria. You'll need to
provide a pull request number, workflow name, and either an exact
artifact name or a pattern. When providing a pattern only the first
match will be returned so make sure your regular expression is robust.
`pipeline hcp get image` will return the image information for an HCP
image. You will need to supply auth via the `HCP_USERNAME` and
`HCP_PASSWORD` environment variables in order to query the image
service. It also takes an enviroment flag so you can query the image
service in different environments.
`pipeline hcp wait image` is like `pipeline hcp get image` except that
it will continue to retry for a given timeout and with a given delay
between requests. In this way it can be used to wait for an image to be
available.
As part of this we also update our Go modules to the latest versions
that are compatible.
* [VAULT-39158]: actions(build-hcp-image): add workflow for building HCP images
* copywrite: add missing headers
* remove unused output
* address feedback
* allow prerelease artifacts
---------
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
* always fetch HEAD repo with it's clone URL
* use the number instead of ID when generating the summary table
Signed-off-by: Ryan Cragun <me@ryan.ec>
Co-authored-by: Ryan Cragun <me@ryan.ec>
Add a new `pipeline github sync branches` command that can synchronize
two branches. We'll use this to synchronize the
`hashicorp/vault-enterprise/ce/*` branches with `hashicorp/vault/*`.
As the community repository is effectively a mirror of what is hosted in
Enterprise, a scheduled sync cadence is probably fine. Eventually we'll
hook the workflow and sync into the release pipeline to ensure that
`hashicorp/vault` branches are up-to-date when cutting community
releases.
As part of this I also fixed a few static analysis issues that popped up
when running `golangci-lint` and fixed a few smaller bugs.
Signed-off-by: Ryan Cragun <me@ryan.ec>
After the merge workflow has been reversed and branches hosted in
`hashicorp/vault` are downstream from community branches hosted in
`hashicorp/vault-enterprise`, most contributions to the source code
will originate in `hashicorp/vault-enterprise` and be backported to
a community branch in hosted in `hashicorp/vault-enterprise`. These
community branches will be considered the primary source of truth and
we'll automatically push changes from them to mirrors hosted in
`hashicorp/vault`.
This workflow ought to yield a massive efficiency boost for HashiCorp
contributors with access to `hashicorp/vault-enterprise`. Before the
workflow would look something like:
- Develop a change in vault-enterprise
- Manually extract relevant changes from your vault-enterprise branch
into a new community vault branch.
- Add any stubs that might be required so as to support any enterprise
only changes.
- Get the community change reviewed. If changes are necessary it often
means changing and testing them on both the enteprise and community
branches.
- Merge the community change
- Wait for it to sync to enterprise
- *Hope you changes have not broken the build*. If they have, fix the
build.
- Update your enterprise branch
- Get the enterprise branch reviewed again
- Merge enterprise change
- Deal with complicated backports.
After the workflow will look like:
- Develop the change on enterprise
- Get the change reviewed
- Address feedback and test on the same branch
- Merge the change
- Automation will extract community changes and create a community
backport PR for you depending on changes files and branch
activeness.
- Automation will create any enterprise backports for you.
- Fix any backport as necessary
- Merge the changes
- The pipeline will automatically push the changes to the community
branch mirror hosted in hashicorp/vault.
No more
- Duplicative reviews
- Risky merges
- Waiting for changes to sync from community to enterprise
- Manual decompistion of changes from enterprise and community
- *Doing the same PR 3 times*
- Dealing with a different backport process depending on which branches
are active or not.
These changes do come at cost however. Since they always originate from
`vault-enterprise` only HashiCorp employees can take advatange of the
workflow. We need to be able to support community contributions that
originate from the mirrors but retain attribution.
That's what this PR is designed to do. The community will be able to
open a pull request as normal and have it reviewed as such, but rather
than merging it into the mirror we'll instead copy the PR and open it
against the corresponding enterprise base branch and have it merged it
from there. The change will automatically get backported to the
community branch if necessary, which eventually makes it back to the
mirror in hashicorp/vault.
To handle our squash merge workflow while retaining the correct
attribution, we'll automatically create merge commits in the copied PR
that include `Co-Authored-By:` trailers for all commit authors on the
original PR.
We also take care to ensure that the HashiCorp maitainers that approve
the PR and/or are assigned to it are also assigned to the copied PR.
This change is only the tooling to enable it. The workflow that drives
it will be implemented in VAULT-34827.
Signed-off-by: Ryan Cragun <me@ryan.ec>
