* hcl2template: err on malformed local/data dep
When introducing the DAG for locals and datasources, we forgot to handle
one limit case: if a dependency for a local or data is malformed, we
didn't check that a vertex was associated to it, leading to the final
DAG being malformed, and the DAG library would crash in this case.
This commit fixes this problem by checking that the dependency does
exist before attempting to add it to the graph as an edge for a vertex,
so that it is reported accurately, and do that Packer doesn't crash.
* error message change
---------
Co-authored-by: anshul sharma <anshul.sharma@hashicorp.com>
* modifying the for loop to preserver user inputted variable files preference
* adding test cases
* pr comments + refactoring the for loop for better readability.
* pr comments + refactoring the for loop for better readability.
* pr comments | handling default case
* adding additional test case
* fixing test cases
* fixing test cases. creating a subdirectory to add a test case for both hcl and json auto var files
* adding test case for json auto var file
This method will be used to validate and fetch the correct
HCPPackerRefistry block to be used in HCP build
hcl: check for multiple HCP Packer registry block
hcl: support for top level HCP Packer Registry
hcl: deprecation warning for build block based hcp config
`packer validate` would output the same error message four times per
unsupported root block type found in a template (e.g., 'src' instead of
'source'). This behavior was due to a function being called four times
for each file on each stage of the parsing.
Evaluating local variables used to be directly written to the
PackerConfig while each variable was created.
This was somewhat of an issue with testing, as we have a bunch of tests
that relied on `PackerConfig.Variables` being set only when we actually
write something.
This is not really a concern for normal use, just for testing, but to
limit the number of changes to the tests in hcl2template, I opted to
change how variables' values are retained, so that evaluating a single
variable returns a Variable in addition to hcl.Diagnostics, so we can
reify the approach and only create the map of variables if there's
something evaluated.
When preparing the datasources to add into the DAG for evaluating the
build prerequisites, we ended-up in a weird situation in which the
datasources for each vertex pointed to the same one.
This is because of the loop semantics of Go, where the same object is
reused over and over again during each loop, so in the end every
datasource vertex pointed to the same instance of a datasource block.
To avoid this, we instead grab them through their reference, making the
reference to the datasource purely local, and pointing to the actual
datasource block, not the one scoped to the function.
Walk uses a reverse topological order to walk on the graph, doing that
visit concurrently if possible.
This is nice as we can speed-up execution of datasources and locals,
however since the `Variables` map stored in the config, and the
production of the context for it, are not meant to be used concurrently,
this means that we end-up in cases where Packer crashes because of
concurrent accesses to that map.
So until we can change this behaviour, we will fallback to using the
sequential visit algorithm for those vertexes, therefore limiting the
risk of those conflicts.
Local variables had an attribute called Name with the name of the local
variable.
However, when producing an error while walking the DAG of
local/datasources, if an error is encountered during validation, the raw
structure of the vertex was printed out, making the error message
produced hard to understand.
Therefore in order to clean it up, we rename the `Name` attribute for
Local variables as `LocalName`, and introduce a `Name()` function for
that block so that the complete name of the variable is clearly
reported.
The implementation of the DAG as extracted from Terraform relied on a
Root vertex being injected into the graph as the last node to visit.
This is used as a sanity check for Terraform, but doesn't apply to our
use-case for now, as we are always executing everything and have no need
for this root node.
Instead, we change how Validate operates so it does not error in case
there is no valid root node for the graph, but enables us calling it to
check for self-referencing edges, and circular dependencies.
Following up on the DAG work, this commit adds a new option for
initialisation that disables DAG on request.
By default we are going to use the DAG approach, with an option to
fallback to using the older algorithm for evaluation in case users
end-up in an edge-case that prevents them from building a template.
As we have finished setting-up the codebase for it, this commit adds the
logic that uses the internal DAG package, and is able to orchestrate
evaluation of datasources and locals in a non-phased way.
Instead, this code acts by first detecting the dependencies for those
components, builds a graph from them, with edges representing the
dependency links between them, and finally walking on the graph
breadth-first to evaluate those components.
This can act as a drop-in replacement for the current phased logic, but
both should be supported until we are confident that the approach works,
and that there are little to no bugs left to squash.
Previously duplicate detection for local variables happened during
`Initialise`, through a call to `checkForDuplicateLocalDefinition`.
This works in a majority of cases, but for commands like `console`, this
was not detected as the return diagnostics for `Initialise` are ignored.
That check can be done as early as during parsing however, as the names
of blocks are not dynamic in the slightest (no interpolation possible),
so we move that detection logic into `Parse`, so that the behaviour is
coherent between all commands.
Since we have both version/version.go and version/VERSION to specify
version strings, both are a bit redundant.
As version/VERSION is supposed to be the source of truth now, we are
using it to derive the version informaiton we used to rely on in Packer
and its subcommands.
Note: doing this prevents us from changing the version/prerelease
through ldflags though as we derive Version/VersionPrerelease from the
rawVersion variable.
Local variables can't have a validation block in their definition, so
this step in not useful and should be removed.
Besides, since the validation was done on the local variables before
evaluation, it did nothing at all, as the PackerConfig.LocalVariables
collection gets populated during evaluation, so this is essentially a
no-op, and can be safely removed.
Since this feature is no longer something we plan to activate later, as
it contradicts with our efforts to remove bundled plugins, and
encouraging users to move to either manually installing plugins, or
managing them through `packer init', we clean-up the code for this
feature.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUSL-1.1
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Since bundled plugins will be removed in an upcoming version of Packer,
this commit adds a new warning message whenever a template uses one such
plugin.
This warning has been implemented on build, validate, console and the
inspect subcommands.
In addition to warning about the upcoming change and potential issue
this will cause, this warning message proposes solutions to the user so
they know what they'll have to do in order not to rely on those bundled
plugins later.
* Update validation options for undeclared variables
In an effort to help users move from JSON to HCL2 templates the support for
variable definitions files are being updated to ignore undeclared
variable warnings on build execution. For legacy JSON templates builds no
warnings are displayed when var-files contain undeclared variables.
Since preferred mode HCL2 templates is to be explicit with variable
declarations - they must be declared to be used - validation for
undeclared variables still warns when running `packer validate`. A new
flag has been added to the validate command that can be used to disable
undeclared variable warnings.
* Update validation test for unused variables
Example Run
```
~> go run . validate -no-warn-undeclared-var -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
The configuration is valid.
~> go run . validate -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl
files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
The configuration is valid.
~> go run . build -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 744 microseconds.
==> Wait completed after 798 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
* Rename Strict field to WarnOnUndeclaredVar
The field name Strict is a bit vague since it is only used for
checking against undeclared variables within a var-file definition.
To mitigate against potential overloading of this field it is
being renamed to be more explicit on its usage.
* command/build: Add warn-on-undeclared-var flag
Now that the default behaviour is to not display warnings for undeclared variables
an optional flag has been added to toggle the old behaviour.
```
~> go run . build -warn-on-undeclared-var -var-file command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 762 microseconds.
==> Wait completed after 799 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
As part of the work to displace everything related to HCP from scattered
places around the Packer code, we move it all to an hcp package.
This in turn reduces the amount of code that the commands have to
integrate, and leaves the HCP details to its own enclave.
Before this commit, some HCP-related logic was embedded in the parser,
making all of it common to all commands.
This complexifies the logic as some parts rely on the template being in
a HCP-valid environment, and the datasources need to be evaluated in
order for some operations to be executed.
To simplify this and avoid those pitfalls, we move this logic to its own
set of functions that will be performed after the parsing is done, on
any valid HCL2 or JSON template.
* prevent duplicate local block creation
* remove duplicate locals block bug
* local variables: first get block, then decode it + simplify retry loop
* Update types.packer_config.go
* revert go get of hcl lib
* Add working registry pkg
* Add custom error for handling the loading of PAR environment variables
* Working Publish to Build, with proper error handling for bucket names
* Update hcp-sdk-go to use branch instead of mod replace directive
* Update Packer build status configuration
* Add support for HCP_PACKER_BUILD_FINGERPRINT env
* Add support for publishing one or more PARtifacts from a single build
* add git shas to this branch
* Add ability to set provider name if available
* Add working RegistryBuilder type
* Add RegistryPostProcessor as wrapper post-processor
* When in PAR mode a empty RegistryPostProcessor is added to the end of
the post-processor list to publish all final image data.
* Add support for updating a build from PAR that is not in a DONE state
* Fix a small issue with creation the initial builds for an empty
iteration.
* Add PAR URL to post-processor display
* Implement hcp_packer_registry block (#11168)
* Update vendored Amazon plugin to v1.0.1-dev
* Fix panic when running a Packer registry build in a clean directory
* Remove the publishing of post-processor metadata from the registry
post-processor.
* Remove metadata add from registry_builder
* Update registry builder to skip a build that was found to be DONE
Co-authored-by: Megan Marsh <megan@hashicorp.com>
Co-authored-by: Sylvia Moss <moss@hashicorp.com>
* HCL2 variables: split validation from getting value, to only
This way we do this only once and log this only once. The errors were being ignored anyways.
* Update types.variables_test.go
This :
* allows to have a `build.dynamic` block
* add tests
* makes sure to show a correct message when a source was not found
* display only name of source (instead of a weird map printout)
* use a "Did you mean %q" feature where possible
Because dynamic blocks need all variables to be evaluated and available, I moved parsing of everything that is not a variable to "after" variables are extrapolated. Meaning that dynamic block get expanded in the `init` phase and then only we start interpreting HCL2 content.
After #10819fix#10657
This adds the new `required_plugins` block to be nested under the packer block.
Example:
```hcl
packer {
required_plugins {
aws = {
version = ">= 2.7.0"
source = "azr/aws"
}
azure = ">= 2.7.0"
}
}
```
For example on darwin_amd64 Packer will install those under :
* "${PACKER_HOME_DIR}/plugin/github.com/azr/amazon/packer-plugin-amazon_2.7.0_x5.0_darwin_amd64"
* "${PACKER_HOME_DIR}/plugin/github.com/hashicorp/azure/packer-plugin-azure_2.7.0_x5.0_darwin_amd64_x5"
+ docs
+ tests
* Allow locals to be delcared as individual blocks, and give them the Sensitive flag
* add docs for new local block
* linting
* add tests
* modified parsing to use schema, check for dupes properly
* update comment
fix wording a liiitle
* add tests for duplicate variables definition in two different files
* remove unnecessary slice initialisation
* fix crash by returning when decode error is hit
* parseLocalVariables: only treat a local vars if its not nil
also return in case of error
return locals in case of error too
* fix duplicate_locals test for windows
Co-authored-by: Adrien Delorme <azr@users.noreply.github.com>
* add the possibility to set the packer.required_version field; to make sure the template file works with that version of Packer
* add tests
* add documentation on packer.required_version
Example:
packer {
required_version = ">= 1.2.0, < 2.0.0"
}
The initialization of packer core in JSON also validates that `null` variables were set, except in the case of `packer validate --syntax-only` , but after the refactor to allow to have all commands work with HCL2 and JSON this subtlety was lost.
This refactors the initialisation of the core in order to allow to have `packer validate --syntax-only` not error in case a variable is not set. Since these calls are refactored this works for HCL2 too.
fix#9478
