The --force option for packer init and packer plugins install enforces
installation of a plugin, even if it is already locally installed.
This will become useful if for some reason a pre-existing plugin
binary/version is already installed, and we want to overwrite it.
This new flag allows the `packer plugins install' command to install a
plugin from a local binary rather than from Github.
This command will only call `describe' on the plugin, and won't do any
further checks for functionality. The SHA256SUM will be directly
computed from the binary, so as with anything manual and potentially
sourced by the community, extra care should be applied when invoking
this.
If a user attempts to remove a plugin through the `packer plugins
remove' subcommand, and the specified plugin is not installed, the
command succeeds, but does nothing, and exits silently.
This is not clear what is happening, and arguably, calling a command
that does nothing, not even explain what went wrong, is not intuitive.
Because of that, this commit changes how the command behaves in this
case, stating what went wrong, and exiting with a non-zero status.
When a user invokes `packer plugins remove', the plugin binary gets
removed, but not the corresponding SHA256SUM file.
This patch changes this so that when a binary is removed, so is its
SHA256SUM file.
This change updates hcl2_upgrade to not flag known plugin components, those used for generating the required plugins
block, when upgrading a legacy JSON template to HCL2. Any unknown plugins will be installed after running packer init
on the generated template so we don't error. We may want to suggest running packer init to install any missing plugins.
* Move knownPluginPrefixes into the hcl2_upgrade command
With the removal of vendored_plugins we are no longer interested in tacking bundled plugins usage.
For plugins such as file, null, or anything bultin into Packer we don't track because there is no
way to install them outside of Packer, for now.
The enumeration for FlagSetFlags, which presumably was added when the
Meta structure was introduced, aims to pre-populate the flagset for a
subcommand with a series of arguments.
However, despite it being documented, it is actually not used, and
therefore can safely be removed from the codebase.
When users call `packer init' on a template that does not specify a
`required_plugin' block, the command succeeds but does nothing, which is
not helpful for users that may expect their plugins to install.
To remedy that problem, we now output a message like what `packer
plugins required' does on templates without such a block, so that users
have an idea of what to change in order to get the command to work.
Since this feature is no longer something we plan to activate later, as
it contradicts with our efforts to remove bundled plugins, and
encouraging users to move to either manually installing plugins, or
managing them through `packer init', we clean-up the code for this
feature.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at https://hashi.co/license-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUSL-1.1
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
`packer init' is meant to be invoked on a template file or a directory.
This was described in the command-line help message as
`[template.pkr.hcl|folder]'.
Because this is encapsulated between square brackets, this could be
taken as an option instead of a positional, mandatory argument.
This commit changes the wording to adopt a similar convention as `packer
build', so the two are consistent in how they are invoked, and to avoid
mistaking this argument as optional.
Since bundled plugins will be removed in an upcoming version of Packer,
this commit adds a new warning message whenever a template uses one such
plugin.
This warning has been implemented on build, validate, console and the
inspect subcommands.
In addition to warning about the upcoming change and potential issue
this will cause, this warning message proposes solutions to the user so
they know what they'll have to do in order not to rely on those bundled
plugins later.
* Update external plugin documentation source
The Parallels plugin for Packer is now maintained by the Parallels team, under
their respective GitHub org. This changes updates the source address for
the external plugin that should be used for pulling new plugin
documentation.
* Removed packer-plugin-parallels for list of vendored plugins
This step removes community plugins from the Packer binary release. These plugins are being maintained and released independently of Packer and have become out of date. To ensure users are using the latest version of the plugins they are being removed in favor of the external plugin installation processes.
This change removes a set of plugins that have not been updated in a while. These
plugins have been archived for some time now. Users wishing to continue using these plugins
should use the `packer plugins install` or `packer init` commands to install the external plugin.
* Remove profitbricks plugin
* Remove oneandone plugin
The following plugins have been unmaintained for some time now, and their
upstream cloud provider has consolidate the services. These plugins
will continue to be available to Packer via direct installation using
packer init or the packer plugins install command. But they will no longer
be bundled with Packer.
The Digital Ocean plugin is now maintained and released by the Digital Ocean team.
To ensure users are downloading the latest available version the plugin will nolonger
be bundled with Packer.
The template type for a builder, provisioner, post-processor or
datasource was not tracked with telemetry data.
This commit adds the infrastructure to track this, bumping the schema
for non-crashes to `beta/packer/6' at the same time.
It is not clear that the console command runs in legacy JSON mode by
default and that users can specify the -config-type flag to toggle HCL2
mode. This change adds the config-type flag to the command's help text
to alert users of the gotcha.
In the next minor Packer release the console command will be updated to
default to HCL2 mode.
Relates to: #12359
Relates to: #10603
* Update external plugin documentation source
The UCloud plugin for Packer is now maintained by the UCloud team, under
their respective GitHub org. This changes updates the source address for
the external plugin that should be used for pulling new plugin
documentation.
* Remove packer-plugin-ucloud from vendored plugins.
* Update external plugin documentation source
The Linode plugin for Packer is now maintained by the Linode team, under
their respective GitHub org. This changes updates the source address for
the external plugin that should be used for pulling new plugin
documentation.
* Removed packer-plugin-linode for list of vendored plugins
The Linode plugin for Packer is now maintained by the Linode team, under
their respective GitHub org. This changes removes Linode from the vendored plugins
lists.
This change fixes a crash that occurs when trying to add undeclared variables into an empty
variables map. The variables block when defined will always be non-nil but in the case
where variables are declared and initialized from a legacy JSON variables file the map
must be initialized first.
Fingerprints are how we link a packer build to an iteration on HCP.
These are computed automatically from the Git SHA in the current state,
and are unique to the bucket/iteration.
The main problem with this approach is that while sound in theory, it
quickly falls apart when users want to run the same build configuration
twice, but expect a new image to be created.
With the current model, this fails, as the iteration with the current
SHA already exists.
While this is solvable through environment variables, or by committing a
change to the repository, we think this is not clear enough, and causes
an extra step to what should otherwise be a simple process.
Therefore, to lower the barrier of entry into HCP, we change this
behaviour with this commit.
Now, fingerprints are randomly generated ULIDs instead of a git SHA, and
a new one is always generated, unless one is already specified in the
environment.
This makes continuation of an existing iteration a conscious choice
rather than something automatic, and virtually eliminates conflicts such
as the ones described above.
* hcl2template: use && for chaining conditions
For deciding if we need to run an error provisioner block, we checked
using a nested if that we didn't need to avoid running it.
This is unnecessary, and adds a superfluous level of inbrication to the
code, so we chain both conditions with &&.
* hcl2: don't warn on excluded post-processor
When a post-processor is excluded through the -except command-line
argument for packer build, we used to print a warning on HCL2 templates
if this except statement did not match a build.
However, since we can use the except option to avoid running
post-processors, we should account for this as well when printing out a
warning that there were no matches.
This commit adds this counter to the except checks for the
post-processors as well.
This commit irons out one of the pain points of the HCP rework by
introducing a HCPPublisher interface, implemented both by the JSON Core,
and the HCL2 PackerConfig, which keeps a map of the build names used by
Packer to the build names pushed on HCP.
This in turn lets us go back to the old implementation of the GetBuilds
function, which returns a list of (filtered) builds, and eventually an
error if something went wrong while processing.
While looking into #11932 I found that Packer will throw an error when
an unknown source is referenced from within the sources attribute for
a build block. The hcl.Diagnostics error includes a Subject, which
should highlight where in the HCL2 configuration file the unknown source resides.
But when creating the BuildBlock no HCL2Ref data is copied over, thus the
Subject is displayed with with zero value for an hcl.Range type.
This change updates the build creation logic to copy the HCL2Ref data
from the build block at decode time so that it can be properly
referenced downstream.
Closes#11932
Failure due to change in output
```
--- FAIL: TestValidateCommand_ShowLineNumForMissing (0.00s)
--- FAIL: TestValidateCommand_ShowLineNumForMissing/test-fixtures/validate-invalid/missing_build_block.pkr.hcl (0.00s)
validate_test.go:377: Unexpected output: (
"""
Error: Unknown source file.cho
- on line 0:
+ on test-fixtures/validate-invalid/missing_build_block.pkr.hcl line 6:
(source code not available)
... // 4 identical lines
"""
)
validate_test.go:379:
FAIL
FAIL github.com/hashicorp/packer/command 1.002s
```
```
Usage: packer validate [options] TEMPLATE
Checks the template is valid by parsing the template and also
checking the configuration with the various builders, provisioners, etc.
If it is not valid, the errors will be shown and the command will exit
with a non-zero exit status. If it is valid, it will exit with a zero
exit status.
Options:
-syntax-only Only check syntax. Do not verify config of the template.
-except=foo,bar,baz Validate all builds other than these.
-only=foo,bar,baz Validate only these builds.
-machine-readable Produce machine-readable output.
-var 'key=value' Variable for templates, can be used multiple times.
-var-file=path JSON or HCL2 file containing user variables, can be used multiple times.
-no-warn-undeclared-var Disable warnings for user variable files containing undeclared variables.
-evaluate-datasources Evaluate data sources during validation (HCL2 only, may incur costs); Defaults to false.
```
Currently the HCL2 upgrade command generates a valid variable block for all variables
within the JSON variables property. However JSON templates in Packer support variable
interpolation for input variables defined within a variable definition file without
it being declared within the variables property. When upgrading a JSON template to HCL2
the user variable reference gets converted to var.<var_name>. The upgraded template ultimately
fails on validation or build execution with undefined variable errors, with this change the upgrade
command will now create a variable block for all input variables referenced within the build template
to ensure all required variables been defined.
When packer validate is invoked, it does not try to evaluate the
datasources before attempting to decide if the template is valid.
In many cases, this works, but sometimes it will fail as the value is
unknown by the validation code.
Since the validation code for all the elements of a Packer template is
left to be implemented by plugins, we cannot rely on checking for
unknown values everywhere, especially since the unknown references are
replaced automatically by a value of the right type for the
configuration expected.
So, in order for such configurations to be validable, we add an extra
option to packer validate, that will let users evaluate the datasources
from a template.
* Update validation options for undeclared variables
In an effort to help users move from JSON to HCL2 templates the support for
variable definitions files are being updated to ignore undeclared
variable warnings on build execution. For legacy JSON templates builds no
warnings are displayed when var-files contain undeclared variables.
Since preferred mode HCL2 templates is to be explicit with variable
declarations - they must be declared to be used - validation for
undeclared variables still warns when running `packer validate`. A new
flag has been added to the validate command that can be used to disable
undeclared variable warnings.
* Update validation test for unused variables
Example Run
```
~> go run . validate -no-warn-undeclared-var -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
The configuration is valid.
~> go run . validate -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl
files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
The configuration is valid.
~> go run . build -var-file
command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl
command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 744 microseconds.
==> Wait completed after 798 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
* Rename Strict field to WarnOnUndeclaredVar
The field name Strict is a bit vague since it is only used for
checking against undeclared variables within a var-file definition.
To mitigate against potential overloading of this field it is
being renamed to be more explicit on its usage.
* command/build: Add warn-on-undeclared-var flag
Now that the default behaviour is to not display warnings for undeclared variables
an optional flag has been added to toggle the old behaviour.
```
~> go run . build -warn-on-undeclared-var -var-file command/test-fixtures/validate/var-file-tests/undeclared.pkrvars.hcl command/test-fixtures/validate/var-file-tests/basic.pkr.hcl
Warning: Undefined variable
The variable "unused" was set but was not declared as an input variable.
To declare variable "unused" place this block in one of your .pkr.hcl files,
such as variables.pkr.hcl
variable "unused" {
type = string
default = null
}
file.chocolate: output will be in this color.
Build 'file.chocolate' finished after 762 microseconds.
==> Wait completed after 799 microseconds
==> Builds finished. The artifacts of successful builds are:
--> file.chocolate: Stored file: chocolate.txt
```
This commit reorganises the code for both the registry/API and the
Orchestrator/Registry.
The main difference with the previous version is how stuff is exposed.
Now we only expose a Registry interface to the outside (previously named
Orchestrator), which has several implementations: null is the default,
and is returned if HCP is not enabled.
The other implementations being HCL/JSON, both private to the hcp
sub-package.
The api (previously `registry') is the set of functionality that
abstracts and calls the HCP API.
It was meant to be merged with the `hcp' package, but because of a
dependency loop with the datasources, both are separated for now.
As part of the work to displace everything related to HCP from scattered
places around the Packer code, we move it all to an hcp package.
This in turn reduces the amount of code that the commands have to
integrate, and leaves the HCP details to its own enclave.
To be able to track HCP builds during a Packer build, we need to
propagate the name used for registering the build to HCP.
This works out-of-the-box for JSON templates, as the build's name is
always the builder's, so there's no confusion on that.
For HCP templates however, there's a possibility for a template to
define a name for the build, which is returned by a CoreBuild's Name()
function, in addition to the source's Type/Name.
The builds are registered to HCP with the source.String() function
however, which does not contain the build's name. This prevents any
build containing a name to work, as their name/source.String does not
match.
The name we registered the build with is stored in a CoreBuild as the
Type attribute, but cannot be safely accessed, as builds are types
within the command as packersdk.Build, which only exposes Name() for
this purpose.
In order to circumvent this problem, and as a way to present a prototype
of solution that will likely have to be discussed before we merge it,
this commit hotfixes the issue.
In azure templates, the shared_image_gallery was mistakenly considered
an attribute while this is supposed to be a block.
This is due to a heuristic we use for deciding whether a JSON object is
to be translated to an attribute or a block that fell short as the
shared_image_gallery does not contain complex types.
This cannot be fixed trivially for the general case, so we add this
entity to the list of workarounds until we can implement something more
robust.
Some literal strings without format-string directives nor arguments were
littered around the hcl2_upgrade command's code, which is pointed out by
the linters.
Since they serve no purpose, we remove them.
In previous versions of hcl2_upgrade, we referenced a list of
passthrough statements from the original JSON template to be replaced by
their equivalent Go-template in HCL2.
This works but requires us to explicitely accept every possible
templated variable in order to not ever encounter <no value> in our
generated HCL2 templates.
This is suboptimal, hence this commit changes approach by pre-visting
the AST from the original Go Template, escaping every sequence that is
not covered by the list of functions we can migrate. Pipes are also
excluded from this function.
* website: fix broken links on /docs/templates
* fix: redirected install-plugins link
* fix: debugging link
* fix: secrets manager link in docs
* fix: secrets manager link in source
* fix: amazon ami plugin link in docs
* fix: amazon ami plugin link in source
* fix: extending plugins link
* fix: plugins/builders/amazon links
* fix: various builders links
* fix: various amazon builder links
* fix: redirected terminology link
* fix: custom-provisioners link
* fix: docker-push redirected plugin link
* fix: googlecompute plugin links
* fix: hyperv iso plugin links
* website: update link to hcl upgrade guide
Co-authored-by: Wilken Rivera <wilken@hashicorp.com>
Co-authored-by: Wilken Rivera <wilken@hashicorp.com>
The HCP error messages were sometimes a bit terse in terms of the
information it bubbles up to the user.
This made them useful for people who already knew what to look for, but
newcomers would almost certainly be lost because of the lack of
information in those.
To improve on this situation, we reword most of those error messages in
this commit.
JSON templates used only to report the builder's type in HCP builds,
even if the name was specified.
This commit changes this behaviour to include both if they're available,
in a similar fashion as what is done on the HCL2 templates.
Fingerprint initialize was previously occurring during the creation of a
bucket. We want to be able to initialize a bucket and defer the setting
of a fingerprint to a later point.
* Update test to reflect new function signatures for Bucket and Iteration
Loadgin the slug from the environment within the Bucket implied that the
Bucket was successfully created, which may fail outside of a Git
directory when the iteration fingerprint is not set through environment
variables.
To make it possible to print as much information regarding the
environment to the users at once rather than stopping immediately when
this step fails, we move the logic to read the default value from the
environment to the HCP setup code.
When HCP is detected to be enabled, but some configuration is missing,
we returned immediately on the first error.
This commit changes this behaviour by reporting every error at once, so
users will know immediately if something is wrong when they invoke
Packer with HCP support, and one or more environment variables is not
defined as we'd expect them.
The HCP_PACKER_REGISTRY environment variable had its behaviour changed
recently, as prior versions of Packer expected the attribute to be
forcefully set to a value that is neither "off" nor "0" in order to
get HCP integration to work, or that a "hcp_packer_registry" block was
defined in an HCL template. Now, this environment variable defaults to
not explicitely enable, but instead to explicitely disable HCP
integration, and the feature switch fall upon HCP_PACKER_BUCKET_NAME.
As an extra feature, we keep the prior behaviour alive when it is
explicitely defined as a value to enable it. That way we can report
errors if the rest is not defined, rather than silently ignore it.
This function we add to env is the first stone to enable this behaviour.
When a template describes a build block without a source reference, the
build should be considered invalid as we won't have a CoreBuild produced
as a result of the need to have both.
In current versions of Packer, this will produce an error message
hinting that nothing will happen because of the lack of either build or
source block.
This commit takes the defined block, and points out to it as missing a
source block as being the reason why nothing is happening, making it
clearer what is required for an HCL2 build to be processed.
When a variable is set in a pkrvars file, but isn't defined, an error
message is output, but does not deliver an example of what is expected
by Packer in order to complete a build.
To remedy that, we improve the error message by giving an example of
variable block to include in the build template.
When a template with some builds to run ends its GetBuilds with an
error, and no builds produced, we can exit immediately without printing
more errors.
In order to test the output from a test command, the commandOutput
function exists in the command_test.go file.
Since its behaviour is linked to the test meta that we produce in the
test_utils.go file, we move it there, and rename it to
`GetStdoutAndErrFromTestMeta' to make it clearer what to expect from it.
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
The current behaviour for HCP integration is based on the presence of
the HCP_PACKER_REGISTRY environment variable, where if it is either not
"0" or "off", the HCP integration is enabled.
This commit changes the behaviour to only use this variable to
explicitely disable HCP, and the HCP_PACKER_BUCKET_NAME variable will
condition if the HCP integration is enabled for a build or not in JSON
templates.
For HCL templates, the integration can also be disabled if the
HCP_PACKER_REGISTRY variable is set to "0" or "off", and will be
considered enabled if the HCP_PACKER_BUCKET_NAME is set, or there is a
"hcp_packer_registry" block in a build.
The latest release of the Oracle plugin for Packer removed
Solaris from its list of supported platforms. Since Packer still
releases binaries for Solaris the Oracle plugin will no longer be
bundled and distributed within the Packer binary. Practitioners relying
on the plugin can continue using the plugin by installing it manually
using either the `packer init` or `packer plugins install` commands.
When a `packer build' is run on a config that does not define anything
to be built, the process would run normally, and only state that no
artifacts were produced after builds were run successfully.
This is misleading as we reach the end of the command without having a
proper warning that nothing was even attempted to be built.
This commit adds a warning when that happens, and leaves immediately
after.
Before this commit, some HCP-related logic was embedded in the parser,
making all of it common to all commands.
This complexifies the logic as some parts rely on the template being in
a HCP-valid environment, and the datasources need to be evaluated in
order for some operations to be executed.
To simplify this and avoid those pitfalls, we move this logic to its own
set of functions that will be performed after the parsing is done, on
any valid HCL2 or JSON template.
Since the GetConfig methods are common to all the commands of Packer
that rely on a config to work, we move them from build.go to meta.go in
order to make the intent clearer.
* Add tests for plugins install command
* Add tests for plugins remove command
* Add PACKER_GITHUB_API_TOKEN to all test env
* Update dir hashes for non-darwin os
* Update tests with feedback from review
* allowing to set shell env vars from a key/value map.
* tests
This new map will work alongside the `environment_vars` but it allows using a
datasource value seamlessly. At validation, and because `environment_vars` was
an array of strings containing `KEY=value`, using datsources would not work,
because these values are not known yet and would evaluate to `<unknown>`. With
this, the value and the key can be unknown and will validate correctly for
datasources.
* evaluateLocalVariables: modify code for readability and some (not benchmarked) perfs
* Make default input variable type the DynamicPseudoType
This should be the default, and avoids a panic. This type can represent situations where a type is not yet known. Its meaning is undefined in cty.
* do not take Empty types from default value
* Update types.variables.go
Co-authored-by: Wilken Rivera <wilken@hashicorp.com>
The auto installation of external plugins via packer init only work for
HCL templates. Since we have not officially removed all plugins (slated
for 1.8.0) I am reverting this test change. Once 1.7.9 is released we
can begin removing plugins and optimizing the redirect experience for
JSON-only users.
* Add variable interpolation support to hcp_packer_registry block
* Add tests for variable interpolation in hcp_packer_registry block
* Add variable interpolation support to the build block (#11425)
* Add simple test to build command
* add test for crash
* fix
* show more precise error message when parse of a provisioner fails
* Revert "fix"
This reverts commit fc88e89fbb.
* add better severity
