* lang: stabilise ephemeral values experiment
Stabilise the ephemeral values experiment and ephemeralasnull
experimental function.
* simplify handling of apply time input vars
As the final step in stabilising the ephemeral values experiment, we can
remove the separate code path for handling input variables supplied via
-var and -var-file during apply.
The intent here is conveyed in the tests: when applying a saved plan,
all ephemeral variables set in the plan (the "apply-time
variables"), must be set, and no others.
As per the TODO copied from the prototype, there is some more work to be
done here in making the handling of undeclared variables during apply as
sophisticated as that during plan, emitting helpful warnings (for
example) when input variables are supplied unnecessarily via environment
variables.
* experiments: officially conclude EV experiment
---------
Co-authored-by: Radek Simko <radek.simko@gmail.com>
When the ephemeral_values experiment is active, a module author can
designate individual input variables and output values as being
"ephemeral", which doesn't currently do anything but in future commits
will represent that the values can be used only in locations that don't
require Terraform to persist the value as part of state snapshots or
saved plan files.
Previously we introduced a language experiment that would permit variable
validation rules to refer to other objects declared in the same module
as the variable.
Now that experiment is concluded and its behavior is available for all
modules.
This final version deviates slightly from the experiment: we learned from
the experimental implementation that we accidentally made the "validate"
command able to validate constant-valued input variables in child modules
despite the usual rule that input variables are unknown during validation,
because the previous compromise bypassed the main expression evaluator
and built its own evaluation context directly.
Even though that behavior was not intended, it's a useful behavior that is
protected by our compatibility promises and so this commit includes a
slightly hacky emulation of that behavior, in eval_variable.go, that
fetches the variable value in the same way the old implementation would
have and then modifies the hcl evaluation context to include that value,
while preserving anything else that our standard evaluation context
builder put in there. That narrowly preserves the old behavior for
expressions that compare the variable value directly to a constant, while
treating all other references (which were previously totally invalid) in
the standard way. This quirk was already covered by the existing test
TestContext2Validate_variableCustomValidationsFail, which fails if the
special workaround is removed.
When a module opts in to the variable_validation_crossref experiment,
Terraform will treat variable validation rules in a similar way as any
other dynamic expression, allowing it to refer to other objects that are
declared in the same module as the variable.
Because the check for a validation rule referring only to itself is now
experiments-sensitive we need to do it only once the full module has been
assembled, rather than during assembly of individual files, and so the
"badref" test in package configs is now classified under "invalid-modules"
rather than "invalid-files". It's otherwise unchanged, and the existing
test that a validation rule must refer to at least _something_ is still
present because that remains true even when opted in to the experiment.
This is experimental for now because although it seems pretty plausible
that it should be okay, this is allowing full evaluation in a new place
where it wasn't allowed before and so we'd like to ask those interested
in this feature to try it out with some of their realistic use-cases and
make sure this solution is a good fit before we lock it in as a
compatibility constraint.
Due to some specific circumstances around how this experiment reached
stabilization we made a planned-temporary exception to make it valid to
specify even in non-alpha builds of Terraform, just to be able to return
a more precise error message.
The comment added at the time suggested that we remove this "a release or
two after v1.3", and v1.8 is the actively-developed release at the time
I'm writing this so we've kept this around far longer than we originally
felt we needed to. It's time for it to go.
This particular experiment will now get the same treatment as all others
when used in a non-alpha release: Terraform will return the general error
about experiments only being available in alpha releases.
Since we've both concluded the module_variables_optional_attrs experiment
and made experiments available only in alpha releases in the same minor
release, we accidentally made the more general message about experiments
not being available mask the specific message about the experiment being
concluded.
In order to give better feedback to those who were participating in the
experiment in earlier Terraform releases, we'll retain a minimal exception
to our checks to allow the "experiment has concluded" error message to
shine through if and only if that is the only selected experiment.
We originally introduced the idea of language experiments as a way to get
early feedback on not-yet-proven feature ideas, ideally as part of the
initial exploration of the solution space rather than only after a
solution has become relatively clear.
Unfortunately, our tradeoff of making them available in normal releases
behind an explicit opt-in in order to make it easier to participate in the
feedback process had the unintended side-effect of making it feel okay
to use experiments in production and endure the warnings they generate.
This in turn has made us reluctant to make use of the experiments feature
lest experiments become de-facto production features which we then feel
compelled to preserve even though we aren't yet ready to graduate them
to stable features.
In an attempt to tweak that compromise, here we make the availability of
experiments _at all_ a build-time flag which will not be set by default,
and therefore experiments will not be available in most release builds.
The intent (not yet implemented in this PR) is for our release process to
set this flag only when it knows it's building an alpha release or a
development snapshot not destined for release at all, which will therefore
allow us to still use the alpha releases as a vehicle for giving feedback
participants access to a feature (without needing to install a Go
toolchain) but will not encourage pretending that these features are
production-ready before they graduate from experimental.
Only language experiments have an explicit framework for dealing with them
which outlives any particular experiment, so most of the changes here are
to that generalized mechanism. However, the intent is that non-language
experiments, such as experimental CLI commands, would also in future
check Meta.AllowExperimentalFeatures and gate the use of those experiments
too, so that we can be consistent that experimental features will never
be available unless you explicitly choose to use an alpha release or
a custom build from source code.
Since there are already some experiments active at the time of this commit
which were not previously subject to this restriction, we'll pragmatically
leave those as exceptions that will remain generally available for now,
and so this new approach will apply only to new experiments started in the
future. Once those experiments have all concluded, we will be left with
no more exceptions unless we explicitly choose to make an exception for
some reason we've not imagined yet.
It's important that we be able to write tests that rely on experiments
either being available or not being available, so here we're using our
typical approach of making "package main" deal with the global setting
that applies to Terraform CLI executables while making the layers below
all support fine-grain selection of this behavior so that tests with
different needs can run concurrently without trampling on one another.
As a compromise, the integration tests in the terraform package will
run with experiments enabled _by default_ since we commonly need to
exercise experiments in those tests, but they can selectively opt-out
if they need to by overriding the loader setting back to false again.
This allows precondition and postcondition checks to be declared for
resources and output values as long as the preconditions_postconditions
experiment is enabled.
Terraform Core doesn't currently know anything about these features, so
as of this commit declaring them does nothing at all.
Based on feedback during earlier alpha releases, we've decided to move
forward with the current design for the first phase of config-driven
refactoring.
Therefore here we've marked the experiment as concluded with no changes
to the most recent incarnation of the functionality. The other changes
here are all just updating test fixtures to no longer declare that they
are using experimental features.
In order to handle optional attributes, the Variable type needs to keep
track of the type constraint for decoding and conversion, as well as the
concrete type for creating values and type comparison.
Since the Type field is referenced throughout the codebase, and for
future refactoring if the handling of optional attributes changes
significantly, the constraint is now loaded into an entirely new field
called ConstraintType. This prevents types containing
ObjectWithOptionalAttrs from escaping the decode/conversion codepaths
into the rest of the codebase.
This package level variable can be overridden at link time to allow
temporarily disabling the UI warning when experimental features are
enabled. This makes it easier to understand how UI will render when the
feature is no longer experimental.
This change is only for those developing Terraform.
An earlier commit added logic to decode "moved" blocks and do static
validation of them. Here we now include that result also in modules
produced from those files, which we can then use in Terraform Core to
actually implement the moves.
This also places the feature behind an active experiment keyword called
config_driven_move. For now activating this doesn't actually achieve
anything except let you include moved blocks that Terraform will summarily
ignore, but we'll expand the scope of this in later commits to eventually
reach the point where it's really usable.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
