In the case of now possible long-running terraform processes, we don't
want the cache to grow indefinitely.
Move the handling of the results into the terraform package, and inject
the shared table during the walk into the `BuildFunction` wrapper.
* Update proto schema and provider interfaces with support for moved across resource type RPCs
* address comments
* remove unused functions
* remove support for flatmap format
Store the hashed function calls within the plan file, so that Terraform
can validate the results during apply as well. The error won't be as
good, since we don't have access to the prior results to help debugging,
but it is very helpful to indicate when a function returned an incorrect
result rather than show that the resource attempted to apply a different
value from what was originally planned.
This makes the built-in "remote-exec" and "file" provisioners available
for use in the modules that implement stack components. These are both
relatively easy and low-risk to include because they are builtins and
don't require anything from outside of Terraform itself.
For now this intentionally excludes local-exec because we'll want to think
about what constraints we want to put on it, if any, to help ensure we can
meet the goal of stack configurations being portable between different
execution environments without significant modification, and our current
stacks execution environment doesn't guarantee the availability of any
external software _at all_.
The motivation for adding this now is just to give some better feedback
when someone uses a module using one of these provisioners, since otherwise
they'll see just a confusing generic error message from the modules
runtime about the provisioners not being available. I expect we'll revisit
this later and consider expanding it to at least include local-exec, and
_maybe_ external provisioner plugins, although that's more questionable
because the provisioner plugin mechanism is incredibly legacy and doesn't
have any way to work outside of local Terraform CLI usage today.
There are no tests here yet because these provisioners are not mockable
and would depend on having an SSH or WinRM server to connect to. Later we
should ponder how to make this more testable, which might mean making
another part of the system responsible for actually providing the
provisioner factories and thus our tests here can use fakes. The goal here
is just to get this done in a relatively lightweight way for better
feedback during preview though, so we're not yet ready to make significant
time investments here.
Components can emit sensitive values as outputs, which can be consumed
as inputs to other components. This commit ensures that such values are
correctly processed in order to pass their sensitivity to the modules
runtime.
The stacks runtime interacts directly with the modules runtime's
planning operations, rather than through the usual CLI paths. As a
result, root module input variable values can be marked as sensitive
upon entry.
This commit adds support for such marked values to the modules runtime
and the `plans.Plan` type. This is sufficient for stacks, which does not
use the planfile serialization, but we may in future choose to serialize
these decoded marks also.
When using stacks the provider configurations belong in the stack
configuration rather than inline in the individual modules.
Shared modules with inline provider configurations has been a deprecated
legacy practice for many years now, but traditional Terraform continued
to support it for backward-compatibility with older modules despite the
significant downsides of doing so. Stacks now finally removes that
capability, since it isn't straightforward to continue supporting it once
we've made the stacks runtime be responsible for instantiating and
configuring providers.
This means both that the validate walk can now describe static problems in
the component's module tree, and that we'll catch such problems earlier
in the planning phase and thus avoid reporting them repeatedly in cases
where a component block uses for_each to declare multiple instances.
This includes a fix to a bug in StackConfig.Components, which was
incorrectly using the input variable declarations as the source for its
result, instead of the component declarations.
Until we've updated the module config loader to be sourcebundle-aware and
thus return proper source addresses, this will at least make the paths
we show in diagnostics a little less verbose, and more consistent across
platforms.
To figure out the required order in which we'll apply the plans for all
components we need to know how data flows between them.
We'll use what's added here during the plan phase to record in the plan
which components depend on which other components, and then the apply
phase will use that coarse information to schedule the individual
component apply operations into a suitable order. None of that is
implemented here though, and will follow in future commits.
This commit includes new documentation about the apply scheduling approach
which includes some mechanisms that are not actually yet included in this
commit, but will follow shortly after. This is just a pragmatic compromise
to avoid spending time authoring short-lived partial documentation drafts
for the intermediate steps while still keeping these increments small
enough for practical review.
This allows us to talk about references between expressions in absolute
terms, taking into account the stack where a particular expression would
be resolved.
This isn't an appropriate representation for references like each.key,
since those require a more specific scope than just a stack, but that's
okay because those contextual reference types can't depend on anything
other than what they are embedded inside anyway, and so we never need to
consider them when we're doing global reference analysis.
We chose this name originally to communicate that implementations are only
responsible for checking that an apply was successful, but in future
commits this interface will also gain the additional responsibility of
helping to gather the information required to understand the dependency
relationships between components that will then drive the scheduling of
the apply actions during the apply phase.
Along with full expression/body evaluation, we'll also need to be able to
determine what an expression or body refers to without evaluating it, so
that we can use references to infer dependency relationships between
significant side-effects in the apply phase, such as the relative ordering
of applying the changes for different components.
A downside of using custom collection types is that go-cmp doesn't support
them automatically, and so we need to help it out a little by providing
additional options.
An additional awkwardness is that go-cmp does all of its work at runtime
using reflection, at which point the generic types have all been erased
and lowered to many separate concrete types. Therefore we need some helper
functions to produce a dynamically representation using interface types,
cautiously exposing some internal details of these types only for go-cmp's
use.
The dynamic adapter here unfortunately means that the compiler likely
won't be able to optimize away CmpOptions in non-test builds, but we'll
accept that for now since the cost of that inert data and code is
relatively small. If it turns out to be a problem then we'll need to find
a different strategy, but hopefully not.
- The cloud plugin wants a go-tfe client to get stuff from TFC.
- Terraform knows how to find all the info you need in order to configure a
go-tfe client, but it's sometimes scattered across configs and env vars and
credentials helpers, and trying to re-implement the resolution logic identically
in another codebase would be error-prone.
- Therefore, it'd be best if Terraform did that resolution and just passed the
plugin all the config info it needs.
There are two options for that handoff:
1. Adding it as an additional argument to the primary Execute RPC interface.
2. Passing it as gRPC metadata, which is a pile of arbitrary key => list-of-strings
pairs transmitted via HTTP/2 headers.
The crux of the decision here is whether a proliferation of plugin protocol
versions whenever we need to add more config data is better or worse than a
fuzzy and typeless key/value bag.
This PR is an implementation of option 2, for discussion and evaluation. (And
after all, this is considered an experimental feature anyway.) It assembles the
necessary information, translates it to the required metadata format, and
attaches it to the request at the proper moment to get everything over to the
plugin.
As previous commits foreshadowed, we're leaning on the existing logic in the
Cloud backend for pretty much all of this.
The plugin needs some config info in order to actually build a go-tfe client and
_do_ anything, and the Cloud backend is the one place that already knows how to
look up and reconcile all the possible sources of that info. So, we'll just find
a Cloud backend and pick its pockets.
This also replaces our reimplementations of hostname lookup and service
discovery, using work the Cloud backend already did.
Most of the time, the Cloud backend is the only thing that needs to read the
results of this discovery, and the only thing it cares about is the URL to the
TFE v2 API service. However:
- Other things might need to know about services that your TFC instance
provides, including services other than the TFE v2 API.
- There's a lot of complexity involved in determining which TFC instance you
should be talking to (as well as the credentials for contacting it, the
default org/workspace/project you're interested in, etc.), and it makes sense
to leave the Cloud backend as the owner of that complexity rather than
re-implement it in other potential consumers!
So, since other TFC consumers will likely be configuring a Cloud backend to get
their information anyway, might as well cache the known services so we don't
have to repeat the discovery request.
As part of this commit, note that I removed some code that swallowed a
particular error during discovery. This was vestigial behavior that came over
from the `remote` backend, but we were no longer performing the second part of
the process (i.e. producing a more descriptive error in place of the one we
swallowed). See https://github.com/hashicorp/terraform/pull/19659 for discussion
about the original intent.
