* jsonstate: Marshal identity values
* jsonstate: Test identity marshalling
* Add identity to prepareStateV4
* Check identity schema version when marshaling state
* Marshal identity for deposed resources
* Marshal identity version if `0`
* Check for missing resource identity schema
* Remove handling of ephemeral root outputs
This is effectively reverting ~99% of https://github.com/hashicorp/terraform/pull/35676
The only changes not being reverted are some formatting and deprecation fixes which remain relevant.
The code being removed is basically dead code now in the context of root ephemeral outputs being rejected per https://github.com/hashicorp/terraform/pull/35791
* Remove unrelated changes
Ephemeral root output values must be kept in the in-memory state representation, but not written to the state file. To achieve this, we store ephemeral root outputs separately from non-ephemeral root outputs, so Terraform can access them during a single plan or apply phase.
Ephemeral root outputs always have a value of null in the state file. This means that the "terraform output" command, that reads the state file, reports null values for these outputs. Consumers of 'terraform output -json' should use the presence of '"ephemeral": true' in such output to interpret the value correctly.
In the very first implementation of "sensitive values" we were
unfortunately not disciplined about separating the idea of "marked value"
from the idea of "sensitive value" (where the latter is a subset of the
former). The first implementation just assumed that any marking whatsoever
meant "sensitive".
We later improved that by adding the marks package and the marks.Sensitive
value to standardize on the representation of "sensitive value" as being
a value marked with _that specific mark_.
However, we did not perform a thorough review of all of the mark-handling
codepaths to make sure they all agreed on that definition. In particular,
the state and plan models were both designed as if they supported arbitrary
marks but then in practice marks other than marks.Sensitive would be
handled in various inconsistent ways: dropped entirely, or interpreted as
if marks.Sensitive, and possibly do so inconsistently when a value is
used only in memory vs. round-tripped through a wire/file format.
The goal of this commit is to resolve those oddities so that there are now
two possible situations:
- General mark handling: some codepaths genuinely handle marks
generically, by transporting them from input value to output value in
a way consistent with how cty itself deals with marks. This is the
ideal case because it means we can add new marks in future and assume
these codepaths will handle them correctly without any further
modifications.
- Sensitive-only mark preservation: the codepaths that interact with our
wire protocols and file formats typically have only specialized support
for sensitive values in particular, and lack support for any other
marks. Those codepaths are now subject to a new rule where they must
return an error if asked to deal with any other mark, so that if we
introduce new marks in future we'll be forced either to define how we'll
avoid those markings reaching the file/wire formats or extend the
file/wire formats to support the new marks.
Some new helper functions in package marks are intended to standardize how
we deal with the "sensitive values only" situations, in the hope that
this will make it easier to keep things consistent as the codebase evolves
in future.
In practice the modules runtime only ever uses marks.Sensitive as a mark
today, so all of these checks are effectively covering "should never
happen" cases. The only other mark Terraform uses is an implementation
detail of "terraform console" and does not interact with any of the
codepaths that only support sensitive values in particular.
Our JSON state format has support for sensitive marks in particular but
lacks generalized support and so cannot deal with any other marks.
Previously it just assumed that any mark must be the sensitive mark, but
now we'll reject other marks to ensure that if any new marks are added we
must consider whether and how the JSON state format should handle them.
For a very long time we've had an annoying discrepancy between the
in-memory state model and our state snapshot format where the in-memory
format stores output values for all modules whereas the snapshot format
only tracks the root module output values because those are all we
actually need to preserve between runs.
That design wart was a result of us using the state both as an internal
and an external artifact, due to having nowhere else to store the
transient values of non-root module output values while Terraform Core
does its work.
We now have namedvals.State to internally track all of the throwaway
results from named values that don't need to persist between runs, so now
we'll use that for our internal work instead and reserve the states.State
model only for the data that we will preserve between runs in state
snapshots.
The namedvals internal model isn't really designed to support enumerating
all of the output values for a particular module call, but our expression
evaluator currently depends on being able to do that and so we have a
temporary inefficient implementation of that which just scans the entire
table of values as a stopgap just to avoid this commit growing even larger
than it already is. In a future commit we'll rework the evaluator to
support the PartialEval mode and at the same time move the responsiblity
for enumerating all of the output values into the evaluator itself, since
it should be able to determine what it's expecting by analyzing the
configuration rather than just by trusting that earlier evaluation has
completed correctly.
Because our legacy state string serialization previously included output
values for all modules, some of our context tests were accidentally
depending on the implementation detail of how those got stored internally.
Those tests are updated here to test only the data that is a real part
of Terraform Core's result, by ensuring that the relevant data appears
somewhere either in a root output value or in a resource attribute.
Previously the supported JSON plan and state formats included only
serialized output values, which was a lossy serialization of the
Terraform type system. This commit adds a type field in the usual cty
JSON format, which allows reconstitution of the original value.
For example, previously a list(string) and a set(string) containing the
same values were indistinguishable. This change serializes these as
follows:
{
"value": ["a","b","c"],
"type": ["list","string"]
}
and:
{
"value": ["a","b","c"],
"type": ["set","string"]
}
* command/jsonstate: remove redundant remarking of resource instance
ResourceInstanceObjectSrc.Decode already handles marking values with any marks stored in ri.Current.AttrSensitivePaths, so re-applying those marks is not necessary.
We've gotten reports of panics coming from this line of code, though I have yet to reproduce the panic in a test.
* Implement test to reproduce panic on #29042
Co-authored-by: David Alger <davidmalger@gmail.com>
* jsonplan and jsonstate: include sensitive_values in state representations
A sensitive_values field has been added to the resource in state and planned values which is a map of all sensitive attributes with the values set to true.
It wasn't entirely clear to me if the values in state would suffice, or if we also need to consult the schema - I believe that this is sufficient for state files written since v0.15, and if that's incorrect or insufficient, I'll add in the provider schema check as well.
I also updated the documentation, and, since we've considered this before, bumped the FormatVersions for both jsonstate and jsonplan.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
