Loading schemas from some providers can be particularly expensive, since
providers for large remote platforms tend to have very large schemas.
Since provider schemas are needed for many operations in Terraform,
callers sometimes end up loading schemas themselves anyway. Earlier work
tried to mitigate this by introducing a global schema cache for all
plugin-based providers, but that's problematic because it forces only a
single implementation of each distinct provider source address across the
entire lifetime of a process importing package providers.
This does not remove that global cache yet, but does add a new capability
that will hopefully eventually supplant it: callers of
terraform.NewContext can provide a set of preloaded provider schemas which
they must ensure would match what Terraform Core would find if it loaded
the schemas from an instance of the same provider instantiated through
the corresponding factory function given alongside.
A caller that wishes to avoid the potential cost of multiple schema
lookups can now therefore go look up the schemas itself before calling
terraform.NewContext, and provide frozen schemas that we'll use instead
of fetching from the associated plugins.
As of this commit no callers are actually using this mechanism. The first
caller will be the "stackeval" package, which already loads provider
schemas in order to evaluate provider configuration blocks anyway and so
should always be able to provide a full complement of preloaded schemas
to avoid Terraform Core needing to do any further lookups itself.
A typo put the check for `GetProviderSchemaOptional` into the wrong `if`
expression, and was missed partly because of the named return which we
also remove here.
Reference: https://github.com/hashicorp/terraform/pull/33486
This is a followup to the new provider server capability to make the `GetProviderSchema` RPC optional. While this server capability would perform its intended function when directly talking to a single provider server SDK implementation, provider servers using terraform-plugin-mux need a methodology for the mux server to determine the available resource types of each underlying provider server to properly route resource-specific RPCs. Since the only methodology available to the mux server would be calling the `GetProviderSchema` RPC to each of underlying provider servers, any memory optimization of core caching would be lost.
The choice of adding a new RPC instead of adjusting the existing `GetProviderSchema` RPC with additional request information, such as "only list the type names and not the schema information in the response", is two-fold:
- Prevents the introduction of conditional logic for the existing RPC.
- Clearly delineates the purpose of the RPC and can be documented easier.
The choice of adding this to the existing provider service is two-fold:
- Implementing a separate protocol and/or service only on the provider side of the protocol would be a novel design change. This small of a change does not warrant the potential research and testing effort that would be associated with that implementation.
- While the core implementation will not use the new RPC immediately, there is no reason why it should be restricted from doing so in the future if a valid use case surfaces. Other ecosystem tools, beyond terraform-plugin-mux, can also potentially benefit from the lightweight RPC now.
This is changing the 5.4 and 6.4 protocol versions following the guidance of this comment in the definition files, since it directly relates to the prior intention of the new minor versions:
```protobuf
// Note that only the proto files included in a release tag of Terraform are
// official protocol releases. Proto files taken from other commits may include
// incomplete changes or features that did not make it into a final release.
// In all reasonable cases, plugin developers should take the proto file from
// the tag of the most recent release of Terraform, and not from the main
// branch or any other development branch.
```
As with any Protocol Buffers definition update, protocol compatibility is guaranteed within a major version, however generated protocol source code compatibility is not guaranteed. In this case, implementing the new RPC method in protocol wrapper types and the moving of the `ServerCapabilities` message to the top namespace are considered acceptable changes.
Allow core to always use the global schema cache, so that providers
without GetProviderSchemaOptional are not spun up repeatedly. Rather
than conditionally setting the cache, we just conditionally use the
cache in the client to work around providers without
GetProviderSchemaOptional.
Add a single global schema cache for providers. This allows multiple
provider instances to share a single copy of the schema, and prevents
loading the schema multiple times for a given provider type during a
single command.
This does not currently work with some provider releases, which are
using GetProviderSchema to trigger certain initializations. A new server
capability will be introduced to trigger reloading their schemas, but
not store duplicate results.
This is most easily handled in the plugin code, without involving
Terraform core.
The biggest change here other than checking the PlanDestroy capability,
is the removal of the schema helper methods in the plugins. With the
addition of the capabilities field, combined with the necessity of
checking diagnostics from the schema, the helpers have outlived their
usefulness. Perhaps there's a better pattern for these repetitive calls,
but for now there isn't too extra verbosity involved.
We can no longer be assured that the particular instance of a provider
we are using has had GetProviderSchema called. Always check the
diagnostics even if we're fetching a cached response.
Running the tool this way ensures that we'll always run the version
selected by our go.mod file, rather than whatever happened to be available
in $GOPATH/bin on the system where we're running this.
This change caused some contexts to now be using a newer version of
staticcheck with additional checks, and so this commit also includes some
changes to quiet the new warnings without any change in overall behavior.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
