* refactor: Move chunk size limit constants to new `chunks` package
* refactor: Make `NewPluggable` return a `Pluggable` concrete type, instead of an instance of the `backend.Backend` interface.
* refactor: Configure state stores through the backend.Backend interface, instead of directly using methods related to RPCs. This requires changing where we call `SetStateStoreChunkSize`.
* docs: Add godocs comment to `StateStoreChunkSizeSetter` interface
To summarize, we don't really need the `SetStateStoreChunkSize` method, and instead methods like `(*GRPCProvider).ConfigureStateStore` in the `plugin6` package can directly inspect the negotiation process that passes through that code and pull out the chunk size. However that means that that code would also need to implement validation. And that's just `(*GRPCProvider).ConfigureStateStore`; what about all the test mocks that are used in different types of test? They would all need to be implemented similarly to GRPCProvider to be good, useful mocks, and then a lot of things that fulfil the provider.Interface interface are coupled to each other. Instead, it's easier to have validation in the `grpcClient` struct's methods in the `remote` package, as that code is common to all scenarios. That code needs a method to 'reach into' the provider.Interface value, so we use the `SetStateStoreChunkSize` method.
* chore: Make it clearer that the v6 GRPCProvider implements `SetStateStoreChunkSize`
* fix: Remove unnecessary assignment of chunk size
I'm surprised that removing this doesn't break E2E tests of PSS that use grpcwrap, but I think there's `plugin6` code that runs in that situation, so maybe chunking is handled elsewhere.
* chore: Add panic to try detect unexpected cases when setting chunk size.
* feat: Add `providers.StateStoreChunkSizeSetter` implementation to provider-simple-v6
* docs: Update code comments for NewPluggable describing its intended use
* feat: Implement `inmem` state store in provider-simple-v6
* feat: Add filesystem state store `fs` in provider-simple-v6, no locking implemented
* refactor: Move PSS chunking-related constants into the `pluggable` package, so they can be reused.
* feat: Implement PSS-related methods in grpcwrap package
* test: Add E2E test checking an init and apply (no plan) workflow is usable with both PSS implementations
* fix: Ensure state stores are configured with a suggested chunk size from Core
---------
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* Implement ReadStateBytes + WriteStateBytes
* [WIP] - Testing ReadStateBytes and WriteStateBytes (#37464)
* Fix nil pointer error
* Add WIP test for ReadStateBytes
* Move test mock to separate testing file
* Update mock to send unexpected EOF when there's a problem returning data and it's not a true EOF
* Add test case for when length != expected length
* Add test for when trying to read state from a store type that doesn't exist
* Change symbol names to lowercase
* Add ability to force a diagnostic to be returned from `mockReadStateBytesClient`'s `Recv` method
* Add test showing error diagnostics raised by the ReadStateBytes client are returned
* Add missing header
* Simplify mock by using an embedded type
* Rename `mockOpts` to `mockReadStateBytesOpts`
* Update existing tests to assert what arguments are passed to the RPC method call
* Add mock WriteStateBytesClient which uses `go.uber.org/mock/gomock` to enable assertions about calls to Send
* Add a test for WriteStateBytes that makes assertions about calls to the Send method
* Update test case to explicitly test writing data smaller than the chunk size
* Implement chunking in WriteStateBytes, add test case to assert expected chunking behaviour
* Add generated mock for Provider_WriteStateBytesClient in protocol v6
* Update tests to use new `MockProvider_WriteStateBytesClient`, remove handwritten mock
* Update code comments in test
* Add tests for diagnostics and errors returned during WriteStateBytes
* Add generated mock for Provider_ReadStateBytesClient in protocol v6, replace old mock
* Add test case for grpc errors in ReadStateBytes, fix how error is returned
* Typo in comment
* Add missing warning test, rename some test cases
* Update proto file definition of Read/WriteStateBytes RPCs (#37529)
* Update Read/WriteStateBytes RPCs to match https://github.com/hashicorp/terraform-plugin-go/pull/531
* Run `make protobuf`
* Run `make generate`
* Update use of `proto.ReadStateBytes_ResponseChunk` in tests
* Fix how diagnostics are handled alongside EOF error, update ReadStateBytes test
* More fixes - test setup was incorrect
I think? I assume that a response would be returned full of zero-values when EOF is encountered.
* WIP - avoid crash if chunk is nil
* Sarah's updates to radek/pss-read-write (#37642)
* Update code to not expect a chunk when EOF encountered
* Return early if any grpc errors are encountered during ReadStateBytes
* Close the stream with CloseSend once everything's read without error. Add test case about handling grpc errors from CloseSend.
* Fix test case about warnings: We would expect to receive a chunk with data alongside the warning and have a normal closing of the stream after EOF
* Add log line, remove unneeded type info
* Implement configurable state chunk size
* handle metadata in WriteStateBytes correctly
* validate chunk sizes received from provider
* ReadStateBytes: avoid early return on warnings
---------
Co-authored-by: Sarah French <15078782+SarahFrench@users.noreply.github.com>
* Add method to allow accessing factories from locks that are in memory
* Create new getStateStoreProviderFactory method for accessing a factory from config
* Update getStateStoreProviderFactory to use in memory locks instead of reading them from the deps lock file. Update calling code to accommodate this.
* Add tests for getStateStoreProviderFactory, improve errors returned from method
* Update test following schema change in simple providers
* Move test case into e2etest package, so we protect against environments where building binaries isn't possible
* Fix issues with running test in e2etest package
* Update code comments
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
---------
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* Add ValidateActionConfig to provider interface and protocol
* Update internal/provider-simple-v6/provider.go
Co-authored-by: Daniel Schmidt <danielmschmidt92@gmail.com>
---------
Co-authored-by: Daniel Schmidt <danielmschmidt92@gmail.com>
* Add GetStates RPC method to Plugin Protocol v6
* Add DeleteState RPC method to Plugin Protocol v6
* Add methods to provider-simple
* Fix error messages
* Apply suggestions from code review
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* Change how GetStates RPC returns the list of state names
* Change GetStates implementation to use new data type
* `make protobuf` to accommodate new code comments
---------
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* Implement ListResource RPC via callbacks
* Implement ListResource with iterators
* Change `ListResourceResponse` to slice
* Turn ListResourceResponse into a struct
* Add a limit for ListResource
* explicitly cancel stream when listing resources
---------
Co-authored-by: Samsondeen Dare <samsondeen.dare@hashicorp.com>
* Minor auto-formatting changes
* Add list RPC and schema protobuf definitions
* make protobuf
* make generate
* Add ValidateListResourceConfig implementation
The call site for language functions doesn't currently have a way to
handle complex diagnostics, so rather than appear to support them in the
protocol we remove the concepts of diagnostics for now. We do however
retain the argument index fields, which we can wrap in a
function.ArgError and get a little more precise hcl diagnostic from
expression.
* Update proto schema and provider interfaces with support for moved across resource type RPCs
* address comments
* remove unused functions
* remove support for flatmap format
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
This is part of a general effort to move all of Terraform's non-library
package surface under internal in order to reinforce that these are for
internal use within Terraform only.
If you were previously importing packages under this prefix into an
external codebase, you could pin to an earlier release tag as an interim
solution until you've make a plan to achieve the same functionality some
other way.
* providers.Interface: rename ValidateDataSourceConfig to
ValidateDataResourceConfig
This PR came about after renaming ValidateResourceTypeConfig to
ValidateResourceConfig: I now understand that we'd called it the former
instead of the latter to indicate that the function wasn't necessarily
operating on a resource that actually exists. A possibly-more-accurate
renaming of both functions might then be ValidateManagedResourceConfig
and ValidateDataResourceConfig.
The next commit will update the protocol (v6 only) as well; these are in
separate commits for reviewers and will get squashed together before
merging.
* extend renaming to protov6
* Add support for plugin protocol v6
This PR turns on support for plugin protocol v6. A provider can
advertise itself as supporting protocol version 6 and terraform will
use the correct client.
Todo:
The "unmanaged" providers functionality does not support protocol
version, so at the moment terraform will continue to assume that
"unmanaged" providers are on protocol v5. This will require some
upstream work on go-plugin (I believe).
I would like to convert the builtin providers to use protocol v6 in a
future PR; however it is not necessary until we remove protocol v6.
* add e2e test for using both plugin protocol versions
- copied grpcwrap and made a version that returns protocol v6 provider
- copied the test provider, provider-simple, and made a version that's
using protocol v6 with the above fun
- added an e2etest
