This walks back some of the decisions we made earlier around two-step provider downloading process. However, it should not impact the safety features and it should not have a tangible (functional) impact on the end-user aside from the log output outlined below.
The motivation is to accommodate some needs of the upcoming policy features.
Previously the two steps were:
1. Download providers based on configuration
2. Download providers based on state
The process remains two steps but the new steps are:
1. Download 0-1 provider if one is needed for PSS (PSS is in use)
2. Download all other providers (configuration + state)
End-user facing changes are mainly related to terminal output - what messages are included and their order:
- initializing_provider_plugin_message (re-introduction after removal in 1.14)
- initializing_state_store_provider_plugin_message (net new but PSS specific)
* test: Add pre-release to available providers. Existing tests don't use it because the configuration doesn't specify the prerelease as the version constraint.
This was the gap in my understanding - prerelease providers are handled differently in the installer logic. I believe it boils down to this: 29863fd2ba/versions/set_released.go (L5-L7)
* test: Add failing tests that show expected behaviour when a pre-release of a provider is used
* fix: Ensure that any relevant version constraints from the config are available to use when installing providers present in the state
* test: Add test defining pre-release download behaviour that was present before v1.15
* test: Update test to reflect that 2 step init is no longer experimental or specific to the PSS project
* fix: Limit backend validation to just the backend type.
Also, remove similar validation for state stores, which would cause a similar issue in future.
* test: Add a new test showing behavior when removed backend types are in a config.
* feat: Enable use of developer override providers when using PSS
* feat: Store how a provider was supplied to Terraform in the backend state file. This allows triggering a prompt for state migration if there's a change between using a managed provider versus an unmanaged/override provider.
* test: Update all PSS-related backend state files in test fixtures to include new, necessary provider_supply_mode data.
* fix: Avoid duplicate dev_override warnings
* refactor: Change how we determine how a provider is supplied to Terraform. By using supply mode data we prevent multiple parts of the code checking if a given provider is reattached/builtin/etc...
* test: Add separate tests showing a built-in or dev override provider being used for the main workflow using PSS
* test: Add E2E tests that characterise current behaviour when how a provider is supplied is changed partway through different workflows
* fix: Tolerate missing default workspace when using PSS and init handles minor changes in state store configuration.
* feat: `init`-related code will prompt a user to migrate state if Terraform detects a swap between managed and unmanaged providers being used for PSS.
Internally, what happens is the different version data (null versus a value being present) impacts the hash of the state store. This then triggers logic where TF checks if a migration is needed or if the new hash value should be silently updated in the backend state file. We always prompt for migration.
* refactor: Add `NotManagedByTerraform` method to `ProviderSupplyMode`
* Apply suggestions from code review
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
---------
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
Instead of outputting the raw values from the ModuleCall, we're now
using the evaluated values from the Config. This should keep the JSON
output unchanged, we just need to make sure to persist the values in
`Config` when loading the configuration.
This turns the string representation of the source and version attribute
of a module call into an expression. This leads to a change in the JSON
output:
`"source": "./foo"` --> `"source": {"constant_value": "./foo"},`
We previously used a loader -> BuildConfig flow to load configuration.
This commit changes most (but not all yet) flows to use the new
graph-based approach. Instead of simply recursively loading the modules,
we now need to take a stepped approach:
1. Load the root module
2. Collect the variables and their values
3. Build the configuration with the graph-based approach
Because this approach relies on different parts from different packages,
it can't easliy be done within the `configload` package. So, now we do
most of in the backend or command.
This new field will be consumed in order to redact sensitive values referenced in a list block's input configuration from the query logs. We simply pipe the config schema and the marked values, and include the paths in the query_start log output.
* feat: Make validate command detect when an unknown backend type is in use.
* feat: Make validate command detect when the backend configuration doesn't match the schema.
* fix: Stop suppressing the Required:true parts of the backend schema when validating backend blocks
* test: Add test showing validation fails when a required attribute is missing from a backend's config
* test: Add E2E test for using pluggable state storage with the `providers` command
Note: I've excluded the `terraform providers locks` and `terraform providers mirror` commands as they don't interact with backends.
* test: Add integration test for using pluggable state storage with the `providers` command
* refactor: Change ioutil.ReadDir to os.ReadDir
* test: Add integration test for using pluggable state storage with the `providers schema` command
* feat: Allow state store schema's to be included when schemas are marshalled into JSON output
* test: Assert that state stores are present in provider schemas returned from `providers schema`.
* test: Update existing tests to accommodate state stores being in provider schema output
* test: Update E2E test for `providers` commands to be better scoped to testing use of a state store to access and use state when generating output.
This complements TestProvidersSchema that tests that state stores in a provider are reflected in the JSON representations of the schemas that the command returns.
* chore: Replace `io/ioutil` with `io` in `providers schema` tests
* test: Show how the validate command doesn't detect when an unknown backend type is in the config.
* test: Minor fixes to error messages when test fails
* test: Add some TODOs in code comments
These tests are here to define existing behaviour, not define desired behaviour. I wanted to make that clear!
* command/meta: Enable migration from PSS to a backend
* Address PR feedback
* Update internal/command/meta_backend.go
Co-authored-by: Sarah French <15078782+SarahFrench@users.noreply.github.com>
* meta_backend: Rename stateStore_c_S to stateStore_to_backend
---------
Co-authored-by: Sarah French <15078782+SarahFrench@users.noreply.github.com>
* refactor: Rename Meta's backendState field to backendConfigState
This helps with navigating ambiguity around the word backend. The new name should indicate that the value represents a `backend` block, not a more general interpretation of what a backend is.
* fix: Only set backendConfigState to synthetic object if it's nil due to a lack of data. Don't change it if pluggable state storage is in use.
* feat: Enable recording a state store's details in an Operation, and using that data when creating a plan file.
* fix: Include provider config when writing a plan file using pluggable state storage
* fix: Having `backendConfigState` be nil may be valid, but it definitely isn't valid for `stateStoreConfigState` to be nil
When backendConfigState is nil it means that an implied local backend is in use, i.e. there is no backend block in the config.
* test: Add integration test showing that a plan command creates a plan file with the expected state_store configuration data
* refactor: Apply suggestion from @radeksimko
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* fix: Allow panics to occur if an unexpected implementation of `backend.Backend` is encountered when managing a state store
* docs: Add code comment explaining the current situation with passing backend config state to downstream logic.
In future this should be simplified, either via refactoring or changes affecting the implied local backend
---------
Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>
* test: Add E2E tests for `state list` and `state show` commands
* test: Update `mockPluggableStateStorageProvider` to log a warning during tests where the values in `MockStates` aren't compatible with the `ReadStateBytesFn` default function. Make existing test set an appropriate value in `MockStates`.
* test: Update `mockPluggableStateStorageProvider` helper to include a resource schema
* test: Add command-level test for `state list` showing integration with pluggable state storage code.
* test: Add command-level test for `state show` showing integration with pluggable state storage code.
* test: Add command-level test for `state pull` showing integration with pluggable state storage code.
* test: Add command-level test for `state identities` showing integration with pluggable state storage code.
* test: Add command-level test for `state rm` showing integration with pluggable state storage code.
* test: Add command-level test for `state mv` showing integration with pluggable state storage code.
* test: Add command-level test for `state push` showing integration with pluggable state storage code.
* test: Add command-level test for `state replace-provider` showing integration with pluggable state storage code.
* test: Change shared test fixture to not be named after a specific command under test.
This test fixure is reused across tests that need the config to define a state store but otherwise rely on the mock provider to set up the test scenario.
* test: Update test to use shared test fixture
* test: Remove redundant test fixture
The internal/command/testdata/state-commands-state-store and internal/command/testdata/state-store-unchanged test fixtures are the same.
* fix: Re-add logic for setting chunk size in the context of E2E tests using grpcwrap package
This was removed, incorrectly, in https://github.com/hashicorp/terraform/pull/37899
* refactor: Let panic happen if there's incompatibility between mock returned from `mockPluggableStateStorageProvider` and the `MockStates` that's been set in the mock
* test: Refactor to contain paths in reused variable, remove unnecessary .gitkeep
* test: Remove unneeded test code
* test: Fix typo in test name
* test: Show how the validate command validates resource blocks using their schema.
* test: Show how the validate command doesn't use backends' schemas when validating backend blocks.
* Add a generic method for loading an operations backend in non-init commands
* Refactor commands to use new prepareBackend method: group 1
* Refactor commands to use new prepareBackend method: group 2, where config parsing needs to be explicitly added
* Refactor commands to use new prepareBackend method: group 3, where we can use already parsed config
* Additional, more nested, places where logic for accessing backends needs to be refactored
* Remove duplicated comment
* Add test coverage of `(m *Meta) prepareBackend()`
* Add TODO related to using plans for backend/state_store config in apply commands
* Add `testStateStoreMockWithChunkNegotiation` test helper
* Add assertions to tests about the backend (remote-state, local, etc) in use within operations backend
* Stop prepareBackend taking locks as argument
* Code comment in prepareBackend
* Replace c.Meta.prepareBackend with c.prepareBackend
* Change `c.Meta.loadSingleModule` to `c.loadSingleModule`
* Rename (Meta).prepareBackend to (Meta).backend, update godoc comment to make relationship to (Meta).Backend more obvious.
* Revert change from config.Module to config.Root.Module
* Update `(m *Meta) backend` method to parse config itself, and also to adhere to calling code's viewtype instructions
* Update all tests and calling code following previous commit
* Change how an operations backend is obtained by autocomplete code
* Update autocomplete to return nil if no workspace names are returned from the backend
* Add test coverage for autocompleting workspace names when using a pluggable state store
* Fix output command: pass view type data to new `backend` method
* Fix in plan command: pass correct view type to `backend` method
* Fix `providers schema` command to use correct viewtype when preparing a backend
* Pull determining of PSS provider's version from current locks into a separate method
* Add code for identifying when config and provider version match existing backend state (i.e. no changes)
* Update test - locks are now needed before it hits expected error diag return
* Add test showing successful init when no config changes are detected.
* Update `getStateStorageProviderVersion` to return nil versions for builtin and re-attached providers.
This makes comparison easier when determining if config has changed since last init.
* Add test coverage for `getStateStorageProviderVersion`
* Move testing fixtures around, preparing for different types of changed state_store config changes being tested
* Add test showing that changing the state_store config is detected as a change, but handling this scenario isn't implemented yet
* Update hashes in test fixture backend state file to be accurate
Previously dummy values were fine, but as tests using hashes to identify changes these values need to be accurate!
* Update existing test cases so that Terraform uses the same test provider version as described in the backend state file fixture for the test.
* Add test showing that changing the PSS provider's config is detected as a change, but handling this scenario isn't implemented yet
* Add test showing that swapping to a different state storage implementation in the same provider is detected as a change, but handling this scenario isn't implemented yet
* Add test showing that changing the provider used for PSS is detected as a change, but handling this scenario isn't implemented yet
* Add test showing that upgrading a provider is detected as a change, but handling this scenario isn't implemented yet
* Update test to use v1.2.3 for consistency with other tests
Just to avoid any confusion if copy-pasting happens in future.
* More corrections to existing test fixtures - unset config should be null, and replace dummy hash values with correct values.
* Fix test for using -reconfigure with state_store; the default workspace would already exist in this scenario
* Update TestInit_stateStore_configUnchanged to assert that init was a no-op for backend state
* Remove unused fixture
* Remove test that's replaced by new tests in command/init_test.go
* Replace old references to deleted "state-store-changed" test fixture & update test to not expect a value for region attr in provider config
* Make test fixture coupling a little more understandable
* Refactor detection of no need to migrate into a function
* Add TODO about more involved provider version change tests
We will allow downgrades to succeed as long as the schema version number is unchanged
* Update (configs.StateStore)Hash method to return a single hash that's impacted by: state store config, provider config, state store type, provider source
* Update calling code and test helper code to reflect that the nested provider block no longer has its own hash
* Remove test; there is now a single hash that SHOULD be affected by the provider block!
* Also use provider name, from config, in hash
* Update tests to reflect changes in how hashes are made
* Remove unused `stateStoreConfigNeedsMigration` function
* Remove duplicate isProviderReattached function.
* Fixes to affected tests
* Allow provider version to impact the state storage hash, update impacted tests and test fixtures
* Update tests that now require locks data to be present in test setup
* Update comment for accuracy
* Fixes to other test fixtures - remove excess hash field, set hash to 0 to indicate they're not set accurately.
* Make upgrade test actually use upgrade code path
* Add lock files to test fixture directories that represent a project that's had a successful prior init using PSS
* Minor fixes in diagnostics
This can only be done once modules have been parsed and the required providers data is available. There are multiple places where config is parsed, into either Config or Module structs, so this needs to be implemented in multiple places.
* Rename test to make it specific to use of backend block in config
* Update initBackend to accept whole initArgs collection
* Only process --backend-config data, when setting up a `backend`, if that data isn't empty
* Simplify how mock provider factories are made in tests
* Update mock provider's default logic to track and manage existing workspaces
* Add `ProviderSchema` method to `Pluggable` structs. This allows calling code to access the provider schema when using provider configuration data.
* Add function for converting a providerreqs.Version to a hashicorp/go-version Version.
This is needed for using locks when creating the backend state file.
* Implement initial version of init new working directories using `stateStore_C_s`. Default to creating the default workspace if no workspaces exist.
* Update test fixtures to match the hashicorp/test mock provider used in PSS tests
* Allow tests to obtain locks that include `testingOverrides` providers.
The `testingOverrides` field will only be set in tests, so this should not impact end users.
* Add tests showing TF can initialize a working directory for the first time (and do the same when forced by -reconfigure flag). Remove replaced tests.
* Add -create-default-workspace flag, to be used to disable creating the default workspace by default when -input=false (i.e for use in CI). Refactor creation of default workspace logic. Add tests.
* Allow reattached providers to be used during init for PSS
* Rename variable to `backendHash` so relation to `backend` is clearer
* Allow `(m *Meta) Backend` to return warning diagnostics
* Protect against nil testingOverrides in providerFactoriesFromLocks
* Add test case seeing what happens if default workspace selected, doesn't exist, but other workspaces do exist.
The consequences here are due to using `selectWorkspace` in `stateStore_C_s`, matching what's done in `backend_C_r_s`.
* Address code consistency check failure on PR
* Refactor use of mock in test that's experiencing EOF error...
* Remove test that requires test to supply input for user prompt
This test passes when run in isolation but fails when run alongside other tests, even when skipping all other tests using `testStdinPipe`. I don't think the value of this test is great enough to start changing how we test stdin input.
* Allow -create-default-workspace to be used regardless of whether input is enabled or disabled
* Add TF_SKIP_CREATE_DEFAULT_WORKSPACE environment variable
* Responses to feedback, including making testStdinPipe helper log details of errors copying data to stdin.
Note: We cannot call t.Fatal from a non-test goroutine.
* Use Errorf instead
* Allow backend state files to not include version data when a builtin or reattached provider is in use.
* Add clarifying comment about re-attached providers when finding the matching entry in required_providers
* Report that the default workspace was created to the view
* Refactor: use error comparison via `errors.Is` to identify when no workspaces exist.
* Move handling of TF_ENABLE_PLUGGABLE_STATE_STORAGE into init's ParseInit func.
* Validate that PSS-related flags can only be used when experiments are enabled, enforce coupling of PSS-related flags when in use.
* Slight rewording of output message about default workspace
* Update test to assert new output about default workspace
* Add test coverage for Meta's `savedBackend` method
* Add new Meta `savedStateStore` method and test coverage
* Streamline test - remove unneeded assertions and update comments
* Remove marks from config before configuring the provider
* Remove marks from config before configuring the state store
* Add test case for savedStateStore to assert marks aren't passed
* Fix call to ConfigureStateStore
* Show that tests pass despite not trying to remove marks
* Allow Config methods to add marks when reading pluggable state store config from the backend state file
* This code is now necessary to let the tests pass
* Stop adding marks to PSS-related config when it's parsed from the backend state file
* Stop removing marks that aren't there
* Remove unnecessary test related to marks
