Migrate terraform state to direct before deploy by denik · Pull Request #6749 · databricks/cli

denik · 2026-09-18T10:29:11Z

Changes

Run the terraform→direct state migration before plan/deploy (after the build, so ${artifacts.*} and library paths are resolved) instead of after a terraform deploy, so the deploy itself runs on the direct engine. The migration:

seeds id-composing fields from the deployed terraform state and compares them normalization-aware (identifier case, trailing slash), so a backend-normalized id converges while a genuine change surfaces as a recreate/rename (reconcileIDFields);
is prepared in memory and reversible; a deploy commits it — pushing the direct state and retiring the terraform state — only once the deploy is approved, before applying its own changes. A declined deploy drops it and stays on terraform, so a refused recreate changes nothing (a genuine recreate is gated by --auto-approve like any other);
falls back to terraform for that run (with a warning) if a pre-commit step fails — parse, convert, or plan check — so a failed migration never blocks a deploy that would otherwise succeed;
tags the user agent with the engine the run actually uses.

engine: terraform still opts out. Destroy commits the migration as part of its teardown and then removes all local state, so no terraform state or backup is left behind. The terraform-engine removal is stacked on top of this PR.

Tests

Acceptance tests under acceptance/bundle/migrate/ cover rename, cross-catalog move, normalization-converge, recreate-refused-without---auto-approve (then applied with it), push/plan/backup failure handling, and clean teardown on destroy (no leftover local or remote state), across schemas, volumes, registered models and pipelines. The rename/converge/catalog-move tests also run against a real workspace (Cloud = true, verified on aws-cli); the storage/external-location cases run on the testserver.

This pull request and its description were written by Isaac.

eng-dev-ecosystem-bot · 2026-09-18T11:02:07Z

Integration test report

Commit: e7f1948

Run: 36428007042

	Env	✅pass	🙈skip	Time
✅	aws linux	294	50	6:51
✅	aws windows	296	48	8:07
✅	azure linux	293	50	7:33
✅	azure windows	295	48	6:47
✅	gcp linux	294	50	7:44
✅	gcp windows	296	48	6:32

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
6:04	aws windows	TestAccept
4:46	azure windows	TestAccept
4:26	gcp windows	TestAccept
4:14	azure linux	TestAccept
3:55	aws linux	TestAccept
3:54	gcp linux	TestAccept

## Changes Reorder the steps in `ProcessBundleRet`. The single pre-build `shouldReadState` block is split in two: the state **pull** (which determines the engine) stays up front, while opening the resolved state moves to after Build. **Old:** pull-state → open-state → FastValidate → Validate → Build → deploy **New:** pull-state → FastValidate → Validate → Build → open-state → deploy where: - **pull-state** = `PullResourcesState` — determine the engine, set `MigratingToDirect`, print the deploy notice, and validate `--select`. Needs only the pulled engine descriptor, so it stays before FastValidate (unchanged). - **open-state** = open the direct state, DMS fetch/open, deployment-history enforcement, `InitIDs`, and the `--plan` file load. Needs the opened/resolved state, so it moves to after Build. The reorder is safe because the two sides are independent: - validate/build do not read the deployment state, and - open-state does not depend on validate/build output. (`Validate` may issue workspace API calls — that's why it is separate from `FastValidate` — but it does not need the state, so running it before open-state is fine.) The auto-migration still runs post-deploy, so behavior is unchanged. ## Why Prep for databricks#6749 — move the terraform→direct migration before deploy: that migration converts the bundle config into the direct state and must run **after Build**, because library and `${artifacts.*}` references (e.g. `whl: ./dist/*.whl`) are only resolved during Build. Running migration before Build would bake unexpanded globs into the migrated state instead of resolved remote paths. Landing this reorder on its own keeps databricks#6749's diff to the actual behavior change. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>

## Changes Add an invariant target that deploys on terraform, then deploys on the direct default (which auto-migrates the state), then asserts no drift — across every resource config. Mirrors `invariant/migrate` but exercises the deploy-triggered auto path rather than the explicit `bundle deployment migrate` command. ## Why Prep for databricks#6749 (move terraform→direct state migration before deploy): landing the auto-migration drift coverage first, off main, so that PR stays focused on the behavior change. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>

…cesState (databricks#6808) ## Changes The user-agent `engine` dimension used to be set inside `PullResourcesState`. Move it to the callers (`bundle deploy`/`plan` in process.go and the two `bundle generate` resource paths), so the engine tag lives next to where the resolved engine is known. Follow-on cleanups from that move: - `PullResourcesState` no longer mutates the context, so it now returns only `*StateDesc` instead of `(context.Context, *StateDesc)`. - `generate dashboard` and `generate genie-space` had byte-identical state-loading blocks (initialize, resolve engine, pull state, tag engine, load). Extracted into one `loadStateForGenerate` helper so the tag is set in a single place. ## Why Prep for databricks#6749 (move terraform→direct state migration before deploy): that PR needs the engine tag set from the resolved (post-migration) engine, so consolidating the tagging off main keeps its diff focused on the behavior change. ## Tests Behavior-preserving. `acceptance/bundle/user_agent` asserts the `engine/` tag on every recorded request for both engines; it plus the `generate` and `state` suites and the generate unit tests pass with no golden changes. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>

When the direct engine is requested (the default) and the existing state still uses terraform, convert the state to the direct engine before the run proceeds, instead of after deploy. deploy/destroy commit the migration (resources.json written and pushed, terraform.tfstate backed up); plan and summary keep it in memory. The migration runs after phases.Build so its conversion and plan check see library and ${artifacts.*} references resolved (matching what a direct deploy records), and it falls back to the terraform engine if the plan check fails. Terraform-state cleanup after the commit is best-effort: resources.json already outranks the terraform state by serial, so a failed backup/delete only warns. Co-authored-by: Isaac <no-reply@databricks.com>

Now that the migration runs before deploy, a hard error blocks a deploy that would otherwise succeed on terraform. Treat every failure before resources.json is pushed (parse, conversion, empty-state sweep) as non-fatal: warn and deploy on terraform this time, retrying the migration next run - matching what plan-check and push failures already did. Only a failure after the push (placing/opening the local state) stays an error, since the workspace is already committed to direct; reword those messages to say the migration succeeded and re-running recovers. Co-authored-by: Isaac <no-reply@databricks.com>

The pre-deploy migration plan-checks the converted state. If that plan would recreate (destroy + create) an existing resource, do not commit the migration: a recreate risks data loss, whether it comes from a conversion that did not faithfully reproduce an immutable field or from a real pending config change (which terraform would recreate too). Fall back to terraform this run and retry the migration next run, once the recreate is applied. Records direct_migrate_recreate_planned. checkPlanOnTempState now returns the plan so the caller can inspect the planned actions. Co-authored-by: Isaac <no-reply@databricks.com>

The pre-deploy migration flips the engine terraform->direct, but the SDK user agent only appends dimensions, so tagging engine/terraform in PullResourcesState and then engine/direct after a successful migration left both on a migrated deploy's requests. PullResourcesState now skips the tag on the auto-migration path (direct requested, state still terraform); the caller sets engine from the resolved stateDesc.Engine once the migration has run, been skipped, or fallen back - so a migrated deploy is engine/direct, a fallback is engine/terraform, and there is never a stale second tag. The two generate commands, which never migrate, tag the resolved engine too. auto-migrate-envvar now records the deploy's engine tags to lock this in. Co-authored-by: Isaac <no-reply@databricks.com>

Records (no fix) how the pre-deploy migration treats a schema's named id fields when they change or are backend-normalized: - name backend-lowercased (config MySchema vs deployed myschema): spurious warnOnIDFieldRename warning (also emitted by the terraform deploy's dry-run telemetry), but the follow-up plan converges. - catalog_name legitimately changed (immutable id): only a warning, no recreate - the migration records the config value and the moved-catalog drift is silently stable. - storage_root trailing slash normalized: absorbed by normalize_slash, no warning, converges. Co-authored-by: Isaac <no-reply@databricks.com>

…odels, pipelines Extends the schema characterization (no fix) across more resources: - volume name backend-lowercased: spurious rename warning, converges (same as schema name). - volume_type changed (both a provided id field AND recreate_on_changes): the recreate classification wins - the recreate guard fires and the run stays on terraform (contrast with catalog_name, a pure provided id field, which only warns). - registered_model catalog_name changed (immutable id): warn + silently stable drift, no recreate (same as schema catalog_name). - pipeline storage changed (pure recreate_on_changes): recreate guard fires. Catalogs are direct-only (no terraform converter), so they are never a migration scenario and are intentionally not covered. Co-authored-by: Isaac <no-reply@databricks.com>

The migration seeded id-composing fields (provided_id_fields, updatable_id_fields) from config, which snapshotted a pending id change as already applied: a genuinely-moved/renamed resource then plan-converged and silently drifted from the backend, while a backend-normalized value (identifier case, trailing slash) produced a spurious "rename not applied" warning - also leaked onto plain terraform deploys via the dry-run telemetry. reconcileIDFields now seeds each id field from the deployed terraform value unless it differs from config only by backend normalization (case-insensitive + trailing-slash), in which case the config value is kept. So a real id change surfaces in the plan (recreate for provided_id -> caught by the recreate guard; rename update for updatable_id), and a normalized value converges with no warning. warnOnIDFieldRename is removed (its genuine case is now the recreate guard, its false-positive case is gone). Regenerated the id-field characterization goldens, which now assert the fixed behavior (recreate on genuine change, clean converge on normalization). Co-authored-by: Isaac <no-reply@databricks.com>

The terraform fallback leaves two terraform.tfstate* files, and find lists them in filesystem order, which differs on Windows. Pipe the finds through sort so the golden is deterministic across platforms. Co-authored-by: Isaac <no-reply@databricks.com>

First cloud coverage for the pre-deploy migration: renaming a schema (an immutable provided id field) must recreate. Verifies against a real workspace that the recreate guard detects the recreate, falls back to terraform (which recreates the schema under the new name), and the following deploy migrates the now-matching state to direct. Passed on aws-cli. Co-authored-by: Isaac <no-reply@databricks.com>

Companion to cloud-recreate-schema. A volume name is an updatable id field, so unlike a schema (which recreates) it is renamed in place. Verifies against a real workspace that the migration seeds the deployed name, the plan renames the volume (UpdateWithID) rather than snapshotting the new name as applied, the migration succeeds (no fallback), and the schema alongside it is untouched. Passed on aws-cli. Co-authored-by: Isaac <no-reply@databricks.com>

Extend the two schema recreate tests (local auto-migrate-recreate and cloud cloud-recreate-schema) with a step that deploys the recreate WITHOUT --auto-approve first: the migration guard falls back to terraform, which refuses the destructive recreate (exit 1) and changes nothing. The following --auto-approve step then applies it. The cloud step passed on aws-cli. Co-authored-by: Isaac <no-reply@databricks.com>

Cleanup from a code review of the Migrate/CommitMigration split: - Migrate never used its requiredEngine argument (only CommitMigration needs it); drop it from Migrate and from migrateTerraformToDirect and their callers. - convertTFStateToDirect returned a resourceCount both callers discarded (CommitMigration counts via StateDB.ExportState instead); drop it. - Fix stale comments: the converter no longer renames a temp file into place (the caller reads it into memory), and correct the serial description - a populated conversion lands at tf+2 (its own WAL replay bumps the tf+1 base once more), an empty one stays at tf+1; either way it outranks the terraform state. No behavior change. Co-authored-by: Isaac <no-reply@databricks.com>

…e delete The 403 was injected at OFFSET=0, which hits the artifacts/.internal cleanup delete, not the terraform.tfstate delete - so the tf delete succeeded and the test never exercised the retirement-failure path (the remote state dir showed only terraform.tfstate.backup). The tf delete is the second workspace/delete of a migrating deploy, so fault it at OFFSET=1. The golden now shows the "could not delete terraform.tfstate" warning, the migration succeeding anyway (best-effort), and both the remote terraform.tfstate and its .backup remaining. Co-authored-by: Isaac <no-reply@databricks.com>

…t-approval window) CommitMigration is the point of no return: it runs after approval and before the apply, so a deploy that fails during the apply has still durably migrated. This test forces that window - a job change makes the migrating deploy attempt a jobs/reset, which is faulted (403) after the migration commits - and asserts the bundle is on direct (resources.json present, terraform state retired, direct_migrated_via_env recorded) even though the deploy errored, and that a retry finishes the apply on the direct state without re-migrating. Co-authored-by: Isaac <no-reply@databricks.com>

…s WAL A migrating destroy ran on the in-memory converted state and called UpgradeToWrite without first persisting a base (unlike deploy's CommitMigration). If the destroy was interrupted between UpgradeToWrite and Finalize, it left an orphan .wal with no state file - the next run re-migrated in memory and tripped UpgradeToWrite's O_EXCL on that orphan WAL, wedging the bundle until the WAL was removed by hand. Persist the base first: an interrupted destroy now leaves a committed direct state on disk, so the next run resolves to it (a direct state) and its file-backed Open recovers the WAL instead of re-migrating. destroyCore still removes the local state at the end, so a successful destroy is unchanged. Co-authored-by: Isaac <no-reply@databricks.com>

…tion, no commit) bundle run needs resource ids, so on a terraform state with direct requested it migrates in memory to resolve them - but it is read-only w.r.t. the engine. This test asserts the run fires after an in-memory migration yet leaves nothing durable: no resources.json, no resources.migrating temp, no WAL, terraform.tfstate intact, and no migration-source telemetry. The bundle stays on terraform; a later deploy is what commits. Co-authored-by: Isaac <no-reply@databricks.com>

Migration produces a plain direct state with no deployment_history feature, so deploying with DATABRICKS_BUNDLE_DEPLOYMENT_HISTORY=true against a terraform state is a mismatch. This asserts it is rejected after the in-memory migration but before CommitMigration writes or pushes anything - the bundle stays on terraform with no leaked direct state. Migration is otherwise non-DMS only, so this combination had no coverage. Co-authored-by: Isaac <no-reply@databricks.com>

…ed-after-prep) The migration is prepared in memory (in process.go) before the destroy phase asks for approval, so a refused destroy is the prepared-but-not-committed path. Strengthen the refusal assertions to confirm nothing durable is left behind - no resources.json, no resources.migrating temp, no WAL - only the terraform state remains. (An interactive "no" hits the same drop but is not acceptance-testable, since there is no TTY to prompt.) Co-authored-by: Isaac <no-reply@databricks.com>

A migration committed by deploy retires the terraform state by renaming the local file to terraform.tfstate.backup. A later destroy ran on the direct engine but removed only the direct state, so the .backup lingered after the deployment was gone. Remove it on any direct destroy - it is never read as live state, so this is always safe - which also lets the empty terraform/ dir be pruned. New acceptance test destroy-after-deploy covers deploy-migrate then destroy and asserts no state is left behind, local or remote. Co-authored-by: Isaac <no-reply@databricks.com>

Replace the per-name terraform.tfstate/resources.json/WAL finds in the destroy and run-no-commit tests with a single find.py listing (sorted, forward slashes, cross-platform, excludes the .terraform provider cache) plus a workspace list of the remote state directory. The goldens now show exactly which state is present locally and remotely, so a stray tfstate/.backup would surface. Co-authored-by: Isaac <no-reply@databricks.com>

…t migrate test The standalone destroy-after-deploy test re-deployed the same terraform->direct migration the default test already sets up. Append the destroy plus state-file listing (local find.py + remote workspace list) to default instead, and drop the duplicate test. Still asserts a destroy after a deploy-committed migration leaves no terraform.tfstate.backup, direct state, or remote deployment behind. Co-authored-by: Isaac <no-reply@databricks.com>

The workspace list after a destroy is expected to fail (the state directory was deleted), so assert that with musterr rather than errcode: musterr fails the test if the command unexpectedly succeeds (i.e. the remote deployment was not removed), and drops the stray Exit code line from the golden. Co-authored-by: Isaac <no-reply@databricks.com>

… the suite Extend the pattern already used in destroy/default/run-no-commit to the remaining migrate/auto tests: replace the per-name resources.json/terraform.tfstate*/WAL finds with a single find.py listing (sorted, forward slashes, cross-platform, excludes the .terraform provider cache), and add a remote workspace list of the state dir where it adds signal - the non-cloud, remote-deployed tests that do not already surface remote state via debug states. The remote view now shows things the local find could not, e.g. push-failure has resources.json locally but absent remotely (the faulted push never landed). Co-authored-by: Isaac <no-reply@databricks.com>

…form state Collapse the three separate state-file removals in destroyCore into one unconditional pass over both engines' state files plus terraform.tfstate.backup, keeping terraform-before-direct ordering for crash safety. The migrating-gated terraform.tfstate removal is gone: a destroy tears everything down, and a coexisting terraform state is always superseded (a direct state only wins engine resolution by a higher serial, i.e. it is the newer migrated-from version), so there is nothing worth preserving for a later deploy to resurrect. Drops the now unused migrating parameter from destroyCore. state_present covered the old resurrection behavior; update it: after the destroy the coexisting terraform state is gone too, so the next deploy is a fresh create on the direct engine (serial 1) rather than a resurrect-migrate to serial 6. Co-authored-by: Isaac <no-reply@databricks.com>

janniklasrose · 2026-09-28T14:45:24Z

+	if recreated := recreatedResources(plan); len(recreated) > 0 {
+		b.Metrics.SetBoolValue(metrics.DirectMigrateRecreatePlanned, true)
+		log.Infof(ctx, "migration to the direct engine will recreate %v; the command's approval gates this", recreated)
 	}


why do we need this specific metric & log line? any concerns with recreating during migration that prompt this?

The concern is spurious recreation - caused by differences in state representation between tf and direct and not by user intent to recreate.

Having the metric gives upper bound on this (most recreates hopefully will be legit).

## Release v1.19.0 ### CLI * Honor `CLAUDE_CONFIG_DIR` in `aitools` commands. ([#6838](#6838)) * Added `--ttl` and `--no-expiry` flags to `databricks postgres create-branch` so a branch's expiration can be set without hand-writing a `--json` spec. `--ttl` accepts the REST API duration form (`604800s`), a Go duration (`168h`), or day/week units (`7d`, `3w`); `--no-expiry` creates a branch that never expires. One of `--ttl`, `--no-expiry`, or a spec expiration in `--json` is required. ([#6313](#6313)) * `databricks ssh connect` serverless sessions now provide Claude Code and Codex configured with Unity Gateway out of the box. ([#6885](#6885)) ### AI Runtime * Add `databricks air images push` (Preview) to configure Docker authentication and push container images to Databricks Artifact Registry. ([#6869](#6869)) * `air run` now grants the configured `permissions` on the MLflow experiment as well as the job. ([#6870](#6870)) ### Bundles * Add libraries field to clusters. ([#6831](#6831)) * Error out when a configured `workspace_id` does not match the connected workspace, instead of silently using it in resource URLs emitted by `bundle summary`. ([#6754](#6754)) * Fix spurious recreation of Lakebase (Postgres) branches, roles, and catalogs when the referenced project is updated in place: an in-place project change (e.g. `display_name`) no longer forces a delete + create of resources that reference the project's or branch's `name`. ([#6865](#6865)) * Direct engine now detects and applies an explicitly configured integer zero (e.g. `gcp_attributes.local_ssd_count: 0`) added to a resource first deployed without the field. ([#6867](#6867)) * Migrate existing Terraform deployment state to the direct engine before deploying (previously done after a Terraform deploy), so the deploy runs on the direct engine. ([#6749](#6749)) * The `postgres_snapshot_schedules` resource (introduced in [v1.16.0](https://github.com/databricks/cli/releases/tag/v1.16.0)) is now marked Beta and is no longer available in PyDABs, matching the other `postgres_*` resources; configure it in YAML instead. ([#6887](#6887))

denik mentioned this pull request Sep 21, 2026

WIP terraform engine removal #6746

Closed

denik changed the base branch from main to denik/migration-backup-fix September 21, 2026 14:58

Base automatically changed from denik/migration-backup-fix to main September 22, 2026 09:01

denik mentioned this pull request Sep 22, 2026

internal: resolve state after build phase #6788

Merged

denik changed the base branch from main to denik/process-reorder September 22, 2026 10:37

denik force-pushed the denik/migration-before-deploy branch from 32fe6bb to 14e8c46 Compare September 22, 2026 11:08

Base automatically changed from denik/process-reorder to main September 22, 2026 11:37

denik force-pushed the denik/migration-before-deploy branch from 6c1fd6c to 7b7aabe Compare September 22, 2026 12:21

github-actions Bot added the DABs DABs related issues label Sep 22, 2026

denik mentioned this pull request Sep 23, 2026

acc: add invariant/auto-migrate test #6806

Merged

denik mentioned this pull request Sep 23, 2026

internal: move the user-agent engine tag to the callers of PullResourcesState #6808

Merged

denik force-pushed the denik/migration-before-deploy branch from edc9b68 to 8b1f858 Compare September 23, 2026 11:02

denik force-pushed the denik/migration-before-deploy branch 3 times, most recently from bc89192 to 384c260 Compare September 25, 2026 07:50

denik and others added 11 commits September 27, 2026 19:51

denik and others added 12 commits September 28, 2026 08:24

denik mentioned this pull request Sep 28, 2026

Remove the Terraform deployment engine #6864

Closed

2 of 4 tasks

denik marked this pull request as ready for review September 28, 2026 13:59

denik requested review from a team as code owners September 28, 2026 13:59

denik requested a review from anton-107 September 28, 2026 13:59

andrewnester approved these changes Sep 28, 2026

View reviewed changes

denik requested review from janniklasrose, pietern and shreyas-goenka September 28, 2026 14:27

janniklasrose approved these changes Sep 28, 2026

View reviewed changes

denik added this pull request to the merge queue Sep 28, 2026

Merged via the queue into main with commit 0f52d4c Sep 28, 2026
31 checks passed

denik deleted the denik/migration-before-deploy branch September 28, 2026 15:26

denik mentioned this pull request Sep 28, 2026

Remove terraform dependency in migrate tests #6866

Merged

rugpanov mentioned this pull request Oct 1, 2026

Update Databricks CLI to v1.19.0 databricks/databricks-vscode#2229

Merged

Navigation Menu

Migrate terraform state to direct before deploy - #6749

Migrate terraform state to direct before deploy#6749
denik merged 52 commits into
mainfrom
denik/migration-before-deploy

denik commented Sep 18, 2026 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Sep 18, 2026 •

edited

Loading

Uh oh!

janniklasrose Sep 28, 2026

Uh oh!

denik Sep 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

denik commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

eng-dev-ecosystem-bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Integration test report

Uh oh!

janniklasrose Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

denik Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

denik commented Sep 18, 2026 •

edited

Loading

eng-dev-ecosystem-bot commented Sep 18, 2026 •

edited

Loading