How to Migrate to Databricks Without Breaking What Already Works

Why Migrate to Databricks?

Teams migrate to Databricks to run data engineering, analytics and machine learning on one platform. Most legacy estates split those jobs across separate systems. The warehouse serves BI. A Hadoop cluster or cloud lake holds raw data. Data scientists pull extracts into their own notebooks. Every handoff adds another pipeline to maintain and another copy to govern.

Moving to a lakehouse collapses those layers into one. The usual drivers look like this.

• One copy of the data. Delta Lake stores tables in open formats on your own cloud storage. BI, engineering and ML workloads read the same tables.

• AI on governed data. Models train on the tables analysts already query. No exports to a side environment.

• Central governance. Unity Catalog handles access control, lineage and auditing across workspaces.

• Exit from licenses and hardware. Teradata renewals and on-prem Hadoop refresh cycles often force the timing.

• Compute separate from storage. Clusters scale with demand under usage-based pricing instead of fixed capacity.

The case is weaker for some teams. A small analytics team running SQL dashboards on a well-tuned warehouse may gain little. Migration effort and retraining can outweigh the savings there. Usage-based pricing also cuts both ways. Poorly sized clusters can cost more than the license they replaced. Databricks pays off when the roadmap includes ML, streaming or large-scale engineering work.

Which Platforms Can You Migrate to Databricks From?

Most migrations start from a cloud warehouse, a legacy appliance or a Hadoop cluster. Each source brings a different kind of work. Some are mostly data movement. Others are mostly code conversion

Snowflake to Databricks

Snowflake to Databricks migrations are usually lighter on data and heavier on logic. Both platforms run in the cloud, so tables can be unloaded to Parquet in object storage and registered as Delta tables without much friction. Lakehouse Federation also lets Databricks query Snowflake in place, which helps when you want to run both systems side by side during the transition.

The real effort sits in Snowflake Scripting procedures, tasks and streams. Tasks move to Lakeflow Jobs. Streams usually map to Delta change data feed. Teams still weighing the two platforms should settle that choice first, since our Databricks vs Snowflake comparison shows they suit different workload mixes. If your Snowflake usage is mostly SQL reporting, a move may not justify itself.

Teradata to Databricks

Teradata migrations carry the most code. A typical estate includes the following.

• BTEQ scripts that wrap SQL with control logic

• Stored procedures and macros built over many years

• FastLoad, MultiLoad and TPT jobs for ingestion

• Physical design based on primary indexes

The SQL converts reasonably well with automated tools. Primary index design does not carry over, so tables need rethinking around liquid clustering and partitioning. Data is usually exported through TPT into cloud storage. These projects often run as part of a wider data warehouse modernization effort, because the license renewal date tends to set the timeline.

Hadoop to Databricks

Hadoop is the closest fit. Databricks runs on Spark, so many existing Spark jobs port with modest changes. Most of the work is mapping each Hadoop component to its replacement.

The hidden risk is undocumented work. Old clusters accumulate cron jobs and shell scripts that nobody owns anymore. A full inventory matters more here than on any other source, which is a pattern common to most legacy data migration projects. 

Other Platforms to Databricks

Databricks also supports migrations from Oracle, SQL Server, Amazon Redshift, Azure Synapse, Netezza and Google BigQuery. The pattern stays the same across all of them. Data lands in cloud storage as Delta tables. SQL and stored procedures go through conversion. ETL tools like Informatica or SSIS get rebuilt as notebooks or Lakeflow pipelines. Of these, SSIS and Informatica rebuilds are often the slowest part. Expect them to need more manual effort than the SQL itself.

Step-by-Step Process to Migrate to Databricks

A Databricks migration runs in seven stages. They overlap in practice, especially on large estates that move in waves. The order still matters, because each step depends on decisions made in the one before.

Step 1: Assess the Existing Data Environment

Start by building a complete inventory of what runs today. Skip this and scope creeps the moment forgotten jobs surface mid-project. The inventory should cover the following.

• Databases, schemas and table sizes

• Stored procedures, views and SQL scripts

• ETL jobs, schedules and their dependencies

• BI reports and the tables they read

• Users, roles and access policies

• Query volumes and peak usage windows

Automated profilers speed this up by scanning query logs and code repositories. They also flag code that will need manual conversion. Treat their output as a starting point. Interviews with the teams who own each workload catch what logs miss.

Step 2: Define the Target Architecture

The target architecture decides how data is organized, governed and accessed once it lands. Most teams adopt a medallion layout with bronze for raw data, silver for cleaned data and gold for business-ready tables. Our guide on data pipeline architecture covers how these layers connect.

Unity Catalog setup belongs here too. Catalogs, schemas, workspaces and permission models should exist before the first table moves. Retrofitting governance after go-live is slow and error-prone.

Teams with a small, simple estate can keep a flatter structure. A full medallion design adds overhead that only pays off at scale.

Step 3: Plan Data and Workload Migration

Migration planning sets the order in which workloads move and how much moves at once. There are two broad approaches.

Phased migration is the safer default. It does mean running two platforms in parallel for months, which doubles infrastructure spend for that period. A first wave of one or two representative workloads works as a pilot and exposes problems early. The data migration cost calculator helps size the effort for each wave. 

Step 4: Migrate Data and Tables

Data migration usually happens in two passes. A bulk historical load comes first. Incremental syncs then keep Databricks current until cutover.

Data typically gets exported from the source into cloud object storage as Parquet or CSV files. From there, Auto Loader or COPY INTO loads it into Delta tables. Schema mapping needs care at this stage. Data types, precision and date handling differ across platforms, and small mismatches cause reporting errors later.

For very large tables, partition the export by date. Loading a single multi-terabyte extract makes failures expensive to retry.

Step 5: Convert SQL and Data Pipelines

Code conversion is where most migration time goes. Automated converters translate source SQL dialects into Databricks SQL or PySpark and handle a large share of standard queries. Complex stored procedures, dynamic SQL and platform-specific functions still need engineers to rewrite them by hand.

Pipelines need the same attention. Schedulers like Oozie, Control-M or Snowflake tasks move to Lakeflow Jobs. ETL tools such as Informatica or SSIS are usually rebuilt as notebooks or Lakeflow declarative pipelines. A straight line-by-line conversion often works first. Refactoring for Spark performance can follow once the logic is proven correct.

Step 6: Test and Validate Migrated Workloads

Validation proves the new platform produces the same answers as the old one. Without it, business users find the errors first, and trust in the platform drops fast.

Run checks at several levels.

• Row counts per table between source and target

• Aggregates such as sums and averages on key columns

• Business logic by comparing report outputs side by side

• Performance of critical queries against agreed thresholds

Reconciliation tools can automate the first two checks. Business logic checks need the report owners involved. A data quality framework set up at this stage keeps checks running after go-live too.

Step 7: Cut Over and Optimize

Cutover switches users, reports and downstream systems to Databricks. Schedule it for a low-traffic window. Keep the source system in read-only mode for a fixed period as a fallback.

Optimization starts after cutover. Right-size clusters based on real usage. Turn on auto-termination. Apply liquid clustering to large, frequently filtered tables. Then decommission the legacy platform once the fallback period ends, because every month it stays running adds to the bill.

Databricks Migration Tools

Databricks migration tools fall into three groups. Some assess and convert code. Some move data. Others validate the results. No single tool covers the whole migration, so most projects combine several.

Lakebridge is the tool most teams start with. It grew out of the earlier Remorph and BladeBridge transpiler and analyzer tools, and it works in three stages. The Analyzer scans source code and estimates complexity. The Converter translates legacy SQL and ETL code into Databricks SQL or PySpark. Reconcile then compares source and target data, and supports schema, column and row level checks with exact, aggregate and threshold-based comparisons.

It has real gaps. Lakebridge does not copy data, so loading still happens through the tools above. It is a Databricks Labs project provided as-is without Databricks SLAs. Source support also varies by stage. Alteryx, for example, is supported by the Analyzer but not by the Converter or Reconcile. Complex stored procedures and dynamic SQL are where Lakebridge still needs engineers to step in and rewrite by hand.

Third-party validation tools and partner accelerators fill some of these gaps. They make sense for large estates with thousands of objects. For a small migration, Lakebridge plus native loading tools is usually enough.

Databricks Migration Best Practices

Most migration problems trace back to a handful of planning gaps. The general cloud data migration best practices still apply here. The practices below are the ones that matter most for Databricks specifically.

1. Start With Workload Assessment, Not Data Volume Alone

Plan by workload complexity, because data volume rarely predicts effort. A 50 TB table with a simple daily load can move in a week. A 2 GB schema with 400 stored procedures can take months. Score each workload on code complexity, number of dependencies and business criticality. Keep volume in the score too, since it still drives transfer time and storage cost. It just shouldn’t be the only factor deciding the timeline.

2. Prioritize High-Value, Lower-Risk Workloads

Early waves should prove value without putting core operations at risk. Sorting workloads on two axes makes the order clearer.

This order gives way when a contract forces it. If a Teradata license expires in nine months, that system moves first regardless of its risk score.

3. Validate Business Logic, Not Just Data

Row counts can match perfectly while every revenue report is wrong. Platforms handle certain operations differently, and the differences only show up in calculated results. Common culprits include the following.

• NULL handling in joins and aggregations

• Integer division and decimal precision

• Time zone and date conversions

• Case sensitivity in string comparisons, since Teradata defaults to case-insensitive matching

Have report owners compare outputs side by side and sign off before cutover. They know which numbers look wrong faster than any automated check.

4. Plan Governance and Security Early

Set up Unity Catalog before the first table lands. Retrofitting permissions onto hundreds of migrated tables is slow and leaves gaps. Map source roles to Unity Catalog groups, tag sensitive columns and define row filters and column masks during the architecture phase. These decisions should follow your wider data governance strategy rather than be invented per project. A short proof of concept on non-sensitive data can run on a minimal setup. Anything heading to production cannot.

5. Benchmark Performance Before Production Cutover

Run critical queries and pipelines against production-scale data before switching users over. Converted code can be logically correct and still run slower than it did on the source. Small files, missing clustering and undersized SQL warehouses are the usual causes. Agree on performance thresholds with report owners in advance, so cutover has a clear pass or fail line.

6. Monitor and Optimize Costs After Migration

Costs often rise in the first few months after migration. Parallel running, oversized clusters and idle all-purpose compute all add up. Forecasts made before migration are rarely accurate either, because real usage patterns only appear once teams are on the platform.

Track spend through system billing tables and set budget alerts per workspace. Move scheduled work to job compute. Use serverless SQL warehouses for BI. Turn on auto-termination for interactive clusters. The Databricks cost calculator helps set a baseline, and our breakdown of Databricks migration cost explains which factors drive the bill during and after the move.

7. Keep a Rollback and Recovery Plan

A failed cutover without a rollback plan turns into an outage. Define the plan before go-live.

1. Keep the source system in read-only mode for a fixed period after cutover.

2. Continue incremental syncs until business owners sign off.

3. Set clear rollback triggers, such as reconciliation failures or missed SLAs.

4. Use Delta time travel to recover from bad loads inside Databricks.

Low-risk waves can run with a lighter version of this plan. Finance and customer-facing workloads need the full version.

Databricks Migration Checklist

Run through this checklist at each stage of the migration. Each item maps to a decision or check that commonly gets skipped under deadline pressure.

Before migration

• Full inventory of tables, code, jobs, reports and users is complete

• Each workload has a complexity and criticality score

• Migration approach is chosen, phased or big bang

• Wave plan and pilot workload are agreed

• Target architecture and medallion layers are defined

• Unity Catalog catalogs, groups and permissions are in place

• Sensitive columns are tagged and masking rules are set

• Budget and cost alerts are configured per workspace

During migration

• Historical data is loaded and incremental syncs are running

• Data types and precision are mapped and checked

• SQL, stored procedures and pipelines are converted

• Manually rewritten code is reviewed by a second engineer

• Row counts and aggregates reconcile between source and target

• Report owners have compared outputs and signed off

• Critical queries meet agreed performance thresholds

At and after cutover

• Rollback triggers and owners are documented

• Source system is set to read-only for the fallback period

• Users, reports and downstream systems point to Databricks

• Clusters are right-sized and auto-termination is on

• Spend is reviewed weekly for the first three months

• Legacy platform is decommissioned once the fallback period ends

Conclusion

The decision to migrate should come from the workload roadmap. Teams planning ML, streaming or large-scale engineering get the most from the move. Teams running mostly SQL reporting on a stable warehouse may be better off staying put. For teams that do move, the order is assessment first, governance second and conversion third. Brilworks helps data teams plan and run that sequence through its data modernization services, from workload assessment to cutover.