Data
AI
The Stack Underneath
Hello, we are BRILLIAN’S. Trying to make an effort to put the right people for you to get the best results. Just insight !
McKinsey surveyed roughly 450 CIOs and IT decision makers about their cloud migrations. 75% ran over budget. Just 7% came in under.
Those figures get quoted often enough that they have stopped landing. What they describe is a specific failure pattern, and it is rarely the transfer itself. Moving bytes between systems is a solved problem. The tools that do it have been production-grade for years. What breaks is almost everything decided before the transfer starts.
A legacy warehouse accumulates behaviour that nobody wrote down. Jobs that depend on file arrival order. Reports built on logic someone quietly duplicated years ago. Temporary fixes that hardened into business rules. None of it is visible while the old system keeps running. It becomes visible the week after cutover, when a finance dashboard returns a number nobody can explain.
Most of the work in a cloud data migration happens before any data moves. That holds for cloud data engineering generally, but migration is where the cost of skipping that work shows up fastest.
This guide covers the six strategies teams use to decide what moves and what stays, a nine-step process for running the migration itself, the cloud data migration tools worth evaluating this year, and the security controls that protect data while it is in transit. It assumes you are moving production data that people depend on. That assumption changes most of the advice.
Cloud data migration is the process of moving data from an existing storage system into cloud infrastructure, along with the schemas, pipelines, permissions, and downstream connections that make that data usable once it lands.
The distinction from cloud migration in the broader sense matters more than it first appears. Cloud migration covers servers, applications, networking, and everything else running in a data centre. Cloud data migration is the part concerned with the data estate itself. Same programme, different discipline, and the tooling barely overlaps.
Data migration to cloud usually takes one of three shapes.
On-premises to cloud: A warehouse, database, or file system moves from owned hardware into a managed cloud service. This is the most common shape and the one most guides assume.
Cloud to cloud: Data moves between providers or between regions. The drivers are usually cost, consolidation after an acquisition, or data residency rules that the current provider cannot satisfy.
Hybrid: Part of the estate moves and part stays. Latency requirements, licensing terms, or compliance constraints make a full move impractical, so the two environments run alongside each other.
Whichever shape applies, the scope is wider than most project plans account for. Alongside the historical and incremental data, a migration has to carry table schemas and data types that rarely map one to one across engines, the ETL and ELT jobs that populate those tables, access controls and masking policies, and every downstream system reading from the source.
That last item is where migrations quietly go wrong. The new environment looks complete, and a forgotten nightly job keeps writing to the legacy database for two weeks before anyone notices two sets of numbers that no longer agree.
The six strategies below come from a framework that was never designed for data. Gartner research director Richard Watson published five ways to migrate applications to the cloud around 2010, and AWS later expanded the set, with Stephen Orban adding Retain in 2016.
They still work for data once you translate them. A cloud migration strategy written for applications asks whether to rewrite the code. For a data estate, the question is whether to rewrite the model. Most migrations use three or four of these at once across different parts of the estate.
Lift and shift moves data into the cloud with its existing structure intact. An on-premises PostgreSQL database becomes a managed PostgreSQL instance, and the schema, model, and engine stay the same.
It is the fastest route and the cheapest to plan. It also moves your problems along with your data, since a workload that was inefficient on owned hardware stays inefficient in the cloud at metered rates. Plan the optimisation work as a separate project rather than expecting the move to deliver it.
Replatforming moves the data while making targeted changes that take advantage of the destination. The model stays recognisable, and specific components get swapped for managed equivalents.
In practice that means converting flat CSV extracts into columnar Parquet during transfer, adding partitioning the source never supported, or moving a self-managed SQL Server instance onto a managed service. Each change is contained enough to test on its own, which is why replatforming is the default choice for most production warehouses.
Refactoring rebuilds the data model and its pipelines for a cloud-native architecture. It is the most expensive option and the only one that removes structural constraints rather than relocating them.
The common version is abandoning a nightly batch process that transforms data before loading, and rebuilding around ELT where raw data lands first and transformation happens in the warehouse. Budget generously, because refactoring during a migration means two things can break at once and you will not immediately know which.
Repurchasing replaces the existing system with a commercial platform instead of migrating what you have. The data moves and the surrounding infrastructure gets retired rather than rebuilt.
Most repurchasing decisions involve dropping a self-managed cluster for a managed platform, which is why Snowflake and Databricks appear in so many migration business cases. Operational overhead disappears with the old system, and in exchange your data model now has to fit someone else's platform.
Retiring means the data does not move, because you delete it. Every large estate holds tables nobody has queried in three years, duplicated extracts, and archives kept because deleting them required a decision nobody wanted to own.
This is the highest-return strategy per hour spent and the most frequently skipped. Migrating redundant, obsolete, and trivial data is how compliance debt becomes permanent, since unclassified sensitive records arrive in the new environment with the same problems and a fresh retention clock. Classify before the migration. Volume you decide not to move is volume you never pay to transfer, store, or secure again.
Retaining keeps data where it is, either permanently or until the economics change. Most guides treat it as a footnote, and it deserves more than that.
The workloads that qualify are narrow but real. Steady storage with flat growth, predictable compute, and a team already running hardware competently. Dropbox and 37signals both moved large parts of their estates back off public cloud for roughly these reasons, and both published the arithmetic. Neither case argues against cloud migration. They argue against migrating without checking whether the workload actually benefits.
One date belongs in the plan. Under the EU Data Act, all switching charges including egress fees are prohibited for EU customers from 12 January 2027. Ordinary operational egress stays chargeable. For an EU business weighing a provider change, waiting may be worth more than any discount on offer today.
Nine steps, in order. The sequence matters more than the individual steps, because most migration failures trace back to work skipped early rather than mistakes made late. McKinsey found that teams finishing on time and on budget were markedly more likely to have committed to upfront investment and to have built security and compliance procedures before the move rather than during it.
Start by inventorying what exists, not what the documentation claims exists. Catalogue every table, its size, its growth rate, and its rate of change, then trace every downstream consumer reading from it.
The output that matters is the dependency map. Legacy systems accumulate jobs that depend on file arrival order and reports built on logic duplicated years ago, and none of it appears in a schema diagram. Budget properly for this step, because the hidden costs of cloud migration mostly live in what the assessment missed.
Write down what success means before choosing anything technical. Cost reduction, faster queries, regulatory compliance, and consolidation after an acquisition all lead to different architectures, and a migration optimised for one will disappoint on the others.
Requirements need numbers. Acceptable downtime, maximum tolerable data loss, query latency targets, and retention obligations all belong here. A data migration cost calculator helps set the budget expectation early, which is the number stakeholders remember.
Decide what the data lands in before deciding how it gets there. The choice between a cloud data warehouse and a lake or lakehouse design shapes every step that follows, from file formats to access control.
Match the architecture to the workload rather than the vendor pitch. The data lake and data warehouse comparison usually resolves along one line, which is whether your consumers need governed structure or flexible raw access. Many estates need both, and saying so early avoids retrofitting later.
Apply the strategies from the previous section per workload, not per project. A single migration commonly lifts and shifts the operational database, replatforms the warehouse, and retires a third of the archive.
Tool selection follows from that decision. Connector coverage, change data capture support, and how the tool handles schema drift matter more than the feature list, and the next section covers the options in detail.
Fix data quality before the move, because migration multiplies whatever you carry across. Duplicate records, inconsistent formats, and orphaned keys become harder to trace once they sit in a new system alongside new pipelines.
Gartner's widely quoted figure puts the average annual cost of poor data quality at $12.9 million, though it is worth knowing that number came from reference customers estimating their own losses rather than measured accounting. The direction is right even if the precision is generous. Running a data quality assessment against a defined data quality framework turns this step into something you can sign off rather than argue about.
Build the transfer mechanism as a real pipeline with logging, error handling, and restart capability. One-off scripts work until a transfer fails at hour nine and nobody can tell which records made it.
Handle historical and incremental data separately. Bulk load the history, then use change data capture to keep the target synchronised while the source stays live, which is what makes a low-downtime cutover possible. The same discipline that goes into building a data pipeline applies here, with the difference that this one runs once and has to be right.
Never let production cutover be the first end-to-end run. Full dry runs into a test environment surface mapping errors, performance problems, and data issues while there is still time to fix them.
Test at four levels. Completeness through row counts recorded at database, schema, and table level. Accuracy through field-level comparison on sampled records. Integrity through referential checks for orphaned relationships. Business validation through rules like invoice totals matching the sum of their line items, stored and rerun after the move.
Define rollback triggers before opening the migration window, not during it. Specific thresholds work better than judgement calls at 3am, so agree in advance on acceptable row count variance, error rates, and latency against production SLAs.
Take a verified snapshot of the source first as your recovery point, and record consumer group offsets for anything streaming. Wave-based execution beats a single cutover for large estates, since each wave teaches you something the next one uses. Migrate outside peak hours where the business allows it.
Reconcile before anyone gets access, and produce evidence rather than assurance. Matching aggregate row counts is not proof, so reconcile distinct entities, duplicate keys, orphaned relationships, and quarantined records separately.
Then watch cost and query performance for the first month, because both usually differ from the estimate. Repoint every downstream consumer deliberately and confirm each one, since a forgotten job still writing to the source is the most common way a completed migration turns into two divergent sets of numbers. Ongoing data governance is what keeps the new environment from accumulating the same undocumented behaviour you just spent months untangling.
Cloud data migration tools fall into two groups worth keeping separate. Native services from AWS, Azure, and Google are built for moving data into their own platform and tend to be cheapest and best integrated for that specific job. Platform-agnostic tools cost more and earn it when sources are varied, when the destination might change, or when the pipeline has to keep running after the migration ends.
One filter before the list. Most tool roundups for this keyword mix in server and application rehosting products, which solve a different problem. Everything below moves data.
AWS DMS migrates databases into AWS and supports both homogeneous and heterogeneous moves. Its main strength is change data capture, which keeps the target synchronised with a live source and makes low-downtime cutovers workable.
It runs replication instances inside your VPC and secures endpoint connections with SSL or TLS. Worth knowing that it defaults to an AWS-managed key unless you supply your own KMS key identifier. Performance can degrade at very large volumes, where sync lag between source and target becomes the constraint.
DataSync moves file and object data between on-premises storage and AWS, between AWS storage services, and between AWS and other public clouds. It handles the unstructured half of a migration that DMS does not touch.
AWS now points new customers here for online transfers, which makes it the default rather than an alternative. It supports customer-managed KMS keys, though you have to specify the full ARN.
Azure Data Factory is the workhorse for data movement into Azure and covers both migration and ongoing pipeline work. It handles a wide connector range and supports mapping data flows for transformation during transfer.
The overlap with migration-specific tooling is real, since Azure Database Migration Service uses Data Factory underneath for parts of its own data movement. Teams already invested in Azure usually end up here regardless of what they started with.
Azure DMS handles database migrations into Azure SQL and related targets, with schema migration and data migration as separate stages depending on the path. It pairs assessment tooling with execution.
The fit is clearest for organisations already holding Microsoft licensing and running Azure landing zones. SaaS-to-Azure data moves and unstructured content usually need something alongside it.
Google DMS focuses on database migrations into Cloud SQL and AlloyDB, with managed replication and minimal-downtime cutovers for MySQL, PostgreSQL, and SQL Server. Setup is light compared with the equivalents.
The scope is narrow by design. Storage, file, and SaaS content need separate tooling, which is the same pattern the AWS and Azure database services follow.
Storage Transfer Service is the managed option for moving large volumes of object and file data into Google Cloud Storage from online sources or on-premises infrastructure. It automates scheduling and handles scale without custom code.
For analytics-bound data there is a companion worth knowing. BigQuery Data Transfer Service ingests from Amazon S3, Azure Blob Storage, Redshift, Teradata, and Snowflake, and Google currently charges nothing for ingestion from those sources.
Fivetran is fully managed replication with automatic schema drift handling, which means connector maintenance stops being your team's problem. It suits organisations that would rather pay for convenience than staff the upkeep.
Security features matter for migration specifically. Column blocking excludes sensitive columns from replication, and column hashing protects PII before it reaches the target. Pricing is consumption-based and can climb faster than expected on high-change tables.
Airbyte is the open-source option, with a connector catalogue in the hundreds and the ability to build custom connectors for sources nothing else supports. It runs self-hosted or as a managed cloud service.
The trade is operational effort for flexibility and no vendor lock-in. Teams with engineering capacity get more control and pay less. Teams without it usually find the maintenance burden lands somewhere unhelpful.
Cloud migration data security is mostly your responsibility, not the provider's. Gartner projects that 99% of cloud security failures through 2026 will be the customer's fault, and the Cloud Security Alliance ranks misconfiguration and inadequate change control above zero-day attacks as the top cloud threat.
Migration widens the exposure temporarily. Data sits in two places, credentials get created for systems that never needed them, and staging buckets appear that nobody plans to keep. The six controls below cover that window.
Enforce TLS on every connection between source, migration service, and target. AWS DMS uses SSL or TLS for endpoint connections and encrypts the intermediate S3 bucket the replication task uses as data moves through the pipeline.
Check the intermediate stages specifically. Staging buckets and temporary landing zones are the parts of a transfer path that get configured quickly and reviewed last.
Encrypt the target before the first load rather than after, since retrofitting encryption onto a populated system usually means snapshot, copy, restore, and repoint.
Use customer-managed keys instead of service defaults where you need control over access. AWS DMS falls back to an AWS-managed key unless you pass a KMS key identifier, and that default gives you less say over who can decrypt.
Grant migration accounts the minimum they need and set an expiry date on them. Temporary elevated permissions created during a cutover have a habit of becoming permanent.
Overly permissive IAM roles and wildcard policies are among the most common misconfigurations found in cloud environments. Keep credentials in a managed secrets store rather than in pipeline configuration files, and audit what the migration tooling itself can reach.
Classify sensitive records before the transfer, because you cannot mask what you have not found. Migration is the cheapest opportunity you will get to apply that classification, since everything is already being touched.
The tooling supports it. AWS DMS can mask specified columns before loading them into the target, and Fivetran offers column blocking to exclude columns from replication and column hashing to protect PII before it reaches the destination.
Confirm where the data will physically sit before choosing a region, since residency obligations under GDPR and similar regimes apply to the target and to any intermediate staging location.
Retention rules travel with the data. Records under a legal hold or a defined retention period need those controls reproduced in the new environment, which is a straightforward task if it happens during planning and a difficult one afterwards. A clear data governance strategy makes this a checklist item rather than an investigation.
Log every migration action and keep the logs somewhere separate from the environment being migrated. Disabled logging is a routine finding in breach investigations, and it is usually disabled for convenience during a project exactly like this one.
Alert on the things that indicate a problem rather than everything. Failed authentication attempts, unexpected data volumes, and access from unfamiliar locations are the signals worth waking someone for.
The decision this guide keeps returning to is not which cloud to move to. It is which parts of your estate should move at all.
Most migrations that run over budget do so because that question got answered by default. Everything moved, including the tables nobody queries and the workloads that were cheaper where they were. Working through the six strategies per workload rather than per project is what turns a migration from a transfer exercise into a decision you can defend later.
Two things are worth acting on before the plan is finalised. Classify your data now, because the retiring decision is the highest-return work available and it only stays cheap while everything is already being touched. And check the currency of any tooling advice you are relying on, since the offline transfer landscape has changed materially in the last year and much of the published guidance has not caught up.
If the assessment step is where you expect this to get difficult, that is the correct instinct. Mapping undocumented dependencies across a live estate is the part that consumes the most time and the part teams most often underestimate. Our data engineering services cover that groundwork, from dependency mapping and quality remediation through to pipeline build and cutover.
Start with the inventory. The plan gets easier once you know what you actually have.
Anywhere from a few weeks to over a year, depending on data volume, the number of downstream systems, and how much of the estate needs remodelling rather than copying. The transfer itself is rarely the long part. Assessment, data cleaning, and testing usually consume more calendar time than the move, and skipping them is what turns a three-month project into a nine-month one.
Cloud migration covers everything moving out of a data centre, including servers, applications, and networking. Cloud data migration is the subset concerned with the data estate, meaning the databases, warehouses, files, pipelines, and access controls around them. The two run as part of the same programme but use different tools and different specialists.
Bulk load the historical data first, then use change data capture to keep the target synchronised while the source stays live. When the two are in sync, the cutover becomes a short window rather than an outage. Full zero downtime is rare in practice, so most teams agree an acceptable window in advance and design toward it.
Reconcile at four levels rather than one. Row counts recorded before and after confirm completeness, field-level comparison on sampled records confirms accuracy, referential checks catch orphaned relationships, and business rules like invoice totals matching their line items confirm the data still behaves correctly. Matching aggregate row counts on its own proves very little.
No. Workloads with steady, predictable demand and heavy storage requirements sometimes cost more in the cloud than on owned hardware, and some data cannot move for residency or licensing reasons. Deciding this per workload rather than per project is what keeps a migration business case honest.
You might also like
