Why a Platform Rebuild Stalls

A rebuild is approved because the current platform is expensive, brittle, or both. The business case is written against the end state: one warehouse, one semantic layer, one orchestration tool, lower run cost. The money is released. Eighteen months later the new platform exists, the old platform also exists, and nobody can name the date the second one switches off.

That is the stall. Not a failed build — a build that succeeded and never finished. If you own the platform and the budget, this is the failure mode worth designing against, because it is the one that is invisible on a status report right up until the run cost goes the wrong way.

What actually stalls?

The parallel run. Every rebuild has a period where both platforms are live: the old one because consumers still depend on it, the new one because you have to prove it before you cut over. The plan assumes this period is short and bounded. In practice it expands, and it expands for reasons that have nothing to do with engineering velocity.

It expands because reconciliation is harder than reproduction. Rebuilding a pipeline so it runs is a week. Proving its output matches the old one, for every partition, including the two quarters where someone patched a table by hand, is not a week. It expands because the last ten per cent of consumers are the ones you cannot find — a scheduled export into a finance spreadsheet, a report someone in operations refreshes manually, a downstream tool that authenticates as a service account nobody remembers creating.

And it expands because decommissioning is unfunded. The business case bought a new platform. Nobody was given a budget line, a name against it, or a deadline for turning the old one off. Shutting down a system is unglamorous, carries all the downside risk and none of the upside, and competes against new feature work every sprint. It loses that competition indefinitely unless you make it structurally impossible for it to lose.

Should the rebuild run in place or in parallel?

Both are defensible; they fail differently, and the choice should be made deliberately rather than inherited from whoever drew the first architecture diagram.

	Rebuild in place	Parallel platform
Cutover	Continuous and small	Discrete and large
Rollback	Hard — the old shape is gone	Easy — the old platform is still running
Cost shape	Flat, absorbed into BAU	Doubled while both run
Political risk	Low visibility, easy to defund quietly	High visibility, hard to defund, hard to finish
Typical stall	Refactor becomes permanent background work	Parallel run never ends

In-place refactoring stalls by dissolving into business as usual — real progress, never enough to declare. Parallel builds stall by never retiring the original. If you pick parallel, you are accepting a doubled run cost for a defined window, and the only thing that closes that window is a decommission plan with teeth.

What should the first slice be?

A vertical one, chosen for its consumers rather than its technical interest.

The instinct is to start with the hardest pipeline, on the theory that if you can do that you can do anything. The instinct is wrong. Start with a slice that goes all the way through — source ingestion, transformation, semantic definition, one report a named person looks at — and that has an owner who will accept the new version and let you turn off the old one. The point of the first slice is not to prove the technology. It is to prove the path to decommission, including the reconciliation evidence, the consumer sign-off and the actual switch-off. That path is where the unknown work lives.

Count progress in retired sources. A dashboard showing pipelines delivered is a vanity metric during a migration; it goes up whether or not the old platform is any closer to dark.

What should you buy rather than build?

Build-versus-buy turns on whether the capability is a differentiator or a commodity. During a rebuild, that question gets asked at the worst possible moment — when the team is deep in a tool’s limitations and feeling capable. Orchestration, catalogue, lineage capture, ingestion connectors and data quality frameworks all look buildable from inside a rebuild. They almost never repay the build once you price the years of upgrades, on-call and migrations that follow.

Build or buy this component?

• Does it change what a customer or regulator sees about you?
• Would owning it let you move faster on something that does?
• Can you staff its on-call, upgrades and migrations for years, not months?

Buy. It is a commodity; the integration is the real work.

The same logic applies to the platform itself. Whether you land on Microsoft Fabric, Databricks, Azure Synapse Analytics or a combination is far less consequential than whether your transformation logic is portable — which is the strongest argument for keeping modelling in something like dbt regardless of where it executes.

What can you safely defer?

More than the plan assumes. Deferring is the lever a platform owner actually controls, and using it early is what keeps the parallel run short.

Two items resist deferral. Metric definitions are one: a semantic layer centralises metric definitions so separate reports agree, and if you migrate reports before you settle definitions, you will migrate the disagreements with them and then be asked to explain why the new platform produces different numbers. Access control is the other — retrofitting row-level security after consumers are live means re-testing every report against every role, twice.

History is the most common overreach. Full backfills are expensive, slow, and frequently rebuild years nobody queries. Where the load is genuinely large, incremental refresh processes only changed partitions rather than a full dataset — but the prior question is whether the history needs to move at all, or whether the old platform can stay read-only for a defined retention window and then go.

What team shape gets it finished?

One accountable owner for the end state, and a separate, named owner for decommission. These should not be the same person, because the incentives differ: one is rewarded for shipping the new thing, the other only for the absence of the old thing.

The owner-of-the-end-state role is where a fractional Head of Data & AI earns its keep on a rebuild — it needs continuity of judgement and the authority to say no to scope, but not necessarily five days a week. The decommission owner needs something else entirely: access to the consumer list, the standing to chase people who do not report to them, and a deadline they are measured against.

Avoid splitting the work by technology layer. A team that owns ingestion and another that owns reporting will produce a rebuild that is complete in every layer and finished in none, because no single group can retire anything on their own. Split by slice, keep the slice narrow, and let each team carry its work all the way to a system that is switched off. That is the only definition of done that protects the business case you signed.