Architecting a Fabric Data Platform for Scale - Think Big, Start Small

This is Episode 2 of the From Chaos to Confidence series. In the intro I promised we'd start at the bottom, with the design, because everything else in the series stacks on top of it. So let's talk architecture.

I want to make one thing clear up front, because it shapes everything that follows: I'm not going to hand you a reference architecture to copy. There isn't a single correct one. What I can give you is a way of thinking about the problem, a few patterns that scale from tiny to enterprise, and a strong opinion about the one mistake that actually hurts - designing only for today.

The trap: starting simple with no way forward

Everybody starts simple, and there's nothing wrong with that. A notebook or two, a lakehouse, a report. All sitting together. It works. The problem isn't that it's small, the problem is that it's often built with no path forward. Everything mixed together, no separation between raw and curated data, nothing that resembles environments, no source control, names like Sales_FINAL_v2. Then the platform gets a little success, more people pile in, and suddenly you can't tell what's safe to change, you can't give one team access without giving them everything, and every deploy is a held breath.

Here's the metaphor I keep coming back to: you don't pour the foundation of a three-storey building and then decide to add thirty more floors. The foundation is the thing you can't cheaply change later. In a data platform, the foundation is your architecture. So how you separate concerns, how you name things, how work flows from raw to consumable, and whether automation and promotion were ever part of the plan. Get that roughly right and everything above it gets easier. Get it wrong and you're not iterating, you're rebuilding.

The goal, then, isn't complexity. The goal is optionality - keeping the door open to automation, CI/CD, multiple environments, governance, security and real team collaboration, even if you don't need all of it on day one.

Think in layers

Your architecture, at its heart, is a set of logical layers, the responsibilities in your platform and the boundaries between them. Get those right and everything downstream follows: how you version it, how you secure it, how you promote it. Get them wrong and no amount of tooling will save you.

So this episode is about the layers, and only the layers. How they turn into physical structures, and my strong view that your layers should be reflected 1:1 in the platform you build, is the subject of the next episode. How you replicate them across environments, we'll get to at the end of this post. For now, one question: what are the layers?

A quick word on names

Before I list mine, a disclaimer: the names don't really matter. What matters is that they're consistent and meaningful to your organisation. It's exactly like the medallion architecture with Bronze, Silver and Gold works, but so does Raw, Base, Curated, and honestly so would Underwear, Shirt, Jacket if your whole team genuinely understood it. Pick names, write them down, and use them everywhere.

I'll happily out myself here: in an earlier post I used Ingest, Prepare, Serve and Consume. These days I say Ingest, Prepare, Model and Present. The concepts didn't change, only my labels did. That's fine. Consistency over time matters more than picking the "perfect" word once.

With that said, here are the layers I use:

• Ingest: Get data in, as-is. Pipelines and notebooks landing raw data. No business logic here.

• Store: Where data lives across its lifecycle. Lakehouses holding the medallion flow: Landing, Base and Curated.

• Prepare: The transformation muscle. Notebooks turning Base into Curated, applying business logic and modelling rules.

• Model: Semantic models. The analytical contract between your data and your reports.

• Present: Reports, dashboards and apps. What the business actually opens.

• Orchestrate: The conductor. Pipelines that schedule and sequence everything above, and manage dependencies between layers.

• Core: Shared, cross-cutting building blocks: environments, functions, SQL databases for metadata, Spark job definitions and more.

So how granular should your layers be? That depends entirely on where you are. Your team, your security needs, your organisation, your ambitions. Let me show you four points on the spectrum.

Reference architecture: Minimum

If you're a small team, a single domain, low interdependency between features, and you value simplicity over isolation, start here. This is also my floor: I never go below three layers, because whatever else you collapse, Store always stands on its own. That's the one rule I don't bend on, everything else is negotiable.

• Store: Your lakehouse(s) and/or warehouses and the medallion layers.

• Engineering: Everything that moves and shapes data: ingest, prepare and orchestrate, folded together.

• Analytics: Semantic models plus reports, dashboards and apps.

Pros: simple to understand and explain, low overhead, fast to reason about.
Cons: limited isolation, coarse security boundaries (hard to give someone access to reports without also handing them the engineering), no natural separation of duties, and shared compute across everything.

Reference architecture: Basic

The most valuable split you can make after Store is pulling Analytics apart into Model and Present. It costs almost nothing and buys you a lot: your semantic models, the reusable analytical contract everything else is built on, get their own home, separate from the reports, dashboards and apps that consume them.

• Store: Unchanged, still standing on its own.

• Engineering: Still ingest, prepare and orchestrate, folded together.

• Model: Semantic models, isolated so they can be owned, secured and governed as the shared contract they are.

• Present: Reports, dashboards and apps, built on top of the models.

Pros: a clean boundary between the people who build models and the people who build reports. You can secure and govern the semantic layer independently of the (usually far more numerous) reports on top and still with low overhead and easy to reason about.
Cons: the engineering side is still coarse. Ingest, prepare and orchestrate share one home so there's little separation of duties there, and compute is still largely shared.

Reference architecture: Enterprise

As soon as you have multiple developers working in parallel, real security requirements, or you want to isolate compute between layers, the coarse split starts to hurt. This is where I land for most serious platforms: the full set of seven layers, each with a clear, single responsibility.

Notice this isn't a different architecture from Basic. It's the same layers, split further. Engineering became Ingest + Prepare + Orchestrate. Analytics became Model + Present. And we broke out Core for the shared plumbing.

Pros: clean separation of duties, fine-grained security and ownership boundaries, the ability to isolate compute so a heavy transformation can't starve your reports, and an architecture that lines up naturally with source control and deployment further down the line.
Cons: more parts to name, own and reason about and, importantly, this only stays sane if you automate it. A fine-grained architecture stood up and wired together by hand is its own kind of chaos. (Which is exactly why the rest of this series exists.)

Reference architecture: Enterprise+

Large organisation, multiple business domains, teams that own their own analytics, sensitive data that can't be co-mingled? Then you keep the shared engineering layers as they are, and you split Model and Present by domain: Finance, Sales, HR, each with its own model and presentation layer, all reading from the same curated store.

This gives each domain autonomy and clean boundaries over its own models and reports, while the engineering platform underneath stays shared and DRY. The cost is more surface to manage, and more thought about how the shared curated layer serves multiple owners. This is also where a dedicated Apps layer for delivery starts to make sense, but that's a topic for later.

The point of showing you four isn't to rank them. It's to show they're the same idea at different resolutions. You slide along the spectrum as your requirements demand. You don't leap to the end because a blog post told you to.

What actually pushes you along the spectrum

If you're trying to work out where you belong, don't count boxes. Look at your requirements. These are the questions that decide how granular your layers should be:

• Security & access: Who should see what, and can your current granularity enforce it?

• Separation of duties: Do engineers and report authors need different boundaries?

• Network & connectivity: Private endpoints, gateways, isolation needs?

• Capacity & compute isolation: Do you need to stop a heavy Spark job from starving your reports?

• Governance & compliance: Audit, ownership, sensitivity, data residency?

• Testing & deployment: Do you need to validate changes before they reach users?

• Discoverability: Can people still find things as the platform grows?

• Automation & CI/CD: Do your boundaries line up with how you'll want to branch, review and deploy?

Every "yes, and our current design can't do that cleanly" is a nudge to split one more layer out.

The axis nobody gets to skip: environments

You'll have noticed I didn't mention dev, test or prod once while describing those three architectures. That was deliberate because environments are not a tier you graduate into. They apply to every architecture on the spectrum, from Minimum to Enterprise+.

Whatever shape you land on, you replicate the whole thing across (at least) dev, test and prod. A three-layer Minimum platform gets a dev, a test and a prod. A seven-layer Enterprise platform gets exactly the same. The architecture you designed is the architecture you stamp out - once per environment.

Why this is non-negotiable, at any size:

• You need somewhere to build and break things without anyone noticing.

• You need somewhere to validate a change before real users meet it.

• You need a production environment you actively protect, rather than edit live and hope.

Skipping this is the single fastest route back to the chaos we opened with. And it's the clearest reason to take automation seriously from day one. Replicating an architecture across three environments and keeping them in sync by hand doesn't scale, even at Basic. Design the architecture once; know that it lives in triplicate.

The mechanics, how you promote a change safely from dev to test to prod, are a whole topic in their own right, and we'll go deep on them later in the series. For now it's enough to treat "×3 environments" as a given, whatever architecture you choose.

My recommendation

So what do I actually tell people?

1. Get your layers right first. Decide what responsibilities exist and where the boundaries between them sit. That is the architecture, and everything downstream depends on it.

2. Don't copy a diagram, copy the thinking. The right architecture is a function of your current state, skills, organisation and vision. A three-person team and a three-hundred-person enterprise should not have the same design, and neither should feel bad about that.

3. Bake the non-negotiables in from day one, even at Minimum: consistent and meaningful naming, a dedicated Store layer (never fewer than three layers), replication across dev/test/prod, and above all designing everything with automation in mind. These cost almost nothing early and are painful to retrofit.

4. Think big, start small. Start with a coarse layering if that's genuinely right for you. But name and structure your layers so that growing into a finer architecture later is a split, not a rewrite. That's the whole game.

Where FabricOps fits

If you want to see the enterprise pattern as running code, that's what FabricOps is. An open-source reference implementation of exactly this layered, multi-environment architecture. It's not a product and not a framework to adopt wholesale. It's an accelerator. Use it in full, take the pieces you like, get inspired, or just leave it, all four are correct answers.

I'm deliberately not getting into how to create any of this yet. Design comes first - setting it up comes next.

What's next

We've got the architecture on paper. The layers, how granular to make them, and the fact that it all lives across dev/test/prod. In Episode 3 we make it real and versionable: a Git repo whose structure mirrors these exact layers, and the principle I keep coming back to: That your architecture should be reflected 1:1 in the platform you build. That's where the design stops being a diagram and starts being something you can branch, review and deploy.

Stay tuned!