From Chaos to Confidence: A Fabric Automation, Git & CI/CD Series
Why I'm turning my "Fabric Automation at Scale" session into a series... and what's coming
Fabric makes it wonderfully easy to just start building. That's one of its best qualities - and also how a tenant can quietly drift into a mess. You've probably seen a version of it: everything living in one place, all mixed together: notebooks, pipelines, lakehouses, semantic models and reports, every item type piled into the same workspaces. And piled in with them is every stage of life at once: the pipeline that runs the actual business, right next to a half-finished piece of new development, right next to an experiment someone abandoned six months ago. There's a scratch_anna, a TEST_do_not_delete, a POC_LH and a tmp_2023 that everyone is a little afraid to delete.
No source control, so no history and no way back. No dev, test or prod. Just prod, all the time. And everything gets done by clicking around in the portal, where every change is an act of faith.
I've spent a lot of the last few years helping teams climb out of exactly that hole, and I've done a number of sessions on it, all under the banner Fabric Automation at Scale: From Chaos to Confidence. A talk is great for the "aha," but it's a fire hose. You get 45-60 minutes, a stack of slides and a few demos, and then everyone goes home. So I'm doing what the talk can't: writing it down properly, one topic at a time, going deeper than a live demo ever allows. And backing all of it with working, open-source code you can actually run.
The journey
The whole thing follows one arc: From chaos to confidence. And confidence, in a data platform, isn't a feeling, it's a property you build in. It comes from four things stacked on top of each other:
Architecting for scale: A foundation that doesn't need ripping out the moment you grow.
Workspaces, repos and environments: A structure that mirrors that architecture and lives in Git.
CI/CD and source control in Fabric: Every change built, validated and promoted the same way, every time.
Automating your ways of working and deployment: Branch, review, validate, deploy. No ClickOps.
Each one only really works if the one below it is solid. Which is why we start at the bottom, with design.
FabricOps - The open-source companion
Everything in this series is anchored to my Git repository FabricOps. A Microsoft Fabric DevOps automation solution that shows these practices working end to end: IaC to provision (and tear down) workspaces, feature-workspace automation, BPA validation, and multi-stage deployment across dev/test/prod.
Let me be clear about what FabricOps is and isn't, because it matters for how you read this series. It is not a product, and it is not a full-blown end-to-end framework you're supposed to in full. It's a practical, open-source reference implementation. An accelerator. My honest advice: use it in full if it fits, lift the bits that are useful, let it inspire your own version, or just look and leave it. All four are correct answers. It's provided as inspiration and a boilerplate reference, at your own risk, and I'll keep evolving it as the series goes on, also including a few experiments I want to try out in the open.
The roadmap
Here's roughly where we're headed. I'll link each episode as it lands:
This intro: The why and the map.
Architecting for scale: How to design the platform, the layers, the reference architectures, and why there's no one-size-fits-all.
Repo structure & Git integration: How to organize your Git repository and do Git integration that your architecture.
Automating the setup (IaC): The different approaches (Terraform and friends), with a focus on FabricOps.
Operating the platform: Ways of working, collaboration, feature workspaces, testing and validation.
Deployment: Release patterns, promotion with gates, and getting to production safely.
…and beyond: A rolling thread of experiments, new patterns and best practices as Fabric (and FabricOps) evolve.
Start here
Enough preamble. The foundation comes first, so head straight to Episode 2: Architecting a Fabric Data Platform for Scale - Think Big, Start Small. That's where we get our hands dirty.
Stay tuned for more!
