# Who's Calling? During Microsoft Fabric project implementations, I’m frequently asked a deceptively simple question: **“Under which identity is this running?”** It turns out, the answer isn’t always straightforward - and to be honest, it’s a topic I’ve also found quite complex at times. Just because a schedule was created by you doesn’t necessarily mean the entire job triggered by that schedule runs in your user context - or for that matter, in the context of the identity who created the item. And with the introduction of **Service Principal** support, things haven’t exactly become clearer. In fact, it often adds an extra layer of complexity to the already tricky landscape of execution context in Fabric. In this post, I want to share some of the insights I’ve gathered - especially when working with **data pipelines that trigger child notebooks and other downstream activities**. We’ll look at how identities are used across different components, what you need to be aware of, and how to avoid common pitfalls. Or in short: **Who’s calling?** 📞 Finally, I’ll touch on a **known bug** in the Fabric API and the **SemPy library** that affects **notebook execution in Service Principal contexts**, a setup that’s becoming increasingly common in enterprise-grade, multi-environment data platforms. ## Test Setup: Simulating Real-World Scheduling Scenarios To explore how execution context behaves in Microsoft Fabric, I created a simple but representative setup. Using the **Fabric CLI**, I triggered **on-demand executions** of Fabric items like **data pipelines** that call **child notebooks** as well as triggering notebooks directly. This setup allows us to control exactly **who initiates the run -** be it a **user** or a **Service Principal** - and observe how that identity flows (or doesn’t) through the various components. Key components of the setup: * A Data Pipeline with multiple activities (e.g., Invoke Notebook and Invoke Data Pipeline) * A Notebook which prints identity info as well as runtime properties and other relevant info * A parent Notebook which executes a child notebook (as the one above) * [Fabric CLI](https://learn.microsoft.com/en-us/rest/api/fabric/articles/fabric-command-line-interface)\-triggered job runs using both **user identity** and **Service Principal** This approach mimics many enterprise deployment scenarios, especially in **multi-environment setups**. ## Execution Scenarios: What Identity Is Actually Used? Regardless of whether a job is triggered by a **user** or a **Service Principal**, the same core logic applies when it comes to **execution context** in Microsoft Fabric. However, what happens next depends heavily on the **type of item** being executed and **how** it's executed. Let’s break it down… ### Top-Level Execution: Who Triggers the Job? When a pipeline or notebook is triggered - either manually, via schedule, or through a CLI/API call - the **top-level item** (the pipeline or notebook itself) is executed in the **context of the identity that triggered it**. That could be: * A user account (e.g., developer in dev/test) * A service principal (e.g., a scheduled run in production) So far, so good. But once you go deeper, into **child components and downstream activities**, the picture becomes more complicated. --- ### Notebook Execution from Notebooks When one notebook triggers another (e.g., using [`notebookutils.notebook.run`](http://mssparkutils.notebook.run)`()`), the **child notebooks** always inherit the **execution context of the parent notebook**. ✅ *If a notebook is triggered by a Service Principal, all downstream notebooks will run under the same Service Principal.* ✅ *If a user triggers the parent notebook, all child notebooks will run under that user’s identity.* This behavior is consistent and predictable across environments. --- ### Data Pipelines: A More Complex Story With **Data Pipelines**, execution context is **activity-specific**. Here’s what governs it: #### 🔹 Activities that use **connections** Examples: Copy Data, Invoke Pipeline (preview), Azure Databricks, Semantic model refresh, Web etc. These activities **run under the identity associated with the connection object** used. #### 🔹 Activities that **do not use connections** Examples: Notebook, Invoke Pipeline (Legacy activity), Dataflow, Spark Job Definition etc. These activities run under the identity of the user or service principal who **last modified** the pipeline. This is the identity shown as **"Last Modified By"** in the Data Pipeline settings. > ⚠️ Yes, that means if you last edited a pipeline in dev as yourself, but deploy it in test using a service principal, the execution identity in test will be the service principal - even if the original intent was to run it as a user. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1746795852549/b4148762-3299-4727-a499-20e96d7f7879.png align="center") ### Real-Life Example: A Lakehouse Medallion Architecture Let’s ground this in a practical scenario - a common **Lakehouse Data Platform** with a **3-layer medallion architecture**: 1. A **controller pipeline** kicks off the process. 2. It calls **child pipelines** that ingest raw data into the **bronze** layer. 3. Then it triggers a **notebook** that processes bronze into **silver**. 4. Another notebook handles transformations into **gold** (curated data). 5. Finally, the pipeline refreshes a **semantic model** as the last step. Here’s how execution context breaks down: * Activities using **connections** (e.g., Copy Data or Semantic model refresh) run under the **connection identity**. * Notebooks in the pipeline (with no connection) run as the **last modified identity of the pipeline -** which could be a user or service principal. * If a child pipeline triggers a notebook, the same logic applies: the **last modified identity of that pipeline** determines the execution context of its notebook. So yes, it’s entirely possible that a single run involves: * Data ingestion as one identity (connection) * Silver transformation as another (pipeline author) * Gold orchestration as yet another (child pipeline modifier) --- ### Feeling Lost? You’re Not Alone If you’re scratching your head, you’re not alone. The behavior is by design, but it does mean we need to be **deliberate** about how we: * Modify items * Manage dependencies downstream * Set up connections * Deploy across environments Most importantly: **how things run in development may not reflect how they run in test or production** - especially if you use a service principal for automated deployments. That’s why **understanding execution context is critical** for ensuring consistent behavior across environments in enterprise-grade solutions. ## Known Bug: When Notebooks Fail Under a Service Principal While building enterprise-ready Fabric solutions, it’s increasingly common to run notebooks using **Service Principals**. However, there's a **known bug** that can cause unexpected failures when doing so. ### What’s the Problem? Running a notebook under a Service Principal can break certain functions and environment references, especially those related to **runtime context** and **authentication**. The issue appears to stem from the **scope or limitations of the Service Principal's token**, and Microsoft has acknowledged it as a **bug**. The Fabric product team is actively working on a fix. ### What Fails? Here’s a list of some of the functions and methods that return `None` or throw errors when executed in a notebook under a Service Principal. Note that mssparkutils is going to be deprecated, notebookutils is the way to go. This is just to illustrate the issue: * `mssparkutils.env.getWorkspaceName()` * `mssparkutils.env.getUserName()` * `notebookutils.runtime.context.get('currentWorkspaceName')` * `fabric.resolve_workspace_id()` * `fabric.resolve_workspace_name()` * Any SemPy `FabricRestClient` operations * Manual API calls using tokens from `notebookutils.mssparkutils.credentials.getToken("`[`https://api.fabric.microsoft.com`](https://api.fabric.microsoft.com)`")` ### ⚠️ Importing `sempy.fabric` Under a Service Principal When executing a notebook in the context of a **Service Principal**, simply importing `sempy.fabric` will result in the following exception: ```plaintext Exception: Fetch cluster details returns 401:b'' ## Not In PBI Synapse Platform ## ``` This error occurs because **SemPy** attempts to fetch cluster and workspace metadata using the **execution identity’s token** - which, as mentioned earlier, lacks proper context or scope when it belongs to a Service Principal. In short, **any method that fetches workspace name** **or user name -** or relies on the **executing identity’s token for SemPy** or **REST API calls** - is likely to fail or return `None`. ### What Still Works? Surprisingly, not everything is broken. Here are some functions that still work under a Service Principal: * `spark.conf.get('`[`trident.workspace.id`](http://trident.workspace.id)`')` – this gives you the workspace ID reliably * `sempy.fabric.get_workspace_id()` – still functional, eventhough importing `sempy.fabric` will throw an exception as shown above. * `notebookutils.credentials.getSecret(...)` – useful for pulling secrets like client credentials from a Key Vault Using these, you can still **manually generate a token** and pass it into your REST requests - or even inject a custom `token_provider` into the SemPy `FabricRestClient`. ### Workarounds If you hit this issue, here are some paths forward: * Avoid relying on runtime context methods when running under a Service Principal * Use a **manual token approach**: fetch your own token using credentials from Key Vault and use that in REST requests * Where possible, **shift context resolution logic out of notebooks** and into deployment orchestration or pipeline steps * Watch for updates: Microsoft is aware of the issue and a fix is on the way ## Why This Bug Matters for CI/CD and Execution Context This issue ties directly back to the core topic of this blog post - **execution context in Microsoft Fabric**. Remember that when a **notebook is triggered by a Data Pipeline**, its execution identity depends on **who last modified the data pipeline**. In modern CI/CD workflows - whether you're using **Azure DevOps Pipelines**, **GitHub Actions**, or any other automation platform - you’re most likely deploying with a **Service Principal**. That means after every deployment, **the "Last Modified By" identity on your Data Pipelines becomes the Service Principal**. This wouldn’t be an issue *if* notebooks worked reliably under Service Principal identity. But as we've seen above, **notebooks run into serious limitations when executed in that context** - missing environment properties, failed API calls, and broken logic in dynamic configurations. ### A Practical Workaround: Let a Web Activity Re-Assign Ownership Here’s one way to get around it: Use a **Web activity in a Fabric Pipeline** - configured with an **OAuth2 connection for a specific user -** to **update the description** of your Data Pipelines post-deployment. Why this works: * A Web activity executes in the context of the **connection identity** * Updating the pipeline’s description (even just reapplying the same description) is enough to change the **"Last Modified By"** property * As a result, **all notebooks executed by those pipelines will now run in the context of the user tied to the OAuth2 connection**, not the Service Principal This allows you to: * Deploy pipelines automatically with a Service Principal * Then post-process them to **re-assign their execution identity to a user**, for scenarios where notebook behavior matters This approach also allows you to apply filters to target only specific Data Pipelines, updating the **Last Modified By** property selectively. This way, you can still support notebook execution under a Service Principal where needed. ### Pipeline Template: Available on GitHub You can see a visual of this post-deployment ownership adjustment pipeline below. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1746788273598/de8f1815-9c60-4ef0-9515-65f9ee0aada9.png align="center") I’ve also published the **pipeline definition** on my GitHub including a short description on how to use the 2 parameters: [View on GitHub](https://github.com/gronnerup/Fabric/tree/main/FabricExecutionContext) > **Note:** All activities in the definition are currently **disabled by default** so you can safely copy-paste it into your own Fabric Data Pipeline json definition and adjust the **connection settings**, **pipeline selection logic** etc. as needed.