Automating Microsoft Fabric: Private Endpoint Setup in workspaces

In an exciting development, Microsoft Fabric just announced support for APIs dedicated to managing private endpoints, a crucial feature for organizations prioritizing secure and private data access. Building on my previous posts on automating Fabric workspaces and lakehouses and leveraging Fabric REST APIs, I’ll guide you through automating the creation of managed private endpoints within your Fabric workspaces. In this post, I’ll cover not only how to set up these private connections but also how to streamline approvals via Azure management APIs, if permitted in your environment.

Find the official blog post from Microsoft on APIs for Managed Private Endpoints here: https://blog.fabric.microsoft.com/en-US/blog/apis-for-managed-private-endpoint-are-now-available/

Previous Approach to Automating Managed Private Endpoint Creation

Before official API support for managed private endpoints was available in Microsoft Fabric, our approach relied on using Fabric's internal, undocumented APIs. To automate endpoint creation within a workspace, I would send a POST request to:

https://wabi-north-europe-j-primary-redirect.analysis.windows.net/metadata/workspaces/00000000-0000-0000-0000-000000000000/privateEndpoints

And with the following JSON payload:

{
   "name":"my-private-endpoint",
   "requestMessage":"Auto-generated managed private endpoint",
   "privateLinkResourceId":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-peerinsights-dev/providers/Microsoft.KeyVault/vaults/kv-peerinsights-dev",
   "groupId":"00000000-0000-0000-0000-000000000000"
}

While effective, this approach was less than ideal - it depended on an unsupported API and allowed only user identity for authentication, not service principals or managed identities.

With the recent additions to the Fabric APIs, creating managed private endpoints can now be achieved through officially supported, documented endpoints. Even better, service principal authentication is now supported, offering a more secure and scalable way to automate private endpoint management.

Adding Managed Private Endpoints with Fabric APIs

Building upon my previous blog post on automating your Fabric environment setup, I’ve enhanced the helper functions notebook to support the creation and management of managed private endpoints, including handling the long-running nature of the setup process.

In the fabric_functions.py script, I added a few key functions to streamline this process. Two of the most critical functions are:

• create_workspace_managed_private_endpoint: This function automates the creation of a managed private endpoint within a Microsoft Fabric workspace, monitoring its provisioning status until fully completed.

• approve_private_endpoint: This function automates the approval of a private endpoint connection within Azure, updating its status to "Approved" through an API request.

To integrate this functionality, I extended the staging recipe used in the workspace setup to include private endpoints that should be created and, if desired, automatically approved. Here’s an example of the updated fabric_stages configuration:

fabric_stages = {
    "Prepare": {
        "private_endpoints": [
            {
                "name": "mpe-kv-peerinsights-dev",
                "auto_approve": True,
                "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-peerinsights-dev/providers/Microsoft.KeyVault/vaults/kv-peerinsights-dev"
            }
        ]
    }
}

With this new functionality, private endpoints can be easily integrated into the Fabric setup process. And by using the auto_approve property in the private endpoint definition, we can direct our setup to automatically approve the newly created endpoint. Here’s how it works:

if not stage_props.get("private_endpoints") is None:
    for private_endpoint in stage_props.get("private_endpoints"):
        fabfunc.create_workspace_managed_private_endpoint(
            fabric_access_token, workspace_id, private_endpoint.get("name"), private_endpoint.get("id")
        )
        if private_endpoint.get("auto_approve"):
            connection_name = f"{workspace_id}.{private_endpoint.get('name')}-conn"
            management_access_token = fabfunc.get_access_token(tenant_id, app_id, app_secret, 'https://management.core.windows.net')
            fabfunc.approve_private_endpoint(
                management_access_token, private_endpoint.get("id"), connection_name
            )

And the result…

With this approach, managed private endpoints can now be included as an integrated part of the Fabric setup, ensuring a smooth and automated deployment from start to finish.

Conclusion

Every Fabric API update brings us closer to fully automating and streamlining data platform workflows, steadily checking off my 'must-have' features list—big kudos to the Fabric team!

I’ll keep sharing insights on automating Microsoft Fabric, so stay tuned for more from Peer Insights! As a sneak peek, I’ll be exploring ways of working within Fabric to simplify the setup of feature development workspaces and more.

You can download the enhanced notebooks, now supporting managed private endpoint setup, here: GitHub - FabricSolutionInit.