Jed Laundry

Note: Most of this article is purely by observation, and may or may not be correct. If a Microsoft product manager would like to correct me, you can find me in the Private Preview Teams groups 😁

What are Managed Private Endpoints?

Data infrastructure is complex. Modern best practices relies on ephemeral compute infrastructure, which scales up and down quickly in response to demand.

Building this infrastructure well is not easy, and many organisations would rather spend the time focusing on more core business objectives. As a result, over the last few years, data pipeline Platform-as-a-Service offerings have become very popular, as it allows teams to apply modern practices without needing to understand the underlying infrastructure.

The way that Microsoft have designed various Azure data pipeline services is to build and run compute components in Microsoft-managed Subscriptions, parallel to the actual customer tenancies. At the end of the day, it's still software running on VMs; but these VMs don't (and shouldn't) appear in the customer tenancy.

You can either attach these pipeline services to your own Virtual Networks, or use Managed Virtual Networks (Managed VNets). The intention of Managed VNets is exactly as it sounds: in addition to managing the compute infrastructure, Microsoft will also manage the network infrastructure needed to make the compute useful. This includes surprisingly simple things--like having subnets big enough to handle the large number of ephemeral compute, and making sure network ACLs prevent data exfil--which teams can often struggle to manage properly.

Managed VNets are currently available for:

• Azure Synapse
• Azure Data Factory
• Azure Machine Learning
• Microsoft Graph Data Connect

But, as this compute needs to connect to data in customer environments, including data not normally available on the public Internet, Microsoft also provide Managed Private Endpoints as a way to connect to these resources, such as Storage Accounts, KeyVaults, Cosmos, and SQL Databases; without needing to make these resources available over the public internet.

The problem(s) with Managed Private Endpoints

Apart from issues around additional wait time costs (check out Martin's blog for an excellent illustration of this), the biggest issue with Managed VNets and Managed Private Endpoints is that there is very little documentation or understanding about them. Many experienced Azure professionals struggle to explain what they are, how to identify them, how to troubleshoot them, or what needs to be done to manage them.

For example, in order to know that your Azure Data Factory is using Managed Private Endpoints, you need to view the configuration from inside the Studio, as nothing appears in the Azure Portal.

Unlike a normal Private Endpoint, it won't appear in the Resource Group:

Identifying Managed Private Endpoints

You'll know if you're using Managed Private Endpoints, if:

• You see connections to Storage, KeyVault, Cosmos, and SQL Databases from private IP addresses (that you haven't already allocated), particularly in the 10.0.0.0/16 range.
• When you try to click on a Private Endpoint in the Azure Portal, and you get a 401 No access error with the message "The access token is from the wrong issuer 'https://sts.windows.net/.../'. It must match the tenant 'https://sts.windows.net/33e01921-4d64-4f8c-a055-5bdaffd5e33d/' associated with this subscription.", this is a Managed Private Endpoint. 33e01921-4d64-4f8c-a055-5bdaffd5e33d is the Tenant ID associated with Azure managed infrastructure (MSAzureCloud.onmicrosoft.com).

Unlike traditional Private Endpoints, Managed Private Endpoints will not show up as resources in your tenancy. The only way to spot them is on each resource; by looking at the Private Endpoint Connection attributes of each resource, and check the IDs for subscriptions that aren't yours.

To find where Managed Private Endpoints are connected to your Azure resources, you can use the below Azure Resource Graph query:

resources
| where isnotnull(properties) and properties contains "privateEndpointConnections"
| where array_length(properties.privateEndpointConnections) > 0
| mv-expand properties.privateEndpointConnections
| extend peResourceId = properties_privateEndpointConnections.properties.privateEndpoint.id
| extend peSubscriptionId = substring(peResourceId, 0, 51)
// would prefer to use a let table, but 'Nested subqueries are not allowed.'
| join kind=leftouter (resourcecontainers | where type == "microsoft.resources/subscriptions" | project id) on $left.peSubscriptionId == $right.id
// This is ick, but 'The join kind "LeftAntiSemi" is not supported or not allowed. (Code:UnsupportedJoinFlavor)'
| where id1 == ""
| project name, id, location, type, peSubscriptionId, peResourceId, tags
| order by ['id'] asc