Simplify Your Data Access at Scale with User-Assigned Managed Identities in Azure Data Factory

Introduction

When an organization uses Azure Data Factory (ADF) at scale and needs to connect to dozens—or even hundreds—of data resources, managing credentials becomes a significant challenge. Enter User-Assigned Managed Identities (UAMI): a feature of Azure that helps streamline authentication and reduce administrative overhead when granting your Data Factory access to services like Azure Data Lake Storage Gen2.

In the screenshot below, we see the Linked Service configuration in Azure Data Factory for a storage account. Notice that under Authentication type, we’ve chosen User-assigned managed identity rather than other options such as System-assigned managed identity or Service principal. In this post, we’ll walk through why you might choose a user-assigned managed identity and how it can help when you need to manage permissions for many different resources across your environment.

In the screenshot, you’ll see:

• Name: The Linked Service name (AzureDataLakeStorageADVADF)

• Connect via integration runtime: (AutoResolveIntegrationRuntime)

• Authentication type: User-assigned managed identity

• Account selection method: Set to Enter manually, indicating we’re providing the Data Lake Storage endpoint ourselves.

• URL: Points to the DFS endpoint of your Data Lake (datalakestorage.dfs.core.windows.net

• Credentials: A dropdown selection of user-assigned identities available in the ADF environment.

By selecting a User-Assigned Managed Identity in this Linked Service configuration, you ensure that ADF will use that specific identity when authenticating to your Data Lake—without storing any additional secrets or passwords in your Linked Service definition.

What Is a User-Assigned Managed Identity?

A User-Assigned Managed Identity is an identity you create as a standalone Azure resource. Unlike a System-Assigned Managed Identity—whose lifecycle is tied to the resource (e.g., an ADF instance, a VM, etc.)—a user-assigned identity is reusable across multiple services or resources. You create it once, then assign it wherever needed. This separation provides flexibility and a central place to manage permissions.

Why Use a User-Assigned Managed Identity, Especially at Scale?

1. Centralized Credential Management
   When you have multiple data services—Data Lake Storage, SQL Databases, Key Vaults—sprawled across many subscriptions or resource groups, a user-assigned identity provides a single “point of control.” You create that identity once, apply the necessary role assignments (e.g., Storage Blob Data Reader on your Data Lake), and then reuse that identity across different Data Factories or other Azure services.

2. Simplified Role Assignments
   Instead of creating separate identities for each resource or letting every Data Factory spawn its own identity, you manage permissions for a single user-assigned identity. This reduces duplication and confusion. If you have hundreds of data stores, you can systematically grant that identity the correct access roles—one time per store, rather than repeating the process for each new managed identity that might pop up.

3. Lifecycle Independence
   A system-assigned identity is deleted if you remove the resource to which it’s tied. By contrast, a user-assigned identity can persist independently. If you need to delete and recreate your Data Factory, that user-assigned identity (and its permissions) can be preserved, saving considerable re-work.

4. Enhanced Security with Fewer Secrets
   Managed identities eliminate the need for stored credentials. You don’t store user name/password combinations or client secrets. Instead, Azure securely handles token acquisition behind the scenes. This not only reduces the attack surface but also simplifies secret rotation.

5. Enterprise-Ready Governance
   In large enterprises, identity and access management is critical. Using a single user-assigned identity subject to corporate policies—rather than many ephemeral system identities—makes it easier for IT and security teams to audit who has access to what.

High-Level Setup Steps

Here’s a quick rundown of how you might configure a User-Assigned Managed Identity in Azure Data Factory to access Azure Data Lake Storage Gen2:

1. Create a User-Assigned Managed Identity

   • In the Azure Portal, search for “User Assigned Managed Identities.”

   • Create a new identity and give it a descriptive name that indicates its purpose (e.g., ADF-UAMI).

2. Assign Roles on Azure Data Lake Storage

   • Go to your Data Lake Storage account → Access Control (IAM) → Role assignments.

   • Add the new user-assigned identity as a role assignment, such as Storage Blob Data Reader

3. Assign the UAMI to Azure Data Factory

   • In the ADF → Manage → Access control (IAM) for your Data Factory resource, add the user-assigned identity to your ADF resource. Alternatively, from the ADF interface, create or edit a Linked Service and specify the user-assigned identity.

4. Reference the Identity in Your Linked Service

   • As shown in the screenshot, choose User-assigned managed identity as the Authentication type.

   • Enter manually the storage account endpoint and select the user-assigned identity from the Credentials dropdown.

5. Test Your Connection

   • Click Test connection to verify that Azure Data Factory can connect to the Data Lake using the user-assigned identity.

Wrapping Up

User-Assigned Managed Identities are a powerful and flexible way to handle authentication for your data workflows, especially when you have hundreds of resources to manage. By centralizing permissions, simplifying credential management, and decoupling identity lifecycles from the resources themselves, you save time and reduce the risk of misconfigurations.

As Azure Data Lake Storage and Data Factory deployments grow, harnessing user-assigned managed identities becomes an essential best practice for secure and scalable data engineering. It’s a small configuration change that delivers a big payoff in terms of manageability and security.

Give it a try in your own Azure environment—create one user-assigned identity, assign the required roles, and start connecting your resources. You’ll quickly see why more and more teams are adopting UAMIs for enterprise-scale deployments in the cloud.