Multi-Tenant CI/CD on Azure DevOps with Workload Identity Federation and Bicep

The Problem with Client Secrets in Pipelines

If you've managed Azure DevOps pipelines for any length of time, you've dealt with this: a deployment fails at 2 AM because a service principal secret expired. Someone had set it to 90 days, rotation never got automated, and now a critical release is blocked while someone scrambles to generate a new secret and update the pipeline variable.

Client secrets in CI/CD are a systemic problem. They have to be stored somewhere - usually as pipeline variables or library secrets - which means they're one misconfigured permission away from exposure. They expire. They get copy-pasted into the wrong place. In multi-tenant environments, the problem compounds: you end up managing separate secrets per tenant, per environment, per service principal, all with different expiry dates.

There's a better way. Workload Identity Federation (WIF) lets Azure DevOps authenticate to Azure using a short-lived federated token instead of a long-lived secret. No secrets to store, no rotation schedules, no expiry panic.

What Is Workload Identity Federation?

Workload Identity Federation is an OpenID Connect (OIDC) based mechanism. Instead of a client secret, your pipeline presents a signed JWT issued by Azure DevOps. Azure's identity platform validates that JWT against a pre-configured trust relationship on the App Registration - then issues an access token. The whole exchange happens in milliseconds, entirely in-memory, with no credentials persisted anywhere.

The federated trust is configured on the App Registration as a federated credential - essentially telling Entra ID: "if you receive a token from Azure DevOps org marlin-devops claiming to be pipeline X, trust it." Each tenant gets its own federated credential pointing to the same DevOps org.

Architecture Overview

One shared service principal authenticates into all tenants via federated credentials - no client secrets anywhere.

The key design decision here is one shared service principal across all tenants, rather than one SP per tenant. This simplifies pipeline YAML (single service connection reference), centralizes audit logs, and reduces the number of App Registrations you have to track and govern. Each tenant still gets its own federated credential and its own RBAC assignments - the sharing happens at the identity layer, not the permissions layer.

Prerequisites

• Azure DevOps organization (marlin-devops in this example) with Project Administrator rights
• Entra ID Global Administrator or Application Administrator in the home tenant (where the App Registration lives)
• Privileged Role Administrator or Owner in each target tenant (to assign RBAC to the guest SP)
• Azure CLI installed locally for the setup steps
• Bicep CLI (or use the Azure CLI built-in transpiler)

Step 1: Create the Shared Service Principal

Create the App Registration in your home tenant. This is the identity that all pipelines will authenticate as:

# Create the App Registration
az ad app create \
  --display-name "sp-marlin-multitenant-cicd" \
  --sign-in-audience AzureADMultipleOrgs

# Note the appId from the output - you'll need it throughout
APP_ID=$(az ad app list --display-name "sp-marlin-multitenant-cicd" --query "[0].appId" -o tsv)

# Create the service principal in the home tenant
az ad sp create --id $APP_ID

echo "App ID: $APP_ID"

Do not create a client secret. You won't need one - that's the whole point.

Step 2: Configure Federated Credentials

For each target tenant (and environment combination), add a federated credential to the App Registration. The credential tells Entra ID which Azure DevOps pipelines are allowed to authenticate as this SP.

# Create federated credential for Marlin Tenant 1 sandbox pipelines
az ad app federated-credential create \
  --id $APP_ID \
  --parameters '{
    "name": "marlin-t1-sandbox-pipelines",
    "issuer": "https://vstoken.dev.azure.com/YOUR-DEVOPS-ORG-ID",
    "subject": "sc://marlin-devops/YOUR-PROJECT-NAME/marlin-t1-sandbox-deploy",
    "description": "Marlin Tenant 1 sandbox deployment pipelines",
    "audiences": ["api://AzureADTokenExchange"]
  }'

The subject field is the scope of the trust. The format for an Azure DevOps service connection is:

sc://<org-name>/<project-name>/<service-connection-name>

You can also scope the trust to a specific pipeline, branch, or environment by using the pipeline-level subject format instead. For a shared multi-team setup, service connection scoping is usually the right balance between security and practicality.

Repeat this step for each additional tenant or environment, using a distinct name per credential. One App Registration can hold up to 20 federated credentials.

Step 3: Grant RBAC Across Tenants

In each resource tenant, you need to invite the service principal and assign it the right roles. Since the SP was created as a multi-tenant app, it can be instantiated in any Entra ID tenant simply by consenting to it.

In each resource tenant:

# Switch context to the resource tenant
az login --tenant RESOURCE-TENANT-ID

# Instantiate the SP in this tenant (first-time only)
az ad sp create --id $APP_ID

# Assign Contributor at subscription scope
az role assignment create \
  --assignee $APP_ID \
  --role "Contributor" \
  --scope "/subscriptions/TARGET-SUBSCRIPTION-ID"

# If pipelines need to create role assignments (e.g., for managed identities):
az role assignment create \
  --assignee $APP_ID \
  --role "User Access Administrator" \
  --scope "/subscriptions/TARGET-SUBSCRIPTION-ID" \
  --condition "@Resource[Microsoft.Authorization/roleAssignments]:RoleDefinitionId ForAnyOfAllValues:GuidEquals {ALLOWED-ROLE-GUIDS}" \
  --condition-version "2.0"

Step 4: Create the Service Connection in Azure DevOps

In Azure DevOps, create a service connection that references the App Registration and uses Workload Identity Federation as the authentication method.

1. Go to Project Settings → Service connections → New service connection
2. Choose Azure Resource Manager
3. Select Workload Identity Federation (manual)
4. Enter your Tenant ID, Subscription ID, and App ID (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
5. Name the connection to match what you put in the federated credential subject (e.g., marlin-t1-sandbox-deploy)
6. Verify the connection

For multi-tenant scenarios, create one service connection per target tenant subscription, all pointing to the same App ID but with different Tenant IDs and Subscription IDs.

Step 5: Write the Bicep Templates

With the identity side sorted, the Bicep templates are straightforward. Here's a typical structure for a sandbox subscription deployment:

// main.bicep - subscription-scoped deployment
targetScope = 'subscription'

param environment string
param location string = 'canadacentral'
param ownerEmail string
param expiryDate string

// Core resource group
resource sandboxRg 'Microsoft.Resources/resourceGroups@2023-07-01' = {
  name: 'rg-sandbox-${environment}'
  location: location
  tags: {
    environment: environment
    'owner-email': ownerEmail
    'expiry-date': expiryDate
    'managed-by': 'marlin-cicd'
  }
}

// Deploy networking module into the RG
module networking './modules/networking.bicep' = {
  name: 'networking'
  scope: sandboxRg
  params: {
    environment: environment
    location: location
  }
}

// Deploy RBAC assignments
module rbac './modules/rbac.bicep' = {
  name: 'rbac'
  scope: sandboxRg
  params: {
    ownerEmail: ownerEmail
  }
  dependsOn: [networking]
}

The networking module provisions a VNet and subnets with sensible defaults for sandbox workloads:

// modules/networking.bicep
param environment string
param location string

resource vnet 'Microsoft.Network/virtualNetworks@2023-09-01' = {
  name: 'vnet-sandbox-${environment}'
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: ['10.0.0.0/16']
    }
    subnets: [
      {
        name: 'snet-workloads'
        properties: {
          addressPrefix: '10.0.1.0/24'
          privateEndpointNetworkPolicies: 'Disabled'
        }
      }
      {
        name: 'snet-management'
        properties: {
          addressPrefix: '10.0.2.0/24'
        }
      }
    ]
  }
}

output vnetId string = vnet.id

Step 6: Build the Pipeline YAML

The pipeline YAML ties everything together. The key is the azureSubscription reference in the AzureCLI task, which is the service connection name - that's all you need for WIF authentication to kick in:

trigger:
  branches:
    include:
      - main
  paths:
    include:
      - infra/**

parameters:
  - name: environment
    displayName: Environment Name
    type: string
  - name: ownerEmail
    displayName: Owner Email
    type: string
  - name: expiryDate
    displayName: Expiry Date (YYYY-MM-DD)
    type: string
  - name: targetTenant
    displayName: Target Tenant
    type: string
    values:
      - marlin-tenant-1
      - marlin-tenant-2
      - marlin-tenant-3

variables:
  - name: serviceConnection
    ${{ if eq(parameters.targetTenant, 'marlin-tenant-1') }}:
      value: marlin-t1-sandbox-deploy
    ${{ if eq(parameters.targetTenant, 'marlin-tenant-2') }}:
      value: marlin-t2-sandbox-deploy
    ${{ if eq(parameters.targetTenant, 'marlin-tenant-3') }}:
      value: marlin-t3-sandbox-deploy

stages:
  - stage: Validate
    displayName: Validate Bicep
    jobs:
      - job: Lint
        pool:
          vmImage: ubuntu-latest
        steps:
          - task: AzureCLI@2
            displayName: Bicep lint + what-if
            inputs:
              azureSubscription: $(serviceConnection)
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: |
                az bicep build --file infra/main.bicep
                az deployment sub what-if \
                  --location canadacentral \
                  --template-file infra/main.bicep \
                  --parameters \
                    environment=${{ parameters.environment }} \
                    ownerEmail=${{ parameters.ownerEmail }} \
                    expiryDate=${{ parameters.expiryDate }}

  - stage: Deploy
    displayName: Deploy Infrastructure
    dependsOn: Validate
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeploySandbox
        displayName: Deploy Sandbox
        pool:
          vmImage: ubuntu-latest
        environment: sandbox-${{ parameters.environment }}
        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: self
                - task: AzureCLI@2
                  displayName: Deploy Bicep
                  inputs:
                    azureSubscription: $(serviceConnection)
                    scriptType: bash
                    scriptLocation: inlineScript
                    inlineScript: |
                      az deployment sub create \
                        --location canadacentral \
                        --template-file infra/main.bicep \
                        --parameters \
                          environment=${{ parameters.environment }} \
                          ownerEmail=${{ parameters.ownerEmail }} \
                          expiryDate=${{ parameters.expiryDate }} \
                        --name "deploy-$(Build.BuildId)"

Real-World Gotchas

1. The federated credential subject has to match exactly

The subject claim in the OIDC token issued by Azure DevOps must match the subject field on the federated credential precisely. If you rename your service connection or project, the trust breaks immediately. Treat these names as immutable after setup, or update the federated credential when they change.

2. Guest SP propagation takes a few minutes

After running az ad sp create --id $APP_ID in a resource tenant, there's typically a 2-5 minute replication delay before the SP is available for RBAC assignments. If you're scripting tenant onboarding, add a sleep 120 or poll until the SP appears rather than failing silently.

3. Subscription-scoped Bicep deployments need a location

When deploying at subscription scope (rather than resource group scope), the az deployment sub create command requires a --location parameter. This is the metadata region for the deployment object itself, not necessarily where your resources will land. Use canadacentral or whichever region your org has approved for control plane operations.

4. WIF doesn't work with az login in scripts

Inside an AzureCLI@2 task with a WIF service connection, the Azure CLI is already authenticated - Azure DevOps handles the OIDC exchange before your script runs. If your script tries to call az login explicitly (a common copy-paste habit), it will break the session. Remove any explicit login calls from scripts running in authenticated task contexts.

5. Bicep what-if is optimistic about RBAC changes

The az deployment sub what-if command often reports role assignments as "no change" even when they will be created or modified. Don't rely on what-if alone to validate RBAC sections of your templates - always do a targeted test deployment to a scratch subscription first.

Security Considerations

WIF significantly improves the security posture of your pipelines, but it doesn't eliminate all risk. A few things worth building into your setup:

• Scope federated credentials as tightly as possible. If only one pipeline in one project should be able to authenticate as this SP, use the pipeline-level subject format rather than the service connection level.
• Review the SP's role assignments regularly. In a multi-tenant environment, it's easy for Contributor assignments to accumulate. Set a quarterly review cadence using Azure Policy or a scheduled query against the ARM API.
• Enable Conditional Access for workload identities in tenants that support it. This lets you restrict the SP to specific IP ranges, named locations, or require continuous access evaluation.
• Use Azure DevOps Environments with approval gates on any pipeline stage that touches non-sandbox resources. WIF makes authentication seamless - don't let that seamlessness remove human oversight from impactful changes.
• Monitor sign-in logs in each tenant. The service principal's sign-in logs will show every pipeline run that authenticated through WIF, including the OIDC claims. This is significantly more auditable than a client secret which shows up as a single authentication event with no pipeline context.

Where to Take This Next

This pattern is a foundation. Here's how to extend it:

• Automated tenant onboarding - wrap the SP instantiation, RBAC assignment, and federated credential creation in a pipeline that runs when a new tenant is added to a config file. New tenant = one PR, pipeline handles the rest.
• Drift detection - add a scheduled pipeline that runs az deployment sub what-if against all managed subscriptions nightly and posts a Teams notification if drift is detected.
• Policy-as-code integration - extend the Bicep templates to deploy Azure Policy assignments at subscription scope using the same SP. Since it already has Contributor, a targeted Policy Contributor role assignment at the subscription scope is all that's needed.
• Environment-specific parameter files - use main.bicepparam files per environment instead of inline parameters in the YAML, and commit them to the repo. This makes configuration reviewable as code.
• Deployment stacks - replace az deployment sub create with az stack sub create to get deny assignments and managed resource tracking for free. Stacks make it easy to tear down a sandbox subscription completely without leaving orphaned resources.