How to publish Bicep Modules to ACR

155
16 min to read

Mastering the Publishing Process for Azure Bicep Modules to Azure Container Registry

In my previous post, Test Bicep Modules using DevOps Pipeline for Azure Bicep Registry, I walked you through creating a structured pipeline for validating and deploying Azure Bicep modules. In this follow-up, I’ll dive deep into publishing Azure Bicep modules to Azure Container Registry (ACR) - a critical step in streamlining Infrastructure as Code (IaC) workflows.

If you’re like me, you want your processes to be automated, efficient, and scalable. Publishing Bicep modules to ACR ensures consistency, traceability, and collaboration across teams and projects. This guide will show you how to set up an automated pipeline for publishing your modules, while tackling common challenges and sharing tips from my experience.

Why Should You Publish Bicep Modules to ACR?

If you’re managing IaC workflows, Azure Container Registry (ACR) is a game-changer. It’s a secure, centralized repository that allows you to store, version, and manage reusable Bicep modules. Here’s why people love using ACR:

  • Reusability: Modules in ACR can be shared across teams and environments, saving time and reducing duplication.
  • Versioning: ACR supports tagging, making it easy to trace changes and roll back if needed.
  • Scalability: It’s built to handle growth, so as your infrastructure expands, ACR keeps up.

By automating the publishing process, you can eliminate manual steps, reduce errors, and streamline collaboration across projects.

What You’ll Need to Get Started

Before we jump into the technical details, make sure you have these prerequisites in place:

  1. Azure Bicep CLI:
    To support this setup, you need to configure either a Microsoft-hosted or self-hosted Azure DevOps agent. Alternatively, you can execute the commands locally with a similar setup. Here's what you'll need to get started:

    az bicep install
az config set bicep.use_binary_from_path=true

  2. Azure Container Registry (ACR):
    Set up an active ACR instance to store your Bicep modules. You can deploy ACR using the Azure Portal or by creating a separate Azure DevOps pipeline with Bicep code to automate the setup process. Below you can see a sample Bicep code to deploy an Azure Container Registry inctance

    3. 
@description('The name of the Azure Container Registry')
param acrName string

@description('The SKU of the Azure Container Registry')
@allowed([
  'Basic'
  'Standard'
  'Premium'
])
param sku string = 'Basic'

@description('The location of the Azure Container Registry')
param location string = resourceGroup().location

resource acr 'Microsoft.ContainerRegistry/registries@2023-01-01-preview' = {
  name: acrName
  location: location
  sku: {
    name: sku
  }
  properties: {
    adminUserEnabled: false
  }
}

  3. Azure DevOps Service Connection:
    Configure a service connection in Azure DevOps with permissions AcrPush / AcrPull to access your ACR resources.

  4. Repository Structure:
    Organize your repository to streamline the publishing process.

How to Structure Your Repository

A well-organized repository is the foundation for efficient module management. Here’s how I structure mine:


  /modules
    /storageAccount
        - your-module-name.bicep          # Core Bicep file defining resources
    /keyVault
        - your-module-name.bicep
    /sqlServer
        - your-module-name.bicep
  /pipelines
        - bicep-publish.yaml

File Details:

  • your-module-name.bicep: This is your module’s blueprint, defining all the necessary resources.

Example az-storage-account.bicep including metadata block:

{
@metadata({
  Version: 'v0.1'
  Name: 'Azure Storage Account'
})
param storageAccountName string 
param location string = resourceGroup().location
param skuName string = 'Standard_LRS'
param kind string = 'StorageV2'
param accessTier string = 'Hot'
param enableHttpsTrafficOnly bool = true
param tags object = {}
param allowBlobPublicAccess bool = false  // Added parameter to control public access
param minimumTlsVersion string = 'TLS1_2'  // Added parameter for minimum TLS version

resource storageAccount 'Microsoft.Storage/storageAccounts@2021-04-01' = {
  name: storageAccountName
  location: location
  sku: {
    name: skuName
  }
  kind: kind
  properties: {
    supportsHttpsTrafficOnly: enableHttpsTrafficOnly
    accessTier: accessTier
    allowBlobPublicAccess: allowBlobPublicAccess  // Configured to disable public access
    minimumTlsVersion: minimumTlsVersion  // Set to enforce TLS 1.2 or higher
    networkAcls: {  // Added network ACLs to restrict access
      defaultAction: 'Deny'  // Deny all traffic by default
      bypass: 'AzureServices'  // Allow trusted Azure services by default
      ipRules: [
        {
          value: '203.0.113.0/24'  // Replace with permitted IP ranges
        }
      ]
    }
  }
  tags: tags
}

output storageAccountId string = storageAccount.id

Publishing Pipeline Overview

To automate publishing, I built an Azure DevOps pipeline with the following features:

  1. Version Validation: Ensures every module has a valid version in its metadata block.
  2. Conditional Publishing: Prevents overwriting existing versions in ACR by checking published tags.
  3. Error Handling: Logs skipped modules or publishing errors to improve traceability and reduce guesswork.

Optimized Pipeline YAML

Here’s the YAML configuration I use to publish Bicep modules to ACR:


name: $(Build.SourceBranchName)_$(Date:yyyyMMdd)_$(Rev:.r)
# Manually triggered via DevOps portal
trigger: none

variables:
  serviceConnection: "test01"          # Azure DevOps service connection to authenticate with Azure
  bicepRegistry: "depocacr01.azurecr.io"  # Azure Container Registry (ACR) URL
  modulePrefix: "modules/bicep/"       # Prefix for Bicep modules in ACR

pool:
  vmImage: "windows-latest"            # Use the latest Windows-based build agent

stages:
  - stage: Publish_Modules
    jobs:
      - job: Publish
        steps:
          # Checkout the code repository
          - checkout: self

          # Azure CLI Task to publish Bicep modules
          - task: AzureCLI@2
            displayName: "Publish Bicep Modules to ACR"
            inputs:
              azureSubscription: $(serviceConnection)  # Use the Azure DevOps service connection
              scriptType: "pscore"                     # Use PowerShell Core for the script
              scriptLocation: "inlineScript"           # Inline script to run the Azure CLI commands
              inlineScript: |
                # Ensure the Bicep CLI is set up and available
                az config set bicep.use_binary_from_path=true --only-show-errors
                az bicep install --only-show-errors

                # Define paths and variables for module processing
                $modulesPath = "$(Pipeline.Workspace)/s/Modules"  # Path to the Modules folder. Note: make sure you provide a correct path as it could be different if you are using a self-hosted agent.
                $bicepRegistry = "$(bicepRegistry)"               # Azure Container Registry URL
                $modulePrefix = "$(modulePrefix)"                 # Module prefix in ACR

                # Check if the Modules folder exists
                if (-not (Test-Path "$modulesPath")) {
                    Write-Output "Modules directory not found at $modulesPath. Exiting."
                    exit 1
                }

                Write-Output "Modules directory found at $modulesPath."
                Write-Output "Fetching list of published modules from ACR..."

                # Get the list of existing modules in ACR matching the prefix
                $publishedModules = $(az acr repository list --name $bicepRegistry --query "[?contains(@, '$modulePrefix')]" -o tsv --only-show-errors 2>$null)

                # Iterate through all Bicep files in the Modules directory (excluding test files)
                Get-ChildItem -Recurse -Path "$modulesPath" -Filter *.bicep | Where-Object {
                    -not ($_.FullName -like "*\tests\*")  # Exclude files in 'tests' subdirectories
                } | ForEach-Object {
                    $fileName = $_.BaseName               # Get the base name of the file
                    $bicepFilePath = $_.FullName          # Full file path of the Bicep file

                    Write-Output "Processing module: $fileName ($bicepFilePath)"

                    # Extract version metadata from the Bicep file
                    $bicepContent = Get-Content -Path $bicepFilePath -Raw
                    if ($bicepContent -match '@metadata\({\s*Version:\s*''([^'']+)''') {
                        $moduleVersion = $matches[1]      # Extract the version from the metadata
                        Write-Output "Found version metadata: $moduleVersion for module $fileName"
                    } else {
                        Write-Output "No version metadata found in $fileName. Skipping."
                        return
                    }

                    # Check if the module already exists in the ACR
                    if ($publishedModules -contains "$modulePrefix$fileName") {
                        Write-Output "Module $fileName already exists in the registry. Checking tags..."

                        # Get the list of existing tags for the module
                        $existingTags = az acr repository show-tags --name $bicepRegistry --repository "$modulePrefix$fileName" --query "[]" -o tsv --only-show-errors 2>$null

                        # Check if the version is already published
                        if ($existingTags -contains $moduleVersion) {
                            Write-Output "Module $fileName version $moduleVersion already exists in the registry. Skipping."
                        } else {
                            Write-Output "Publishing new version $moduleVersion for module $fileName..."
                            # Publish the module with the specified version to ACR
                            az bicep publish --file $bicepFilePath --target br:$bicepRegistry/$modulePrefix${fileName}:$moduleVersion --only-show-errors 2>$null
                        }
                    } else {
                        Write-Output "Module $fileName not found in registry. Publishing..."
                        # Publish the module to ACR
                        az bicep publish --file $bicepFilePath --target br:$bicepRegistry/$modulePrefix${fileName}:$moduleVersion --only-show-errors 2>$null
                    }
                }

                # Generate a list of available modules and their versions in ACR
                Write-Output "Generating a list of available modules and versions in the registry..."
                $items = @()
                $(az acr repository list --name $(bicepRegistry) --query "[?contains(@, '$(modulePrefix)')]" -o tsv --only-show-errors 2>$null) | ForEach-Object {
                    # Fetch tags (versions) for each module
                    $tags = $(az acr repository show-tags --name $(bicepRegistry) --repository $_ -o tsv --only-show-errors 2>$null)
                    $items += [PSCustomObject]@{
                        Module = $_            # Module name
                        Tags = $tags -join ", "  # Concatenate tags as a comma-separated string
                    }
                }

                # Output the list of modules and versions
                Write-Output $items | Format-Table -AutoSize

                Write-Output "The following modules are available in the registry:"
                Write-Output $items

Breaking Down the Script: A Detailed Walkthrough

Let’s take a closer look at each key step in the script and understand why it’s crucial for automating the publishing process for Azure Bicep modules.

1. Azure CLI Setup

az config set bicep.use_binary_from_path=true --only-show-errors
az bicep install --only-show-errors

This step ensures that Azure CLI is properly configured to work with Bicep CLI, which is essential for publishing modules.

  • What It Does:

    • Configures Azure CLI to use the local installation of Bicep CLI (bicep.use_binary_from_path=true).
    • Installs or updates the Bicep CLI to the latest version.

  • Why It Matters:

    • Ensures compatibility with the latest Bicep features and syntax.
    • Avoids potential conflicts that could arise from outdated or mismatched versions.
    • By explicitly installing and configuring the Bicep CLI in the pipeline, you reduce the chances of environment-related issues.

  • Tip: If you frequently update Bicep CLI locally, ensure your pipeline mirrors the same version to maintain consistency across environments.

2. Module Validation

Get-ChildItem -Recurse -Path "$modulesPath" -Filter *.bicep | Where-Object {
    -not ($_.FullName -like "*\tests\*")
} | ForEach-Object {
    $bicepContent = Get-Content -Path $_.FullName -Raw
    if ($bicepContent -match '@metadata\({\s*Version:\s*''([^'']+)''') {
        $moduleVersion = $matches[1]
        Write-Output "Found version metadata: $moduleVersion for module $fileName"
    } else {
        Write-Output "No version metadata found in $fileName. Skipping."
        return
    }
}

The script validates that each module has a valid metadata block in its your-module-name.bicep file. The metadata block must include a Version field to ensure proper tagging. If there is no metadata block in your bicep file, the pipeline will notify you about it and will skip publishing such a module.

  • What It Does:

    • Scans all .bicep files in the specified directory (modulesPath).
    • Excludes test files or directories, such as those in a tests subfolder.
    • Extracts the Version from the @metadata block in the Bicep file.

  • Why It Matters:

    • Modules without version metadata are skipped to prevent accidental overwrites.
    • Enforcing version metadata ensures traceability and consistency across published modules.
    • Helps maintain best practices for semantic versioning (1.0.0, 1.1.0, etc.).

  • What Happens If Metadata Is Missing:

    • The module is skipped, and a clear log message is generated, ensuring the issue can be easily identified and fixed.

3. ACR Checks

$publishedModules = $(az acr repository list --name $bicepRegistry --query "[?contains(@, '$modulePrefix')]" -o tsv --only-show-errors)

Before publishing, the script retrieves a list of existing modules from ACR to compare them with the modules in the repository.

  • What It Does:

    • Uses az acr repository list to fetch all repositories in ACR that match the module prefix.
    • Filters results to include only the relevant modules ($modulePrefix).

  • Why It Matters:

    • Avoids redundant uploads by checking if the module already exists in ACR.
    • Ensures that only new or updated modules are published, saving resources and time.
    • Provides a foundation for conditional publishing (next step).

  • Key Benefit:

    • This check helps maintain a clean and organized registry by avoiding duplicate entries or overwrites.

4. Conditional Publishing

if ($publishedModules -contains "$modulePrefix$fileName") {
    $existingTags = az acr repository show-tags --name $bicepRegistry --repository "$modulePrefix$fileName" --query "[]" -o tsv --only-show-errors
    if ($existingTags -contains $moduleVersion) {
        Write-Output "Module $fileName version $moduleVersion already exists. Skipping."
    } else {
        Write-Output "Publishing new version $moduleVersion for module $fileName..."
        az bicep publish --file $bicepFilePath --target br:$bicepRegistry/$modulePrefix${fileName}:$moduleVersion --only-show-errors
    }
} else {
    Write-Output "Publishing new module $fileName..."
    az bicep publish --file $bicepFilePath --target br:$bicepRegistry/$modulePrefix${fileName}:$moduleVersion --only-show-errors
}

This is where the decision-making happens. The script determines whether to publish a module based on its existence and version in ACR.

  • What It Does:

    • Checks for Module Existence: If the module exists in ACR, the script retrieves its tags (versions).
    • Compares Versions:
      • If the version already exists, the module is skipped.
      • If the version doesn’t exist, the module is published as a new version.
    • Publishes New Modules: If the module doesn’t exist at all, it’s added to ACR.

  • Why It Matters:

    • Prevents overwriting existing versions, ensuring data integrity.
    • Optimizes publishing by avoiding unnecessary operations.
    • Ensures that every version in ACR reflects a unique and valid module state.

  • Best Practice:

    • Use meaningful and consistent versioning in your metadata to make version management easier.

5. Logging

Write-Output "Processing module: $fileName ($bicepFilePath)"
Write-Output "Found version metadata: $moduleVersion for module $fileName"
Write-Output "Module $fileName version $moduleVersion already exists. Skipping."
Write-Output "Publishing new module $fileName..."

The script logs every significant action to provide transparency and traceability.

  • What It Logs:

    • Start of Processing: Indicates which module is being processed.
    • Validation Results: Logs whether a module has valid version metadata.
    • Publishing Decisions: Logs whether a module was skipped, updated, or published.
    • Errors or Warnings: Reports issues like missing metadata or failed publishing attempts.

  • Why It Matters:

    • Troubleshooting: Detailed logs make it easy to identify and fix issues in the publishing process.
    • Audit Trail: Logs provide a record of all actions, which is valuable for compliance and debugging.
    • Clarity: By clearly stating why a module was skipped or published, the script reduces guesswork.

  • Example Log Messages:

    • Processing module: keyVault (/path/to/keyVault/your-module-name.bicep)
    • Found version metadata: v1.0.1 for module keyVault
    • Module keyVault version v1.0.1 already exists. Skipping.
    • Publishing new version v1.0.2 for module keyVault...

Why This Script is Effective

By combining validation, conditional publishing, and detailed logging, this script ensures a robust and efficient publishing process:

  • Accuracy: Validates every module before publishing.
  • Efficiency: Publishes only new or updated versions.
  • Traceability: Provides clear, actionable logs for every step.

Whether you’re managing a handful of modules or hundreds, this script lays the groundwork for a scalable and reliable pipeline.

Visualizing the Pipeline Run

 

 

A screenshot of the Azure DevOps pipeline run showcasing the successful publishing of Bicep modules.

 

 

A screenshot of the Azure Portal run showcasing the successful publishing of Bicep modules.

 

Successful publishing of Bicep modules

 

A screenshot of the Azure Portal run showcasing the successful publishing of Bicep modules - Module Tags (versions).

 

Overcoming Challenges in Automating Azure Bicep Module Publishing

Automating the publishing of Azure Bicep modules to Azure Container Registry (ACR) is transformative, but it comes with its own set of challenges. Below is a breakdown of the hurdles I faced and the strategies I employed to resolve them.

1. Metadata Management: Ensuring Accurate Versioning

The Challenge:
Every .bicep file must include a well-structured metadata block. Without this block, the pipeline cannot determine the module's version, leading to skipped modules during publishing. This oversight can create confusion and reintroduce older versions unintentionally.

Example Metadata Block:

@metadata({
  Version: 'v0.1'
  Name: 'Azure SQL Server'
})

Key Issues:

  • Manual Updates: After modifying a module, the Version field in the metadata block must be manually updated. Forgetting this step can cause significant problems, such as overwriting older versions or losing track of changes.
  • Validation Needs: Modules without valid metadata often fail silently in pipelines, leading to wasted time debugging.

The Solution:

  • Manual Version Updates: Always update the Version field in your metadata block when making changes. A pre-commit hook or linter can flag version mismatches before committing to version control.
  • Automated Validation: Integrate a validation step in your pipeline to ensure every module includes a properly formatted metadata block. This proactive approach minimizes errors and ensures consistency.

2. Registry Organization: Maintaining a Clean Structure

The Challenge:
Managing multiple Bicep modules can quickly become chaotic without a structured approach. An unorganized registry makes locating and managing specific modules cumbersome, especially as the number of modules grows.

Key Issues:

  • Ambiguous Naming: Without a consistent naming convention, modules can become hard to identify or might overwrite each other.
  • Search Difficulties: Modules mixed with other container images in ACR can be challenging to locate.

The Solution:

  • Consistent Prefixes: Use clear, hierarchical prefixes to categorize modules. For example:
    • modules/bicep/keyvault
    • modules/bicep/sqlserver
    • modules/bicep/storage
  • Descriptive Names: Avoid generic names; instead, ensure module names are specific and meaningful.

By maintaining these conventions, your ACR remains scalable and easy to navigate, even as your infrastructure expands.

3. Agent Setup: AzureCLI vs. PowerShell Tasks

The Challenge:
Setting up the Azure DevOps pipeline agent posed unique challenges, especially with the choice between using the AzureCLI task and the PowerShell task for interacting with ACR.

AzureCLI@2 Task:

  • Advantages:

    • Simplifies authentication by leveraging Azure DevOps' service connection.
    • Provides direct access to Azure resources without requiring additional setup.
    • Executes commands like az acr repository list seamlessly.

  • Best Practice:

    • Use the AzureCLI@2 task wherever possible for simplicity and reliability.

PowerShell@5 Task:

  • Authentication Complexity:

    • Requires manual authentication using Azure Service Principal (SPN) credentials, such as:
      • Azure Client ID
      • Client Secret
      • Tenant ID
    • Without proper authentication, commands like az acr repository list may behave unexpectedly or fail entirely.

  • Example Authentication for PowerShell:

    $clientId = $env:AZURE_CLIENT_ID
$clientSecret = $env:AZURE_CLIENT_SECRET
$tenantId = $env:AZURE_TENANT_ID

az login --service-principal -u $clientId -p $clientSecret --tenant $tenantId

  • Key Issues:

    • Overhead: Manually managing SPN credentials increases complexity.
    • Security Risks: Improper handling of SPN credentials could expose sensitive information.

The Solution:

  • For AzureCLI Users:
    Stick with AzureCLI@2 for tasks like az acr repository list or az bicep publish. It abstracts authentication complexities and integrates seamlessly with Azure DevOps.

  • For PowerShell Users:
    If you must use PowerShell@5, ensure SPN credentials are securely stored in Azure DevOps variable groups or Azure Key Vault. Also, be aware of the differences in how az acr commands operate in PowerShell.

Why This Matters

Addressing these challenges has not only improved the reliability of my pipeline but also simplified the process of maintaining and publishing Bicep modules. Here's how these solutions make a difference:

  • Metadata Management: Ensures every module is accurately versioned and traceable.
  • Registry Organization: Keeps ACR clean and scalable, even as the number of modules grows.
  • Agent Setup: Reduces the risk of errors and ensures seamless authentication across environments.

Let’s Collaborate!

Have you faced similar challenges while automating your Azure Bicep module publishing? What strategies worked for you? Share your experiences in the comments below, and let’s create a better workflow together!

Useful Links

Here are some essential resources to help you dive deeper into automating Azure Bicep module publishing and related topics:

Official Azure Bicep Resources

Azure Container Registry (ACR)

Azure DevOps

About me

Azure Cloud Solutions Engineer with a passion for cloud automation, DevOps, Kubernetes , monitoring, identity management (IAM), cloud security and O365. Working with Azure, Amazon AWS, System Center, OMS/Log Analytics and related cloud services and software.

My certifications
Recent articles
Should You Use a Separate Stage for Bicep Linting in Your CI/CD Pipeline?
Should You Use a Separate Stage for Bicep Linting in Your CI/CD Pipeline?
How To Share Bicep Templates and Modules
How To Share Bicep Templates and Modules
How to publish Bicep Modules to ACR
How to publish Bicep Modules to ACR
Azure Blob Storage Lifecycle Management Policies with Terraform
Azure Blob Storage Lifecycle Management Policies with Terraform
Topics
Tags