Automating Azure Blob Storage Backups with Advanced Backup Policies Using Terraform and the AzAPI Provider

Table of Contents

Introduction

Setting Azure Blob Storage Backup Frequency with Advanced Backup Policies Using Terraform and the AzAPI Provider

Continuing from my previous blog post on automating Azure Blob Storage backups with Terraform, let's explore a significant update: the ability to set backup frequencies and retention rules for Azure Blob Storage, enabled through the AzAPI provider in Terraform. With this approach, we can fully automate complex backup schedules beyond the AzureRM provider's current capabilities, including daily, weekly, and monthly backups with custom retention policies.

The Issue with Terraform's AzureRM Provider and Backup Frequency Support

As of this writing, Terraform's native azurerm_data_protection_backup_policy_blob_storage resource does not yet support frequency-based backup scheduling for Azure Blob Storage, due to limitations in the AzureRM provider. This limitation has led to challenges for many users looking to implement robust, frequency-based backup policies directly through Terraform.

The ongoing GitHub issue #16711 highlights community demand for backup frequency support in azurerm_data_protection_backup_policy_blob_storage. While contributors continue discussing potential enhancements, this feature remains in an open state with no official implementation in the AzureRM provider as of now.

To overcome this, the AzAPI provider offers a workaround by allowing direct access to Azure's Data Protection APIs, enabling advanced backup configurations, including customized scheduling and retention.

Enhanced Solution: Automating Azure Blob Storage Backup Frequency with AzAPI

The AzAPI provider allows us to define a comprehensive backup policy, including daily, weekly, and monthly backups. Below is a Terraform configuration that leverages the AzAPI provider to configure these features within the azurerm_data_protection_backup_vault and azapi_resource resources.

Step 1: Define the Backup Vault using azurerm_data_protection_backup_vault

The azurerm_data_protection_backup_vault resource sets up a backup vault to store Blob backups with options for redundancy and system-assigned identity.

resource "azurerm_data_protection_backup_vault" "this" {
  name                = var.backup_vault_name
  resource_group_name = var.resource_group_name
  location            = var.location
  datastore_type      = var.datastore_type
  redundancy          = var.redundancy
  tags                = local.tags

  identity {
    type = "SystemAssigned"
  }
}

Step 2: Define the Backup Policy and Configure Backup Frequency with AzAPI

With the azapi_resource, we configure the backup policy directly using Azure’s API capabilities, including complex scheduling and retention rules.

resource "azapi_resource" "backup_policy" {
  type      = "Microsoft.DataProtection/backupVaults/backupPolicies@2022-11-01-preview"
  name      = var.backup_vault_policy_name
  parent_id = azurerm_data_protection_backup_vault.this.id
  body = jsonencode({
    properties = {
      datasourceTypes = [
        "Microsoft.Storage/storageAccounts/blobServices"
      ]
      objectType = "BackupPolicy"
      policyRules = [
        {
          objectType = "AzureRetentionRule"
          name       = "DailyRetention"
          isDefault  = true
          lifecycles = [
            {
              deleteAfter = {
                objectType = "AbsoluteDeleteOption"
                duration   = var.daily_retention_duration_days
              }
            }
          ]
        },
        {
          objectType = "AzureRetentionRule"
          name       = "MonthlyRetention"
          isDefault  = false
          lifecycles = [
            {
              deleteAfter = {
                objectType = "AbsoluteDeleteOption"
                duration   = "P${var.monthly_backup_retention_duration_months}M"
              }
            }
          ]
        },
        {
          objectType = "AzureRetentionRule"
          name       = "YearlyRetention"
          isDefault  = false
          lifecycles = [
            {
              deleteAfter = {
                objectType = "AbsoluteDeleteOption"
                duration   = "P${var.yearly_backup_retention_duration_years}Y"
              }
            }
          ]
        },
        {
          objectType = "AzureBackupRule"
          name       = "DailyBackup"
          backupParameters = {
            objectType = "AzureBackupParams"
            backupType = "Discrete"
          },
          trigger = {
            objectType = "ScheduleBasedTriggerContext"
            schedule = {
              repeatingTimeIntervals = [
                "R/2023-01-01T00:00:00+10:00/P1D"
              ],
              timeZone = "AUS Eastern Standard Time"
            }
          },
          dataStore = {
            objectType    = "DataStoreInfoBase"
            dataStoreType = "VaultStore"
          }
        }
      ]
    }
  })
  schema_validation_enabled = false
}

Explanation of the Backup Policy Components

1. Daily, Monthly, and Yearly Retention Rules: We specify multiple retention rules with customized lifecycles and set the frequency for backups according to the data needs.

2. Scheduled Backup Trigger: We use a ScheduleBasedTriggerContext to define daily backups, specifying the repeatingTimeIntervals to run backups every day at the desired time. The AzAPI provider allows us to customize this with time zone support.

3. Flexible Deletion Policy: For each retention rule, we set deleteAfter to ensure data is retained based on your operational, monthly, and yearly requirements.

Conclusion

By leveraging the AzAPI provider alongside Terraform, we achieve complete automation for Azure Blob Storage backups, including frequency scheduling. Although Terraform's AzureRM provider currently lacks official support for frequency-based backup policies, the AzAPI provider fills this gap, enabling advanced policies directly through Azure's Data Protection APIs. Let me know in comments below what is your approach when you are dealing with Zzure Backup Policies .. ? 

Useful Links

To deepen your understanding and explore related concepts, here’s a curated list of useful links:

1. Terraform Documentation:

   • Terraform AzAPI Provider
   • Using Azure Blob Storage with Terraform

2. Azure Data Protection and Backup:

   • Azure Data Protection Overview
   • Backup Policies for Azure Blob Storage

3. GitHub Discussions and Issues:

   • Terraform AzureRM GitHub Issues
   • Issue #16711: Support for Backup Scheduling in AzureRM

4. Azure CLI Reference:

   • Azure CLI Command Guide

5. Azure Storage Lifecycle Policies:

   • Azure Storage Management Policies

     JSON Structure Validation:

     • JSON Formatter and Validator
     • Azure Resource Manager (ARM) Schema Reference

6. Azure Blob Storage Tutorials:

   Getting Started with Azure Blob Storage