𝐏𝐫𝐨𝐃𝐞𝐯𝐎𝐩𝐬𝐆𝐮𝐲 ♾️ 𝐅𝐫𝐞𝐞 𝐃𝐞𝐯𝐎𝐩𝐬/𝐂𝐥𝐨𝐮𝐝 𝐖𝐨𝐫𝐥𝐝‌

date

Apr 23, 2024

slug

terraform-with-azure-devops-ci-cd-pipelines

author

status

Public

Prerequisites

Before we get started, make sure you have the following in place:

Azure Subscription: To host your resources provisioned by Terraform. If you don't have one, you can sign up for a free trial.

Azure DevOps Account: To create CI/CD pipelines. If you don't have one, you can sign up for free at https://azure.microsoft.com/en-us/services/devops/.

Service Principal (SP): Setup a Service Principal in Azure Entra ID (Formally known as Azure Active Directory (AAD) with Contributor access to the Azure subscription or Resource Group where you want to deploy resources. The SP will be used by Terraform to authenticate and interact with Azure. Instructions on how to do that can be found on the Microsoft docs pages here.

How to run Terraform in an Azure DevOps pipeline

Create an Azure DevOps Project

With your Azure DevOps account setup, you next need to sign into Azure DevOps and create a project where you'll be setting up your CI/CD pipelines.

Go to the Azure DevOps website (https://dev.azure.com/) and sign in with your Microsoft account or organizational account.

Navigate to Organization: If you're part of multiple organizations, select the organization where you want to create the project from the top right corner of the page.

Click on the "New Project" button.

In the "Create a new project" form, fill in the Project Name and visibility (whether the project should be public or private) and optionally, provide a description for your project, then click on the "Create" button.

Once the project is created, you'll be redirected to the project's dashboard. Depending on your requirements, you may want to customize your project further by adding team members, configuring permissions, setting up repositories, boards, and pipelines, and integrating with other services.

Create Terraform configuration files

Create your Terraform configuration files and store them in a repository. This repo could be in your Azure DevOps project, or hosted elsewhere. In this example, we define a configuration file to provision a resource group containing a VNET and subnet.

network.tf


Copyprovider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "UK South"
}

resource "azurerm_virtual_network" "example" {
  name                = "example-vnet"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  address_space       = ["172.16.0.0/16"]
}

resource "azurerm_subnet" "example" {
  name                 = "example-subnet"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["172.16.1.0/24"]
}

Configure the Azure DevOps pipeline & Define CI/CD steps in YAML

As a best practice, your pipelines should be created in code. Create the pipeline in code in YAML format and store it in the same repository:

azure-pipelines.yml


Copytrigger:
- main

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Plan
  displayName: 'Terraform Plan'
  jobs:
  - job: TerraformPlan
    displayName: 'Terraform Plan'
    steps:
    - task: TerraformInstaller@0
      inputs:
        terraformVersion: 'latest'

    - script: |
        terraform --version
        terraform init -backend-config="storage_account_name=$(storageAccountName)" -backend-config="container_name=$(containerName)" -backend-config="key=$(key)" -backend-config="access_key=$(accessKey)"
        terraform plan -out=tfplan
      displayName: 'Terraform Init and Plan'

- stage: Deploy
  displayName: 'Terraform Apply'
  jobs:
  - job: TerraformApply
    displayName: 'Terraform Apply'
    steps:
    - task: TerraformInstaller@0
      inputs:
        terraformVersion: 'latest'

    - script: |
        terraform --version
        terraform init -backend-config="storage_account_name=$(storageAccountName)" -backend-config="container_name=$(containerName)" -backend-config="key=$(key)" -backend-config="access_key=$(accessKey)"
        terraform apply -auto-approve tfplan
      displayName: 'Terraform Init and Apply'

The pipeline is triggered whenever changes are pushed to the main branch.

The pipeline runs on an Ubuntu agent (ubuntu-latest).

TerraformInstaller: Installs the latest version of Terraform on the build agent.

Script: Executes Terraform commands to initialize the Terraform working directory, generate a plan, and apply changes to Azure. It initializes Terraform with a backend configuration to store the state in an Azure Storage Account (replace <your-azure-service-connection> with your actual Azure service connection details).

TerraformTaskV1: Uses the Terraform task to apply changes to the infrastructure. It first specifies the plan command in the first stage, and then an apply stage, utilizing the Azure service connection for authentication (environmentServiceNameAzureRM and backendServiceArm), and the working directory where the Terraform files are located. The commandOptions: '-auto-approve' automatically approves the changes without prompting for confirmation.

To use the pipeline:

In Azure DevOps, go to "Pipelines" > "Pipelines" and click on "New Pipeline".

Point your pipeline to the Git repository containing the azure-pipelines.yml file.

Azure DevOps will automatically detect the YAML configuration file. Review and confirm the configuration, then save.

Run the pipeline

To run your new pipeline, first navigate to Pipelines in the left sidebar. Choose the pipeline that you want to run from the list of available pipelines. If you have multiple pipelines, you'll see them listed here. Once you've selected the pipeline, you'll see a "Run pipeline" button at the top right corner. Click on this button to initiate a manual run of the pipeline. If your pipeline is set up to trigger on specific branches, you may be prompted to select the branch you want to run the pipeline for. If not, you can skip this step. Review the configuration and settings for the pipeline run, and then click on the "Run" button to confirm and start the pipeline execution.

Once the pipeline run is started, you'll be taken to the pipeline run page where you can monitor the progress of the pipeline execution. You'll see the various stages and tasks being executed in real time. As the pipeline runs, you can view detailed logs for each task to see the output and any error messages. You can also access any artifacts generated by the pipeline, such as build outputs or deployment packages.

After the pipeline run completes, you can review the results to see if the pipeline executed successfully or if there were any failures. You can view the overall status, individual task status, and any associated warnings or errors.

Best practices for Terraform in Azure DevOps pipelines

Here are 10 best practices you'll want to think about when writing your Terraform pipelines in Azure DevOps!

Define your Azure DevOps pipelines using YAML as code (e.g., azure-pipelines.yml) alongside your Terraform configurations. This practice enables versioning, review, and audit of pipeline changes. Store your Terraform configurations in version control (e.g., Git) to track changes, collaborate with team members, and maintain a history of infrastructure modifications.

Configure pipeline triggers to automatically execute pipelines upon code changes (e.g., commits to specific branches or pull requests, or on an overnight schedule to run a plan). This ensures timely validation and deployment of infrastructure changes.

Optimize pipeline performance by parallelizing tasks whenever possible. For example, run Terraform plan and apply stages concurrently for different environments or resource groups.

Maintain separate pipelines and environments (e.g., Dev, QA, Prod) to isolate infrastructure changes and avoid unintended impacts across environments. These can be set up in the Environments section of Azure DevOps.

Integrate automated testing into your pipelines for Terraform configurations using tools like Terratest to validate infrastructure changes before deployment.

Implement robust error handling and logging mechanisms within pipelines to capture and report errors effectively. Utilize Azure DevOps pipeline features like logging commands and error-handling tasks.

Include validation steps in your pipelines to verify Terraform configurations against best practices, policies, and compliance requirements using tools like terraform validate to confirm your syntax is correct, terraform fmtto validate your formatting or call static analysis tools.

Monitor pipeline execution, performance metrics, and resource usage to identify bottlenecks, optimize workflows, and ensure the reliability of infrastructure deployments.

Apply security best practices to your pipelines, including access control, role-based permissions, and regular security reviews to mitigate risks associated with pipeline configurations and execution.

Document pipeline configurations, deployment processes, and infrastructure designs to facilitate collaboration, knowledge sharing, and onboarding of team members.

Key Points

Terraform tasks in the Terraform workflow including deployments can be automated with Azure DevOps CI/CD Pipelines. Azure DevOps is a good choice to seamlessly integrate with Azure and integrate with the Azure ecosystem in particular. Using pipelines means you can repeat the workflow tasks defined in your pipeline in a repeatable and controlled manner, giving you consistency and removing manual effort.

Thank you for reading my blog …:)

Support 🫶

Help spread the word about ProDevOpsGuy by sharing it on social media and recommending it to your friends. 🗣️

You can also sponsor 🏅 on GitHub Sponsors // 🎗️ Support Our Work 🎗️

Pull down to refresh

Terraform with Azure DevOps CI/CD Pipelines