Terraform Modules, lets make some automation's

James
James Cloud Architect working mainly in Azure and blogging his journey and hopefully dropping some useful bits of information on the way.

What do I mean by automation in modules?

When it comes to automation in modules there are some key scenarios where you may want to automate the operations inside the module. The two areas that are work considering are:

Documentation

One of the easiest areas to automate is your Documentation with Terraform Docs, in Azure DevOps its pretty simple to make a pipeline, I’ve put the pipeline that is used by myself below.

The pipeline runs on all Branches so that even Branches documentation is kept up to date, also a good indication of when my code is not yet working as the README fails to generate!

trigger:
- "*"

pool:
  vmImage: ubuntu-latest

steps:
- checkout: self
  persistCredentials: true

- script: |
    curl -sSLo ./terraform-docs.tar.gz https://terraform-docs.io/dl/v0.16.0/terraform-docs-v0.16.0-linux-amd64.tar.gz
    tar -xzf terraform-docs.tar.gz
    chmod +x terraform-docs
    ./terraform-docs ./ -c terraform-docs.yml

    git config --global user.email "ADO-NoReply@rawritscloud.onmicrosoft.com"
    git config --global user.name "Documentation Pipeline"
    git add README.md
    git commit -m "Documentation Autogenerated  [skip ci]"
    git push origin HEAD:$(Build.SourceBranchName)
  displayName: 'Terraform Docs'

Testing

This one is slightly more controversial as this could turn out to be pretty expensive depending on how many modules you are creating.

There are plenty of ways to automate Testing, one of the more popular ways is to use Terratest but that can add additional skills requirements and depending on your team size that means learning Go as well as HCL.

The way I have found not only the easiest but is a real test by actually standing up the resources, a generic pipeline in our modules project and a connection to a single developer subscription that solely used for modules to apply and destroy.

Calling a modified Pipeline we use in our Customer projects and then as part of our documentation we have an example.tf and this is used as the code that should be applied, the example contains blank values for anything that is reliant on previous modules. The example also has all variables specified with their “default” example values.

You can see we only trigger on main branch as we don’t want it constantly going off when developing the module, we also have it set to be run when a Pull Request is started. We Run in stages and in Environments so that we can add guard rails in the Customer environments.

trigger:
- main

pool:
  vmImage: windows-latest

variables:
- group: bte-management-platform
- group: customer-variables

stages:
- stage: plan
  displayName: 'Terraform Initilize and Plan an Apply'
  jobs:
  - job: init_and_plan
    displayName: 'Initilize and Plan Deployment' 
    steps:
    - checkout: self
      persistCredentials: true

    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
      displayName: 'Install Terraform latest'

    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2
      displayName: 'Terraform: init'
      inputs:
        workingDirectory: '$(System.DefaultWorkingDirectory)/examples'
        commandOptions: '-reconfigure -upgrade'
        backendServiceArm: 'Azure DevOps - AzureRM Connection'
        backendAzureRmResourceGroupName: '$(storage-account-resource-group)'
        backendAzureRmStorageAccountName: '$(storage-account-name)'
        backendAzureRmContainerName: terraform
        backendAzureRmKey: $(Build.Repository.Name).tfstate

    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2
      displayName: 'Terraform: plan'
      inputs:
        command: plan
        workingDirectory: '$(System.DefaultWorkingDirectory)'
        commandOptions: '-out $(Build.BuildNumber)apply.plan'
        environmentServiceNameAzureRM: 'Azure DevOps - AzureRM Connection'

    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: '$(System.DefaultWorkingDirectory)'
        artifact: 'terraformPlanStage'
        publishLocation: 'pipeline'

- stage: Deploy_Apply
  displayName: 'Deploy Terraform Plan'
  jobs:
  - deployment: deploy
    displayName: 'Terraform Deploy'
    environment: $(Build.Repository.Name)
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: terraformPlanStage

          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
            displayName: 'Install Terraform latest'

          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2
            displayName: 'Terraform: init'
            inputs:
              workingDirectory: '$(Pipeline.Workspace)\terraformPlanStage\'
              commandOptions: '-reconfigure -upgrade'
              backendServiceArm: 'Azure DevOps - AzureRM Connection'
              backendAzureRmResourceGroupName: '$(storage-account-resource-group)'
              backendAzureRmStorageAccountName: '$(storage-account-name)'
              backendAzureRmContainerName: terraform
              backendAzureRmKey: $(Build.Repository.Name).tfstate

          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2
            displayName: 'Terraform : apply'
            inputs:
              command: apply
              workingDirectory: '$(Pipeline.Workspace)\terraformPlanStage\'
              commandOptions: '"$(Build.BuildNumber)apply.plan"'
              environmentServiceNameAzureRM: 'Azure DevOps - AzureRM Connection'


- stage: plan_destory
  displayName: 'Terraform Initilize and Plan a Destroy'
  condition: eq('$', 'destroy')
  jobs:
  - job: init_and_plan
    displayName: 'Initilize and Plan a Destroy' 
    steps:
    - checkout: self
      persistCredentials: true

    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
      displayName: 'Install Terraform latest'

    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2
      displayName: 'Terraform: init'
      inputs:
        workingDirectory: '$(System.DefaultWorkingDirectory)/examples'
        commandOptions: '-reconfigure -upgrade'
        backendServiceArm: 'Azure DevOps - AzureRM Connection'
        backendAzureRmResourceGroupName: '$(storage-account-resource-group)'
        backendAzureRmStorageAccountName: '$(storage-account-name)'
        backendAzureRmContainerName: terraform
        backendAzureRmKey: $(Build.Repository.Name).tfstate

    - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2
      displayName: 'Terraform: plan'
      inputs:
        command: plan
        workingDirectory: '$(System.DefaultWorkingDirectory)'
        commandOptions: '-destroy -out $(Build.BuildNumber)destroy.plan'
        environmentServiceNameAzureRM: 'Azure DevOps - AzureRM Connection'

    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: '$(System.DefaultWorkingDirectory)'
        artifact: 'terraformPlanStage'
        publishLocation: 'pipeline'

- stage: Deploy_Destroy
  displayName: 'Deploy Terraform Plan'
  jobs:
  - deployment: deploy
    displayName: 'Terraform Deploy'
    environment: $(Build.Repository.Name)
    strategy:
      runOnce:
        deploy:
          steps:
          - download: current
            artifact: terraformPlanStage

          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller@0
            displayName: 'Install Terraform latest'

          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2
            displayName: 'Terraform: init'
            inputs:
              workingDirectory: '$(Pipeline.Workspace)\terraformPlanStage\'
              commandOptions: '-reconfigure -upgrade'
              backendServiceArm: 'Azure DevOps - AzureRM Connection'
              backendAzureRmResourceGroupName: '$(storage-account-resource-group)'
              backendAzureRmStorageAccountName: '$(storage-account-name)'
              backendAzureRmContainerName: terraform
              backendAzureRmKey: $(Build.Repository.Name).tfstate

          - task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV2@2
            displayName: 'Terraform : apply'
            inputs:
              command: apply
              workingDirectory: '$(Pipeline.Workspace)\terraformPlanStage\'
              commandOptions: '"$(Build.BuildNumber)destroy.plan"'
              environmentServiceNameAzureRM: 'Azure DevOps - AzureRM Connection'

comments powered by Disqus