Member-only story
Provision Azure Databricks using Azure Pipelines and Terraform— Part 1
4 min readFeb 11, 2022
–
Press enter or click to view image in full size
Azure Databricks using Terraform
In the first part of the article, we will explore the provisioning of Azure Databricks using terraform. Azure Databricks is a unified data and analytics PaaS platform that helps Data engineers, Data Scientists, and Data Analysts to collaborate and work together, without worrying about the infrastructure, tools, and libraries. Behind Azure Databricks, it runs an Apache-Spark-based system that splits its workloads and is distributed among the different nodes forming a cluster. In this article, we will provision a Databricks instance and in the second part of the article, we will continue…
Member-only story
Provision Azure Databricks using Azure Pipelines and Terraform— Part 1
4 min readFeb 11, 2022
–
Press enter or click to view image in full size
Azure Databricks using Terraform
In the first part of the article, we will explore the provisioning of Azure Databricks using terraform. Azure Databricks is a unified data and analytics PaaS platform that helps Data engineers, Data Scientists, and Data Analysts to collaborate and work together, without worrying about the infrastructure, tools, and libraries. Behind Azure Databricks, it runs an Apache-Spark-based system that splits its workloads and is distributed among the different nodes forming a cluster. In this article, we will provision a Databricks instance and in the second part of the article, we will continue to run a fully integrated CI/CD pipeline. The contents of this article are focused on the audience who is already familiar with Azure DevOps and Databricks. The complete source code of the article can be found here. So let’s get started.
1. Pre-requisites
- Azure Subscription and Service Principal created in Azure AD to programmatically interact with Azure Resource Manager(ARM)
- A PAT token from Azure DevOps Organization.
- The remote backend for Terraform used in this article is Azure Blob Storage. If you don’t have it ready, please create it by referring to this repo.