10 min read1 day ago

–

Setup Terraform Efficiently using Backends, Parallelism & Modules.

Press enter or click to view image in full size

🌟 Introduction

Welcome, Devs, to the exciting world of Infrastructure and Cloud computing!

If you have been navigating the cloud landscape recently, you have undoubtedly encountered the industry buzzword: Infrastructure as Code (IaC). Simply put, IaC is the practice of specifying your infrastructure requirements in a code format rather than manually configuring servers and networks. This approach is the bedrock of modern DevOps, ensuring reliability and consistency across all your environments.

While there is a robust ecosystem of tools offering IaC capabilities — including Pulumi, Chef, Puppet, and AWS CloudFormation — today we are zeroing in on the undisputed heavyweight of this segment: Terraform.

If you are just starting out or want to dive deeper into the core concepts, I have covered a lot of ground regarding Terraform in my previous posts. I highly recommend checking out the full series here to get up to speed:

👉 Terraform Series

In today’s guide, we are shifting our focus to Optimisation. As your infrastructure grows, so does the complexity and the time required for deployments. We will dive into the best practices to improve the performance of your Terraform configurations, ensuring your infrastructure remains efficient, fast, and secure.

So, without further ado, let’s get started!

🌟 Pre-Requisites

Before we jump into the optimisation techniques, let’s ensure you have the necessary tools ready to roll.

As usual, for this guide, you will need the following set up on your machine:

• AWS CLI (Configured with an appropriate IAM user)
• Terraform CLI
• Docker and Docker-compose

If you don’t have these installed yet or aren’t sure how to configure them, don’t worry! I have walked through the entire process step-by-step in my previous guide. Just follow the instructions there, and you will be good to go:

👉 How to Install AWS CLI and Terraform

Once you are all set up, proceed to the next section where we start optimizing!

🌟 How to Optimise Terraform

Now that we are set up, let’s get into the meat of the matter. Here are 5 Best Practices to turbocharge your Terraform performance.

1. Use Remote Backends for State Management

By default, Terraform stores its “state” (the file that maps your code to real-world resources) locally on your machine (terraform.tfstate). While this works for solo side projects, it kills performance and collaboration in real teams.

The Fix: Store your Terraform State file in a remote location, preferably AWS S3.

• Why? It prevents conflicts when multiple team members (Dev, Stage, Prod) try to apply changes at the same time.
• The Performance Boost: Moving to a remote backend can improve I/O performance by 10–30% for large state files.
• State Locking: When configured correctly with State Lock enabled, you ensure that no two people can write to the state simultaneously. (Note: While traditionally done with DynamoDB, ensuring your backend supports locking is key to preventing corruption).

How to do it:

You simply need to create an S3 bucket and reference it in your configuration:

terraform {  backend "s3" {    bucket       = "my-terraform-state-bucket"    key          = "prod/terraform.tfstate"    region       = "us-east-1"    use_lockfile = true  }}

It’s that simple!

2. Modularise Your Code

Writing one giant [main\.tf](http://main.tf) file is a common rookie mistake. If you are defining a secure EC2 instance, you likely also need a VPC, Security Groups, Subnets, etc. Lumping this all together makes Terraform work harder to calculate dependencies.

The Fix: Break your code down into Modules.

Instead of rewriting the same sub-components for every service, use the DRY (Don’t Repeat Yourself) principle. Create a module for your network stack, a module for your compute stack, etc.

• Benefit: This reduces complex dependency graphs.
• Speed: It enables better parallel execution because Terraform can process distinct modules concurrently.

3. Unlock True Parallelism

Terraform is capable of walking through the dependency graph and creating non-dependent resources at the same time. However, the default setting is often too conservative.

The Fix: Adjust the -parallelism flag.

By default, Terraform limits concurrent operations to 10. For modern systems, this is quite low.

• Guideline: You can safely increase this to 30–100 for normal development changes.
• Constraint: Don’t go too high, or you might hit AWS API rate limits (throttling). Aim for roughly 512MB of system memory per 1,000 resources.

The Result:

Using this flag can cut down plan and apply times significantly — potentially dropping a 3–5 minute operation down to just 30–60 seconds.

terraform apply -parallelism=30

4. Optimise Provider Configuration

Sometimes, Terraform performs checks that aren’t strictly necessary for every single run, especially in non-production environments.

The Fix: Tune your AWS Provider.

In your Testing, Staging, or Dev environments, you can tell the provider to skip certain validations to save time on API calls. You can also configure retry behaviors to handle network blips gracefully without failing the whole run.

Example Configuration:

provider "aws" {  region                      = "us-east-1"  skip_credentials_validation = true  skip_metadata_api_check     = true  max_retries                 = 5}

Note:* While this is a great time-saver for Dev/Test environments, check your organization’s policy before using strict validation skipping in Production.*

5. Use Resource Targeting

When managing large infrastructure, Terraform’s default behavior is to analyze the entire configuration graph. For routine, independent updates (e.g., adding a new, isolated logging bucket), you might want to limit the scope of the operation to save time.

The Use Case: Use the -target flag when dealing with literally independent resources that are not connected to other resources that are changing. This allows you to apply changes only to specific resources, acting like a surgical strike rather than a full deployment.

# Only apply changes to the specific bucket, ignoring the restterraform apply -target=aws_s3_bucket.my_website_bucket

🌟 Practical Demonstration

Enough with the theory! Let’s get our hands dirty with some real Terraform code and the AWS Console.

To demonstrate these optimizations, we are going to deploy a Request Counter Application on AWS Elastic Beanstalk. This isn’t just a “Hello World”; it is a multi-container setup running on an ECS Cluster involving:

1. Nginx: As a reverse proxy and load balancer.
2. Node.js: The application server (running two instances).
3. Redis: To store the request count data persistently.

Step 1: Create Your ECR Repositories

First, we need a place to store our Docker images. We will create two Public Repositories in AWS Elastic Container Registry (ECR) — one for our web app and one for our custom Nginx image.

Run the following commands in your terminal:

aws ecr-public create-repository --repository-name nginx-node-redis-webaws ecr-public create-repository --repository-name nginx-node-redis-nginx

Step 2: Get the Code

The complete source code for this project is available on my GitHub.

1. Fork the repo.
2. Clone it to your local system.
3. Open it in your code editor (like VS Code).

👉 GitHub Repo: Nginx-Node-Redis Project

Step 3: Test Locally (Get the Vibe)

Before we deploy to the cloud, let’s make sure it works on your machine. Inside the project directory, run:

docker-compose up --build

This will spin up the Nginx, Redis, and two Web containers. Open your browser and go to http://localhost:8080.

Press enter or click to view image in full size

Press enter or click to view image in full size

You should see a nice Request Counter app. Every time you refresh, the counter increments, and you will see the server ID toggle between web1 and web2 as Nginx load-balances the traffic.

Once you are satisfied, press CTRL+C in your terminal to stop the application.

Step 4: Build and Push to Cloud

Now, let’s move these images to AWS.

1. Navigate to the web directory in your terminal.
2. Go to AWS Console → ECR → Public Repositories.
3. Select nginx-node-redis-web and click the “View Push Commands” button.
4. Execute those commands one by one to build and push your web image.

Press enter or click to view image in full size

Press enter or click to view image in full size

Press enter or click to view image in full size

Repeat for Nginx:

1. Navigate to the nginx directory.
2. In the AWS Console, select nginx-node-redis-nginx.
3. Click “View Push Commands” and execute them to push your Nginx image.

Press enter or click to view image in full size

Step 5: The Terraform Configuration

Navigate to the terra-performance-config directory. This is where we have applied all the optimizations we discussed earlier:

• provider.tf: We are skipping unnecessary validation checks (skip_metadata_api_check, etc.) to speed up the provider.
• backend.tf: We are using an S3 bucket to host our state file remotely and enabling lock_file to prevent write conflicts.
• main.tf: We are using a Terraform Module for Elastic Beanstalk. It utilizes a Dockerrun.aws.json file (similar to docker-compose but for ECS) which is uploaded to S3 and referenced in the module.
• dockerrun.aws.json: Skelton of your Docker containers, make sure to replace the image URI of web1, web2 and Nginx, you can find it in your ECR public Repo.
• variables.tf: Contains our standard configuration using the default VPC and subnets in us-east-1.

⚠️ CRITICAL: Architecture Check (ARM vs AMD)

I built the default Docker images on a MacBook Air M3 (ARM64 architecture).

If you are using Windows or Linux (Intel/AMD), your architecture is likely AMD64. You must make these two small changes in the code before proceeding, or the deployment will fail:

• In variables.tf: Change the instance type from t4g.micro (ARM-based) to t3.micro or t3.small.

Press enter or click to view image in full size

• In Dockerrun.aws.json: Locate the Redis container definition. Change the image from the specific ARM tag to generic: "image": "redis:alpine".

Press enter or click to view image in full size

Step 6: Testing Parallelism

Now for the moment of truth. Let’s see how fast we can provision this stack using the Parallelism optimization.

Run the following commands:

cd terra-performance-config# Create the S3 bucket for our artifactsaws s3 mb s3://pravesh-ebs-terra-performance-bucket# Initialize Terraformterraform init# Apply with high parallelismtime terraform apply --auto-approve -parallelism=3

The Result: Usually, a multi-container Elastic Beanstalk environment takes 10–30 minutes to provision. With -parallelism=30, you will likely see this drop to 3-8 minutes.

Press enter or click to view image in full size

Press enter or click to view image in full size

Press enter or click to view image in full size

Once finished, Terraform will output the Elastic Beanstalk URL. Click it to see your live app!

Step 7: Testing Resource Targeting

Finally, let’s look at the power of Resource Targeting.

We have made a small change to the Dockerrun.aws.json file (e.g., adding an Environment Variable).

Press enter or click to view image in full size

If we ran a normal apply, Terraform might check every single resource. Instead, we will target only the specific components we are updating:

time terraform apply -parallelism=30 \  -target=aws_s3_object.dockerrun \  -target=aws_elastic_beanstalk_application_version.v1 \  -target=module.elastic-beanstalk-environment \  -auto-approve

The Result: Without targeting, Terraform might attempt to refresh the state of the entire VPC and Security Group structure. With targeting, this update happens in under a minute.

Press enter or click to view image in full size

Cleanup

Don’t forget to tear down your infrastructure to avoid unexpected AWS bills!

# Destroy Terraform resourcesterraform destroy --auto-approve# Clean up S3aws s3 rm s3://pravesh-ebs-terra-performance-bucket --recursiveaws s3 rb s3://pravesh-ebs-terra-performance-bucket# Delete ECR Repositoriesaws ecr-public delete-repository --repository-name nginx-node-redis-web --forceaws ecr-public delete-repository --repository-name nginx-node-redis-nginx --force

🌟 Conclusion

And that’s a wrap, folks!

We have successfully journeyed through the landscape of Terraform Optimization. We didn’t just learn how to write Infrastructure as Code; we learned how to make it efficient, scalable, and blazing fast.

From moving our state to a Remote Backend to unlocking the speed of Parallelism, and from Modularising our logic to performing surgical strikes with Resource Targeting — you now possess the toolkit to take your DevOps game to the next level.

Remember, optimization isn’t just about saving a few minutes here and there. It’s about creating a developer experience where feedback loops are short, deployments are reliable, and your infrastructure can scale without becoming a bottleneck.

I hope you enjoyed this deep dive and the hands-on project. Go ahead and apply these techniques to your own infrastructure, and let me know how much time you saved on your next deployment!

If you found this guide helpful or have any questions, feel free to connect with me. I talk about Cloud, DevOps, and everything in between.

🚀 Let’s Connect:

💼 LinkedIn: linkedin.com/in/pravesh-sudha

🐦 Twitter/X: x.com/praveshstwt

📹 YouTube: youtube.com/@pravesh-sudha

🌐 Website: blog.praveshsudha.com

Happy Coding and Happy Clouding! ☁️🚀