Oct 15, 2025

#Kubernetes

#Rant

#AWS

---

Intro

When learning something, I need to experiment a lot. All of this happens in labs, and most of the time they are provided by my employer, where resources aren’t really something I have to think about. But recently, I had to pay for my own labs out of pocket, and oh boy, did I feel it.

Well, not really, I haven’t been a broke person paying for labs, but every time I’ve spun up some lab, there was that anxiety in my mind: how much does this cost? Did I turn it off, or have I left something running ? Can I delete this now, or will I need it later ? Oh, I need to test more, but I’ve already removed the lab and don’t want to bring it back, etc.. It was super annoying, I had to fix it, which I did.

So this is my take on how to build a lab that will cost so little you won’t have to worry about it.

I do have to clarify something: this post is not about a person who has no money, such as a student just starting out studying for a job. There are a lot of options to study for free, including credits for labs, both vendors and third-party companies provide such options. I am not a broke person who found a loophole for free compute; what I am is an obsessive, anxious engineer with a knack for overcomplication. I look at something and overengineer it to the point of obsessiveness. Regardless, this post is for anyone who wants to study systems and how they are engineered, and it is more about that, than saving a buck.

Target labs

The nature of labs and what we want to learn from them are a major contributor of the cost and study path, as stated this blog is not a way to acquire credits or funding. I am not going to cover a lot of possible labs that heavily rely on specific SaaS services; such services are priced by providers as they see fit, with dedicated teams for promoting and training their clientele. If a targeted lab heavily relies on such offerings, please consider contacting the provider for credits, study paths and other learning resources.

What I discuss in this blog is how to utilize compute resources in AWS to spin up self-hosted services.

Why self-hosted? Self-hosting something is a sure way to learn about it. All of this lab talk comes from me learning something, this is probably the case for others as well. We are creating labs to experience our theoretical knowledge in practice. This is the most important part of the blog, while the cost side is simply that, a cost we pay for it.

Why AWS ? They provide an API for everything, and cost is calculated based on usage. This makes labs on AWS very, very cheap compared to other offerings that might offer 1/10th of AWS’s price for the same compute.

Does that make any sense? Well, it should, but here’s an example.

If AWS has a monthly cost of 4.5 USD for a 1/1 CPU/RAM compute node, and another platform has the same compute node for 0.50 USD on paper, the second offering seems cheaper, right? Sure, it is, but most of the time such offerings are prepaid or fully paid per month, meaning if I spin up 20 nodes of that 1/1 CPU/RAM node on AWS for 5 hours, the cost will come out to about 0.06 USD, while the other platform will charge me for the whole month (10 USD) even if I remove the node after those 5 hours.

This can be true for other clouds as well, that’s true; AWS does not hold a singular, unique advantage over other clouds. No, it’s more of a whole-package thing: Infrastructure-as-a-Service APIs; well-documented and widespread in the community; complex and mature services with a lot of flexibility; an experience that can be referenced and used in an actual job; etc..

Just make sure, when choosing a cloud service, to choose a serious player who is mature enough to provision everything automatically. This is in contrast to smaller players who might offer a service for much less but try to automate their offerings and end up manually approving or even provisioning resources will prove difficult. I once had an experience with a cloud provider that gave me a duplicate IP of another user’s machine, I have no idea how to account for such issues and fail to see how it would cost less after accounting unexpected problems, even if I don’t account for wasted time.

Cloud Resources

Dynamically managing resources is mandatory to achieve cheapness at the level where we don’t even feel the cost; it is truly the key in the equation. We have to provision a bunch of resources and remove them as soon as possible, and I don’t mean provisioning something for weeks and then removing it, no, no, we are talking hours here.

Provisioning something for hours certainly has its own quirks, with one mandatory thing originating from the nature of the targeted usage of said labs itself. When we provision a lab for study, we usually have to invest some time working on parts of the lab that have nothing to do with what we are studying just to get to the study part, and redoing that over and over just to save a buck seems so annoying.

But in actuality, this is a blessing in disguise. Just think about it this way: we don’t want to do anything manually, especially if we have to repeat it, so we are going to automate it. And to automate it, we have to understand it on such a level that we learn the inner workings of our study target from an outside perspective.

Really, I am very serious about this. Applications are created similarly; there is so much architectural difference out there, and the majority of the time these applications will share behaviors, things like how they store data, cache it, talk with other systems, or whatnot. And if we understand this for a few systems out there, not only will we have much better knowledge of those systems, but we can pretty much adjust the formula for others.

When provisioning a lab to study, we ignore all of this. We open some documentation provided by the vendor on how to spin up a demo environment or some getting started guide, and while they are definitely helpful for getting started fast, they lack a lot when it comes to really understanding the system itself.

instead of running a single Docker Compose command that will download a monstrosity of applications and present a web interface for us to access, where we are missing a lot and have to imagine magical unicorns doing their bubu baba magic to perform a simple application routine, I offer to be a blockhead, just like me. To analyze the application’s inner workings and automate each and every resource with their dependencies. every time we spin up our environment it will provision everything as stated by us and then demolish them. This way we will both study and save money, and trust me, nothing feels better (in this field?) than that feeling when you want to change some part of system and your brain basically simulates the whole exchange with implemented system. There are no magical unicorns, so why give them any power at all?

The tool for dynamically provisioning labs is Terraform. Now, this post really is not about Terraform, I will not explain how it works or argue alternatives, but I have a hard time imagining a scenario where this tool changes. Also, don’t worry about knowing Terraform; it is mandatory for what I will do in this post, and yes, to use it you have to know it. But trust me on this: there is only one way to learn Terraform, and that is by using it. So why not start here?

Engineers who are experienced in automation out there, sorry, but you are not using the tool you are familiar with. I have had to hear so much from engineers who already know X and refuse to use Y, thus butchering X’s functionality to technically cover for what Y would provide with their precious X. Get out of your comfort zone and get okay with it. Study never stops; it will always have to be done one way or another. And no, there is no such thing as useless knowledge, everything has a use. You can study inefficiently, but you cannot study nothing. And if you are not able to tell yourself how to study efficiently, then you are not allowed to refuse to study inefficiently. Be patient, study, and time will come when you don’t have to question or argue your choice.

I am not here to cover Terraform, but I have to create some common ground for the reader to follow. Terraform is an overengineered tool that allows us to define targeted infrastructure in a declarative way. The declaration is feed to Terraform cli tool, which will create an tree of dependencies that will error-check and predict the end state of our declared infrastructure, creation of resources is handled via APIs of the hosting platform. Meaning, we can tell Terraform how many instances we need, where they will sit, and how they will be configured. Terraform will analyze our meaning and, if possible, realize it by asking AWS to create each required service as needed in the correct order. After our meaning is realized, it will manage the state of created resources and their lifecycle.

This allows us to create very complicated sets of dependencies that can be created within a minute or two and later cleaned up just as fast. This is a very powerful utility to have, especially when modifying declared infrastructure. Where it falls short is in configuring the operating system. Terraform’s bread and butter is calling an API. Creating an instance ? Sure, we can call an API that will tell us if creation is possible and, if so, create it for us. But telling the OS what to do ? Well, there is no API for that. We can feed commands to the instance, but it does not have that exquisite feedback provided by APIs, so we have to keep it very simple. Unfortunately, simple is not enough to build a lab. Here we have to get creative. Normally, we use a lot of different tools to generate configuration files, but due to our limited options, we kind of have to generate files outside of the OS instead and feed the generated files. Does that sound scary? Yeah, it does, I know, but it’s not scary at all. It does get complicated, but trust me on this.

Here I have to introduce another tool: CoreOS, an immutable operating system configured using Ignition files. CoreOS is not actually mandatory for this, but it is certainly fit the usage. This all comes down to the way the OS is configured: CoreOS is an immutable system where configuration needs to be preloaded via Ignition files, and because of this nature, we can, and are forced to, automate services in a simple manner instead of, you know, creating a machine and running a bunch of Ansible/Bash scripts to create a service. No, no, we have to suffer so that what we create (which will never be seen by another person) is a minimalistic marvel that will work years after writing, even though it will be collecting dust somewhere in a private Git before we forget our credentials.

CoreOS will act as our operating system. We will configure it by feeding prerendered configuration files. This allows us to create files, services, transfer data, configure access, and so on. This should raise a question: what if my choice of lab has a binary program that takes control of nodes and configures them appropriately? Here our approach falls short. Running a binary with Terraform is a horrible experience, it can be done, but I don’t recommend it. But this is a non-issue. While it certainly can be a blocker, most of the time we can and should do what this binary does ourselves. Yeah, I know, crazy concept, right? Give me time; I’ll get there.

Another alternative to Ignition type configuration can be CloudInit, but I prefer Ignition for how simple and punishing it can be, either reliably working or simply not.

Making progress

With the introduction and crazy talk out of the way, I want to get in there and demonstrate what I’m on about.

For a demonstration, we will follow a scenario, a scenario to create a very simple Kubernetes cluster. This will allow me to demonstrate my reasoning for why the presented approach is awesome and totally not me coping with the amount of hours I wasted obsessing over stupid stuff. No, this is important and was needed. Everything for education, right? This is what I should be using my time for. No issues here…

Yes, scenario, right. Kubernetes, really a monster of a thing to choose here. I could and should have gone for something simpler, but who cares, I’ll suffer some more. So let’s do a scenario like this: provisioning a simple three-node cluster, bare bones, no networking, no nothing, just a Kubernetes cluster.

Before tackling the Kubernetes cluster, we should create some servers, just a few where we can go in and run stuff.

Servers would need a network, so let’s create that first. The Virtual Private Cloud (VPC) is Amazon’s take on software-defined networks. It allows us to define subnets, assign addresses, configure routing tables, etc. VPC allows us to experiment with network-isolated scenarios, which will also come up in our real work where we have to comply with air-gapped environments for government and enterprise entities. But leaving specific scenarios aside, it will also save us money.

resource "aws_vpc" "lab" {
cidr_block = "10.10.16.0/20"
}
resource "aws_subnet" "instance_subnet" {
vpc_id = aws_vpc.lab.id
cidr_block = "10.10.16.0/24"
}

This gives us the VPC with a defined range of 10.10.16.0-10.10.31.254. Under that, we created an isolated subnet of 10.10.16.0–10.10.16.255, where we will put our instances. Let’s put some EC2 instances inside our glorious VPC.

resource "aws_instance" "lab_instance" {
count = 2
ami = local.ami
instance_type = "t2.micro"
key_name = local.ssh_key
network_interface {
network_interface_id = aws_network_interface.instance_int[count.index].id
device_index = 0
}
}
resource "aws_security_group" "instance_sec" {
name        = "instace_sec"
description = "sec group for the instance server"
vpc_id      = aws_vpc.lab.id
}

resource "aws_network_interface" "instance_int" {
count = 2
subnet_id = aws_subnet.instance_subnet.id
security_groups = [ aws_security_group.instance_sec.id ]
}

Now we have two instances, but they only have internal IPs assigned by the VPC. We cannot really interact with them; they can talk to each other, but that’s pretty much it.

So far, the VPC hasn’t cost us anything, we are only paying for the created instances and a small additional cost for traffic (if generated). Accessing a host can be done in a few ways. For example, we can attach a public IP to the instance, but this will cost an additional $0.50 a month per instance. Fortunately, we are not stuck with this cost.

The problem of available public IPs should be familiar to system and network engineers, and they are also familiar with workarounds such as IPv6, NAT, and VPN services. Let’s focus on NAT (Network Address Translation), where we route traffic via a single public IP. The NAT gateway, where traffic is terminated, tracks calls and records them in a NAT table. Thus, when our internal instance accesses the internet, it will hit the NAT gateway, which in turn takes note of the source IP/port, modifies the source IP to the public IP, and sends the modified packet to the destination IP. Later, when we get a response, the NAT gateway will receive it on the public IP (previously modified source), then look at the NAT table and similarly handle the destination IP, replacing it with the internal IP that matches the previously recorded source port as the current destination port.

AWS provides its own implementation of NAT as a VPC feature, but it costs money, around $34/month ($0.045/hour), plus an additional $0.45 ($0.045 per GB) for 10 GB of traffic. Well, that’s too much money for our little lab. So what can we do as an alternative? Simple: we can create a single instance with a public IP, configure NAT rules on it, and then tell the VPC to route traffic through that instance, which will in turn route traffic to the internet via its own public IP. That should come out to about $5 a month (t2.nano, $0.0058/hour, public IP, $0.005/hour). Little to no money.

We can put this instance in the same subnet, but I will create another subnet called GW, just to have clear roles.

resource "aws_subnet" "gw_subnet" {
vpc_id = aws_vpc.lab.id
cidr_block = "10.10.31.0/24"
}

we also need Public IP, and some routing.

resource "aws_eip" "nat_public_ip" {
instance = aws_instance.gw_instance.id
domain = "vpc"
}

resource "aws_internet_gateway" "internet_gw" {
vpc_id = aws_vpc.lab.id
}

resource "aws_route_table" "gw_routing" {
vpc_id = aws_vpc.lab.id

route {
cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.internet_gw.id
}
}

resource "aws_route_table_association" "gw_rb" {
subnet_id      = aws_subnet.gw_subnet.id
route_table_id = aws_route_table.gw_routing.id
}

resource "aws_security_group" "gw_sec" {
name        = "gw_sec"
description = "sec group for the gw server"
vpc_id      = aws_vpc.lab.id
}
resource "aws_network_interface" "gw_int" {
subnet_id = aws_subnet.gw_subnet.id
private_ips = ["10.10.31.10"]
source_dest_check = false
security_groups = [ aws_security_group.gw_sec.id ]
}

now we can create Gateway instance, and define access rules.

resource "aws_instance" "gw_instance" {
ami = local.ami
instance_type = "t2.micro"
key_name = local.ssh_key
network_interface {
network_interface_id = aws_network_interface.gw_int.id
device_index = 0
}
}

resource "aws_vpc_security_group_ingress_rule" "gw_allow_ssh" {
security_group_id = aws_security_group.gw_sec.id
cidr_ipv4         = "0.0.0.0/0"
from_port         = 22
ip_protocol       = "tcp"
to_port           = 22
}

resource "aws_vpc_security_group_egress_rule" "gw_allow_outbound" {
security_group_id = aws_security_group.gw_sec.id
cidr_ipv4         = "0.0.0.0/0"
ip_protocol       = "-1"
}
resource "aws_vpc_security_group_ingress_rule" "instance_allow_ssh" {
security_group_id = aws_security_group.instance_sec.id
cidr_ipv4         = "10.10.31.0/24"
from_port         = 22
ip_protocol       = "tcp"
to_port           = 22
}

resource "aws_vpc_security_group_egress_rule" "instance_allow_outbound" {
security_group_id = aws_security_group.instance_sec.id
cidr_ipv4         = "0.0.0.0/0"
ip_protocol       = "-1"
}

Okay, cool, now we have an instance that has both a public IP and an internal IP. We can directly SSH into this machine and access our other instances from there, which is cool, but the other instances still don’t have access to the internet. And honestly, accessing one machine just to use it to SSH into yet another machine is annoying.

Let’s address the internet part first, and while we’re at it, let’s also introduce some drawings to make sure my rambling makes a bit more sense.

Here we have three machines and two subnets: the GW machine, which is part of the GW subnet, and two machines that are part of the instance subnet. Leaving access lists aside, the subnets are already aware of each other and able to communicate via a router.

Well, not really, this is SDN, not a legacy network. In an SDN scenario, it’s more of a distributed set of edge devices taking action than a router sitting in the middle. But for the sake of keeping the explanation simple, suspend disbelief and focus on the idea itself.

Now let’s introduce the public IP to the drawings. Previously, I mentioned the GW instance hosting two IPs, this is technically true, but in practice, we are bound by VPC rules. In this case, what we actually get is a single IP on the instance, the internal IP assigned by the VPC.

When we attach a public IP to the instance, AWS goes to an entity called the IGW (Internet Gateway) and puts the assigned IP on it. The IGW acts as a NAT gateway by creating a static NAT rule to translate the assigned public IP to the internal IP of the instance, and vice versa.

The IGW only does static translation of IPs, mapping a single public IP to a single internal IP. To keep costs low, we have to do dynamic translation using an IP/Port combination.

This is where our GW host comes into play. We modify the routing table to send traffic to that host, which will perform dynamic translation and then route it to the IGW, where static translation happens.

So, kind of like this.

Now we have to tell Linux not to drop packets with a destination IP that doesn’t match its own IP, and instead NAT them. This can be done with two commands.

sysctl -w net.ipv4.ip_forward=1
iptables -t nat -A POSTROUTING -o enX0 -j MASQUERADE

Ignition version.

variant: fcos
version: 1.6.0
storage:
files:
- path: /etc/sysctl.d/90-ipv4-ip-forward.conf
mode: 0644
contents:
inline: |
net.ipv4.ip_forward = 1

systemd:
units:
- name: nat-rule.service
enabled: true
contents: |
[Unit]
Description=nat rule
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/iptables -t nat -A POSTROUTING -o enX0 -j MASQUERADE

[Install]
WantedBy=multi-user.target

Okay, now we have a functioning NAT gateway, and our instances have internet access. Let me also drop a link to the GitHub repository with this specific code. github

| | | | | | | Cost | 0.015$(Hour) | | Cost under free tier | free (250 Hours, per month) | | Time to provision lab | minute or two | | Time to Destruct lab | minute or two | | Coammnds to init Lab | Two | | Commands to Create/Destruct Lab | Single |

cost calc: (0.0047(t3a.nano) * 3) + 0.0007(Public IP)

To test the instances, just SSH into instance-01 or instance-02 and try accessing the internet. This can be done by using the GW instance as a jump host:

ssh core@<instance 01/02 internal IP> -J core@<gw public ip>

Improving Experience

Using the GW as a jump host is annoying, doing it for SSH is bearable, but as soon as we introduce web based applications, it becomes unusable.

To solve this, we have to implement a VPN solution. A Virtual Private Network allows us to interact with our private network in a secure manner. This is done by introducing a virtual interface in our operating system; that virtual interface forwards traffic to a local application, which in turn handles routing of said packets to our private network over the internet.

We have a wide variety of VPN solutions. AWS itself offers one, but as expected, it costs money, around $73 a month per user ($0.10 an hour). This is too much. Like with the NAT gateway, we should set up our own VPN service.

WireGuard is one of the best options for our scenario. It is easy to set up, has little to no latency, and provides everything we might need.

Basically, install and configure WireGuard on a server, then install and configure the client on your laptop.

We need an instance to set up this service. To keep costs down, we will run it on the same host we used for NAT. Usually, we have to install the server binary on the host, but CoreOS includes WireGuard binaries out of the box. The only thing left is configuration.

here is a simple wireguard configuration:

[Interface]
PrivateKey =
Address = 10.70.88.0/24
ListenPort = 443

PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o enX0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o enX0 -j MASQUERADE

[Peer]
PublicKey =
AllowedIPs = 10.70.88.101

it contains two sections, Interface and Peer.

• Interface: contains main configuration

• PrivateKey of server, generated with wg genkey

• Address, allowed IP addresses for the peer

• Listening, the Port server will bound to

• PostUp, command run when services starts

• PostDown, command run when services stops

• Peer: contains configuration of peer(Client)

• PublicKey of Peer, generated with wg pubkey

• AllowedIps, list of allowed IPs for specified Peer

and configuration for client,

[Interface]
PrivateKey =
Address = 10.70.88.101/24

[Peer]
PublicKey =
AllowedIPs = 10.10.16.0/24, 10.70.88.0/24
Endpoint = <Public IP>:443

again, contains two sections, Interface and Peer.

• Interface: contains main configuration

• PrivateKey of Client, generated with wg genkey

• Address, Static IP set on Virtual Interface

• Peer: contains configuration of Peer(Server)

• PublicKey of Peer(server) generated with wg pubkey

• AllowedIPs, list of subnetes to be routed via Virtual Interface

• Endpoint, Public IP of server

simple right? basicly same as server, but reverse.

lets put this in work,

variant: fcos
version: 1.6.0
storage:
files:
- path: /etc/sysctl.d/90-ipv4-ip-forward.conf
mode: 0644
contents:
inline: |
net.ipv4.ip_forward = 1

- path: /etc/wireguard/wg0.conf
mode: 0600
contents:
inline: |
[Interface]
PrivateKey =
Address = 10.70.88.0/24
ListenPort = 443

PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o enX0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o enX0 -j MASQUERADE

[Peer]
PublicKey =
AllowedIPs = 10.70.88.10

systemd:
units:
- name: wg-quick@wg0.service
enabled: true

thats it, server is configured.

github with this specific code. github

| | | | | | | Cost | 0.015$(Hour) | | Cost under free tier | free (250 Hours, per month) | | Time to provision lab | minute or two | | Time to Destruct lab | minute or two | | Coammnds to init Lab | tenish | | Commands to Create/Destruct Lab | Single |

cost calc: (0.0047(t3a.nano) * 3) + 0.0007(Public IP)

cost have not increased, init lab commands went up a lot, but that can be easily fixed.

now we can directly access internal IPs of VPC, awesome.

Back to Scenario

Now that we have instances and some idea of how to provision the lab, we can finally focus on the Kubernetes part of the scenario. As our scenario is a bare-bones Kubernetes cluster, let’s hit the source of Kubernetes.

Depending on our understanding of Kubernetes, we will either jump straight to setting it up or just analyze how it works on a theoretical level. I don’t want to explain Kubernetes here, but given the bound complexity and variety of the target actor (Kubernetes), I’m forced to create a leveled playing field. Why did I even pick it…

Anyways, Kubernetes was created to address issues that would emerge during the development and operation of applications. Basically, when we want to develop or run an application, we’re setting up a bunch of resources that will be used by that application, for example: DNS records, SSL certificates, compute, operating systems, networking, etc. Without Kubernetes, we configure all of these separately; most of the time, they have no direct relationship with each other and are controlled by inherently different actors (network, systems, security, etc.).

This caused a lot of issues where the development and distribution of applications slowed to a halt, with plenty of finger-pointing. Kubernetes addresses this by introducing a platform where each such resource is managed by a centralized system that exposes an API to control the process as needed. How Kubernetes achieves this is through complex distributed systems following an outline created by the Kubernetes team. This outline has changed a lot over the years, but the idea remains the same.

Now, with little to no knowledge, let’s try to spin it up. The documentation recommends the kubeadm utility. This utility runs on a node, takes some parameters, and spins up a cluster for us. Nice, right? Well sure, but before I start my rant, let me actually spin up a cluster this way.

The database of Kubernetes has a quorum-type architecture, meaning we need to have an odd number of instances. Let’s address that first and while we are at it move to variables and loops to reduce duplicated Terraform code.

variable "control-instances" {
type = map
default = {
control-01 = { ip = "10.10.16.11" },
control-02 = { ip = "10.10.16.12" },
control-03 = { ip = "10.10.16.13" },
}
}

resource "aws_instance" "control" {
for_each = var.control-instances
ami = local.ami
instance_type = "t2.micro"
key_name = local.ssh_key
network_interface {
network_interface_id = aws_network_interface.control_int[each.key].id
device_index = 0
}
}

resource "aws_network_interface" "control_int" {
for_each = var.control-controls
subnet_id = aws_subnet.control_subnet.id
security_groups = [ aws_security_group.control_sec.id ]
private_ips = [each.value["ip"]]
}

Before running kubeadm, we have a set of prerequisites to handle:

• Swap needs to be off
• kubeadm expects a container runtime to be present
• Network access has to be granted between nodes
• We have to install Kubernetes binaries: kubelet and kubeadm
• Nodes should be able to resolve each other by hostname
• A load balancer for the API

Lucky for us, CoreOS is an operating system optimized for containers, so let’s eliminate some of them.

Swap is off by default; the container runtime is already present as the containerd package (I will use cri-o); and access is controlled via security groups.

This leaves us with the load balancer, Kubernetes binaries, and name resolution.

We can set up the load balancer like the VPN/NAT service or simply use AWS’s ELB. As the ELB service doesn’t add much cost, I’ll go with that.

resource "aws_security_group" "kube_lb" {
name        = "kube_lb"
description = "sec group for the lb server"
vpc_id      = aws_vpc.kube.id
tags = {
Name = "kube_lb"
}
}


resource "aws_vpc_security_group_ingress_rule" "kube_lb_allow_kapi" {
security_group_id = aws_security_group.kube_lb.id
cidr_ipv4         = "10.10.16.0/24"
from_port         = 6443
ip_protocol       = "tcp"
to_port           = 6443
}

resource "aws_vpc_security_group_ingress_rule" "kube_lb_allow_kapi_p" {
security_group_id = aws_security_group.kube_lb.id
cidr_ipv4         = "10.10.31.0/24"
from_port         = 6443
ip_protocol       = "tcp"
to_port           = 6443
}

resource "aws_vpc_security_group_egress_rule" "kube_lb_allow_outbound" {
security_group_id = aws_security_group.kube_lb.id
cidr_ipv4         = "10.10.16.0/20"
ip_protocol       = "-1"
}

resource "aws_lb" "kube" {
name               = "kube"
load_balancer_type = "network"
internal = true
security_groups    = [ aws_security_group.kube_lb.id ]
subnet_mapping {
subnet_id = aws_subnet.instance_subnet.id
private_ipv4_address = "10.10.16.10"
}

}

resource "aws_lb_target_group" "kube-control" {
name        = "kube-control"
port        = 6443
protocol    = "TCP"
target_type = "ip"
vpc_id      = aws_vpc.lab.id
ip_address_type = "ipv4"
health_check {
port     = 6443
protocol = "TCP"
}
}

resource "aws_lb_listener" "kube_api" {
load_balancer_arn = aws_lb.kube.arn
port = 6443
protocol = "TCP"
default_action {
type             = "forward"
target_group_arn = aws_lb_target_group.kube-control.arn
}
}

resource "aws_lb_target_group_attachment" "kube-controlers" {
for_each = aws_network_interface.control
target_group_arn = aws_lb_target_group.kube-control.arn
target_id =  each.value.private_ip
port = 6443
}

For name resolution, we can either spin up a DNS service or use AWS Route 53.

AWS Route 53 is cheap, but it’s charged at a flat rate (I think $0.50?) per month after the first 24 hours are up (up to 24 hours, the service is free and considered as testing). However, the cost can add up(i think ?) considering how many times we’ll create and tear it down. This can be addressed in two ways: either have a permanent DNS zone and tell Terraform to simply add or remove records under it, or remove any zones before the 24-hour mark is up.

Spinning up a DNS service is simple, it can be done similarly to the NAT/VPN setup, but there’s actually a third option: simply generate a hosts file and distribute it. I’ll use that one here.

variant: fcos
version: 1.6.0
storage:
files:
- path: /etc/hosts
mode: 0644
overwrite: true
contents:
inline: |
127.0.0.1 localhost
10.10.16.10 api.kubelius
10.10.16.11 control-01
10.10.16.12 control-02
10.10.16.13 control-03

Installing binaries is a weird experience in CoreOS. The operating system is designed as an immutable system; installing things isn’t as simple as running apt or dnf install. CoreOS recommends not modifying the OS itself, instead to run everything as a layer on top, in simpler words, as containers. But we’re talking about low-level tools that will later allow us to manage containers, so we have to install these packages directly. This can be done either by layering packages onto the base OS (similar to a Dockerfile) or by using OSTree to modify the underlying image.

The OSTree option is simpler in this case, so let’s do that. I don’t want to run the install command manually, so let’s put it in an Ignition config. This is actually kind of tricky, the Butane configs aren’t meant to be used for dynamic automation; they’re more static in nature, like creating files, configuring OS services, and so on. Installing a package is dynamic, we need to run a command that will dynamically analyze dependencies and pull them from the internet.

As a workaround, we can generate a script with our command in it, then create a systemd oneshot service that calls our install script. A oneshot service is meant to run a command that exits after its job is done. To make sure it only runs a single time, we can introduce some persistent state that checks before the script is executed and skips it if needed, for example, creating a file after our script finishes installation and telling the oneshot service to run only when that file is absent.

variant: fcos
version: 1.6.0
storage:
files:
- path: /etc/yum.repos.d/kubernetes.repo
mode: 0644
contents:
inline: |
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.33/rpm/repodata/repomd.xml.key

- path: /etc/hosts
mode: 0644
overwrite: true
contents:
inline: |
127.0.0.1 localhost
10.10.16.10 api.kubelius
10.10.16.11 control-01
10.10.16.12 control-02
10.10.16.13 control-03

- path: /etc/sysctl.d/kube.conf
mode: 0644
contents:
inline: |
net.ipv4.ip_forward=1
systemd:
units:
- name: rpm-ostree-install-kube.service
enabled: true
contents: |
[Unit]
Description=Layer with kubelet and crio
Wants=network-online.target
After=network-online.target
# We run before `zincati.service` to avoid conflicting rpm-ostree
# transactions.
Before=zincati.service
ConditionPathExists=!/var/lib/%N.stamp

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/rpm-ostree install -y --allow-inactive kubelet kubeadm crio
ExecStart=/bin/touch /var/lib/%N.stamp
ExecStart=/bin/systemctl --no-block reboot

[Install]
WantedBy=multi-user.target

Okay, this gives us all the prep, well, kind of. We haven’t told each instance what its hostname is. This is important because the nodes will advertise their hostname as their node name. We’re providing a static file to all control nodes, meaning the hostname will be the same for all of them, we have to address this.

Well, what I’m about to introduce is kind of complicated, especially for something as simple as setting a hostname, but we’ll need to do similar stuff for other parts of the lab, might as well get it over with. What am I talking about? Generating Ignition files. To generate an Ignition file, we write a human-readable declaration in YAML format, then feed this YAML file to the Butane tool, which generates a JSON manifest (Ignition).

We cannot ask Terraform to template YAML. Stop. Let us manually generate the Ignition files from templated YAMLs and ask it to continuou from there.

We have a few options here. The first should be to check if Terraform has an integration for Butane, and indeed, there are providers for it. Unfortunately for us, they are community providers.

The community provider for Terraform gives me anxiety. Why, you ask?

When we get the option of extending functionality via third-party additions, that option needs to be implemented exceptionally. Think of it this way: when we use a third-party provider, we have to make sure it isn’t harmful, intentionally or unintentionally. How?

Well, here are a few ways I can think of:

Simply trust the source of the provider. For example, if a big company is behind the provider, like AWS, we can (kind of) trust it.

Or trust the publisher, in this case, HashiCorp, but that depends on how much responsibility the publisher takes for published content. What do I mean? Well, if HashiCorp simply allows anyone to publish a provider and directs users to use it with caution because they take no responsibility for what’s published, then obviously that option is not viable.

What’s the alternative? Well, let’s look at other community stores. For example, Red Hat hosts the OpenShift Operator Marketplace. The marketplace has tiers of published operators: starting with Community, where Red Hat takes little to no responsibility; then Partners, where Red Hat takes responsibility for verifying the source of operators by partnering with them, probably putting them under contract with restrictions and responsibilities; and finally, the Certified operator, where Red Hat provides guidelines on how the operator needs to be implemented and tested, including review/audit from Red Hat.

We can also trust the engineering, meaning the platform where third-party code runs is engineered in such a way that the code cannot act maliciously without explicit permission. When we install an application on our mobile device, we’re not worried about its code. Why? Because the application is constrained within the OS environment. For example, on iOS, if an application wants to record our voice, it has to ask for permission, and when recording starts, iOS provides a visual indication that recording is taking place. The same goes for the camera, file access, and so on.

I have to clarify that I’m not familiar with how HashiCorp addresses this, but I can say this: with the little research and limited information HashiCorp provides on the topic, I’m not willing to use or recommend community providers.

So, what do I recommend? Well, as always, be a blockhead. Terraform provides the External data source, which allows us to define a wrapper around shell scripts. It calls a local executable file, passes JSON data, and expects JSON data back.

How can we use it here? Simple: we can have Butane installed locally and call it via the External data source, passing the templated Butane config and expecting Ignition data back. Now, this isn’t as simple as calling Butane and piping data, but it’s still straightforward. What complicates things is the way the External data source passes and expects data back. It uses JSON, and JSON isn’t the standard format Unix-type tools use to pass data. Unix tools rely more on raw stdin/stdout data streams. To use this with Butane (or any other program that doesn’t support JSON input/output), we first have to pass that data to something that will convert it to raw data, send it to Butane, capture the output, and return it to the External provider.

Let me demonstrate. First, we have to create a script.

#!/bin/bash
set -e

eval "$(jq -r '@sh "CONFIG64=\(.config64)"')"

BUTANE64=`echo $CONFIG64 | base64 -d | butane | base64`

jq -n --arg config "$BUTANE64" '{"base64":$config}'

this will accept, json, extract it via jq and passit to butane, capture output, and return it as single level json {"base64":"igntion-data"}. now we can interact with this from terraform code via external data source.

like this;

data "external" "ignition_control" {
for_each = var.control-instances
program = ["sh", "scripts/butane.sh"]
query = {
config64 = base64encode(templatefile("templates/control.tftpl", {
name = each.key
}))
}
}

now we can introduce dyamic data to butane config.

variant: fcos
version: 1.6.0
storage:
files:
- path: /etc/hostname
mode: 0644
contents:
inline: |
${name}

and finnaly feed it to instance.

resource "aws_instance" "control" {
for_each = var.control-instances
ami = local.ami
instance_type = "t2.micro"
key_name = local.ssh_key
user_data_base64 = data.external.ignition_control[each.key].result.base64
network_interface {
network_interface_id = aws_network_interface.control_int[each.key].id
device_index = 0
}
}

Now, with the instances prepared, we can start the initialization of the cluster.

kubeadm init --config  /etc/kube-cluster.config  --ignore-preflight-errors=NumCPU,Mem

This will bootstrap the first node of the cluster. After the initial setup is done, it will output instructions to join the other control nodes.

something like copying the certificate files and running the kubeadm join command a token.

ssh core@10.10.16.12 "sudo mkdir -p /etc/kubernetes/pki/etcd/"
ssh core@10.10.16.13 "sudo mkdir -p /etc/kubernetes/pki/etcd/"

files=(
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
/etc/kubernetes/pki/front-proxy-ca.key
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/etcd/ca.key
/etc/kubernetes/pki/etcd/ca.crt
)
mkdir -p /tmp/rsyncc/etc/kubernetes/pki/etcd/

for i in "${files[@]}"; do
rsync --rsync-path="sudo rsync" core@10.10.16.11:$i /tmp/rsyncc$i

rsync --rsync-path="sudo rsync" /tmp/rsyncc$i core@10.10.16.12:$i
rsync --rsync-path="sudo rsync" /tmp/rsyncc$i core@10.10.16.13:$i
done

now we can run join command

kubeadm join api.kubelius:6443 --token <token> --discovery-token-ca-cert-hash <sha> --control-plane --ignore-preflight-errors=NumCPU,Mem

retrive superadmin config.

#add in local hosts file.
#10.10.16.10 api.kubelius
rsync --rsync-path="sudo rsync" core@10.10.16.11:/etc/kubernetes/super-admin.conf /tmp/kubelius.conf
export KUBECONFIG=/tmp/kubelius.conf

And voila, we have a Kubernetes cluster! Granted, it’s only the masters, no networking, ingress, UI, etc… but it’s a fully functional Kubernetes cluster. The other components are additions to Kubernetes, not part of what makes the Kubernetes.

github

| | | | | | | Cost | 0.045$(Hour) | | Cost under free tier | free (180 Hours, per month) | | Time to provision lab | minute or two | | Time to Destruct lab | five minutes | | Coammnds to init Lab | tenish | | Commands to Create/Destruct Lab | Single |

cost calc: (0.0047(t3a.nano) * 4) + 0.0007(Public IP) + 0.025(ELB)

Rant

Let’s start by addressing the obvious annoyance of running commands on nodes to provision a cluster. It gets annoying so fast, and every time I’ve had a similar setup, I’ve dreaded starting it up. Of course, we can automate kubeadm commands, but automating commands on Linux is such a hassle. The cluster will become unstable, not provision correctly, or not function consistently, in short, it won’t solve the underlying issue.

Leaving all of that aside, what did we just provision? How does it work? How are nodes talking to each other? Where is the data stored? How are nodes authenticated? Provisioning the lab certainly didn’t provide any of these details. Creating the infrastructure gave us some information, but to troubleshoot it? To discuss and describe how it’s working? Forget it, when an issue arises, we’ll be left scratching our heads.

This is my main issue with engineers learning systems: we provisioned Kubernetes but have no idea how it works. kubeadm stole precious experience from us. Now, let’s dwell in blockheadedness. This is optional, but what I like to do first is hit a hard tabletop with my head once or twice, just to make the wall-hitting sessions a bit easier down the line.

Now, with a newly acquired (optional) slight head curve, let’s address the elephant of abstraction: what is Kubernetes, and how does it work?

Kubernetes is an API service that takes requests, stores them in a database, and directs the operating system to complete the requested operation. For example, if we ask Kubernetes to run an application, it will take that request via the REST API, record it in etcd, and tell the OS to run a container.

How does Kubernetes do this?

By running a set of small applications, each handling its own specific role. Some of these applications are maintained by the Cloud Native Computing Foundation (the Kubernetes team), while others are third-party tools, either directly used by Kubernetes or later standardized by the community as part of the Kubernetes platform.

The list of applications is entirely dependent on us and what we want from it, but for our bare-bones cluster, it comes down to: etcd, kube-apiserver, kubelet, cri-o, scheduler, and controller(s).

All of this can be found in the official documentation. Now, with this information in mind, let’s talk about provisioning the cluster.

Previously, we used the kubeadm tool. So, what role does kubeadm play in provisioning the cluster?

Well, I could explain it in a few words, but where’s the fun in that? Instead, listen to my unintelligent, borderline deranged rant.

kubeadm automates the provisioning of Kubernetes by running prechecks, configuring the OS, generating certificates, pulling binaries, configuring services, etc.

Why is it needed?

Tricky question here. kubeadm is not needed to create a cluster, but it is needed, and was created (in my opinion), to address the shitty community of engineers.

What am I on about? Well, let me tell you a tale. A tale of a young technology, freshly reheated by Google and presented to the world as a new hot cake. When this microwaved youngling was becoming popular, the term “native Kubernetes” also became popularized.

“Native Kubernetes” was used by wannabe engineers who didn’t understand what Kubernetes was but refused to shut up about it. Think about it this way: I listed a bunch of applications and claimed that they were enough to build a cluster, which is true.

Based on that list, what makes a Kubernetes cluster “native Kubernetes”? If I swapped etcd with another database, would it still be Kubernetes? What about the scheduler?

Leaving swapping things aside, what about adding things? The cluster is bare bones, right? We want networking. What options do we have?

How about Cilium, Calico, Flannel, OVN, Multus… Which one of these is the so-called “native Kubernetes”?

What about the UI? We can install the Kubernetes Dashboard, Rancher, OpenShift Console, K9s… which one of these is “native”?

Stupid, right? The term “native Kubernetes” was never anything logical, it was just engineers with little to no understanding talking about it.

These days, “native Kubernetes” has slowly lost attention. There are a lot of reasons for this, mainly because the community got better, but the CNCF certainly hasn’t just been sitting around. kubeadm is one of the pushes CNCF put out.

What am I talking about? Well, all this “native Kubernetes” talk was hurting Kubernetes.

I had direct experience with this from the standpoint of a company partnered with Red Hat, trying to sell the OpenShift solution. When we went to a client, they always claimed to prefer “native Kubernetes” instead of OpenShift and didn’t want to get locked in. Weirdly, their definition of “native Kubernetes” was another distribution, mainly community ones. This was hurting sales of OpenShift and other enterprise-grade Kubernetes solutions, which in turn hurt the Kubernetes project itself.

You may ask: why would the sales of OpenShift and others hurt Kubernetes? Well, Kubernetes is a community project maintained by the Cloud Native Computing Foundation (CNCF). Who are the contributors, and donors? Companies like Google, Red Hat, Microsoft, IBM, Amazon, etc. The list can be found here.

How did CNCF address this? Well, I’m sure they took several approaches, but here’s one of them:

They introduced the Certified Kubernetes. The program basically allowed CNCF to define guidelines for what a distribution should support to qualify as “Certified Kubernetes.” As a vendor, you have to follow these guidelines and present your distribution for certification.

How did this change the situation? CNCF basically said that if you hold the “Certified Kubernetes” badge, you’re able to host or migrate any application that runs on the so-called “native Kubernetes.”

This allowed sales teams to reference the program whenever a client brought up “native Kubern