3 min read4 days ago

–

Running Kubernetes on nodes with two network interfaces often leads to subtle routing issues — especially when the OS is a minimal cloud image and K3s picks the wrong interface for the cluster network. This article documents the exact configuration that reliably separates internal cluster traffic from public internet traffic using systemd-networkd on Debian 12.

This guide applies to:

• Single-NIC nodes (172.20.1.x)
• Dual-NIC nodes (172.20.1.x internal + 192.168.6.x public)
• K3s clusters using Flannel

Press enter or click to view image in full size

🎯 Goals

We want:

1. Internal LAN traffic (172.20.1.0/24, 172.20.0.0/16) → use gateway 172.20.1.254
2. Kubernetes pod & cluster CIDR traffic (10.0.0.0/8) → also use gateway 172.20.1.254
3. All public internet traffic → use gateway 192.168.6.1
4. Flannel must bind to ens18, the internal NIC.

Without the last step, pods on dual-NIC nodes are unreachable from other nodes.

1. Disable cloud-init networking

Minimal Debian cloud images often override your networking unless you disable it:

echo "network: {config: disabled}" | sudo tee /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg

Reboot afterward.

2. Configure Internal NIC (ens18)

Create /etc/systemd/network/10-eth0-cluster.network

[Match]Name=ens18[Network]Address=172.20.1.57/24IPForward=yes# Route all internal traffic through 172.20.1.254[Route]Destination=172.20.0.0/16Gateway=172.20.1.254

This ensures:

• All internal networking and inter-node traffic flows through the internal gateway
• Kubernetes control-plane traffic stays on the right NIC

3. Configure Public NIC (ens19)

Create /etc/systemd/network/20-eth1-internet.network

[Match]Name=ens19[Network]Address=192.168.6.16/25Gateway=192.168.6.1DNS=172.20.1.116IPForward=yes

This interface handles only:

• Public internet egress
• Package downloads
• External load balancer traffic

At this point:

✔ ping 172.20.1.x works ✔ ping 192.168.6.x works ✔ ping websites works

But… K3s traffic still won’t work properly.

4. Fix: Tell K3s Which Interface to Use

By default, K3s (Flannel) automatically chooses an interface. On dual-NIC systems, it often picks ens19 (public NIC), breaking inter-pod networking.

Solution: explicitly set Flannel to bind to ens18.

Edit /etc/systemd/system/k3s-agent.service

Add:

--flannel-ifaceens18

Your final ExecStart section is:

ExecStart=/usr/local/bin/k3s \    agent \        '--node-label' 'group=platform' \        '--node-label' 'environment=vm' \        '--node-label' 'provider=pve' \        '--node-label' 'owner=scc' \        '--node-label' 'net=public' \        '--node-label' 'location=suzhou.scc' \        '--node-ip' '172.20.1.57' \        '--flannel-iface' 'ens18'

Reload and restart:

sudo systemctl daemon-reloadsudo systemctl restart k3s-agent

5. Result

After applying the above configuration:

✔ Pods on dual-NIC nodes become reachable from other nodes

Flannel now uses 172.20.1.x instead of 192.168.6.x.

✔ Kubernetes internal networking works

All 10.x.x.x traffic routes correctly via 172.20.1.254.

✔ Internet traffic stays on the public NIC

No interference with cluster traffic.

✔ Routing table is clean and predictable

No asymmetric routing or pod unreachable issues.

Conclusion

Dual-NIC Kubernetes nodes require carefully separating the routing domains. The key steps are:

1. Use systemd-networkd to explicitly declare gateways per NIC.
2. Add a route for internal subnets to the internal gateway.
3. Tell K3s/Flannel to use — flannel-iface=ens18.

This ensures reliable inter-node communication and stable pod networking.