5 min readJust now

–

Eliminate Static AWS Credentials On-Prem with the EKS Pod Identity Webhook

Press enter or click to view image in full size

What is the EKS Pod Identity Webhook?

The EKS Pod Identity Webhook is a mutating webhook that injects a service account token and its associated IAM role into a pod. This allows the pod to authenticate with the AWS STS endpoint using the injected token, which is verified against a configured OIDC identity provider. Once verified, short-lived credentials are issued to the pod, granting access to the AWS resources defined by the associated IAM policies.

See the EKS Pod Identity Webhook GitHub for more info.

Why use the EKS Pod Identity Webhook?

For many on-premise services, it’s necessary to interact with AWS resources. For instance, in my cluster, I use DynamoDB as the backend for Vault and a KMS key for auto-unseal. I also use S3 buckets as targets for CloudNativePG backups. Creating long-lived access keys and IDs introduces operational overhead for key rotation and increases secret sprawl at best. More often, these secrets end up overly privileged and rarely, if ever, rotated. By leveraging the Pod Identity Webhook, we achieve production-ready authentication and authorization with AWS from on-premises environments.

How does it all work?

Press enter or click to view image in full size

By hosting your Kubernetes API’s .well-known OIDC endpoint in a public S3 bucket, AWS STS can verify the identity of your service account tokens. I’ve published a GitHub repository with a Makefile that demonstrates how to get this running on KIND — including running aws sts get-caller-identity successfully from within a pod.

A working example with KIND

This tutorial assumes you have KIND working on your machine as well as the awscli configured credentials to create S3 buckets, identity providers and manage IAM.

This document supplements the repo eks-pod-identity-webhook-tutorial

Feel free to just run the make file in the repo with make all to create a working example. What follows is a detailed explanation of all the pieces involved to make the webhook work.

Create a kind cluster — step 1

make create creates a KIND cluster and configures the service-account-issuer with a hostname that you manage. You must be able to create a CNAME with this service-account-issuer value.

kind: ClusterapiVersion: kind.x-k8s.io/v1alpha4name: oidckubeadmConfigPatches:- |  apiVersion: kubeadm.k8s.io/v1beta3  kind: ClusterConfiguration  apiServer:    extraArgs:      service-account-issuer: "https://oidc.example.com"nodes:  - role: control-plane

kind create cluster --name kind-oidc --config /tmp/kind-config.yaml

Get the OpenId configuration and prep for the identity provider

make fetch-oidc fetches the OpenId configuration endpoints and prepares them to be used as a public identity provider. They should look like this.

cat oidc_bucket/.well-known/openid-configuration {  "issuer": "https://oidc.example.com",  "jwks_uri": "https://oidc.example.com/openid/v1/jwks",  "response_types_supported": [    "id_token"  ],  "subject_types_supported": [    "public"  ],  "id_token_signing_alg_values_supported": [    "RS256"  ]}cat oidc_bucket/openid/v1/jwks | jq{  "keys": [    {      "use": "sig",      "kty": "RSA",      "kid": "xkW35chyStGopSQuW6Y_-rPosnPn__XdjMi8srhyJlo",      "alg": "RS256",      "n": "9Aa5-o3ug6xyhB-3oPaFQzKJJaAkdSat5e2-FG5jdpfnEmwDSVhGzyY-DIv3w6gaDPhWhbKWKL3pzukFqwnDPGqOKB0Jh38bW-yyvu5l_beL2jZ0TtYiDMqyU9-cm30ATqr9EgTY4SwIHura2JC3M-Qa4DtM9nLcU5ozpuAL0hy54EZsxhgOTRcNlsXcQna8TCw1HuT5376k9WRpRqDYN39xqQe68sfjvKPvzTcV794JUAFP61zJEGcg1kPKexLkO2PiBmy8BxxZzjPis2UArVMPzyAtKdWXE9GW9PWe7H8nUu5t1HmlEXh215AVZeiAnPRepnkLAtX5DX7-N1AR5w",      "e": "AQAB"    }  ]}

Create a public S3 bucket and sync the openid configuration

make create-bucket will create a bucket using the name of your service-account-issuer. It does not create the CNAME for you. It will instruct you to do that manually.

Please create a CNAME oidc.example.com => https://oidc.example.com.s3.us-west-2.amazonaws.comEnter to continue after creating dns...

After you’ve created the bucket you then configure this address as an identity provider.

make create-oidc-provider

Test it all out…

Now it’s necessary to create a role that can use our identity provider. Here is a an example policy that can assume there the oidc-kind-role from the **default **namespace as the **test-sa **service account.

make create-test-role preps the oidc-kind-role to be used with the pod identity webhook. We first add the assume-role-policy such that it can use sts:AssumeRoleWithWebIdentity and then specify which namespace and service account can use it.

{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Principal": {                "Federated": "arn:aws:iam::012345678910:oidc-provider/oidc.example.com"            },            "Action": "sts:AssumeRoleWithWebIdentity",            "Condition": {                "StringEquals": {                    "oidc-kind.anacondas.io:sub": "system:serviceaccount:default:test-sa"                }            }        }    ]}

And we attach a policy so this role can call sts get-caller-identity

{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Action": "sts:GetCallerIdentity",            "Resource": "*"        }    ]}

Now everything is in place on the infrastructure side to have a container capable of assuming the oidc-kind-role from the KIND cluster.

Install cert-manager and amazon-eks-pod-identity-webhook

The next step installs, configures and tests the webhook with a basic call to aws.

make install-webhook

Install cert-manager

helm repo add jetstack https://charts.jetstack.io helm repo update helm install \  cert-manager jetstack/cert-manager \  --namespace cert-manager \  --create-namespace \  --version v1.9.1 \  --set installCRDs=true

Install eks-pod-identity-webhook

helm repo add jkroepke https://jkroepke.github.io/helm-charts/ helm repo update helm install amazon-eks-pod-identity-webhook \  jkroepke/amazon-eks-pod-identity-webhook \  -n amazon-eks-pod-identity-webhook \  -f values.yaml \  --create-namespace

Create the service account and test deployment.

apiVersion: v1kind: ServiceAccountmetadata:  creationTimestamp: null  name: test-sa  namespace: default  annotations:    eks.amazonaws.com/role-arn: arn:aws:iam::012345678910:role/oidc-kind-role---apiVersion: v1kind: Podmetadata:  name: aws-cli  namespace: defaultspec:  serviceAccountName: test-sa  containers:    - name: aws      image: amazon/aws-cli:2.15.0      command: ["sleep"]      args: ["infinity"]  restartPolicy: Never

From there we can verify everything is working…

kubectl exec -n default aws-cli -- aws sts get-caller-identity{    "UserId": "AROARHJJNFDQ2J5TOICZM:botocore-session-1760475439",    "Account": "084375578849",    "Arn": "arn:aws:sts::012345678910:assumed-role/oidc-kind-role/botocore-session-1760475439"}

😮‍💨 Success! It worked!

Setting up a self-hosted EKS Pod Identity Webhook can feel daunting at first, but once configured, you can sit back and enjoy production-level authentication and authorization in your homelab or on-prem Kubernetes clusters. Rotating access IDs and secret keys becomes a thing of the past.

The included Makefile demonstrates a working example of the EKS Pod Identity Webhook. It’s highly recommended to integrate the infrastructure components into your IaC stack and deploy cert-manager and the Pod Identity Webhook using your GitOps tool of choice, such as ArgoCD or Flux.