3 min readFeb 3, 2025

–

Press enter or click to view image in full size

Kubernetes Monitoring

Monitoring Kubernetes clusters is crucial for maintaining performance, troubleshooting issues, and ensuring availability. This guide provides a step-by-step approach to deploying a full monitoring stack in Azure Kubernetes Service (AKS) using Grafana, Loki, and Prometheus, with persistent storage configured using an Azure Storage Account.

Why This Setup?

• Loki: A horizontally scalable, highly available log aggregation system.
• Grafana: A powerful tool for visualizing logs and metrics.
• Prometheus: A leading open-source monitoring and alerting toolkit.
• Azure Storage: Provides durable and scalable storage for logs and metrics.

This combination is perfect for monitoring AKS clusters in production environments.

Step 1: Verify Available Storage Classes in AKS

Azure Kubernetes Service provides multiple storage classes. To check the available ones, run:

kubectl get sc

🔹 We will use azureblob-nfs-premium, as it provides reliable storage for Loki, Prometheus, and Grafana.

Step 2: Deploy Loki with Persistent Storage

We’ll start by deploying Loki using Helm. Loki will use Azure Storage for persistent storage.

loki-values.yaml

loki:  enabled: true  persistence:    enabled: true    storageClassName: azureblob-nfs-premium    size: 10Gigrafana:  enabled: true  image:    tag: latest  persistence:    enabled: true    size: 10Gi    storageClassName: azureblob-nfs-premium  sidecar:    datasources:      enabled: true

2.1 Install Loki using Helm

helm repo add grafana https://grafana.github.io/helm-chartshelm repo updatehelm install loki-new-release grafana/loki-stack -f loki-values.yaml -n monitoring --create-namespace

Loki is now installed in the monitoring namespace!

Step 3: Deploy Prometheus with Persistent Storage

Next, we’ll deploy Prometheus for metrics collection.

prometheus-values.yaml

prometheus:  persistence:    enabled: true    storageClassName: azureblob-nfs-premium    size: 10Gi

Deploy Prometheus

helm install prometheus prometheus-community/kube-prometheus-stack -f prometheus-values.yaml -n monitoring

This command installs Prometheus with persistent storage using the same azureblob-nfs-premium storage class.

Step 4: Add Azure Monitor as a Data Source

To integrate Azure Monitor with Grafana, create a service principal with the Monitoring Reader role.

Create Service Principal

az ad sp create-for-rbac --role="Monitoring Reader" --scopes="/subscriptions/--id--/resourceGroups/rg-aks-dev"

Use the credentials from the service principal to add Azure Monitor as a data source in Grafana.

Step 5: Configure SMTP for Grafana Alerts

To enable email alerts in Grafana, configure the SMTP settings.

Edit Grafana ConfigMap

kubectl edit configmap loki-new-release-grafana -n monitoring

Add the following SMTP configuration:

[smtp]enabled = truehost = <<smtp>>:587 user = <<email>> password = <<Password>> from_address = <<email>>from_name = Grafanaskip_verify = falsestarttls = true

This configuration allows Grafana to send email alerts.

Step 6: Access Grafana Dashboard

Retrieve the Grafana admin password:

kubectl get secret loki-new-release-grafana -n monitoring -o jsonpath="{.data.admin-password}" | base64 --decode; echAccess the Grafana dashboard using the AKS cluster’s external IP or domain.

Explore Logs in Grafana

Press enter or click to view image in full size

Navigate to the “Explore” section in Grafana (left sidebar). In the top-left dropdown, select “Loki” as the data source. Then select the namespace or filters according to your requirements.

Conclusion

By following these steps, you’ve set up a comprehensive monitoring solution for your AKS cluster using Loki, Grafana, Prometheus, and Azure Storage. This setup ensures that you have scalable, reliable, and efficient monitoring for your Kubernetes workloads.

Feel free to ask any questions in case of any issue or confusion. You can customize this setup further based on your specific requirements. Happy monitoring!