Logging and Monitoring in Kubernetes

1 min read

🧭 Introduction

When you run applications on Kubernetes, you need to observe what’s happening inside your cluster — how much CPU and memory your nodes and pods are consuming, and what logs your containers are generating. Without visibility, troubleshooting becomes guesswork. This post covers both monitoring and logging in Kubernetes — focusing on Metrics Server, cAdvisor, and kubectl tools for real-world usage.

📊 Monitoring Cluster Components

Kubernetes doesn’t store metrics by default. It relies on Metrics Server and cAdvisor for gathering and exposing metrics.

🧠 What is Metrics Server?

Metrics Server is a cluster-wide aggregator of resource usage data. It collects CPU and memory metrics from each node and pod through Kubelet’s Summary API, which in turn gathers data from cAdvisor (Container Advisor).

These metrics are used by:

The kubectl top command.
The Horizontal Pod Autoscaler (HPA).
The Vertical Pod Autoscaler (VPA).

Unlike Prometheus, Metrics Server doesn’t store historical data — it only provides current point-in-time metrics.

⚙️ Installing Metrics Server

Let’s install Metrics Server directly from the official GitHub repository.

# Clone the Metrics Server repo (optional)
git clone https://github.com/kubernetes-sigs/metrics-server.git
cd metrics-server

# Apply the components YAML directly from GitHub
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

After installation, verify it’s working:

kubectl get deployment metrics-server -n kube-system
kubectl get apiservice | grep metrics

If it’s not reporting metrics (common in self-hosted clusters), you might need to patch the deployment to allow insecure TLS connections to Kubelets:

kubectl patch deployment metrics-server -n kube-system   --type='json'   -p='[{
    "op": "add",
    "path": "/spec/template/spec/containers/0/args/-",
    "value": "--kubelet-insecure-tls"
  }]'

🧩 How cAdvisor Works with Metrics Server

cAdvisor (Container Advisor) runs as part of Kubelet on every node. It collects real-time metrics about containers, such as:

CPU usage
Memory usage
Filesystem I/O
Network traffic

Here’s how the data flows:

Container → cAdvisor → Kubelet Summary API → Metrics Server → kubectl top / HPA

Kubernetes Resource Metrics Pipeline

cAdvisor doesn’t store metrics permanently. It’s a runtime component that feeds live data to the Kubelet, which the Metrics Server aggregates.

🗂️ Metrics Server Data Storage

Metrics Server doesn’t persist metrics. It keeps them in-memory, refreshing periodically. It’s lightweight by design — meant for autoscaling decisions, not long-term monitoring.

If you need long-term metrics storage and visualization, tools like Prometheus and Grafana are your next step.

🧮 Using `kubectl top`

Once Metrics Server is running, you can use kubectl top to check resource usage.

For Nodes:

kubectl top nodes

For Pods:

kubectl top pods

With Sorting:

# Sort nodes by CPU usage
kubectl top nodes --sort-by=cpu

# Sort pods by memory usage
kubectl top pods --sort-by=memory

For Specific Namespace:

kubectl top pods -n kube-system

For a Specific Pod:

kubectl top pod my-pod -n default

These commands help you quickly spot which nodes or pods are consuming the most resources.

📜 Monitoring Applications (Logs)

Metrics give you numbers, but logs give you stories — the who, what, and why behind your app’s behavior.

🔍 Viewing Pod Logs

To view logs from a specific pod:

kubectl logs <pod-name>

If a pod has multiple containers, you must specify the container:

kubectl logs <pod-name> -c <container-name>

🌀 View Logs of All Containers in a Pod

Since Kubernetes v1.10, you can use --all-containers=true:

kubectl logs <pod-name> --all-containers=true

This is super useful for multi-container pods like sidecars (e.g., Envoy, Fluentd, etc.).

⚙️ Useful `kubectl logs` Options

Option	Description
`--previous`	Shows logs from the previously terminated container in the pod.
`--tail=<N>`	Displays only the last N lines of logs.
`--since=5m`	Shows logs from the last 5 minutes.
`--timestamps`	Includes timestamps in the output.
`-f` or `--follow`	Streams logs live (similar to `tail -f`).
`--limit-bytes=<N>`	Limits the total bytes of logs shown.

Example:

kubectl logs -f my-pod --since=10m --tail=50 --timestamps

This streams the last 50 log lines from the last 10 minutes — perfect for real-time debugging.

🧩 Combining Monitoring and Logging Tools

While kubectl top and kubectl logs are handy, they’re just the beginning.

In production environments, you’d typically add:

Prometheus → For advanced metrics and alerting.
Grafana → For beautiful visual dashboards.
ELK Stack (Elasticsearch, Logstash, Kibana) or EFK (Fluentd) → For centralized log storage and search.
Loki + Promtail + Grafana → Lightweight alternative to ELK.

These tools integrate with Kubernetes seamlessly and scale horizontally with your workloads.

🧾 Summary

Component	Purpose	Persistence
cAdvisor	Collects container-level metrics	No
Kubelet Summary API	Provides node/pod metrics to Metrics Server	No
Metrics Server	Aggregates live resource metrics	No
Prometheus	Stores and queries metrics over time	Yes
kubectl logs	Shows container logs	Yes (short-term in kubelet log rotation)
EFK Stack	Centralized logging and visualization	Yes

🚀 Final Thoughts

Start small — install Metrics Server and get comfortable with kubectl top and kubectl logs. Once you understand the basics, move on to Prometheus and Grafana for a full observability stack.

Remember, monitoring tells you when something’s wrong. Logging tells you why. You need both to run a healthy Kubernetes cluster.

📚 References

💡 Pro tip: Use aliases like klogs and ktop in your shell to speed up your debugging workflow.