Lesson 11: Monitoring and Logging in Kubernetes

Monitoring and logging are crucial aspects of managing Kubernetes clusters and applications. They help you keep track of the health of your cluster and the performance of your applications.

Why Monitor and Log?

Monitoring allows you to observe the state of your applications and infrastructure in real-time, while logging provides a historical record of events and errors. Together, they enable you to: - Identify performance bottlenecks - Troubleshoot issues - Ensure reliability and availability

Key Components of Monitoring and Logging

1. Metrics: Quantitative measures of system performance (e.g., CPU usage, memory usage).
2. Logs: Textual records of events that occur within your applications and infrastructure.
3. Alerting: Notifications based on specific conditions or thresholds.

Setting Up Monitoring with Prometheus

Prometheus is a popular open-source monitoring tool designed for reliability and scalability. Here's how to set it up in your Kubernetes cluster:

Step 1: Install Prometheus

You can use the Prometheus Operator to simplify the deployment. First, create a namespace for monitoring:

kubectl create namespace monitoring

Then, apply the Prometheus Operator manifest:

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml -n monitoring

Step 2: Deploy Prometheus Instance

Create a prometheus.yaml file:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      app: my-app
  resources:
    requests:
      cpu: 100m
      memory: 400Mi
  retention: 10d

Apply the Prometheus instance:

kubectl apply -f prometheus.yaml

Step 3: Accessing Prometheus

To access the Prometheus UI, you can port-forward the service:

kubectl port-forward svc/prometheus-k8s 9090 -n monitoring

Visit http://localhost:9090 in your web browser to see the Prometheus dashboard.

Setting Up Logging with Fluentd and Elasticsearch

Fluentd is a data collector that helps you unify logging. Elasticsearch is a search and analytics engine that stores logs.

Step 1: Install Elasticsearch

You can deploy Elasticsearch using Helm:

helm repo add elastic https://helm.elastic.co
helm install elasticsearch elastic/elasticsearch -n logging

Step 2: Install Fluentd

Create a fluentd-configmap.yaml file to configure Fluentd:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging
data:
  fluent.conf: |
    <source>
      @type kubernetes
      @id input_kubernetes
      @log_level info
      <parse>
        @type json
      </parse>
    </source>
    <match **>
      @type elasticsearch
      @id output_elasticsearch
      host elasticsearch.logging.svc.cluster.local
      port 9200
      logstash_format true
    </match>

Apply the ConfigMap:

kubectl apply -f fluentd-configmap.yaml

Step 3: Deploy Fluentd

Deploy Fluentd as a DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.8.0-debian-ubuntu-20.04
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch.logging.svc.cluster.local"
        volumeMounts:
        - name: fluentd-config
          mountPath: /fluentd/etc/fluent.conf
          subPath: fluent.conf
      volumes:
      - name: fluentd-config
        configMap:
          name: fluentd-config

Apply the Fluentd DaemonSet:

kubectl apply -f fluentd-daemonset.yaml

Best Practices

• Use Labels: Label your resources to make it easier to query metrics and logs.
• Set Alerts: Configure alerts for critical metrics to proactively manage issues.
• Log Rotation: Implement log rotation to manage disk space.

Common Mistake: Not aggregating logs can lead to difficulties in troubleshooting. Always centralize logs for easier access.

Summary

• Monitoring and logging are essential for maintaining Kubernetes applications.
• Use Prometheus for metrics collection and alerting.
• Use Fluentd and Elasticsearch for centralized logging.
• Follow best practices to enhance observability and manageability of your applications.

• Exercise 1: Set up Prometheus in your Kubernetes cluster and monitor a sample application. Create a service monitor for it.
• Exercise 2: Deploy Fluentd and Elasticsearch in your cluster. Verify that logs from your applications are being collected and stored.
• Exercise 3: Configure alerts in Prometheus for high CPU usage and test the alerting mechanism.

• Monitoring and logging are critical for application performance and reliability.
• Prometheus is used for metrics collection and alerting.
• Fluentd and Elasticsearch provide a powerful logging solution.
• Centralizing logs and metrics helps in troubleshooting and performance tuning.