Kubernetes Observability with EFK

Page content

Welcome

I’ve decided recently that I would like to be a part of the crowd that populates the Internet with even more information that is currently existing today. As such, I have decided that I will be posting information pertaining to things that I learn throughout my career in tech, in hopes that other may find the information helpful. As there are an overabundance of blogs out there already, my goal is to simplify things and explain things in more understandable terms. As such, I will try and develop different tiers of blogs which contain either high-level (101-series) down to low-level (501-series) in hopes that I can reach all different types of audiences!

My first blog will include one of the most important aspects of managing a Kubernetes environment, and that is how to observe and monitor the platform and applications that are running on the platform.

Motivation

Why do we need to monitor and observe the platform? Well, for obvious reasons, us IT folks have jobs today due to things that fail and need to be fixed. In order for us to fix those things, we need a variety of accesses to the aforementioned failed systems including system logs. Infrastructures today are also highly-sprawled, containing many different flavors of applications, running on different versions, running in different datacenters, running on different flavors of operating systems, and likely running on different types of infrastructure and infrastructure providers. Additionally, the shear number of systems can be so overwhelming that we need a centralized information source to consolidate the data from these sources.

As it stands now, Kubernetes is the next big thing as of this writing. My motivation is to give a simple view into this common use case, and how to accomplish it in Kubernetes.

Common Tools

There are myriads of different ways that people monitor Kubernetes. One of the most popular ways is through the use of the EFK stack. EFK consists of three (or four) main components and is named that way due to its underlying components. The following make up the EFK stack:

  • Elasticsearch: the data storage and search engine for logs.
  • Fluentd: a log aggregator.
  • Fluent Bit: a log collector and forwarder.
  • Kibana: the web interface which interacts with elasticsearch and allows users to visualize its data.

You notice above that Fluent Bit and Fluentd make up the F in the EFK stack. I point this out because there are several patterns that I’ve seen. That pattern can make use of both Fluentd and Fluent Bit to accomplish observability in Kubernetes. That pattern can also make use of either to accommplish observability in Kubernetes. To keep it simple, I don’t want to talk about use cases and why you would chose one or the other, I simply want to show you how to deploy a common, simple pattern of Elasticsearch, Fluent Bit, and Kibana. There are plenty of examples of different models and how to choose the appropriate one.

Prerequisites

Before we get started, it is always important to understand what items need to be in place before we can accomplish anything.

  1. A running Kubernetes cluster
  2. The kubectl context set to the correct cluster that we will deploy our application into (see kubectl config set-context)
  3. Helm installed (see Helm Quickstart Guide if you are unfamiliar)

Step 1: Deploy Elasticsearch

The first thing we want to do is ensure that our data storage mechanism is available before we start forwarding information to it. To accomplish this, I will be using a tool called Helm.

Setup the repository. This tells us where we can obtain our “charts” from. I’m using Bitnami charts as I feel they have the most flexible deployment options for Elasticsearch. Many others exist as well.

helm repo add bitnami https://charts.bitnami.com/bitnami

Ensure the we have Elasticsearch available to us:

helm search repo bitnami/elasticsearch
NAME                 	CHART VERSION	APP VERSION	DESCRIPTION
bitnami/elasticsearch	11.0.12      	7.6.1      	A highly scalable open-source full-text search ...

Create a elasticsearch/helm-values.yml file. Values files in Helm allow us the ability to customize the application. There are many different way to customize Elasticsearch, so I do not want to overwhelm anyone with that data. The values that are available for the deployment can be found here.

DISCLAIMER: this is not meant for a production type deployment. Other, more detailed, considerations should be taken rather than using this as a step-by-step guide.

Here is an example file for our deployment, stored at ./elasticsearch/helm-values.yml.

---
image:
  registry: docker.io
  repository: bitnami/elasticsearch
  tag: 7.6.1-debian-10-r22
client:
  replicas: 1
master:
  replicas: 1
  persistence:
    storageClass: efk-master
    size: 5Gi
cluster:
  env:
    MINIMUM_MASTER_NODES: 1
    RECOVER_AFTER_MASTER_NODES: 1
    EXPECTED_MASTER_NODES: 1
data:
  replicas: 1
  heapSize: 300m
  persistence:
    storageClass: efk-data
    size: 5Gi

Additionally, we need some storage available that match our storageClass above. We want our data to persist in the instance our Elasticsearch container(s) get destroyed, so we will create persistent volume resource definitions to accomplish this at ./elasticsearch/pv/pv-data.yml and ./elasticsearch/pv/pv-master.yml:

DISCLAIMER: I am using hostPath for this walkthrough, however a more reliable source should be used for Production deployments. In this scenario, we are assuming that the location of the host path exists on the worker nodes.

---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: efk-data-volume
  labels:
    type: local
spec:
  storageClassName: efk-data
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt/data
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: efk-master-volume
  labels:
    type: local
spec:
  storageClassName: efk-master
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt/master

Finally, we likely want to namespace this off from our other resources. To do so, we will create a namespace definition at ./common/ns-logging.yml.

---
kind: Namespace
apiVersion: v1
metadata:
  name: logging
spec:

In order to create the object in Kubernetes, we will run a few commands:

# create the namespace
kubectl create -f ./common/ns-logging.yml

# create the persistent volumes
kubectl create -f ./elasticsearch/pv/

# deploy the helm chart
helm install elasticsearch bitnami/elasticsearch --namespace=logging -f elasticsearch/helm-values.yml

We will assume that our elasticsearch deployment is successful when all objects are available and running:

kubectl get all -n logging
NAME                                                                READY   STATUS    RESTARTS   AGE
pod/elasticsearch-elasticsearch-coordinating-only-bb898d758-6fb8d   1/1     Running   0          113s
pod/elasticsearch-elasticsearch-coordinating-only-bb898d758-sbpzq   1/1     Running   0          113s
pod/elasticsearch-elasticsearch-data-0                              1/1     Running   0          113s
pod/elasticsearch-elasticsearch-master-0                            1/1     Running   0          113s

NAME                                                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/elasticsearch-elasticsearch-coordinating-only   ClusterIP   10.99.129.164    <none>        9200/TCP   113s
service/elasticsearch-elasticsearch-discovery           ClusterIP   None             <none>        9300/TCP   113s
service/elasticsearch-elasticsearch-master              ClusterIP   10.102.122.195   <none>        9300/TCP   113s

NAME                                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/elasticsearch-elasticsearch-coordinating-only   2/2     2            2           113s

NAME                                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/elasticsearch-elasticsearch-coordinating-only-bb898d758   2         2         2       113s

NAME                                                  READY   AGE
statefulset.apps/elasticsearch-elasticsearch-data     1/1     113s
statefulset.apps/elasticsearch-elasticsearch-master   1/1     113s

Step 2: Deploy Fluent Bit

For this 101 series blog, we will skip Fluentd and simply make sure that our containers are forwarding their log information to Elasticsearch.

Before we start, I want to point out that we need to know where to forward the logs to. Luckily, the helm chart we deployed in the last step created a service that we can use to forward logs to. It is described below:

kubectl describe svc elasticsearch-elasticsearch-coordinating-only -n logging

Name:              elasticsearch-elasticsearch-coordinating-only
Namespace:         logging
Labels:            app=elasticsearch
                   chart=elasticsearch-11.0.12
                   heritage=Helm
                   release=elasticsearch
                   role=coordinating-only
Annotations:       <none>
Selector:          app=elasticsearch,release=elasticsearch,role=coordinating-only
Type:              ClusterIP
IP:                10.99.129.164
Port:              http  9200/TCP
TargetPort:        http/TCP
Endpoints:         10.40.0.2:9200,10.44.0.2:9200
Session Affinity:  None
Events:            <none>

Before we deploy the application we need to enable the repo. For this, we can use the standard stable helm chart repo:

helm repo add stable https://kubernetes-charts.storage.googleapis.com/

Create a fluent-bit/helm-values.yml file. The values that are available for the deployment can be found here.

DISCLAIMER: this is not meant for a production type deployment. Other, more detailed, considerations should be taken rather than using this as a step-by-step guide.

Here is an example file for our deployment, stored at ./fluent-bit/helm-values.yml. It should be noted that we are using a special toleration for this application deployment to allow the Fluent Bit application to pull logs from the master node. Without the toleration, the DaemonSet created will not be able to tolerate the taint of the master node and we will be left without the ability to pull logs from the master node.

image:
  fluent_bit:
    repository: fluent/fluent-bit
    tag: 1.4.1
backend:
  type: es
  es:
    host: 'elasticsearch-elasticsearch-coordinating-only'
tolerations:
  - operator: Exists

Finally, we can deploy Fluent Bit. This will ensure that a pod exists on each of our nodes and will begin the process of forwarding our logs to the Elasticsearch service:

helm install fluentbit stable/fluent-bit --namespace=logging -f fluent-bit/helm-values.yml

We can ensure that data is making it into our Elasticsearch cluster by running a command from within the Elasticsearch pod:

kubectl -n logging exec elasticsearch-elasticsearch-coordinating-only-bb898d758-6fb8d -- curl -XPOST "http://elasticsearch-elasticsearch-coordinating-only:9200/_search" -d '{"query": {"match_all": {}}}' -H 'Content-Type: application/json'

Step 3: Deploy Kibana

Finally, once we have confirmed that we get data into our cluster, we need a way to visualize it. Kibana is the tool that we use to provide an intuitive web interface so that we can get insight into our data.

This is the easiest step in the process. Create a kibana/helm-values.yml file. The values that are available for the deployment can be found here. Of note, we are exposing the Kibana service outside of our cluster on port 31000. This allows us to access the Kibana dashboard easily by accessing the IP address or hostname of any of our Kubernetes nodes at the port in which we’ve exposed, which is 31000 in this instance.

DISCLAIMER: this is not meant for a production type deployment. Other, more detailed, considerations should be taken rather than using this as a step-by-step guide.

Here is an example file for our deployment, stored at ./kibana/helm-values.yml.

---
image:
  registry: docker.io
  repository: bitnami/kibana
  tag: 7.6.1-debian-10-r21
elasticsearch:
  hosts:
    - elasticsearch-elasticsearch-coordinating-only
  port: 9200
service:
  type: NodePort
  nodePort: 31000

Once we have set the parameters for the Kibana deployment, we can deploy it with Helm:

helm install kibana bitnami/kibana --namespace=logging -f kibana/helm-values.yml

Step 5: Access the Kibana Dashboard

We can now access our dashboard by typing in the IP or Hostname of any of our Kubernetes nodes using the port in which we specified for the NodePort in the previous step. Once we setup an Index Pattern, we can view our logs which will look like so:

Kibana Interface

View This Demo on GitHub

If you feel like following along, feel free to check out the code used for this blog in GitHub at https://github.com/scottd018/ihazcloudthings_demos/tree/master/efk-stack.