Testing Kubernetes Manifests with CI/CD

Page content

Introduction

A common pattern that consumers of Kubernetes apply is to store their manifests in a git repository. This provides a way to document (by way of YAML) the configuration of the applications that reside on the platform, and also enables an easy onramp to things like CI/CD.

One of the main purposes of CI/CD is to ensure that the code (in this case manifests) being delivered conform to standards, is free of errors, and quickly makes its way into a production ready state. It eliminates a lot of the things that we usually do as developers and platform operators like applying the manifests to a Kubernetes test cluster, prior to applying them into a production environment.

The purpose of this article is to show you how you can insert testing of the application of manifests inline with your current CI/CD processes. This would allow you to smoke test an application prior to it’s deployment to production.

Tools

For this walkthrough, I will be using Gitlab with Gitlab CI as my CI/CD tool of choice. The framework can apply to multiple other CI tools as well, but as my code is already stored in a Gitlab instance, Gitlab CI is a natural choice for my needs. Additionally, I will be using KIND (Kubernetes-in-Docker) in a pipeline to quickly spin up a Kubernetes cluster for testing.

Configuration

Create our Manifests

For this walkthrough, I simply want to show two (2) manifests. One is configured correctly, and the other has mismatched selector labels.

Good manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      containers:
      - image: nginx:latest
        name: nginx

Bad manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      containers:
      - image: nginx:latest
        name: nginx

Configure KIND in a Pipeline

Gitlab CI uses a .gitlab-ci.yml file to declare what it’s pipeline is. In this case, I simply want to tell the tool to deploy a KIND (Kuberentes-in-Docker) cluster first, and then deploy the manifests when my Kubernetes cluster is available.

Here is what the configuration looks like to run KIND in a Docker-in-Docker image:

#kind-config.yml
---
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
networking:
  apiServerAddress: "0.0.0.0"

# add to the apiServer certSANs the name of the docker (dind) service in order to be able to reach the cluster through it
kubeadmConfigPatchesJSON6902:
  - group: kubeadm.k8s.io
    version: v1beta2
    kind: ClusterConfiguration
    patch: |
      - op: add
        path: /apiServer/certSANs/-
        value: docker
nodes:
  - role: control-plane
  - role: worker
    extraPortMappings:
    - containerPort: 30000
      hostPort: 30000

Here is what the code looks like to deploy KIND using Gitlab CI:

#.gitlab-ci.yml
---
variables:
  KIND_VERSION: 'v0.7.0'
  KUBECTL_VERSION: 'v1.17.3'

stages:
  - 'smoke_test'
  - 'deploy'

#
# smoke test the kubernetes manifests
#
smoke_test:
  stage: 'smoke_test'
  image: docker:19.03.8
  services:
    - docker:19.03.8-dind
  tags:
    - 'gitlab-org-docker'
  script:
    - apk add -U curl build-base

    # setup kubectl
    - |
      if [ -z $(which kubectl) ]; then
        echo "kubectl executable not found...installing"
        curl https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl -L
        chmod +x /usr/local/bin/kubectl
      else
        echo "found kubectl executable at $(which kubectl)...not installing"
      fi

    # setup kind
    - |
      if [ -z $(which kind) ]; then
        echo "kind executable not found...installing"
        curl https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-linux-amd64 -o /usr/local/bin/kind -L
        chmod +x /usr/local/bin/kind
      else
        echo "found kind executable at $(which kind)...not installing"
      fi

    # create our kind cluster
    - kind create cluster --name ${CI_PROJECT_ID}
    - sed -i -E -e 's/localhost|0\.0\.0\.0/docker/g' "$HOME/.kube/config"

    # apply our manifests
    - kubectl apply -f manifests/

#
# deploy the manifests into production
#
deploy:
  stage: 'deploy'
  image: 'bitnami/kubectl:1.18.6-debian-10-r9'

  # don't actually deploy for this walkthrough
  script:
    - echo "kubectl apply -f manifests/"

It is important to note above that we must use tags: 'gitlab-org-docker' to instruct Gitlab to run this CI workflow on a Docker enabled host. We accomplish the KIND install by running a Docker-in-Docker instance for KIND and we need a priviliged container to do this.

Run Pipeline

Of course, with any DevOps model, a commit of our source code (in this case YAML manifests) will trigger the pipeline to run automatically. That commit will represent a change in the manifest and will allow our pipeline to let us know if we have made a good or bad change.

Pipeline with Good Manifest

In a normal operating scenario, our manifests are golden and our pipeline run passes without problems.

Pipeline Success:

Passing Pipeline

Pipeline with Bad Manifest

When we introduce a change that is bad, we get an error in the pipeline and a nice graphic displaying our workflow showing that the deploy didn’t happen due to a problem with the smoke test stage.

Error:

Error from server (Invalid): error when creating "manifests/bad-deploy.yaml": Deployment.apps "nginx" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"nginx"}: `selector` does not match template `labels`
Error from server (NotFound): error when creating "manifests/good-deploy.yaml": namespaces "nginx" not found

Pipeline Fail:

Failing Pipeline

Summary

CI tools can provide a nice workflow to test and ensure your manifests are proper before deploying them into a more sensitive environment, such as production. The above is an overly simplistic example of how providing a smoke test cluster prior to deploying to any managed infrastructure could be used. This can be extended as well to other tools (e.g. helm) as well.