Testing Kubernetes Manifests with CI/CD
Introduction
A common pattern that consumers of Kubernetes apply is to store their manifests in a git repository. This provides a way to document (by way of YAML) the configuration of the applications that reside on the platform, and also enables an easy onramp to things like CI/CD.
One of the main purposes of CI/CD is to ensure that the code (in this case manifests) being delivered conform to standards, is free of errors, and quickly makes its way into a production ready state. It eliminates a lot of the things that we usually do as developers and platform operators like applying the manifests to a Kubernetes test cluster, prior to applying them into a production environment.
The purpose of this article is to show you how you can insert testing of the application of manifests inline with your current CI/CD processes. This would allow you to smoke test an application prior to it’s deployment to production.
Tools
For this walkthrough, I will be using Gitlab with Gitlab CI as my CI/CD tool of choice. The framework can apply to multiple other CI tools as well, but as my code is already stored in a Gitlab instance, Gitlab CI is a natural choice for my needs. Additionally, I will be using KIND (Kubernetes-in-Docker) in a pipeline to quickly spin up a Kubernetes cluster for testing.
Configuration
Create our Manifests
For this walkthrough, I simply want to show two (2) manifests. One is configured correctly, and the other has mismatched selector labels.
Good manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx
namespace: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
name: nginx
spec:
containers:
- image: nginx:latest
name: nginx
Bad manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: test
template:
metadata:
labels:
app: nginx
name: nginx
spec:
containers:
- image: nginx:latest
name: nginx
Configure KIND in a Pipeline
Gitlab CI uses a .gitlab-ci.yml
file to declare what it’s pipeline is. In this case, I simply
want to tell the tool to deploy a KIND (Kuberentes-in-Docker) cluster first, and then deploy
the manifests when my Kubernetes cluster is available.
Here is what the configuration looks like to run KIND in a Docker-in-Docker image:
#kind-config.yml
---
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
networking:
apiServerAddress: "0.0.0.0"
# add to the apiServer certSANs the name of the docker (dind) service in order to be able to reach the cluster through it
kubeadmConfigPatchesJSON6902:
- group: kubeadm.k8s.io
version: v1beta2
kind: ClusterConfiguration
patch: |
- op: add
path: /apiServer/certSANs/-
value: docker
nodes:
- role: control-plane
- role: worker
extraPortMappings:
- containerPort: 30000
hostPort: 30000
Here is what the code looks like to deploy KIND using Gitlab CI:
#.gitlab-ci.yml
---
variables:
KIND_VERSION: 'v0.7.0'
KUBECTL_VERSION: 'v1.17.3'
stages:
- 'smoke_test'
- 'deploy'
#
# smoke test the kubernetes manifests
#
smoke_test:
stage: 'smoke_test'
image: docker:19.03.8
services:
- docker:19.03.8-dind
tags:
- 'gitlab-org-docker'
script:
- apk add -U curl build-base
# setup kubectl
- |
if [ -z $(which kubectl) ]; then
echo "kubectl executable not found...installing"
curl https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl -L
chmod +x /usr/local/bin/kubectl
else
echo "found kubectl executable at $(which kubectl)...not installing"
fi
# setup kind
- |
if [ -z $(which kind) ]; then
echo "kind executable not found...installing"
curl https://github.com/kubernetes-sigs/kind/releases/download/${KIND_VERSION}/kind-linux-amd64 -o /usr/local/bin/kind -L
chmod +x /usr/local/bin/kind
else
echo "found kind executable at $(which kind)...not installing"
fi
# create our kind cluster
- kind create cluster --name ${CI_PROJECT_ID}
- sed -i -E -e 's/localhost|0\.0\.0\.0/docker/g' "$HOME/.kube/config"
# apply our manifests
- kubectl apply -f manifests/
#
# deploy the manifests into production
#
deploy:
stage: 'deploy'
image: 'bitnami/kubectl:1.18.6-debian-10-r9'
# don't actually deploy for this walkthrough
script:
- echo "kubectl apply -f manifests/"
It is important to note above that we must use tags: 'gitlab-org-docker'
to instruct
Gitlab to run this CI workflow on a Docker enabled host. We accomplish the KIND install by
running a Docker-in-Docker instance for KIND and we need a priviliged container to do this.
Run Pipeline
Of course, with any DevOps model, a commit of our source code (in this case YAML manifests) will trigger the pipeline to run automatically. That commit will represent a change in the manifest and will allow our pipeline to let us know if we have made a good or bad change.
Pipeline with Good Manifest
In a normal operating scenario, our manifests are golden and our pipeline run passes without problems.
Pipeline Success:
Pipeline with Bad Manifest
When we introduce a change that is bad, we get an error in the pipeline and a nice graphic displaying our workflow showing that the deploy didn’t happen due to a problem with the smoke test stage.
Error:
Error from server (Invalid): error when creating "manifests/bad-deploy.yaml": Deployment.apps "nginx" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"nginx"}: `selector` does not match template `labels`
Error from server (NotFound): error when creating "manifests/good-deploy.yaml": namespaces "nginx" not found
Pipeline Fail:
Summary
CI tools can provide a nice workflow to test and ensure your manifests are proper before deploying them into a more sensitive environment, such as production. The above is an overly simplistic example of how providing a smoke test cluster prior to deploying to any managed infrastructure could be used. This can be extended as well to other tools (e.g. helm) as well.