Demonstrating a Production Application Platform with the Reference Platform for Kubernetes Project

Page content

Introduction

Recently, I was able to get involved with an internal project while working at VMware that allowed a group of field engineers to help to enable the broader Kubernetes community by developing a method to allow consumers of said project to demonstrate what a production viable application platform may look like. Specifically, those who have been working with Kubernetes for a while understand the potholes and the decisions that go into producing a production viable platform. However, Kubernetes has been emerging and gaining steam recently, and new users are often faced with the questions like:

  1. What would this look like in Production?
  2. What makes up a Production viable platform?
  3. There are so many projects out there that could make up an application platform, which ones should I use?

This blog is set to outline the findings of that project and demonstrate how users can quickly get hands on with a platform which demonstrates what a production platform might look like.

The project was initially referred to as “Project Ivy”, but is now avaiable on GitHub via the name “Reference Platform for Kubernetes” or RPK (https://github.com/vmware-tanzu-labs/reference-platform-for-kubernetes).

Beginning the Project

Getting Started

There are so many different projects within the CNCF landscape, how could we possibly settle on what our platform looked like to achieve the highest resonance with our consumers? At the time, we had an internal initiative to document field best practices and opinions. That internal initiative outlined each “building block” within a production-style platform in an extremely well-organized manner. The decision we made was to align our platform with the opinions and projects of our internal initiative. That initiative is now openly available via the Tanzu Developer Center at https://tanzu.vmware.com/developer/guides/kubernetes/.

In addition, we had to take into account that helm install <package> would not be a viable methodology, as helm does not have a good mechanism to resolve and deploy dependencies and to orchestrate the integration between its charts/packages. Helm is a fantastic tool for quickly installing individual components, however it relies on user input to a values file which makes it difficult to orchestrate the integration between components.

Another area of focus for RPK was layering components on top of a pre-existing Kubernetes installation. An anti-goal of the project was coming up with another way of installing Kubernetes, rather to focus on the ability to extend a day zero Kubernetes platform.

Finally, we had to understand that not all two platforms will look the same, so users would need a way to easily install individual components as well as customize which components would be installed when deploying their sample platform.

Lessons Learned

  1. Align opinions with documented best practices (https://tanzu.vmware.com/developer/guides/kubernetes/)
  2. Pick most commonly used components to fit the 80% use case
  3. Strong opinions on what each component contained to avoid increased scope
  4. Orchestrate integration between components is key
  5. Resolve dependencies between components is key
  6. Focus on layering components on post-installation

Tooling

Orchestration (Ansible)

Very quickly, we learned that we were a talented group of people with vastly different backgrounds. As such, selecting the tooling would be a challenge. Furthermore, the tooling should be easy enough for us to use and/or learn quickly, as well as produce the outcome of demonstrating a sample production-viable application platform atop Kubernetes.

We landed as Ansible as our tooling of choice to give us the best balance of:

  1. Robust orchestration language
  2. Tool which matches the background of the individuals involved
  3. Tool which quickly allows for a path to success (internal project do not last forever!)
  4. Tool which would allow us to meet above requirements

Deployment (Ansible)

In addition to orchestration, Ansible offers a k8s suite of modules to deploy manifests to clusters and to do things like wait for said resources in manifests to be completely deployed before continuing. Although tools like Carvel’s kapp does solve for this use case, we wanted to avoid tool overload and focus on making the automation obvious as to what was happening, rather than introduce extra complexities where it may be unneeded.

Starting Point (Manifests)

This was the real challenge. We wanted to ensure that we didn’t reinvent the wheel, so we had to scrounge the internet for helm charts and internal manifests which met our needs, and adjust them to our liking. At the time we started, we accepted that we would be managing those manifests for the life of the project. However, there are other tools out now like Carvel’s vendir which would have allowed us to declaratively specify the state of our manifests directory, given an upstream source.

CLI (Make/Bash/Docker)

We wanted to keep this incredibly simple. As such, we chose to use simple make targets that would run a docker command to instantiate a versioned image which would use a bash script as an entrypoint to initiate our Ansible playbook. This seemed like the simplest way short of writing our own custom CLI using something like Cobra for GoLang.

Distribution and Developer Experience (Docker)

We had the additional challenge of noting that Ansible, by way of Python, has an inconsistent way of managing library dependencies (e.g. MacOS doesn’t play the same as RHEL, etc), even in sitatuions where Python virtual environments are used. Additionally, how do we control which version of Ansible is used and supported by the project?

This made the choice easy, which allowed us to double down on the decision to use Docker as an entrypoint for execution of RPK, as we could simply create a Dockerfile for our project and package our dependencies and entrypoint script together in a convenient image. An added benefit is that users only need 2 tools to be installed to use RPK:

  1. Bash w/ Make (commonly distributed in most *nix distributions by default)
  2. Docker

Using the Tool

Install the Default Platform

As stated initially, the “Default Platform” needed to be quick and easy to deploy and offer a quick path to allowing users to see what a sample platfrom might look like. The steps to do so are relatively simple:

  1. Clone the repository and navigate to the directory:
git clone git@github.com:vmware-tanzu-labs/reference-platform-for-kubernetes.git
cd reference-platform-for-kubernetes
  1. Instantiate the Ansible inventory, which is also used as a global variable store:
make setup.<aws|vmware|azure|kind>
  1. Review the sensible defaults in the inventory, being sure to input the kubectl config context in which you wish to use and reviewing which profile is being deployed (default: platform):
vim build/inventory.yaml

...
tanzu_kubectl_context: "my-kubeconfig-kubectl-context"
rpk_profile:           "platform"
...
  1. Review the components being installed by the profile:
vim profiles/platform.yaml
  1. Deploy RPK!!!:
make deploy

Install a Single Component

Considering steps 1-3 above from above, deploy the component AND its dependencies in which you wish to install to your platform:

  1. List the components:
make list.components.all
  1. Review the dependencies which get installed with the component:
vim roles/components/<core|extensions>/<component_name>/.dependencies.yaml
  1. Deploy the component:
ROLE=<component_name> make deploy.role

Install a Custom Profile

Considering steps 1-3 above from above, create a custom profile with the components in which you wish to install:

  1. List the components:
make list.components.all
  1. Add components to your profile:
vim profiles/components.yaml

...
  - name:    "workload-tenancy"
    enabled: true
    profiles:
      - "platform"
      - "my-custom-profile
...
  1. Build your profile:
make build.profiles
  1. Add the profile to your inventory:
vim build/inventory.yaml

...
rpk_profile: "my-custom-profile"
...
  1. Deploy your profile:
make deploy

Summary

Thanks for reading! Feel free to contribute to RPK after reviewing our contributing guide!