TL;DR
Are you tired of copy-pasting and editing a ton YAML files? In this post I suggest using TypeScript to define your services and either Handlebars templates or the Kubernetes NodeJS client to more easily manage your deployments. You can find some sample code that demonstrates this at https://github.com/nabsul/k8s-yaml-alternative.
Introduction
I've been using YAML files checked into a Git repo to manage my Kubernetes deployment for many years now. But as the number of deployments grows, this approach starts to run into problems:
- A lot of the YAML is boilerplate that is repeated over and over again.
- Global changes are tedious, requiring editing all the files in your repo.
I've recently started experimenting with managing my deployments in a different way. In this post I will describe the steps I take to reach this design. The example that I will be using is a cluster running three application:
- An NGINX service with two replicas exposing port 80
- A single instance of NATS exposing ports 4222 and 8222
- A custom application that doesn't have ports but requires some environment variables
Start with your Own Definition
I usually start building out my deployments by first deciding what language/system I will use. For my cluster this has been the YAML files, but it could also have been something like Terraform or Helm charts. With that approach, my application requirements take a back seat and the focus becomes: "What does this system require to work?". Take for example a Kubernetes deployment specified in YAML:
apiVersion: apps/v1
kind: Deployment
metadata:
name: service1
namespace: k8s-ts-test
labels:
app: service1
spec:
replicas: 2
selector:
matchLabels:
app: service1
template:
metadata:
labels:
app: service1
spec:
containers:
- name: service1
image: nginx:latest
ports:
- containerPort: 80
name: port80
What parts of of the above specification do I really care about or control? Number of replicas, image tag, and the application port. That's it, three lines out of twenty. Everything else is boilerplate.
Instead of trying to fit our application into what Kubernetes wants, let's start with a specification that only includes the details that we care about. Based on the initial list of services that I want to run, we can define everything in as follows:
const services = {
service1: {
image: 'nginx:latest',
replicas: 2,
ports: [80]
},
service2: {
image: 'nats:latest',
replicas: 1,
ports: [4222, 8222]
},
service3: {
image: 'app:latest',
replicas: 1,
env: {
VAR1: 'Some Value',
VAR2: 'another value',
}
}
}
Starting with the specification like this keeps me focused on the details that I care about. You can clearly see how many applications I plan to deploy, and how those applications differ from each other. We can worry about YAML/Terraform/Helm details later. Moreover, I can change my mind about the YAML/Terraform/Helm question and still keep this definition as the starting point for everything I do later.
From Custom Definition to YAML
Converting this custom specification into regular YAML can easily be done using Handlebars templates. You can see the all of the templates here, but here is a small sample:
spec:
containers:
- name: {{name}}
image: {{image}}
{{#if ports}}
ports:
{{#each ports}}
- containerPort: {{this}}
name: port{{this}}
{{/each}}
The handlebars library is used in /generate.ts to create all of the YAML in one go.
With this approach notice that:
- All of my deployments are guaranteed to look the same.
- If I need to make a change to all of my services (API version or namespace change for example) I can easily do it with one template change.
You can generate /generated.yaml by running npm run generate.
You can then deploy all of the services in one go with kubectl apply -f generated.yaml.
Skipping YAML Altogether
In this repo I also demonstrate how to eliminate the need for YAML completely.
Instead of generating YAML and running kubectl, I use the @kubernetes/client-node npm package to directly deploy to Kubernetes.
This requires a little more work than YAML generation, but has several advantages.
The biggest advantage is that the npm package includes TypeScript definitions for V1Secret, V1Service, and so on.
This makes your deployment strongly typed and reduces the possibility of errors compared to authoring YAML.
You can follow the /deploy.ts script to see how this all works, but at the heart of it is a simple loop:
for (var [n, s] of Object.entries(services)) {
console.log(`Deploying ${getName(namespace, n)} started`)
await makeSecret(namespace, n, s)
await makeDeployment(namespace, n, s)
await makeService(namespace, n, s)
console.log(`Deploying ${getName(namespace, n)} complete\n`)
}
What Next?
I'm only starting to rethink how I want to manage and deploy my Kubernetes clusters. I think this is a good start, and I hope to share more learnings as I continue to experiment. As next steps I'm going to be looking into removing more of the "click-ops" that I do when creating my cluster:
- Logging into the digital ocean dashboard to create a new cluster
- Manually setting up the load balancer
- Finding the load balancer IP address and configuring DNS to point to the new cluster
- Configuring all the infrastructure (docker repo, AWS) and application secrets
As for challenges I foresee: In this example, the applications are defined generically such that you could use them even if you wanted to skip Kubernetes altogether. However, I have two Kubernetes-specific applications that I run in the cluster: KCert and ecr-login-renewal. These applications require special Kubernetes configurations around service accounts and permissions. I'm not yet sure how to cleanly encode those.
If you like this idea, give it a try in your own setup. If you're comfortable writing code in NodeJS/TypeScript, try out the Kubernetes client approach. Or if you're just looking to simplify your templates, give Handlebars templates a try.