Local Development in Kubernetes

In this second post of my Exploring Kubernetes series we're going to make our k8s development environment less onerous. The code can be found here.

There's continuing debate whether you should local k8s development at all. One camp says that you should develop on a remote cluster in order to mimic production as much as possible. While this has merits, there are two big disadvantages to remote development: cost and availability.

Even many large organizations may not want the financial overhead of a remote development cluster or clusters. Also, with a local set up, you can develop without an internet connection. So just know that with local cluster development we trade economy and convenience for 100% fidelity. Anyway, our development environment (no matter what we do) is never 100% like our production environment, unless we develop in production!

The Problem

We saw at the end of the last post the typical development workflow in k8s:

change app code
build and tag a new image
push the image
set the new image via kubectl set image. This will trigger creation of new pods with the new image, and terminate the existing pods with the old image.

The problems are the manual steps and the delay between when we make changes to the code and when the new code is deployed to the cluster. You might find yourself making shell scripts to automate this, but there are already several battle-proven third-party tools that address this:

I've chosen skaffold because it's maintained by Google, it uses a yaml configuration file, and it's lightweight. After installing skaffold, you can do skaffold init in the hello-k8s directory that we created in the last post. This will generate a skaffold.yaml file which skaffold uses to configure your k8s deployments (both local and remote). Or you can copy mine:

apiVersion: skaffold/v2beta19
kind: Config
metadata:
  name: hello-k8s
build:
  artifacts:
  - image: {USERNAME}/hello-k8s
    docker:
      dockerfile: Dockerfile
deploy:
  kubectl:
    manifests:
    - manifests/*.yaml

The skaffold manifest is self-explanatory. It tells skaffold how to build your images and where your manifests are. skaffold uses this to create your k8s objects, to tag your images, and to deploy them to a local/remote cluster if necessary. skaffold watches your app code and runs through these steps every time your app code changes. Once you press CONTROL-C, skaffold cleans up all the resources it created.

If you are connected to a local cluster, skaffold does not push your image(s) unless you explicitly tell it to.

Once you have skaffold installed and you create your skaffold.yaml file, do:

$ skaffold dev
Listing files to watch...
 - kahunacohen/hello-k8s
Generating tags...
 - kahunacohen/hello-k8s -> index.docker.io/v1/kahunacohen_hello-k8s:f1d95a0-dirty
Checking cache...
 - kahunacohen/hello-k8s: Found Locally
Starting test...
Tags used in deployment:
 - kahunacohen/hello-k8s -> index.docker.io/v1/kahunacohen_hello-k8s:30947c057d58b80ec6a5a202b00cfad7047e5bb2c02e192ea10d7443c8bb9968
Starting deploy...
 - configmap/web-configmap created
 - service/web-service created
 - deployment.apps/web-deployment created
Waiting for deployments to stabilize...
 - deployment/web-deployment is ready.
Deployments stabilized in 2.107 seconds
Press Ctrl+C to exit
Watching for changes...
[web] Listening on http://localhost:3000

Skaffold has created all your k8s objects for you, and it's watching your files.

Change the heading in server.js. You'll see that skaffold catches the change. Within ten seconds or so you should see your change after refreshing the browser (remember minikube service web-service to launch your browser if you haven't already).

There's still a lag, but at least you don't have to manage the deploy manually or create scripts. You'll also notice that after five minutes, you'll start seeing output from the cronjob in the terminal.

skaffold does much more, including automating tests, tagging images, pushing images image repos etc. Learn more here.

In the next part of this series, we're going to learn how to manage secrets in k8s.