Creating a Mutating Webhook in OpenShift

By Mark DeNeve

If you have ever used tools like Istio, or OpenShift Service Mesh, you may have noticed that they have an ability to modify your Kubernetes deployments automatically injecting “side-cars” into your application definitions. Or perhaps you have come across tools that add certificates to your deployment, or add special environment variables to your definitions. This magic is brought to you by Kubernetes Admission Controllers. There are multiple types of admission controllers, but today we will focus on just one of them, “Mutating Webhooks”. Mutating Webhooks are the specific class of Admission Controller that can inject changes into your Kubernetes definitions.

This blog post will be making use of a simple Go based application to act as our Webhook, and we will be using OpenShift API and RBAC controls to deploy it into OpenShift. The code and sample app can also be used in a non-OpenShift Kubernetes cluster, but you will need to handle the certificate handling and RBAC in another way.

When researching this blog post I found many examples of creating a mutating webhook, or an admission controller that used the admissionregistration.k8s.io/v1beta1 APIs that were released a few years ago. Since then Kubernetes has continued to evolve, and as of Kubernetes 1.22, admissionregistration.k8s.io/v1beta1 has been not only deprecated, but removed (see the Kubernetes 1.22 deprecation guide for details). This means that existing example code will no longer work in Kubernetes 1.22 or later. The code used in this example has been updated to use the admissionregistration.k8s.io/v1 API and should be supported within Kubernetes for the foreseeable future.

The Goal

In order to show what a Mutating Webhook is, and how it works, we need a goal in mind. For this blog post, we are going to create a Mutating Webhook that adds a small side-car to any application that is annotated with sidecar-injector-webhook.xphyr.net/inject: “yes”. (More on this shortly.) We will inject the following side-car which contains the tcpdump utility which will allow us to better inspect traffic going into and out of our pod:

containers:
  - name: sidecar-tcpdump
    image: corfr/tcpdump
    imagePullPolicy: IfNotPresent
    command:
      - /bin/sleep
      - infinity

Keep in mind, there are much better ways to enable tcpdump on a pod. If you are here looking for a way to use tcpdump in your cluster I would suggest you see the excellent article TCPDump for OpenShift Workloads on the Cloud Cult DevOps site.

WARNING

WARNING The use of Admission Controllers and Mutating Webhooks can have an adverse effect on your OpenShift (or Kubernetes) cluster. By adding any admission controller into your cluster you are adding a failure point to the deployment of any pod or container in your cluster. You must take caution in how you deploy and configure your admission controllers to ensure that you do not adversely effect the stability of your cluster.

Consider yourself Warned!

Prerequisites

We will be creating a sample webhook application from source, so you will at a minimum need the following tools available:

  • git
  • docker or podman
  • oc client version v4.9+
  • Access to a OpenShift 4.9 or newer cluster with the admissionregistration.k8s.io/v1 API enabled.

We can validate that your cluster has the admissionregistration.k8s.io/v1 API enabled using the following command:

$ oc api-versions | grep admissionregistration.k8s.io
admissionregistration.k8s.io/v1

If the admissionregistration.k8s.io/v1 API is not listed, check to ensure that your cluster is running OpenShift 4.9+ or Kubernetes 1.22+

$ oc version
Client Version: 4.9.8
Server Version: 4.9.21
Kubernetes Version: v1.22.3+fdba464

The Code

The code we will be using is a fork of kube mutating webhook tutorial. The source repo for this blog post is https://github.com/xphyr/kube-mutating-webhook-tutorial. It has been updated to use the current APIs available in Kubernetes. The program is made up of two files main.go and webhook.go. The main.go file sets up the http web server required to run the webhook, the real work is done inside the webhook.go file.

We will not go through the entire webhook.go file, but I want to point a few key functions out:

  1. mutationRequired() - this function checks to see if it should apply a mutation. This is what ensures that we only modify objects in a specified namespace, or namespaces, and only those objects that have an annotation “sidecar-injector-webhook.xphyr.net/inject: yes”. It also ensures that we only apply a mutation once.
  2. createPatch() - this function creates a patch that Kubernetes will apply to the pod. The webhook does not directly edit the deployment, but must supply a patch that will be applied to the kubernetes object by kubernetes itself
  3. mutate() - this is the main function of the application and pulls the other functions together in order to achieve the mutation

A Note about Annotations:

The use of “sidecar-injector-webhook.xphyr.net” as well as other annotations in this blog post that reference xphyr.net is arbitrary. If you use this example code as a base, you will want to ensure that you use something that is relevant and UNIQUE to your use case, and update as appropriate.

Building the example code

NOTE: The instructions below are for building your own copy of this example webhook locally. You can also use the container image published in Quay or GHCR without having to build your own copy if you would like. If you wish to try out the prebuilt example, skip to the Deployment section below.

Building in a Container

This project is set up to allow building the application inside a container so yo do not need to have the Go toolchain installed. You can use Podman or Docker to build the container image. In this blog post we will be using Podman.

Start by checking out the code from github:

$ git clone https://github.com/xphyr/kube-mutating-webhook-tutorial.git
Cloning into 'kube-mutating-webhook-tutorial'...
remote: Enumerating objects: 250, done.
remote: Counting objects: 100% (79/79), done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 250 (delta 30), reused 65 (delta 23), pack-reused 171
Receiving objects: 100% (250/250), 242.13 KiB | 1.24 MiB/s, done.
Resolving deltas: 100% (114/114), done.
$ cd kube-mutating-webhook-tutorial

With the code cloned locally, we can build the application using containers. We will be using a “multistage” Dockerfile that runs the build for our application in one container, and then uses the results from the first container to populate the second container. Additional information on running a multistage Dockerfile can be found here: Build your Go image Run the following command to build our container:

$ podman build -f Dockerfile.multistage -t mwhexample:latest .
[1/2] STEP 1/7: FROM golang:1.17-buster AS build
[1/2] STEP 2/7: WORKDIR /app
--> Using cache 400ea9b25da7730b8cc40d317b6274d898ec5e68a94f9472cdacaf9042675716
--> 400ea9b25da
[1/2] STEP 3/7: COPY go.mod ./
...
[1/2] STEP 7/7: RUN go build -o /mwh-tutorial
--> 107ce1573f1
[2/2] STEP 1/7: FROM gcr.io/distroless/base-debian10
[2/2] STEP 2/7: ENV SIDECAR_INJECTOR=/usr/local/bin/sidecar-injector   USER_UID=1001   USER_NAME=sidecar-injector
--> Using cache b520ffbe03b36eb9a338393f8d907c568e780d15217fbea40e7218260cacc28e
--> b520ffbe03b
...
[2/2] STEP 7/7: ENTRYPOINT ["/mwh-tutorial"]
[2/2] COMMIT mwhexample:latest
--> 19616f6fc07
Successfully tagged localhost/mwhexample:latest
19616f6fc079831ab7f469a4906bc74f73db8a7136ba0975f08ce5ce77111eaa

Building Locally

You can also build the code locally on your machine without the use of containers.

$ go build -o build/_output/linux/bin/mwh-tutorial ./cmd/
$ podman build -f build/Dockerfile -t mwhexample:latest .

Push Containerimage to Registry

Once the container is built, you will need to publish the container image to an image repository. Your steps may vary based on your particular container registry, in the example below we will push to quay.io.

$ podman tag mwhexample:latest quay.io/xphyr/mwhexample:latest
$ podman login quay.io
Authenticating with existing credentials for quay.io
Existing credentials are valid. Already logged in to quay.io
$ podman push quay.io/xphyr/mwhexample:latest

Our mutating webhook application is now ready for deployment.

Deployment

Now that we have a container image to work with, we can deploy the Mutating Webhook.

To start, we will create a new project called “sidecar-injector” to install our mutating webhook into:

$ oc login
Authentication required for https://api.ocp49.xphyrlab.net:6443 (openshift)
Username: markd
Password: 
Login successful.
$ oc new-project sidecar-injector
Now using project "sidecar-injector" on server "https://api.ocp49.xphyrlab.net:6443".

With the new project created, we need to setup the security required to make this all work. This means the creation of a service account as well as the proper RBAC controls to allow the webhook to work.

$ oc create -f deploy/serviceaccount.yaml
serviceaccount/mutator created
$ oc get sa
NAME       SECRETS   AGE
builder    2         2m37s
default    2         2m37s
deployer   2         2m37s
mutator    2         43s
$ oc auth reconcile -f deploy/rbac.yaml
clusterrolebinding.rbac.authorization.k8s.io/auth-delegator-mutator reconciled
clusterrole.rbac.authorization.k8s.io/xphyr-net-mutatator reconciled
clusterrole.rbac.authorization.k8s.io/xphyr-net-mutatator reconciled
clusterrolebinding.rbac.authorization.k8s.io/mutator-mutating-webhook reconciled
rolebinding.rbac.authorization.k8s.io/extension-mutator-authentication-reader-mutator reconciled
clusterrole.rbac.authorization.k8s.io/mutator-role reconciled
clusterrolebinding.rbac.authorization.k8s.io/mutator-role reconciled

So what did we just do? As part of our overall deployment process, we created a serviceAccount as well as a set of cluster roles, and role bindings that will be used to control access to the webhook, and prevents token information from other API servers from being disclosed to the webhook. Our webhook application will run as this service account.

With the proper roles and service account in place, we will create a Kubernetes service. In order to ease the deployment of our webhook we will use a feature built into OpenShift called “service.beta.openshift.io/inject-cabundle”, which will handle the creation of our x509 certificates and make them available as a secret in the current namespace. By adding an annotation service.beta.openshift.io/serving-cert-secret-name: webhook-certs OpenShift will automatically create a TLS certificate and store it in the secret that we defined called “webhook-certs”.

$ oc create -f deploy/service.yaml
service/sidecar-injector-webhook-svc created
$ oc get secrets
NAME                       TYPE                                  DATA   AGE
webhook-certs              kubernetes.io/tls                     2      27s
$ oc describe secret/webhook-certs
Name:         webhook-certs
Namespace:    sidecar-injector
Labels:       <none>
Annotations:  service.alpha.openshift.io/expiry: 2024-02-23T16:59:21Z
             service.beta.openshift.io/expiry: 2024-02-23T16:59:21Z
             service.beta.openshift.io/originating-service-name: sidecar-injector-webhook-svc
             service.beta.openshift.io/originating-service-uid: a2eb4897-6950-4fb0-9cd1-d2659de1edab

Type:  kubernetes.io/tls

Data
====
tls.crt:  2709 bytes
tls.key:  1679 bytes

We will now deploy our application. This will consist of configuration data for the side-car we want to inject as well as the deployment of the application itself. If you created your own version of the webhook application be sure to update the image definition in the deployment.yaml file before proceeding.

$ oc create -f deploy/configmap.yaml
configmap/sidecar-injector-webhook-configmap created
$ oc create -f deploy/deployment.yaml
deployment.apps/sidecar-injector-webhook-deployment created
$ oc get pods
NAME                                                  READY   STATUS    RESTARTS   AGE
sidecar-injector-webhook-deployment-689c48cd6-jhfrg   1/1     Running   0          16s

We can now create a Kubernetes API service that will be used by the webhook configuration. This API service is what will help to connect our webhook application, with the internal admission controllers. The APIService is annotated with service.beta.openshift.io/inject-cabundle=true which will populate its spec.caBundle field with the service CA bundle. This allows the Kubernetes API server to validate the service CA certificate used to secure the targeted endpoint.

$ oc create -f deploy/apiservice.yaml
$ oc get apiservice.apiregistration.k8s.io/v1.admission.xphyr.net
NAME                     SERVICE                                         AVAILABLE   AGE
v1.admission.xphyr.net   sidecar-injector/sidecar-injector-webhook-svc   True        90s

Ensure that the api service shows as being “AVAILABLE” before continuing.

With all the pieces in place to create our new webhook, we can define the webhook itself. Review the yaml for deploy/mutatingwebhook.yaml, and take note of the webhooks.rules section which defines what actions our webhook will be called on as well as the webhooks.namespaceSelector section which defines which namespaces (Projects) our webhook should apply to. The webhooks.namespace section is particularly important, as this is how we ensure that we do not break our cluster by applying this webhook cluster wide, but only to specific namespaces we want.

$ oc create -f deploy/mutatingwebhook.yaml
mutatingwebhookconfiguration.admissionregistration.k8s.io/sidecar-injector-webhook-cfg created

CONGRATULATIONS you have deployed your first webhook.

Testing

So now what? Well lets try it out. First, lets open a new terminal window, and we will watch the logs for our webhook application while we deploy a test application. In the new terminal window run the following commands to watch the logs from our pod. Be sure to update the command to point to your pod. This command will continue to run and show the logs coming from our pod in real time:

$ oc get pods
NAME                                                  READY   STATUS    RESTARTS   AGE
sidecar-injector-webhook-deployment-689c48cd6-jhfrg   1/1     Running   0          17m
$ oc log sidecar-injector-webhook-deployment-689c48cd6-jhfrg --follow

Now, lets create a new namespace that we will use to test our webhook.

$ oc new-project mutatetest
Now using project "mutatetest" on server "https://api.ocp49.xphyrlab.net:6443".

Now lets deploy a test pod. We will use a small application called “Portunus” which can test for network connectivity to endpoints outside you Kubernetes cluster.

$ oc create -f test/pod.yaml
pod/portunus-test created

Create a service to connect to our portunus application and export the service as an OpenShift route

$ oc create -f test/service.yaml
$ oc expose svc/portunus
route.route.openshift.io/portunus exposed
$ oc get route
NAME       HOST/PORT                                     PATH   SERVICES   PORT               TERMINATION   WILDCARD
portunus   portunus-mutatetest.apps.ocp49.xphyrlab.net          portunus   portunus-webport                 None

Open a web browser to the route you just created and test our portunus application. Click “Submit” under the DNS tester section and you should get an IP address for example.com.

Now we can take a look at our pod…

$ oc get po
NAME            READY   STATUS    RESTARTS   AGE
portunus-test   1/1     Running   0          5m24s

So where is our side-car? You can see that there is only one container in the pod, so no side car. Switch over and check the other terminal you have running and check to see if our webhook was called. You will note that it was not. This is because we configured our webhook to only work on namespaces (Projects) that have a label “sidecar-injection=enabled”. Let’s enable the webhook on our project.

$ oc label namespace/mutatetest sidecar-injection=enabled
namespace/mutatetest labeled

Now that the namespace has been labeled, lets check our pod.

$ oc get po
NAME            READY   STATUS    RESTARTS   AGE
portunus-test   1/1     Running   0          5m24s

You will see that it is still a single container pod. Remember that when we set up our webhook, it was only to be called on “CREATE” and “UPDATE” operations, we have not taken these actions yet, so lets delete and recreate the pod.

$ oc delete pod/portunus-test
pod "portunus-test" deleted
$ oc create -f test/pod.yaml
pod/portunus-test created
$ oc get po
NAME            READY   STATUS    RESTARTS   AGE
portunus-test   1/1     Running   0          23s

We still have a pod with a single container in it, but lets check the logs from our webhook container.

I0223 18:39:13.146642       1 webhook.go:202] AdmissionReview for Kind=/v1, Kind=Pod, Namespace=mutatetest Name=portunus-test (portunus-test) UID=189580a6-36bd-49d2-a2e2-b07730ddfadd patchOperation=CREATE UserInfo={markd 50f29bca-66c5-4d26-bf53-728880cbf1e4 [system:authenticated:oauth system:authenticated] map[scopes.authorization.openshift.io:[user:full]]}
I0223 18:39:13.146894       1 webhook.go:110] Mutation policy for mutatetest/portunus-test: status: "" required:false
I0223 18:39:13.146906       1 webhook.go:207] Skipping mutation for mutatetest/portunus-test due to policy check
I0223 18:39:13.147107       1 webhook.go:289] Ready to write response ...
I0223 18:39:13.147128       1 webhook.go:290] {"kind":"AdmissionReview","apiVersion":"admission.k8s.io/v1","response":{"uid":"189580a6-36bd-49d2-a2e2-b07730ddfadd","allowed":true}}

Note the line Skipping mutation for mutatetest/portunus-test due to policy check, so what is missing? Remember that our webhook is configured to act ONLY on pods with the annotation sidecar-injector-webhook.xphyr.net/inject: yes. Edit the test/pod.yaml file and remove the comments from the annotation so that it looks like this:

kind: Pod
apiVersion: v1
metadata:
  name: portunus-test
  labels:
    app: portunus-test
  annotations:
    sidecar-injector-webhook.xphyr.net/inject: "yes"

REQUIRED STEP: Because we will be injecting a sidecar that uses tcpdump which requires elevated privileges to run, we need to do a little extra work that has nothing to do with webhooks. This step is only required to allow the tcpdump command to work within OpenShift.

$ oc adm policy add-scc-to-user privileged -z default -n mutatetest
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:privileged added: "default"

Now we will delete and redeploy our pod one more time

$ oc delete pod/portunus-test
pod "portunus-test" deleted
$ oc create -f test/pod.yaml
pod/portunus-test created
$ oc get po
NAME            READY   STATUS    RESTARTS   AGE
portunus-test   2/2     Running   0          23s

SUCCESS! We now have a pod that contains two containers, our initial Portunus container, as well as the injected side-car container sidecar-tcpdump. You can connect to the sidecar-tcpdump and run the tcpdump command in the same network namespace.

NOTE: If your pod does not have a sidecar, or you get errors on deployment check the “REQUIRED STEP” above and ensure that you have given the proper permissions to the default service account to run the tcpdump sidecar.

$ oc rsh -c sidecar-tcpdump  portunus-test
# tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

Go back to the portunus web page one more time and run a dns lookup again. You will now see the tcp packets from the portunus test application creating a DNS query.

Summary

Kubernetes Admission Controllers are a very powerful tool to help control and maintain your OpenShift cluster. This post focused on modifying objects that are submitted to Kubernetes but only scratches the surface of what is possible. Mutations can be used to ensure that pods only run in certain contexts, or inject additional configuration data. You can also write a Validating Admission webhook that can control if a Pod is allowed to be created at all based on a set of your own logic. It is this customization that makes Kubernetes such a powerful platform.

References

In no specific order, the following articles helped to contribute to this post:

The following code was also consulted: