Is it possible to create an OpenShift cluster that contains one Control Plane node and multiple worker nodes? This is something that came up in a recent project, and as it turns out the answer is YES. Since OpenShift 4.11, it has been possible to add worker nodes to SNO OpenShift cluster. When you follow this process, you end up with a cluster that contains one or more worker nodes, along with the original SNO cluster.
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ocp-worker-1 Ready worker 38h v1.27.16+03a907c
ocp-worker-2 Ready worker 38h v1.27.16+03a907c
ocp-control-plane-1 Ready control-plane,master,worker 42h v1.27.16+03a907c
But what if we don’t want the ocp-control-plane-1 node to be schedulable as a worker? Can we set it up so that it is ONLY a control-plane/master node? The answer is yes. In a non-OpenShift cluster we would simply remove the label node-role.kubernetes.io/worker from the node using the command oc adm label ocp-control-plane-1 node-role.kubernetes.io/worker- and the role would be removed. However if you try this with an OpenShift node, you will find that the change does not take effect. This is because the cluster scheduler operator is overriding your change based on the default configuration. So how do we make the change permanent, read on.
NOTE: Remember that building an OpenShift cluster with only one control-plane node means that when the control-plane node is down (be it due to failure, or maintenance), you will be unable to make any changes to the workloads running on the cluster, and any applications that leverage the “OpenShift Router” for exposing the application will not work, as all traffic must go through the control-plane node.
Prerequisites
The rest of this blog post will assume that you already have a
Cluster Configuration Changes Required
There are two changes that need to be made to your cluster to make your control-plane node not schedulable. The first is to edit the cluster scheduler, and the second is to update the ingress operator to run on the master node. We will take each of these changes one at a time.
Updating the Cluster Scheduler
We will start be editing the default Cluster Scheduler and changing the “mastersSchedulable” option from true to false. Log into the cluster using the oc login command, and then use the command oc edit schedulers.config.openshift.io cluster and change the spec.mastersSchedulable option to false as shown below.
apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
name: cluster
spec:
mastersSchedulable: false
policy:
name: ""
status: {}Save your changes, and then we can check the state of the nodes:
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ocp-worker-1 Ready worker 38h v1.27.16+03a907c
ocp-worker-2 Ready worker 38h v1.27.16+03a907c
ocp-control-plane-1 Ready control-plane,master 42h v1.27.16+03a907c
SUCCESS, but don’t celebrate just yet. If you reboot the ocp-control-plane-1 node, you will find that when the node starts back up, the OpenShift Ingress does not start. Lets check the pod status:
$ oc get po -n openshift-ingress
NAME READY STATUS RESTARTS AGE
router-default-7f9ddc5fcc-xkcl7 0/1 Pending 0 39s
You will note that the pod shows as pending, lets take a look and see why.
$ oc describe po router-default-7f9ddc5fcc-xkcl7 -n openshift-ingress
<many lines omitted>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 40m default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
From the output we can see that there is a taint on the master node, that is keeping it from being scheduled, so we will need to resolve this for the ingress controller to properly start. In order to do this, we will need to update the Ingress Controller and add a toleration so that the Ingress pod can run on the control-plane node.
Updating the Ingress Controller
The ingress controller needs to be updated so that it can run on the control-plane where the shared Ingress IP and API IP exist from deploying the SNO cluster. In order to do this we will need to edit the default IngressController oc edit ingresscontroller -n openshift-ingress-operator, update the section highlighted below:
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
clientTLS:
clientCA:
name: ""
clientCertificatePolicy: ""
httpCompression: {}
httpEmptyRequestsPolicy: Respond
httpErrorCodePages:
name: ""
nodePlacement:
nodeSelector:
matchLabels:
node-role.kubernetes.io/master: ""
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: ExistsSave your changes, and then we can check the state of the ingress pod:
$ oc get po -n openshift-ingress
NAME READY STATUS RESTARTS AGE
router-default-6bbd55d79-nfbnw 1/1 Running 0 3s
Success! The router pod is now running on our control-plane node. You should now be able to access your cluster console at console-openshift-console.apps.<clustername>.<domainname>.
Checking the results
With these changes made we can go ahead and take a look at our cluster and make sure that it is up and running, and that the ocp-control-plane-1 node no longer has the “worker” role.
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ocp-worker-1 Ready worker 38h v1.27.16+03a907c
ocp-worker-2 Ready worker 38h v1.27.16+03a907c
ocp-control-plane-1 Ready control-plane,master 42h v1.27.16+03a907c
Congratulations, you now have an OpenShift cluster with one control-plane node and two master nodes.
References
Configuring control plane nodes as schedulable