OpenShift Windows Containers - Bring Your Own Host

By Mark DeNeve

November 15, 2021

OpenShift has supported Windows Containers with the Windows Machine Config Operator for the past year, starting with OCP 4.6. Initial Windows Container support required running your platform in Azure or AWS. With the release of 4.7, the WMCO also supported hosting machines in VMWare. However, when deploying in a VMWare environment you had to spend time configuring a base Windows image, using tools such as sysprep and VMware templates. What if you wanted to use a bare metal host(s), or wanted to take advantage of existing Windows servers that you already manage. Or perhaps you just wanted to try out Windows Containers without going through all the steps of setting up a Windows template to deploy a single machine?

With the release of Windows Machine Config Operator 4.0, now you can “Bring Your Own Host”. What does it mean to Bring Your Own Host (BYOH)? If you have a Windows server (or servers) that are already built that you would like to leverage to run Windows Containers you can now add these machines to your OpenShift cluster and deploy Windows based containers.

In this post, we will go over how to add a Windows Server 2019 LTS machine to your existing OpenShift cluster. We will not cover the OpenShift install, or how to install Windows 2019, many articles already exist that cover this information. One important thing to note is that your OpenShift cluster must be using OVNKubernetes in Hybrid mode. An excellent article on building a cluster using OVNKubernetes in Hybrid mode can be found on the “My OpenShift Blog” site Running Windows Containers in OpenShift.

Prerequisites

Before we get into how to properly configure your Windows Server and OpenShift cluster to use BYOH Windows machines, we need to ensure you have the proper prerequisites.

Working OpenShift 4.8+ cluster configured with OVNKubernetes in Hybrid mode
Windows Server 2019, Windows Server 2004 or 20H2
Trial Versions of Windows Server can be found here: https://www.microsoft.com/en-us/evalcenter/evaluate-windows-server-2019

NOTE: If you choose to use the trial/eval software be aware that the OS is only good for 180 days. There is no way to upgrade or move this to a supported Windows version. You have been warned!

A few other requirements to keep in mind before proceeding:

The Windows Server instance must be on the same network as the Linux worker nodes in the cluster.
The hostname of the Windows Server instance must follow the RFC 1123 DNS label requirements, which include the following standards:
Contains only lowercase alphanumeric characters or ‘-’.
Starts with an alphanumeric character.
Ends with an alphanumeric character.
An “A record” as well as a “PTR record” must exist in DNS for each BYOH Server you plan to add to the cluster.

Finally, you will need to ensure that DNS records are in place that resolves both api.<clustername>.basedomain AND api-int.<clustername>.basename. If you are unable to resolve both these records, be sure to create the missing records before continuing. A quick test would be to ping both names to ensure they resolve and resolve to the same IP address.

$ ping -c1 api.ocp49.xphyrlab.net
PING api.ocp49.xphyrlab.net (172.16.25.81): 56 data bytes
64 bytes from 172.16.25.81: icmp_seq=0 ttl=62 time=68.624 ms
$ ping -c1 api-int.ocp49.xphyrlab.net
PING api-int.ocp49.xphyrlab.net (172.16.25.81): 56 data bytes
64 bytes from 172.16.25.81: icmp_seq=0 ttl=62 time=5.695 ms

Windows Server Prep

Before we can add a BYOH machine to our OpenShift cluster, we need to prepare the target Windows machine to join the cluster. The following steps will need to be completed:

properly name the server
patch the server
install OpenSSH
install Docker for Windows
open port 10250 to retrieve logs
enable remote ssh connections with a private key

The instructions in this post will show how to configure a freshly installed Windows Server that has no prior configuration. You can use an existing Windows Server if you already have one, however some of the instructions here will not apply to you. The steps below will use the PSWindowsUpdate PowerShell module to ease the install of the Windows updates. You will note that we rename the server to “win2019-pet”, you may use any name you wish as long as it meets the needs listed in the Prerequisites section.

NOTE: All of the steps below are PowerShell commands so ensure that you are running them from a PowerShell console.

# Start by Renaming the Server
Rename-Computer -NewName "win2019-pet1" -Force
 
#Install PS Windows Update Module
 
Get-PackageProvider -name nuget -force
Install-Module PSWindowsUpdate -confirm:$false -force
Get-WindowsUpdate -Install -acceptall -IgnoreReboot
Restart-Computer -Force
# Rinse and repeat until your host is patched
Get-WindowsUpdate -Install -acceptall -IgnoreReboot
Restart-Computer -Force

Now that your server is properly named and patched, we will enable RDP for remote diagnostics. This will make it easier to connect to the machine and copy/paste the remaining commands below.

NOTE: Enabling RDP is NOT a requirement of using BYOH but will make it easier to complete the remaining commands and ease troubleshooting down the road should it be required.

Set-ItemProperty -Path 'HKLM:\System\CurrentControlSet\Control\Terminal Server' -name "fDenyTSConnections" -value 0
Enable-NetFirewallRule -DisplayGroup "Remote Desktop"

With RDP enabled, connect to the server and run the following commands to install OpenSSH Server (https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse) on your host.

# Install the OpenSSH Server
Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
 
# Start the sshd service
Start-Service sshd
 
# OPTIONAL but recommended:
Set-Service -Name sshd -StartupType 'Automatic'
 
# Confirm the Firewall rule is configured. It should be created automatically by setup. Run the following to verify
if (!(Get-NetFirewallRule -Name "OpenSSH-Server-In-TCP" -ErrorAction SilentlyContinue | Select-Object Name, Enabled)) {
   Write-Output "Firewall Rule 'OpenSSH-Server-In-TCP' does not exist, creating it..."
   New-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
} else {
   Write-Output "Firewall rule 'OpenSSH-Server-In-TCP' has been created and exists."
}

We will complete the software install steps by installing Docker on your Windows server. Full documentation for Docker on Windows can be found here: Windows Containers - Quick Start. Use the following commands to quickly install Docker:

Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
Install-Package -Name docker -ProviderName DockerMsftProvider
# You may be prompted to trust the package source "DockerDefault" select "A" to continue
Restart-Computer -Force

After your server reboots from installing Docker, do a quick test to ensure that Docker is running properly. We will pull down a base Windows image which we will use later to test that Windows Containers are working properly from within OpenShift. Pulling this image now will speed up testing later on.

NOTE: Windows containers are NOT small. This example base container will pull down over 5Gb of data, so it may take some time to complete.

# This command will take a WHILE Windows Containers are not small
docker pull mcr.microsoft.com/windows/servercore:ltsc2019

Finally, we will open port 10250 to allow for log collection from our Windows Containers:

New-NetFirewallRule -DisplayName 'Windows Container Logs' `
                   -LocalPort 10250 -Action Allow `
                   -Profile 'Public' `
                   -Protocol TCP `
                   -Direction Inbound

Configuring Windows for Remote SSH Access

Our Windows server now has all the prerequisites installed but we still need to enable remote access to the Windows box using an SSH key. It is suggested that you generate an ssh key specifically for use with your Windows BYOH host. The following steps are based on the official Microsoft documentation for OpenSSH Key Management

Start by creating a new ssh key pair for administrator use. We will create this new key as ~/.ssh/winserver and use this key pair for the remainder of the blog post. If you are creating the key pair from a Windows host you will need to slightly modify your commands below:

# Generate a new SSH key file
ssh-keygen -t ed25519 -N '' -f ~/.ssh/winserver
# Copy the public key over to your windows box
scp ~/.ssh/winserver administrator@<BYOH IP address>

Now SSH to your BYOH server and run the following commands to set up the ssh key to have administrator rights:

ssh administrator@<BYOH IP address>
# by default, Windows will put you in the command shell, switch to powershell
powershell
 
New-Item -Type File -Path C:\ProgramData\ssh\administrators_authorized_keys
get-acl C:\ProgramData\ssh\ssh_host_dsa_key | set-acl C:\ProgramData\ssh\administrators_authorized_keys
Get-Content winserver.pub | Out-File C:\ProgramData\ssh\administrators_authorized_keys -Encoding UTF8 -Append

As the last step, let’s check to ensure that the ssh keypair is working and you can remotely access the BYOH Windows server without the use of a password. The following command will use your ~/.ssh/winserver private key to authenticate to the Windows host and will connect you to the Windows server without the use of a password:

ssh -i ~/.ssh/winserver administrator@<>
Microsoft Windows [Version 10.0.17763.2300]
(c) 2018 Microsoft Corporation. All rights reserved.
 
administrator@WIN2019-PET1 C:\Users\Administrator>

You should get connected without the use of a password. If you are still prompted for a password revisit the steps in this section to ensure you didn’t miss any steps, or review the Microsoft documentation on OpenSSH Key Management for additional assistance.

Adding your new node

Our BYOH Windows server is ready to be added to our OpenShift cluster. If you have not already done so, install the Windows Machine Config Operator from Operator Hub:

Log into OpenShift as a Cluster Administrator
Select “Operator Hub”
Search for “Windows”
Select “Windows Machine Config Operator”, and click Install
Use all default options and click Install

Ensure that the Windows Machine Config Operator installs successfully before continuing onto the next steps.

We will now use the SSH key that we created earlier to create a secret in the “openshift-windows-machine-config-operator” namespace:

$ oc create secret generic cloud-private-key --from-file=private-key.pem=${HOME}/.ssh/winserver \
   -n openshift-windows-machine-config-operator

With our ssh private key properly configured in OpenShift, we can now create a new ConfigMap called “windows-instances” and add our BYOH Windows server(s) into the cluster. Create a file called “windows-instances.yaml” and put the following content in it:

kind: ConfigMap
apiVersion: v1
metadata:
 name: windows-instances
 namespace: openshift-windows-machine-config-operator
data:
 172.16.25.175: |-
   username=Administrator   
 # additional machines can be added by listing additional entries like so
 #instance.dns.com: |-
 #  username=core

NOTE: Be sure to update the YAML to reflect the IP address of YOUR machine that you are adding.

Apply the file we just created to your cluster using the oc command:

oc create -f windows-instances.yaml

The Windows Machine Config Operator will now install additional software and add the machine as a new node in your cluster. You can check the status using the oc get nodes command. Wait until the new node shows as being in a Ready state:

oc get nodes
NAME                       STATUS   ROLES    AGE   VERSION
ocp49-qdb8j-master-0       Ready    master   40h   v1.22.0-rc.0+a44d0f0
ocp49-qdb8j-master-1       Ready    master   40h   v1.22.0-rc.0+a44d0f0
ocp49-qdb8j-master-2       Ready    master   40h   v1.22.0-rc.0+a44d0f0
ocp49-qdb8j-worker-frgkq   Ready    worker   39h   v1.22.0-rc.0+a44d0f0
ocp49-qdb8j-worker-pp5q5   Ready    worker   39h   v1.22.0-rc.0+a44d0f0
ocp49-qdb8j-worker-rh8kp   Ready    worker   39h   v1.22.0-rc.0+a44d0f0
win2019-pet1               Ready    worker   23h   v1.22.1-1676+af080cb8d127b3

Deploying a Windows Container

At this point, we can now deploy a Windows Container within your OpenShift cluster. We will deploy a simple Windows container that uses PowerShell to serve a simple HTML web page that says “Windows Container Web Server”.

Start by creating a new project for our Windows Container image to run in:

oc new-project wincontainer

Now create a new file called windows-hello.yml and paste the following contents into that file:

apiVersion: v1
kind: Service
metadata:
 name: win-webserver
 labels:
   app: win-webserver
spec:
 ports:
   # the port that this service should serve on
 - port: 80
   targetPort: 80
 selector:
   app: win-webserver
 type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
   app: win-webserver
 name: win-webserver
spec:
 selector:
   matchLabels:
     app: win-webserver
 replicas: 1
 template:
   metadata:
     labels:
       app: win-webserver
     name: win-webserver
   spec:
     tolerations:
     - key: "os"
       value: "Windows"
       Effect: "NoSchedule"
     containers:
     - name: windowswebserver
       image: mcr.microsoft.com/windows/servercore:ltsc2019
       imagePullPolicy: IfNotPresent
       command:
       - powershell.exe
       - -command
       - $listener = New-Object System.Net.HttpListener; $listener.Prefixes.Add('http://*:80/'); $listener.Start();Write-Host('Listening at http://*:80/'); while ($listener.IsListening) { $context = $listener.GetContext(); $response = $context.Response; $content='<html><body><H1>Hello from a Windows Container Web Server</H1></body></html>'; $buffer = [System.Text.Encoding]::UTF8.GetBytes($content); $response.ContentLength64 = $buffer.Length; $response.OutputStream.Write($buffer, 0, $buffer.Length); $response.Close(); };
     nodeSelector:
       beta.kubernetes.io/os: windows

Finally, we will create our Windows container, service and expose the application using an OpenShift route:

oc create -f windows-hello.yml
oc expose svc/win-webserver
oc get route
NAME           HOST/PORT                                           PATH   SERVICES        PORT   TERMINATION   WILDCARD
wincontainer   wincontainer-wincontainer.apps.ocp49.xphyrlab.net          win-webserver   80                   None

Open your web browser of choice and connect to the hostname listed in the output from the oc get route command above. You should see “Hello from a Windows Container Web Server”. You have successfully deployed a Windows Container image on a Windows Server managed by OpenShift!

Conclusion

This blog post showed how you can add “Pet” Windows Server machines to an OpenShift cluster to enable the use of Windows based containers. Keep in mind that if you look to operationalize this process and use Windows Server hosts in production you still need to maintain these servers. This includes things like patching, anti-virus, and other maintenance tasks that come along with managing Windows Servers. It will be up to you to determine how to handle these maintenance activities and coordinate them with the OpenShift scheduler.