Using gMSA with Windows Containers in OCP
By Mark DeNeve
gMSA and OpenShift
In previous articles, we have shown how you can manage Windows Containers in OpenShift using the Windows Machine Config Operator. By configuring this feature, we are able to deploy and manage Windows Container Images just like any other Container Image with OpenShift. This gives us additional paths to application modernization allowing app developers to move over things like .Net legacy apps to OpenShift without having to re-write large portions of code.
Being able to host native Windows code within OpenShift brings with it, a need to be able to support Windows constructs such as Group Managed Service Accounts (gMSA). Group managed service accounts (gMSAs) are domain accounts that help to secure services. gMSAs can run on one server, or in a server farm, such as systems behind a network load balancing or Internet Information Services (IIS) server. After you configure your services to use a gMSA principal, account password management is handled by the Windows operating system (OS).
In this blog post we will show how to enable gMSA in your OpenShift cluster and run a test workload to validate that the gMSA is working properly. We will do this by leveraging the windows-gmsa CRD and Admission Webhook supplied by the Kubernetes Windows SIG.
NOTE: Support for the gMSA webhook and CRD are supplied by the Kubernetes Windows SIG and the open source community. This is not a Red Hat/OpenShift supported project.
The instructions below assume you already have a working Windows AD Domain in place and have the Windows Machine Config Operator installed on your cluster, and 1 or more Windows Worker Nodes running. Your Windows Worker Nodes will need to be joined to the domain for gMSA to work with OpenShift.
One last thing to keep in mind is that this will enable gMSA for Windows Containers ONLY. gMSA is not supported for Linux containers.
Prerequisites
- OpenShift 4.13 Cluster with the Windows Machine Config Operator installed and configured
- OpenShift Command Line tool (oc)
- Helm 3
Install the cert-manager Operator for Red Hat OpenShift
The Helm chart for windows-gmsa leverages cert-manager to automatically create and assign internal certificates for the admission webhook. We will need to install cert-manager prior to installing the windows-gmsa webhook. Follow the instructions below to install the webhook:
- Log in to the OpenShift Container Platform web console.
- Navigate to Operators → OperatorHub.
- Enter cert-manager Operator for Red Hat OpenShift into the filter box.
- Select the cert-manager Operator for Red Hat OpenShift and click Install.
- On the Install Operator page, select all defaults, and click Install
Wait for the Operator to install before proceeding to the next section.
Install the gMSA CRD
With cert-manager installed, we can move onto installing the gMSA CRD. We will start by cloning the windows-gmsa GitHub repo locally and then changing into the new directory created.
$ git clone https://github.com/kubernetes-sigs/windows-gmsa.git
$ cd windows-gmsa
Now, as an OpenShift administrator, log into the cluster and run the following command to install the gMSA CRD:
$ oc login
$ oc create -f windows-gmsa/admission-webhook/deploy/gmsa-crd.yml
customresourcedefinition.apiextensions.k8s.io/gmsacredentialspecs.windows.k8s.io created
Validate that the CRD was successfully installed:
$ oc get crd/gmsacredentialspecs.windows.k8s.io
NAME CREATED AT
gmsacredentialspecs.windows.k8s.io 2023-06-27T19:35:52Z
Installing the gMSA Web Hooks
gMSA requires Web Hooks to be installed. These can be installed using a Helm Chart. It is also possible to install via a shell script. See How to Deploy for a shell script that can be used to deploy the gMSA WebHook. The Shell script has not been tested against OCP. Your Mileage May Vary.
Installing the gMSA Web Hooks via Helm
We will create a new project/namespace called gmsa-webhook for the gMSA WebHook to run from, and use helm to install the WebHook:
$ oc new-project gmsa-webhook
Now using project "gmsa-webhook" on server "https://api.example.example.com:6443".
$ helm repo add windows-gmsa https://raw.githubusercontent.com/kubernetes-sigs/windows-gmsa/master/charts/repo
"windows-gmsa" has been added to your repositories
Validate that the helm repo was successfully added by getting the versions available in the helm chart
$ helm search repo -l gmsa
NAME CHART VERSION APP VERSION DESCRIPTION
windows-gmsa/gmsa 0.7.0 0.6.0 Windows GMSA Configuration
windows-gmsa/gmsa 0.6.0 0.4.0 Windows GMSA Configuration
windows-gmsa/gmsa 0.5.0 0.4.0 Windows GMSA Configuration
We can now deploy the helm chart. We need to override the default containerPort that the webhook server runs as due to OpenShift security requirements. This is easily achieved with the --set containerPort=8443 option.
$ helm install gmsa windows-gmsa/gmsa --namespace gmsa-webhook --set containerPort=8443
W0627 16:31:39.175325 27903 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (container "gmsa" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "gmsa" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "gmsa" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "gmsa" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
NAME: gmsa
LAST DEPLOYED: Tue Jun 27 16:31:35 2023
NAMESPACE: gmsa-webhook
STATUS: deployed
REVISION: 1
TEST SUITE: None
We have now completed the install phase and gMSA is available within your cluster. We now need to do some configuration in order to use this new application authentication method.
NOTE: The certificate issued for the gmsa-webhook will expire in 90d. Certmanager will renew it 30 days prior to expiration, HOWEVER the webhook application itself will not pick up the renewed certificate automatically. You will need to manually cycle the gmsa-webhook to pick up the new certificate. You can also look at using Reloader to handle the pod restart automatically. There is a current Issue Certificate renewal requires a restart of the gmsa pods that is tracking this and it may be resolved by the time you read this article.
Keep in mind that because the webhook is installed as a mutating admission webhook, it is called every time a new pod is started. If the certificate is expired, the cluster will not talk to the webhook, but can also not proceed with deploying any new pods. Be sure to keep the webhook deployment up to date with certificates, or you will run into issues with your cluster.
Windows Configuration Steps
You will need to have an ActiveDirectory domain pre-configured and gMSA configured in your domain. Proper configuring Windows AD objects and gMSA groups is beyond the scope of this post. See Getting Started with Group Managed Service Accounts for additional details. The commands listed in the Create gMSA account section below are for demo purposes only. Be sure to fully understand the security implications of this setup prior to implementing them in your domain.
You will also need the CredentialSpec PowerShell module installed on your Windows node. This can be done by running Install-Module -Name CredentialSpec as Administrator on your Windows host.
NOTE: The commands shown in this section that begin with “PS >” must be run from a Windows Host that is joined to the Domain, and run by a user with Domain Admin credentials.
Create gMSA account
From a Windows server, you can create a new gMSA account using the following command. We will create an account with the name “WebApp01”, in the example command below.
PS > New-ADServiceAccount -Name "WebApp01" -DnsHostName "WebApp01.example.com" -ServicePrincipalNames "host/WebApp01", "host/WebApp01.example.com" -PrincipalsAllowedToRetrieveManagedPassword windowsNode1$,windowsNode2$
Using the CredentialSpec applet retrieve the credentialSpec file for the “WebApp01” gMSA we just created
## Create a directory to store the file in
PS > New-Item -Path "c:\" -Name "MyCreds" -ItemType "directory"
## Retrieve the credentials and place in a file called "C:\MyCreds\WebApp01_CredSpec.json"
PS > New-CredentialSpec -AccountName WebApp01 -Path "C:\MyCreds\WebApp01_CredSpec.json"
The following is an example of what that JSON will look like
{
"CmsPlugins": [
"ActiveDirectory"
],
"DomainJoinConfig": {
"Sid": "S-1-5-21-1943239671-2162678975-3927069038",
"MachineAccountName": "WebApp01",
"Guid": "48d8fd20-3ee0-48c7-9b5e-3fb6cad6a9ff",
"DnsTreeName": "example.com",
"DnsName": "example.com",
"NetBiosName": "EXAMPLE"
},
"ActiveDirectoryConfig": {
"GroupManagedServiceAccounts": [
{
"Name": "WebApp01",
"Scope": "example.com"
},
{
"Name": "WebApp01",
"Scope": "EXAMPLE"
}
]
}
}
Keep the information from the C:\MyCreds\WebApp01_CredSpec.json file handy, you will need them for the next section.
Create gMSA Credential Spec
With our Windows gMSA credentials created, its time to configure OpenShift to take advantage of them. We will create a GMSACredentialSpec which defines our credential, and tells OpenShift how to get it. One thing to take note of, gMSA Credential Specs are NOT namespaced. This means that they exist at the cluster level. We will still need to create a Role, and RoleBinding to use the credential so the use of the gMSA can still be controlled, but it is visible to all namespaces. There is a bug/feature request Namespaced gMSA credential spec doesn’t work but no progress has been made on this since November of 2021.
Create a new file called gmsa-webapp01.yaml and fill with the contents below, making sure to update with YOUR information from the output of the C:\MyCreds\WebApp01_CredSpec.json from the previous section called Create gMSA account.
apiVersion: windows.k8s.io/v1
kind: GMSACredentialSpec
metadata:
name: gmsa-webapp01 # This is an arbitrary name but it will be used as a reference
credspec:
ActiveDirectoryConfig:
GroupManagedServiceAccounts:
- Name: WebApp01 # Username of the GMSA account
Scope: EXAMPLE # NETBIOS Domain Name
- Name: WebApp01 # Username of the GMSA account
Scope: example.com # DNS Domain Name
CmsPlugins:
- ActiveDirectory
DomainJoinConfig:
DnsName: example.com # DNS Domain Name
DnsTreeName: example.com # DNS Domain Name Root
Guid: 244818ae-87ac-4fcd-92ec-e79e5252348a # GUID
MachineAccountName: WebApp01 # Username of the GMSA account
NetBiosName: EXAMPLE # NETBIOS Domain Name
Sid: S-1-5-21-2126449477-2524075714-3094792973 # SID of GMSA
Now apply this GMSACredentialSpec to the cluster.
$ oc create -f gmsa-webapp01.yaml
Create Cluster Role to access gMSA
Since the GMSACredentialSpec is not namespaced, we are going to need to create a ClusterRole that will control access to the spec. This role will be resourceName specific and will grant access only to the named gMSA. Create a new file called gmsa-webapp01-role.yaml and put the following in it:
# Create the Role to read the credspec
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: webapp01-role
rules:
- apiGroups: ["windows.k8s.io"]
resources: ["gmsacredentialspecs"]
verbs: ["use"]
resourceNames: ["gmsa-webapp01"]
Now create the new role in your cluster:
$ oc create -f gmsa-webapp01-role.yaml
We are now ready to move onto testing of the gMSA.
Testing gMSA
Create role binding
In order to test everything that we have just worked to configure we will create a new project called “gmsatest” and deploy our test workload here. In order for the default service account in the “gmsatest” project to be able to access the GMSACredentialSpec we created previously, we need to create a rollBinding to our webapp01-role we created previously. We will start by creating the project, and then create the RoleBinding using the oc command:
oc new-project gmsatest
oc adm policy add-cluster-role-to-user webapp01-role -z default
Deploy test pod
In order to test our gMSA credentials, we will start up an instance of an IIS container. If you are using Windows server 2019 for your Windows Worker Nodes, be sure to update the container image below to use mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019. Create a file called iis-deployment.yml with the following contents. Note the securityContext.windowsOptions section where we specify the name of the GMSACredentialSpec we created earlier.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: with-creds
name: with-creds
spec:
replicas: 1
selector:
matchLabels:
run: with-creds
template:
metadata:
labels:
run: with-creds
spec:
tolerations:
- key: "os"
value: "Windows"
Effect: "NoSchedule"
securityContext:
windowsOptions:
gmsaCredentialSpecName: gmsa-webapp01
containers:
- image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2022
imagePullPolicy: Always
name: iis
nodeSelector:
kubernetes.io/os: windows
Using the file we just created, create an instance of the IIS server container:
$ oc create -f iis-deployment.yml
Now, we can watch the status of the deployment and wait for the IIS container to start. Keep in mind that Windows containers are larger than standard Linux containers and can take significantly longer to start up the first time.
$ oc get po --watch
NAME READY STATUS RESTARTS AGE
with-creds-679bbf947d-cct6j 0/1 ContainerCreating 0 5m2s
with-creds-679bbf947d-cct6j 1/1 Running 0 1m
Running Auth Tests
With gMSA installed, and your test IIS Deployment container running, you can connect to the Windows Container Console and run example commands to test that gMSA is working. Use oc exec -it win-nerddinner-7cb4478779-blf6l -- cmd.exe making sure to update the pod name for your pod. Then run the commands listed below to check for proper authentication:
Microsoft Windows [Version 10.0.20348.1668]
(c) Microsoft Corporation. All rights reserved.
C:\inetpub\wwwroot>nltest /parentdomain
example.com. (1)
The command completed successfully
C:\inetpub\wwwroot>nltest /sc_verify:example
Flags: b0 HAS_IP HAS_TIMESERV
Trusted DC Name \\win22adc.example.com
Trusted DC Connection Status Status = 0 0x0 NERR_Success
Trust Verification Status = 0 0x0 NERR_Success
The command completed successfully
C:\inetpub\wwwroot>klist get krbtgt
Current LogonId is 0:0x74b399
A ticket to krbtgt has been retrieved successfully.
Cached Tickets: (2)
#0> Client: WebApp01$ @ EXAMPLE.COM
Server: krbtgt/EXAMPLE.COM @ EXAMPLE.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags 0x40a10000 -> forwardable renewable pre_authent name_canonicalize
Start Time: 6/28/2023 13:43:18 (local)
End Time: 6/28/2023 23:43:18 (local)
Renew Time: 7/5/2023 13:43:18 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: win22adc.example.com
#1> Client: WebApp01$ @ EXAMPLE.COM
Server: krbtgt/EXAMPLE.COM @ EXAMPLE.COM
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags 0x40e10000 -> forwardable renewable initial pre_authent name_canonicalize
Start Time: 6/28/2023 13:43:18 (local)
End Time: 6/28/2023 23:43:18 (local)
Renew Time: 7/5/2023 13:43:18 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0x1 -> PRIMARY
Kdc Called: win22adc.example.com
C:\inetpub\wwwroot>exit
You are looking for the nltest and klist get krbgt commands to complete successfully and return valid data. If you get back valid data, congratulations you have successfully configured gMSA for OpenShift and Windows Containers.
Clean Up
If you no longer need gMSA in your cluster, the following steps will help you to remove the WebHook and CRD from your cluster.
NOTE: Be sure to uninstall the webHook itself via helm PRIOR to uninstalling the CRD.
Removing the gMSA Web Hooks via Helm
$ helm uninstall gmsa -n gmsa-webhook
Removing the gMSA CRD
$ oc delete -f windows-gmsa/admission-webhook/gmsa-crd.yml
Conclusion
The ability to leverage gMSA authentication in OpenShift with Windows Containers support, opens up the number of applications that can be migrated into OpenShift/Kubernetes. Once the gMSA WebHook is installed, the configuration of gMSA is straightforward, and granting individual Pods use of the gMSA credentials is easily achieved by adding securityContext.windowsOptions.gmsaCredentialSpecName section to your Kubernetes spec. This allows Windows applications to leverage gMSA for authentication to other Windows hosted applications and components.
References
- Kubernetes Sig Windows gMSA github repo
- Configure gMSA for Windows Pods and containers
- Configure group Managed Service Account Support
- gMSA CRD
- gMSA Webhook Template