Handling secrets in Kubernetes and more specifically OpenShift is an ever evolving space. There are many secrets managers available including Google Secrets Manger, HashiCorp Vault, CyberArk and Azure Key Vault, just to name a few. In this post we will be testing out a new player in the secrets management arena called Infisical.

Infisical is based on an OpenSource (maybe more OpenCore) development model. The core of the platform is MIT Licensed and they have add-on features such as password history for paying customers. They have a SaaS offering as well as a self hosted version. If you want to try installing and running Infisical locally be sure to check out Infisical Deployment Options page for ways to deploy the application locally.

We will be installing the Infisical Secrets Operator in our OpenShift cluster, and testing some of the functionality that comes with Infisical as well as this operator. This includes the ability to create secrets from other secrets, and the ability to restart pods who’s secrets have been updated in Infisical. Infisical can also manage secrets through the use of the infisical cli tool which can inject secrets into environment variables for applications not hosted in OpenShift. I encourage you to look at all the features of Infisical on their website to see everything this tool can do.

We will be using the SaaS (hosted) version of Infisical as a part of this blog post, but I will call out where things need to change if you want to use a local install instead.

Prerequisites

• OpenShift 4 cluster - tested with OCP 4.13
• Infisical Hosted account - Get Started
• OpenShift Command Line tool (oc)
• Helm 3

Installing Secrets Operator

We will start by installing the Infisical Secrets Operator. They supply a helm chart to make this install quick and easy. If you do not wish to use Helm to install and manage the Infisical Secrets operator, there is also a manual install documented in their official documentation here. We will create a new project/namespace called infisical and then use helm to complete the install:

$ oc new-project infisical
$ helm repo add infisical-helm-charts 'https://dl.cloudsmith.io/public/infisical/helm-charts/helm/charts/'
"infisical-helm-charts" has been added to your repositories
$ helm repo update
$ helm install --generate-name infisical-helm-charts/secrets-operator

NOTE: it is recommended if using Helm to deploy the Infisical Secrets Operator in prod to set a specific target versions: helm install --generate-name infisical-helm-charts/secrets-operator --version=0.3.1 --set controllerManager.manager.image.tag=v0.2.0

Validate that the Infisical CRD has been installed:

$ oc get crd/infisicalsecrets.secrets.infisical.com
NAME                                     CREATED AT
infisicalsecrets.secrets.infisical.com   2023-09-01T18:17:42Z

Next up we will populate Infisical with some secrets that we will use with our test application.

Populate Infisical With Secrets

If you haven’t already, create an account with the Infisical SaaS and log into Infisical. start by clicking Add New Project. We will create a new Project, which we will call OCPDemo.

Inside this project we will create three secrets:

• DB_USERNAME
• DB_PASSWORD
• DB_DSN - (Data Source Name) which contains all the information to connect to the database including, server name, database name as well as username and password

Start by creating the DB_USERNAME, and DB_PASSWORD entries. Feel free to use anything for the username name and password values at this time, we will “fix” this later.

Now, for our DSN. If you are not familiar with the term DSN (Data Source Name) it is similar to a URL, and it looks like the following mysql:host=mariadb;dbname=guestbook;user=username;password=password.

There are multiple ways to format a DSN, but in our use-case the DSN contains the HOST, DatabaseName, UserName and Password all in one string. If we were to store that string in the DB_DSN, when we change the database password we would need to make sure to update both DB_PASSWORD and the DB_DSN entry.

Infisical offers a different way to handle this. We can create a secret that references other secrets, so for the DB_DSN create an entry like so mysql:host=mariadb;dbname=guestbook;user=${DB_USERNAME};password=${DB_PASSWORD}

This entry, will now use the DB_USERNAME, and DB_PASSWORD entries to build and return a DB_DSN entry.

Create Service Token for access by the Infisical Secrets Operator

With our secrets created in Infisical, we need a way to access these secrets from OpenShift. To do this, we need to create an Infisical service token. Start by selecting the “OCPDemo” project we created, and then select “Project Settings” and then switch to the Service Tokens tab and click + Create token.

Be sure to set an expiration that you are comfortable with and give it a name, and then select “Create”. Copy the token that is created and store it someplace safe, we will use this in our next step.

Apply Service Token to OpenShift

In order for the Infisical Secret Operator to access the secrets we need to supply the Service Token to operator. We will do this by creating a Kubernetes secret in the infisical project/namespace. Use the oc command line tool to create the required secret, be sure to update the token with your token from the previous step.

$ oc create secret generic infisical-hosted-service-token  -n infisical --from-literal=infisicalToken=<your token here>
secret/infisical-hosted-service-token created

Next up, we will deploy a test application that can make use of the secrets that we have stored in Infisical.

Testing the Operator with a demo application

We will use a demo application that leverages MariaDB, and a simple PHP application called guestbook-php. To deploy guestbook-php start by cloning the source, and creating a new project for the application called “guestbookphp”.

$ git clone https://github.com/rh-telco-tigers/guestbook-php.git
$ cd guestbook-php
$ oc new-project guestbookphp

With our new project created, we will deploy a MariaDB instance and an instance of our Guestbook application:

$ oc apply -f ./kubernetes//ocp
service/guestbook-service created
deployment.apps/guestbook created
secret/guestbook-config created
service/mariadb created
deployment.apps/mariadb created
secret/mariadb created
$ oc expose svc/guestbook
route.route.openshift.io/guestbook exposed
$ oc get route
NAME        HOST/PORT                                  PATH   SERVICES    PORT   TERMINATION   WILDCARD
guestbook   guestbook-guestbook.apps.acm.example.com          guestbook   8080                 None

If you open your favorite web-browser and goto the URL listed in the oc get route command (ie http://guestbook-guestbook.apps.acm.example.com in the above example) you should see our Guestbook application.

With our test application up and running, lets go ahead an create a secret using the Infisical Secret Operator.

Creating a InfisicalSecret

We will now create an “InfisicalSecret” object, which will connect to the hosted Infisical instance and retrieve our secret. We will put the values retrieved into a new secret called “ismanaged-db-secret”. We will be able to swap this with the secret that is currently in use, once it is fully populated. Create a new file called ismanaged-db-secret.yml and then apply it to our cluster.

Note: If you are using a local instance of Infisical, be sure to uncomment and update the hostAPI field in the YAML below.

apiVersion: secrets.infisical.com/v1alpha1
kind: InfisicalSecret
metadata:
  # Name of of this InfisicalSecret resource
  name: guestbook-infisicalsecret
spec:
  # The host that should be used to pull secrets from. If left empty, the value specified in Global configuration will be used
  # hostAPI: https://infisical.example.com/api
  resyncInterval: 60
  authentication:
    serviceToken:
      serviceTokenSecretReference:
        secretName: infisical-hosted-service-token
        secretNamespace: infisical
      secretsScope:
        envSlug: dev
        secretsPath: "/"
  managedSecretReference:
    secretName: ismanaged-db-secret # <-- the name of kubernetes secret that will be created
    secretNamespace: guestbookphp # <-- where the kubernetes secret should be created

Apply the InfisicalSecret yaml using the oc tool

$ oc create -f ismanaged-db-secret.yml 
infisicalsecret.secrets.infisical.com/guestbook-infisicalsecret created

With the InfisicalSecret created, lets take a look at the status:

$ oc describe infisicalsecret.secrets.infisical.com
Name:         guestbook-infisicalsecret
Namespace:    default
API Version:  secrets.infisical.com/v1alpha1
Kind:         InfisicalSecret
...
Status:
  Conditions:
    Last Transition Time:  2023-09-07T15:22:10Z
    Message:               Infisical controller has located the Infisical token in provided Kubernetes secret
    Reason:                OK
    Status:                True
    Type:                  secrets.infisical.com/LoadedInfisicalToken
    Last Transition Time:  2023-09-10T22:40:08Z
    Message:               Infisical controller has started syncing your secrets
    Reason:                OK
    Status:                True
    Type:                  secrets.infisical.com/ReadyToSyncSecrets
    Last Transition Time:  2023-09-10T22:40:08Z
    Message:               Infisical has found 0 deployments which are ready to be auto redeployed when secrets change
    Reason:                OK
    Status:                True
    Type:                  secrets.infisical.com/AutoRedeployReady

In the output above, not that the conditions shows “OK” for locating the Infisical token, and for syncing the secrets. You can also validate this by looking for the secret we told Infisical to create:

$ oc get secret/ismanaged-db-secret -n guestbookphp
NAME                       TYPE                                  DATA   AGE
ismanaged-db-secret        Opaque                                3      39s

With the secret in place, lets edit our guestbookphp deployment to start using the new managed secret. Use the oc edit deployment/guestbook command to

Update the deployment to use the new secret by using oc edit deployment/guestbook -n guestbookphp and update the REMOTE_DSN value as show below:

       - name: REMOTE_DSN
          valueFrom:
            secretKeyRef:
              key: DB_DSN
              name: ismanaged-db-secret

Validate that the pods are being updated:

$ oc get po
NAME                         READY   STATUS        RESTARTS   AGE
guestbook-5dd68d8778-62kxw   1/1     Running       0          13s
guestbook-7c78775555-cx7rj   1/1     Terminating   0          9m14s
mariadb-57c4f45fbb-mfn4k     1/1     Running       0          21h

Go ahead and check the guestbook application and try to add new entry:

WAIT, what happened? Well if you remember back in Populate Infisical with Secrets, we just put any random information in the secret. This doesn’t match what we are using for our MariaDB instance, and that is OK. we are going to enable the “Automatic Redeploy of Applications” feature of the Infisical Secrets Operator, and then update the secret in Infisical to match what MariaDB is configured with, and watch the magic happen.

Automatic Redeploy of applications

We will start by annotating the guestbook deployment with the auto-reload tag. This tag is used by the Infisical Secrets Operator to redeploy any pod that is using a secret that is managed by Infisical:

$ oc annotate deployment/guestbook secrets.infisical.com/auto-reload=true
deployment.apps/guestbook annotated

With the auto-reload annotation on our deployment, log into Infisical, and update the DB_USER and DB_PASSWORD entries to the following:

• DB_USER=myuser
• DB_PASSWORD=apassword1

Now, watch the pods in our namespace and see that the pods are automatically re-deployed after 1 minute.

$ oc get po --watch
NAME                         READY   STATUS        RESTARTS   AGE
guestbook-5dd68d8778-62kxw   1/1     Running       0          13s
guestbook-7c78775555-cx7rj   1/1     Terminating   0          9m14s
mariadb-57c4f45fbb-mfn4k     1/1     Running       0          21h

Rememeber, that while we are using the “DB_DSN” secret, we did not need to update that, as it will use the variable expansion for DSN_USER and DSN_PASSWORD automatically.

Once the application has been re-deployed, go back to the guestbook application and try to create a new entry.

SUCCESS!

If you want to take this to the next level, update the MariaDB deployment to leverage the same secret, and then try rotating the secret to see how the secret can be rotated end to end automatically.

Conclusion

Using a secrets manager can help you better manage secrets in your enterprise/organization. By leveraging the Infisical Secrets Operator you can automatically update secrets in your OpenShift cluster and ensure that your applications are always up to date with the proper secrets.

• openshift
• secrets
• infisical