Creating Custom Operator Hub Catalogs
By Mark DeNeve
Introduction
By default every new OpenShift cluster has a fully populated Operator Hub, filled with various Operators from the Kubernetes community and Red Hat partners, curated by Red Hat. These Operators can be installed by the cluster administrator in order to expand the features and functions of a given cluster. While this is great in many cases, not all enterprises want these Operators made available for install. OpenShift OperatorHub is fully configurable, and some of the various options on how to work with the Operator Catalog will be the topic of this blog post.
It is possible to remove all the Operators from Operator Hub, but some times a company will want to make use of a subset of the Operators that are available. In this case you will want to define a custom Operator Catalog for your specific use case.
If you are running OpenShift in a disconnected environment, or have chosen to use a local registry to cache all the OpenShift required images, creating a custom Operator catalog can also significantly cut back on the amount of storage required to maintain this local copy.
In this post we will briefly discuss how to completely disable all Operators in Operator Hub, but then focus on the other options that you have available including creating a customized catalog and mirroring that catalog to an internal registry.
Prerequisites
You will need an OpenShift cluster up and running to fully test this procedure. In addition you will require the following:
- Cluster Administrator privileges on an already installed OpenShift cluster
- opmcli - Download the binary from here
- podman - You will need to have podman already installed on your machine.
- oc - The OpenShift client, download from here
- grpcurl - Use your package manager to install or download the latest version from here
You will also need access to a container image registry that you can push/publish your new Operator Index container image to. If you have not previosuly configured your OpenShift cluster to trust an external registry, follow the steps Configuring additional trust stores for image registry access to enable your cluster to pull from an internal registry.
Disabling Default Operator Hub
If you are just looking for a way to disable the Operator Hub in your OpenShift cluster, you need to only run one command. You will need to have Cluster Admin privileges in order to run the commands in the following section.
$ oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'
operatorhub.config.openshift.io/cluster patched
At this point, you can validate that the Operator Hub in OpenShift is empty by checking in the UI:
If disabling Operator Hub in your cluster was all you were looking to do, Congrats, you have made it to the end of the blog. However if you are looking to be able to have access to some of the operators in the catalog then continue reading, to see how you can create a custom Operator Catalog and how to mirror it to an internal registry if required.
Configure Registry Authentication
The first thing we will need to do is set up authentication to the Red Hat registry and your private container registry. To do this, we will start with a copy of your OpenShift pull-secret. Download your registry.redhat.io pull secret from the Red Hat OpenShift Cluster Manager and save it to your home directory as “pull-secret.json”.
With our pull-secret retrieved, we need to add additional authentication to the this file to allow us to connect to your private container registry. Run the command below to log into YOUR private container registry using a username and password that works for your internal registry. Be sure to update the path to the pull-secret.json as well if you did not store it in your home directory.
$ podman login --authfile ~/pull-secret.json \
-u <username> \
-p <password> \
<host_example_com>:8443> \
--tls-verify=false
This will update the pull-secret.json file to include the authentication for your local registry to ensure you can push images to your private registry.
Creating a Custom Catalog
Identify the Operators you will need for your deployment
With our auth file setup to connect to both the Red Hat Registry as well as your local registry, we can now get a copy of all the Operators that are in the current release of the OpenShift Operator Catalog. Run the command below, making sure to update the “v4.10” tag to the proper release you are targeting.
$ podman run -p50051:50051 \
--authfile ~/pull-secret.json \
-d --name operator-index registry.redhat.io/redhat/redhat-operator-index:v4.10
$ grpcurl -plaintext localhost:50051 api.Registry/ListPackages > packages.out
$ cat packages.out
# ensure that packages.out contains the list of packages
$ podman kill operator-index
Review the packages.out file and identify all the Operators that you want to install, in this blog post we will target the following three Operators:
- Performance Addon Operator - performance-addon-operator
- Local Storage - local-storage-operator
- OpenShift Data Foundation - odf-operator, odf-lvm-operator, odf-csi-addons-operator
Record the name(s) of the required operators you wish to keep, and then move onto the next step.
Create new Catalog Image
Now that we have identified the Operators that we would like to have in our catalog, we will create a new custom operator index. These steps assume that you have already followed steps in Configure Auth to Red Hat Registry locally section to ensure you can pull from the Red Hat Registry. If your pull-secret.json is located in a different directory be sure to update that path. Also ensure that you update “v4.10” with the current release you are planning to target.
$ export REGISTRY_AUTH_FILE=~/pull-secret.json
$ opm index prune \
-f registry.redhat.io/redhat/redhat-operator-index:v4.10 \
-p performance-addon-operator,local-storage-operator,odf-operator,odf-lvm-operator,odf-csi-addons-operator \
-t registry.xphyrlab.net:8443/privoperator/my-operator-index:v4.10
INFO[0000] pruning the index packages="[performance-addon-operator local-storage-operator odf-operator odf-lvm-operator odf-csi-addons-operator]"
INFO[0000] Pulling previous image registry.redhat.io/redhat/redhat-operator-index:v4.10 to get metadata packages="[performance-addon-operator local-storage-operator odf-operator odf-lvm-operator odf-csi-addons-operator]"
INFO[0000] running /usr/bin/podman pull registry.redhat.io/redhat/redhat-operator-index:v4.10 packages="[performance-addon-operator local-storage-operator odf-operator odf-lvm-operator odf-csi-addons-operator]"
INFO[0005] running /usr/bin/podman pull registry.redhat.io/redhat/redhat-operator-index:v4.10 packages="[performance-addon-operator local-storage-operator odf-operator odf-lvm-operator odf-csi-addons-operator]"
...
INFO[0070] [podman build --format docker -f index.Dockerfile501796183 -t registry.xphyrlab.net/privoperator/my-operator-index:v4.10 .] packages="[performance-addon-operator local-storage-operator odf-operator odf-lvm-operator odf-csi-addons-operator]"
$ podman push registry.xphyrlab.net:8443/privoperator/my-operator-index:v4.10
At this point we now have an custom Operator Index that contains just the operators that you want being available in your cluster.
Optional - Mirror Required Operator Images
If you are operating your cluster in a “disconnected state” (no direct access to the Red Hat Registries), and you are using an internal mirror registry, you can create a mirror of just the containers required to use our custom operator catalog. This will save on the total amount of storage that is required to keep a mirror of the Operator Hub Images (a full mirror can be over 500GB).
NOTE: This step is Optional, if you do not need a local mirror of the container images, you do not need to run these steps and can skip to the next section.
When running the command below, be sure to specify the custom operator catalog that we created in the previous steps:
$ oc adm catalog mirror \
jumpbox.xphyrlab.net:8443/pruneolm/redhat-operator-index:v4.10 \
jumpbox.xphyrlab.net:8443/pruneolm \
-a ${LOCAL_SECRET_JSON} \
--index-filter-by-os='linux/amd64'
...
sha256:9be245f935741d6f653040dad904cf4b7f4184cf4b6440acad8dd42c5a93f92a jumpbox.xphyrlab.net:8443/pruneolm/odf4-odf-topolvm-rhel8
sha256:34bd9881ae207838866ac1c31f2be9324903cf952568339bd7d99e1231a44825 jumpbox.xphyrlab.net:8443/pruneolm/odf4-odf-topolvm-rhel8
sha256:69425447c0b97716a0075cfe7c3c9f8022edf51e266a3f0577092db9e2de0b0d jumpbox.xphyrlab.net:8443/pruneolm/odf4-odf-topolvm-rhel8:ed1d843a
info: Mirroring completed in 17m23.34s (13.66MB/s)
no digest mapping available for jumpbox.xphyrlab.net:8443/pruneolm/redhat-operator-index:v4.10, skip writing to ImageContentSourcePolicy
wrote mirroring manifests to manifests-redhat-operator-index-1660743954
deleted dir /tmp/1168044877
Capture the files in the manifests-redhat-operator-index-1660743954 (your directory name will be different), these will be used later when configuring OpenShift to use the mirror images.
Enabling Custom Operator Catalog
With the default Operator Catalog disabled, we can now add our custom Operator Catalog to the cluster. Create a file called “my-operator-catalog.yml” with the following information, making sure to update the image value with the new operator index that you created in the Create new Catalog section:
Deploy new catalog
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: my-operator-catalog
namespace: openshift-marketplace
spec:
sourceType: grpc
image: registry.xphyrlab.net/privoperator/my-operator-index:v4.10
displayName: My Operator Catalog
publisher: Xphyr
updateStrategy:
registryPoll:
interval: 30m
You can now apply this catalog to the cluster
$ oc create -f my-operator-catalog.yml
catalogsource.operators.coreos.com/my-operator-catalog created
NOTE: If you created a local mirror of the Operator container images, you will also need to also apply the imageContentSourcePolicy.yaml file that is located in the directory that was created when you ran the OPTIONAL Steps for Mirroring commands above.
$ oc create -f manifests-redhat-operator-index-1660743954/imageContentSourcePolicy.yaml
Check the status of the operatorcatalog
$ oc get catalogsource -n openshift-marketplace
NAMESPACE NAME DISPLAY TYPE PUBLISHER AGE
openshift-marketplace my-operator-catalog My Operator Catalog grpc Xphyr 35s
$ oc get po -n openshift-marketplace
NAME READY STATUS RESTARTS AGE
marketplace-operator-6fc4c9d8f-x6nsx 1/1 Running 14 18d
my-operator-catalog-jvqrh 1/1 Running 0 42s
NOTE: If you see any errors for your catalog pod, check the Troubleshooting section below.
Assuming that everything starts up OK, if you log into the OpenShift UI you can now see in the Operator Hub that only the Operators that you selected for your Custom catalog are available in OpenShift.
Note that now there are only four Operators listed in the catalog for use. Congratulations, you have created your own custom Operator Catalog and deployed it into your cluster.
Troubleshooting
If you apply your new Operator Catalog and your catalog does not come on line, this can be traced back to one of two issues:
- Your cluster does not trust your registry. You will see errors like
rpc error: code = Unknown desc = pinging container registry: x509: certificate signed by unknown authority. Configuring additional trust stores for image registry access to enable your cluster to pull from an internal registry. - Your cluster does not have the proper credentials to access your registry. You will see errors like
rpc error: code = Unknown desc = reading manifest v4.10 unauthorized: access to the requested resource is not authorized. You need to either make this image “public” in your registry or add additional credentials to the cluster to pull from this registry.
Conclusion
It is possible to remove the default OperatorHub catalog from your cluster and replace it with a custom operator catalog. This process can also be used to minimize the number of Operator images that you need to mirror when using a disconnected, or cached local on site registry.