OpenShift FileIntegrity Scanning

By Mark DeNeve

June 11, 2021

Introduction

The File Integrity Operator is used to watch for changed files on any node within an OpenShift cluster. Once deployed and configured, it will watch a set of pre-configured locations and report if any files are modified in any way that were not approved. This operator works in sync with MachineConfig so if you update a file through MachineConfig, once the files are updated, the File Integrity Operator will update its database of signatures to ensure that the approved changes do not trigger an alert. The File Integrity Operator is based on the OpenSource project AIDE Advanced Intrusion Detection Environment.

We will use Operator Hub to install an instance of the File Integrity Operator on our cluster and we will then test it by modifying files in multiple ways to see how changes are reported.

Installing File Integrity Operator

Using the OpenShift Console:

Log into your cluster
Select Operators->Operator Hub
Search for File Integrity Operator and Select it.
Select Install and leave all defaults, and click install.

From the command line:

Login to your cluster with the oc command
Apply the yaml located in the install directory oc apply -f install/
Verify that the install worked

$ oc get csv -n openshift-file-integrity
NAME                              DISPLAY                   VERSION   REPLACES   PHASE
file-integrity-operator.v0.1.13   File Integrity Operator   0.1.13               Succeeded
$ oc get deploy -n openshift-file-integrity
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
file-integrity-operator   1/1     1            1           41h

Deploying the File Integrity Operator

Using the OpenShift Console:

Log into your cluster
Select Operators->Installed Operators
Ensure that the Project Dropdown has “All Projects” selected
Search for File Integrity Operator and Select it.
Select “FileIntegrity”
Select “Create FileIntegrity”
Enter a name for your FileIntegrity instance. Use “cluster-fileintegrity”, as the name. This will be used for the remainder of this demo.
Leave all other options as default, and click Apply

From the command line:

Login to your cluster with the oc command
Apply the fio-cr.yaml to create a default instance oc create -f fio-cr.yml
Validate that the default config is running oc get fileintegrities/cluster-fileintegrity -n openshift-file-integrity -o jsonpath="{ .status.phase }"

The status will be one of three phases:

Pending - This is the initial state, while the FileIntegrity instance starts
Active - This is the primary state and indicates that the AIDE daemon is active
Initializing - This state indicates that the AIDE database is being initialized

Once the status is “Active”, you can move onto the next steps.

Retrieving the File Integrity Operator Results

The first round of data collection will take some time. The amount of time it takes for the scan to run will be hardware dependent. Once complete we can view the status of every node:

$ oc get fileintegritynodestatuses -n openshift-file-integrity
 NAME                                            NODE                       STATUS
cluster-fileintegrity-ocp47-jzgl9-master-0       ocp47-jzgl9-master-0       Succeeded
cluster-fileintegrity-ocp47-jzgl9-master-1       ocp47-jzgl9-master-1       Succeeded
cluster-fileintegrity-ocp47-jzgl9-master-2       ocp47-jzgl9-master-2       Succeeded
cluster-fileintegrity-ocp47-jzgl9-worker-6svd6   ocp47-jzgl9-worker-6svd6   Succeeded
cluster-fileintegrity-ocp47-jzgl9-worker-jt5ng   ocp47-jzgl9-worker-jt5ng   Succeeded
cluster-fileintegrity-ocp47-jzgl9-worker-wrddr   ocp47-jzgl9-worker-wrddr   Succeeded
cluster-fileintegrity-ocp47-jzgl9-worker-x5btd   ocp47-jzgl9-worker-x5btd   Succeeded

The important information here is the STATUS. You will see one of three states:

Succeeded - No files have been changed
Failed - Files have been changed
Errored - The AIDE application had an internal failure

Reviewing the AIDE configuration

We can take a look at the AIDE configuration file by reviewing the ConfigMap used by the operator:

oc describe cm/cluster-fileintegrity -n openshift-file-integrity

aide.conf:
----
@@define DBDIR /hostroot/etc/kubernetes
@@define LOGDIR /hostroot/etc/kubernetes
database=file:@@{DBDIR}/aide.db.gz
database_out=file:@@{DBDIR}/aide.db.gz
gzip_dbout=yes
verbose=5
report_url=file:@@{LOGDIR}/aide.log
report_url=stdout
PERMS = p+u+g+acl+selinux+xattrs
CONTENT_EX = sha512+ftype+p+u+g+n+acl+selinux+xattrs

/hostroot/boot/        CONTENT_EX
/hostroot/root/\..* PERMS
/hostroot/root/   CONTENT_EX
!/hostroot/usr/src/
!/hostroot/usr/tmp/

/hostroot/usr/    CONTENT_EX

# OpenShift specific excludes
!/hostroot/opt/
!/hostroot/var
!/hostroot/etc/NetworkManager/system-connections/
!/hostroot/etc/mtab$
!/hostroot/etc/.*~
!/hostroot/etc/kubernetes/static-pod-resources
!/hostroot/etc/kubernetes/aide.*
!/hostroot/etc/kubernetes/manifests
!/hostroot/etc/docker/certs.d
!/hostroot/etc/selinux/targeted
!/hostroot/etc/openvswitch/conf.db

# Catch everything else in /etc
/hostroot/etc/    CONTENT_EX

Simulating a failure

We will edit a file to simulate a failure. Select one of the nodes that is being monitored and connect to it using the oc debug command.

$ oc debug node/<node>
echo "# someone was here" >> /host/etc/resolv.conf
chmod 666 /host/etc/resolv.conf
exit

You will now need to wait for the tool to find the change. The AIDE scan can be resource intensive. By default, the scan will occur every 15 minutes.

$ oc get fileintegritynodestatuses -n openshift-file-integrity
 NAME                                            NODE                       STATUS
cluster-fileintegrity-ocp47-jzgl9-master-0       ocp47-jzgl9-master-0       Succeeded
cluster-fileintegrity-ocp47-jzgl9-master-1       ocp47-jzgl9-master-1       Succeeded
cluster-fileintegrity-ocp47-jzgl9-master-2       ocp47-jzgl9-master-2       Succeeded
cluster-fileintegrity-ocp47-jzgl9-worker-6svd6   ocp47-jzgl9-worker-6svd6   Failed
cluster-fileintegrity-ocp47-jzgl9-worker-jt5ng   ocp47-jzgl9-worker-jt5ng   Succeeded
cluster-fileintegrity-ocp47-jzgl9-worker-wrddr   ocp47-jzgl9-worker-wrddr   Succeeded
cluster-fileintegrity-ocp47-jzgl9-worker-x5btd   ocp47-jzgl9-worker-x5btd   Succeeded

Note that in the output above you can now see that the node we edited shows that the STATUS is “Failed”. Let’s check and see what the results are by retrieving the failed fileintegritynodestatuses with the oc get fileintegritynodestatuses/<report>.

$ oc get fileintegritynodestatuses/cluster-fileintegrity-ocp47-jzgl9-worker-6svd6 -n openshift-file-integrity -ojsonpath='{.results}' | jq -r
[
  {
    "condition": "Succeeded",
    "lastProbeTime": "2021-06-09T14:57:56Z"
  },
  {
    "condition": "Failed",
    "filesChanged": 1,
    "lastProbeTime": "2021-06-09T15:28:20Z",
    "resultConfigMapName": "aide-ds-cluster-fileintegrity-ocp47-jzgl9-worker-6svd6-failed",
    "resultConfigMapNamespace": "openshift-file-integrity"
  }
]

The results of the failed scan are stored in a ConfigMap stored in the same NameSpace as the operator. We can retrieve the configmap using the oc get cm <configMap name>. Lets get our results:

$ oc describe cm aide-ds-cluster-fileintegrity-ocp47-jzgl9-worker-6svd6-failed -n openshift-file-integrity
Name:         aide-ds-cluster-fileintegrity-ocp47-jzgl9-worker-6svd6-failed
Namespace:    openshift-file-integrity
...
====
integritylog:
----
Start timestamp: 2021-06-09 15:43:20 +0000 (AIDE 0.16)
AIDE found differences between database and filesystem!!

Summary:
  Total number of entries:  33901
  Added entries:              0
  Removed entries:            0
  Changed entries:            1

---------------------------------------------------
Changed entries:
---------------------------------------------------

f   p..    .CA.. : /hostroot/etc/resolv.conf

---------------------------------------------------
Detailed information about changes:
---------------------------------------------------

File: /hostroot/etc/resolv.conf
  Perm     : -rw-r--r--                       | -rw-rw-rw-
  SHA512   : 34i+lObhTukttHi3EZW31S1+gdXRtB3O | yxgwQGS8o1wMONLgCiDLd6akJrtHqE9v
             RhyxDst2S8JMo/+IMxAuiierKI1TFMeN | R8tG2hOEJkNfgLzlcDytpqHCmbYQJl4L
             +RwyzBwFN4+epz6ElkSSQA==         | 7o8lwP4ehkKneMxN945nJg==
  ACL      : A: user::rw-                     | A: user::rw-
             A: group::r--                    | A: group::rw-
             A: other::r--                    | A: other::rw-

End timestamp: 2021-06-09 15:43:32 +0000 (run time: 0m 12s)

Events:  <none>

With the information above we can now identify the file that was changed. You can then review the file and see what has changed, and deal with it as appropriate. The summary for each file change that is detected looks like so:

f   p..    .CA.. : /hostroot/etc/resolv.conf

Breaking down the report above:

a file has changed
the permissions have changed
the Checksum (or Contents) have changed
the Access control list has changed

The fileintegrity operator can detect many different file and directory changes. The full list looks like this:

l means that the link name has changed.
b means that the block count has changed.
p means that the permissions have changed.
u means that the uid has changed.
g means that the gid has changed.
a means that the access time has changed.
m means that the modification time has changed.
c means that the change time has changed.
i means that the inode has changed.
n means that the link count has changed.
C means that one or more checksums have changed.
A means that the access control list has changed.
X means that the extended attributes have changed.
S means that the SELinux attributes have changed.

For more details on the AIDE output check the man page for AIDE, or check here for a cached web version: aide.conf(5)

Reinitializing the AIDE database

Lets say that your configuration was changed, but it was done intentionally, and you do not want to be alerted to the change any more. What do you do to reset the AIDE database so that it knows this change was intentional. Using the oc command we will annotate the fileintegrity and mark it for “re-init”, which will re-scan all the hosts that are managed by the fileintegrity instance, and update the database to have the new signatures.

$ oc annotate fileintegrities/cluster-fileintegrity file-integrity.openshift.io/re-init= -n openshift-file-integrity

Conclusion

The FileIntegrity Operator is just one of many additional tools that are available to you to manage security in your OpenShift cluster. Another tool that can help you manage the security of your cluster is the “Openshift Compliance Operator”. To learn more about the OpenShift Compliance Operator and what it can provide check out the blog post at “My OpenShift Blog” here.