Detect Malicious Behaviour on Kubernetes API Server through Audit Logs
Introduction
We might not know that Falco is not just for detecting malicious behavior that involves making Linux system calls, in addition to that, Falco v0.13.0 adds Kubernetes Audit Events to the list of supported event sources. That means that, once your cluster is configured with audit log enabled, you can send audit logs as events to Falco, then, you can write Falco rules that can read these events and detect malicious or other notable activity. After that, you can send notifications about the malicious activities or set up a Kubernetes Response Engine to take any action based on them. If you want to take a look at the rules which Falco provides for Kubernetes Audit logs, you can refer to this link. Also, if you want to learn more about that Kubernetes Response Engine is, there is series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.
- Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless
- Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas
- Kubernetes Response Engine, Part 3 : Falcosidekick + Knative
- Kubernetes Response Engine, Part 4: Falcosidekick + Tekton
- Kubernetes Response Engine, Part 5: Falcosidekick + Argo
Without further ado, let's explain what Audit Logs are, and which useful pieces of information that they have for us.
We can assume that Kubernetes API Server is like the brain of the cluster. We all communicate with the cluster through Kubernetes API Server, which means that all the requests hit Kubernetes API Server first. So, sometimes we might want to know that what’s happening inside of that brain. At this point, the Kubernetes Audit Log comes into the picture.
Kubernetes Audit Logs are the records that explain what’s happening inside of this brain but if we look at them more technically the official documentation says Kubernetes Audit logs provide a security-relevant, chronological set of records documenting the sequence of actions in a cluster., so by collecting and analyzing them we can answer these following questions:
- What happened?
- When did it happen?
- Who initiated it?
- On what did it happen?
To get more detail about them, you can refer to this link.
In this guide, I'll explain the following topics:
- How you can enable Audit log feature in Kubernetes.
- How you can enable the embedded web server available within the Falco project.
- The purpose of the Falcosidekick project.
Prerequisites
We need tools with the following minimum versions to achieve this demo:
- multipass v1.6.2
- k3sup v0.11.0
- kubectl v1.21.1
- helm v3.5.4+g1b5edb6
Tutorial
Provision local Kubernetes Cluster
There are various ways to provision a local Kubernetes cluster such as, KinD, k3s, k0s, Minikube etc. We are going to use k3s in this walkthrough, we're also going to use some tooling to make the installation process of k3s cluster easier and quicker such as k3sup and multipass.
Let's get provision a local Kubernetes cluster:
Verify if everything is working before move on to the next step.
Install Falco and Falcosidekick with Auditing Feature Enabled
We'll use Helm to install the Falco, so, there is an option that we can enable or disable the audit log feature called auditLog.enabled, once we set the value of the option as true, the embedded webserver will be started within the Falco to consume audit events from the port 8765 and behind k8s-audit endpoint.
Firstly, we'll create the namespace that will host both Falco and Falcosidekick:
We add the helm repo:
In a real project, you should get the whole chart with helm pull falcosecurity/falco --untar and then configure
the values.yaml. For this tutorial, will try to keep thing as easy as possible and set configs directly
by passing arguments to helm install command line:
You should get this output:
And you can see your new Falco and Falcosidekick pods:
Check your services within the namespace falco, and you should see that Falco service has 8765 port, let's verify that.
The argument falcosidekick.enabled=true sets the following settings in Falco for you:
The arguments --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falco-falcosidekick:2801 are there to configure the format of events and the URL where Falco will send them. As Falco and Falcosidekick will be in the same namespace, it can directly use the name of the service(falco-falcosidekick) above Falcosidekick pods.
We check the logs:
WebUI is displayed as enabled output, everything is good 👍.
Enabling Kubernetes Audit Logs
To enable Kubernetes audit logs, you need to change the arguments to the kube-apiserver process to add --audit-policy-file and --audit-webhook-config-file arguments and provide files that implement an audit policy/webhook configuration.
Let's do that.
Once you created the necessary files, we should update k3s.service Unit file.
The final k3s.service Unit file should looks like the following:
Save this file and restart k3s service.
Verify if everything is working before move on to the next step.
Now we are ready to test everything.
Test
There is a rule called Create/Modify Configmap With Private Credentials in Falco for detecting ConfigMaps that contains private credentials from Kubernetes Audit Logs, you can see the details of the rule from this link. In order to test this, we should create a ConfigMap that contains private credentials, but before doing that, let's access the Falcosidekick Web UI to see what will happen once we create the malicious ConfigMap.
Now, let's create the ConfigMap.
As soon as you created the ConfigMap, Falco will detect that from the audit logs, and send an alert for that to the Falcosidekick. Finally, Falcosidekick forwards the alert to the Web UI. So, you should see a screen similar to the following:
Conclusion
With this really simple example, we only scratched the surface of possibilities, so don't hesitate to share with us on Slack (https://kubernetes.slack.com #falco) your comments, ideas and successes. You're also always welcome to contribute.