APIServer Audit Feature

The audit feature enables the cluster administrator to answer the following questions:

What happened?
When did it happen?
Who triggered it?
On which object(s) did the activity happen?
Where was it observed?
Where was it triggered from?
What was the subsequent behavior of the activity?

The audit log record function will increase the memory consumption of the API server as it needs to store some context required for the audit for each request. In addition, the memory consumption depends on the configuration of the audit log recording.

1. Audit Policy

Set your own audit policy by editing the file /etc/kubernetes/audit-policy.yaml.


apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  # The cluster contains a large number of the following low-risk requests, it is recommended not to audit (do not log)
  # Watch request of kube-proxy
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" # core
        resources: ["endpoints", "services", "services/status"]
  # Get request for configmap under kube-system namespace
  - level: None
    users: ["system:unsecured"]
    namespaces: ["kube-system"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["configmaps"]
  # kubelet's get request for node
  - level: None
    users: ["kubelet"] # legacy kubelet identity
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes", "nodes/status"]
  # system:node user group's get request for node
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes", "nodes/status"]
  # Get/update requests for endpoints in kube-system namespace by system components
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["endpoints"]
  # apiserver's get request for namespace
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["namespaces", "namespaces/status", "namespaces/finalize"]
  # Get/update requests for configmap, endpoint in kube-system namespace by cluster-autoscaler
  - level: None
    users: ["cluster-autoscaler"]
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["configmaps", "endpoints"]
  # HPA's requests for metrics information through controller manager
  - level: None
    users:
      - system:kube-controller-manager
    verbs: ["get", "list"]
    resources:
      - group: "metrics.k8s.io"
  # The following read-only URL
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # event
  - level: None
    resources:
      - group: "" # core
        resources: ["events"]
 
  # Update and patch requests for nodes by kubelet, system:node-problem-detector and system:nodes, level set to Request, recording metadata and request body
  - level: Request
    users: ["kubelet", "system:node-problem-detector", "system:serviceaccount:kube-system:node-problem-detector"]
    verbs: ["update","patch"]
    resources:
      - group: "" # core
        resources: ["nodes/status", "pods/status"]
  - level: Request
    userGroups: ["system:nodes"]
    verbs: ["update","patch"]
    resources:
      - group: "" # core
        resources: ["nodes/status", "pods/status"]
 
  # The log level for Secrets, ConfigMaps, tokenreviews interfaces that may contain sensitive information or binary files is set to Metadata
  - level: Metadata
    resources:
      - group: "" # core
        resources: ["secrets", "configmaps", "serviceaccounts/token"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
 
  # For some get, list, watch requests with a large return body, set to Request
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apiextensions.k8s.io"
      - group: "apiregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "metrics.k8s.io"
      - group: "networking.k8s.io"
      - group: "node.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "scheduling.k8s.io"
      - group: "storage.k8s.io"
 
  # Set to RequestResponse for Known Kubernetes API
  - level: RequestResponse
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apiextensions.k8s.io"
      - group: "apiregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "metrics.k8s.io"
      - group: "networking.k8s.io"
      - group: "node.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "scheduling.k8s.io"
      - group: "storage.k8s.io"
 
  # Set all other requests as Metadata 
  - level: Metadata

1.1 Phase (omitStages)

Stage	Indicates
RequestReceived	This stage corresponds to the event generated after the audit processor receives the request and before delegating to other processors
ResponseStarted	The event generated after the response message header is sent and before the response body is sent. Only long-running requests (such as watch) generate this stage
ResponseComplete	When the response body is complete and no more data needs to be transferred
Panic	Generated when panic occurs

1.2 Audit Level (level)

Level	Indicates
None	Logs that comply with this rule will not be recorded
Metadata	Records the metadata of the request (user, timestamp, resource, verb, etc.) but does not record the request or response body
Request	Records the metadata of the event and request body, but not the response body. This does not apply to non-resource requests
RequestResponse	Records event metadata, request and response body. This does not apply to non-resource requests

2. Audit Log Configuration

Log in to the 3 Master nodes separately, add the following parameters in the APIServer configuration file /etc/kubernetes/apiserver, and restart APIServer with systemctl restart kube-apiserver:


# Specifies the log file path to write the audit events. Not specifying this flag will disable the log backend.
--audit-log-path=/var/log/audit.log
# Specifies the audit policy configuration file
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
# Specifies the maximum number of days to keep old audit log files.
--audit-log-maxage=7
# Specifies the maximum number of audit log files to keep.
--audit-log-maxbackup=10
# Specifies the maximum size of an audit log file in megabytes.
--audit-log-maxsize=1000

3. Reference

Kubernetes Official Documentation - Audit