Log Writer and Log Reader Pod

Today I Learned

Kubernetes, and more specifically Docker, push you to use a pattern of logging to standard out (stdout/stderr). If you have applications that instead log to a file, it can be tricky to get these logs where you want them to go.

A sidecar will run a new container inside your pod, and both your existing container (e.g. log-writer) and the new container (e.g. log-reader) will mount a shared volume. Once you have the shared volume, a writer can write data here and the reader can output wherever it needs to go (e.g. remote SIEM, syslog, etc.).

Example Config

It’ll run the initContainers before the others start, making sure the audit file exists. This will prevent the healthchecks on the other containers from failing.

Using tail to read the audit log

kubectl apply -f sidecar-example.yml

# Read the log-writer logs from the log-reader container
kubectl logs log-writer-and-reader -c log-reader

sidecar-example.yml

apiVersion: v1
kind: Pod
metadata:
  name: log-writer-and-reader
spec:
  initContainers:
    - name: init-log-file
      image: busybox:1.31
      command: ["touch", "/var/log/audit.log"]
      volumeMounts:
        - name: logs
          mountPath: /var/log
  containers:
    - name: log-reader
      image: ubuntu:latest
      command: ["tail", "-f", "/var/log/audit.log"]
      volumeMounts:
        - name: logs
          mountPath: /var/log
    - name: log-writer
      image: busybox:1.31
      command:
        ["sh", "-c", "while true; do echo $(date +'%Y-%m-%dT%H:%M:%S') >> /var/log/audit.log; sleep 2; done"]
      volumeMounts:
        - name: logs
          mountPath: /var/log
  volumes:
    - name: logs
      emptyDir: {}

Using fluentd to read the log

kubectl apply -f sidecar-example-with-fluentd.yml

# Read the log-writer logs from the log-reader container
kubectl logs log-writer-and-reader -c log-reader

sidecar-example-with-fluentd.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluentd.conf: |
    <source>
      @type tail
      tag audit.logs
      path /var/log/audit.log
      pos_file /var/log/audit.log.pos
      format none
    </source>

    <match **>
      @type stdout
    </match>
---
apiVersion: v1
kind: Pod
metadata:
  name: log-writer-and-fluentd
spec:
  initContainers:
    - name: init-log-file
      image: busybox:1.31
      command: ["touch", "/var/log/audit.log"]
      volumeMounts:
        - name: logs
          mountPath: /var/log
  containers:
    - name: log-writer
      image: busybox:1.31
      command:
        [
          "sh",
          "-c",
          "while true; do echo $(date +'%Y-%m-%dT%H:%M:%S') >> /var/log/audit.log; sleep 2; done",
        ]
      volumeMounts:
        - name: logs
          mountPath: /var/log
    - name: fluentd
      image: fluent/fluentd:v1.16-1
      env:
        - name: FLUENTD_CONF
          value: "fluentd.conf"
      volumeMounts:
        - name: logs
          mountPath: /var/log
        - name: fluentd-config
          mountPath: /fluentd/etc/fluentd.conf
          subPath: fluentd.conf
      ports:
        - containerPort: 24224
  volumes:
    - name: logs
      emptyDir: {}
    - name: fluentd-config
      configMap:
        name: fluentd-config

Hashicorp Vault Audit Logs

How can we use this to solve a real problem?

Let’s look at Hashicorp Vault. Vault supports setting up an audit log for all actions that take place, but the current options for remote logging are limited. It’s also against best practice to add an additional service (e.g. logging agent) to a container. This leads us to using a sidecar to ship audit logs from Vault.

Vault Helm with Log Reader

Using tail to read the audit log

These values override the default values of the Helm chart.

# See full values.yml below
helm install vault hashicorp/vault --version 0.25.0 --values values.yml

# Connect to pod in order to enable audit logging
kubectl exec -it vault-0 -c vault -- /bin/sh
vault audit enable file file_path=/vault_audit/audit.log

# you should be able to see all the logs from your log-reader container
kubectl logs vault-0 -c log-reader

values.yml

global:
  enabled: true
server:
  affinity: ""
  updateStrategyType: RollingUpdate
  # Use dev mode so vault is already initialized and unsealed for testing
  dev:
    enabled: true
  volumes:
    - name: logs
      emptyDir: {}
  volumeMounts:
    - name: logs
      mountPath: /vault_audit
  extraInitContainers:
  # Create the audit.log file so health checks on other containers don't fail
  - name: init-log-file
    image: busybox:1.31
    command: ["touch", "/vault_audit/audit.log"]
    volumeMounts:
    - name: logs
      mountPath: /vault_audit
  extraContainers:
  # Replace with a container like fluentd that will read and ship logs to external systems
    - name: log-reader
      image: ubuntu:latest
      command: ["tail", "-f", "/vault_audit/audit.log"]
      volumeMounts:
        - name: logs
          mountPath: /vault_audit

Using fluentd to read the log

First set up a ConfigMap storing your fluentd config.

kubectl apply -f fluentd-configmap.yaml

helm install vault hashicorp/vault --version 0.25.0 --values values.yml
kubectl exec -it vault-0 -c vault -- /bin/sh
vault audit enable file file_path=/vault_audit/audit.log
kubectl logs vault-0 -c fluentd

fluentd-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluentd.conf: |
    <source>
      @type tail
      tag audit.logs
      path /vault_audit/audit.log
      pos_file /vault_audit/audit.log.pos
      format none
    </source>

    <match **>
      @type stdout
    </match>

values.yml

global:
  enabled: true
server:
  affinity: ""
  updateStrategyType: RollingUpdate
  dev:
    enabled: true
  volumes:
    - name: logs
      emptyDir: {}
    - name: fluentd-config
      configMap:
        name: fluentd-config
  volumeMounts:
    - name: logs
      mountPath: /vault_audit
  extraInitContainers:
    - name: init-log-file
      image: busybox:1.31
      command: ["touch", "/vault_audit/audit.log"]
      volumeMounts:
        - name: logs
          mountPath: /vault_audit
  extraContainers:
    - name: fluentd
      image: fluent/fluentd:v1.16-1
      env:
        - name: FLUENTD_CONF
          value: "fluentd.conf"
      volumeMounts:
        - name: logs
          mountPath: /vault_audit
        - name: fluentd-config
          mountPath: /fluentd/etc/fluentd.conf
          subPath: fluentd.conf
      ports:
        - containerPort: 24224

For the reader

With this, you can set up a sidecar container running Fluentd to read the audit logs created by your other containers. Now, you can update the Fluentd configuration to send these logs to a remote collection point using your favorite output plugin.