Shipping Kasten K10 Logs to Graylog Across Clusters

James Tate

• 7 min read
Share
Shipping Kasten K10 Logs to Graylog Across Clusters

A practical guide to forwarding Kasten K10's control-plane logs from an OpenShift cluster into a Graylog instance running on a separate RKE2 cluster, using Fluent Bit and GELF.

In the previous post we walked through standing up Graylog itself on an RKE2 cluster — MongoDB and OpenSearch as backing services, all on Longhorn storage — and the string of environment-specific gotchas that came with it: a pre-baked config file hidden by an over-broad PVC mount, an OpenSearch major version mismatch, a security plugin that wouldn't take a password, and a journal directory sized just a little too small. By the end of it, Graylog was up, reachable, and ready to actually receive logs from something. This post picks up from there and covers getting a real workload's logs into it.

Why bother

Kasten K10's own log viewer is fine for a quick look, but it doesn't persist history, doesn't let you search across time windows, and disappears the moment a pod restarts. If you're running K10 in a pre-sales lab, a customer POC, or just want a searchable record of backup/restore activity, forwarding its logs into a proper log aggregator pays off quickly — especially the first time something misbehaves and you need to know exactly when it started and how often it's recurring.

This guide covers the whole path: exposing a GELF input on Graylog, making it reachable across cluster boundaries, and deploying a scoped Fluent Bit collector on OpenShift that ships only Kasten's own namespace.

Assumes:

  • Graylog already running (any reasonably recent version) on one cluster — this guide uses RKE2 as the example, but the steps are the same for any Kubernetes distribution
  • Kasten K10 running on a separate OpenShift cluster
  • Both clusters reachable over the same network (same LAN, VPN, or otherwise routable)

Part 1: Graylog side — create a GELF UDP input

Graylog needs an actual Input object listening before anything can send it messages — the Kubernetes Service alone doesn't do this.

Via the Graylog web UI:

  1. Log in to Graylog
  2. System → Inputs
  3. Select GELF UDP from the dropdown, click Launch new input
  4. Set:
    • Title: something identifiable, e.g. Kasten K10 GELF UDP
    • Bind address: 0.0.0.0
    • Port: 12201 (GELF's conventional default)
    • Leave the rest as default
  5. Click Save

Or via the REST API, if you'd rather script it:

curl -sk -u admin:'<your-admin-password>' -X POST \
  https://<your-graylog-host>/api/system/inputs \
  -H 'Content-Type: application/json' \
  -H 'X-Requested-By: cli' \
  -d '{
    "title": "Kasten K10 GELF UDP",
    "type": "org.graylog2.inputs.gelf.udp.GELFUDPInput",
    "configuration": {
      "bind_address": "0.0.0.0",
      "port": 12201,
      "recv_buffer_size": 262144,
      "decompress_size_limit": 8388608
    },
    "global": true
  }'

Confirm it's actually running (not just created):

curl -sk -u admin:'<your-admin-password>' -H 'X-Requested-By: cli' \
  https://<your-graylog-host>/api/system/inputstates | jq

Look for "state": "RUNNING" against the input's ID.

Part 2: RKE2 side — expose the GELF port externally

If Graylog runs as a ClusterIP Service (the usual default), it's only reachable from inside its own cluster. A pod on a different cluster needs a NodePort (or a LoadBalancer, if your environment has one) to reach it.

Patch the existing Service to add a NodePort for the GELF UDP port, alongside whatever ports it already exposes:

kubectl -n <graylog-namespace> patch svc graylog -p '{
  "spec": {
    "type": "NodePort",
    "ports": [
      {"name": "http", "port": 9000, "targetPort": 9000},
      {"name": "gelf-udp", "port": 12201, "targetPort": 12201, "protocol": "UDP", "nodePort": 31221}
    ]
  }
}'

Verify:

kubectl -n <graylog-namespace> get svc graylog

You should see TYPE: NodePort and 12201:31221/UDP in the ports list.

Test the path end-to-end before touching the other cluster. From any host on the same network as the RKE2 node:

echo '{"version":"1.1","host":"test-host","short_message":"manual gelf test","level":6}' | nc -u -w1 <rke2-node-ip> 31221

Then check Graylog's search UI for a message from test-host in the last few minutes. If it shows up, the network path, firewall rules, and input are all confirmed working — worth doing before deploying anything to OpenShift, since UDP failures are silent and much easier to debug from this side than after adding a collector into the mix.

(If your RKE2 host runs firewalld or a similar host firewall, make sure the NodePort range — 30000–32767 by default — is allowed, or the packet will be dropped before it ever reaches kube-proxy.)

Part 3: OpenShift side — deploy Fluent Bit, scoped to Kasten's namespace

We'll run Fluent Bit as a DaemonSet, tailing only the kasten-io namespace's container logs, enriching them with Kubernetes metadata, and forwarding as GELF.

3.1 Namespace, ServiceAccount, and RBAC

# fluent-bit-rbac.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: logging
roleRef:
  kind: ClusterRole
  name: fluent-bit-read
  apiGroup: rbac.authorization.k8s.io
EOF

oc apply -f fluent-bit-rbac.yaml

Worth verifying the binding actually took, since a raw ClusterRoleBinding manifest can silently fail to apply on some OpenShift clusters depending on admission configuration:

oc get clusterrolebinding fluent-bit-read
oc auth can-i get pods --as=system:serviceaccount:logging:fluent-bit -n kasten-io
oc auth can-i list pods --as=system:serviceaccount:logging:fluent-bit -n kasten-io

Both can-i checks should return yes. If the binding is missing or the checks return no, create it directly via OpenShift's own RBAC helper instead:

oc adm policy add-cluster-role-to-user fluent-bit-read system:serviceaccount:logging:fluent-bit

3.2 Security Context Constraint (SCC)

Fluent Bit needs hostPath access to read node-level container logs — OpenShift's default restricted SCC blocks this outright.

oc adm policy add-scc-to-user privileged -z fluent-bit -n logging

Granting the SCC alone isn't enough — the pod's own securityContext also has to explicitly request privileged mode (see the DaemonSet spec below). Both sides — the grant and the request — are required.

3.3 Fluent Bit configuration

This tails only kasten-io container logs, attaches pod/namespace metadata via the kubernetes filter, and ships everything as GELF.

# fluent-bit-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Daemon        off

    [INPUT]
        Name              tail
        Path              /var/log/containers/*_kasten-io_*.log
        Parser            cri
        Tag               kube.*
        Refresh_Interval  5
        Mem_Buf_Limit     20MB
        Skip_Long_Lines  On

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Merge_Log           On
        Keep_Log            On

    [OUTPUT]
        Name                    gelf
        Match                   kube.*
        Host                    <rke2-node-ip>
        Port                    31221
        Mode                    udp
        Gelf_Short_Message_Key  log

oc apply -f fluent-bit-config.yaml

A couple of details worth flagging, since they're easy to get wrong:

  • Use Fluent Bit's built-in cri parser (as above — just reference Parser cri with no custom parsers.conf). It's tempting to hand-write a CRI-format regex parser, but the built-in one names its raw-log capture group log, which is exactly the field name Merge_Log and Gelf_Short_Message_Key both expect. A hand-rolled parser using a different field name (e.g. message) will silently break both.
  • Path filtering by namespace works because of how CRI-O names log files — <pod-name>_<namespace>_<container-name>-<container-id>.log — so *_kasten-io_*.log cleanly matches only that namespace without needing a separate filter step.

3.4 The DaemonSet

# fluent-bit-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels: { app: fluent-bit }
  template:
    metadata:
      labels: { app: fluent-bit }
    spec:
      serviceAccountName: fluent-bit
      containers:
        - name: fluent-bit
          image: cr.fluentbit.io/fluent/fluent-bit:3.1
          securityContext:
            privileged: true
            runAsUser: 0
          volumeMounts:
            - name: config
              mountPath: /fluent-bit/etc/
            - name: varlog
              mountPath: /var/log
            - name: varlibcontainers
              mountPath: /var/lib/containers
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: fluent-bit-config
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibcontainers
          hostPath:
            path: /var/lib/containers

oc apply -f fluent-bit-daemonset.yaml

3.5 Verify

oc -n logging get pods -w
oc -n logging logs -f daemonset/fluent-bit

You're looking for:

  • [input:tail:tail.0] inotify_fs_add(): ... name=/var/log/containers/<kasten-pod>_kasten-io_...log lines for each Kasten component — confirms the path filter found the right files
  • No [error] [input:tail:tail.0] read error, check permissions — if you see this, the SCC/securityContext privilege step didn't take effect
  • No [error] [flb_msgpack_to_gelf] missing short_message key — if you see this, double-check the parser and Gelf_Short_Message_Key settings match (both should reference log)

Part 4: Confirm messages are landing in Graylog

Simplest check is the web UI:

  1. Search bar → kubernetes_namespace_name:kasten-io
  2. Time range → "Last 5 minutes"
  3. You should see live log lines from K10's components — frontend-svc, jobs-svc, state-svc, kanister-svc, etc.

Expand any message to see the full field list Fluent Bit's kubernetes filter attached — kubernetes_pod_name, kubernetes_namespace_name, kubernetes_container_name, kubernetes_labels_*, and so on (nested JSON gets flattened with underscores once GELF-indexed).

Part 5: Organize it — a dedicated Stream

Once messages are flowing, route them into their own Stream so they don't mix into the Default Stream with everything else Graylog might be collecting:

  1. Streams → Create Stream → name it something like Kasten K10
  2. Inside the new stream, Create Rule:
    • Field: kubernetes_namespace_name
    • Type: must match exactly
    • Value: kasten-io
  3. Save, then Start Stream

From here, searching that stream directly gives a clean, scoped view of everything Kasten-related — and it's the natural foundation for setting up Alerts later if you want to be notified the moment something starts erroring repeatedly, rather than discovering it days later during an unrelated search.

Wrapping up

That's the whole path: a GELF input on Graylog, a NodePort to make it reachable across cluster boundaries, and a namespace-scoped Fluent Bit DaemonSet on OpenShift doing the collection. The pattern generalizes easily beyond Kasten — the same approach works for any namespace on any cluster you want centralized logging for, just by changing the Path glob and the stream's matching rule.

One practical note from having built this in a real lab environment: application logs aren't always clean, consistent JSON. Different components — and even different versions of the same product — can emit structured logs in genuinely inconsistent formats. If you're planning to filter or route based on log severity or content, expect to spend some time validating your filter rules against real traffic rather than assuming the first pass catches everything.