Deploying Graylog on RKE2: A Debugging Marathon

James Tate

• 5 min read
Share
Deploying Graylog on RKE2: A Debugging Marathon

Or: how six discrete infrastructure decisions can each independently break a container in a different, misleading way.

The Goal

Deploy Graylog 6.1 onto a single-node RKE2 cluster (rke2-prod) already running GitLab and a Ghost blog, backed by:

  • MongoDB (bitnami chart, standalone) for Graylog's metadata
  • OpenSearch (opensearch-project Helm chart) as the search/indexing backend
  • Longhorn for persistent storage
  • The cluster's existing bundled ingress-nginx for external access

On paper, three helm installs and a Deployment manifest. In practice, it took nine distinct failure modes before a single pod reached Running and stayed there.

Attempt One: Permission Denied

First pass at the Graylog Deployment mounted a Longhorn PVC straight at /usr/share/graylog/data. Immediate crash:

mkdir: cannot create directory '/usr/share/graylog/data/journal': Permission denied

Longhorn provisions volumes owned by root; the Graylog image runs as UID 1100. Standard fix — add an fsGroup to the pod's securityContext so Kubernetes chowns the mount on attach:

securityContext:
  fsGroup: 1100

That cleared the permission error. It also introduced the next one, because I got greedy and added runAsUser: 1100 alongside it.

Attempt Two: The Vanishing Config File

With runAsUser: 1100 forcing the whole pod to run as non-root from the start, Graylog now failed differently:

Couldn't load configuration: Properties file /usr/share/graylog/data/config/graylog.conf doesn't exist!

Turns out the official Graylog image's entrypoint script needs to start as root, do its bootstrap work (generate config, fix ownership), then drop privilege internally — runAsUser at the pod level skips that step entirely. Removing runAsUser and keeping only fsGroup got past this.

Except it didn't, immediately — the same error kept appearing. This is where things got genuinely confusing.

Attempt Three: Reading the Actual Entrypoint

After two more rounds of guessing — wrong wait-for-it target hostname (chart-default opensearch-cluster-master, not the Helm release name graylog-opensearch), and passing graylog as an entrypoint arg (should have been graylog server) — I stopped guessing and pulled /docker-entrypoint.sh straight out of the image:

kubectl run inspect --image=graylog/graylog:6.1 --restart=Never --command -- sleep 3600
kubectl exec -it inspect -- cat /docker-entrypoint.sh

That was the turning point of the whole exercise. The script revealed something no amount of trial-and-error would have surfaced: the image ships with a pre-populated config/graylog.conf baked in at build time. There's no templating step from GRAYLOG_* env vars into a fresh config file — Graylog's Java process reads the baked-in file directly and layers env var overrides on top at runtime.

Which meant the real bug had been sitting there since step one: mounting the PVC at the entire /usr/share/graylog/data directory was hiding that pre-populated config/ folder behind an empty volume. Every single "config doesn't exist" error, across three different attempted fixes, had the same root cause.

The actual fix — mount only the subdirectory that needs persistence:

volumeMounts:
  - name: graylog-data
    mountPath: /usr/share/graylog/data/journal
    subPath: journal

With that, plus command: ["/usr/bin/tini", "--"] / args: [..., "/docker-entrypoint.sh", "server"], and the corrected OpenSearch service name, Graylog finally moved past config loading and into real startup — MongoDB connected on the first try.

Attempt Four: OpenSearch Is Too New

Progress, then a new wall:

Unsupported (Elastic/Open)Search version <OpenSearch:3.8.0>.
Supported versions: [OpenSearch ^1.0.0, OpenSearch ^2.0.0, Elasticsearch ^7.0.0, Datanode ^5.2.0]

The opensearch/opensearch Helm chart's current default pulls OpenSearch 3.x. Graylog 6.1 doesn't support it. Pin the image tag explicitly:

helm upgrade graylog-opensearch opensearch/opensearch -n graylog \
  --set image.tag="2.19.1" \
  ...

Downgrading the image doesn't downgrade data already committed to disk, though — the PVC already had 3.x-format Lucene segments written to it. Had to wipe the OpenSearch PVC and reinstall clean.

Attempt Five: The Security Plugin Gauntlet

Fresh volume, pinned to 2.19.1, and immediately:

No custom admin password found. Please provide a password via the
environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.

OpenSearch 2.12+ mandates an admin password before the security plugin's demo config will install at all. Set one via extraEnvs, upgrade — and hit the exact same error again, because helm upgrade doesn't touch existing PVCs, and once opensearch.yml exists on disk, the installer never re-runs regardless of what the env var says on a later deploy. Every password change from here on required a full PVC wipe to actually take effect. That single fact cost more debugging cycles than anything else in this whole process.

Once that pattern was understood, the actual password needed several iterations of its own:

  1. First attempt failed the security plugin's strength regex (pwgen -A generates lowercase-only by default — no uppercase, no digit, no special character)
  2. Second attempt (with real complexity) worked for OpenSearch, but broke Graylog's URI parser — the password contained a literal @, which collided with the @ separating credentials from host in https://admin:password@host:9200
  3. Even with a clean password, Graylog's JVM then refused the connection outright: None of the TrustManagers trust this certificate chain — the security plugin's self-signed demo cert wasn't trusted by Graylog's default truststore, and VERIFY_HOSTNAMES=false only skips hostname matching, not certificate chain validation

At that point, importing a self-signed CA into a JVM truststore inside someone else's container image was clearly more effort than the security plugin was worth for an internal, non-exposed connection between two pods in the same namespace. Pivoted to disabling it outright:

extraEnvs:
  - name: DISABLE_SECURITY_PLUGIN
    value: "true"

— on, again, a genuinely fresh PVC. That single change eliminated the password complexity problem, the URI-encoding problem, and the TLS trust problem simultaneously, because there was no longer anything to authenticate or encrypt on that connection at all.

Attempt Six: Out of Journal Space

Last hurdle. With OpenSearch finally healthy on plain HTTP and MongoDB already connected, Graylog got further than ever before and then stopped on:

Journal directory has not enough free space (4939 MB) available.
You need to provide additional 180 MB to contain
'message_journal_max_size = 5120 MB'

The original graylog-data PVC was sized at 5Gi — just barely too small for a 5GB default journal once filesystem overhead was accounted for. Longhorn supports online volume expansion, so a live patch fixed it with zero downtime:

kubectl -n graylog patch pvc graylog-data \
  -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}'

Restarted the pod, and:

Graylog server up and running.

What Actually Fixed It — Final State

OpenSearch values (opensearch-values.yaml):

singleNode: true
replicas: 1
persistence:
  storageClass: longhorn
  size: 30Gi
resources:
  requests:
    memory: 2Gi
  limits:
    memory: 2Gi
opensearchJavaOpts: "-Xmx1g -Xms1g"
image:
  tag: "2.19.1"
extraEnvs:
  - name: DISABLE_SECURITY_PLUGIN
    value: "true"

Graylog Deployment — the load-bearing lines:

spec:
  template:
    spec:
      securityContext:
        fsGroup: 1100
      containers:
        - name: graylog
          image: graylog/graylog:6.1
          command: ["/usr/bin/tini", "--"]
          args: ["wait-for-it", "opensearch-cluster-master:9200", "--", "/docker-entrypoint.sh", "server"]
          env:
            - name: GRAYLOG_ELASTICSEARCH_HOSTS
              value: "http://opensearch-cluster-master:9200"
            # ... GRAYLOG_PASSWORD_SECRET, GRAYLOG_ROOT_PASSWORD_SHA2, etc.
          volumeMounts:
            - name: graylog-data
              mountPath: /usr/share/graylog/data/journal
              subPath: journal

PVC sized to 10Gi, no runAsUser anywhere in the pod spec.

Lessons Worth Keeping

  1. When an entrypoint's behavior doesn't match its documentation, read the script. Every theory about templated config generation was wrong; the actual answer — a pre-baked graylog.conf — was one cat away and would have saved four separate debugging rounds if I'd pulled it first.
  2. Never mount a PVC over an entire application directory unless you mean to. A subPath mount targeting just the stateful subdirectory would have avoided the whole config-file saga from the start.
  3. helm upgrade does not reset persistent state. Any change that depends on "first boot" logic — passwords, security plugin installation, index format versions — needs a PVC wipe to actually take effect, not just a values change.
  4. Passwords destined for a URI need URI-safe characters. @, :, and # will break userinfo parsing regardless of how strong the password otherwise is.
  5. For an internal, non-exposed service in a homelab, disabling security wholesale is often more correct than half-configuring it. Chasing password regexes and TLS trust chains for a connection nothing outside the namespace ever touches was solving a problem that didn't need to exist.