Getting OpenCost Running on OpenShift: SCC Fixes, a Persistent Prometheus, and What Actually Needs a PVC

Why Cost Visibility Matters On-Prem

Cloud cost tooling is a solved problem. AWS, Azure, and GCP all ship native cost explorers, and a whole ecosystem of FinOps tooling exists to slice that billing data by team, project, or cost center. On-prem and private cloud infrastructure has no equivalent out of the box — there's no invoice to parse, no billing API to query. You own the hardware, so nobody sends you a bill.

That doesn't mean the cost went away. It just went invisible.

For any organization running Kubernetes on-prem — whether that's bare metal, VMware, or OpenShift on private infrastructure — the actual cost of a namespace, a team's workloads, or a specific application is buried in depreciation schedules, power and cooling contracts, storage array pricing, and network egress agreements that live in someone's spreadsheet, not in the cluster. When finance asks "what does Team X's workload actually cost us," the honest answer is usually a shrug.

This is where chargeback and showback models come in, and why they matter more as organizations scale. Chargeback allocates infrastructure cost directly to the business unit consuming it — the same way a cloud bill would, just synthesized from your own hardware costs rather than a vendor invoice. Showback does the same allocation without the actual billing step, purely for visibility and accountability. Both require the same foundational thing: a per-namespace, per-workload cost model that maps CPU, memory, storage, and network consumption back to a dollar (or GPU-hour, or whatever unit matters to you) figure.

OpenCost solves this for Kubernetes. It's a CNCF project, originally built by Kubecost, that computes real-time cost allocation for anything running in a cluster — broken down by namespace, deployment, label, or any other dimension you care about — using metrics it already has access to via Prometheus. Crucially, it supports custom pricing models, which is what makes it viable on-prem: you're not tied to AWS/GCP/Azure list prices, you tell it what your CPU-hour, RAM-GB-hour, and storage actually cost you, and it does the allocation math against real usage.

Why Prometheus Is a Hard Requirement (and Why It's the Thing That Must Persist)

OpenCost doesn't do its own metrics collection. It's a consumer, not a collector — it computes cost allocation from time-series data it queries out of Prometheus: CPU and memory requests/usage per pod (via kube-state-metrics and cAdvisor/kubelet), persistent volume sizes, and node-level pricing inputs. No Prometheus, no OpenCost. The OpenCost docs describe Prometheus as a prerequisite that it relies on for both scraping metrics and storing data.

That last point has a practical consequence that's easy to miss: the cost history lives in Prometheus's TSDB, not in OpenCost. Every allocation request becomes a set of range queries against Prometheus. If Prometheus loses its data, OpenCost loses its history, no matter how many volumes OpenCost itself has mounted. Making the setup durable means giving Prometheus a PersistentVolume.

On OpenShift specifically, there's a further wrinkle. OpenShift ships its own built-in monitoring stack (openshift-monitoring), which is real Prometheus under the hood — but it's not trivially exposed for arbitrary external consumers like OpenCost without extra auth plumbing (bearer tokens against Thanos Querier, specifically). For a straightforward setup, deploying a dedicated kube-prometheus-stack instance into its own namespace is the path of least resistance, and it's what this walkthrough covers.

The OpenCost values.yaml

Here's the OpenCost Helm values file used for this deployment:

# values.yaml
opencost:
  prometheus:
    internal:
      namespaceName: monitoring
      serviceName: prometheus-kube-prometheus-prometheus
      port: 9090

  dataRetention:
    dailyResolutionDays: 30  # default: 15

  exporter:
    defaultClusterId: openshift2
    replicas: 1
    resources:
      requests:
        cpu: "10m"
        memory: "55Mi"
      limits:
        memory: "1Gi"
    persistence:
      enabled: true
      storageClass: "rook-ceph-block"  # adjust based on your environment
      accessMode: ReadWriteOnce
      size: 5Gi

  customPricing:
    enabled: true
    provider: aws
    costModel:
      description: Modified pricing configuration.
      CPU: 0.031611
      spotCPU: 0.006655
      RAM: 0.004237
      spotRAM: 0.000892
      GPU: 0.95
      storage: 0.00005479452
      zoneNetworkEgress: 0.01
      regionNetworkEgress: 0.01
      internetNetworkEgress: 0.143

  metrics:
    kubeStateMetrics:
      emitKsmV1Metrics: false
      emitKsmV1MetricsOnly: false
    serviceMonitor:
      enabled: true
      additionalLabels:
        release: prometheus

  ui:
    enabled: true

A few things worth calling out:

• prometheus.internal tells OpenCost exactly where to find Prometheus — namespace, service name, and port. This has to match whatever you actually deploy, byte for byte.
• customPricing is where the on-prem story lives. Rather than pulling AWS/Azure/GCP list pricing, this hardcodes your own derived cost-per-unit figures — CPU-hour, RAM-GB-hour, storage-GB-hour, egress — however you've calculated them (amortized hardware cost, power, colocation fees, whatever your finance team uses).
• serviceMonitor.additionalLabels.release: prometheus ties the whole thing together. Prometheus Operator only picks up ServiceMonitors matching its configured label selector, and that selector is keyed off the Helm release name used for the Prometheus install. Get this label wrong (or install Prometheus under a different release name) and OpenCost's own metrics will silently never get scraped.
• exporter.persistence creates a PVC that OpenCost mounts (as opencost-export on the pod). It is harmless and worth keeping, but don't mistake it for the durability story. It is not the query cache or the cost history — that's Prometheus's job, covered below.

Standing Up Prometheus on OpenShift

The serviceName: prometheus-kube-prometheus-prometheus in the values above is the tell — this is written for the prometheus-community/kube-prometheus-stack Helm chart, installed as a release named prometheus in a monitoring namespace. That chart isn't officially certified for OpenShift, and getting it running cleanly meant working through several classes of OpenShift-specific friction.

The overrides file

# values-openshift-overrides.yaml
#
# Overrides needed on top of prometheus-community/kube-prometheus-stack's
# defaults to run under OpenShift's restricted-v2 SCC, avoid colliding with
# OpenShift's own built-in monitoring stack, and feed OpenCost.

# Top-level toggle actually disables the node-exporter subchart (condition:
# nodeExporter.enabled in Chart.yaml). Not needed for OpenCost, and its
# DaemonSet fights OpenShift's own node-exporter for hostPort 9100/hostNetwork/
# hostPID, none of which restricted-v2 allows.
nodeExporter:
  enabled: false

# Top-level toggle for kube-state-metrics (condition: kubeStateMetrics.enabled).
# Needed for OpenCost - leave enabled.
kubeStateMetrics:
  enabled: true

# kube-state-metrics is a separately maintained chart with its own values
# schema: securityContext is a map with its own "enabled" key, not a bare
# security context - {} or null breaks its template. Setting enabled: false
# lets OpenShift assign a UID/fsGroup from the namespace's own range.
kube-state-metrics:
  securityContext:
    enabled: false

prometheusOperator:
  # This chart's own securityContext is a plain map (no "enabled" key) -
  # null drops it entirely so OpenShift assigns UID/fsGroup automatically.
  securityContext: null
  # Hardcodes runAsUser: 2000 in its hook jobs, which SCC rejects.
  # Disabling sidesteps it - you lose admission-time schema validation on
  # Prometheus/Alertmanager/ServiceMonitor CRs.
  admissionWebhooks:
    enabled: false
    patch:
      enabled: false
  # The operator still serves TLS for the webhook by default even with the
  # webhook job disabled, and expects a cert secret that job would have
  # created. Turn this off too or the operator pod hangs in ContainerCreating
  # waiting on a secret that will never exist.
  tls:
    enabled: false

prometheus:
  prometheusSpec:
    # A fresh Ceph RBD volume is root-owned, so with securityContext: null the
    # pod's random UID can't write to /prometheus ("queries.active: permission
    # denied"). Explicit IDs inside the monitoring namespace's SCC range make
    # Kubernetes chown the volume to fsGroup on mount. If your namespace range
    # differs from 1001130000-1001139999, change all four values to match.
    securityContext:
      runAsNonRoot: true
      runAsUser: 1001130000
      runAsGroup: 1001130000
      fsGroup: 1001130000
    # Match OpenCost's 30 day daily-resolution window. retentionSize is a
    # safety cap so the TSDB can't fill the PVC (keep it below the PVC size).
    retention: 30d
    retentionSize: 18GB
    # Observed usage before trimming the config: ~8Gi RAM / ~0.4 CPU with no
    # limits. The limit is deliberately above that so it can't OOM on restart;
    # tighten it once you've seen the real footprint of your trimmed config.
    resources:
      requests:
        cpu: 500m
        memory: 4Gi
      limits:
        memory: 10Gi
    # Without this the TSDB lives on an emptyDir and all history is lost
    # whenever the Prometheus pod is deleted or rescheduled.
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: rook-ceph-block
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

# Alertmanager and Grafana are disabled, so the default alert/recording rules
# have nowhere to send anything and only cost CPU and memory.
defaultRules:
  create: false

# Control-plane scrapes OpenCost doesn't use (and which mostly can't be
# reached on OpenShift anyway). kubelet/cAdvisor stays enabled.
kubeApiServer:
  enabled: false
kubeControllerManager:
  enabled: false
kubeScheduler:
  enabled: false
kubeEtcd:
  enabled: false
kubeProxy:
  enabled: false
coreDns:
  enabled: false

# Not required just to feed OpenCost - disabled to keep the footprint small.
alertmanager:
  enabled: false

grafana:
  enabled: false

kubelet:
  serviceMonitor:
    # cAdvisor emits one series per container per network interface and per
    # block device, none of which OpenCost's allocation model reads (it needs
    # CPU/memory requests and usage, PV sizes, and node pricing - not per-
    # interface network counters or blkio breakdowns). On a small cluster
    # these three groups accounted for roughly half of all scraped series.
    cAdvisorMetricRelabelings:
      - sourceLabels: [__name__]
        regex: 'container_network_.*'
        action: drop
      - sourceLabels: [__name__]
        regex: 'container_blkio_device_usage_total'
        action: drop
      - sourceLabels: [__name__]
        regex: 'prober_probe_duration_seconds.*'
        action: drop

Why each of these was necessary

Node-exporter and the built-in monitoring collision. kube-prometheus-stack deploys its own node-exporter DaemonSet, requiring hostNetwork, hostPID, and a hostPort binding on 9100. OpenShift's own monitoring stack already runs a node-exporter with exactly the same host-level requirements, and none of them are permitted under restricted-v2 anyway. Since OpenCost's cost allocation doesn't depend on node-exporter metrics (it uses kube-state-metrics and cAdvisor/kubelet data), the clean fix is disabling it outright rather than trying to grant it a privileged SCC. Note the toggle: the subchart is switched off by the top-level nodeExporter.enabled, not by prometheus-node-exporter.enabled — the latter only configures the subchart once it's already running.

SCC and fixed UIDs, repeatedly. This was the recurring theme. restricted-v2 — OpenShift's default SCC — refuses to run any pod requesting a specific UID or GID outside the namespace's own auto-assigned range (visible directly in the SCC denial message: must be in the ranges: [1001130000, 1001139999]). The chart hardcodes UID 65534 (nobody) across most of its components, and UID 2000 specifically in the admission-webhook hook jobs. Helm's value merging matters here: setting a value to {} merges with the chart's defaults and changes nothing, whereas setting it to null actually drops the key so OpenShift can assign its own UID. The one exception was kube-state-metrics, a separately maintained chart that restructured its securityContext value into a map with its own enabled flag — passing null there broke the template outright with a nil-pointer error, so it needed enabled: false instead. The lesson: don't assume every subchart under one umbrella chart uses the same values schema. A cheap habit that would have saved several install/uninstall cycles: render locally with helm template ... -f values-openshift-overrides.yaml --skip-crds first, which catches template errors without touching the cluster.

The admission webhook. The operator's admission-webhook mechanism runs a short-lived Job to self-sign TLS certs and patch the webhook configuration. That Job's pod spec also hardcodes a UID, hits the same SCC wall, and — because it's a Helm pre-upgrade hook — a stuck hook job causes Helm itself to hang or fail with context deadline exceeded, and its cleanup logic deletes the RBAC objects it created, which briefly makes debugging attempts look like resources are randomly vanishing. Disabling the webhook (admissionWebhooks.enabled: false) avoids the whole problem; the trade-off is losing admission-time schema validation on Prometheus/Alertmanager/ServiceMonitor custom resources, which is an acceptable loss for a single-tenant monitoring install feeding a cost tool. One easily-missed follow-up: disabling the webhook job doesn't disable the operator's own TLS serving for it — prometheusOperator.tls.enabled needs to be turned off separately, or the operator pod hangs indefinitely in ContainerCreating waiting to mount a certificate secret that the (now-disabled) job would have created.

CRD ownership conflicts. OpenShift's cluster-version-operator already owns the Prometheus Operator CRDs cluster-wide as part of the built-in monitoring stack. A fresh Helm install trying to apply its own copies of those CRDs fails with a server-side-apply field-manager conflict. The fix is --skip-crds on the Helm install/upgrade commands — the CRDs already on the cluster are compatible, and the new operator instance just uses them.

RBAC and OwnerReferencesPermissionEnforcement. Kubernetes has an admission plugin, OwnerReferencesPermissionEnforcement, that OpenShift enables by default (many vanilla clusters don't). It requires explicit RBAC — an update verb on the relevant CRD's /finalizers subresource — before a controller can set blockOwnerDeletion: true on an owner reference. The bundled kube-prometheus-stack ClusterRole doesn't grant this for its own CRDs, so on OpenShift the operator would successfully create the Prometheus custom resource, then fail every reconciliation loop trying to own its child ConfigMaps, with the deeply unhelpful error cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on. The fix is an additive ClusterRole/ClusterRoleBinding granted directly to the operator's ServiceAccount:

# prometheus-operator-finalizers-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-operator-finalizers
rules:
  - apiGroups: ["monitoring.coreos.com"]
    resources:
      - alertmanagers/finalizers
      - prometheuses/finalizers
      - prometheusagents/finalizers
      - thanosrulers/finalizers
    verbs: ["update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-operator-finalizers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-operator-finalizers
subjects:
  - kind: ServiceAccount
    name: prometheus-kube-prometheus-operator
    namespace: monitoring

Since this RBAC isn't something Helm manages, it survives a helm uninstall and needs cleaning up manually if the stack is ever torn down. The operator also only retries on a resource change or its resync interval, so after applying the RBAC, restart the operator (kubectl rollout restart) to make it retry immediately rather than waiting.

Persistence, and the fsGroup trap. This is the part the first version of this setup got wrong. Out of the box, kube-prometheus-stack gives Prometheus an emptyDir for its TSDB — fine for a demo, but any pod deletion or reschedule wipes all metric history, and with it OpenCost's cost history. The fix is a storageSpec.volumeClaimTemplate on the Prometheus spec, sized for your retention. Two knobs worth setting alongside it: retention (here 30d, matching OpenCost's dailyResolutionDays) and retentionSize (a safety cap below the PVC size so the TSDB can't fill the volume and wedge Prometheus).

The trap: with securityContext: null (which we used to let OpenShift pick a UID), the pod runs as an arbitrary UID from the namespace range — but a freshly provisioned Ceph RBD volume is root-owned. Prometheus then fails at startup with open /prometheus/queries.active: permission denied. The fix is to stop relying on OpenShift's automatic assignment for this one component and specify runAsUser, runAsGroup, and fsGroup explicitly, using IDs from the namespace's allowed range. Kubernetes then recursively fixes ownership on the volume at mount time. Find your namespace's range with:

oc get ns monitoring -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.uid-range}{"\n"}{.metadata.annotations.openshift\.io/sa\.scc\.supplemental-groups}{"\n"}'

Trimming what you don't use, and sizing what you keep. Before persistence, the default install was consuming roughly 8Gi of memory and around 0.4 CPU on a small cluster, with no limits set. With Alertmanager and Grafana disabled, the chart's default alerting/recording rules and its control-plane scrape jobs (API server, etcd, scheduler, controller-manager, kube-proxy, CoreDNS — most of which can't be scraped on OpenShift anyway) produce data nothing consumes, so the overrides turn them off. Kubelet/cAdvisor and kube-state-metrics stay on, since those are what OpenCost actually reads. A caveat: this trim is based on what OpenCost is documented to need, not an exhaustive audit — after applying it, confirm the allocation API still returns sensible data.

On sizing: measure before you limit. The memory limit in the overrides (10Gi) is intentionally set above the observed 8Gi so a restart can't OOM-loop; tighten it once you've watched the trimmed config settle. If memory stays high on a small cluster, that points at series cardinality — Prometheus will tell you which metrics are responsible:

kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9091:9090 &
sleep 2
curl -s localhost:9091/api/v1/status/tsdb | python3 -c "import sys,json; d=json.load(sys.stdin)['data']; print('head series:', d['headStats']['numSeries']); [print(x['value'], x['name']) for x in d['seriesCountByMetricName'][:15]]"

The Install, Start to Finish

# 1. Add the Helm repo, create the namespace
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
oc new-project monitoring

# 2. Sanity-check the values render before touching the cluster
helm template prometheus prometheus-community/kube-prometheus-stack \
  -f values-openshift-overrides.yaml --skip-crds > /tmp/rendered.yaml && echo "render OK"

# 3. Install kube-prometheus-stack, skipping CRDs OpenShift already owns
helm install prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring \
  -f values-openshift-overrides.yaml \
  --skip-crds \
  --timeout 5m

# 4. Grant the finalizers RBAC the chart doesn't ship
kubectl apply -f prometheus-operator-finalizers-rbac.yaml

# 5. Force the operator to retry reconciliation with the new permission
kubectl rollout restart deployment/prometheus-kube-prometheus-operator -n monitoring

# 6. Confirm Prometheus is up and its volume is bound
kubectl get prometheus -n monitoring
kubectl get pvc -n monitoring
kubectl get pods -n monitoring

# 7. Install OpenCost against it
helm repo add opencost-charts https://opencost.github.io/opencost-helm-chart
helm repo update
helm install opencost opencost-charts/opencost \
  --namespace opencost --create-namespace \
  -f values.yaml

# 8. Confirm it's pulling real data
kubectl port-forward -n opencost svc/opencost 9003 9090
curl -s "http://localhost:9003/allocation/compute?window=1h"

Verifying It's Healthy (and Staying That Way)

A 200 with allocation data from that last curl confirms the chain: Prometheus scraping cluster metrics, OpenCost's ServiceMonitor matched and scraped in turn, and the custom pricing model applied against real usage. Three further checks are worth making, because a working first response doesn't prove the setup is durable:

# 1. The TSDB is on a PVC, not an emptyDir (this should print a claimName, not {"emptyDir":{}})
kubectl get pod -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
  -o jsonpath='{.spec.volumes[?(@.name=="prometheus-prometheus-kube-prometheus-prometheus-db")]}{"\n"}'

# 2. Every scrape target is up - kubelet in particular, or OpenCost gets no usage data
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9091:9090 &
sleep 2
curl -s localhost:9091/api/v1/targets | python3 -c "
import sys,json
for t in json.load(sys.stdin)['data']['activeTargets']:
    print(t['health'].ljust(5), t['labels'].get('job'), t['scrapeUrl'])
"

# 3. Query latency for short vs long windows, with the OpenCost port-forward running
time curl -s "http://localhost:9003/allocation/compute?window=1h" -o /dev/null -w "%{http_code}\n"
time curl -s "http://localhost:9003/allocation/compute?window=7d" -o /dev/null -w "%{http_code}\n"

Expect long windows to return partial data until Prometheus has accumulated that much history. If short windows are fast but long ones are slow, that's query cost over a wide time range against a busy TSDB rather than missing data — OpenCost's own docs note that Prometheus-based queries may not scale well over large time windows on large clusters.

Closing Thoughts

None of the individual fixes here are exotic — disable what you don't need, clear hardcoded UIDs, grant the RBAC OpenShift additionally requires, and give Prometheus a real volume. What makes this genuinely time-consuming is that kube-prometheus-stack bundles several independently maintained subcharts, each with slightly different conventions for security contexts and enable/disable toggles, and OpenShift's SCC and admission-plugin defaults surface every inconsistency as a distinct, cryptic failure.

The subtler lesson is about where state lives. It's tempting to see OpenCost's own PVC and assume the cost data is safe, but OpenCost is a lens over Prometheus. A "working" install that leaves Prometheus on an emptyDir will look perfect on day one and quietly lose its history the first time the pod moves. Check the TSDB volume, not just the pods.

Once you've been through it, the values-openshift-overrides.yaml and RBAC manifest above are reusable for the next OpenShift cluster that needs the same stack.

Sources: OpenCost documentation, OpenCost Prometheus exporter docs.