Getting OpenCost Running on OpenShift

James Tate

• 8 min read
Share
Getting OpenCost Running on OpenShift

Why Cost Visibility Matters On-Prem

Cloud cost tooling is a solved problem. AWS, Azure, and GCP all ship native cost explorers, and a whole ecosystem of FinOps tooling exists to slice that billing data by team, project, or cost centre. On-prem and private cloud infrastructure has no equivalent out of the box — there's no invoice to parse, no billing API to query. You own the hardware, so nobody sends you a bill.

That doesn't mean the cost went away. It just went invisible.

For any organisation running Kubernetes on-prem — whether that's bare metal, VMware, or OpenShift on private infrastructure — the actual cost of a namespace, a team's workloads, or a specific application is buried in depreciation schedules, power and cooling contracts, storage array pricing, and network egress agreements that live in someone's spreadsheet, not in the cluster. When finance asks "what does Team X's workload actually cost us," the honest answer is usually a shrug.

This is where chargeback and showback models come in, and why they matter more as organisations scale. Chargeback allocates infrastructure cost directly to the business unit consuming it — the same way a cloud bill would, just synthesized from your own hardware costs rather than a vendor invoice. Showback does the same allocation without the actual billing step, purely for visibility and accountability. Both require the same foundational thing: a per-namespace, per-workload cost model that maps CPU, memory, storage, and network consumption back to a dollar (or GPU-hour, or whatever unit matters to you) figure.

OpenCost solves this for Kubernetes. It's a CNCF project, originally built by Kubecost, that computes real-time cost allocation for anything running in a cluster — broken down by namespace, deployment, label, or any other dimension you care about — using metrics it already has access to via Prometheus. Crucially, it supports custom pricing models, which is what makes it viable on-prem: you're not tied to AWS/GCP/Azure list prices, you tell it what your CPU-hour, RAM-GB-hour, and storage actually cost you, and it does the allocation math against real usage.

Why Prometheus Is a Hard Requirement

OpenCost doesn't do its own metrics collection. It's a consumer, not a collector — it computes cost allocation entirely from time-series data it queries out of Prometheus: CPU and memory requests/usage per pod (via kube-state-metrics and cAdvisor/kubelet), persistent volume sizes, and node-level pricing inputs. No Prometheus, no OpenCost. This is non-negotiable, and it's the first thing worth checking before attempting an install: do you already have a Prometheus Operator-managed instance on the cluster, or do you need to stand one up?

On OpenShift specifically, this question has a wrinkle. OpenShift ships its own built-in monitoring stack (openshift-monitoring), which is real Prometheus under the hood — but it's not trivially exposed for arbitrary external consumers like OpenCost without extra auth plumbing (bearer tokens against Thanos Querier, specifically). For a straightforward setup, deploying a dedicated kube-prometheus-stack instance into its own namespace is the path of least resistance, and it's what this walkthrough covers.

The OpenCost values.yaml

Here's the OpenCost Helm values file used for this deployment:

# values.yaml
opencost:
  prometheus:
    internal:
      namespaceName: monitoring
      serviceName: prometheus-kube-prometheus-prometheus
      port: 9090

  dataRetention:
    dailyResolutionDays: 30  # default: 15

  exporter:
    defaultClusterId: openshift2
    replicas: 1
    resources:
      requests:
        cpu: "10m"
        memory: "55Mi"
      limits:
        memory: "1Gi"
    persistence:
      enabled: true
      storageClass: "rook-ceph-block"  # adjust based on your environment
      accessMode: ReadWriteOnce
      size: 5Gi

  customPricing:
    enabled: true
    provider: aws
    costModel:
      description: Modified pricing configuration.
      CPU: 0.031611
      spotCPU: 0.006655
      RAM: 0.004237
      spotRAM: 0.000892
      GPU: 0.95
      storage: 0.00005479452
      zoneNetworkEgress: 0.01
      regionNetworkEgress: 0.01
      internetNetworkEgress: 0.143

  metrics:
    kubeStateMetrics:
      emitKsmV1Metrics: false
      emitKsmV1MetricsOnly: false
    serviceMonitor:
      enabled: true
      additionalLabels:
        release: prometheus

  ui:
    enabled: true

A few things worth calling out:

  • prometheus.internal tells OpenCost exactly where to find Prometheus — namespace, service name, and port. This has to match whatever you actually deploy, byte for byte.
  • customPricing is where the on-prem story lives. Rather than pulling AWS/Azure/GCP list pricing, this hardcodes your own derived cost-per-unit figures — CPU-hour, RAM-GB-hour, storage-GB-hour, egress — however you've calculated them (amortized hardware cost, power, colocation fees, whatever your finance team uses).
  • serviceMonitor.additionalLabels.release: prometheus is the detail that ties this whole thing together. Prometheus Operator only picks up ServiceMonitors matching its configured label selector, and that selector is keyed off the Helm release name used for the Prometheus install. Get this label wrong (or install Prometheus under a different release name) and OpenCost's own metrics will silently never get scraped.

Standing Up Prometheus on OpenShift

The serviceName: prometheus-kube-prometheus-prometheus in the values above is the tell — this is written for the prometheus-community/kube-prometheus-stack Helm chart, installed as a release named prometheus in a monitoring namespace. That chart isn't officially certified for OpenShift, and getting it running cleanly meant working through several classes of OpenShift-specific friction.

The overrides file

# values-openshift-overrides.yaml
#
# Overrides needed on top of prometheus-community/kube-prometheus-stack's
# defaults to run under OpenShift's restricted-v2 SCC, avoid colliding with
# OpenShift's own built-in monitoring stack, and feed OpenCost.

# Top-level toggle actually disables the node-exporter subchart (condition:
# nodeExporter.enabled in Chart.yaml). Not needed for OpenCost, and its
# DaemonSet fights OpenShift's own node-exporter for hostPort 9100/hostNetwork/
# hostPID, none of which restricted-v2 allows.
nodeExporter:
  enabled: false

# Top-level toggle for kube-state-metrics (condition: kubeStateMetrics.enabled).
# Needed for OpenCost — leave enabled.
kubeStateMetrics:
  enabled: true

# kube-state-metrics is a separately maintained chart with its own values
# schema: securityContext is a map with its own "enabled" key, not a bare
# security context — {} or null breaks its template. Setting enabled: false
# lets OpenShift assign a UID/fsGroup from the namespace's own range.
kube-state-metrics:
  securityContext:
    enabled: false

prometheusOperator:
  # This chart's own securityContext is a plain map (no "enabled" key) —
  # null drops it entirely so OpenShift assigns UID/fsGroup automatically.
  securityContext: null
  # Hardcodes runAsUser: 2000 in its hook jobs, which SCC rejects.
  # Disabling sidesteps it — you lose admission-time schema validation on
  # Prometheus/Alertmanager/ServiceMonitor CRs.
  admissionWebhooks:
    enabled: false
    patch:
      enabled: false
  # The operator still serves TLS for the webhook by default even with the
  # webhook job disabled, and expects a cert secret that job would have
  # created. Turn this off too or the operator pod hangs in ContainerCreating
  # waiting on a secret that will never exist.
  tls:
    enabled: false

prometheus:
  prometheusSpec:
    securityContext: null

# Not required just to feed OpenCost — disabled to keep the footprint small.
alertmanager:
  enabled: false

grafana:
  enabled: false

Why each of these was necessary

Node-exporter and the built-in monitoring collision. kube-prometheus-stack deploys its own node-exporter DaemonSet, requiring hostNetwork, hostPID, and a hostPort binding on 9100. OpenShift's own monitoring stack already runs a node-exporter with exactly the same host-level requirements, and none of them are permitted under restricted-v2 anyway. Since OpenCost's cost allocation doesn't depend on node-exporter metrics (it uses kube-state-metrics and cAdvisor/kubelet data), the clean fix is disabling it outright rather than trying to grant it a privileged SCC.

SCC and fixed UIDs, repeatedly. This was the recurring theme. restricted-v2 — OpenShift's default SCC — refuses to run any pod requesting a specific UID or GID outside the namespace's own auto-assigned range (visible directly in the SCC denial message: must be in the ranges: [1001130000, 1001139999]). The chart hardcodes UID 65534 (nobody) across most of its components, and UID 2000 specifically in the admission-webhook hook jobs. Kubernetes' own securityContext handling matters here too: setting a value to {} merges with the chart's defaults and changes nothing, whereas setting it to null actually drops the key so OpenShift can assign its own UID. The one exception was kube-state-metrics, a separately maintained chart that recently restructured its securityContext value into a map with its own enabled flag — passing null there broke the template outright with a nil-pointer error, so it needed enabled: false instead of the null pattern used everywhere else. The lesson: don't assume every subchart under one umbrella chart uses the same values schema, even when they're bundled together.

The admission webhook. The operator's admission-webhook mechanism runs a short-lived Job to self-sign TLS certs and patch the webhook configuration. That Job's pod spec also hardcodes a UID, hits the same SCC wall, and — because it's a Helm pre-upgrade hook — a stuck hook job causes Helm itself to hang or fail with context deadline exceeded, and its cleanup logic deletes the RBAC objects it created, which briefly makes debugging attempts look like resources are randomly vanishing. Disabling the webhook (admissionWebhooks.enabled: false) avoids the whole problem; the trade-off is losing admission-time schema validation on Prometheus/Alertmanager/ServiceMonitor custom resources, which is an acceptable loss for a single-tenant monitoring install feeding a cost tool. One easily-missed follow-up: disabling the webhook job doesn't disable the operator's own TLS serving for it — prometheusOperator.tls.enabled needs to be turned off separately, or the operator pod hangs indefinitely in ContainerCreating waiting to mount a certificate secret that the (now-disabled) job would have created.

CRD ownership conflicts. OpenShift's cluster-version-operator already owns the Prometheus Operator CRDs cluster-wide as part of the built-in monitoring stack. A fresh Helm install trying to apply its own copies of those CRDs fails with a server-side-apply field-manager conflict. The fix is --skip-crds on the Helm install/upgrade commands — the CRDs already on the cluster are compatible, and the new operator instance just uses them.

RBAC and OwnerReferencesPermissionEnforcement. The final and least obvious issue: Kubernetes has an admission plugin, OwnerReferencesPermissionEnforcement, that OpenShift enables by default (many vanilla clusters don't). It requires explicit RBAC — an update verb on the relevant CRD's /finalizers subresource — before a controller can set blockOwnerDeletion: true on an owner reference. The bundled kube-prometheus-stack ClusterRole doesn't grant this for its own CRDs, so on OpenShift the operator would successfully create the Prometheus custom resource, then fail every subsequent reconciliation loop trying to own its child ConfigMaps, with the deeply unhelpful error cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on. The fix is an additive ClusterRole/ClusterRoleBinding granted directly to the operator's ServiceAccount:

# prometheus-operator-finalizers-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-operator-finalizers
rules:
  - apiGroups: ["monitoring.coreos.com"]
    resources:
      - alertmanagers/finalizers
      - prometheuses/finalizers
      - prometheusagents/finalizers
      - thanosrulers/finalizers
    verbs: ["update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-operator-finalizers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-operator-finalizers
subjects:
  - kind: ServiceAccount
    name: prometheus-kube-prometheus-operator
    namespace: monitoring

Since this RBAC isn't something Helm manages, it survives a helm uninstall and needs cleaning up manually if the stack is ever torn down.

The Install, Start to Finish

With the groundwork explained, here's the actual sequence:

# 1. Add the Helm repo, create the namespace
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
oc new-project monitoring

# 2. Install kube-prometheus-stack, skipping CRDs OpenShift already owns
helm install prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring \
  -f values-openshift-overrides.yaml \
  --skip-crds \
  --timeout 5m

# 3. Grant the finalizers RBAC the chart doesn't ship
kubectl apply -f prometheus-operator-finalizers-rbac.yaml

# 4. Force the operator to retry reconciliation with the new permission
kubectl rollout restart deployment/prometheus-kube-prometheus-operator -n monitoring

# 5. Confirm Prometheus is actually up
kubectl get prometheus -n monitoring
kubectl get pods -n monitoring

# 6. Install OpenCost against it
helm repo add opencost-charts https://opencost.github.io/opencost-helm-chart
helm repo update
helm install opencost opencost-charts/opencost \
  --namespace opencost --create-namespace \
  -f values.yaml

# 7. Confirm it's pulling real data
kubectl port-forward -n opencost svc/opencost 9003 9090
curl -s "http://localhost:9003/allocation/compute?window=1h"

A successful response from that last curl — a 200 with allocation data broken down by cluster, namespace, and service — confirms the entire chain is wired correctly: Prometheus scraping cluster metrics, OpenCost's ServiceMonitor matched and scraped in turn, and the custom pricing model applied against real usage.

Closing Thoughts

None of the individual fixes here are exotic — disable what you don't need, clear hardcoded UIDs, grant the RBAC OpenShift additionally requires. What makes this genuinely time-consuming is that kube-prometheus-stack bundles several independently maintained subcharts, each with slightly different conventions for how their security contexts and enable/disable toggles are structured, and OpenShift's SCC and admission-plugin defaults surface every one of those inconsistencies as a distinct, cryptic failure. Once you've been through it once, though, the values-openshift-overrides.yaml and RBAC manifest above are entirely reusable for the next OpenShift cluster that needs the same stack.