Keeping Kasten Happy When VBR Rotates Its Passwords

James Tate

• 5 min read
Share
Keeping Kasten Happy When VBR Rotates Its Passwords

If you protect Kubernetes with Kasten K10 and export to Veeam Backup & Replication (VBR) repositories, there's a gotcha waiting for you the day your VBR password policy kicks in.

The problem

Kasten talks to VBR using a service account. Each VBR repository you add to Kasten becomes a location profile, and each profile references a Kubernetes secret holding the credentials for that account.

Now add a scripted password rotation on the VBR side. The moment the password changes, every Kasten export to VBR starts failing, because the secret still holds the old one. And if you have several VBR repositories on the same VBR server, that means finding and updating each profile's secret individually.

Doing that by hand once is annoying. Doing it on every rotation is not a plan.

How Kasten stores it

A VBR location profile looks like this (trimmed):

kind: Profile
apiVersion: config.kio.kasten.io/v1alpha1
metadata:
  name: vbr
  namespace: kasten-io
spec:
  type: Location
  locationSpec:
    type: VBR
    vbr:
      serverAddress: 192.168.1.153
      serverPort: 9419
      repoName: VBR
      skipSSLVerify: true
    credential:
      secretType: VBRKey
      secret:
        apiVersion: v1
        kind: secret
        name: k10secret-xchdd
        namespace: kasten-io
status:
  validation: Success

The part that matters is spec.locationSpec.credential.secret. That points at the secret to update, and spec.locationSpec.type: VBR tells us which profiles are VBR ones.

To list the profiles and their secrets:

kubectl -n kasten-io get profiles.config.kio.kasten.io -o json |
  jq -r '.items[]
         | select(.spec.locationSpec.type=="VBR")
         | [.metadata.name, .spec.locationSpec.credential.secret.name] | @tsv'

To see the key names inside a secret (names only, not the values):

kubectl -n kasten-io get secret k10secret-xchdd -o json | jq -r '.data | keys[]'

The script

The script does the following:

  1. Finds every VBR location profile in the Kasten namespace.
  2. Resolves the secret each one references and locates the password key.
  3. Shows you exactly what it is about to change.
  4. Asks for the new password (hidden, entered twice).
  5. Asks for confirmation, then patches each secret once, even if several profiles share it.
#!/usr/bin/env bash
# update-kasten-vbr-password.sh
# Finds every Kasten location profile that uses a VBR repository, locates the
# secret holding the VBR account password, and updates it with a new password
# entered at run time.
#
# Usage: ./update-kasten-vbr-password.sh [-n namespace] [-u vbr-username-filter] [-y]
#   -n  Kasten namespace (default: kasten-io)
#   -u  Only update profiles whose secret username matches this value
#   -y  Skip the final confirmation prompt (not recommended)

set -euo pipefail

NS="kasten-io"
USER_FILTER=""
ASSUME_YES="false"

while getopts "n:u:yh" opt; do
  case "$opt" in
    n) NS="$OPTARG" ;;
    u) USER_FILTER="$OPTARG" ;;
    y) ASSUME_YES="true" ;;
    h|*) sed -n '2,11p' "$0"; exit 0 ;;
  esac
done

for bin in kubectl jq base64; do
  command -v "$bin" >/dev/null || { echo "ERROR: $bin is required" >&2; exit 1; }
done

echo "Scanning location profiles in namespace '$NS'..."

# Profiles whose location type is VBR, with the secret they reference
mapfile -t ROWS < <(
  kubectl -n "$NS" get profiles.config.kio.kasten.io -o json |
  jq -r '.items[]
         | select(.spec.type=="Location" and .spec.locationSpec.type=="VBR")
         | [.metadata.name,
            (.spec.locationSpec.credential.secret.name // ""),
            (.spec.locationSpec.credential.secret.namespace // "'"$NS"'")]
         | @tsv'
)

if [[ ${#ROWS[@]} -eq 0 ]]; then
  echo "No VBR location profiles found in '$NS'." >&2
  exit 1
fi

# Build the work list: profile | secret ns | secret name | username | password key
declare -a PROFILES SECRET_NS SECRET_NAME PW_KEY USERNAMES

for row in "${ROWS[@]}"; do
  IFS=$'\t' read -r prof sname sns <<<"$row"
  if [[ -z "$sname" ]]; then
    echo "  SKIP $prof: no credential secret referenced"
    continue
  fi
  if ! sjson=$(kubectl -n "$sns" get secret "$sname" -o json 2>/dev/null); then
    echo "  SKIP $prof: secret $sns/$sname not found"
    continue
  fi

  # Password key: prefer exact 'password', otherwise any key containing 'pass'
  key=$(jq -r '.data | keys[] ' <<<"$sjson" | grep -ix 'password' | head -n1 || true)
  [[ -z "$key" ]] && key=$(jq -r '.data | keys[]' <<<"$sjson" | grep -i 'pass' | head -n1 || true)
  if [[ -z "$key" ]]; then
    echo "  SKIP $prof: no password-like key in $sns/$sname (keys: $(jq -r '.data|keys|join(",")' <<<"$sjson"))"
    continue
  fi

  ukey=$(jq -r '.data | keys[]' <<<"$sjson" | grep -iE '^(username|user)$' | head -n1 || true)
  uname="(unknown)"
  [[ -n "$ukey" ]] && uname=$(jq -r --arg k "$ukey" '.data[$k]' <<<"$sjson" | base64 -d)

  if [[ -n "$USER_FILTER" && "$uname" != "$USER_FILTER" ]]; then
    echo "  SKIP $prof: username '$uname' does not match filter"
    continue
  fi

  PROFILES+=("$prof"); SECRET_NS+=("$sns"); SECRET_NAME+=("$sname")
  PW_KEY+=("$key");    USERNAMES+=("$uname")
done

if [[ ${#PROFILES[@]} -eq 0 ]]; then
  echo "Nothing to update." >&2
  exit 1
fi

echo
echo "The following secrets will be updated:"
printf '  %-35s %-35s %-20s %s\n' "PROFILE" "SECRET" "USERNAME" "KEY"
for i in "${!PROFILES[@]}"; do
  printf '  %-35s %-35s %-20s %s\n' \
    "${PROFILES[$i]}" "${SECRET_NS[$i]}/${SECRET_NAME[$i]}" "${USERNAMES[$i]}" "${PW_KEY[$i]}"
done
echo

# Prompt for the new password (hidden, entered twice)
read -rsp "Enter new VBR password: " NEWPW; echo
read -rsp "Confirm new VBR password: " NEWPW2; echo
if [[ -z "$NEWPW" || "$NEWPW" != "$NEWPW2" ]]; then
  echo "ERROR: passwords empty or do not match." >&2
  exit 1
fi

if [[ "$ASSUME_YES" != "true" ]]; then
  read -rp "Update ${#PROFILES[@]} secret(s) now? [y/N] " ans
  [[ "$ans" =~ ^[Yy]$ ]] || { echo "Aborted, nothing changed."; exit 0; }
fi

ENC=$(printf '%s' "$NEWPW" | base64 -w0 2>/dev/null || printf '%s' "$NEWPW" | base64 | tr -d '\n')
unset NEWPW NEWPW2

# Several profiles can share one secret, so only patch each secret once
declare -A DONE
fail=0
for i in "${!PROFILES[@]}"; do
  id="${SECRET_NS[$i]}/${SECRET_NAME[$i]}/${PW_KEY[$i]}"
  if [[ -n "${DONE[$id]:-}" ]]; then
    echo "  OK   ${PROFILES[$i]} (secret already updated)"
    continue
  fi
  if kubectl -n "${SECRET_NS[$i]}" patch secret "${SECRET_NAME[$i]}" --type merge \
       -p "{\"data\":{\"${PW_KEY[$i]}\":\"$ENC\"}}" >/dev/null; then
    DONE[$id]=1
    echo "  OK   ${PROFILES[$i]} -> ${SECRET_NS[$i]}/${SECRET_NAME[$i]}"
  else
    echo "  FAIL ${PROFILES[$i]} -> ${SECRET_NS[$i]}/${SECRET_NAME[$i]}" >&2
    fail=1
  fi
done
unset ENC

echo
if [[ $fail -eq 0 ]]; then
  echo "Done. Re-run a backup or check the profile status in the K10 dashboard to validate."
else
  echo "Finished with errors, see above." >&2
  exit 1
fi

Using it

chmod +x update-kasten-vbr-password.sh
./update-kasten-vbr-password.sh

A run looks like this:

Scanning location profiles in namespace 'kasten-io'...

The following secrets will be updated:
  PROFILE                             SECRET                              USERNAME             KEY
  vbr                                 kasten-io/k10secret-xchdd           k10-svc              password

Enter new VBR password:
Confirm new VBR password:
Update 1 secret(s) now? [y/N] y
  OK   vbr -> kasten-io/k10secret-xchdd

Done. Re-run a backup or check the profile status in the K10 dashboard to validate.

Useful options:

  • -n <namespace> if Kasten isn't in kasten-io.
  • -u <account> to update only secrets belonging to a particular VBR account, handy if you have more than one service account in play.
  • -y to skip the confirmation prompt. I'd leave it off unless you are driving this from automation.

Checking it worked

The profile's validation status should stay Success:

kubectl -n kasten-io get profiles.config.kio.kasten.io -o custom-columns=NAME:.metadata.name,VALIDATION:.status.validation

Then kick off an export to the VBR repository and confirm it completes.

Caveats

  • The script assumes the profile layout shown above and a password key whose name is password or contains "pass". If your Kasten version stores the credentials differently, it skips that profile and tells you which keys it found rather than guessing.
  • It writes the secret directly, so it needs RBAC permission to get and patch secrets in the Kasten namespace.
  • Try it against a single non-critical repository first.
  • Nothing in it touches VBR itself. If you script the rotation on the VBR side, you could chain the two: generate the new password once, set it in VBR, and feed the same value into this script.