Mirroring Kasten K10 Images into Harbor (and the DNS Rabbit Hole That Followed)

James Tate

16 Jul 2026 • 6 min read

If you run Kasten K10 anywhere without a straight line to the internet — an
air-gapped cluster, a restricted network segment, or just a homelab behind a
proxy you don't fully trust — sooner or later you need every K10 image
sitting in your own registry instead of gcr.io. I do this often enough
across both my homelab and customer environments that it was worth turning
into a proper script instead of re-typing the same podman pull loop every
time. This post covers the script, the Harbor setup around it, and — because
no infrastructure post is complete without one — a DNS problem that took
longer to diagnose than the actual scripting did.

The starting point

Kasten ships a small helper image, k10tools, that already knows how to
enumerate every image a given K10 release depends on:

podman run --rm gcr.io/kasten-images/k10tools:9.0.1 image list

That's the foundation of an air-gapped install: pull that list, retag each
image under your own registry, push, then point Helm at it with
global.airgapped.repository. My first pass at automating this was
functional but rough — a straight bash loop with no confirmation before it
downloaded several GB of images, no real progress feedback, and credentials
either typed at a prompt or left sitting in shell history.

Building something more fluid

The rewrite kept the same core mechanism — k10tools image list to resolve
the image set, then pull/retag/push — but wrapped it in a few things that
matter more the longer a script like this lives:

Credentials from a file, not a prompt or an argument. A .credentials
file with the username on line one and the password (or Harbor CLI secret,
if you're on OIDC/LDAP) on line two, chmod 600'd and gitignored. Nothing
sensitive touches shell history or ps.

Confirmation before anything expensive or destructive. Three checkpoints:
before logging in, before the pull (which can be several GB), and before the
push. Given how easy it is to fat-finger a version number or point at the
wrong registry, I'd rather the script paused and showed me exactly what it
was about to do than assume I got it right.

An actual progress UI. podman pull and docker pull don't hand you a
clean, parseable percentage when their output isn't going to a live TTY —
the fancy progress bars they show interactively disappear the moment you
redirect stdout. Rather than fight that, the script tracks overall job
progress instead: how many of N images are done, with a spinner and elapsed
timer for whichever one is currently in flight.

[####################----------------]  56%  (11/20)  pull  kanister-tools:9.0.1 | 14s

It's not byte-level progress, but it's honest about what it's actually
measuring, and it means you're never staring at a frozen terminal wondering
if the script died.

A retag mapping you can see before it happens. Before anything gets
pushed, the script prints the full source → destination mapping and waits
for a yes:

gcr.io/kasten-images/kanister-tools:9.0.1 -> harbor.apps.openshift2.lab.home/kasten-images/kanister-tools:9.0.1

Setting up the Harbor side

The script pushes into a Harbor project, and Harbor won't create one for
you on the fly — it has to exist first, with the pushing account granted at
least Developer-level access:

Projects → New Project in the Harbor console
Name it to match what the script expects (kasten-images by default)
Private access level, unless you're happy with the images being
world-readable
Default storage quota is generally fine — a full K10 image set is a few
GB, not dozens

One thing I fixed while documenting this: the destination project name
started out purely cosmetic in the script. The actual push path was being
derived by swapping gcr.io for the Harbor hostname, and it only landed
under kasten-images because that string already happened to be part of
the upstream image path. Changing the project variable didn't actually
change where images went — a bug that would only bite the day someone tried
to mirror into a differently-named project. Worth a reminder that "looks
like it works" and "is actually driven by the variable you think it is"
aren't the same thing, especially in scripts you don't touch for months at
a stretch.

Podman vs Docker: the TLS gotcha

I run podman day to day, but plenty of environments still standardise on
Docker, so a second version of the script exists for that. The engines
aren't quite drop-in replacements here: podman lets you skip TLS
verification per-command with --tls-verify=false, but Docker has no
equivalent flag. A self-signed or internal-CA registry has to be added to
the daemon's trust list globally:

sudo tee /etc/docker/daemon.json <<'JSON'
{ "insecure-registries": ["harbor.apps.openshift2.lab.home"] }
JSON
sudo systemctl restart docker

The Docker variant checks for this on startup and warns rather than just
failing opaquely partway through a push — trusting a registry at the
podman/containers-common level doesn't imply Docker trusts it too; they
maintain entirely separate trust stores.

The DNS saga

This is the part that actually ate the afternoon. Two separate machines,
two unrelated DNS misconfigurations, both producing the same symptom: the
script failing at the login step because the registry hostname wouldn't
resolve.

Round one: Tailscale quietly owns your resolver

On my Rocky box, podman login failed with:

dial tcp: lookup ui.harbor.apps.openshift2.lab.home on 100.100.100.100:53: no such host

100.100.100.100 isn't a broken value — it's Tailscale's MagicDNS resolver.
By default, Tailscale takes over system DNS resolution entirely
(--accept-dns=true), and MagicDNS has no idea about internal lab.home
hostnames, so anything not on the tailnet just fails to resolve. Two ways to
handle it, depending on how much you want Tailscale involved:

# Stop Tailscale from overriding DNS at all
sudo tailscale set --accept-dns=false

or, better if you still want MagicDNS for tailnet hosts, add a split-DNS
entry in the Tailscale admin console scoped to the lab.home domain,
pointing at the actual internal resolver (Pi-hole, in my case). That way
*.ts.net still resolves via MagicDNS and everything under lab.home gets
forwarded correctly, without one clobbering the other.

Somewhat separately, this also turned up that the registry hostname itself
was wrong — ui.harbor.apps.openshift2.lab.home routes to Harbor's web
console, which doesn't serve the registry API and returned a 503 once DNS
was no longer the blocker. Dropping the ui. prefix fixed that; a quick way
to tell the two failure modes apart is:

curl -sk -o /dev/null -w "%{http_code}\n" https://<host>/v2/

401 means you've found the registry endpoint (just unauthenticated —
expected). 503 or 404 means you're hitting something else entirely.

Round two: two DNS servers, one bad tiebreak

Different box, same symptom, different cause. On the jumpbox, resolvectl status showed the interface configured with both servers:

Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.8 192.168.1.1

192.168.1.8 is the Pi-hole that actually knows about lab.home.
192.168.1.1 is the router, which doesn't. systemd-resolved wasn't
malfunctioning — it was doing exactly what it's designed to do, which is
trust whichever configured server answers first, even if that answer is a
negative one (NXDOMAIN counts as a valid response, not a reason to try the
next server). Since the router happened to answer first, every internal
hostname lookup died right there regardless of the Pi-hole sitting one
address over, perfectly willing to answer correctly.

Fix, once you've confirmed the Pi-hole itself has the record:

dig @192.168.1.8 harbor.apps.openshift2.lab.home   # confirm it resolves directly
sudo resolvectl dns ens160 192.168.1.8              # temporary — drops the ambiguity

For anything persistent, that goes into netplan:

network:
  ethernets:
    ens160:
      nameservers:
        addresses: [192.168.1.8]

The real long-term fix, though, is upstream at the DHCP source: if the
router is handing out itself as a DNS server alongside the Pi-hole, either
stop it advertising DNS at all, or have it forward unknown queries to the
Pi-hole so every device on the network behaves consistently without needing
a netplan override on each one individually.

Takeaways

Neither DNS issue had anything to do with Harbor, Kasten, podman, or
Docker — they were both pre-existing quirks of home network DNS that had
simply never been exercised against an internal hostname from those
particular machines before. That's the annoying thing about DNS bugs: they
sit dormant until the one day a script needs to resolve something it's
never had to resolve before, and then they look exactly like the thing
you just changed is broken.

The actual scripting turned out to be the easy part. Enumerate images via
k10tools, pull, retag, push, wrap it in confirmations and a progress bar
so it's pleasant to run more than once. The harder part, as usual, was
making sure the network agreed on what a hostname meant before any of that
logic got a chance to run.