I run Ghost for this blog on an RKE2 cluster in the homelab, sitting on top of MariaDB via the bitnami Helm chart. It's been solid for a long time — until I tried to upgrade Ghost itself and the app came up broken. No obvious error, just a site that wouldn't render properly post-upgrade.

The root cause turned out to be simple to state and considerably less simple to fix: Ghost dropped MariaDB support a while back. Versions past a certain point require MySQL 8 specifically — Ghost's migration tooling (knex/knex-migrator) relies on JSON column behavior and SQL semantics that MariaDB implements differently under the hood, so upgrades either fail outright or leave the app in a half-migrated state. That's exactly what I was seeing.

This post walks through the diagnosis, the migration from MariaDB to a MySQL 8 StatefulSet, and — the part that ate most of the afternoon — a collation bug in the upgrade path that took four attempts to actually fix, because the two most "official" looking fixes both turned out not to apply to my situation.

The setup

• Ghost deployed as a plain Kubernetes Deployment (not Helm) in the ghost namespace on RKE2
• MariaDB via the bitnami chart, credentials in a ghost-mariadb Secret with the standard bitnami key names (mariadb-root-password, mariadb-password)
• Ghost's DB connection configured via plain env vars (database__connection__host, __user, __password, __database) — Ghost's own config convention, unrelated to which DB engine is behind it
• Kasten K10 available for application-consistent backups before touching anything

Step 1: stand up MySQL 8 alongside the existing MariaDB

Rather than replacing MariaDB in place, I added a MySQL 8 StatefulSet in the same namespace — headless Service, Secret for credentials, PVC-backed pod — and left the existing Ghost Deployment and MariaDB pod untouched until the new DB was ready and populated.

args:
  - "--default-authentication-plugin=mysql_native_password"
  - "--character-set-server=utf8mb4"
  - "--collation-server=utf8mb4_unicode_ci"

The mysql_native_password flag matters for older Ghost/mysql2 driver combinations — without it you can hit an unsupported-auth-mode error on first connect.

Step 2: dump and restore

Standard mysqldump / restore, with the app quiesced first for a consistent snapshot:

kubectl -n ghost scale deployment ghost --replicas=0
kubectl -n ghost exec <mariadb-pod> -- \
  mysqldump --single-transaction --no-tablespaces --routines --triggers \
  -uroot -p"$MARIADB_ROOT_PW" ghost > ghost-backup.sql

First snag: the restore into MySQL 8 failed immediately with Unknown collation: 'utf8mb4_uca1400_ai_ci'. This is a MariaDB-only collation (added in MariaDB 10.10+ for Unicode 14 support) that MySQL has never heard of. One sed pass fixed it:

sed -i 's/utf8mb4_uca1400_ai_ci/utf8mb4_unicode_ci/g' ghost-backup.sql

With that cleaned up, the restore went through cleanly, and repointing Ghost's database__connection__host env var at the new MySQL Service brought the site back up on the current Ghost version, unchanged — the migration itself hadn't broken anything yet. Good checkpoint before touching the version.

Step 3: attempt the actual Ghost upgrade

This is where it got interesting. Bumping the Ghost image to 6.61.0 triggered its migration run, which failed on:

ER_FK_INCOMPATIBLE_COLUMNS
alter table `gifts` add constraint `gifts_buyer_member_id_foreign`
foreign key (`buyer_member_id`) references `members` (`id`)
on delete SET NULL - Referencing column 'buyer_member_id' and
referenced column 'id' ... are incompatible.

Ghost creates the gifts table as part of a routine schema migration and tries to add a foreign key back to members.id. MySQL was refusing the FK, claiming the columns were incompatible — despite both being varchar(24).

Attempt 1: collation mismatch (partially right, wrong target)

Checking members.id's actual collation turned up utf8mb4_general_ci — the original MariaDB collation, carried through the dump — while the database's default collation was utf8mb4_unicode_ci (set by the StatefulSet's collation-server flag). A mismatch, definitely, so I aligned the database default to match the existing data:

ALTER DATABASE ghost CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

Same error, unchanged. Setting the database default doesn't touch the collation already stored on 400+ existing columns, and as it turned out, wasn't actually the mechanism causing the failure anyway.

Attempt 2: convert the whole schema

Next, a full-schema conversion — every one of 78 tables — from general_ci to unicode_ci, generated dynamically from information_schema and applied with FK checks disabled:

mysql ... -N -e "SELECT CONCAT('ALTER TABLE \`', TABLE_NAME,
  '\` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;')
  FROM information_schema.TABLES WHERE TABLE_SCHEMA='ghost';" > convert.sql

Verified zero columns left on the old collation. Retried the upgrade. Identical error, byte-for-byte. At this point collation looked like a dead end.

Attempt 3: column type, ruled out on inspection

members.id turned out to be varchar(24) rather than what I assumed was Ghost's schema definition of char(24). Tempting fix, but MySQL's actual FK-compatibility rules treat CHAR and VARCHAR as interchangeable for foreign keys as long as length, charset, and collation match — so this was never going to be the real cause, and converting to CHAR would likely have just introduced a new mismatch against tables Ghost's own migrations create as VARCHAR. Ruled out without touching anything, on inspection rather than a failed test.

Attempt 4: the documented fix (also didn't work)

A search turned up several GitHub issues against Ghost describing this exact error on this exact migration file, with a documented fix: MySQL 8 assigns new tables a different collation depending on the connection's session collation, not the schema default, so the fix is to force it explicitly via init_connect:

args:
  - "--character-set-client-handshake=OFF"
  - "--init-connect=SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci"

(character-set-client-handshake=OFF matters — without it, a client's own charset preference during connection handshake silently overrides init_connect.)

Confirmed the ghost DB user had no SUPER privilege (which would have bypassed init_connect entirely), confirmed SELECT @@collation_connection reported the correct value for that user's session — and the upgrade still failed with the identical error. The documented, officially-referenced fix simply didn't apply to this case.

Finding it for real: reproduce, don't theorize

After three misses, I stopped guessing and reproduced the exact conditions Ghost's migration uses:

CREATE TABLE test_collation_check (
  id VARCHAR(24) CHARACTER SET utf8mb4 NOT NULL
);
SHOW FULL COLUMNS FROM test_collation_check;

The column came back utf8mb4_0900_ai_ci — despite collation-server, the database default, and init_connect all being set to utf8mb4_unicode_ci. The mechanism: when a CREATE TABLE statement specifies CHARACTER SET explicitly but omits COLLATE, MySQL 8 falls back to that charset's hardcoded default collation, ignoring server, connection, and schema defaults entirely. Ghost's migration defines its columns exactly this way, which is why nothing at the server or connection level ever had a chance of working.

The actual fix

Convert the whole schema to the collation MySQL was always going to assign, rather than fighting it:

mysql ... -N -e "SELECT CONCAT('ALTER TABLE \`', TABLE_NAME,
  '\` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;')
  FROM information_schema.TABLES WHERE TABLE_SCHEMA='ghost';" > convert_0900.sql

Applied the same way, FK checks off during the run, verified zero columns remained on the old collation, retried the upgrade — and it went straight through. Ghost 6.61.0, Database: mysql8, site back up.

Takeaways

• Dumping MariaDB into MySQL 8 isn't a drop-in operation. Collation defaults differ, and MariaDB has collations MySQL doesn't recognize at all.
• "Compatible-looking" foreign keys can still fail on collation alone — same type, same length, wrong collation is enough to trip ER_FK_INCOMPATIBLE_COLUMNS.
• The documented fix for a known error isn't guaranteed to be the fix for your instance. GitHub issues describing the identical error and even the identical migration file led me to a plausible, widely-referenced solution that didn't hold up under an actual reproduction.
• When a theory survives two failed fixes, stop theorizing and reproduce the exact conditions. A single throwaway CREATE TABLE statement told me more in ten seconds than three rounds of documentation-driven guessing.
• Keep the old database around. MariaDB stayed live and untouched through all of this, and Ghost's own migration tooling wrote a fresh JSON backup to disk before every failed attempt — cheap insurance that made it safe to keep iterating.