Hunting the Ghost in the Machine...

Hunting the Ghost in the Machine...

It started, as these things always do, with something small. Brave kept crashing. A tab would freeze, the page would go unresponsive, and eventually the whole browser would fall over. Annoying, but hardly a mystery — browsers crash, you shrug, you restart.

Except it kept happening. Two or three times a day, every day. And when I finally sat down to diagnose it properly, I discovered my Linux Mint desktop wasn't suffering from one fault. It was suffering from six — a bad RAM stick, a bad SATA cable, a counterfeit Bluetooth dongle, a failing USB extension lead, a genuine desktop-environment bug, and a motherboard with opinions about new memory — all overlapping to produce a machine that behaved like it was haunted.

This post is the full write-up: every error, every dead end, the command-line forensics that finally cracked it, and — most usefully — the lessons that apply to any machine behaving strangely. The single biggest one is worth stating up front: memtest86+ passed overnight while my RAM was demonstrably faulty. If you take nothing else away, take that.

The symptom: browsers dying randomly

The kernel journal painted the first picture:

kernel: brave[388534]: segfault at 4008 ip 00005fcf9f2a15cc ... error 4 in brave[...]
kernel: traps: brave[387160] trap invalid opcode ip:5fcfa04fc8e6 ...
kernel: brave[168956]: segfault at 1ebf33333332 ip 00005fd39ed23ce1 ...

Two distinct signatures, and both matter:

Segfaults at scattered addresses. Some near-null (0x8, 0x48, 0x4008 — a struct field read through a garbage pointer), some completely wild (0x1ebf33333332 — which looks suspiciously like floating-point bit patterns being dereferenced as a pointer). That's the smell of type confusion or heap corruption.

"Trap invalid opcode" at identical offsets across days. This one is subtle and worth knowing: Chromium deliberately plants ud2 (an invalid instruction) at the site of every failed internal CHECK(). An invalid-opcode trap at a repeating offset isn't the CPU going mad — it's the software detecting inconsistent internal state and aborting on purpose. The program's own smoke alarm was going off.

coredumpctl list confirmed the pattern: a mix of SIGSEGV and SIGILL, day after day, sometimes six crashes in five minutes (session restore re-loading whatever had killed the browser — a crash loop).

Reading the crash dumps

Brave (like all Chromium browsers) writes Crashpad minidumps to ~/.config/BraveSoftware/Brave-Browser/Crash Reports/completed/. You don't need vendor tools to get value out of a .dmp — the minidump format is documented and even strings yields gold:

strings -a crash.dmp | grep -aE "ptype|gpu-gl-renderer|renderer"

That surfaced ptype = renderer (so these were renderer-process crashes, not GPU or browser process) plus the GPU annotation showing the machine was running ANGLE over Mesa's radeonsi — useful for eliminating a whole class of graphics theories later. Parsing the exception stream properly gave the faulting instruction pointer, the fault address, and a stack walk showing valid return-address chains right up to the moment a garbage pointer was dereferenced. The stacks weren't shredded; the data was.

Eliminating software, one variable at a time

The golden rule of the whole investigation: change one thing, verify it actually changed, observe, then move on. Every suspect got its day in court:

The V8 JIT. Heap corruption in a JS engine is classically either a JIT bug or hardware. I disabled the JIT with --js-flags=--jitless — and immediately learned two things. First, a gotcha worth its own paragraph: the official Brave .deb does not read ~/.config/brave-flags.conf. That file is an Arch-community convention. My flags had been silently ignored for months (including an earlier graphics workaround I thought had been active). Always verify flags actually applied in brave://version → Command Line. The reliable way to add flags on Mint/Ubuntu is a user copy of the desktop entry:

cp /usr/share/applications/brave-browser.desktop ~/.local/share/applications/
sed -i 's|Exec=/usr/bin/brave-browser-stable|Exec=/usr/bin/brave-browser-stable --js-flags=--jitless|g' \
    ~/.local/share/applications/brave-browser.desktop

Second gotcha: --jitless disables WebAssembly entirely, which broke Bitwarden (its crypto runs on WASM). The middle ground is --js-flags=--no-opt, which disables only the optimising compiler tiers and keeps WASM alive. Result of the experiment either way: crashes continued. JIT exonerated.

The kernel. The box was on a very fresh kernel. I pinned the previous series via GRUB (GRUB_DEFAULT="Advanced options for Linux Mint...>...6.17.0-40-generic", update-grub) and ran for days. Crashes continued. Kernel exonerated.

Chromium itself. The freezes hit Brave, Chrome and Helium — but that's one codebase three times. So I ran Firefox as a control... and Firefox crashed too, a content-process SIGSEGV at a near-null address, in a completely unrelated code path. Two independent browser engines — V8 and SpiderMonkey, zero shared code — both corrupting memory the same way. When that happens, the fault lives in what they share: kernel, glibc, the disk serving the binaries, or the hardware itself.

The disk. lsblk confirmed the OS lived on an NVMe drive, and dpkg -V brave-browser firefox verified the on-disk binaries matched their package checksums. Not disk corruption. (Though this check found something else — see the side quests.)

The test that finally caught it

Memtest86+ had run overnight and passed. RAM acquitted, right?

Wrong. Memtest walks relatively gentle patterns through memory with modest concurrency. It's good at finding dead cells. It is notoriously poor at finding marginal memory — sticks that only corrupt bits under real-world pressure: heavy concurrent access, bursty AVX-driven traffic, thermal load. Which is exactly what a browser under YouTube-and-Google-Maps load generates.

The tool that does generate that pressure is Prime95/mprime. It computes enormous FFTs where every result is mathematically verifiable — if the hardware returns a wrong answer, there is no software excuse available. I ran the Blend torture test:

FATAL ERROR: Rounding was 0.499652173, expected less than 0.4
Hardware failure detected running 560K FFT size, consult stress.txt file.
Torture Test completed 0 tests in 0 minutes - 1 errors, 0 warnings.

Three workers, wrong arithmetic, in under a minute. No kernel, no driver, no browser can make a CPU compute a verifiable FFT incorrectly. Hardware, convicted — and mprime's own documentation points the way to isolating which hardware: fail Blend but pass the small in-cache FFTs, and the problem is memory or memory controller.

So I ran the isolation ladder:

Smallest FFTs (in-cache, CPU-only): 19 minutes, all workers clean. The CPU cores compute perfectly when data never leaves cache. Cores acquitted; the fault lives on the CPU↔RAM path.

Per-DIMM testing. One stick at a time in slot A2 (the standard single-stick slot — second from the CPU on almost every four-slot board; the silkscreen labels next to the slots confirm it), XMP disabled, and — critically — speed verified before every single run:

sudo dmidecode -t memory | grep "Configured Memory Speed"

This verification step earned its keep twice. Once when it revealed XMP had silently re-enabled itself after an earlier BIOS session — meaning a previous "XMP off made no difference" data point was worthless because it had never actually been off. And once because it later proved the failing stick was failing at stock JEDEC 2133 and 1.2V, removing any overclocking excuse.

Results: Stick 1, alone, at verified JEDEC speed — failed Blend inside the first minute. Stick 2, same slot, same settings — 30 minutes clean. One defective DIMM, isolated beyond argument.

A note on interpreting these runs: a fail is conclusive instantly; a pass needs time. The moment hardware returns one wrong answer, you can stop — the conviction is in. Proving the absence of errors is what needs the full 30–60 minutes.

And here's the satisfying part: one bad stick retroactively explained the machine's entire rap sheet. The browser crashes (V8's heap is the biggest, most pointer-dense target on the system — it loses first). The invalid-opcode CHECKs (V8's sanity checks catching corrupted state). A months-old mystery where sudo had segfaulted repeatedly at a fixed offset and a reboot "fixed" it — a flipped bit in a cached executable page, cured by re-reading clean pages from disk. Filesystem corruption I'd blamed on an ageing drive. Random hard freezes. One root cause, many costumes.

Corsair honoured the lifetime warranty (they take the whole kit back, so plan for interim RAM). The replacement kit went through what is now house policy — an acceptance test before trust: 20 minutes of Blend at the full 3200 XMP profile, the exact configuration that exposed the original fault in seconds. Clean. Case closed.

One interim-RAM footnote: the borrowed stopgap sticks initially wouldn't POST at all. Before declaring incompatibility, know that boards store memory-training parameters for the old DIMMs and can take a disturbingly long first boot (or a CMOS clear plus two patient minutes) to retrain for new ones. Mine eventually sulked its way into booting and then passed Blend happily.

Side quest #1: the "failing" hard drive that wasn't

While hunting the crashes, the kernel log kept showing this for the data drive:

ata5: SError: { UnrecovData CommWake 10B8B BadCRC }
ata5.00: failed command: READ FPDMA QUEUED
ata5: limiting SATA link speed to 3.0 Gbps
I/O error, dev sda, sector 260049424

I'd been mentally writing this drive's obituary for months — it had produced genuine ext4 directory corruption. But look at the SMART data:

  5 Reallocated_Sector_Ct   ... 0
197 Current_Pending_Sector  ... 0
198 Offline_Uncorrectable   ... 0
199 UDMA_CRC_Error_Count    ... 29

Zero reallocated sectors, zero pending sectors — the platters are healthy. But 29 CRC errors plus BadCRC link faults and an automatic link-speed downgrade is the textbook signature of a bad SATA cable or connector: data corrupted in transit on the wire, not on the disk. The distinction matters enormously — one is a £3 cable, the other is a drive replacement and a restore from backup. Attributes 5/197/198 indict the drive; attribute 199 indicts the cable. Learn to read the difference before you buy a disk you don't need.

(Related hard-won advice: never run filesystem repairs on a machine with suspect RAM. fsck making decisions based on corrupted memory can make the damage worse. Fix the RAM first, then repair the filesystem.)

Side quest #2: the desktop that crashed when the monitor turned off

Every evening, pressing the monitor's power button would crash Cinnamon. coredumpctl had five SIGABRTs on record, and coredumpctl info <PID> handed over a beautiful backtrace:

#4  g_assertion_message_expr (libglib-2.0.so.0)
#5  clutter_actor_real_unrealize (libmuffin-clutter-0.so.0)
...
#15 clutter_actor_destroy_all_children (libmuffin-clutter-0.so.0)

paired with this in ~/.xsession-errors:

JS ERROR: TypeError: m is undefined
  update@/usr/share/cinnamon/js/ui/hotCorner.js:359
  _updateBoxes@/usr/share/cinnamon/js/ui/layout.js:360
  _monitorsChanged@/usr/share/cinnamon/js/ui/layout.js:368

The mechanism: a DisplayPort monitor doesn't just blank when powered off — it electrically disconnects. The desktop sees a hot-unplug, reflows onto zero monitors, Cinnamon's hot-corner code indexes into a now-empty monitor list (m is undefined), the reconfiguration is left half-done, and a Clutter assertion aborts the process one second later. Deterministic, reproducible, and entirely in stock desktop code — HDMI users rarely see it because HDMI monitors usually stay "connected" when off, which is why bugs like this survive for years.

The workaround: never let the monitor disconnect. Software-blank the display instead of using the power button. I bound a keyboard shortcut to:

sh -c 'cinnamon-screensaver-command --lock; sleep 0.6; xset dpms force off'

Lock first, then blank (so waking lands on the password prompt); the sleep stops your key-release from instantly re-waking the screen. The monitor sits in DPMS standby drawing about a watt, the DP link stays alive, _monitorsChanged never fires, and the crash simply cannot occur. Validated with multi-hour blanks: zero new coredumps. Also worth checking in the monitor's OSD: LG's "Automatic Standby" and similar deep-sleep timers can drop the link on their own after a few hours — turn them off.

One more thing Cinnamon does after crashes: it disables your extensions as self-protection. If your desktop looks oddly plain after a crash, check System Settings → Extensions before assuming anything was uninstalled — re-enabling is instant, no logout required.

Side quest #3: the Bluetooth mouse — a saga in four cables

The mouse story deserves its own confession, because I got the diagnosis wrong twice and the tooling is genuinely useful.

Chapter 1: the wedging dongle. The mouse (an MX Anywhere 2 over Bluetooth) would die whenever the monitor slept, needing systemctl restart bluetooth to recover. The adapter turned out to be a 0a12:0001 "Cambridge Silicon Radio" USB dongle — probably the most counterfeited USB ID in existence, and the clones' firmware is infamous for wedging on power events. btusb enable_autosuspend=0 in modprobe.d helped but didn't cure it. Replaced with a TP-Link UB5A (Realtek RTL8761B — genuine, mainline-supported, firmware ships in linux-firmware). The wedging stopped.

Chapter 2: the one-second pauses. But now the mouse would freeze for a second at random, mid-use. Time to actually measure the radio. btmon is the packet-level microscope for BlueZ, and it produced the first hard evidence:

> HCI Event: Disconnect Complete
        Reason: LMP Response Timeout / LL Response Timeout (0x22)
> LE Extended Advertising Report
        RSSI: -66 dBm

Two crucial numbers there. Reason 0x22 means the peer stopped answering at the radio level — the link starved and the supervision timeout killed it. (Contrast with reason 0x13, "Remote User Terminated" — that's a BLE mouse deliberately disconnecting to save battery during idle, completely benign, and if you don't know the difference you'll chase ghosts in your kernel log forever.) And RSSI -66 dBm for a mouse an arm's length away is poor — a healthy short-range link runs -40 to -55. The nano dongle was buried in the rear I/O behind a steel case, and the link was living right on the failure threshold.

So began an afternoon of quantified antenna engineering with a USB extension lead and a live meter: sudo btmon | grep RSSI in one terminal, bluetoothctl disconnect <MAC> in another to force a fresh advertising report per reposition. The numbers told the whole story: behind the case, -66. On a 15cm lead still in the case's RF shadow, -64 (position matters more than distance). Taped behind the monitor, -54 — ten dB better, but 10 dB short of what free-space maths predicted, because the panel's metal frame was detuning the antenna. Hanging 3cm clear below the bezel with line of sight: -46 dBm. That last 3cm of clearance bought 8 dB on its own. Total improvement: 20 dB — one hundred times the received power. The inverse-square law showed up on cue at every step; radio is physics, and you can measure your way through it.

Chapter 3: the plot twist. The freezes continued at -46. Signal was no longer a viable explanation, so I ran correlation logging — Bluetooth events in one terminal, USB events in the other:

sudo btmon | grep -iE "Disconnect Complete" -A3 --line-buffered | tee ~/bt-drops.log
sudo dmesg -wT | grep -iE "usb|hci0|btusb"    | tee ~/usb-events.log

And the USB log confessed:

usb 1-9: device descriptor read/64, error -71
usb 1-9: device not accepting address, unable to enumerate
Bluetooth: hci0: RTL: download fw command failed (-110)
btusb: Resetting usb device

Error -71 is EPROTO — protocol-level corruption on the USB wire itself. The dongle wasn't losing the mouse; the PC was losing the dongle. The extension lead's data lines were electrically failing, the kernel was re-flashing the dongle's firmware after each dropout, and the radio disconnects were downstream symptoms of USB stalls. Every 0x22 in the Bluetooth log had a USB reset standing behind it. The radio had been framed.

Chapter 4: the retreat. A known-good shielded lead still produced dropouts — leaving the motherboard port or the dongle's own plug as the last suspects. And this is where engineering judgment beats stubbornness: after RAM forensics, RSSI surveying and packet capture, chasing a flaky USB port for a £25 problem is a poor trade. I bought a tri-mode mouse (Bluetooth, its own 2.4GHz receiver, and true wired USB-C data — rarer than you'd think: most "USB-C" mice, the entire MX Master line included, wire the port for charging only, no data lines at all). Its dedicated 2.4GHz receiver, plugged into a different port, has been flawless — which itself is a final diagnostic result: same desk, same band, no BlueZ stack, no problem. A strategic retreat, fully mapped, is a legitimate outcome.

Why these faults happen

None of this was exotic. Marginal RAM develops with age and heat — XMP profiles that were stable for years drift marginal as DIMMs and memory controllers wear, and a stick can degrade to where it fails even at stock speed while still sailing through pattern testers. Cables corrode and fatigue — SATA and USB both carry high-speed differential signals with little tolerance for oxidised contacts or broken strands, and both fail marginally: working most of the time, corrupting under load, which is far harder to spot than dead. Two independent bad cables on one desk sounds unlucky until you count how many cables a desk accumulates and how few ever get questioned. Counterfeit silicon is endemic in cheap USB dongles, and its firmware fails in ways no host-side setting can fix. And rare desktop bugs survive in exactly the code paths most users never hit — like what happens when the monitor count reaches zero.

The toolkit, and the lessons

The commands that did the real work, for your bookmarks: journalctl and coredumpctl for what crashed and the backtraces of why; dmidecode -t memory for what your RAM is actually doing right now (never assume — this saga contains two occasions where the BIOS wasn't doing what I believed); smartctl -a with an educated eye on attributes 5/197/198 versus 199 to tell a dying drive from a dying cable; dpkg -V to verify installed binaries against package checksums; mprime -t as the real memory test, with Blend-vs-small-FFT as the built-in fault isolator; btmon for Bluetooth packet truth, including disconnect reason codes and live RSSI; and dmesg -wT running in a spare terminal whenever anything USB is suspect.

And the principles, which outlast any particular fault. Change one variable at a time, and verify the change took effect before trusting the result — my two worthless data points (ignored browser flags, self-re-enabling XMP) both came from skipping the verification step. A passing test only means the test passed — memtest's clean bill of health cost me a week of chasing software. Match the test to the failure mode: pattern testers find dead cells, computational load finds marginal ones. Failures are instant convictions; passes need soak time. Correlate across layers — the mouse mystery cracked only when radio logs and USB logs ran side by side with timestamps; single-layer logging had me confidently wrong twice. Read what the error actually says: 0x22 vs 0x13, CRC errors vs pending sectors, EPROTO vs a timeout — the specifics are the diagnosis. Suspect the copper: cables are the least glamorous and most-ignored components in any system, and they were guilty twice here. And finally: acceptance-test everything that enters the machine. New RAM, replacement RAM, borrowed RAM — thirty minutes of Blend before trust. It's the cheapest insurance in computing.

Epilogue

The machine now runs its replacement RAM at full XMP speed, acceptance-tested. The browsers haven't crashed since the bad stick left the building. The desktop survives every screen-off. The mouse glides on its own receiver. The kernel journal — for months a nightly horror anthology — is boring.

Six faults, six root causes, zero guesses. The ghost was never one ghost; it was a committee. But every member of a committee leaves paperwork — in the journal, in the dumps, in the SMART counters, in the packet captures. You just have to go and read it.