Running Local Image Generation on ComfyUI with Nvidia CUDA

James Tate

• 8 min read
Share
Running Local Image Generation on ComfyUI with Nvidia CUDA

I already run a fair amount of local LLM work through Ollama on my homelab box — a Linux Mint machine with an RTX 4070 Ti (12GB VRAM). Text generation is a solved problem at this point: pull a model, run it, done. Image generation turned out to be a different story entirely. This post walks through the full process — the dead end I hit with Ollama, the pivot to ComfyUI, the VRAM math that shaped every model choice, and the handful of sharp edges that cost the most time to figure out.

Starting point: trying Ollama's native image generation

Ollama added experimental text-to-image support in early 2026, backed by two model families: Z-Image Turbo (6B parameters, Alibaba Tongyi Lab) and FLUX.2 Klein (Black Forest Labs, 4B/9B). Support was announced for macOS first, with Linux (CUDA) following.

Running the current version:

ollama --version
# ollama version is 0.34.2

That's well past the version where Linux support was supposed to land, so:

ollama run x/z-image-turbo

Result:

Error: this model requires MLX support, but the MLX runtime is not available

Digging into Ollama's build docs, the MLX engine — the piece that actually powers these image models — isn't bundled in the standard prebuilt Linux binary from install.sh. It's an optional component that has to be compiled from source, and on Linux/Windows that means:

  • CUDA 13+ SDK (the full toolkit, not just the driver)
  • cuDNN 9+
  • A specific cmake --preset "MLX CUDA 13" build step, separate from the normal Ollama build

That's a meaningfully bigger commitment than a standard Ollama install — and it risks conflicting with how Ollama normally manages its own bundled CUDA libraries for everything else. Given the effort-to-payoff ratio, I parked Ollama's native image gen and moved to a more mature, battle-tested path: ComfyUI.

Why ComfyUI over AUTOMATIC1111

AUTOMATIC1111 was the default answer for local Stable Diffusion for years, but active development has stalled and it can't run newer architectures like Flux. ComfyUI is the actively maintained option — it picks up new model architectures first, and it's generally more VRAM-efficient on the same hardware, which matters a lot at 12GB.

Installing ComfyUI

git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI
python3 -m venv venv
source venv/bin/activate

First snag: installing PyTorch.

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

ERROR: Could not find a version that satisfies the requirement torch

PyTorch had already dropped the cu124 index by this point — current builds only ship for cu126, cu128, and cu130 (plus ROCm and CPU-only). Checking the driver's actual CUDA support first is worth doing before picking an index:

nvidia-smi
# CUDA Version: 13.0

With CUDA 13.0 on the driver side, cu128 was the safe pick (PyTorch's CUDA runtime is forward-compatible with newer drivers):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

Sanity check before going further:

python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
# True NVIDIA GeForce RTX 4070 Ti

Then the rest of ComfyUI's dependencies and the launch itself:

pip install -r requirements.txt
python main.py

This opens a local web UI at http://127.0.0.1:8188.

One thing worth knowing from nvidia-smi at idle: a normal desktop session (Xorg, Cinnamon, background apps) already sits on close to 1GB of VRAM before ComfyUI even starts. On a 12GB card, budget for ~11GB usable, not 12.

The default workflow isn't SDXL anymore

I expected the default ComfyUI workflow to be a simple SDXL checkpoint setup. Instead, the current version ships Z-Image Turbo as its built-in "get started" template — a three-file split model rather than a single checkpoint:

File Role Folder
z_image_turbo_bf16.safetensors Diffusion model models/diffusion_models/
qwen_3_4b.safetensors Text encoder (Qwen3) models/text_encoders/
ae.safetensors VAE (Flux 1 VAE) models/vae/

The UI's "Missing Models" panel offered a one-click "Download All" for these — 18.96GB combined, with the diffusion model alone at 11.46GB. On a 12GB card with ~1GB already claimed by the desktop, that leaves effectively no headroom for VAE decode and activation memory. It's OOM territory, or at best forced heavy CPU offload.

The GGUF detour

The fix was the same approach that works for Flux on constrained VRAM: swap the full-precision weights for GGUF-quantized versions of both the diffusion model and the text encoder.

Install the custom node:

cd ~/ComfyUI/custom_nodes
git clone https://github.com/city96/ComfyUI-GGUF

Download quantized replacements:

cd ~/ComfyUI/models/text_encoders
wget https://huggingface.co/Qwen/Qwen3-4B-GGUF/resolve/main/Qwen3-4B-Q5_K_M.gguf

cd ~/ComfyUI/models/diffusion_models
wget https://huggingface.co/jayn7/Z-Image-Turbo-GGUF/resolve/main/z_image_turbo-Q5_K_S.gguf

cd ~/ComfyUI/models/vae
wget https://huggingface.co/Comfy-Org/z_image_turbo/resolve/main/split_files/vae/ae.safetensors

That's roughly 5.2GB + 2.9GB instead of 11.46GB + 7.49GB — a huge reduction, and comfortably inside the VRAM budget.

The missing Python dependency

Restarting ComfyUI after cloning the custom node, the "Load Diffusion Model" dropdown showed no results found — it can only see .safetensors files, not .gguf. That's expected; GGUF needs its own loader node from the custom package. But when searching for it, nothing showed up at all. The startup log had the answer:

File ".../ComfyUI-GGUF/ops.py", line 2, in <module>
    import gguf
ModuleNotFoundError: No module named 'gguf'

The custom node's own dependency hadn't installed. Fix:

cd ~/ComfyUI/custom_nodes/ComfyUI-GGUF
pip install -r requirements.txt

After a restart, the log showed a clean import (0.0 seconds: /home/.../ComfyUI-GGUF, no failure), and the node search turned up the full GGUF node family: Unet Loader (GGUF), Unet Loader (GGUF/Advanced), CLIPLoader (GGUF), DualCLIPLoader (GGUF), TripleCLIPLoader (GGUF), QuadrupleCLIPLoader (GGUF).

Wiring the graph

For Z-Image Turbo, only the single-encoder variant applies — CLIPLoader (GGUF), not Dual/Triple/Quadruple (those are for models like Flux that combine multiple text encoders).

The graph:

  • Unet Loader (GGUF/Advanced) → z_image_turbo-Q5_K_S.gguf → MODEL output feeds into ModelSamplingAuraFlow
  • CLIPLoader (GGUF) → Qwen3-4B-Q5_K_M.gguf
  • One non-obvious detail: the CLIPLoader's type parameter has to be set to lumina2, not qwen. This is a Z-Image-specific quirk in how ComfyUI routes the Qwen text encoder — easy to miss, and the workflow won't behave correctly if left on default.
  • Load VAE → ae.safetensors (note: the default dropdown value is pixel_space, a placeholder — it needs to be explicitly changed)
  • Two CLIP Text Encode (Prompt) nodes feed the KSampler's positive and negative inputs
  • EmptySD3LatentImage sets the output resolution (1024×1024 default)
  • KSampler → VAE Decode → Save Image

KSampler settings for the Turbo variant

Because this is the "Turbo" (fast-inference) model, the sampler settings differ from a standard SDXL/Flux setup:

  • Steps: 8 (Turbo models are distilled for very few steps)
  • CFG: 1.0 (pushing CFG higher tends to break Turbo-tuned models rather than improve them)
  • Sampler: res_multistep, Scheduler: simple

First generation

[INFO] gguf qtypes: F32 (145), Q6_K (37), Q5_K (216)
[WARNING] Dequantizing token_embd.weight to prevent runtime OOM.
[INFO] Requested to load ZImageTEModel_
[INFO] loaded completely; 9229.67 MB usable, 3187.56 MB loaded, full load: True
[INFO] Requested to load Lumina2
[INFO] loaded completely; 6583.11 MB usable, 5033.71 MB loaded, full load: True
100%|██████████| 8/8 [00:11<00:00,  1.46s/it]
[INFO] Prompt executed in 19.33 seconds

19.33 seconds total, with the actual 8-step sampling loop taking only 11 seconds. VRAM usage stayed well within budget — CLIP loaded ~3.2GB, the diffusion model ~5GB — leaving headroom to try a higher-fidelity quant (Q6_K/Q8_0) later if desired. The "Dequantizing token_embd.weight to prevent runtime OOM" warning is expected GGUF-loader behavior, not an error.

Prompting Z-Image Turbo

A couple of things that differ from older SD1.5/SDXL habits:

  • It responds well to natural, descriptive sentences, not comma-separated tag soup
  • It handles bilingual English/Chinese text rendering well if you need readable text baked into an image
  • Keep steps ≈8 and CFG ≈1.0 — this is a distilled/Turbo model, not a general-purpose one

Running it without leaving it running

The instinct is to wrap ComfyUI in a systemd service so it starts on boot:

# /etc/systemd/system/comfyui.service
[Unit]
Description=ComfyUI
After=network.target

[Service]
Type=simple
User=jtate
WorkingDirectory=/home/jtate/ComfyUI
ExecStart=/home/jtate/ComfyUI/venv/bin/python main.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

But this creates a real problem for anyone who also games or uses local LLMs on the same GPU. ComfyUI's memory manager keeps the last-used model resident in VRAM after a generation finishes — it does not free it just because the workflow goes idle. In this setup, that's roughly 8GB parked in VRAM indefinitely (5GB diffusion model + 3GB text encoder), unavailable to anything else, including a game, until something forces it out. There's no built-in "unload after N minutes idle" setting, and --disable-smart-memory doesn't fully solve it either — per ComfyUI's own maintainers, it still offloads to VRAM rather than releasing it.

The practical fix: don't run it permanently. Start it on demand, and stop it — explicitly freeing VRAM — when done.

start-comfyui.sh

#!/bin/bash
COMFY_DIR="$HOME/ComfyUI"
PIDFILE="$COMFY_DIR/comfyui.pid"
LOGFILE="$COMFY_DIR/comfyui.log"

if [ -f "$PIDFILE" ] && kill -0 "$(cat "$PIDFILE")" 2>/dev/null; then
    echo "ComfyUI is already running (PID $(cat "$PIDFILE"))"
    exit 0
fi

cd "$COMFY_DIR" || exit 1
source venv/bin/activate

nohup python main.py --enable-cors-header "*" > "$LOGFILE" 2>&1 &
echo $! > "$PIDFILE"

echo "Starting ComfyUI (PID $(cat "$PIDFILE"))..."

for i in {1..30}; do
    if curl -s -H "Origin: http://127.0.0.1:8188" http://127.0.0.1:8188 > /dev/null; then
        echo "ComfyUI is up: http://127.0.0.1:8188"
        exit 0
    fi
    sleep 1
done

echo "ComfyUI hasn't responded after 30s — check $LOGFILE for errors"
exit 1

stop-comfyui.sh

#!/bin/bash
COMFY_DIR="$HOME/ComfyUI"
PIDFILE="$COMFY_DIR/comfyui.pid"

if [ ! -f "$PIDFILE" ] || ! kill -0 "$(cat "$PIDFILE")" 2>/dev/null; then
    echo "ComfyUI doesn't appear to be running."
    rm -f "$PIDFILE"
    exit 0
fi

PID=$(cat "$PIDFILE")

echo "Freeing VRAM..."
curl -s -X POST http://127.0.0.1:8188/free \
    -H "Content-Type: application/json" \
    -H "Origin: http://127.0.0.1:8188" \
    -d '{"unload_models": true}' > /dev/null

echo "Stopping ComfyUI (PID $PID)..."
kill "$PID"

for i in {1..10}; do
    if ! kill -0 "$PID" 2>/dev/null; then
        echo "ComfyUI stopped, VRAM freed."
        rm -f "$PIDFILE"
        exit 0
    fi
    sleep 1
done

echo "Didn't exit gracefully, forcing kill..."
kill -9 "$PID"
rm -f "$PIDFILE"

chmod +x start-comfyui.sh stop-comfyui.sh

The CORS 403 that broke the start script

First run of the start script's health check produced:

403 Forbidden — access denied

Recent ComfyUI versions ship an origin-checking security middleware (create_origin_only_middleware) that rejects requests where Host/Origin headers don't match its expectations — and curl doesn't send an Origin header by default. There's even a CVE (CVE-2026-6589) documenting flaws in this exact middleware, so this isn't a misconfiguration so much as a rough edge in a fairly new security feature.

The fix is the two changes already baked into the scripts above: launch with --enable-cors-header "*", and have curl send a matching Origin header on every request. Worth noting: --enable-cors-header "*" opens CORS to any origin, which is fine bound to 127.0.0.1 but should not be paired with exposing the port beyond localhost without adding authentication in front of it.

Usage

~/ComfyUI/start-comfyui.sh
# ... generate images via the web UI ...
~/ComfyUI/stop-comfyui.sh

The stop script calls /free before killing the process (belt-and-suspenders — killing the process alone also releases everything), then confirms the process has actually exited before declaring victory.

Takeaways

  • 12GB VRAM is workable for modern local image generation, but only with quantized models — full-precision bf16 weights for current-generation models (Z-Image Turbo, Flux) routinely exceed what a 12GB card can hold alongside desktop overhead.
  • GGUF quantization is the same lever for image models as it is for LLMs, and the tooling (ComfyUI-GGUF) transfers directly.
  • Ollama's native image gen isn't there yet for Linux/CUDA without a from-source MLX build — worth revisiting as it matures, but ComfyUI is the more reliable path today.
  • Don't run ComfyUI as an always-on service on a gaming or local LLM rig. VRAM stays claimed after generation until explicitly freed. Start-on-demand scripts with an explicit /free call on shutdown are the better fit.
  • Recent ComfyUI security hardening (origin-checking middleware) can silently break simple curl-based automation — --enable-cors-header plus a matching Origin header resolves it for local-only use.