Docs / Advanced & Developer Hub / GPU Hardware Acceleration (CUDA, DirectML, CoreML)

GPU Hardware Acceleration (CUDA, DirectML, CoreML)

Complete guide for configuring NVIDIA CUDA + cuDNN on Linux, DirectML on Windows, and CoreML on Apple Silicon.

Preview Content

Please note that the details on this page are pre-content drafts. More detailed technical specifications, architectural breakdowns, and updated guides will be added soon.

1. Windows: DirectML Acceleration (Zero-Setup)

On Windows, XianScan utilizes DirectML (ort/directml) by default. DirectML accelerates ONNX neural models across:

NVIDIA GeForce / RTX GPUs (GTX 1060+, RTX 20/30/40/50 series)
AMD Radeon RX GPUs (RX 5000/6000/7000 series)
Intel Arc / Iris Xe GPUs and dedicated NPUs

XianScan uses Win32 DXGI adapter enumeration to detect all installed GPUs and automatically select your dedicated graphics card with dedicated VRAM. DirectML requires zero CUDA toolkit or SDK installations.

2. Linux: NVIDIA CUDA 12 & cuDNN 9 Setup

For dedicated Linux machines and cloud servers (Ubuntu 22.04/24.04/26.04, Debian 12+, AWS EC2 G4dn/G5/G6, Hetzner GPU):

Step 1: Install NVIDIA Driver & CUDA Toolkit

bash

# 1. Disable conflicting nouveau driver
echo "blacklist nouveau" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
echo "options nouveau modeset=0" | sudo tee -a /etc/modprobe.d/blacklist-nouveau.conf
sudo update-initramfs -u

# 2. Install kernel headers and driver
sudo apt-get update -y
sudo apt-get install -y linux-headers-$(uname -r) nvidia-driver-550-server nvidia-cuda-toolkit
sudo reboot

Step 2: Install NVIDIA cuDNN 9 Libraries

ONNX Runtime dynamically links against libcudnn.so for Koharu RF-DETR and RapidOCR convolution kernels:

bash

# Install cuDNN 9 via official NVIDIA wheel
sudo apt-get install -y python3-pip
pip install --break-system-packages nvidia-cudnn-cu12

# Symlink cuDNN shared objects to system library path
sudo ln -sf ~/.local/lib/python3.*/site-packages/nvidia/cudnn/lib/libcudnn*.so* /usr/lib/x86_64-linux-gnu/
sudo ldconfig

Step 3: Launch with CUDA Library Path

bash

export LD_LIBRARY_PATH="$HOME/xianscan-app:/usr/lib/x86_64-linux-gnu:/usr/local/cuda/lib64:$LD_LIBRARY_PATH"
./xianscan

3. macOS: Apple Silicon (CoreML / Metal)

On macOS with Apple Silicon (M1, M2, M3, M4), XianScan routes neural tensor operations through Apple's CoreML Execution Provider, offloading compute to the 16-core Apple Neural Engine (ANE) and Metal graphics with unified memory access.

4. Systemd Daemon with GPU Environment

For 24/7 background operation on Linux servers, configure a systemd service with LD_LIBRARY_PATH and ORT_CUDA_MEM_LIMIT_MB:

ini

[Unit]
Description=XianScan Translation Server
After=network.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/home/ubuntu/xianscan
ExecStart=/home/ubuntu/xianscan/xianscan
Restart=always
RestartSec=5
Environment=PORT=8124
Environment=LD_LIBRARY_PATH=/home/ubuntu/xianscan:/usr/lib/x86_64-linux-gnu:/usr/local/cuda/lib64
Environment=ORT_CUDA_MEM_LIMIT_MB=8192
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

5. Dynamic Device Switching & Telemetry

Switch hardware execution providers or set VRAM limits dynamically at runtime via REST API or Web Settings:

bash

# Query active hardware telemetry and GPU info
curl http://localhost:8124/api/system/hardware

# Switch active provider to CUDA with 8GB VRAM cap via Web API (Port 8124)
curl -X POST http://localhost:8124/api/system/hardware \
  -H "Content-Type: application/json" \
  -d '{"device": "cuda", "vram_limit_mb": 8192}'

# Or switch directly on the Axum ML sidecar (Port 8123)
curl -X POST http://localhost:8123/system/device \
  -H "Content-Type: application/json" \
  -d '{"provider": "cuda", "vram_limit_mb": 8192}'

Previous: Japanese Manga (RTL) Next: ML Pipeline & Inpainting Engine