2025-05-05 · 3 min read

Running LoRA Adapters in llama.cpp

llama.cpp is a C++ inference engine optimized for CPU. It supports LoRA adapters on quantized (GGUF) base models, allowing CPU inference of adapted models on machines without a GPU.

The workflow

Base model (GGUF) + LoRA (converted to GGML format) = full inference.

The key constraint: both must be for the same original architecture. A LoRA trained on Qwen/Qwen2.5-7B-Instruct works with a GGUF quantized from the same checkpoint.

Converting LoRA to GGML

The LoRA must be converted from safetensors to GGML format:

# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Convert safetensors LoRA to GGML
python convert_lora_to_ggml.py \
  --model-dir /path/to/qwen25-7b-support-tone \
  --output adapter.ggml

The result is adapter.ggml (~30–100 MB depending on rank).

Quantizing the base model

Download or quantize the base model to GGUF:

# Using llama.cpp's quantize tool
./quantize /path/to/Qwen2.5-7B-Instruct-fp16.gguf qwen-q4.gguf Q4_K_M

Quantization levels: Q2 (smallest, ~3.5 bits/param), Q4 (4-bit, ~4.5 GB for 7B), Q5 (5-bit, ~5.5 GB), Q6 (6-bit, ~7 GB).

For CPU inference, Q4 is the sweet spot: good quality, reasonable speed.

Running inference

./server -m qwen-q4.gguf --lora adapter.ggml --lora-scale 0.8

The --lora-scale parameter controls adapter strength (0.0–1.0).

The server opens on http://localhost:8080 with an OpenAI-compatible API:

curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "The system is down. ",
    "max_tokens": 50
  }'

Performance expectations

On a recent MacBook Air (M2, 8-core CPU):

LoRA adds ~15–20% latency due to matrix multiplication overhead. Still reasonable for non-real-time use.

On larger CPUs (16+ cores), throughput can reach 20+ tok/sec even with LoRA.

Real example: phi3-mini-sql-helper

The registry includes phi3-mini-sql-helper with format: gguf. Perfect candidate for llama.cpp:

# Download base + adapter
mfl pull Phi-3-mini-4k-instruct  # hypothetical base checkpoint
mfl pull phi3-mini-sql-helper

# Quantize base if not already
python convert_lora_to_ggml.py \
  --model-dir ./phi3-mini-sql-helper \
  --output phi3-sql.ggml

# Run server
./server -m phi3-mini-4k-q4.gguf --lora phi3-sql.ggml --lora-scale 0.9

Phi-3-mini is only 3.8B parameters; with Q4 quantization it's ~2 GB + 50 MB LoRA. Runs comfortably on a Raspberry Pi 4 (8 GB RAM).

Multi-adapter (advanced)

llama.cpp supports loading multiple adapters, but only one active at a time:

./server -m base.gguf \
  --lora adapter1.ggml \
  --lora adapter2.ggml

Switching adapters requires restarting the inference thread; true hot-swap is not supported.

For production multi-adapter serving, use vLLM or TGI instead.

Limitations

• Architecture lock-in: The LoRA must match the exact base model. GGUF conversion can be fragile.
• No quantization of LoRA: The adapter matrices always run in fp32, adding some memory overhead.
• Limited training support: Some edge cases in GGML LoRA loading (rare, but possible with exotic architectures).

Ollama frontend

Ollama wraps llama.cpp and makes LoRA loading friendlier:

ollama create custom \
  -f Modelfile

# Inside Modelfile:
FROM phi3-mini:4k
ADAPTER ./phi3-sql.ggml

Then:

ollama run custom "Write a SQL query to fetch user data."

Ollama handles the LoRA conversion and model management automatically.

CPU inference strategy

Use llama.cpp for: - Development on consumer hardware (MacBook, Linux laptop) - Edge deployment (Raspberry Pi, embedded systems) - Cost-sensitive inference (no GPU rental) - Privacy (everything runs locally)

The tradeoff: 5–10x slower than GPU inference. Acceptable for chatbots, not for real-time applications.

llama.cpp proves that adapter inference is not GPU-exclusive. A well-quantized base model + LoRA runs surprisingly well on CPU.