2024-10-11 · 3 min read

Merging a LoRA into the Base Model

Merging a LoRA adapter into the base model creates a single artifact with all weights combined. The tradeoff is simplicity and inference speed against flexibility and hot-swapping.

Why merge?

A LoRA adapter adds computation. During inference, the forward pass must apply the adapter matrices: output = x + (alpha/r) * (BA^T)x. For a 7B model, that's a small but measurable overhead.

Merged adapters skip this step entirely. The weights are baked in: W_merged = W_base + (alpha/r) * B*A^T. One matrix multiply instead of two. Throughput gains are 5–10% depending on batch size and model.

Additional benefits: - Single file to ship and version - Simpler deployment (no peft required at inference) - Easy to export to GGUF, ONNX, or TensorRT - No questions about adapter compatibility

The cost

Once merged, you cannot: - Adjust alpha at runtime - Swap the adapter without reloading - Recover the original base model weights - Mix this adapter with others

If you trained a merged adapter and later want to do A/B testing on alpha values, you must retrain.

How to merge

from peft import PeftModel
from transformers import AutoModelForCausalLM

# Load base + adapter
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(base_model, "path/to/qwen25-7b-support-tone")

# Merge and unload peft
merged_model = model.merge_and_unload()

# Save as standard safetensors
merged_model.save_pretrained("./merged-support-tone-fp16")

The result is a standard transformer model indistinguishable from any other fine-tuned checkpoint. No peft metadata, no special loading code.

File size comparison

The merged adapter is the same size as the base but includes the learned delta.

When to merge

Merge if your workflow is: - Deploy once, don't change - Need absolute minimum inference latency - Will export to GGUF or mobile later - Ship as a self-contained checkpoint

Don't merge if you plan to: - Run A/B tests on alpha values - Combine with other adapters - Quickly iterate and update - Hot-swap during serving

Practical example: sdxl-watercolor-lora

The sdxl-watercolor-lora adapter (223 MB) could be merged into SDXL (6 GB). The result is a 6.2 GB single-file checkpoint:

python merge_adapter.py \
  --base stabilityai/stable-diffusion-xl-base-1.0 \
  --adapter sdxl-watercolor-lora-1.4.0 \
  --output sdxl-watercolor-merged-fp16

This merged checkpoint: - Loads 5% faster than base + adapter - Requires no peft at runtime - Cannot be un-watercolored - Cannot be blended with other styles

Exporting merged adapters

After merging, you can convert to other formats:

# To GGUF (for llama.cpp / Ollama)
python convert_to_gguf.py sdxl-watercolor-merged-fp16 --outfile model.gguf

# To ONNX (for mobile / edge)
python -m transformers.onnx --model-dir sdxl-watercolor-merged-fp16 model_onnx/

Only merged adapters should be exported. LoRA checkpoints cannot be directly converted to GGUF or ONNX; you must merge first.

Versioning merged adapters

If you release a merged adapter, treat it as a distinct artifact: - Slug: sdxl-watercolor-merged-fp16 (different from sdxl-watercolor-lora) - Version: tied to the original adapter version - Metadata: note that it's pre-merged, cannot be disabled

ModelForgeLab can store both: the slim LoRA (100 MB) and the pre-merged version (6.2 GB) as separate downloads. Users choose based on their deployment.

Partial merging

Advanced: merge only certain layers, keep others as adapters. This is rare and usually not worth the complexity. Unless you have a specific reason (extreme memory constraint, mixed precision), stick to full merge or no merge.

The safest policy: keep adapters separate for development, merge only before final production deployment.