2025-01-14 · 3 min read

DoRA: When It Beats Vanilla LoRA

DoRA (Weight-Decomposed Low-Rank Adaptation) refactors how the adapter matrices are composed. Instead of learning low-rank updates directly, DoRA decomposes each weight matrix into magnitude and direction, training only the direction. The result is modest improvements in low-rank regimes.

The core idea

Standard LoRA updates weights as: W_new = W + ΔW, where ΔW = B*A^T and rank(A), rank(B) << dim(W).

DoRA splits the update differently: W_new = m * (V / ||V||), where m is a magnitude vector and V is the learned direction. During training, m stays frozen (initialized from the base weight) and only V is updated.

The practical effect is that DoRA learns "direction of change" rather than "delta magnitude," which can stabilize training when the rank is small.

When DoRA helps

Performance gains are most visible with rank ≤ 8. For a Qwen2.5-7B model on classification tasks, DoRA-8 often outperforms LoRA-8 by 1–3% absolute accuracy.

The improvements shrink at higher ranks. DoRA-32 vs LoRA-32 show negligible difference.

Configuration in peft

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    use_dora=True,  # Enable DoRA
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, config)

The API is identical to standard LoRA; only the internal computation changes.

File size and parameter count

DoRA adds a magnitude vector per target module. For a 7B model with rank 8:

The overhead is ~8%, negligible for most purposes.

Inference and compatibility

At inference, DoRA weights are just LoRA weights. The magnitude and direction are composed into a single update matrix before loading. No special runtime support is needed.

However, not all inference stacks understand DoRA checkpoints. vLLM and TransformersOnnx support it, but llama.cpp and older versions may fail. When in doubt, verify compatibility before distributing a DoRA adapter.

When DoRA doesn't help

For style and image adapters, DoRA shows no clear advantage over LoRA. Training a watercolor style adapter with DoRA-4 vs LoRA-4 produces visually identical outputs. The per-layer magnitude learning in DoRA is most useful when the task requires precise feature-space adjustments, not stylistic shifts.

For large ranks (r ≥ 32), standard LoRA is simpler and equally effective.

A practical example

The mistral-7b-json-extract adapter in the registry benefits from DoRA. The task is structured output under distributional shift, where tight weight control matters:

# DoRA config for extraction task
LoraConfig(
    r=8,
    lora_alpha=16,
    use_dora=True,
    target_modules=["q_proj", "v_proj", "up_proj"],  # More modules for structure
    lora_dropout=0.1,
    task_type="CAUSAL_LM",
)

Training on 500 JSON extraction examples with this config and DoRA typically reaches 81% exact-match accuracy vs 79% with vanilla LoRA.

Debugging with DoRA

If a DoRA adapter trains but shows no improvement, check the loss trajectory. If DoRA loss diverges quickly while LoRA loss is stable, the learning rate may be too high for the magnitude decomposition. Try reducing by half.

If accuracy is identical between DoRA and LoRA, the task may not benefit from magnitude learning. Rank and learning rate matter more than DoRA for such tasks.

When to use DoRA

Start with DoRA if: - Your rank is 4–16 - Your task is structured (classification, extraction, reasoning) - You have time to test both and compare

Use vanilla LoRA if: - Your rank is > 32 - Your task is stylistic or generative - You need maximum inference compatibility

For most cases, LoRA remains the safe default. DoRA is a refinement, not a breakthrough.