2024-08-15 · 4 min read

RAG or a Fine-Tuned Adapter: Choosing the Path

RAG (Retrieval-Augmented Generation) and LoRA adapters solve different problems. Neither is universally better; the choice depends on what your task requires.

Core distinction

RAG: Augment the model's knowledge at query time. Retrieve relevant documents, append to prompt, generate response. Trade-off: extra latency (retrieval hop), lower hallucination, requires fresh documents.

LoRA: Retrain the model on your task. Embedded behavior is learned into weights. Trade-off: no access to new info post-training, no retrieval hop, stable inference latency.

Decision matrix

Requirement	RAG	LoRA	Notes
Freshness (data updates daily)	✓	✗	RAG retrieves latest docs; LoRA is static
Structured output (JSON, SQL)	△	✓	LoRA more reliable for format control
Zero-shot (no training data)	✓	✗	RAG works without data; LoRA requires examples
Offline capability	✗	✓	LoRA works anywhere; RAG needs retriever + docs
Low latency (<100ms)	✗	✓	Retrieval adds overhead; LoRA is direct
Large knowledge base (>10k docs)	✓	✗	LoRA cannot memorize that much

Task-by-task recommendation

Internal FAQ/documentation QA: RAG wins. Documents change, users ask variations of known questions. Retrieve FAQ entries, feed to model.

Brand voice / tone adaptation: LoRA wins. Style is consistent, doesn't change per query. Train qwen25-7b-support-tone once.

SQL generation from natural language: LoRA wins. Output format is strict. phi3-mini-sql-helper learns the schema and query patterns.

Product recommendation system: RAG wins. Catalog changes weekly. Retrieve user profile + similar products, generate explanation.

Image generation style: LoRA wins (for images, fusion of LoRA + base model weights). Style doesn't change; consistency is paramount.

Legal document summarization: Hybrid. RAG retrieves precedents (freshness matters), LoRA ensures summary format matches legal standards.

The hybrid approach

Combine them:

Client query → Retrieve relevant docs (RAG) → Augment prompt → LoRA model → Response

Example: support agent. - RAG retrieves FAQ and ticket history - LoRA model applies supportive tone - Response is empathetic and factually accurate

from transformers import AutoModelForCausalLM
from peft import PeftModel
from elasticsearch import Elasticsearch

es = Elasticsearch()
base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(base_model, "qwen25-7b-support-tone")

def support_response(ticket):
    # RAG: retrieve similar tickets
    results = es.search(index="tickets", body={
        "query": {"match": {"description": ticket["description"]}}
    })
    context = "\n".join([hit["_source"]["resolution"] for hit in results["hits"]["hits"][:3]])

    # LoRA: generate supportive response
    prompt = f"Context:\n{context}\n\nTicket: {ticket['description']}\n\nResponse:"
    inputs = tokenizer(prompt, return_tensors="pt")
    output = model.generate(**inputs, max_length=200)
    return tokenizer.decode(output[0], skip_special_tokens=True)

Cost comparison

Training one qwen25-7b-support-tone LoRA: - Data: 500 examples (cheap to collect) - Compute: 1 GPU-hour on RTX 4090 - Cost: ~$0.50 (self-hosted) or $5 (cloud) - Artifact size: 100 MB - Distribution: easy (single file)

Setting up RAG for support: - Embedding model: required (BAAI/bge-small, ~100 MB) - Vector database: Milvus, Pinecone, or Elasticsearch (setup overhead) - Documents: must be curated and indexed - Compute (per query): ~50–100ms retrieval latency - Cost: initial setup + operational infrastructure

For small use cases, LoRA is cheaper. For large knowledge bases, RAG scales better.

Failure modes

RAG fails when: - Retriever is bad (irrelevant documents augment the prompt with noise) - Document quality is poor (LLM can only hallucinate better) - Latency is critical (retrieval hop kills performance) - Documents are private (cannot send to vector database)

LoRA fails when: - Task is too broad (adapter cannot learn everything) - Distribution shifts after deployment (new patterns emerge, no retraining) - Knowledge must be updated frequently (retraining is expensive) - Zero examples available (cannot bootstrap without data)

Model size matters

For small models (3–7B), LoRA is practical and effective. For 70B+, RAG is often preferred because: - Training a 70B adapter costs more - Retrieval cost becomes proportional (larger model = slower per token) - Bigger models have better in-context learning; RAG feels natural

But a 70B model trained with LoRA can replace several 7B RAG systems, so the calculation is complex.

Practical heuristic

Do you have training data? Yes → LoRA. No → RAG.
Does your data change? Yes → RAG. No → LoRA.
Is latency critical? Yes → LoRA. No → RAG is fine.
Is output format strict? Yes → LoRA. No → RAG is fine.

If 3+ answers are "yes" for a category, lean that way.

ModelForgeLab perspective

The registry hosts adapters (LoRA) because the legend is a self-hosted registry and training service. It implicitly assumes tasks amenable to LoRA: style, tone, format, narrow domain.

A real system might use both: LoRA for style/tone/format, RAG for knowledge/facts. The playground could demonstrate this: image generation uses LoRA (style), but a text playground might use RAG (factuality) or LoRA (format).

Most teams start with one or the other, then add the second when the first hits its limits. RAG + LoRA is the norm at scale.