2025-02-24 · 4 min read

Using Synthetic Data to Bootstrap a LoRA

When you have 20 examples and need 500, synthetic data can bridge the gap. The risk is distribution shift: an adapter trained on synthetic data may generalize poorly to real requests.

The bootstrap workflow

1. Start with N real seed examples (20–50)
2. Use a larger or instruction-tuned model to generate variations
3. Filter synthetically-generated examples for diversity
4. Mix synthetic + real (typical: 70% synthetic, 30% real)
5. Train the adapter
6. Evaluate heavily on real data

Practical generation

from transformers import pipeline
import json
import random

# Seed examples (real data)
seeds = [
    "The system is down and users cannot log in.",
    "Performance is degraded in the evening.",
    "Dashboard shows incorrect numbers.",
]

# Generator (instruction-tuned base model)
generator = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct")

# Generate variations
synthetics = []
for seed in seeds:
    for _ in range(10):  # 10 variations per seed
        prompt = f"Generate a similar but different support ticket: '{seed}'"
        output = generator(prompt, max_length=100, do_sample=True)[0]["generated_text"]
        synthetics.append({"input": output, "is_synthetic": True})

# Deduplicate by text similarity (basic: exact match)
seen = set()
deduped = []
for record in synthetics:
    if record["input"] not in seen:
        seen.add(record["input"])
        deduped.append(record)

print(f"Synthetic examples: {len(seeds)} seeds → {len(deduped)} unique variations")

Filtering synthetic data

Not all generations are good. A simple ranking helps:

from transformers import AutoModel, AutoTokenizer
import torch

# Load embedding model for similarity
embedding_model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")

def embed(texts):
    """Get sentence embeddings."""
    inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
    with torch.no_grad():
        outputs = embedding_model(**inputs)
    return outputs.last_hidden_state.mean(dim=1)

# Seeds as reference
seed_embeds = embed(seeds)

# Score synthetic examples by diversity from seeds
synthetic_texts = [r["input"] for r in deduped]
synthetic_embeds = embed(synthetic_texts)

diversity_scores = []
for syn_embed in synthetic_embeds:
    # Distance to nearest seed (higher = more novel)
    distances = torch.norm(syn_embed - seed_embeds, dim=1)
    min_distance = distances.min()
    diversity_scores.append(min_distance.item())

# Keep top 50% by diversity
cutoff = sorted(diversity_scores)[len(diversity_scores) // 2]
filtered = [r for r, score in zip(deduped, diversity_scores) if score > cutoff]
print(f"After diversity filter: {len(deduped)} → {len(filtered)}")

Blending real and synthetic

# Mix 30% real + 70% synthetic
real_data = [json.loads(line) for line in open("real_data.jsonl")]
dataset = random.sample(real_data, int(0.3 * 500)) + random.sample(filtered, int(0.7 * 500))
random.shuffle(dataset)

# Write mixed dataset
with open("mixed_dataset.jsonl", "w") as f:
    for record in dataset:
        f.write(json.dumps(record) + "\n")

print(f"Mixed dataset: {len([r for r in dataset if r.get('is_synthetic')])} synthetic, {len([r for r in dataset if not r.get('is_synthetic')])} real")

License concerns

Synthetic data generated from proprietary models may inherit restrictions. If you use GPT-4 or Claude to generate training data, the terms of service typically prohibit using the output to train competing models.

For public models (Mistral, Llama, Qwen), generation is usually permitted as long as you respect the base model's license.

Practical rule: Use small, openly-licensed models (Mistral 7B, Qwen2.5-7B) for generation. Avoid OpenAI/Anthropic proprietary models unless you've read the fine print.

Common failure modes

Overfitting to synthetic style: An adapter trained 90% on synthetic data learns the generator's phrasing, not the task. Real requests feel off.

Test for style leakage: If the output always includes phrases or structures not in your seeds, the synthetic data is teaching spurious patterns.

Distribution mismatch: If all synthetic examples are very short but real ones are long, the adapter won't handle real requests well.

Citation debt: You can't cite generated examples as training data in a model card. Mark them as synthetic; be transparent.

Evaluation

Test the synthetic-boosted adapter on real data exclusively:

# Real held-out set (never touch this during training or synthetic generation)
real_eval = [json.loads(line) for line in open("eval_real.jsonl")]

# Load trained adapter
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = PeftModel.from_pretrained(model, "runs/synthetic-adapter")

# Evaluate
correct = 0
for record in real_eval:
    inputs = tokenizer(record["prompt"], return_tensors="pt")
    output = model.generate(**inputs, max_length=128)
    prediction = tokenizer.decode(output[0], skip_special_tokens=True)
    # Compare against ground truth
    if prediction == record["expected"]:
        correct += 1

accuracy = correct / len(real_eval)
print(f"Accuracy on real eval: {100*accuracy:.1f}%")

If accuracy is significantly lower than your training loss suggested, synthetic data is at fault. Add more real examples or retrain without synthetic boosting.

When synthetic data makes sense

Use synthetic data for: - Rapid prototyping (validate task feasibility before expensive data collection) - Edge cases (generate rare examples) - Augmentation (supplement scarce real data)

Do not use synthetic data for: - High-stakes tasks (legal, medical, financial) - When real data is abundant - If you cannot evaluate thoroughly on held-out real data

The best synthetic-boosted adapters are those where real and synthetic distributions nearly match. That's rare; most of the time, a smaller real-only adapter beats a large synthetic-heavy one.