2025-01-29 · 3 min read

Preparing a Dataset for Adapter Training

A good dataset determines adapter quality more than any hyperparameter. Preparation is not glamorous, but it is the most time-sensitive part of the workflow.

Dataset size by task

No universal rule, but guidelines:

Smaller datasets work if they are very clean. Larger datasets work even if noisier. The sweet spot is 500–2000 examples for most tasks.

Format standards

For text, use JSONL (one JSON object per line):

{"instruction": "Rewrite in a supportive tone.", "input": "This is broken.", "output": "I understand your frustration. Let's work through this together."}
{"instruction": "Extract JSON from text.", "input": "The user Bob has email bob@example.com", "output": "{\"name\": \"Bob\", \"email\": \"bob@example.com\"}"}

For chat-based tasks, use the model's native chat template:

{"messages": [{"role": "user", "content": "Summarize this ticket."}, {"role": "assistant", "content": "Issue: Login fails. Status: Investigating."}]}

For image adapters, structure as tar/zip with metadata:

dataset.tar.gz
├── images/
│   ├── image_001.jpg
│   ├── image_002.jpg
│   ...
└── metadata.jsonl

Where metadata.jsonl is:

{"image": "image_001.jpg", "caption": "a watercolor painting of a mountain landscape"}
{"image": "image_002.jpg", "caption": "watercolor style, soft edges, muted colors"}

Chat template alignment

For text models, the base model's chat template is critical. Qwen2.5-7B and Mistral have different formats. Training on misaligned templates teaches the adapter to work around the mismatch.

Check the base model's expected format:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
print(tokenizer.chat_template)

Then format your data to match. For Qwen2.5:

messages = [
    {"role": "user", "content": "Draft a support reply."},
    {"role": "assistant", "content": "I'd be happy to help..."}
]

formatted = tokenizer.apply_chat_template(messages, tokenize=False)
# Use formatted text in your JSONL

Preprocessing pipeline

from datasets import Dataset
import json

def preprocess_dataset(jsonl_path):
    # Load
    data = [json.loads(line) for line in open(jsonl_path)]

    # Deduplicate by text content
    seen = set()
    deduped = []
    for record in data:
        key = record.get("output", "")
        if key not in seen:
            seen.add(key)
            deduped.append(record)

    # Filter by length (tokens)
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

    filtered = []
    for record in deduped:
        text = record.get("output", "")
        tokens = len(tokenizer.encode(text))
        if 10 < tokens < 2048:  # reasonable range
            filtered.append(record)

    # Shuffle
    import random
    random.shuffle(filtered)

    return Dataset.from_dict({k: [r[k] for r in filtered] for k in filtered[0].keys()})

dataset = preprocess_dataset("data.jsonl")

Quality audit checklist

Before training, spot-check:

• [ ] 20 random samples: Do outputs make sense for the task?
• [ ] Chat template: Does the prompt/response format match the base model's expectations?
• [ ] No test leakage: Is the eval set completely held out (not in training)?
• [ ] Balance: For classification, are classes roughly equal?
• [ ] Length distribution: Use percentiles (p50, p95, p99) to spot extreme outliers
• [ ] PII: Search for email addresses, phone numbers, names; remove or mask them
• [ ] Duplicates: wc -l before and after dedup; should not lose >5%

Image-specific notes

For style adapters (watercolor, pixel art), caption quality matters more than quantity. 300 well-described images beat 2000 generic ones.

Captions should describe the visual property you're encoding:

Good: - "watercolor style, soft edges, muted earth tones, wet blending" - "pixel art, 8-bit aesthetic, blocky shapes, limited palette"

Bad: - "painting" - "image of a landscape"

For subject adapters (product photography, character style), consistency is key. All 1000 images should be the same aesthetic context.

Upload to ModelForgeLab

The /upload endpoint accepts gzipped archives:

tar -czf dataset.tar.gz data/
curl -X POST -F "file=@dataset.tar.gz" http://localhost:8080/upload

The response includes a job_id. Check /v1/jobs/{job_id} for progress. The uploaded data is streamed to /dev/null and never persisted (privacy by design).

Common mistakes

• Mixing tasks: One dataset = one narrowly-defined task. Do not mix sentiment + summarization examples.
• Instruction creep: If your instruction column is vague ("improve this"), the adapter learns noise.
• No held-out test: Always keep 10–20% of examples for post-training evaluation.
• Template mismatch: Base model's expected format is non-negotiable. Check before you spend 6 hours training.

The best dataset is boring: consistent format, consistent task, consistent quality. Surprises come later, during evaluation.