2024-07-01 · 3 min read

Train and Validation Splits That Catch Real Regressions

A validation split is only useful if it measures something the model has not already seen. If the split leaks duplicates, near-duplicates, or tightly related samples, the validation score becomes decorative. ModelForgeLab should show split rules because they are part of the artifact quality, not an implementation footnote.

The easiest split to get wrong is random splitting on a noisy dataset. If the same prompt appears multiple times with tiny variations, random splitting will spread similar records across train and validation. The model appears stronger than it is because it is essentially seeing the same example twice. A cleaner split groups by source, conversation, user, document, or time window when possible.

If the data comes from conversations, keep full conversations together. Do not split a single thread across train and validation. If the data comes from documents, keep document sections together when they share context. If the data comes from image captions, keep closely related variants together so the validation set does not mirror the training set too closely.

import json
import random
from pathlib import Path

random.seed(42)
rows = [json.loads(line) for line in Path("data.jsonl").read_text(encoding="utf-8").splitlines()]
random.shuffle(rows)
split = int(len(rows) * 0.9)
train, valid = rows[:split], rows[split:]
print(len(train), len(valid))

That simple split is fine only when the dataset has already been deduplicated and the examples are independent. Otherwise, group-aware splitting is safer. A dataset with repeated customer tickets, repeated prompts, or templated examples should not be split blindly.

Validation size matters too. If validation is too small, the score is noisy and unhelpful. If it is too large, training loses too much signal. The right size depends on the dataset, but a small adapter project should usually reserve enough examples to produce a stable signal across multiple checkpoints. A tiny validation set that changes dramatically from run to run is not much better than no validation at all.

One useful pattern is to create several validation slices rather than one monolithic split. For example, hold out easy cases, hard cases, and boundary cases separately. That makes it easier to see what kind of regression a new run introduces. The model might improve on straightforward prompts but get worse on edge cases. A single average score can hide that.

A good split also respects output diversity. If all the long-form answers end up in training and all the short ones end up in validation, the comparison becomes skewed. Try to preserve the distribution of prompt types, lengths, and output styles across the splits. The goal is not perfect statistical symmetry. The goal is to avoid obvious distortions.

from collections import Counter
from pathlib import Path

def bucket(row):
    text = row.get("output", "")
    return "long" if len(text.split()) > 50 else "short"

rows = [json.loads(line) for line in Path("data.jsonl").read_text(encoding="utf-8").splitlines()]
counts = Counter(bucket(r) for r in rows)
print(counts)

For fine-tuning workflows, it helps to store split metadata next to the dataset: source rules, seed, date, and any grouping keys. If a future run behaves differently, the first question should be whether the split changed. That is especially useful in ModelForgeLab because users may retrain with slightly different data over time.

The best validation split is the one that reliably catches the mistakes you care about. If the task is style transfer, validate style consistency and prompt adherence. If the task is structured output, validate format and exact field coverage. If the task is image generation, use fixed prompts and seeds so visual regressions are easy to compare.