2024-06-10 · 4 min read

Dataset Cleaning Checklist for Fine-Tuning

The quality of a fine-tune run is usually limited by the dataset long before it is limited by the model. A small clean dataset often beats a larger noisy one because the model can actually learn the pattern you intended. ModelForgeLab should treat dataset preparation as a first-class step, not as an upload form with a progress bar.

The first pass is structural. Check that every record has the fields the trainer expects, that the file encoding is valid UTF-8, and that the line format is consistent. JSONL is a good default because it is easy to stream and easy to inspect. A broken JSON object in the middle of a file can waste an entire run if it is not caught early.

import json
from pathlib import Path

path = Path("data/train.jsonl")
bad_lines = 0
total = 0

for lineno, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1):
    total += 1
    try:
        obj = json.loads(line)
    except json.JSONDecodeError:
        bad_lines += 1
        print(f"bad json at line {lineno}")
        continue
    for key in ("instruction", "input", "output"):
        if key not in obj or not str(obj[key]).strip():
            print(f"missing {key} at line {lineno}")

print({"total": total, "bad_lines": bad_lines})

After structure, look for duplicates. Duplicate examples inflate the apparent size of the dataset without adding new information. They also make evaluation less honest because the model may memorize repeated phrasing. Near-duplicates matter too, especially when the same prompt appears with only tiny wording changes. A quick dedup pass based on normalized text is usually enough to catch the worst offenders.

Normalization should be simple and conservative. Lowercasing may help in some tasks, but do not apply aggressive transforms unless they preserve meaning. Remove leading and trailing whitespace. Collapse repeated spaces. Normalize line endings. Keep punctuation when it carries semantics, because punctuation often matters in instruction data and support transcripts.

Some datasets need content filtering. If your task is customer support, remove raw secrets, session IDs, API keys, and accidental logs. If your task is code generation, make sure the examples are syntactically valid. If your task is summary generation, check that the input is actually longer and more detailed than the output. The model learns from examples, so broken examples teach broken patterns.

Label consistency is another weak point. In chat data, the role structure should be stable. If one sample uses system, user, and assistant, then all comparable samples should do the same. Mixing templates from different tooling is one of the easiest ways to produce a dataset that looks valid but trains poorly. The structure can vary across tasks, but it should not vary by accident.

A useful review step is to sample a few dozen records and read them by hand. That is still the fastest way to catch weird formatting, repeated boilerplate, and mismatched inputs. Machines are good at counting problems. People are better at noticing when a dataset feels off.

Train and validation splits need to be honest. Put near-duplicates in the same split. If a prompt appears in both training and validation, the validation score is inflated. Split by source, by customer, by document family, or by time when that is available. Random splitting is fine only when the dataset is already diverse and de-duplicated.

If the data contains multi-turn conversations, keep full conversations together. Do not split a single conversation across training and validation. That leaks context and makes the evaluation look better than it should. The same rule applies to documents with sequential sections that share strong local dependencies.

Length distribution matters too. Very long examples can dominate memory usage and make training less stable. Very short examples can cause the model to overfit to terse responses. It helps to compute token counts before the run and inspect the tail. If a few examples are extreme outliers, decide whether to trim, split, or remove them.

python - <<'PY'
import json
from pathlib import Path

path = Path("data/train.jsonl")
lengths = []
for line in path.read_text(encoding="utf-8").splitlines():
    obj = json.loads(line)
    text = " ".join(str(obj.get(k, "")) for k in ("instruction", "input", "output"))
    lengths.append(len(text.split()))

print("count", len(lengths))
print("max", max(lengths))
print("avg", sum(lengths) / len(lengths))
PY

Once the dataset is clean, freeze it. Fine-tuning works best when the exact training set is reproducible. Store the file hash, the split rules, and the cleaning notes next to the artifact. If the result looks good, you want to be able to trace it back to the exact input that produced it. That is the sort of discipline a registry like ModelForgeLab should make visible in the UI, not hide in a notebook.

The final pass is a sanity check against the intended behavior. Ask whether the dataset really teaches one thing, whether the labels are consistent, and whether the examples are representative of real use. If the answer is fuzzy, the model will usually be fuzzy too. A clean dataset is not glamorous, but it is the part that keeps the whole pipeline honest.