2024-12-02 · 3 min read

Learning Rate Schedules for Adapter Training

Learning rate is one of the fastest ways to ruin a fine-tune run if it is too aggressive and one of the easiest ways to waste time if it is too timid. A LoRA adapter has less capacity than a full model, so the schedule needs to be chosen with some care. The goal is to move quickly enough to learn the task but not so quickly that the adapter memorizes or destabilizes.

The simplest schedule is a constant learning rate with a small warmup period. That is a sensible starting point for many adapter jobs because it is easy to reason about and easy to debug. If the run behaves badly, the issue is less likely to be hidden in scheduler complexity.

def lr(step, warmup=100, base=2e-4):
    if step < warmup:
        return base * step / warmup
    return base

Warmup is especially useful when the adapter begins training on a small dataset. It prevents the earliest updates from being too sharp. A short warmup can make the difference between a stable run and a run that jumps around in the first few hundred steps.

Cosine and linear decay schedules are also common. They can help when the run needs a stronger push early on and a gentler finish later. That is useful for many adapter jobs because the model often learns the broad shape of the task early and then needs refinement rather than more force. A schedule that tapers off can preserve generalization better than a flat aggressive rate.

The right schedule depends on dataset size. Small datasets generally prefer smaller learning rates and shorter runs. Larger datasets can tolerate more steps and sometimes a stronger initial rate. Rank also matters because higher-rank adapters can absorb more signal, which can justify a slightly different schedule. The main rule is to keep the schedule simple until the run proves it needs more sophistication.

It is also useful to distinguish between the base learning rate and the effective update size. A run can use a moderate nominal rate and still act aggressively if the batch is tiny or the dataset is extremely repetitive. That is why learning rate should be considered alongside batch size, accumulation, and rank rather than in isolation.

One practical workflow is to run a small schedule sweep. If the adapter underfits at 1e-4, try 2e-4. If it overfits or becomes unstable, drop to 5e-5. This does not need to be elaborate. The point is to find the narrow band where the model learns fast enough but still keeps generalization intact.

Validation is the best guide. If loss drops smoothly but validation degrades, the rate may be too high or the run too long. If neither loss moves, the rate may be too low. If training is unstable, the schedule may be too aggressive at the start. Those are useful signals because they tell you whether to change the rate, the warmup, or the number of steps.

Warmup is one of the cheapest stability tools available. If the early batches are noisy or the dataset is small, a short warmup can keep the first few optimizer updates from pushing the model too hard. That often matters more than a fancy decay curve. A plain schedule with a sensible warmup is better than a sophisticated schedule that nobody can explain later.

The schedule should also match the amount of training data. A tiny dataset rarely needs a long decay tail because the run should end before it drifts too far from the useful range. A larger dataset may benefit from a more gradual decay so later examples can refine the learned behavior without oversteering it.

ModelForgeLab should surface the learning rate and scheduler choice alongside rank and alpha so users can compare runs. A model card that records these settings is much easier to trust than one that only lists the final artifact.