2024-09-23 · 3 min read

Choosing Target Modules for LoRA Training

Target modules determine where the LoRA update is applied. That choice has a direct effect on file size, training speed, and how strongly the adapter changes the base model. For many text models, the common starting point is attention projections such as q_proj and v_proj. For some tasks, that is enough. For others, you need a wider attachment pattern.

Attention modules are often the first choice because they influence how information is routed. If the task is mostly tone, formatting, or a narrow response style, attention-only LoRA can be a clean fit. It keeps the adapter compact and often gives enough leverage to shape the output.

If the task requires deeper behavior changes, adding more modules can help. MLP layers can capture a different kind of transformation, and including them may improve capacity at the cost of a larger artifact. That tradeoff matters because every extra module increases training memory and inference footprint.

from peft import LoraConfig

config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
)

That configuration is a reasonable default, but it is not universal. Model naming conventions vary. Some architectures use different projection names, and some frameworks expect different module identifiers. Before training, inspect the model structure instead of assuming the common names will always work.

The main question is not which modules are popular. The question is which modules carry the behavior you want to change. If the adapter has to influence generation style or response form, attention modules may be enough. If the model must adapt to richer domain behavior, adding MLP coverage can help. If the model becomes too large or training gets unstable, back off and simplify.

It also helps to think in terms of debugging. If you start with a small target set and the adapter is weak, you can widen the target modules. If you start with too many modules and the adapter overfits, you can narrow them. That iterative process is easier when the registry stores the exact target modules in metadata and release notes.

There is a practical review step before training: inspect the module names in the model architecture. Different model families expose different naming conventions, and assumptions can break silently. A target module list that works for one transformer may be ignored by another if the names do not line up exactly. That is one of the easiest mistakes to make when moving between architectures.

It also helps to think about the artifact you want to ship. If the registry is expected to distribute compact adapters, you should prefer the smallest target set that still produces the intended behavior. If the artifact is private and training budget is larger, a wider target set may be acceptable. The decision is not only about quality. It is also about whether the final file is practical to store, download, and reuse.

A small target set also makes evaluation easier. If the adapter changes too many modules at once, it becomes harder to tell which part of the architecture is driving the improvement. Narrower target modules give you a cleaner diagnostic path. That matters when you are comparing several experiments or trying to explain a release note to other users.

For image and multimodal models, the same logic still applies, but the module names and layer groups can differ. The registry should never imply that a single target-module recipe works for every family. A useful adapter page tells users what was actually trained, not what a generic guide would have suggested.

ModelForgeLab should make target modules visible on the adapter page because they explain a lot of the artifact's behavior. A user who sees q_proj and v_proj can infer something useful about capacity and compatibility. A user who sees a generic LoRA adapter label cannot.