2025-08-14 · 3 min read

Multi-Adapter Fusion and Task Routing

A single base model can support many task-specific adapters simultaneously. The challenge is composing them intelligently and routing each request to the right set.

Three strategies

Sequential stacking: Load adapter A, then adapter B on top. The forward pass is x → A → B. Simple but order-dependent; results change if you swap A and B.

Weighted merging: Combine adapters with scalar weights. W = W_base + w_1ΔW_1 + w_2ΔW_2. Non-commutative blending; useful for task ensembles.

Dynamic routing: Keep multiple adapters in memory, select one per request based on input classification.

Weighted fusion example

Imagine we want a single model that balances qwen25-7b-support-tone (0.7 weight) and mistral-7b-json-extract (0.3 weight). The second adapter handles structured output, the first handles tone.

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

# Load both adapters
model = PeftModel.from_pretrained(base, "qwen25-7b-support-tone")
model.add_weighted_adapter(
    adapters=["default", "extract"],
    weights=[0.7, 0.3],
    adapter_name="fused",
    combination_type="linear",
)

model.set_adapter("fused")
output = model.generate(input_ids, max_length=128)

The fused adapter inherits traits from both: supportive tone (from the first) with structured constraints (from the second).

Dynamic routing with a classifier

For multi-intent systems, a lightweight classifier routes requests:

import torch
from transformers import pipeline

# Main model
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

# Intent classifier (lightweight)
classifier = pipeline(
    "zero-shot-classification",
    model="MoritzLaurer/DeBERTa-v3-small-mnli-faithful-v1",
)

adapters = {
    "support_tone": "qwen25-7b-support-tone",
    "json_extract": "mistral-7b-json-extract",
    "sql_helper": "phi3-mini-sql-helper",
}

def route_and_infer(prompt):
    # Classify intent
    result = classifier(prompt, list(adapters.keys()))
    intent = result["labels"][0]

    # Load appropriate adapter
    model = PeftModel.from_pretrained(base, adapters[intent])

    # Generate
    inputs = tokenizer(prompt, return_tensors="pt")
    output = model.generate(**inputs, max_length=128)
    return tokenizer.decode(output[0], skip_special_tokens=True)

response = route_and_infer("Please write a JSON summary of this support ticket.")

The classifier overhead is negligible; the DebERTa-small model runs in <50ms on CPU.

Fusion constraints

Not all adapters can be fused safely:

• Same base model: All adapters must target the same base (Qwen2.5-7B, not Qwen2.5-7B + Llama).
• Non-overlapping target modules: If two adapters modify the same q_proj, fusion may amplify errors. Check target_modules in their configs.
• Compatible precision: Mixing fp16 and bf16 adapters is fine; adding a quantized adapter to an unquantized base requires careful alignment.

When in doubt, test the fused adapter on a held-out eval set before deployment.

Performance cost

Stacking adapters adds overhead:

Fused adapters are faster than sequential stacking because they're merged into a single computation graph.

Practical flow for ModelForgeLab

The /v1/jobs endpoint can accept multiple adapters:

curl -X POST http://localhost:8080/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "type": "generate",
    "base_model": "Qwen/Qwen2.5-7B-Instruct",
    "adapters": [
      {"name": "qwen25-7b-support-tone", "weight": 0.7},
      {"name": "mistral-7b-json-extract", "weight": 0.3}
    ],
    "prompt": "Draft a JSON summary of this support ticket.",
    "max_tokens": 256
  }'

The backend composes the weighted adapter and streams back the job ID. The /events/{id} endpoint delivers progress updates.

Image adapter fusion

For sdxl-watercolor-lora and sdxl-product-photo-lora, blending is visual:

from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0")

# Load both adapters
pipe.load_lora_weights("path/to/sdxl-watercolor-lora", weight_name="sdxl-watercolor-lora")
pipe.load_lora_weights("path/to/sdxl-product-photo-lora", weight_name="sdxl-product-photo-lora", adapter_name="product")

# Weighted fusion
pipe.set_adapters(["default", "product"], adapter_weights=[0.6, 0.4])

image = pipe("a watercolor painting of a product on a table").images[0]

The result: watercolor aesthetic (0.6) tempered by product-photography realism (0.4). Quality depends on both the adapters and the interpolation weights.

Debugging multi-adapter workflows

If the fused output is incoherent: - Test each adapter individually on the same prompt; ensure both work - Try w_1=1.0, w_2=0.0 (should match first adapter alone) - Gradually increase w_2; watch for degradation - If fusion is always worse than individual adapters, they may be conflicting in concept

The best multi-adapter systems typically combine orthogonal tasks: tone + format, style + subject, routing + specialization.