2025-03-19 · 3 min read

Serving Text Adapters with vLLM

vLLM is a production inference server that natively supports multiple LoRA adapters. Load 4–8 adapters and route requests dynamically with minimal overhead.

Why vLLM

vLLM provides: - Continuous batching (requests don't wait for each other) - Paged attention (memory efficient) - LoRA support with hot-swapping - OpenAI-compatible API (drop-in replacement) - Throughput 10–20x higher than naive inference

Starting vLLM with adapters

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-lora \
  --max-loras 4 \
  --max-lora-rank 16 \
  --lora-modules \
    qwen25-support=/path/to/qwen25-7b-support-tone \
    qwen25-extract=/path/to/mistral-7b-json-extract \
    qwen25-sql=/path/to/phi3-mini-sql-helper \
  --port 8000

The --max-loras 4 parameter reserves 4 adapter slots. Adapters are hot-loaded on request; no restart needed.

Routing with the OpenAI-compatible API

vLLM exposes an OpenAI-compatible interface. To use a specific adapter:

from openai import OpenAI

client = OpenAI(api_key="dummy", base_url="http://localhost:8000/v1")

response = client.completions.create(
    model="qwen25-support",  # adapter name from --lora-modules
    prompt="The system is down. ",
    max_tokens=50,
    temperature=0.7,
)

print(response.choices[0].text)

The backend loads the qwen25-support adapter and applies it to requests automatically.

Batching different adapters

vLLM can batch requests with different adapters in a single step. If 8 requests arrive with adapters ["support", "extract", "support", "sql", ...], vLLM runs them together, applying the right adapter per request.

Performance still degrades with many active adapters in a batch:

For production, keep the number of "hot" adapters (actively used) ≤ 4. Less-used adapters can be loaded on-demand.

Example: multi-task service

from openai import OpenAI
import json

client = OpenAI(api_key="dummy", base_url="http://localhost:8000/v1")

def process_support_ticket(ticket_text):
    """Extract JSON from a support ticket using adapter."""
    response = client.completions.create(
        model="qwen25-extract",  # JSON extraction adapter
        prompt=f"Extract structured data: {ticket_text}",
        max_tokens=256,
        temperature=0.0,  # deterministic
    )
    return json.loads(response.choices[0].text)

def draft_support_reply(ticket_text):
    """Draft a supportive tone reply."""
    response = client.completions.create(
        model="qwen25-support",  # tone adapter
        prompt=f"Draft a reply: {ticket_text}",
        max_tokens=200,
        temperature=0.7,
    )
    return response.choices[0].text

# Pipeline: extract + tone
ticket = "User says the app crashes on startup."
structured = process_support_ticket(ticket)
reply = draft_support_reply(ticket)
print(f"Extracted: {structured}")
print(f"Reply: {reply}")

Troubleshooting adapter loading

Adapter not found: Ensure the --lora-modules name=path matches the actual LoRA directory. Check:

ls -la /path/to/qwen25-7b-support-tone/
# Should contain: adapter_config.json, adapter_model.safetensors

Incompatible base model: The adapter must match the base model exactly. LoRA trained on Qwen/Qwen2.5-7B-Instruct will not work on Qwen/Qwen2.5-7B.

Out of adapter slots: If --max-loras 4 and you try to load a 5th, vLLM returns a 503. Increase the limit or unload a less-used adapter.

Scaling with vLLM

For production workloads:

1. Single GPU (24 GB): Base model + 2–3 adapters, batch size 8–16
2. Multi-GPU (2x A100, 80 GB): Base model + 4–8 adapters, batch size 32+
3. vLLM proxy: Front multiple vLLM instances with a router (nginx, Envoy); distribute adapters across servers

Example 3-instance setup: - Instance A: qwen25-support + qwen25-extract - Instance B: qwen25-sql + phi3-mini-sql-helper - Instance C: backup of Instance A

A router frontend directs requests: qwen25-support → Instance A, qwen25-sql → Instance B.

ModelForgeLab integration

The /v1/jobs endpoint can delegate to vLLM:

# Start vLLM backend with ModelForgeLab adapters
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-lora \
  --max-loras 8 \
  --lora-modules \
    qwen25-support=/data/adapters/qwen25-7b-support-tone \
    mistral-extract=/data/adapters/mistral-7b-json-extract \
    phi3-sql=/data/adapters/phi3-mini-sql-helper \
  --port 8001

ModelForgeLab's app.py can proxy:

@app.post("/v1/jobs")
async def api_create_job(request: Request):
    payload = await request.json()
    adapter = payload.get("adapter", "qwen25-support")
    prompt = payload.get("prompt", "")

    # Call vLLM backend
    vllm_response = requests.post(
        "http://localhost:8001/v1/completions",
        json={"model": adapter, "prompt": prompt, "max_tokens": 256}
    )

    # Stream result back via SSE
    job_id = "job_" + secrets.token_hex(8)
    _register_job(job_id, "completion")
    return JSONResponse({"id": job_id, "stream_url": f"/events/{job_id}"})

Performance tips

• Disable safety transformers if not needed (speed boost)
• Use smaller max-lora-rank if adapters have rank ≤ 8
• Batch requests before sending (higher throughput)
• Monitor adapter cache hit rate; hot adapters should stay in VRAM

vLLM is the production standard for multi-adapter serving. It handles the complexity of hot-loading, batching, and GPU memory management so you don't have to.