2025-05-05 · 4 min read

Prompt Engineering for Adapter-Tuned Models

An adapter changes the base model's behavior. But the prompt must cooperate. A poor prompt can neutralize a good adapter; a good prompt amplifies it.

Core principle

An adapter imposes a bias toward certain behaviors. The prompt should steer in the same direction, not fight it.

Example: qwen25-7b-support-tone was trained on empathetic support replies. A prompt like "Be harsh and direct" contradicts the adapter. It won't work.

Trigger tokens

Some adapters respond to specific words or phrases in the prompt:

For sdxl-watercolor-lora: - Effective: "watercolor style", "soft edges", "muted colors" - Neutral: "a mountain" - Counterproductive: "photorealistic", "sharp details"

For llama3-8b-changelog-writer: - Effective: "changelog", "release notes", "what changed" - Neutral: "write this" - Counterproductive: "detailed commit history"

The adapter encodes which words matter. Using trigger tokens aligns the forward pass with training distribution.

Minimal system prompts

When a LoRA encodes task behavior, explicit system instructions are often redundant:

Poor (fighting the adapter):

System: "You are a helpful support agent. Always be empathetic and professional."

User: "The system is down."

Better (let the adapter work):

System: "You are a support agent."

User: "The system is down."

Best (trust the adapter):

System: ""  (empty or minimal)

User: "The system is down."

The adapter already learned tone and professionalism from 500 support examples. Over-specifying the system prompt introduces conflicting signals.

Temperature calibration

Adapted models often need lower temperature:

Example: phi3-mini-sql-helper at temp 0.7 sometimes mixes SQL dialects or adds prose. At temp 0.2, pure SQL is reliable.

Testing adapter alignment

from transformers import AutoModelForCausalLM
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
base_model = PeftModel.from_pretrained(base_model, "qwen25-7b-support-tone")

test_prompts = [
    "The system is down.",
    "Users cannot log in.",
    "Everything is broken!",
]

# Compare with and without adapter
for prompt in test_prompts:
    inputs = tokenizer(prompt, return_tensors="pt")

    # Without adapter (unload)
    base_only = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
    output_base = base_only.generate(**inputs, max_length=100, temperature=0.7)[0]

    # With adapter
    output_adapted = base_model.generate(**inputs, max_length=100, temperature=0.5)[0]

    print(f"\nPrompt: {prompt}")
    print(f"Base: {tokenizer.decode(output_base, skip_special_tokens=True)[:150]}")
    print(f"Adapted: {tokenizer.decode(output_adapted, skip_special_tokens=True)[:150]}")

Look for differences in tone, not just content. The adapter should shift phrasing and emotional register.

Negative prompts for images

Style adapters benefit from negative guidance:

pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0")
pipe.load_lora_weights("path/to/sdxl-watercolor-lora")

prompt = "a mountain landscape"
negative_prompt = "photorealistic, oil painting, sharp edges, detailed"

image = pipe(
    prompt,
    negative_prompt=negative_prompt,
    guidance_scale=7.0
).images[0]

The negative prompt tells the model what NOT to do. Combined with the LoRA bias toward watercolor, the result is more coherent.

Adapter bleed

Sometimes the adapter injects its training phrases into outputs unintentionally:

Prompt: "Write a SQL query."
Output: "Here's a helpful SQL query to fetch user data: SELECT * FROM users..."

The phrase "helpful SQL query" was not in the prompt but appears in outputs. It's learned phrasing from the training data.

Mitigate by using no_repeat_ngram_size or by checking generated text for patterns:

output = model.generate(
    **inputs,
    max_length=100,
    no_repeat_ngram_size=3,  # Prevents repetitive phrases
)

Or post-process: if the output contains a known training phrase, re-generate.

When a prompt doesn't work

If an adapter produces generic or poor outputs:

1. Check temperature: Maybe too high. Lower to 0.5.
2. Check trigger tokens: Is the prompt aligned with training themes?
3. Check system prompt: Too much specification? Simplify.
4. Check base model compatibility: Is the adapter even loaded? Verify with model.peft_config.
5. Eval the adapter: Maybe it's just weak. Test on training data reproduction.

The adapter and prompt should feel like a single system, not adversarial.

Real example: support conversation

# Good prompt + adapter
system = "You are a support agent."
user_message = "The login page is broken."

full_prompt = f"System: {system}\n\nUser: {user_message}\n\nAgent:"

inputs = tokenizer(full_prompt, return_tensors="pt")
output = model.generate(**inputs, max_length=150, temperature=0.5)

print(tokenizer.decode(output[0], skip_special_tokens=True))
# Expected: empathetic, actionable response

# Bad prompt + same adapter
system = "You are a technical documentation writer."
user_message = "The login page is broken."

# This system prompt contradicts the tone adapter; output is stilted.

The first prompt aligns with the adapter. The second works against it.

Key insight: After training an adapter, spend time on prompt engineering. A great adapter with a mediocre prompt underperforms. A good adapter with excellent prompts outperforms.