Llama 3 8B Instruct (GGUF Q4)

Published by @quant_lab · Community adapter

Q4_K_M quantisation of Llama 3 8B Instruct for edge deployment. Runs on 4 GB VRAM or 8 GB RAM CPU-only. Retains ~97 % of full-precision benchmark scores.

Usage

Load with Ollama:

# Download the adapter file, then create a Modelfile:
FROM meta-llama-3-8b-instruct
ADAPTER ./llama3-8b-instruct-gguf-q4.gguf

# Build and run:
ollama create my-model -f Modelfile
ollama run my-model "Your prompt here"

Or with llama.cpp:

./llama-cli \
  -m base-model.gguf \
  --lora llama3-8b-instruct-gguf-q4.gguf \
  -p "Your prompt here"

Compatibility

• llama.cpp
• Ollama
• LM Studio
• Jan