Phi-3 Mini (GGUF Q8)

Published by @edge_llm · Community adapter

Near-lossless Q8_0 quantisation of Phi-3 Mini 128k. 3.8B parameters in under 4 GB. Exceptional coding and reasoning for its size class.

Usage

Load with Ollama:

# Download the adapter file, then create a Modelfile:
FROM phi-3-mini-128k-instruct
ADAPTER ./phi3-mini-gguf-q8.gguf

# Build and run:
ollama create my-model -f Modelfile
ollama run my-model "Your prompt here"

Or with llama.cpp:

./llama-cli \
  -m base-model.gguf \
  --lora phi3-mini-gguf-q8.gguf \
  -p "Your prompt here"

Compatibility

• llama.cpp
• Ollama
• LM Studio