Llama 3.1 70B Chat (GGUF Q3)

Published by @big_model_hub · Community adapter

Aggressively quantised 70B for servers with 32 GB RAM. Q3_K_M offers the largest context window at a reasonable quality/size tradeoff for bulk inference.

Usage

Load with Ollama:

# Download the adapter file, then create a Modelfile:
FROM meta-llama-3.1-70b-instruct
ADAPTER ./llama31-70b-chat-gguf-q3.gguf

# Build and run:
ollama create my-model -f Modelfile
ollama run my-model "Your prompt here"

Or with llama.cpp:

./llama-cli \
  -m base-model.gguf \
  --lora llama31-70b-chat-gguf-q3.gguf \
  -p "Your prompt here"

Compatibility

• llama.cpp
• Ollama