Mixtral 8x7B

46.7B parametersMoE
reasoningcodingmoe
Parameters
46.7B
12.9B active
Context Length
32K
Architecture
mixtral
Release Date
2023

Available Quantizations

QuantizationFile SizeQuality
Q2_K17.5 GBLower
Q3_K_M21.8 GBLower
Q4_K_M26.5 GBGood
Q5_K_M30.5 GBGood
Q8_049.0 GBHigh

Can it run? (Q4_K_M quantization)

GeForce RTX 4090
Runs slowly
~5 tok/s • 24GB VRAM
GeForce RTX 4070
Won't run
~0 tok/s • 12GB VRAM
GeForce RTX 3060 12GB
Won't run
~0 tok/s • 12GB VRAM
Radeon RX 7900 XTX
Runs slowly
~5 tok/s • 24GB VRAM
Check with your specific GPU →