Yi 6B

6B parameters
reasoning
Parameters
6B
Context Length
4K
Architecture
llama
Release Date
2023

Available Quantizations

QuantizationFile SizeQuality
Q2_K2.4 GBLower
Q3_K_M3.0 GBLower
Q4_K_M3.7 GBGood
Q5_K_M4.2 GBGood

Can it run? (Q4_K_M quantization)

GeForce RTX 4090
Runs great
~194 tok/s • 24GB VRAM
GeForce RTX 4070
Runs great
~97 tok/s • 12GB VRAM
GeForce RTX 3060 12GB
Runs great
~69 tok/s • 12GB VRAM
Radeon RX 7900 XTX
Runs great
~185 tok/s • 24GB VRAM
Check with your specific GPU →