RoleQ: Role-Aligned Scales for Converting Language Models to Ternary

Johannes Knittel, Alec Helbling, Hanspeter Pfister

Weight quantization below four bits shrinks memory and bandwidth at a perplexity that looks acceptable, but often at the cost of factual knowledge and task accuracy. Ternary weight…