Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing

Aviv Bick, Tobias Katsch, Nimit Sohoni +2

We introduce Llamba, a family of efficient recurrent language models distilled from Llama-3.x into the Mamba architecture. The series includes Llamba-1B, Llamba-3B, and Llamba-8B,…