Tuning Language Models by Mixture-of-Depths Ensemble

Haoyan Luo, Lucia Specia

Transformer-based Large Language Models (LLMs) traditionally rely on final-layer loss for finetuning and final-layer representations for predictions, potentially overlooking the pr…