5 papers
CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry
Huzama Ahmad, Cao Viet Hai Nam, Se-Young Yun
Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value. We present two efficiency methods that exploit this asymmetry.…
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Huzama Ahmad, Se-Young Yun
Long contexts have become standard in pretrained LLMs, yet they remain expensive to run: prefill compute grows quadratically with sequence length, and every decode step re-reads a…
Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting
Soowon Oh, Nam Cao, Yujin Kim +4
Block-diffusion drafters have recently emerged as a powerful alternative for speculative decoding by predicting multiple future-token distributions in a single parallel step. Howev…
Diffusion Models Through a Global Lens: Are They Culturally Inclusive?
Zahra Bayramli, Ayhan Suleymanzade, Na Min An +5
Text-to-image diffusion models have recently enabled the creation of visually compelling, detailed images from textual prompts. However, their ability to accurately represent vario…
When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contexts
Jun Seong Kim, Kyaw Ye Thu, Javad Ismayilzada +6
In a highly globalized world, it is important for multi-modal large language models (MLLMs) to recognize and respond correctly to mixed-cultural inputs. For example, a model should…