2 citations · 2 across the 4 of their papers we have counts for
7 papers
Cross-Lingual Alignment for Decoder-Only Models using MoE Routers
Lucas Bandarkar, Clark Peng, Ahmed Haj Ahmed +2
Cross-lingual contrastive learning has been a core component of multilingual encoder training, but the ability to explicitly align representations is not possible in decoder-only L…
From Routing Signals to Selective Review: Visual regrounding in MoE VLMs
Hongzhu Guo, Mohsen Fayyaz, Nanyun Peng
Vision-language models (VLMs) may accept false visual premises, answering questions about a target object's color, count, location, or state even when it is absent. We call this re…
Multilinguality in Hybrid Attention LLMs
Lucas Bandarkar, Junlin Hu, Chenyuan Yang +2
In response to the growing demand for long sequences in agentic and reasoning use cases, many state-of-the-art LLMs combine multiple variants of attention to mitigate the quadratic…
The Network Structure of Mathlib
Xinze Li, Nanyun Peng, Simone Severini +1
The ongoing development of Lean 4's Mathlib has produced a macroscopic structural complexity that interweaves logical, mathematical, and infrastructural dependencies. We present a…
Multilingual Routing in Mixture-of-Experts
Lucas Bandarkar, Chenyuan Yang, Mohsen Fayyaz +2
Mixture-of-Experts (MoE) architectures have become the key to scaling modern LLMs, yet little is understood about how their sparse routing dynamics respond to multilingual data. In…
Steering MoE LLMs via Expert (De)Activation
Mohsen Fayyaz, Ali Modarressi, Hanieh Deilamsalehy +5
Mixture-of-Experts (MoE) in Large Language Models (LLMs) routes each token through a subset of specialized Feed-Forward Networks (FFN), known as experts. We present SteerMoE, a fra…
Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence
Mohsen Fayyaz, Ali Modarressi, Hinrich Schuetze +1
Dense retrieval models are commonly used in Information Retrieval (IR) applications, such as Retrieval-Augmented Generation (RAG). Since they often serve as the first step in these…