9 papers
Variable-Width Transformers
Zhaofeng Wu, Oliver Sieberling, Shawn Tan +3
Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across…
Implicit Representations of Grammaticality in Language Models
Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu +2
Grammaticality and likelihood are distinct notions in human language. Pretrained language models (LMs), which are probabilistic models of language fitted to maximize corpus likelih…
Parallel-SFT: Improving Zero-Shot Cross-Programming-Language Transfer for Code RL
Zhaofeng Wu, Shiqi Wang, Boya Peng +5
Modern language models demonstrate impressive coding capabilities in common programming languages (PLs), such as C++ and Python, but their performance in lower-resource PLs is ofte…
Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages
Danlu Chen, Ka Sing He, Jiahe Tian +4
The landscape of extremely low-resource machine translation (MT) is characterized by perplexing variability in reported performance, often making results across different language…
Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning
Yihong Tang, Ao Qu, Zhaokai Wang +7
Vision language models (VLMs) perform well on many tasks but often fail at spatial reasoning, which is essential for navigation and interaction with physical environments. Many spa…
reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs
Zhaofeng Wu, Michihiro Yasunaga, Andrew Cohen +3
Reward models have become a staple in modern NLP, serving as not only a scalable text evaluator, but also an indispensable component in many alignment recipes and inference-time al…