6 papers
Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy
Shmuel Berman, Jia Deng
Memory is widely viewed as an important unsolved problem for LLMs and VLMs, and current benchmarks typically evaluate it by testing accuracy over long text or video. However, accur…
q0: Primitives for Hyper-Epoch Pretraining
Bishwas Mandal, Shmuel Berman, Akshay Vegesna +1
Multi-epoch training is becoming the standard now that compute is growing faster than the supply of high-quality text. But pretraining a single model saturates within a few passes,…
VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs
Shmuel Berman, Jia Deng
Vision-Language Models (VLMs) excel at complex visual tasks such as VQA and chart understanding, yet recent work suggests they struggle with simple perceptual tests. We present an…
Facts Do Care About Your Language: Assessing Answer Quality of Multilingual LLMs
Yuval Kansal, Shmuel Berman, Lydia Liu
Factuality is a necessary precursor to useful educational tools. As adoption of Large Language Models (LLMs) in education continues of grow, ensuring correctness in all settings is…
Solving Zebra Puzzles Using Constraint-Guided Multi-Agent Systems
Shmuel Berman, Kathleen McKeown, Baishakhi Ray
Prior research has enhanced the ability of Large Language Models (LLMs) to solve logic puzzles using techniques such as chain-of-thought prompting or introducing a symbolic represe…
Programming-By-Example by Programming-By-Example: Synthesis of Looping Programs
Shmuel Berman, Mark Santolucito
Program synthesis has seen many new applications in recent years, in large part thanks to the introduction of SyGuS. However, no existing SyGuS solvers have support for synthesizin…