4 citations · 7 across the 20 of their papers we have counts for
22 papers
CT: Counterfactual Causal Reasoning for Sentiment Shifts in Social-Media Conversation Trees
S M Rafiuddin, Atriya Sen
Sentiment in social-media threads does not only vary across posts; it shifts as users react to claims, corrections, evidence, and hostility within a branching reply tree. We study…
Gender Bias in MT for a Genderless Language: New Benchmarks for Basque
Amaia Murillo, Olatz-Perez-de-Viñaspre, Naiara Perez
Large language models (LLMs) and machine translation (MT) systems are increasingly used in our daily lives, but their outputs can reproduce gender bias present in the training data…
Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs
Rachit Bansal, Aston Zhang, Rishabh Tiwari +8
Progress on training and architecture strategies has enabled LLMs with millions of tokens in context length. However, empirical evidence suggests that such long-context LLMs can co…
The Art of Scaling Reinforcement Learning Compute for LLMs
Devvrit Khatri, Lovish Madaan, Rishabh Tiwari +6
Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pr…
Rethinking Thinking Tokens: LLMs as Improvement Operators
Lovish Madaan, Aniket Didolkar, Suchin Gururangan +6
Reasoning training incentivizes LLMs to produce long chains of thought (long CoT), which among other things, allows them to explore solution strategies with self-checking. This res…
From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation
Seokhee Hong, Sunkyoung Kim, Guijin Son +3
The development of Large Language Models (LLMs) requires robust benchmarks that encompass not only academic domains but also industrial fields to effectively evaluate their applica…
Balancing Knowledge Delivery and Emotional Comfort in Healthcare Conversational Systems
Shang-Chi Tsai, Yun-Nung Chen
With the advancement of large language models, many dialogue systems are now capable of providing reasonable and informative responses to patients' medical conditions. However, whe…
Dialect Normalization using Large Language Models and Morphological Rules
Antonios Dimakis, John Pavlopoulos, Antonios Anastasopoulos
Natural language understanding systems struggle with low-resource languages, including many dialects of high-resource ones. Dialect-to-standard normalization attempts to tackle thi…
Jailbreak Distillation: Renewable Safety Benchmarking
Jingyu Zhang, Ahmed Elgohary, Xiawei Wang +5
Large language models (LLMs) are rapidly deployed in critical applications, raising urgent needs for robust safety benchmarking. We propose Jailbreak Distillation (JBDistill), a no…
ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition
Hisham A. Alyahya, Haidar Khan, Yazeed Alnumay +2
We introduce ZeroSumEval, a dynamic, competition-based, and evolving evaluation framework for Large Language Models (LLMs) that leverages competitive games. ZeroSumEval encompasses…
Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data
Yunhao Tang, Sid Wang, Lovish Madaan +1
We propose to scale RL to unverifiable data with a novel algorithm JEPO (Jensen's Evidence lower bound Policy Optimization). While most prior efforts on scaling RL for LLMs focus o…
Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks
Rylan Schaeffer, Punit Singh Koura, Binh Tang +11
The explosion of high-performing conversational language models (LMs) has spurred a shift from classic natural language processing (NLP) benchmarks to expensive, time-consuming and…
Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study
Menglong Cui, Pengzhi Gao, Wei Liu +2
Large language models (LLMs) have shown continuously improving multilingual capabilities, and even small-scale open-source models have demonstrated rapid performance enhancement. I…
MLGym: A New Framework and Benchmark for Advancing AI Research Agents
Deepak Nathani, Lovish Madaan, Nicholas Roberts +14
We introduce Meta MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing LLM agents on AI research tasks. This is the first Gym environment for machine…
HARP: A challenging human-annotated math reasoning benchmark
Albert S. Yue, Lovish Madaan, Ted Moskovitz +2
Math reasoning is becoming an ever increasing area of focus as we scale large language models. However, even the previously-toughest evals like MATH are now close to saturated by f…
Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models
Lovish Madaan, David Esiobu, Pontus Stenetorp +2
In the recent past, a popular way of evaluating natural language understanding (NLU), was to consider a model's ability to perform natural language inference (NLI) tasks. In this p…
LLäMmlein: Transparent, Compact and Competitive German-Only Language Models from Scratch
Jan Pfister, Julia Wunderle, Andreas Hotho
We create two German-only decoder models, LLäMmlein 120M and 1B, transparently from scratch and publish them, along with the training data, for the German NLP research community to…
The Llama 3 Herd of Models
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri +556
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models th…
Quantifying Variance in Evaluation Benchmarks
Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer +5
Evaluation benchmarks are the cornerstone of measuring capabilities of large language models (LLMs), as well as driving progress in said capabilities. Originally designed to make c…
Treeformer: Dense Gradient Trees for Efficient Attention Computation
Lovish Madaan, Srinadh Bhojanapalli, Himanshu Jain +1
Standard inference and training with transformer based architectures scale quadratically with input sequence length. This is prohibitively large for a variety of applications espec…
Field Study in Deploying Restless Multi-Armed Bandits: Assisting Non-Profits in Improving Maternal and Child Health
Aditya Mate, Lovish Madaan, Aparna Taneja +6
The widespread availability of cell phones has enabled non-profits to deliver critical health information to their beneficiaries in a timely manner. This paper describes our work t…
Selective Intervention Planning using Restless Multi-Armed Bandits to Improve Maternal and Child Health Outcomes
Siddharth Nishtala, Lovish Madaan, Aditya Mate +11
India has a maternal mortality ratio of 113 and child mortality ratio of 2830 per 100,000 live births. Lack of access to preventive care information is a major contributing factor…