14 citations · 58 across the 43 of their papers we have counts for
51 papers
Algorithmic Principles For Multiclass Learning Are Hard To Come By: Limits of Regularization and Proper Learning
Julian Asilis, Shaddin Dughmi, Vatsal Sharan +3
Two of the most fundamental questions in statistical learning theory are the following: which prediction problems are learnable, and how should they be learned? For the former, ele…
Convergent Evolution: How Different Language Models Learn Similar Number Representations
Deqing Fu, Tianyi Zhou, Mikhail Belkin +2
Language models trained on natural text learn to represent numbers using periodic features with dominant periods at . In this paper, we identify a two-tiered hierarchy…
Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge
Bhavya Vasudeva, Puneesh Deora, Alberto Bietti +2
Transformer-based language models excel at in-context learning (ICL), where they can adapt to new tasks based on contextual examples, without parameter updates. In a specific form…
A Unified Approach to Memory-Sample Tradeoffs for Detecting Planted Structures
Sumegha Garg, Jabari Hastings, Chirag Pabbaraju +1
We present a unified framework for proving memory lower bounds for multi-pass streaming algorithms that detect planted structures. Planted structures -- such as cliques or biclique…
An External Fairness Evaluation of LinkedIn Talent Search
Tina Behzad, Siddartha Devic, Vatsal Sharan +2
We conduct an independent, third-party audit for bias of LinkedIn's Talent Search ranking system, focusing on potential ranking bias across two attributes: gender and race. To do s…
Efficient Swap Multicalibration of Elicitable Properties
Lunjia Hu, Haipeng Luo, Spandan Senapati +1
Multicalibration [HJKRR18] is an algorithmic fairness perspective that demands that the predictions of a predictor are correct conditional on themselves and membership in a collect…
How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
Bhavya Vasudeva, Puneesh Deora, Yize Zhao +2
The growing adoption of spectrum-aware matrix-valued optimizers such as Muon and Shampoo in deep learning motivates a systematic study of their generalization properties and, in pa…
Transformers Provably Learn Algorithmic Solutions for Graph Connectivity, But Only with the Right Data
Qilin Ye, Deqing Fu, Robin Jia +1
Transformers often fail to learn generalizable algorithms, instead relying on brittle heuristics. Using graph connectivity as a testbed, we explain this phenomenon both theoretical…
Auditability and the Landscape of Distance to Multicalibration
Nathan Derhake, Siddartha Devic, Dutch Hansen +2
Calibration is a critical property for establishing the trustworthiness of predictors that provide uncertainty estimates. Multicalibration is a strengthening of calibration which r…
Limitations on Accurate, Trusted, Human-level Reasoning
Rina Panigrahy, Vatsal Sharan
We identify a fundamental incompatibility between the goals of accuracy, trust, and human-level reasoning in artificial intelligence (AI) systems, for strict mathematical definitio…
On the Inherent Privacy of Zeroth Order Projected Gradient Descent
Devansh Gupta, Meisam Razaviyayn, Vatsal Sharan
Differentially private zeroth-order optimization methods have recently gained popularity in private fine tuning of machine learning models due to their reduced memory requirements.…
Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning
Ang Li, Charles Wang, Deqing Fu +9
Humans often use visual aids, for example diagrams or sketches, when solving complex problems. Training multimodal models to do the same, known as Visual Chain of Thought (Visual C…
From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered
Siddartha Devic, Tejas Srinivasan, Jesse Thomason +2
Large Language Models (LLMs) are increasingly assisting users in the real world, yet their reliability remains a concern. Uncertainty quantification (UQ) has been heralded as a too…
Latent Concept Disentanglement in Transformer-based Language Models
Guan Zhe Hong, Bhavya Vasudeva, Vatsal Sharan +3
When large language models (LLMs) use in-context learning (ICL) to solve a new task, they must infer latent concepts from demonstration examples. This raises the question of whethe…
Improved Bounds for Swap Multicalibration and Swap Omniprediction
Haipeng Luo, Spandan Senapati, Vatsal Sharan
In this paper, we consider the related problems of multicalibration -- a multigroup fairness notion and omniprediction -- a simultaneous loss minimization paradigm, both in the dis…
Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models
Woody Haosheng Gan, Deqing Fu, Julian Asilis +5
Steering methods have emerged as effective and targeted tools for guiding large language models' (LLMs) behavior without modifying their parameters. Multimodal large language model…
The Rich and the Simple: On the Implicit Bias of Adam and SGD
Bhavya Vasudeva, Jung Whan Lee, Vatsal Sharan +1
Adam is the de facto optimization algorithm for several deep learning applications, but an understanding of its implicit bias and how it differs from other algorithms, particularly…
Simultaneous Swap Regret Minimization via KL-Calibration
Haipeng Luo, Spandan Senapati, Vatsal Sharan
Calibration is a fundamental concept that aims at ensuring the reliability of probabilistic predictions by aligning them with real-world outcomes. There is a surge of studies on ne…
Proper Learnability and the Role of Unlabeled Data
Julian Asilis, Siddartha Devic, Shaddin Dughmi +2
Proper learning refers to the setting in which learners must emit predictors in the underlying hypothesis class , and often leads to learners with simple algorithmic forms (e.g.…
FoNE: Precise Single-Token Number Embeddings via Fourier Features
Tianyi Zhou, Deqing Fu, Mahdi Soltanolkotabi +2
Large Language Models (LLMs) typically represent numbers using multiple tokens, which requires the model to aggregate these tokens to interpret numerical values. This fragmentation…
Discovering Data Structures: Nearest Neighbor Search and Beyond
Omar Salemohamed, Laurent Charlin, Shivam Garg +2
We propose a general framework for end-to-end learning of data structures. Our framework adapts to the underlying data distribution and provides fine-grained control over query and…
Pre-trained Large Language Models Use Fourier Features to Compute Addition
Tianyi Zhou, Deqing Fu, Vatsal Sharan +1
Pre-trained large language models (LLMs) exhibit impressive mathematical reasoning capabilities, yet how they compute basic arithmetic, such as addition, remains unclear. This pape…
When is Multicalibration Post-Processing Necessary?
Dutch Hansen, Siddartha Devic, Preetum Nakkiran +1
Calibration is a well-studied property of predictors which guarantees meaningful uncertainty estimates. Multicalibration is a related notion -- originating in algorithmic fairness…
Optimal Multiclass U-Calibration Error and Beyond
Haipeng Luo, Spandan Senapati, Vatsal Sharan
We consider the problem of online multiclass U-calibration, where a forecaster aims to make sequential distributional predictions over classes with low U-calibration error, tha…