37 citations · 86 across the 18 of their papers we have counts for
19 papers
TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning
Alliot Nagle, Jakhongir Saydaliev, Dhia Garbaya +3
Large Reasoning Models (LRMs) achieve impressive performance on complex reasoning tasks via Chain-of-Thought (CoT) reasoning, which enables them to generate intermediate thinking t…
What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains
Chanakya Ekbote, Marco Bondaschi, Nived Rajaraman +4
In-context learning (ICL) is a hallmark capability of transformers, through which trained models learn to adapt to new tasks by leveraging information from the input context. Prior…
From Markov to Laplace: How Mamba In-Context Learns Markov Chains
Marco Bondaschi, Nived Rajaraman, Xiuying Wei +5
While transformer-based language models have driven the AI revolution thus far, their computational complexity has spurred growing interest in viable alternatives, such as structur…
Transformers on Markov Data: Constant Depth Suffices
Nived Rajaraman, Marco Bondaschi, Kannan Ramchandran +2
Attention-based transformers have been remarkably successful at modeling generative processes across various domains and modalities. In this paper, we study the behavior of transfo…
Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
Alliot Nagle, Adway Girish, Marco Bondaschi +3
We formalize the problem of prompt compression for large language models (LLMs) and present a framework to unify token-level prompt compression methods which create hard prompts fo…
Local to Global: Learning Dynamics and Effect of Initialization for Transformers
Ashok Vardhan Makkuva, Marco Bondaschi, Chanakya Ekbote +4
In recent years, transformer-based models have revolutionized deep learning, particularly in sequence modeling. To better understand this phenomenon, there is a growing interest in…
Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains
Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish +4
Attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. To deepen our understanding of their sequential modeling…
LASER: Linear Compression in Wireless Distributed Optimization
Ashok Vardhan Makkuva, Marco Bondaschi, Thijs Vogels +3
Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its pe…
Machine Learning-Aided Efficient Decoding of Reed-Muller Subcodes
Mohammad Vahid Jamali, Xiyang Liu, Ashok Vardhan Makkuva +3
Reed-Muller (RM) codes achieve the capacity of general binary-input memoryless symmetric channels and are conjectured to have a comparable performance to that of random codes in te…
CRISP: Curriculum based Sequential Neural Decoders for Polar Code Family
S Ashwin Hebbar, Viraj Nadkarni, Ashok Vardhan Makkuva +3
Polar codes are widely used state-of-the-art codes for reliable communication that have recently been included in the 5th generation wireless standards (5G). However, there remains…
TinyTurbo: Efficient Turbo Decoders on Edge
S Ashwin Hebbar, Rajesh K Mishra, Sravan Kumar Ankireddy +3
In this paper, we introduce a neural-augmented decoder for Turbo codes called TINYTURBO . TINYTURBO has complexity comparable to the classical max-log-MAP algorithm but has much be…
KO codes: Inventing Nonlinear Encoding and Decoding for Reliable Wireless Communication via Deep-learning
Ashok Vardhan Makkuva, Xiyang Liu, Mohammad Vahid Jamali +3
Landmark codes underpin reliable physical layer communication, e.g., Reed-Muller, BCH, Convolution, Turbo, LDPC and Polar codes: each is a linear code and represents a mathematical…
Reed-Muller Subcodes: Machine Learning-Aided Design of Efficient Soft Recursive Decoding
Mohammad Vahid Jamali, Xiyang Liu, Ashok Vardhan Makkuva +3
Reed-Muller (RM) codes are conjectured to achieve the capacity of any binary-input memoryless symmetric (BMS) channel, and are observed to have a comparable performance to that of…
Barracuda: The Power of -polling in Proof-of-Stake Blockchains
Giulia Fanti, Jiantao Jiao, Ashok Makkuva +3
A blockchain is a database of sequential events that is maintained by a distributed group of nodes. A key consensus problem in blockchains is that of determining the next block (da…
Optimal transport mapping via input convex neural networks
Ashok Vardhan Makkuva, Amirhossein Taghvaei, Sewoong Oh +1
In this paper, we present a novel and principled approach to learn the optimal transport between two distributions, from samples. Guided by the optimal transport theory, we learn t…
Learning in Gated Neural Networks
Ashok Vardhan Makkuva, Sewoong Oh, Sreeram Kannan +1
Gating is a key feature in modern neural networks including LSTMs, GRUs and sparsely-gated deep neural networks. The backbone of such gated networks is a mixture-of-experts layer,…
Learning One-hidden-layer Neural Networks under General Input Distributions
Weihao Gao, Ashok Vardhan Makkuva, Sewoong Oh +1
Significant advances have been made recently on training neural networks, where the main challenge is in solving an optimization problem with abundant critical points. However, exi…
Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms
Ashok Vardhan Makkuva, Sewoong Oh, Sreeram Kannan +1
Mixture-of-Experts (MoE) is a widely popular model for ensemble learning and is a basic building block of highly successful modern neural networks as well as a component in Gated R…
Equivalence of additive-combinatorial linear inequalities for Shannon entropy and differential entropy
Ashok Vardhan Makkuva, Yihong Wu
This paper addresses the correspondence between linear inequalities of Shannon entropy and differential entropy for sums of independent group-valued random variables. We show that…