2 citations · 4 across the 7 of their papers we have counts for
7 papers
Muon: Training and Trade-offs with Latent Attention and MoE
Sushant Mehta, Raj Dandekar, Rajat Dandekar +1
We present a comprehensive theoretical and empirical study of the Muon optimizer for training transformers only with a small to medium decoder (30M - 200M parameters), with an emph…
A study of Universal ODE approaches to predicting soil organic carbon
Satyanarayana Raju G. V., Prathamesh Dinesh Joshi, Raj Abhijit Dandekar +2
Soil Organic Carbon (SOC) is a foundation of soil health and global climate resilience, yet its prediction remains difficult because of intricate physical, chemical, and biological…
BULL-ODE: Bullwhip Learning with Neural ODEs and Universal Differential Equations under Stochastic Demand
Nachiket N. Naik, Prathamesh Dinesh Joshi, Raj Abhijit Dandekar +2
We study learning of continuous-time inventory dynamics under stochastic demand and quantify when structure helps or hurts forecasting of the bullwhip effect. BULL-ODE compares a f…
Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models
Sushant Mehta, Raj Dandekar, Rajat Dandekar +1
We present MoE-MLA-RoPE, a novel architecture combination that combines Mixture of Experts (MoE) with Multi-head Latent Attention (MLA) and Rotary Position Embeddings (RoPE) for ef…
Latent Multi-Head Attention for Small Language Models
Sushant Mehta, Raj Dandekar, Rajat Dandekar +1
We present the first comprehensive study of latent multi-head attention (MLA) for small language models, revealing interesting efficiency-quality trade-offs. Training 30M-parameter…
CBEval: A framework for evaluating and interpreting cognitive biases in LLMs
Ammar Shaikh, Raj Abhijit Dandekar, Sreedath Panat +1
Rapid advancements in Large Language models (LLMs) has significantly enhanced their reasoning capabilities. Despite improved performance on benchmarks, LLMs exhibit notable gaps in…