13 citations · 32 across the 19 of their papers we have counts for
7 papers · 1 filter
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
Soumajyoti Sarkar, Yuxin Tang, Sheng Zha
In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and train…
Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation
Joshua Shay Kricheli, Alexander Lawrence Reid, Soumajyoti Sarkar +2
Neural scaling laws approximate a language model's loss as a power-law function of parameter count and token count . Following Chinchilla-style compute-optimal training, man…
Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
Soumajyoti Sarkar, Leonard Lausen, Volkan Cevher +3
Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense models in language modeling. These models use conditionally activated feedforward subnetworks…
EMC: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence
Chung-Yiu Yau, Hoi-To Wai, Parameswaran Raman +2
A key challenge in contrastive learning is to generate negative samples from a large sample set to contrast with positive samples, for learning better encoding of the data. These n…
HYTREL: Hypergraph-enhanced Tabular Data Representation Learning
Pei Chen, Soumajyoti Sarkar, Leonard Lausen +4
Language models pretrained on large collections of tabular data have demonstrated their effectiveness in several downstream tasks. However, many of these models do not take into ac…
An Algorithm For Adversary Aware Decentralized Networked MARL
Soumajyoti Sarkar
Decentralized multi-agent reinforcement learning (MARL) algorithms have become popular in the literature since it allows heterogeneous agents to have their own reward functions as…
Bandit based centralized matching in two-sided markets for peer to peer lending
Soumajyoti Sarkar
Sequential fundraising in two sided online platforms enable peer to peer lending by sequentially bringing potential contributors, each of whose decisions impact other contributors…