Publications (28)
Beyond Pattern Recognition: Probing Mental Representations of LMs
Moritz Miller, Kumar Shridhar
Language Models (LMs) have demonstrated impressive capabilities in solving complex reasoning tasks, particularly when prompted to generate intermediate explanations. However, it re…
The ART of LLM Refinement: Ask, Refine, and Trust
Kumar Shridhar, Koustuv Sinha, Andrew Cohen +6
In recent years, Large Language Models (LLMs) have demonstrated remarkable generative abilities, but can they judge the quality of their own generations? A popular concept, referre…
HyperEmbed: Tradeoffs Between Resources and Performance in NLP Tasks with Hyperdimensional Computing enabled Embedding of n-gram Statistics
Pedro Alonso, Kumar Shridhar, Denis Kleyko +2
Recent advances in Deep Learning have led to a significant performance increase on several NLP tasks, however, the models become more and more computationally demanding. Therefore,…
SMART: Self-learning Meta-strategy Agent for Reasoning Tasks
Rongxing Liu, Kumar Shridhar, Manish Prajapat +2
Tasks requiring deductive reasoning, especially those involving multiple steps, often demand adaptive strategies such as intermediate generation of rationales or programs, as no si…
SCREWS: A Modular Framework for Reasoning with Revisions
Kumar Shridhar, Harsh Jhamtani, Hao Fang +3
Large language models (LLMs) can improve their accuracy on various tasks through iteratively refining and revising their output based on feedback. We observe that these revisions c…
One to rule them all: Towards Joint Indic Language Hate Speech Detection
Mehar Bhatia, Tenzin Singhay Bhotia, Akshat Agarwal +5
This paper is a contribution to the Hate Speech and Offensive Content Identification in Indo-European Languages (HASOC) 2021 shared task. Social media today is a hotbed of toxic an…
SIKeD: Self-guided Iterative Knowledge Distillation for mathematical reasoning
Shivam Adarsh, Kumar Shridhar, Caglar Gulcehre +2
Large Language Models (LLMs) can transfer their reasoning skills to smaller models by teaching them to generate the intermediate reasoning process required to solve multistep reaso…
Distilling Reasoning Capabilities into Smaller Language Models
Kumar Shridhar, Alessandro Stolfo, Mrinmaya Sachan
Step-by-step reasoning approaches like chain of thought (CoT) have proved to be very effective in inducing reasoning capabilities in large language models. However, the success of…
A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models
Alessandro Stolfo, Zhijing Jin, Kumar Shridhar +2
We have recently witnessed a number of impressive results on hard mathematical reasoning problems with language models. At the same time, the robustness of these models has also be…
ProbAct: A Probabilistic Activation Function for Deep Neural Networks
Kumar Shridhar, Joonho Lee, Hideaki Hayashi +6
Activation functions play an important role in training artificial neural networks. The majority of currently used activation functions are deterministic in nature, with their fixe…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…
Distilling LLMs' Decomposition Abilities into Compact Language Models
Denis Tarasov, Kumar Shridhar
Large Language Models (LLMs) have demonstrated proficiency in their reasoning abilities, yet their large size presents scalability challenges and limits any further customization.…
Learning to Drop Out: An Adversarial Approach to Training Sequence VAEs
ÄorÄe MiladinoviÄ, Kumar Shridhar, Kushal Jain +4
In principle, applying variational autoencoders (VAEs) to sequential data offers a method for controlled sequence generation, manipulation, and structured representation learning.…
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
Kumar Shridhar, Felix Laumann, Marcus Liwicki
Artificial Neural Networks are connectionist systems that perform a given task by learning on examples without having prior knowledge about the task. This is done by finding an opt…
Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages
Kushal Jain, Adwait Deshpande, Kumar Shridhar +2
Language models based on the Transformer architecture have achieved state-of-the-art performance on a wide range of NLP tasks such as text classification, question-answering, and t…
Calibrating Large Language Models with Sample Consistency
Qing Lyu, Kumar Shridhar, Chaitanya Malaviya +6
Accurately gauging the confidence level of Large Language Models' (LLMs) predictions is pivotal for their reliable application. However, LLMs are often uncalibrated inherently and…
Automatic Generation of Socratic Subquestions for Teaching Math Word Problems
Kumar Shridhar, Jakub Macina, Mennatallah El-Assady +3
Socratic questioning is an educational method that allows students to discover answers to complex problems by asking them a series of thoughtful questions. Generation of didactical…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1144
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…
Subword Semantic Hashing for Intent Classification on Small Datasets
Kumar Shridhar, Ayushman Dash, Amit Sahu +6
In this paper, we introduce the use of Semantic Hashing as embedding for the task of Intent Classification and achieve state-of-the-art performance on three frequently used benchma…
Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
Kumar Shridhar, Felix Laumann, Marcus Liwicki
We introduce a novel uncertainty estimation for classification tasks for Bayesian convolutional neural networks with variational inference. By normalizing the output of a Softplus…
Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
Project Apertus, Alejandro Hernández-Cano, Alexander Hägele +100
We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingu…
Translational Equivariance in Kernelizable Attention
Max Horn, Kumar Shridhar, Elrich Groenewald +1
While Transformer architectures have show remarkable success, they are bound to the computation of all pairwise interactions of input element and thus suffer from limited scalabili…
First-Step Advantage: Importance of Starting Right in Multi-Step Math Reasoning
Kushal Jain, Moritz Miller, Niket Tandon +1
Language models can solve complex reasoning tasks better by learning to generate rationales for their predictions. Often these models know how to solve a task but their auto-regres…
Longtonotes: OntoNotes with Longer Coreference Chains
Kumar Shridhar, Nicholas Monath, Raghuveer Thirukovalluru +4
Ontonotes has served as the most important benchmark for coreference resolution. However, for ease of annotation, several long documents in Ontonotes were split into smaller parts.…
BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
Terry Yue Zhuo, Xiaolong Jin, Hange Liu +37
Crowdsourced model evaluation platforms, such as Chatbot Arena, enable real-time evaluation from human perspectives to assess the quality of model responses. In the coding domain,…
End to End Binarized Neural Networks for Text Classification
Harshil Jain, Akshat Agarwal, Kumar Shridhar +1
Deep neural networks have demonstrated their superior performance in almost every Natural Language Processing task, however, their increasing complexity raises concerns. In particu…
EMAFusion: A Self-Optimizing System for Seamless LLM Selection and Integration
Soham Shah, Kumar Shridhar, Surojit Chatterjee +1
While recent advances in large language models (LLMs) have significantly enhanced performance across diverse natural language tasks, the high computational and financial costs asso…
UNDO: Understanding Distillation as Optimization
Kushal Jain, Piyushi Goyal, Kumar Shridhar
Knowledge distillation has emerged as an effective strategy for compressing large language models' (LLMs) knowledge into smaller, more efficient student models. However, standard o…