196 citations · 203 across the 12 of their papers we have counts for
12 papers
Fine-tuning vs. In-context Learning in Large Language Models: A Formal Language Learning Perspective
Bishwamittra Ghosh, Soumi Das, Till Speicher +5
Large language models (LLMs) operate in two fundamental learning modes - fine-tuning (FT) and in-context learning (ICL) - raising key questions about which mode yields greater lang…
A Family of LLMs Liberated from Static Vocabularies
Aleph Alpha, :, Adnen Abdessaied +35
Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although lea…
Revisiting Privacy, Utility, and Efficiency Trade-offs when Fine-Tuning Large Language Models
Soumi Das, Camila Kolling, Mohammad Aflah Khan +5
We study the inherent trade-offs in minimizing privacy risks and maximizing utility, while maintaining high computational efficiency, when fine-tuning large language models (LLMs).…
Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications
Till Speicher, Mohammad Aflah Khan, Qinyuan Wu +5
Understanding whether and to what extent large language models (LLMs) have memorised training data has important implications for the reliability of their output and the privacy of…
Understanding the Role of Invariance in Transfer Learning
Till Speicher, Vedant Nanda, Krishna P. Gummadi
Transfer learning is a powerful technique for knowledge-sharing between different tasks. Recent work has found that the representations of models with certain invariances, such as…
Towards Reliable Latent Knowledge Estimation in LLMs: Zero-Prompt Many-Shot Based Factual Knowledge Extraction
Qinyuan Wu, Mohammad Aflah Khan, Soumi Das +7
In this paper, we focus on the challenging task of reliably estimating factual knowledge that is embedded inside large language models (LLMs). To avoid reliability concerns with pr…
Diffused Redundancy in Pre-trained Representations
Vedant Nanda, Till Speicher, John P. Dickerson +3
Representations learned by pre-training a neural network on a large dataset are increasingly used successfully to perform a variety of downstream tasks. In this work, we take a clo…
Investigating the Effects of Fairness Interventions Using Pointwise Representational Similarity
Camila Kolling, Till Speicher, Vedant Nanda +2
Machine learning (ML) algorithms can often exhibit discriminatory behavior, negatively affecting certain populations across protected groups. To address this, numerous debiasing me…
Measuring Representational Robustness of Neural Networks Through Shared Invariances
Vedant Nanda, Till Speicher, Camila Kolling +3
A major challenge in studying robustness in deep learning is defining the set of ``meaningless'' perturbations to which a given Neural Network (NN) should be invariant. Most work o…
Unifying Model Explainability and Robustness via Machine-Checkable Concepts
Vedant Nanda, Till Speicher, John P. Dickerson +2
As deep neural networks (DNNs) get adopted in an ever-increasing number of applications, explainability has emerged as a crucial desideratum for these models. In many real-world ta…
A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices
Till Speicher, Hoda Heidari, Nina Grgic-Hlaca +4
Discrimination via algorithmic decision making has received considerable attention. Prior work largely focuses on defining conditions for fairness, but does not define satisfactory…
A Generalized Language Model as the Combination of Skipped n-grams and Modified Kneser-Ney Smoothing
Rene Pickhardt, Thomas Gottron, Martin Körner +3
We introduce a novel approach for building language models based on a systematic, recursive exploration of skip n-gram models which are interpolated using modified Kneser-Ney smoot…