7 papers · 1 filter
Weight Tying Biases Token Embeddings Towards the Output Space
Antonio Lopardo, Avyukth Harish, Catherine Arnett +1
Weight tying, i.e. sharing parameters between input and output embedding matrices, is common practice in language model design, yet its impact on the learned embedding space remain…
Sylber: Syllabic Embedding Representation of Speech from Raw Audio
Cheol Jun Cho, Nicholas Lee, Akshat Gupta +4
Syllables are compositional units of spoken language that efficiently structure human speech perception and production. However, current neural speech representations lack such str…
PokerBench: Training Large Language Models to become Professional Poker Players
Richard Zhuang, Akshat Gupta, Richard Yang +3
We introduce PokerBench - a benchmark for evaluating the poker-playing abilities of large language models (LLMs). As LLMs excel in traditional NLP tasks, their application to compl…
Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing
Akshat Gupta, Sidharth Baskaran, Gopala Anumanchipalli
Recent work using Rank-One Model Editing (ROME), a popular model editing method, has shown that there are certain facts that the algorithm is unable to edit without breaking the mo…
FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression
Fazal Mittu, Yihuan Bu, Akshat Gupta +4
While the language modeling objective has been shown to be deeply connected with compression, it is surprising that modern LLMs are not employed in practical text compression syste…
Model Editing at Scale leads to Gradual and Catastrophic Forgetting
Akshat Gupta, Anurag Rao, Gopala Anumanchipalli
Editing knowledge in large language models is an attractive capability to have which allows us to correct incorrectly learnt facts during pre-training, as well as update the model…