7 papers
Do Large Language Models Know How Much They Know?
Gabriele Prato, Jerry Huang, Prasanna Parthasarathi +2
Large Language Models (LLMs) have emerged as highly capable systems and are increasingly being integrated into various uses. However, the rapid pace of their deployment has outpace…
Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination
Jerry Huang, Prasanna Parthasarathi, Mehdi Rezagholizadeh +2
The growth in prominence of large language models (LLMs) in everyday life can be largely attributed to their generative abilities, yet some of this is also owed to the risks and co…
Resona: Improving Context Copying in Linear Recurrence Models with Retrieval
Xinyu Wang, Linrui Ma, Jerry Huang +5
Recent shifts in the space of large language model (LLM) research have shown an increasing focus on novel architectures to compete with prototypical Transformer-based models that h…
Manifold Metric: A Loss Landscape Approach for Predicting Model Performance
Pranshu Malviya, Jerry Huang, Aristide Baratin +2
Determining the optimal model for a given task often requires training multiple models from scratch, which becomes impractical as dataset and model sizes grow. A more efficient alt…
How Well Can a Long Sequence Model Model Long Sequences? Comparing Architechtural Inductive Biases on Long-Context Abilities
Jerry Huang
Long sequences occur in abundance within real-world scenarios, hence properly modelling them opens numerous down-stream use-cases. Deep neural networks, however, have often struggl…
ZETA: Leveraging Z-order Curves for Efficient Top-k Attention
Qiuhao Zeng, Jerry Huang, Peng Lu +4
Over recent years, the Transformer has become a fundamental building block for sequence modeling architectures. Yet at its core is the use of self-attention, whose memory and compu…