49 citations · 130 across the 12 of their papers we have counts for
4 papers · 1 filter
Parallel Context Windows for Large Language Models
Nir Ratner, Yoav Levine, Yonatan Belinkov +7
When applied to processing long text, Large Language Models (LLMs) are limited by their context window. Existing efforts to address this limitation involve training specialized arc…
What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary
Ori Ram, Liat Bezalel, Adi Zicher +3
Dual encoders are now the dominant architecture for dense retrieval. Yet, we have little understanding of how they represent text, and why this leads to good performance. In this w…
Standing on the Shoulders of Giant Frozen Language Models
Yoav Levine, Itay Dalmedigos, Ori Ram +10
Huge pretrained language models (LMs) have demonstrated surprisingly good zero-shot capabilities on a wide variety of tasks. This gives rise to the appealing vision of a single, ve…
Transformer Language Models without Positional Encodings Still Learn Positional Information
Adi Haviv, Ori Ram, Ofir Press +2
Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, we show that LMs without any ex…