419 citations · 2.1k across the 46 of their papers we have counts for
47 papers
When Does In-Context Search Help? A Sampling-Complexity Theory of Reflection-Driven Reasoning
Yotam Wolf, Noam Wies, Amnon Shashua
Training large language models (LLMs) with extended reasoning has enabled in-context search, in which models iteratively generate, critique, and revise solution attempts. We provid…
From Reasoning to Super-Intelligence: A Search-Theoretic Perspective
Shai Shalev-Shwartz, Amnon Shashua
Chain-of-Thought (CoT) reasoning has emerged as a powerful tool for enhancing the problem-solving capabilities of large language models (LLMs). However, the theoretical foundations…
FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming
Gal Beniamini, Yuval Dor, Alon Vinnikov +10
Frontier AI models demonstrate formidable breadth of knowledge. But how close are they to true human -- or superhuman -- expertise? Genuine experts can tackle the hardest problems…
Artificial Expert Intelligence through PAC-reasoning
Shai Shalev-Shwartz, Amnon Shashua, Gal Beniamini +6
Artificial Expert Intelligence (AEI) seeks to transcend the limitations of both Artificial General Intelligence (AGI) and narrow AI by integrating domain-specific expertise with cr…
Compositional Hardness of Code in Large Language Models -- A Probabilistic Perspective
Yotam Wolf, Binyamin Rothberg, Dorin Shteyman +1
A common practice in large language model (LLM) usage for complex analytical tasks such as code generation, is to sample a solution for the entire task within the model's context w…
Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods
Yotam Wolf, Noam Wies, Dorin Shteyman +3
Language model alignment has become an important component of AI safety, allowing safe interactions between humans and language models, by enhancing desired behaviors and inhibitin…
Generating Benchmarks for Factuality Evaluation of Language Models
Dor Muhlgay, Ori Ram, Inbal Magar +7
Before deploying a language model (LM) within a given domain, it is important to measure its tendency to generate factually incorrect information in that domain. Existing methods f…
Align With Purpose: Optimize Desired Properties in CTC Models with a General Plug-and-Play Framework
Eliya Segev, Maya Alroy, Ronen Katsir +8
Connectionist Temporal Classification (CTC) is a widely used criterion for training supervised sequence-to-sequence (seq2seq) models. It enables learning the relations between inpu…
Fundamental Limitations of Alignment in Large Language Models
Yotam Wolf, Noam Wies, Oshri Avnery +2
An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by…
The Learnability of In-Context Learning
Noam Wies, Yoav Levine, Amnon Shashua
In-context learning is a surprising and important phenomenon that emerged when modern language models were scaled to billions of learned parameters. Without modifying a large langu…
In-Context Retrieval-Augmented Language Models
Ori Ram, Yoav Levine, Itay Dalmedigos +4
Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significa…
Parallel Context Windows for Large Language Models
Nir Ratner, Yoav Levine, Yonatan Belinkov +7
When applied to processing long text, Large Language Models (LLMs) are limited by their context window. Existing efforts to address this limitation involve training specialized arc…
MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning
Ehud Karpas, Omri Abend, Yonatan Belinkov +14
Huge language models (LMs) have ushered in a new era for AI, serving as a gateway to natural-language-based knowledge tasks. Although an essential element of modern AI, LMs are als…
Standing on the Shoulders of Giant Frozen Language Models
Yoav Levine, Itay Dalmedigos, Ori Ram +10
Huge pretrained language models (LMs) have demonstrated surprisingly good zero-shot capabilities on a wide variety of tasks. This gives rise to the appealing vision of a single, ve…
Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks
Noam Wies, Yoav Levine, Amnon Shashua
The field of Natural Language Processing has experienced a dramatic leap in capabilities with the recent introduction of huge Language Models. Despite this success, natural languag…
The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design
Yoav Levine, Noam Wies, Daniel Jannai +3
Pretraining Neural Language Models (NLMs) over a large corpus involves chunking the text into training examples, which are contiguous text segments of sizes processable by the neur…
Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
Noam Wies, Yoav Levine, Daniel Jannai +1
After their successful debut in natural language processing, Transformer architectures are now becoming the de-facto standard in many domains. An obstacle for their deployment over…
Neural tensor contractions and the expressive power of deep neural quantum states
Or Sharir, Amnon Shashua, Giuseppe Carleo
We establish a direct connection between general tensor networks and deep feed-forward artificial neural networks. The core of our results is the construction of neural-network lay…
The Depth-to-Width Interplay in Self-Attention
Yoav Levine, Noam Wies, Or Sharir +2
Self-attention architectures, which are rapidly pushing the frontier in natural language processing, demonstrate a surprising depth-inefficient behavior: previous works indicate th…
On the Ethics of Building AI in a Responsible Manner
Shai Shalev-Shwartz, Shaked Shammah, Amnon Shashua
The AI-alignment problem arises when there is a discrepancy between the goals that a human designer specifies to an AI learner and a potential catastrophic outcome that does not re…
SenseBERT: Driving Some Sense into BERT
Yoav Levine, Barak Lenz, Or Dagan +6
The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision t…
Deep autoregressive models for the efficient variational simulation of many-body quantum systems
Or Sharir, Yoav Levine, Noam Wies +2
Artificial Neural Networks were recently shown to be an efficient representation of highly-entangled many-body quantum states. In practical applications, neural-network states inhe…
Vision Zero: on a Provable Method for Eliminating Roadway Accidents without Compromising Traffic Throughput
Shai Shalev-Shwartz, Shaked Shammah, Amnon Shashua
We propose an economical, viable, approach to eliminate almost all car accidents. Our method relies on a mathematical model of safety and can be applied to all modern cars at a mil…
Quantum Entanglement in Deep Learning Architectures
Yoav Levine, Or Sharir, Nadav Cohen +1
Modern deep learning has enabled unprecedented achievements in various domains. Nonetheless, employment of machine learning for wave function representations is focused on more tra…