6 citations · 21 across the 21 of their papers we have counts for
24 papers
To Copy or Not to Copy: Controlling Speculative Decoding via Intrinsic Model Signals
Roy Eisenstadt, Ido Cohen, Edo Cohen-Karlik +2
Speculative Decoding (SD) has significantly accelerated Large Language Model (LLM) inference, yet existing approaches face a fundamental tradeoff between two drafting strategies: n…
TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors
Ido Andrew Atad, Itamar Zimerman, Shahar Katz +1
Attention matrices are fundamental to transformer research, supporting a broad range of applications including interpretability, visualization, manipulation, and distillation. Yet,…
CLIMP: Contrastive Language-Image Mamba Pretraining
Nimrod Shabtay, Itamar Zimerman, Eli Schwartz +1
Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and scales quadratically with resolu…
Neural Brain Fields: A NeRF-Inspired Approach for Generating Nonexistent EEG Electrodes
Shahar Ain Kedem, Itamar Zimerman, Eliya Nachmani
Electroencephalography (EEG) data present unique modeling challenges because recordings vary in length, exhibit very low signal to noise ratios, differ significantly across partici…
Efficient Decoding Methods for Language Models on Encrypted Data
Matan Avitan, Moran Baruch, Nir Drucker +2
Large language models (LLMs) power modern AI applications, but processing sensitive data on untrusted servers raises privacy concerns. Homomorphic encryption (HE) enables computati…
Differential Mamba
Nadav Schneider, Itamar Zimerman, Eliya Nachmani
Sequence models like Transformers and RNNs often overallocate attention to irrelevant context, leading to noisy intermediate representations. This degrades LLM capabilities by prom…
Overclocking LLM Reasoning: Monitoring and Controlling Thinking Path Lengths in LLMs
Roy Eisenstadt, Itamar Zimerman, Lior Wolf
Recently, techniques such as explicit structured reasoning have demonstrated strong test-time scaling behavior by enforcing a separation between the model's internal "thinking" pro…
Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability
Yarden Bakish, Itamar Zimerman, Hila Chefer +1
The development of effective explainability tools for Transformers is a crucial pursuit in deep learning research. One of the most promising approaches in this domain is Layer-wise…
Overflow Prevention Enhances Long-Context Recurrent LLMs
Assaf Ben-Kish, Itamar Zimerman, M. Jehanzeb Mirza +4
A recent trend in LLMs is developing recurrent sub-quadratic models that improve long-context processing efficiency. We investigate leading large long-context models, focusing on h…
On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach
Edo Cohen-Karlik, Itamar Zimerman, Liane Galanti +3
Recent advances in efficient sequence modeling have introduced selective state-space layers, a key component of the Mamba architecture, which have demonstrated remarkable success i…
Power-Softmax: Towards Secure LLM Inference over Encrypted Data
Itamar Zimerman, Allon Adir, Ehud Aharoni +7
Modern cryptographic methods for implementing privacy-preserving LLMs such as \gls{HE} require the LLMs to have a polynomial form. Forming such a representation is challenging beca…
DeciMamba: Exploring the Length Extrapolation Potential of Mamba
Assaf Ben-Kish, Itamar Zimerman, Shady Abu-Hussein +4
Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrate…
Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation
Itamar Zimerman, Ameen Ali, Lior Wolf
Recent advances in efficient sequence modeling have led to attention-free layers, such as Mamba, RWKV, and various gated RNNs, all featuring sub-quadratic complexity in sequence le…
The Hidden Attention of Mamba Models
Ameen Ali, Itamar Zimerman, Lior Wolf
The Mamba layer offers an efficient selective state space model (SSM) that is highly effective in modeling multiple domains, including NLP, long-range sequence processing, and comp…
On the Long Range Abilities of Transformers
Itamar Zimerman, Lior Wolf
Despite their dominance in modern DL and, especially, NLP domains, transformer architectures exhibit sub-optimal performance on long-range tasks compared to recent layers that are…
Converting Transformers to Polynomial Form for Secure Inference Over Homomorphic Encryption
Itamar Zimerman, Moran Baruch, Nir Drucker +3
Designing privacy-preserving deep learning models is a major challenge within the deep learning community. Homomorphic Encryption (HE) has emerged as one of the most promising appr…
Multi-Dimensional Hyena for Spatial Inductive Bias
Itamar Zimerman, Lior Wolf
In recent years, Vision Transformers have attracted increasing interest from computer vision researchers. However, the advantage of these transformers over CNNs is only fully manif…
Efficient Skip Connections Realization for Secure Inference on Encrypted Data
Nir Drucker, Itamar Zimerman
Homomorphic Encryption (HE) is a cryptographic tool that allows performing computation under encryption, which is used by many privacy-preserving machine learning solutions, for ex…
2-D SSM: A General Spatial Layer for Visual Transformers
Ethan Baron, Itamar Zimerman, Lior Wolf
A central objective in computer vision is to design models with appropriate 2-D inductive bias. Desiderata for 2D inductive bias include two-dimensional position awareness, dynamic…
Decision S4: Efficient Sequence-Based RL via State Spaces Layers
Shmuel Bar-David, Itamar Zimerman, Eliya Nachmani +1
Recently, sequence learning methods have been applied to the problem of off-policy Reinforcement Learning, including the seminal work on Decision Transformers, which employs transf…
Focus Your Attention (with Adaptive IIR Filters)
Shahar Lutati, Itamar Zimerman, Lior Wolf
We present a new layer in which dynamic (i.e.,input-dependent) Infinite Impulse Response (IIR) filters of order two are used to process the input sequence prior to applying convent…
Algorithms for Reconstructing DDoS Attack Graphs using Probabilistic Packet Marking
Dina Barak-Pelleg, Daniel Berend, Thomas J. Robinson +1
DoS and DDoS attacks are widely used and pose a constant threat. Here we explore Probability Packet Marking (PPM), one of the important methods for reconstructing the attack-graph…
Training Large Scale Polynomial CNNs for E2E Inference over Homomorphic Encryption
Moran Baruch, Nir Drucker, Gilad Ezov +5
Training large-scale CNNs that during inference can be run under Homomorphic Encryption (HE) is challenging due to the need to use only polynomial operations. This limits HE-based…
Recovering AES Keys with a Deep Cold Boot Attack
Itamar Zimerman, Eliya Nachmani, Lior Wolf
Cold boot attacks inspect the corrupted random access memory soon after the power has been shut down. While most of the bits have been corrupted, many bits, at random locations, ha…