29 citations · 36 across the 14 of their papers we have counts for
15 papers
Long-Context Drafting Does Not Require Long-Context Drafters
Aref Jafari, Vansh Bhatia, Matt Qin +5
Speculative decoding pairs a cheap drafter with parallel verification against the target. At ultra-long context it stops paying, because a drafter that attends over the full prefix…
Large Language Models with At Most One Spike per Neuron
Zhuoya Zhao, Parsa Omidi, Aref Jafari +1
Leveraging their inherent sparse event-driven computation, spiking neural networks (SNNs) offer a promising path toward energy-efficient large language models (LLMs). Time-to-first…
Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
Parsa Ashrafi Fashi, Utkarsh Saxena, Mehdi Rezagholizadeh +7
Hybrid sequence models that combine efficient Transformer components with linear sequence modeling blocks are a promising alternative to pure Transformers, but most are still pretr…
SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers
Aref Jafari, Yuhe Fan, Benyamin Jamialahmadi +3
Transformers have demonstrated strong performance across a wide range of sequence modeling tasks, but their quadratic attention complexity limits scalability to long sequences. Lin…
DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers
Aman Sharma, Saeed Najafi, Parsa Farinneya +6
Transformers achieve state-of-the-art results across many tasks, but their uniform application of quadratic self-attention to every token at every layer makes them computationally…
Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models
Benyamin Jamialahmadi, Parsa Kavehzadeh, Mehdi Rezagholizadeh +5
Deploying large language models (LLMs) in real-world applications is often hindered by strict computational and latency constraints. While dynamic inference offers the flexibility…
EchoAtt: Attend, Copy, then Adjust for More Efficient Large Language Models
Hossein Rajabzadeh, Aref Jafari, Aman Sharma +5
Large Language Models (LLMs), with their increasing depth and number of parameters, have demonstrated outstanding performance across a variety of natural language processing tasks.…
HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution
Ehsan Kamalloo, Aref Jafari, Xinyu Zhang +2
The rise of large language models (LLMs) had a transformative impact on search, ushering in a new era of search engines that are capable of generating search results in natural lan…
Improved knowledge distillation by utilizing backward pass knowledge in neural networks
Aref Jafari, Mehdi Rezagholizadeh, Ali Ghodsi
Knowledge distillation (KD) is one of the prominent techniques for model compression. In this method, the knowledge of a large network (teacher) is distilled into a model (student)…
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization
Aref Jafari, Ivan Kobyzev, Mehdi Rezagholizadeh +2
Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowl…
Do we need Label Regularization to Fine-tune Pre-trained Language Models?
Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh +5
Knowledge Distillation (KD) is a prominent neural model compression technique that heavily relies on teacher network predictions to guide the training of a student model. Consideri…
Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
Mehdi Rezagholizadeh, Aref Jafari, Puneeth Salad +3
With ever growing scale of neural models, knowledge distillation (KD) attracts more attention as a prominent tool for neural model compression. However, there are counter intuitive…
How to Select One Among All? An Extensive Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language Understanding
Tianda Li, Ahmad Rashid, Aref Jafari +3
Knowledge Distillation (KD) is a model compression algorithm that helps transfer the knowledge of a large neural network into a smaller one. Even though KD has shown promise on a w…
Annealing Knowledge Distillation
Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma +1
Significant memory and computational requirements of large deep neural networks restrict their application on edge devices. Knowledge distillation (KD) is a prominent model compres…
Segmentation Approach for Coreference Resolution Task
Aref Jafari, Ali Ghodsi
In coreference resolution, it is important to consider all members of a coreference cluster and decide about all of them at once. This technique can help to avoid losing precision…