Publications (25)
BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMs
Abbas Ghaddar, Ivan Kobyzev, Boxing Chen +1
Post-training hybridization of large language models (LLMs) often replaces quadratic self-attention with sliding-window attention (SWA) to reduce KV cache usage and improve latency…
Normalizing Flows: An Introduction and Review of Current Methods
Ivan Kobyzev, Simon J. D. Prince, Marcus A. Brubaker
Normalizing Flows are generative models which produce tractable distributions where both sampling and density evaluation can be efficient and exact. The goal of this survey article…
Anti-Yetter-Drinfeld Modules for Quasi-Hopf Algebras
Ivan Kobyzev, Ilya Shapiro
We apply categorical machinery to the problem of defining anti-Yetter-Drinfeld modules for quasi-Hopf algebras. While a definition of Yetter-Drinfeld modules in this setting, extra…
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization
Aref Jafari, Ivan Kobyzev, Mehdi Rezagholizadeh +2
Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowl…
Attribute Controlled Dialogue Prompting
Runcheng Liu, Ahmad Rashid, Ivan Kobyzev +2
Prompt-tuning has become an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks. However, both discrete prompting and…
Integral Transformer: Denoising Attention, Not Too Much Not Too Little
Ivan Kobyzev, Abbas Ghaddar, Dingtao Hu +1
Softmax self-attention often assigns disproportionate weight to semantically uninformative tokens such as special tokens and punctuation, a phenomenon known as attention noise. Whi…
OTTAWA: Optimal TransporT Adaptive Word Aligner for Hallucination and Omission Translation Errors Detection
Chenyang Huang, Abbas Ghaddar, Ivan Kobyzev +3
Recently, there has been considerable attention on detecting hallucinations and omissions in Machine Translation (MT) systems. The two dominant approaches to tackle this task invol…
A Short Study on Compressing Decoder-Based Language Models
Tianda Li, Yassir El Mesbahi, Ivan Kobyzev +6
Pre-trained Language Models (PLMs) have been successful for a wide range of natural language processing (NLP) tasks. The state-of-the-art of PLMs, however, are extremely large to b…
Learning Functions on Multiple Sets using Multi-Set Transformers
Kira Selby, Ahmad Rashid, Ivan Kobyzev +2
We propose a general deep architecture for learning functions on multiple permutation-invariant sets. We also show how to generalize this architecture to sets of elements of any di…
Mathematical Challenges in Deep Learning
Vahid Partovi Nia, Guojun Zhang, Ivan Kobyzev +8
Deep models are dominating the artificial intelligence (AI) industry since the ImageNet challenge in 2012. The size of deep models is increasing ever since, which brings new challe…
Representation Learning for Dynamic Graphs: A Survey
Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain +4
Graphs arise naturally in many real-world applications including social networks, recommender systems, ontologies, biology, and computational finance. Traditionally, machine learni…
Resonance RoPE: Improving Context Length Generalization of Large Language Models
Suyuchen Wang, Ivan Kobyzev, Peng Lu +2
This paper addresses the challenge of train-short-test-long (TSTL) scenarios in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE), where models pre-traine…
Polarized-VAE: Proximity Based Disentangled Representation Learning for Text Generation
Vikash Balasubramanian, Ivan Kobyzev, Hareesh Bahuleyan +2
Learning disentangled representations of real-world data is a challenging open problem. Most previous methods have focused on either supervised approaches which use attribute label…
ReGLA: Refining Gated Linear Attention
Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh +2
Recent advancements in Large Language Models (LLMs) have set themselves apart with their exceptional performance in complex language modelling tasks. However, these models are also…
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh +3
Knowledge Distillation (KD) is a commonly used technique for improving the generalization of compact Pre-trained Language Models (PLMs) on downstream tasks. However, such methods i…
Hyperparameter Optimization for Large Language Model Instruction-Tuning
Christophe Tribes, Sacha Benarroch-Lelong, Peng Lu +1
The fine-tuning of Large Language Models (LLMs) has enabled them to recently achieve milestones in natural language processing applications. The emergence of ever larger LLMs has p…
DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev +1
With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretraine…
Equivariant Finite Normalizing Flows
Avishek Joey Bose, Marcus Brubaker, Ivan Kobyzev
Generative modeling seeks to uncover the underlying factors that give rise to observed data that can often be modeled as the natural symmetries that manifest themselves through inv…
Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity
Michael R. Metel, Peng Lu, Boxing Chen +2
We present a simple on the fly method for faster inference of large language models. Unlike other (self-)speculative decoding techniques, our method does not require fine-tuning or…
A categorical approach to cyclic cohomology of quasi-Hopf algebras and Hopf algebroids
Ivan Kobyzev, Ilya Shapiro
We apply categorical machinery to the problem of defining cyclic cohomology with coefficients in two particular cases, namely quasi-Hopf algebras and Hopf algebroids. In the case o…
LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization
Peng Lu, Ahmad Rashid, Ivan Kobyzev +2
Regularization techniques are crucial to improving the generalization performance and training efficiency of deep neural networks. Many deep learning algorithms rely on weight deca…
Generating Emotionally Aligned Responses in Dialogues using Affect Control Theory
Nabiha Asghar, Ivan Kobyzev, Jesse Hoey +2
State-of-the-art neural dialogue systems excel at syntactic and semantic modelling of language, but often have a hard time establishing emotional alignment with the human interacta…
KronA: Parameter Efficient Tuning with Kronecker Adapter
Ali Edalati, Marzieh Tahaei, Ivan Kobyzev +3
Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task has been a well-known paradigm in Natural Language Processing. However, with the ever-growing size of P…
Do we need Label Regularization to Fine-tune Pre-trained Language Models?
Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh +5
Knowledge Distillation (KD) is a prominent neural model compression technique that heavily relies on teacher network predictions to guide the training of a student model. Consideri…
Tails of Lipschitz Triangular Flows
Priyank Jaini, Ivan Kobyzev, Yaoliang Yu +1
We investigate the ability of popular flow based methods to capture tail-properties of a target density by studying the increasing triangular maps used in these flow methods acting…