31 citations · 45 across the 10 of their papers we have counts for
10 papers
Automatic Pruning of Fine-tuning Datasets for Transformer-based Language Models
Mohammadreza Tayaranian, Seyyed Hasan Mozafari, Brett H. Meyer +2
Transformer-based language models have shown state-of-the-art performance on a variety of natural language understanding tasks. To achieve this performance, these models are first…
KronA: Parameter Efficient Tuning with Kronecker Adapter
Ali Edalati, Marzieh Tahaei, Ivan Kobyzev +3
Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task has been a well-known paradigm in Natural Language Processing. However, with the ever-growing size of P…
BD-KD: Balancing the Divergences for Online Knowledge Distillation
Ibtihel Amara, Nazanin Sepahvand, Brett H. Meyer +2
We address the challenge of producing trustworthy and accurate compact models for edge devices. While Knowledge Distillation (KD) has improved model compression in terms of achievi…
CES-KD: Curriculum-based Expert Selection for Guided Knowledge Distillation
Ibtihel Amara, Maryam Ziaeefard, Brett H. Meyer +2
Knowledge distillation (KD) is an effective tool for compressing deep classification models for edge devices. However, the performance of KD is affected by the large capacity gap b…
Efficient Fine-Tuning of Compressed Language Models with Learners
Danilo Vucetic, Mohammadreza Tayaranian, Maryam Ziaeefard +3
Fine-tuning BERT-based models is resource-intensive in memory, computation, and time. While many prior works aim to improve inference efficiency via compression techniques, e.g., p…
Clustered Saliency Prediction
Rezvan Sherkati, James J. Clark
We present a new method for image salience prediction, Clustered Saliency Prediction. This method divides subjects into clusters based on their personal features and their known sa…
Efficient Fine-Tuning of BERT Models on the Edge
Danilo Vucetic, Mohammadreza Tayaranian, Maryam Ziaeefard +3
Resource-constrained devices are increasingly the deployment targets of machine learning applications. Static models, however, do not always suffice for dynamic environments. On-de…
Consistency driven Sequential Transformers Attention Model for Partially Observable Scenes
Samrudhdhi B. Rangrej, Chetan L. Srinidhi, James J. Clark
Most hard attention models initially observe a complete scene to locate and sense informative glimpses, and predict class-label of a scene based on glimpses. However, in many appli…
Standard Deviation-Based Quantization for Deep Neural Networks
Amir Ardakani, Arash Ardakani, Brett Meyer +2
Quantization of deep neural networks is a promising approach that reduces the inference cost, making it feasible to run deep networks on resource-restricted devices. Inspired by ex…
Kronecker Decomposition for GPT Compression
Ali Edalati, Marzieh Tahaei, Ahmad Rashid +3
GPT is an auto-regressive Transformer-based pre-trained language model which has attracted a lot of attention in the natural language processing (NLP) domain due to its state-of-th…