26 citations · 97 across the 29 of their papers we have counts for
35 papers
Mathematical Challenges in Deep Learning
Vahid Partovi Nia, Guojun Zhang, Ivan Kobyzev +8
Deep models are dominating the artificial intelligence (AI) industry since the ImageNet challenge in 2012. The size of deep models is increasing ever since, which brings new challe…
Scaling Deep Networks with the Mesh Adaptive Direct Search algorithm
Dounia Lakhmiri, Mahdi Zolnouri, Vahid Partovi Nia +2
Deep neural networks are getting larger. Their implementation on edge and IoT devices becomes more challenging and moved the community to design lighter versions with similar perfo…
On the Convergence of Stochastic Gradient Descent in Low-precision Number Formats
Matteo Cacciola, Antonio Frangioni, Masoud Asgharian +2
Deep learning models are dominating almost all artificial intelligence tasks such as vision, text, and speech processing. Stochastic Gradient Descent (SGD) is the main tool for tra…
EuclidNets: An Alternative Operation for Efficient Inference of Deep Learning Models
Xinlin Li, Mariana Parazeres, Adam Oberman +3
With the advent of deep learning application on edge devices, researchers actively try to optimize their deployments on low-power and restricted memory devices. There are establish…
Training Integer-Only Deep Recurrent Neural Networks
Vahid Partovi Nia, Eyyüb Sari, Vanessa Courville +1
Recurrent neural networks (RNN) are the backbone of many text and speech applications. These architectures are typically made up of several computationally complex components such…
KronA: Parameter Efficient Tuning with Kronecker Adapter
Ali Edalati, Marzieh Tahaei, Ivan Kobyzev +3
Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task has been a well-known paradigm in Natural Language Processing. However, with the ever-growing size of P…
SeKron: A Decomposition Method Supporting Many Factorization Structures
Marawan Gamal Abdel Hameed, Ali Mosleh, Marzieh S. Tahaei +1
While convolutional neural networks (CNNs) have become the de facto standard for most image processing and computer vision applications, their deployment on edge devices remains ch…
DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization
Xinlin Li, Bang Liu, Rui Heng Yang +3
Efficiently deploying deep neural networks on low-resource edge devices is challenging due to their ever-increasing resource requirements. To address this issue, researchers have p…
Is Integer Arithmetic Enough for Deep Learning Training?
Alireza Ghaffari, Marzieh S. Tahaei, Mohammadreza Tayaranian +2
The ever-increasing computational complexity of deep learning models makes their training and deployment difficult on various cloud and edge platforms. Replacing floating-point ari…
Rethinking Pareto Frontier for Performance Evaluation of Deep Neural Networks
Vahid Partovi Nia, Alireza Ghaffari, Mahdi Zolnouri +1
Performance optimization of deep learning models is conducted either manually or through automatic architecture search, or a combination of both. On the other hand, their performan…
Demystifying and Generalizing BinaryConnect
Tim Dockhorn, Yaoliang Yu, Eyyüb Sari +2
BinaryConnect (BC) and its many variations have become the de facto standard for neural network quantization. However, our understanding of the inner workings of BC is still quite…
Kronecker Decomposition for GPT Compression
Ali Edalati, Marzieh Tahaei, Ahmad Rashid +3
GPT is an auto-regressive Transformer-based pre-trained language model which has attracted a lot of attention in the natural language processing (NLP) domain due to its state-of-th…
KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
Marzieh S. Tahaei, Ella Charlaix, Vahid Partovi Nia +2
The development of over-parameterized pre-trained language models has made a significant contribution toward the success of natural language processing. While over-parameterization…
Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition
Marawan Gamal Abdel Hameed, Marzieh S. Tahaei, Ali Mosleh +1
Modern Convolutional Neural Network (CNN) architectures, despite their superiority in solving various problems, are generally too large to be deployed on resource constrained edge…
iRNN: Integer-only Recurrent Neural Network
Eyyüb Sari, Vanessa Courville, Vahid Partovi Nia
Recurrent neural networks (RNN) are used in many real-world text and speech applications. They include complex modules such as recurrence, exponential-based activation, gate intera…
: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks
Xinlin Li, Bang Liu, Yaoliang Yu +3
Shift neural networks reduce computation complexity by removing expensive multiplication operations and quantizing continuous weights into low-bit discrete values, which are fast a…
A Twin Neural Model for Uplift
Mouloud Belbahri, Olivier Gandouet, Alejandro Murua +1
Uplift is a particular case of conditional treatment effect modeling. Such models deal with cause-and-effect inference for a specific factor, such as a marketing intervention or a…
Tensor train decompositions on recurrent networks
Alejandro Murua, Ramchalam Ramakrishnan, Xinlin Li +2
Recurrent neural networks (RNN) such as long-short-term memory (LSTM) networks are essential in a multitude of daily live tasks such as speech, language, video, and multimodal lear…
A Causal Direction Test for Heterogeneous Populations
Vahid Partovi Nia, Xinlin Li, Masoud Asgharian +3
A probabilistic expert system emulates the decision-making ability of a human expert through a directional graphical model. The first step in building such systems is to understand…
Batch Normalization in Quantized Networks
Eyyüb Sari, Vahid Partovi Nia
Implementation of quantized neural networks on computing hardware leads to considerable speed up and memory saving. However, quantized deep networks are difficult to train and batc…
Importance of Data Loading Pipeline in Training Deep Neural Networks
Mahdi Zolnouri, Xinlin Li, Vahid Partovi Nia
Training large-scale deep neural networks is a long, time-consuming operation, often requiring many GPUs to accelerate. In large models, the time spent loading data takes a signifi…
Qini-based Uplift Regression
Mouloud Belbahri, Alejandro Murua, Olivier Gandouet +1
Uplift models provide a solution to the problem of isolating the marketing effect of a campaign. For customer churn reduction, uplift models are used to identify the customers who…
Random Bias Initialization Improves Quantized Training
Xinlin Li, Vahid Partovi Nia
Binary neural networks improve computationally efficiency of deep models with a large margin. However, there is still a performance gap between a successful full-precision training…
Adaptive Binary-Ternary Quantization
Ryan Razani, Grégoire Morin, Vahid Partovi Nia +1
Neural network models are resource hungry. It is difficult to deploy such deep networks on devices with limited resources, like smart wearables, cellphones, drones, and autonomous…