14 citations · 19 across the 5 of their papers we have counts for
6 papers
Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
Yonggan Fu, Xin Dong, Shizhe Diao +12
Efficient deployment of small language models (SLMs) is essential for numerous real-world applications with stringent latency constraints. While previous work on SLM design has pri…
Hymba: A Hybrid-head Architecture for Small Language Models
Xin Dong, Yonggan Fu, Shizhe Diao +10
We propose Hymba, a family of small language models featuring a hybrid-head parallel architecture that integrates transformer attention mechanisms with state space models (SSMs) fo…
Compressing 1D Time-Channel Separable Convolutions using Sparse Random Ternary Matrices
Gonçalo Mordido, Matthijs Van Keirsbilck, Alexander Keller
We demonstrate that 1x1-convolutions in 1D time-channel separable convolutions may be replaced by constant, sparse random ternary matrices with weights in . Such layer…
Rethinking Full Connectivity in Recurrent Neural Networks
Matthijs Van Keirsbilck, Alexander Keller, Xiaodong Yang
Recurrent neural networks (RNNs) are omnipresent in sequence modeling tasks. Practical models usually consist of several layers of hundreds or thousands of neurons which are fully…
Instant Quantization of Neural Networks using Monte Carlo Methods
Gonçalo Mordido, Matthijs Van Keirsbilck, Alexander Keller
Low bit-width integer weights and activations are very important for efficient inference, especially with respect to lower power consumption. We propose Monte Carlo methods to quan…
Resource aware design of a deep convolutional-recurrent neural network for speech recognition through audio-visual sensor fusion
Matthijs Van keirsbilck, Bert Moons, Marian Verhelst
Today's Automatic Speech Recognition systems only rely on acoustic signals and often don't perform well under noisy conditions. Performing multi-modal speech recognition - processi…