126 citations · 236 across the 5 of their papers we have counts for
8 papers
VANDAM: Viewing a nucleotide sequence with DNA molecular priors
Jeremy Levy, Ariel Larey, Yury Nahshan +18
Contemporary Genomic Foundation Models (GFMs) rely on a DNA-as-a-string paradigm that employs masked token prediction objectives for pretraining. However, this abstraction does not…
Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
Yury Nahshan, Nati Daniel, Jacob Goldberger +1
Sparse mixture-of-experts (MoE) large language models scale model capacity by routing each token to a small subset of experts. Their routers are regularized with load balancing ter…
Linear Log-Normal Attention with Unbiased Concentration
Yury Nahshan, Joseph Kampeas, Emir Haleva
Transformer models have achieved remarkable results in a wide range of applications. However, their scalability is hampered by the quadratic time and memory complexity of the self-…
Rotation Invariant Quantization for Model Compression
Joseph Kampeas, Yury Nahshan, Hanoch Kremer +4
Post-training Neural Network (NN) model compression is an attractive approach for deploying large, memory-consuming models on devices with limited memory resources. In this study,…
Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
Itay Hubara, Yury Nahshan, Yair Hanani +2
Lately, post-training quantization methods have gained considerable attention, as they are simple to use, and require only a small unlabeled calibration set. This small dataset can…
Robust Quantization: One Model to Rule Them All
Moran Shkolnik, Brian Chmiel, Ron Banner +4
Neural network quantization methods often involve simulating the quantization process during training, making the trained model highly dependent on the target bit-width and precise…
Loss Aware Post-training Quantization
Yury Nahshan, Brian Chmiel, Chaim Baskin +4
Neural network quantization enables the deployment of large models on resource-constrained devices. Current post-training quantization methods fall short in terms of accuracy for I…
Post-training 4-bit quantization of convolution networks for rapid-deployment
Ron Banner, Yury Nahshan, Elad Hoffer +1
Convolutional neural networks require significant memory bandwidth and storage for intermediate computations, apart from substantial computing resources. Neural network quantizatio…