6 papers
More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding
Noam Elata, Itay Lamprecht, Mikey Shechter +3
Autoregressive generation in Large Language Models (LLMs) is constrained by the memory and computational demands of attention mechanisms. Sparse attention methods mitigate this cos…
Block Sparse Flash Attention
Daniel Ohayon, Itay Lamprecht, Itay Hubara +3
Modern large language models increasingly require long contexts for reasoning and multi-document tasks, but attention's quadratic complexity creates a severe computational bottlene…
Toward Cybersecurity-Expert Small Language Models
Matan Levi, Daniel Ohayon, Ariel Blobstein +3
Large language models (LLMs) are transforming everyday applications, yet deployment in cybersecurity lags due to a lack of high-quality, domain-specific models and training dataset…
Automatic mixed precision for optimizing gained time with constrained loss mean-squared-error based on model partition to sequential sub-graphs
Shmulik Markovich-Golan, Daniel Ohayon, Itay Niv +1
Quantization is essential for Neural Network (NN) compression, reducing model size and computational demands by using lower bit-width data types, though aggressive reduction often…
Faster Inference of LLMs using FP8 on the Intel Gaudi
Joonhyung Lee, Shmulik Markovich-Golan, Daniel Ohayon +9
Low-precision data types are essential in modern neural networks during both training and inference as they enhance throughput and computational capacity by better exploiting avail…
CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions
Matan Levi, Yair Alluouche, Daniel Ohayon +1
Large Language Models (LLMs) have significantly advanced natural language processing (NLP), providing versatile capabilities across various applications. However, their application…