57 citations · 57 across the 8 of their papers we have counts for
10 papers
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
Ali Janati, Kaoutar El Maghraoui, Andrei Kanavalau +1
Muon-trained modular-arithmetic transformers can lose accuracy while retaining linearly decodable task information. Adjacent swaps localize five captured unnormalized failures to A…
Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai +4
Sparse Mixture-of-Experts (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input. While this reduces computation, the…
Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees
Mohammed Nowaz Rabbani Chowdhury, Hsinyu Tsai, Geoffrey W. Burr +3
Sparse Mixture-of-Experts (MoE) models enable efficient scalability by activating only a small sub-set of experts per input, yet their massive parameter counts lead to substantial…
Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systems
Zehao Fan, Zhenyu Liu, Yunzhen Liu +4
Mixture-of-Experts (MoE) models scale large language models through conditional computation, but inference becomes memory-bound once expert weights exceed the capacity of GPU memor…
SparseST: Exploiting Data Sparsity in Spatiotemporal Modeling and Prediction
Junfeng Wu, Hadjer Benmeziane, Kaoutar El Maghraoui +2
Spatiotemporal data mining (STDM) has a wide range of applications in various complex physical systems (CPS), i.e., transportation, manufacturing, healthcare, etc. Among all the pr…
Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
Thomas Joshi, Herman Saini, Neil Dhillon +2
Large Language Models (LLMs) encounter severe memory inefficiencies during long-context inference due to conventional handling of key-value (KV) caches. In this work, we introduce…
EvidenceMoE: A Physics-Guided Mixture-of-Experts with Evidential Critics for Advancing Fluorescence Light Detection and Ranging in Scattering Media
Ismail Erbas, Ferhat Demirkiran, Karthik Swaminathan +6
Fluorescence LiDAR (FLiDAR), a Light Detection and Ranging (LiDAR) technology employed for distance and depth estimation across medical, automotive, and other fields, encounters si…
Analog Foundation Models
Julian Büchel, Iason Chalas, Giovanni Acampa +7
Analog in-memory computing (AIMC) is a promising compute paradigm to improve speed and power efficiency of neural network inference beyond the limits of conventional von Neumann-ba…
A Comprehensive Survey on Hardware-Aware Neural Architecture Search
Hadjer Benmeziane, Kaoutar El Maghraoui, Hamza Ouarnoughi +3
Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process…
Dependability in a Multi-tenant Multi-framework Deep Learning as-a-Service Platform
Scott Boag, Parijat Dube, Kaoutar El Maghraoui +7
Deep learning (DL), a form of machine learning, is becoming increasingly popular in several application domains. As a result, cloud-based Deep Learning as a Service (DLaaS) platfor…