4 papers
STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization
Marco Federici, Riccardo Del Chiaro, Boris van Breugel +2
Quantization is the key method for reducing inference latency, power and memory footprint of generative AI models. However, accuracy often degrades sharply when activations are qua…
HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations
Marco Federici, Riccardo Del Chiaro, Boris van Breugel +2
Diffusion models represent the cutting edge in image generation, but their high memory and computational demands hinder deployment on resource-constrained devices. Post-Training Qu…
Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking
Marco Federici, Davide Belli, Mart van Baalen +5
While mobile devices provide ever more compute power, improvements in DRAM bandwidth are much slower. This is unfortunate for large language model (LLM) token generation, which is…
Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference
Andrii Skliar, Ties van Rozendaal, Romain Lepert +5
Mixture of Experts (MoE) LLMs have recently gained attention for their ability to enhance performance by selectively engaging specialized subnetworks or "experts" for each input. H…