6 papers
Predicting LLM Compression Degradation from Spectral Statistics
Mingxue Xu
Matrix-level low-rank compression is a promising way to reduce the cost of large language models, but running compression and evaluating the resulting models on language tasks can…
Reconstructing Training Data from Adapter-based Federated Large Language Models
Silong Chen, Yuchuan Luo, Guilin Deng +4
Adapter-based Federated Large Language Models (FedLLMs) are widely adopted to reduce the computational, storage, and communication overhead of full-parameter fine-tuning for web-sc…
TensorSLM: Energy-efficient Embedding Compression of Sub-billion Parameter Language Models on Low-end Devices
Mingxue Xu, Yao Lei Xu, Danilo P. Mandic
Small Language Models (SLMs, or on-device LMs) have significantly fewer parameters than Large Language Models (LLMs). They are typically deployed on low-end devices, like mobile ph…
How can representation dimension dominate structurally pruned LLMs?
Mingxue Xu, Lisa Alazraki, Danilo P. Mandic
Pruning assumes a subnetwork exists in the original deep neural network, which can achieve comparative model performance with less computation than the original. However, it is unc…
Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models
Mingxue Xu, Sadia Sharmin, Danilo P. Mandic
Matrix and tensor-guided parametrization for Natural Language Processing (NLP) models is fundamentally useful for the improvement of the model's systematic efficiency. However, the…
TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition
Mingxue Xu, Yao Lei Xu, Danilo P. Mandic
High-dimensional token embeddings underpin Large Language Models (LLMs), as they can capture subtle semantic information and significantly enhance the modelling of complex language…