98 citations · 103 across the 13 of their papers we have counts for
13 papers
RiverONE: Generating Knowledge-Intensive VLM by Simulated Quantum Machines
Xindian Ma, Xinyu Long, Yefei Zhang +11
Quantum computing provides a powerful paradigm for representing and transforming high-dimensional information through superposition, entanglement, and measurement-induced nonlinear…
SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
Haoqian Meng, Yilun Luo, Yafei Zhao +4
Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging…
ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
Xindian Ma, Rundong Kong, Peng Zhang +2
LoRA has become a universal Parameter-Efficient Fine-Tuning (PEFT) technique that equips Large Language Models (LLMs) to adapt quickly to new tasks. However, when these models are…
Hierarchical Adaptive Eviction for KV Cache Management in Multimodal Language Models
Xindian Ma, Yidi Lu, Peng Zhang +1
The integration of visual information into Large Language Models (LLMs) has enabled Multimodal LLMs (MLLMs), but the quadratic memory and computational costs of Transformer archite…
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
Haoqian Meng, Yilun Luo, Yafei Zhao +3
The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference. However, it is difficult to adapt existing…
OMEGA: Optimized Multimodal Position Encoding Index Derivation with Global Adaptive Scaling for Vision-Language Models
Ruoxiang Huang, Xindian Ma, Rundong Kong +2
Vision-Language Models (VLMs) have demonstrated strong performance across various multimodal tasks, where position encoding plays a vital role in modeling both the sequential struc…
MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models
Wenyuan Liu, Haoqian Meng, Yilun Luo +3
Quantization significantly accelerates inference in large language models (LLMs) by replacing original high-precision matrices with low-precision counterparts. Recent advances in w…
SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression
Jing Zhang, Shuzhen Sun, Peng Zhang +5
Transformer-based large language models exhibit groundbreaking capabilities, but their storage and computational costs are prohibitively high, limiting their application in resourc…
CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression
Wenyuan Liu, Xindian Ma, Peng Zhang +1
Post-Training Quantization (PTQ) is an effective technique for compressing Large Language Models (LLMs). While many studies focus on quantizing both weights and activations, it is…
3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding
Xindian Ma, Wenyuan Liu, Peng Zhang +1
Inspired by the Bloch Sphere representation, we propose a novel rotary position encoding on a three-dimensional sphere, named 3D Rotary Position Encoding (3D-RPE). 3D-RPE is an adv…
TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language Modeling
Shuai Zhang, Peng Zhang, Xindian Ma +3
Transformer has been widely-used in many Natural Language Processing (NLP) tasks and the scaled dot-product attention between tokens is a core module of Transformer. This attention…
A Tensorized Transformer for Language Modeling
Xindian Ma, Peng Zhang, Shuai Zhang +4
Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, h…
A Generalized Language Model in Tensor Space
Lipeng Zhang, Peng Zhang, Xindian Ma +3
In the literature, tensors have been effectively used for capturing the context information in language models. However, the existing methods usually adopt relatively-low order ten…