Publications (43)
Small Language Models in the Real World: Insights from Industrial Text Classification
Lujun Li, Lama Sleem, Niccolo' Gentile +2
With the emergence of ChatGPT, Transformer models have significantly advanced text classification and related tasks. Decoder-only models such as Llama exhibit strong performance an…
LPZero: Language Model Zero-cost Proxy Search from Zero
Peijie Dong, Lujun Li, Xiang Liu +4
In spite of the outstanding performance, Neural Architecture Search (NAS) is criticized for massive computation. Recently, Zero-shot NAS has emerged as a promising approach by expl…
Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models
Peijie Dong, Lujun Li, Zhenheng Tang +4
Despite the remarkable capabilities, Large Language Models (LLMs) face deployment challenges due to their extensive size. Pruning methods drop a subset of weights to accelerate, bu…
Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs
Binxing Xu, Hao Gu, Lujun Li +8
Training LLMs at ultra-low precision remains a formidable challenge. Direct low-bit QAT often suffers from convergence instability and substantial training costs, exacerbated by qu…
NORM: Knowledge Distillation via N-to-One Representation Matching
Xiaolong Liu, Lujun Li, Chao Li +1
Existing feature distillation methods commonly adopt the One-to-one Representation Matching between any pre-selected teacher-student layer pair. In this paper, we present N-to-One…
Progressive Meta-Pooling Learning for Lightweight Image Classification Model
Peijie Dong, Xin Niu, Zhiliang Tian +5
Practical networks for edge devices adopt shallow depth and small convolutional kernels to save memory and computational cost, which leads to a restricted receptive field. Conventi…
IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference
Xintong Yang, Hao Gu, Binxing Xu +6
Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, qui…
DisWOT: Student Architecture Search for Distillation WithOut Training
Peijie Dong, Lujun Li, Zimian Wei
Knowledge distillation (KD) is an effective training strategy to improve the lightweight student models under the guidance of cumbersome teachers. However, the large architecture d…
GP-NAS-ensemble: a model for NAS Performance Prediction
Kunlong Chen, Liu Yang, Yitian Chen +3
It is of great significance to estimate the performance of a given model architecture without training in the application of Neural Architecture Search (NAS) as it may take a lot o…
Agent Skill Framework: Perspectives on the Potential of Small to Medium Language Models in Industrial Environments
Yangjie Xu, Lujun Li, Lama Sleem +6
Agent skills are widely supported by major agentic frameworks and perform well with proprietary models, yet their effectiveness for small and medium-sized open source language mode…
Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
Lujun Li, Zhu Qiyuan, Jiacheng Wang +4
Mixture of Experts (MoE) LLMs face significant obstacles due to their massive parameter scale, which imposes memory, storage, and deployment challenges. Although recent expert merg…
Uncovering Zero-Shot Generalization Gaps in Time-Series Foundation Models Using Real-World Videos
Lujun Li, Lama Sleem, Yiqun Wang +3
Recent research on time-series foundation models (TSFMs) has underscored the scarcity of real-world data, often supplemented with synthetic sources in existing datasets, whose gene…
Delta Decompression for MoE-based LLMs Compression
Hao Gu, Wei Li, Lujun Li +5
Mixture-of-Experts (MoE) architectures in large language models (LLMs) achieve exceptional performance, but face prohibitive storage and memory requirements. To address these chall…
BTC-LLM: Efficient Sub-1-Bit LLM Quantization via Learnable Transformation and Binary Codebook
Hao Gu, Lujun Li, Hao Wang +7
Binary quantization represents the most extreme form of compression, reducing weights to +/-1 for maximal memory and computational efficiency. While recent sparsity-aware binarizat…
Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression
Te Zhang, Yuheng Li, Junxiang Wang +1
Large multimodal models (LMMs) have advanced significantly by integrating visual encoders with extensive language models, enabling robust reasoning capabilities. However, compressi…
NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models
Cheng Lin, Lujun Li, Dezhi Li +3
In this paper, we introduce Nested Low-Rank Adaptation (NoRA), a novel approach to parameter-efficient fine-tuning that extends the capabilities of Low-Rank Adaptation (LoRA) techn…
The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models
Lujun Li, Lama Sleem, Niccolo Gentile +4
Recent vision-language models (VLMs) excel at multimodal understanding and reasoning, yet their fine-grained visual perception remains underexplored. A natural extension of ``How m…
The Necessity of Setting Temperature in LLM-as-a-Judge
Lujun Li, Lama Sleem, Yangjie Xu +4
Using large language models (LLMs) as judges for evaluating model outputs has emerged as an important paradigm for automated evaluation. However, the choice of decoding temperature…
Regularized Forward-Backward Decoder for Attention Models
Tobias Watzel, Ludwig Kürzinger, Lujun Li +1
Nowadays, attention models are one of the popular candidates for speech recognition. So far, many studies mainly focus on the encoder structure or the attention module to enhance t…
You Know What I'm Saying: Jailbreak Attack via Implicit Reference
Tianyu Wu, Lingrui Mei, Ruibin Yuan +3
While recent advancements in large language model (LLM) alignment have enabled the effective identification of malicious objectives involving scene nesting and keyword rewriting, o…
Multi-trial Neural Architecture Search with Lottery Tickets
Zimian Wei, Hengyue Pan, Lujun Li +4
Neural architecture search (NAS) has brought significant progress in recent image recognition tasks. Most existing NAS methods apply restricted search spaces, which limits the uppe…
Audio Adversarial Examples for Robust Hybrid CTC/Attention Speech Recognition
Ludwig Kürzinger, Edgar Ricardo Chavez Rosas, Lujun Li +2
Recent advances in Automatic Speech Recognition (ASR) demonstrated how end-to-end systems are able to achieve state-of-the-art performance. There is a trend towards deeper neural n…
QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch
Hao Gu, Hao Wang, Jiacheng Liu +9
Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by runnin…
Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgish
Lujun Li, Yewei Song, Lama Sleem +7
Grammar refers to the system of rules that governs the structural organization and the semantic relations among linguistic units such as sentences, phrases, and words within a give…
Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
Peijie Dong, Zhenheng Tang, Xiang Liu +3
Post-training compression reduces the computational and memory costs of large language models (LLMs), enabling resource-efficient deployment. However, existing compression benchmar…
Is Small Language Model the Silver Bullet to Low-Resource Languages Machine Translation?
Yewei Song, Lujun Li, Cedric Lothritz +6
Low-resource languages (LRLs) lack sufficient linguistic resources and are underrepresented in benchmark datasets, resulting in persistently lower translation quality than high-res…
CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition
Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li +2
Recent end-to-end Automatic Speech Recognition (ASR) systems demonstrated the ability to outperform conventional hybrid DNN/ HMM ASR. Aside from architectural improvements in those…
NegBLEURT Forest: Leveraging Inconsistencies for Detecting Jailbreak Attacks
Lama Sleem, Jerome Francois, Lujun Li +3
Jailbreak attacks designed to bypass safety mechanisms pose a serious threat by prompting LLMs to generate harmful or inappropriate content, despite alignment with ethical guidelin…
EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization
Peijie Dong, Lujun Li, Zimian Wei +3
Mixed-Precision Quantization~(MQ) can achieve a competitive accuracy-complexity trade-off for models. Conventional training-based search methods require time-consuming candidate tr…
Adversarial Joint Training with Self-Attention Mechanism for Robust End-to-End Speech Recognition
Lujun Li, Yikai Kang, Yuchen Shi +3
Lately, the self-attention mechanism has marked a new milestone in the field of automatic speech recognition (ASR). Nevertheless, its performance is susceptible to environmental in…
Exploring the Impact of Temperature on Large Language Models:Hot or Cold?
Lujun Li, Lama Sleem, Niccolo' Gentile +2
The sampling temperature, a critical hyperparameter in large language models (LLMs), modifies the logits before the softmax layer, thereby reshaping the distribution of output toke…
TVT: Training-Free Vision Transformer Search on Tiny Datasets
Zimian Wei, Hengyue Pan, Lujun Li +4
Training-free Vision Transformer (ViT) architecture search is presented to search for a better ViT with zero-cost proxies. While ViTs achieve significant distillation gains from CN…
Vision Transformer-Based Time-Series Image Reconstruction for Cloud-Filling Applications
Lujun Li, Yiqun Wang, Radu State
Cloud cover in multispectral imagery (MSI) poses significant challenges for early season crop mapping, as it leads to missing or corrupted spectral information. Synthetic aperture…
RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies
Peijie Dong, Xin Niu, Lujun Li +5
Neural architecture search (NAS) has made tremendous progress in the automatic design of effective neural network structures but suffers from a heavy computational burden. One-shot…
ParZC: Parametric Zero-Cost Proxies for Efficient NAS
Peijie Dong, Lujun Li, Xinglin Pan +4
Recent advancements in Zero-shot Neural Architecture Search (NAS) highlight the efficacy of zero-cost proxies in various NAS benchmarks. Several studies propose the automated desig…
DMFormer: Closing the Gap Between CNN and Vision Transformers
Zimian Wei, Hengyue Pan, Lujun Li +4
Vision transformers have shown excellent performance in computer vision tasks. As the computation cost of their self-attention mechanism is expensive, recent works tried to replace…
Temporal-Spatial Tubelet Embedding for Cloud-Robust MSI Reconstruction using MSI-SAR Fusion: A Multi-Head Self-Attention Video Vision Transformer Approach
Yiqun Wang, Lujun Li, Meiru Yue +1
Cloud cover in multispectral imagery (MSI) significantly hinders early-season crop mapping by corrupting spectral information. Existing Vision Transformer(ViT)-based time-series re…
Catch-Up Distillation: You Only Need to Train Once for Accelerating Sampling
Shitong Shao, Xu Dai, Lujun Li +3
Diffusion Probability Models (DPMs) have made impressive advancements in various machine learning domains. However, achieving high-quality synthetic samples typically involves perf…
CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing
Yixuan Wang, Haoyu Qiao, Lujun Li +2
Large Language Models (LLMs) confront significant memory challenges due to the escalating KV cache with increasing sequence length. As a crucial technique, existing cross-layer KV…
STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
Peijie Dong, Lujun Li, Yuedong Zhong +8
In this paper, we present the first structural binarization method for LLM compression to less than 1-bit precision. Although LLMs have achieved remarkable performance, their memor…
Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery
Zimian Wei, Lujun Li, Peijie Dong +6
The substantial success of Vision Transformer (ViT) in computer vision tasks is largely attributed to the architecture design. This underscores the necessity of efficient architect…
Activation Modulation and Recalibration Scheme for Weakly Supervised Semantic Segmentation
Jie Qin, Jie Wu, Xuefeng Xiao +2
Image-level weakly supervised semantic segmentation (WSSS) is a fundamental yet challenging computer vision task facilitating scene understanding and automatic driving. Most existi…
Prior-Guided One-shot Neural Architecture Search
Peijie Dong, Xin Niu, Lujun Li +5
Neural architecture search methods seek optimal candidates with efficient weight-sharing supernet training. However, recent studies indicate poor ranking consistency about the perf…