114 citations · 413 across the 46 of their papers we have counts for
21 papers · 1 filter
Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8B
Rahul Chowdhury, Timothy A Rupprecht, Senhao Cao +5
Recent work has shown that large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction. While LLMs display in-co…
Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality
Zhenglun Kong, Yize Li, Fanhu Zeng +7
In Transformer architectures, tokens\textemdash discrete units derived from raw data\textemdash are formed by segmenting inputs into fixed-length chunks. Each token is then mapped…
Structured Agent Distillation for Large Language Model
Jun Liu, Zhenglun Kong, Peiyan Dong +10
Large language models (LLMs) exhibit strong capabilities as decision-making agents by interleaving reasoning and actions, as seen in ReAct-style frameworks. Yet, their practical de…
Squat: Quant Small Language Models on the Edge
Xuan Shen, Peiyan Dong, Zhenglun Kong +9
A growing trend has emerged in designing high-quality Small Language Models (SLMs) with a few million parameters. This trend is driven by the increasing concerns over cloud costs,…
Detection and Recovery Against Deep Neural Network Fault Injection Attacks Based on Contrastive Learning
Chenan Wang, Pu Zhao, Siyue Wang +1
Deep Neural Network (DNN) models when implemented on executing devices as the inference engines are susceptible to Fault Injection Attacks (FIAs) that manipulate model parameters t…
Less is More: Data Pruning for Faster Adversarial Training
Yize Li, Pu Zhao, Xue Lin +2
Deep neural networks (DNNs) are sensitive to adversarial examples, resulting in fragile and unreliable performance in the real world. Although adversarial training (AT) is currentl…
All-in-One: A Highly Representative DNN Pruning Framework for Edge Devices with Dynamic Power Management
Yifan Gong, Zheng Zhan, Pu Zhao +6
During the deployment of deep neural networks (DNNs) on edge devices, many research efforts are devoted to the limited hardware resource. However, little attention is paid to the i…
Advancing Model Pruning via Bi-level Optimization
Yihua Zhang, Yuguang Yao, Parikshit Ram +5
The deployment constraints in practical applications necessitate the pruning of large-scale deep learning models, i.e., promoting their weight sparsity. As illustrated by the Lotte…
Efficient Multi-Prize Lottery Tickets: Enhanced Accuracy, Training, and Inference Speed
Hao Cheng, Pu Zhao, Yize Li +4
Recently, Diffenderfer and Kailkhura proposed a new paradigm for learning compact yet highly accurate binary neural networks simply by pruning and quantizing randomly weighted full…
Automatic Mapping of the Best-Suited DNN Pruning Schemes for Real-Time Mobile Acceleration
Yifan Gong, Geng Yuan, Zheng Zhan +9
Weight pruning is an effective model compression technique to tackle the challenges of achieving real-time deep neural network (DNN) inference on mobile devices. However, prior pru…
A Compression-Compilation Framework for On-mobile Real-time BERT Applications
Wei Niu, Zhenglun Kong, Geng Yuan +7
Transformer-based deep learning models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. In this paper, we propose a compression-compila…
High-Robustness, Low-Transferability Fingerprinting of Neural Networks
Siyue Wang, Xiao Wang, Pin-Yu Chen +2
This paper proposes Characteristic Examples for effectively fingerprinting deep neural networks, featuring high-robustness to the base model against model pruning as well as low-tr…
NPAS: A Compiler-aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Acceleration
Zhengang Li, Geng Yuan, Wei Niu +13
With the increasing demand to efficiently deploy DNNs on mobile edge devices, it becomes much more important to reduce unnecessary computation and increase the execution speed. Pri…
Learning to Generate Image Source-Agnostic Universal Adversarial Perturbations
Pu Zhao, Parikshit Ram, Songtao Lu +4
Adversarial perturbations are critical for certifying the robustness of deep learning models. A universal adversarial perturbation (UAP) can simultaneously attack multiple images,…
Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
Pu Zhao, Pin-Yu Chen, Payel Das +2
Mode connectivity provides novel geometric insights on analyzing loss landscapes and enables building high-accuracy pathways between well-trained neural networks. In this work, we…
Towards Real-Time DNN Inference on Mobile Platforms with Model Pruning and Compiler Optimization
Wei Niu, Pu Zhao, Zheng Zhan +3
High-end mobile platforms rapidly serve as primary computing devices for a wide range of Deep Neural Network (DNN) applications. However, the constrained computation and storage re…
BLK-REW: A Unified Block-based DNN Pruning Framework using Reweighted Regularization Method
Xiaolong Ma, Zhengang Li, Yifan Gong +8
Accelerating DNN execution on various resource-limited computing platforms has been a long-standing problem. Prior works utilize l1-based group lasso or dynamic regularization such…
Towards Query-Efficient Black-Box Adversary with Zeroth-Order Natural Gradient Descent
Pu Zhao, Pin-Yu Chen, Siyue Wang +1
Despite the great achievements of the modern deep neural networks (DNNs), the vulnerability/robustness of state-of-the-art DNNs raises security concerns in many application domains…
Fault Sneaking Attack: a Stealthy Framework for Misleading Deep Neural Networks
Pu Zhao, Siyue Wang, Cheng Gongye +3
Despite the great achievements of deep neural networks (DNNs), the vulnerability of state-of-the-art DNNs raises security concerns of DNNs in many application domains requiring hig…
Structured Adversarial Attack: Towards General Implementation and Better Interpretability
Kaidi Xu, Sijia Liu, Pu Zhao +6
When generating adversarial examples to attack deep neural networks (DNNs), Lp norm of the added perturbation is usually used to measure the similarity between original image and a…
An ADMM-Based Universal Framework for Adversarial Attacks on Deep Neural Networks
Pu Zhao, Sijia Liu, Yanzhi Wang +1
Deep neural networks (DNNs) are known vulnerable to adversarial attacks. That is, adversarial examples, obtained by adding delicately crafted distortions onto original legal inputs…