5 citations · 14 across the 15 of their papers we have counts for
15 papers
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
Runlin Shi, Bojian Yin, Guoqi Li
Hybrid architectures combining Full Attention (FA) and Linear Attention (LA) are increasingly prominent, yet their allocation remains heuristic. We seek an evidence-grounded basis…
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Siyu Ding, Mingchuan Ma, Jiabo Tong +3
Recent NVFP4 pretraining work has primarily optimized Transformer linear projections, leaving persistent optimizer states, optimizer computation, and low-precision attention forwar…
NeuroCogMap Reveals Cognitive Organization of Large Language Models
Zhongxiang Sun, Haolang Lu, Qiang Ma +11
Understanding how complex cognitive functions are organized within artificial systems is central to interpreting large language models (LLMs) and relating them to biological cognit…
Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modeling
Bojian Yin, Shurong Wang, Haoyu Tan +3
Real-world sequential signals, such as audio or video, contain critical information that is often embedded within long periods of silence or noise. While recurrent neural networks…
PretrainZero: Reinforcement Active Pretraining
Xingrun Xing, Zhiyuan Fan, Jie Lou +3
Mimicking human behavior to actively learning from general experience and achieve artificial general intelligence has always been a human dream. Recent reinforcement learning (RL)…
Breaking the Modality Wall: Time-step Mixup for Efficient Spiking Knowledge Transfer from Static to Event Domain
Yuqi Xie, Shuhan Ye, Yi Yu +7
The integration of event cameras and spiking neural networks (SNNs) promises energy-efficient visual intelligence, yet scarce event data and the sparsity of DVS outputs hinder effe…
Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
Weigao Sun, Jiaxi Hu, Yucheng Zhou +12
Large Language Models (LLMs) have delivered impressive results in language understanding, generation, reasoning, and pushes the ability boundary of multimodal models. Transformer m…
SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and Complexity
Shihao Zou, Qingfeng Li, Wei Ji +4
Spiking Neural Networks (SNNs) have shown competitive performance to Artificial Neural Networks (ANNs) in various vision tasks, while offering superior energy efficiency. However,…
Autoregressive Image Generation with Randomized Parallel Decoding
Haopeng Li, Jinyue Yang, Guoqi Li +1
We introduce ARPG, a novel visual Autoregressive model that enables Randomized Parallel Generation, addressing the inherent limitations of conventional raster-order approaches, whi…
EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models
Xingrun Xing, Zheng Liu, Shitao Xiao +6
Modern large language models (LLMs) driven by scaling laws, achieve intelligence emergency in large model sizes. Recently, the increasing concerns about cloud costs, latency, and p…
Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directions
Yangfan Hu, Qian Zheng, Guoqi Li +2
Deep learning has revolutionized artificial intelligence (AI), achieving remarkable progress in fields such as computer vision, speech recognition, and natural language processing.…
Topology Optimization of Random Memristors for Input-Aware Dynamic SNN
Bo Wang, Shaocong Wang, Ning Lin +12
There is unprecedented development in machine learning, exemplified by recent large language models and world simulators, which are artificial neural networks running on digital co…
Integer-Valued Training and Spike-Driven Inference Spiking Neural Network for High-performance and Energy-efficient Object Detection
Xinhao Luo, Man Yao, Yuhong Chou +2
Brain-inspired Spiking Neural Networks (SNNs) have bio-plausibility and low-power advantages over Artificial Neural Networks (ANNs). Applications of SNNs are currently limited to s…
SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
Xingrun Xing, Boyan Gao, Zheng Zhang +5
Recent advancements in large language models (LLMs) with billions of parameters have improved performance in various applications, but their inference processes demand significant…
SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms
Xingrun Xing, Zheng Zhang, Ziyi Ni +6
Towards energy-efficient artificial intelligence similar to the human brain, the bio-inspired spiking neural networks (SNNs) have advantages of biological plausibility, event-drive…