5 citations · 14 across the 15 of their papers we have counts for
6 papers · 1 filter
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
Runlin Shi, Bojian Yin, Guoqi Li
Hybrid architectures combining Full Attention (FA) and Linear Attention (LA) are increasingly prominent, yet their allocation remains heuristic. We seek an evidence-grounded basis…
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Siyu Ding, Mingchuan Ma, Jiabo Tong +3
Recent NVFP4 pretraining work has primarily optimized Transformer linear projections, leaving persistent optimizer states, optimizer computation, and low-precision attention forwar…
Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modeling
Bojian Yin, Shurong Wang, Haoyu Tan +3
Real-world sequential signals, such as audio or video, contain critical information that is often embedded within long periods of silence or noise. While recurrent neural networks…
EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models
Xingrun Xing, Zheng Liu, Shitao Xiao +6
Modern large language models (LLMs) driven by scaling laws, achieve intelligence emergency in large model sizes. Recently, the increasing concerns about cloud costs, latency, and p…
Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directions
Yangfan Hu, Qian Zheng, Guoqi Li +2
Deep learning has revolutionized artificial intelligence (AI), achieving remarkable progress in fields such as computer vision, speech recognition, and natural language processing.…
SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
Xingrun Xing, Boyan Gao, Zheng Zhang +5
Recent advancements in large language models (LLMs) with billions of parameters have improved performance in various applications, but their inference processes demand significant…