Publications (42)
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
Yilong Zhao, Chien-Yu Lin, Kan Zhu +7
The growing demand for Large Language Models (LLMs) in applications such as content generation, intelligent chatbots, and sentiment analysis poses considerable challenges for LLM s…
Pure Tensor Program Rewriting via Access Patterns (Representation Pearl)
Gus Henry Smith, Andrew Liu, Steven Lyubomirsky +5
Tensor kernels in machine learning (ML) often correspond to pure mathematical expressions, making term rewriting an attractive strategy for optimization and mapping to specialized…
VibeTensor: System Software for Deep Learning, Fully Generated by AI Agents
Bing Xu, Terry Chen, Fengzhe Zhou +12
VIBETENSOR is an open-source research system software stack for deep learning, generated by LLM-powered coding agents under high-level human guidance. In this paper, "fully generat…
FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems
Shanli Xing, Yiyan Zhai, Alexander Jiang +10
Recent advances show that large language models (LLMs) can act as autonomous agents capable of generating GPU kernels, but integrating these AI-generated kernels into real-world in…
TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
Tianqi Chen, Thierry Moreau, Ziheng Jiang +9
There is an increasing need to bring machine learning to a wide diversity of hardware devices. Current frameworks rely on vendor-specific operator libraries and optimize for a narr…
Punica: Multi-Tenant LoRA Serving
Lequn Chen, Zihao Ye, Yongji Wu +3
Low-rank adaptation (LoRA) has become an important and popular method to adapt pre-trained models to specific domains. We present Punica, a system to serve multiple LoRA models in…
Accelerating SpMM Kernel with Cache-First Edge Sampling for Graph Neural Networks
Chien-Yu Lin, Liang Luo, Luis Ceze
Graph neural networks (GNNs), an emerging deep learning model class, can extract meaningful representations from highly expressive graph-structured data and are therefore gaining p…
Arch2030: A Vision of Computer Architecture Research over the Next 15 Years
Luis Ceze, Mark D. Hill, Thomas F. Wenisch
Application trends, device technologies and the architecture of systems drive progress in information technologies. However, the former engines of such progress - Moore's Law and D…
SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits
Edward Lin, Sahil Modi, Siva Kumar Sastry Hari +30
As agentic AI systems become increasingly capable of generating and optimizing GPU kernels, progress is constrained by benchmarks that reward speedup over software baselines rather…
Computer Security Risks of Distant Relative Matching in Consumer Genetic Databases
Peter M. Ney, Luis Ceze, Tadayoshi Kohno
Consumer genetic testing has become immensely popular in recent years and has lead to the creation of large scale genetic databases containing millions of dense autosomal genotype…
Srifty: Swift and Thrifty Distributed Training on the Cloud
Liang Luo, Peter West, Arvind Krishnamurthy +1
Finding the best VM configuration is key to achieve lower cost and higher throughput, two primary concerns in cloud-based distributed neural network (NN) training today. Optimal VM…
A Hardware-Software Blueprint for Flexible Deep Learning Specialization
Thierry Moreau, Tianqi Chen, Luis Vega +8
Specialized Deep Learning (DL) acceleration stacks, designed for a specific set of frameworks, model architectures, operators, and data types, offer the allure of high performance…
Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
Liang Luo, Jacob Nelson, Luis Ceze +2
Distributed deep neural network (DDNN) training constitutes an increasingly important workload that frequently runs in the cloud. Larger DNN models and faster compute engines are s…
MATIC: Learning Around Errors for Efficient Low-Voltage Neural Network Accelerators
Sung Kim, Patrick Howe, Thierry Moreau +3
As a result of the increasing demand for deep neural network (DNN)-based services, efforts to develop dedicated hardware accelerators for DNNs are growing rapidly. However,while ac…
SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning
Zihao Ye, Ruihang Lai, Junru Shao +2
Sparse tensors are rapidly becoming critical components of modern deep learning workloads. However, developing high-performance sparse operators can be difficult and tedious, and e…
Correlation Manipulating Circuits for Stochastic Computing
Vincent T. Lee, Armin Alaghi, Luis Ceze
Stochastic computing (SC) is an emerging computing technique that promises high density, low power, and error tolerant solutions. In SC, values are encoded as unary bitstreams and…
Enumerating Hardware-Software Splits with Program Rewriting
Gus Smith, Zachary Tatlock, Luis Ceze
A core problem in hardware-software codesign is in the sheer size of the design space. Without a set ISA to constrain the hardware-software interface, the design space explodes. Th…
VSS: A Storage System for Video Analytics [Technical Report]
Brandon Haynes, Maureen Daum, Dong He +4
We present a new video storage system (VSS) designed to decouple high-level video operations from the low-level details required to store and efficiently retrieve video data. VSS i…
Vignette: Perceptual Compression for Video Storage and Processing Systems
Amrita Mazumdar, Brandon Haynes, Magdalena Balazinska +3
Compressed videos constitute 70% of Internet traffic, and video upload growth rates far outpace compute and storage improvement trends. Past work in leveraging perceptual cues like…
vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs
Size Zheng, Renze Chen, Meng Li +3
IoT devices based on microcontroller units (MCU) provide ultra-low power consumption and ubiquitous computation for near-sensor deep learning models (DNN). However, the memory of M…
TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval
Chien-Yu Lin, Keisuke Kamahori, Yiyu Liu +11
Retrieval-augmented generation (RAG) extends large language models (LLMs) with external data sources to enhance factual correctness and domain coverage. Modern RAG pipelines rely o…
Learning to Optimize Tensor Programs
Tianqi Chen, Lianmin Zheng, Eddie Yan +5
We introduce a learning-based framework to optimize tensor programs for deep learning workloads. Efficient implementations of tensor operators, such as matrix multiplication and hi…
xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin +8
Long-context Large Language Models (LLMs) enable powerful applications but incur high memory costs due to the key-value states (KV-Cache). Recent studies attempt to share KV-Cache…
Characterizing and Taming Resolution in Convolutional Neural Networks
Eddie Yan, Liang Luo, Luis Ceze
Image resolution has a significant effect on the accuracy and computational, storage, and bandwidth costs of computer vision model inference. These costs are exacerbated when scali…
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
Zihao Ye, Lequn Chen, Ruihang Lai +8
Transformers, driven by attention mechanisms, form the foundation of large language models (LLMs). As these models scale up, efficient GPU attention kernels become essential for hi…
Stochastic Synthesis for Stochastic Computing
Vincent T. Lee, Armin Alaghi, Luis Ceze +1
Stochastic computing (SC) is an emerging computing technique which offers higher computational density, and lower power over binary-encoded (BE) computation. Unlike BE computation,…
The Impact of Memory Models on Software Reliability in Multiprocessors
Alexander Jaffe, Thomas Moscibroda, Laura Effinger-Dean +2
The memory consistency model is a fundamental system property characterizing a multiprocessor. The relative merits of strict versus relaxed memory models have been widely debated i…
Automating Generation of Low Precision Deep Learning Operators
Meghan Cowan, Thierry Moreau, Tianqi Chen +1
State of the art deep learning models have made steady progress in the fields of computer vision and natural language processing, at the expense of growing model sizes and computat…
21st Century Computer Architecture
Mark D. Hill, Sarita Adve, Luis Ceze +7
Because most technology and computer architecture innovations were (intentionally) invisible to higher layers, application and other software developers could reap the benefits of…
Palu: Compressing KV-Cache with Low-Rank Projection
Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin +7
Post-training KV-Cache compression methods typically either sample a subset of effectual tokens or quantize the data into lower numerical bit width. However, these methods cannot e…
Making data center computations fast, but not so furious
Daniel Porto, João Loff, Rui Duarte +2
We propose an aggressive computational sprinting variant for data center environments. While most of previous work on computational sprinting focuses on maximizing the sprinting pr…
AVO: Agentic Variation Operators for Autonomous Evolutionary Search
Terry Chen, Zhifan Ye, Bing Xu +20
Agentic Variation Operators (AVO) are a new family of evolutionary variation operators that replace the fixed mutation, crossover, and hand-designed heuristics of classical evoluti…
Cloud Collectives: Towards Cloud-aware Collectives forML Workloads with Rank Reordering
Liang Luo, Jacob Nelson, Arvind Krishnamurthy +1
ML workloads are becoming increasingly popular in the cloud. Good cloud training performance is contingent on efficient parameter exchange among VMs. We find that Collectives, the…
Automated Backend-Aware Post-Training Quantization
Ziheng Jiang, Animesh Jain, Andrew Liu +4
Quantization is a key technique to reduce the resource requirement and improve the performance of neural network deployment. However, different hardware backends such as x86 CPU, N…
Application-Driven Near-Data Processing for Similarity Search
Vincent T. Lee, Amrita Mazumdar, Carlo C. del Mundo +3
Similarity search is a key to a variety of applications including content-based search for images and video, recommendation systems, data deduplication, natural language processing…
Democratizing Design for Future Computing Platforms
Luis Ceze, Mark D. Hill, Karthikeyan Sankaralingam +1
Information and communications technology can continue to change our world. These advances will partially depend upon designs that synergistically combine software with specialized…
Parameter Box: High Performance Parameter Servers for Efficient Distributed Deep Neural Network Training
Liang Luo, Jacob Nelson, Luis Ceze +2
Most work in the deep learning systems community has focused on faster inference, but arriving at a trained model requires lengthy experiments. Accelerating training lets developer…
Energy-Efficient Hybrid Stochastic-Binary Neural Networks for Near-Sensor Computing
Vincent T. Lee, Armin Alaghi, John P. Hayes +2
Recent advances in neural networks (NNs) exhibit unprecedented success at transforming large, unstructured data streams into compact higher-level semantic information for tasks suc…
Similarity Search on Automata Processors
Vincent T. Lee, Justin Kotalik, Carlo C. Del Mundo +3
Similarity search is a critical primitive for a wide variety of applications including natural language processing, content-based search, machine learning, computer vision, databas…
Synthesizing Number Generators for Stochastic Computing using Mixed Integer Programming
Vincent T. Lee, Samuel Archibald Elliot, Armin Alaghi +1
Stochastic computing (SC) is a high density, low-power computation technique which encodes values as unary bitstreams instead of binary-encoded (BE) values. Practical SC implementa…
Exploring Computation-Communication Tradeoffs in Camera Systems
Amrita Mazumdar, Thierry Moreau, Sung Kim +5
Cameras are the defacto sensor. The growing demand for real-time and low-power computer vision, coupled with trends towards high-efficiency heterogeneous systems, has given rise to…
SAP: an Architecture for Selectively Approximate Wireless Communication
Benjamin Ransford, Luis Ceze
Integrity checking is ubiquitous in data networks, but not all network traffic needs integrity protection. Many applications can tolerate slightly damaged data while still working…