Tearing Down the Memory Wall

Abstract

We present a vision for the Erudite architecture that redefines the compute and memory abstractions such that memory bandwidth and capacity become first-class citizens along with compute throughput. In this architecture, we envision coupling a high-density, massively parallel memory technology like Flash with programmable near-data accelerators, like the streaming multiprocessors in modern GPUs. Each accelerator has a local pool of storage-class memory that it can access at high throughput by initiating very large numbers of overlapping requests that help to tolerate long access latency. The accelerators can also communicate with each other and remote memory through a high-throughput low-latency interconnect. As a result, systems based on the Erudite architecture scale compute and memory bandwidth at the same rate, tearing down the notorious memory wall that has plagued computer architecture for generations. In this paper, we present the motivation, rationale, design, benefit, and research challenges for Erudite.

SRC Techcon 2020 paper. Discusses vision of GPU-Centric architecture, Erudite

References in corpus (7)

• Deep Residual Learning for Image Recognition
• Language Models are Few-Shot Learners
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• MOLIERE: Automatic Biomedical Hypothesis Generation System
• EMOGI: Efficient Memory-access for Out-of-memory Graph-traversal In GPUs

Cited by in corpus (2)

• Bandwidth-Effective DRAM Cache for GPUs with Storage-Class Memory
• LLAMA: The Low-Level Abstraction For Memory Access