Leveraging Apache Arrow for Zero-copy, Zero-serialization Cluster Shared Memory

Abstract

This paper describes a distributed implementation of Apache Arrow that can leverage cluster-shared load-store addressable memory that is hardware-coherent only within each node. The implementation is built on the ThymesisFlow prototype that leverages the OpenCAPI interface to create a shared address space across a cluster. While Apache Arrow structures are immutable, simplifying their use in a cluster shared memory, this paper creates distributed Apache Arrow tables and makes them accessible in each node.

arXiv commentPresented at the 3rd Workshop on Heterogeneous Composable and Disaggregated Systems (HCDS 2024)