Embedding Apache Arrow inside Storage Systems

Speaker: Jayjeet Chakraborty, University of California, Santa Cruz
Department of Computer Science and Engineering

With the recent advances in the I/O performance of storage and network devices, the traditional concept of a “A fast CPU and slow disk” have changed, and CPUs have become the new bottleneck. When we filter a large dataset stored in distributed storage systems like Ceph or Amazon S3, all the data is first transformed to the client, and the processing then happens on the Clients CPUs. A lot of the data is thrown out in the end and this unnecessary data transfer wastes network bandwidth and CPU cycles. Meanwhile, the storage layer CPUs sits idle and just ends up serving “dumb bytes.”

Skyhook was developed to embed the Apache Arrow Dataset API in the object-storage layer of Ceph and allows offloading simple query
operations to the storage layer, thereby removing the CPU bottleneck on the client. Skyhook makes selection and projection queries faster and more scalable. The main advantage of the design is that it follows a plugin architecture and does not require any code changes in either
Ceph or Arrow. This makes the design portable to other pairs of storage systems and data access libraries.

In this talk, we discuss how Apache Arrow can be embedded inside storage
systems to make queries faster and scale better.