Flexible Rule-Based Decomposition and Metadata Independence in Modin: A Parallel Dataframe System

Dataframes have become universally popular as a means to represent data in various stages of structure, and manipulate it using a rich set of operators-thereby becoming an essential tool in the data scientists' toolbox. However, dataframe systems. such as pandas. scale poorly-and are non-interactive on moderate to large datasets. We discuss our experiences developing MODIN, our first cut at a parallel dataframe system, which already has users across several industries and over 1M downloads. MODIN translates pandas functions into a core set of operators that are individually parallelized via columnar, row-wise, or cell-wise decomposition rules that we formalize in this paper. We also introduce metadata independence to allow metadata-such as order and type-to be decoupled from the physical representation and maintained lazily. Using rule-based decomposition and metadata independence, along with careful engineering, MODIN is able to support pandas operations across both rows and columns on very large dataframes-unlike Koalas and Dask DataFrames that either break down or are unable to support such operations, while also being much faster than pandas.