Users have many options for big dataframes today, including Apache Spark, Dask DataFrame, DuckDB, and Polars. These libraries have different architectures, languages, data models, and communities, but they’re all useful in this space. Which should you choose?

The answer to this question is complex and involves application area, familiarity, personal style, thoughts on typing, and so on. But for the purposes of this talk, we’ll just consider performance, and in particular, performance on the TPC-H benchmark set, focused on database queries with heavy joins and groupbys.

We’ll run this benchmark suite across compute environments ranging from a Macbook Pro to hundreds of cores on the cloud. We’ll also explore ranges in scale from 10 GiB up to 10 TiB, and see how the different libraries respond at different scales.

No library performs perfectly across this range, and by comparing performance we’ll be able to gain insight into what to use when. We'll dive more deeply into Dask performance in order to learn generally what's important at large scale. Finally we'll also talk about deploying these projects in the cloud with Coiled.