11-02, 11:40–12:20 (America/New_York), Radio City (Room 6604)
Polars is an OLAP query engine that focusses on the DataFrame use case. Machines have changed a lot in the last decade and Polars is a query engine that is written from scratch in Rust to benefit from the modern hardware.
Effective parallelism, cache efficient data structures and algorithms are ingrained in its design. Thanks to those efforts Polars is among the fastest single node OSS query engines out there. Another goal of polars is rethinking the way DataFrame's should be interacted with. Polars comes with a very declarative and versatile API that enables users to write readable. This talk will focus on how Polars can be used and what you gain from using it idiomatically.
DataFrames are a data structure that naturally fit the way many users want to interact with data.
Polars is a new DataFrame library that has combined DataFrames with query engines. Polars inherits decades of research in databases and query engines and comes with an optimizer and a very fast vectorized, parallel query engine.
This means that users can write their queries idiomatically and the optimizer will take care of improving the performance.
Polars doesn't primarily runs fast due to optimizations, Polars is also a parallel vectorized query engine that runs your queries on all available cores without any serialization overhead. Another benefit is that Polars engine is able to process queries Out-Of-Core, meaning it can process datasets that don't fit into RAM.
Polars is completely written in Rust and has been exposed as an API to many front ends. Namely Python, Rust, NodeJS, SQL, R, (and Scala and PHP are in the works).
Python and Rust are the main focus of our team and we've seen great adoption in both. For instance; the pypi downloads have reached over 2 million downloads/ month.
Polars Github: https://github.com/pola-rs/polars
Polars PyPI Download Statistics: https://pypistats.org/packages/polars
No previous knowledge expected
Ritchie Vink is the Author of the Polars DataFrame library. Originally he has a background in Civil Engineering, but he soon made the switch to Data/Software development. He has worked as a Machine Learning Engineer and a Software Engineer for 5 years, before he spent all of his time to Polars project. Those years have been filled with side projects to feed his curiosity. In present times he is the CEO of the newly started Polars Inc.