Polars vs pandas: What's the Difference?

Polars and pandas both provide DataFrame-based data analysis in Python, but they differ in syntax, performance, and features. In this tutorial on Polars vs pandas, you’ll compare their method chaining styles, run timed performance tests, explore LazyFrame optimizations in Polars, convert data between the two libraries, and create plots with their built-in tools. You’ll also examine scenarios where each library’s strengths make it the better choice.

By the end of this tutorial, you’ll understand that:

Polars expressions and contexts let you build clear, optimized query pipelines without mutating your original data.
LazyFrames with query optimization in Polars can outperform pandas for grouped and aggregated workloads.
Streaming in Polars enables processing datas…

By the end of this tutorial, you’ll understand that:

Polars expressions and contexts let you build clear, optimized query pipelines without mutating your original data.
LazyFrames with query optimization in Polars can outperform pandas for grouped and aggregated workloads.
Streaming in Polars enables processing datasets that don’t fit in memory, which pandas can’t handle natively.
.to_pandas() and from_pandas() let you convert between DataFrame formats, and Narwhals offers a library-agnostic API.
Built-in plotting uses Altair for Polars and Matplotlib for pandas, allowing quick visualization directly from DataFrames.

To get the most out of this tutorial, it’s recommended that you already have a basic understanding of how to work with both pandas and Polars DataFrames, as well as Polars LazyFrames.

To complete the examples in this tutorial, you’ll use various tools and the Python REPL. You’ll use the command line to run some scripts that time your code and reveal how pandas and Polars compare. You’ll also take advantage of the plotting capabilities of Jupyter Notebook.

Much of the data you’ll use will be random and self-generated. You’ll also use a cleansed and reformatted Apache Parquet version of some freely available retail data from the UC Irvine Machine Learning Repository. Parquet files are optimized to store data and analyze it efficiently. This enables you to achieve optimal performance from the pandas and Polars libraries.

Before you start, you should download the online_retail.parquet file from the tutorial downloadables and place it into your project directory.

You’ll need to install the pandas and Polars libraries, as well as PyArrow, Matplotlib, Vega-Altair, and Narwhals, to make sure your code has everything it needs to run. You’ll also use NumPy, which is currently installed automatically when you install pandas.

You may also want to consider creating your own virtual environment within your project folder to install the necessary libraries. This will prevent them from interfering with your current setup.

You can install the required libraries using these commands at your command prompt:

All the code examples are provided in the downloadable materials for this tutorial, which you can download by clicking the link below:

Now that you’re set up, it’s time to get started and learn about the main differences between Polars and pandas.

** Take the Quiz:** Test your knowledge with our interactive “Polars vs pandas: What’s the Difference?” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Polars vs pandas: What’s the Difference?

Take this quiz to test your knowledge of the Polars vs pandas tutorial and review the key differences between these open-source Python libraries.

Do Polars and pandas Use the Same Syntax?

There are similarities between Polars and pandas. For example, they both support Series and DataFrames and can perform many of the same data analysis computations. However, there are some differences in their syntax.

To explore this, you’ll use the order details in your online_retail.parquet file to analyze both pandas and Polars DataFrames. This file contains the following data:

Column Name	Description
InvoiceNo	Invoice number
StockCode	Stock code of item
Description	Item description
Quantity	Quantity purchased
InvoiceDate	Date invoiced
UnitPrice	Item price
CustomerID	Customer identifier
Country	Country of purchase made

Next, you’ll analyze some of this data with pandas and then with Polars.

Using Index-Based Syntax in pandas

Suppose you want a DataFrame with a new Total column that contains the total cost of each purchase. You also want to apply filtering so you can concentrate on specific data.

To achieve this, you might write the following pandas code in your REPL:

This code uses pandas index-based syntax, inspired by NumPy, on which pandas was originally built. First, you add a new Total column to your DataFrame. The column is calculated by multiplying the values of the Quantity and UnitPrice columns together. This operation permanently changes your original DataFrame.

Next, you index the DataFrame using a list containing the four columns you want to see. You use the square-bracket notation ([]) in pandas for indexing, which is standard across the Python ecosystem. You then use the square brackets again, but this time to apply a filter so that only rows with totals greater than $100 are displayed. Finally, for brevity, you use .head(3) to restrict the output to the first three records.

This approach to writing pandas code mutates the DataFrame during processing. Once you’ve added the Total column, it becomes a permanent part of your DataFrame. If you make several such changes, your code can become difficult to follow, and the changes to the DataFrame become difficult to track. Also, your starting point for further processing is the updated DataFrame, which you may not always want.

Additionally, you can’t merge your code into a single operation since the Total column wouldn’t be recognized when you select it for display or filtering. That’s why you need a second operation to complete your analysis.

Using Method-Chaining Syntax in pandas

In addition to its index-based syntax, pandas also supports method chaining. This syntax allows you to specify multiple steps in a single operation and provides a clear audit trail of what happens to your DataFrame during its analysis.

You can rewrite the code above using method chaining. You need to read the data from the Parquet file and assign it to orders_pandas again:

Method chaining works because each method call, or link in the chain, returns a new DataFrame.

When you take your original orders_pandas DataFrame and apply .assign(Total=orders_pandas["Quantity"] * orders_pandas["UnitPrice"]) to it, you get a new DataFrame with a new Total column containing the product of the Quantity and UnitPrice columns. The original orders_pandas DataFrame remains unchanged.

Then, when you apply .filter() to this new DataFrame, you create an updated DataFrame containing only the four columns: InvoiceNo, Quantity, UnitPrice, and Total. This operation recognizes the Total column since you apply .filter() to the DataFrame that .assign() returns.

Finally, you use .query() to restrict the rows output to only those with values greater than one hundred in their Total column. It’s this final version of the DataFrame that contains your results. The intermediate versions created by each link in the chain have all been overwritten and the original orders_pandas hasn’t changed.

As in the previous example, you use .head(3) to restrict the output to the first three rows. You can add .head() directly to the chain of methods, but as it’s not part of the analysis, you can also add it at the end of the expression as shown above.

If you want to see more examples of method chaining in pandas, then be sure to check out the examples in Using Python for Data Analysis.

Using Method-Chaining Syntax in Polars

Polars syntax was designed with method chaining in mind. Here’s the same analysis you performed earlier, but this time using Polars. You’ll notice some differences from the pandas example shown above:

When you write Polars code, you create it using expressions and contexts. In Polars, you use an expression to represent data or use it in a calculation. For example, pl.col("Quantity") * pl.col("UnitPrice") will instruct Polars to multiply together the "Quantity" and "UnitPrice" values from each column in a DataFrame.

To use your expressions productively, you need to put them into contexts. For example, expressions within the .select() context allow you to view specific columns, while those within the .filter() context enable you to restrict your output to only what you need.

Here you use the .select() context to choose the columns you want to see and to create and display the new Total column. You then use the .filter() context to apply your filter. The result is achieved in a single operation. As with the pandas chaining code you saw earlier, the original orders_polars DataFrame never changes.

Chaining creates code that can be more readable and maintainable. While pandas’ index-based syntax may feel intuitive for programmers familiar with NumPy, method chaining is often easier for beginners to learn.

Syntax differences aside, both pandas and Polars produce the same results, so you might think they both do the same thing behind the scenes. Read on and you’ll soon realize that they don’t.

What Are the Differences in Query Execution Between Polars vs pandas?

To understand how query execution differs between Polars and pandas, you need to consider not only their DataFrames but also Polars LazyFrames. In this section, you’ll create a pandas DataFrame, a Polars DataFrame, and a Polars LazyFrame to compare their data analysis capabilities.

Creating a Data Generation Script

To compare the time performance for pandas and Polars, you’ll create a Python script that can generate different quantities of data.

Create a new file in your project folder and name it data_generation.py, then add the following code to your file:

You use the NumPy library in this code, which was installed when you installed pandas. However, if you’re using NumPy in a different Python environment, you’ll need to install it separately.

Your script produces a Python dictionary that you’ll use to populate DataFrames. When you call generate_data(), you pass it the number of rows you want your DataFrame to have. Then, when you create a DataFrame with the dictionary returned by generate_data(), it’ll contain five columns and the number of rows you specified, each filled with random data.

Timing Operations Using DataFrames and LazyFrames

To see how well pandas DataFrames perform against both Polars DataFrames and LazyFrames, you’ll time a grouping operation. In this example, you’ll measure how long it takes to work out the total sales income for each salesperson within each product, within each region, using data generated with the generate_data() function you created earlier.

To begin, you’ll write a script to time your analysis. To keep the tests fair, you’ll make sure each one runs against the same data.

Importing the Libraries

Create a new file named benchmark.py, then add the following code:

You start by importing the built-in functools and sys modules.

The functools module provides you with various higher-order functions, which are functions that allow you to pass other functions to them as parameters.

The sys module allows you to pass command-line arguments to your script, which gives you more control over how it runs.

The timeit module enables you to measure the execution time of your code to determine its speed. This allows you to compare the performance between pandas and Polars. You’ll use a Timer object to do this.

You also import the pandas and polars libraries, as well as the generate_data() function you created earlier.

Creating the Functions to Be Tested

Next, add in three functions that will create a pandas DataFrame, a Polars DataFrame, and a Polars LazyFrame:

Each function has a test_data parameter. You’ll pass the Python dictionary returned by generate_data() to these functions, and this is used to create the DataFrames and LazyFrames used in the time analysis. Each function returns the appropriate object. You also use .convert_dtypes() to ensure pandas uses the PyArrow data types instead of its default NumPy versions, since this newer backend is more performant.

Next, you create three more functions, each of which will perform the identical data analysis that you’ll time:

Each function analyzes the data by calculating total sales income for each group of salespeople, within a product and region. You call .collect() on analyze_polars_lazyframe() to materialize the LazyFrame into a DataFrame. This way, all tests are fair because each produces a physical dataset.

Adding the Time Tests

Finally, add in the code that will perform the time tests for creating the DataFrames and the LazyFrame:

To keep the results clear, you start each test with a line that tells the user what’s being timed and how many rows are in the object being timed. To pick out the row count, you use sys.argv[1], which is the value passed to the script when you run it.

To actually time the analysis, you time the three create_*() functions you wrote earlier. To do this, you create a Timer object. This will organize the timing operation.

You then need to pass the name of the function you want to time, along with any parameters it requires, to your Timer. To do this, you make use of the partial() function from the functools library. This creates a partial object containing both the function you want to call and any arguments you want to pass to it.

For example, the partial object created by functools.partial(create_pandas_dataframe, test_data) contains both the create_pandas_dataframe function reference and the test_data dictionary. This is then passed to Timer, which measures the time taken to run the function.

As a finishing touch, you use .timeit(100) to time the running of each function one hundred times and take an average time reading. This will level out any outlier results.

In your final part of the script, you create the time tests for the data analysis:

This time, you aim to time each of your three analyze_*() functions. To begin with, you create the DataFrames and LazyFrame, again using the same test data for each. Once each object has been created, you again pass it to the appropriate analyze_*() function using the same techniques as before.

Running the Script

Finally, it’s time to run your script and see what happens:

The timings you get will depend on your setup and may be different from the ones shown here. The first three results show that pandas can create its DataFrame faster than Polars.

The most striking performance differences emerge from the data analysis. The last three results show that Polars is significantly faster at running this analysis than pandas, and that the LazyFrame version of the query runs a bit faster than the Polars DataFrame version due to its optimization. Performance differences will also depend on the type of analysis you perform.

You can also confirm that the analysis returns the same results. To avoid displaying the full results, you can limit the output to the entries matching the "East" region for "Boots" sales:

You can run the code using only 500 lines this time to speed things up. For conciseness, the printouts with the performance data are omitted in the output shown below:

The total sales values for Boots in the East region across all three salespeople are the same in all three analyses. This sanity check gives confidence that the analyses are performing the same tasks on the data.

Polars can usually run its queries faster than pandas. However, if this is a one-off analysis, you may get your results faster using pandas. If you only need to create the DataFrame once and run multiple queries against it, then you might be better off with a LazyFrame in the long run.

As before, the best strategy is to run tests using the different libraries before deciding which one suits your particular use case.

Using Streaming With Polars LazyFrames

Polars can also use something called streaming with its LazyFrames to allow it to cope when your data is too large to fit into your computer’s RAM. Instead of reading all of the data at once, it reads it in multiple chunks. The pandas library doesn’t support this feature.

To investigate streaming, you’ll time test streaming against a non-streamed equivalent for the same LazyFrame. To start, create a new script named streaming_test.py that looks like this:

This script uses code similar to the previous benchmark.py script. You create a new analyze_polars_streaming() function. To request that it uses streaming, you pass engine="streaming" to .collect(). Streaming only works on a LazyFrame, so you pass polars_lf to analyze_polars_streaming() for timing.

For comparison, you also time a LazyFrame without streaming.

Then, you run your streaming_test.py script. You test this script with 5,000,000 rows this time since you’re using Polars, which is significantly faster than pandas for this analysis:

Notice that for this test with 5,000,000 rows, the streaming analysis is faster. This is because memory is being used more efficiently. For extremely large data volumes, streaming tests may be slower due to the additional time Polars requires to combine multiple batches of data. However, it may be your only option to prevent your system from hanging.

As with the earlier tests, the best option is to try it and see if it suits your use case. Additionally, certain operations don’t support streaming, so even if you request it, it may not be possible.

Can You Convert Between Polars and pandas?

It’s possible for you to convert between Polars and pandas DataFrames in either direction. You’ll use the Python REPL to investigate how this is done.

To begin with, you’ll create a Polars DataFrame:

Can You Produce Plots From Both Polars and pandas?

Python provides you with a range of plotting libraries such as seaborn and Bokeh. While you can use any of them to plot your pandas and Polars data, using the default libraries within pandas and Polars is also possible. The pandas library uses Matplotlib, while Polars uses Vega-Altair.

In this section, you’ll use either a standalone Jupyter Notebook or one within JupyterLab to define and display your plots. This is easier than using the Python REPL.

Start up a new Jupyter Notebook in your project folder to give it access to the libraries you’ve already installed, and then add the following code into a cell:

You’ll use the sales_data for your plots.

Suppose you wanted a bar plot of each region’s sales_income figures. You can do this from within Polars using the DataFrame’s .plot, from which it can pass its plotting requirements to Altair.

Create a new cell in your notebook and add the following code:

You analyze your DataFrame using the same code as before to calculate the total regional sales. Once the data has been analyzed, you use .plot.bar() to define your bar plot. You define the data to be plotted in the x and y axes, as well as the plot’s width, height, and title.

The resulting plot will look something like this:

Remember, your figures will be different because their values are randomly generated.

Next, you’ll use pandas to produce a plot using the same data. Add a new cell to your Notebook, and add in the following code:

As before, you set up the same labels and titles. The pandas version of your plot looks like this:

This plot should be similar to the previous version, but your plot will again differ from this one because of the random data used to construct them. It also looks slightly different from the previous one because you’re using a different library to produce it.

As you can see, you can quickly generate plots from within both pandas and Polars. It’s usually better to learn how to use the full features of these plotting libraries directly and feed your DataFrame columns into them. You need to use this technique if you choose to use the other plotting libraries that Python provides.

Are There Any Other Differences?

The pandas code is designed to be run on a computer’s CPU, and it does this very well. Polars also does this but can use your resources more efficiently. For example, it can use the cuDF library to delegate some of its processing to your computer’s GPU.

It’s possible to utilize parallelization in both pandas and Polars. This allows your computer to allocate different parts of the query across the other CPU cores in your system. However, to do this using pandas, you need to use a library such as Dask or Modin since pandas is single-threaded by default. Parallelization is built into Polars.

A vectorized query engine enables queries to be executed on groups of rows simultaneously, rather than processing them row by row. Both Polars and pandas support this. Currently, pandas relies on NumPy for vectorization, but starting with version 3 it will switch to PyArrow’s vectorization capabilities. Polars, by contrast, has a built-in SIMD (Single Instruction, Multiple Data) engine.

Although pandas is still the most popular of the two libraries because it’s been around for a lot longer and is more established, Polars is catching up. However, both are designed for single-node environments. If you need a solution that works in a distributed environment, then consider investigating a library such as PySpark.

Conclusion

By completing this tutorial, you’ve learned about the key differences between pandas and Polars, along with techniques to help you decide which one best fits your needs.

In this tutorial, you’ve learned that:

Polars expressions and contexts let you build clear, optimized query pipelines without mutating your original data.
LazyFrames with query optimization in Polars can outperform pandas for grouped and aggregated workloads.
Streaming in Polars enables processing datasets that don’t fit in memory, which pandas can’t handle natively.
.to_pandas() and from_pandas() let you convert between DataFrame formats, and Narwhals offers a library-agnostic API.
Built-in plotting uses Altair for Polars and Matplotlib for pandas, allowing quick visualization directly from DataFrames.

You now understand the differences between the pandas and Polars libraries. To dive deeper, why not follow the links in this tutorial to explore the related topics in more detail?

Frequently Asked Questions

Now that you have some experience with Polars vs pandas in Python, you can use the questions and answers below to check your understanding and recap what you’ve learned.

These FAQs are related to the most important concepts you’ve covered in this tutorial. Click the Show/Hide toggle beside each question to reveal the answer.

You often see Polars run analysis faster than pandas, especially when you use LazyFrames and query optimization. For quick one-off tasks, you may find pandas faster to create the DataFrame, but Polars usually wins on grouped and aggregated workloads.

Yes, you may find Polars easier because it encourages clear method chaining with expressions and contexts. If you already think in NumPy-style indexing, you might prefer pandas, but chaining in Polars tends to read more cleanly for beginners.

Yes, you can chain pandas operations with methods like .assign(), .filter(), and .query() to build readable pipelines. You create new DataFrames at each step, so you avoid mutating your original data.

You call the DataFrame method .to_pandas() to get a pandas DataFrame. For the reverse process, you use Polars from_pandas(), and for mixed backends, you can wrap your data with Narwhals.

Yes, you can use Polars LazyFrames with streaming to process data in chunks without loading everything into RAM. pandas doesn’t offer built-in streaming, so you rely on Polars for this scenario.