Pandas Tutorial for Beginners: DataFrames, Filtering, and GroupBy

An analyst receives a 300,000-row export every Monday and spends an hour in a spreadsheet filtering, adding a formula column, and building a pivot table. The same work is about ten lines of pandas, runs in under a second, and gives the same answer every week, with no manual clicks to get wrong.

pandas is the standard Python library for working with tabular data. This pandas tutorial covers its central object, the DataFrame: a table with labeled columns, where each column holds one type of data. A single column is called a Series.

You need Python 3.11 and pandas 2.0.x. You should know basic Python, including lists and dictionaries. The dataset is small enough to check every result by hand, which is the best way to learn what each operation does.

My position: the skill in pandas is not memorizing methods. It is learning to describe what you want for a whole column, instead of what to do for each row.

Install pandas and create the data

python --version
python -m venv .venv
source .venv/bin/activate          # macOS and Linux
.venv\Scripts\Activate.ps1         # Windows PowerShell
python -m pip install "pandas>=2.0,<2.1"
python -c "import pandas; print(pandas.__version__)"

2.0.2

Save the following as orders.csv. Note that order 1006 has no quantity.

order_id,region,product,quantity,unit_price
1001,north,keyboard,2,50.0
1002,south,mouse,1,20.0
1003,north,monitor,1,200.0
1004,east,keyboard,3,50.0
1005,south,monitor,2,200.0
1006,north,mouse,,20.0
1007,east,mouse,5,20.0
1008,south,keyboard,1,50.0

Load and inspect a DataFrame

Start a Python session in the same folder, or put the lines in a script. By convention, pandas is imported as pd.

import pandas as pd

df = pd.read_csv('orders.csv')
print(df.head())

   order_id region   product  quantity  unit_price
0      1001  north  keyboard       2.0        50.0
1      1002  south     mouse       1.0        20.0
2      1003  north   monitor       1.0       200.0
3      1004   east  keyboard       3.0        50.0
4      1005  south   monitor       2.0       200.0

The unlabeled column on the left is the index, a label for each row. By default it counts from 0. head() shows the first five rows. Always inspect the shape, types, and missing values before you compute anything.

print(df.shape)
print(df.dtypes)
print(df.isna().sum())

(8, 5)
order_id        int64
region         object
product        object
quantity      float64
unit_price    float64
dtype: object
order_id      0
region        0
product       0
quantity      1
unit_price    0
dtype: int64

Two details matter here. Text columns have the dtype object. The quantity column is float64, although the file holds whole numbers. The missing value in order 1006 became NaN (“not a number”), which is a float, so pandas made the whole column float. A surprising dtype is often your first hint of dirty data.

Handle the missing value

You have two honest choices: remove the incomplete row, or fill the gap with a value you can defend. An order without a quantity cannot produce revenue, so remove it.

df = df.dropna(subset=['quantity'])
df = df.astype({'quantity': 'int64'})
print(len(df), df['quantity'].dtype)

7 int64

Most pandas methods return a new DataFrame and leave the original unchanged. That is why the code assigns the result back to df. Forgetting the assignment is a classic beginner bug: the line runs, and nothing changes.

Select columns and filter rows

Square brackets with a column name return a Series. A list of names returns a smaller DataFrame.

regions = df['region']                    # a Series
subset = df[['order_id', 'quantity']]     # a DataFrame

To filter rows, build a boolean mask: a Series of True and False values, one per row. Then use the mask to select.

is_south = df['region'] == 'south'
print(df[is_south])

   order_id region   product  quantity  unit_price
1      1002  south     mouse         1        20.0
4      1005  south   monitor         2       200.0
7      1008  south  keyboard         1        50.0

Combine conditions with & (and), | (or), and ~ (not). Each condition needs its own parentheses.

# WRONG: Python's "and" cannot combine two Series
df[df['region'] == 'south' and df['quantity'] > 1]

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

# RIGHT: element-wise operators, with parentheses
big_south = df[(df['region'] == 'south') & (df['quantity'] > 1)]
print(big_south)

   order_id region  product  quantity  unit_price
4      1005  south  monitor         2       200.0

loc versus iloc

These two accessors select rows and columns together, and people mix them up constantly.

After the dropna above, the index labels are 0 to 4, then 6 and 7, because row 5 is gone. So df.loc[5] raises KeyError, while df.iloc[5] returns the sixth remaining row. Labels and positions stop matching as soon as you filter. Use .loc with conditions for most work.

Add columns with vectorized operations

To compute revenue, multiply two columns. pandas applies the operation to every row at once, in compiled code. This is called vectorization.

df['revenue'] = df['quantity'] * df['unit_price']
print(df[['order_id', 'quantity', 'unit_price', 'revenue']])
print(df['revenue'].sum())

   order_id  quantity  unit_price  revenue
0      1001         2        50.0    100.0
1      1002         1        20.0     20.0
2      1003         1       200.0    200.0
3      1004         3        50.0    150.0
4      1005         2       200.0    400.0
6      1007         5        20.0    100.0
7      1008         1        50.0     50.0
1020.0

Check one row by hand: order 1004 is 3 times 50.0, which is 150.0. Checking a few rows against the source is a habit worth keeping on real data.

The myth: pandas is slow

You will hear that pandas is slow. What is slow is using pandas like a list of rows. This is the single most common performance mistake:

# WRONG: a Python loop over rows
revenue = []
for _, row in df.iterrows():
    revenue.append(row['quantity'] * row['unit_price'])
df['revenue'] = revenue

# RIGHT: one vectorized expression
df['revenue'] = df['quantity'] * df['unit_price']

Both produce the same column. iterrows() builds a new Series object for every row and runs the arithmetic in the interpreter. The vectorized form runs one loop in compiled code. You can measure the gap with my illustrative script, which uses 100,000 rows:

import time

import pandas as pd


def main() -> None:
    df = pd.DataFrame({'quantity': range(100_000), 'unit_price': 2.5})

    start = time.perf_counter()
    looped = [row['quantity'] * row['unit_price'] for _, row in df.iterrows()]
    loop_time = time.perf_counter() - start

    start = time.perf_counter()
    vectorized = df['quantity'] * df['unit_price']
    vector_time = time.perf_counter() - start

    assert looped == vectorized.tolist()
    print(f'iterrows:   {loop_time:.3f}s')
    print(f'vectorized: {vector_time:.5f}s')


if __name__ == '__main__':
    main()

On my laptop, the loop takes a few seconds and the vectorized line takes about a millisecond: a difference of three orders of magnitude. Run it yourself, because the ratio is what matters. Even the list comprehensions that are idiomatic elsewhere in Python lose badly here.

My heuristic: writing for over a DataFrame is a signal to stop and look for a column operation. Arithmetic, comparisons, string methods through .str, and date parts through .dt all work on whole columns. If the code still feels slow after that, profile the code before changing more.

Summarize with groupby

groupby answers questions of the form “for each X, what is the total or average Y?”. It works in three steps: split the rows into groups, apply a calculation to each group, and combine the results.

split by region            apply sum(revenue)        combine

north: 100, 200      -->   north  300
south: 20, 400, 50   -->   south  470          -->   one row per region
east:  150, 100      -->   east   250

by_region = (
    df.groupby('region')
    .agg(orders=('order_id', 'count'), revenue=('revenue', 'sum'))
    .sort_values('revenue', ascending=False)
)
print(by_region)

        orders  revenue
region                 
south        3    470.0
north        2    300.0
east         2    250.0

The agg call uses named aggregation. Each keyword becomes an output column, defined by a pair: the input column and the function. The three revenues add up to 1,020, which matches the earlier total. That cross-check takes five seconds and catches most grouping mistakes.

Other everyday summaries:

print(df['product'].value_counts())                    # how many orders per product
print(df.groupby('product')['quantity'].mean())        # average quantity per product
print(df.groupby(['region', 'product'])['revenue'].sum())   # two grouping keys

Write it as one readable chain

Because most methods return a new DataFrame, you can chain the whole analysis into one expression that reads top to bottom. This complete script creates the CSV file and produces the report. Save it as sales.py.

from pathlib import Path

import pandas as pd

CSV_TEXT = """order_id,region,product,quantity,unit_price
1001,north,keyboard,2,50.0
1002,south,mouse,1,20.0
1003,north,monitor,1,200.0
1004,east,keyboard,3,50.0
1005,south,monitor,2,200.0
1006,north,mouse,,20.0
1007,east,mouse,5,20.0
1008,south,keyboard,1,50.0
"""


def summarize(path: Path) -> pd.DataFrame:
    return (
        pd.read_csv(path)
        .dropna(subset=['quantity'])
        .astype({'quantity': 'int64'})
        .assign(revenue=lambda frame: frame['quantity'] * frame['unit_price'])
        .groupby('region', as_index=False)
        .agg(orders=('order_id', 'count'), revenue=('revenue', 'sum'))
        .sort_values('revenue', ascending=False)
    )


def main() -> None:
    path = Path('orders.csv')
    path.write_text(CSV_TEXT, encoding='utf-8')
    summary = summarize(path)
    print(summary)
    summary.to_csv('revenue_by_region.csv', index=False)


if __name__ == '__main__':
    main()

python sales.py

  region  orders  revenue
2  south       3    470.0
1  north       2    300.0
0   east       2    250.0

assign adds a column inside a chain. The lambda receives the DataFrame as it exists at that step, after the cleaning. as_index=False keeps region as a normal column, which is friendlier for export. index=False in to_csv stops pandas from writing the row labels as an extra column.

The chain has a cost. You cannot inspect intermediate results without breaking it apart. When a chain misbehaves, comment out lines from the bottom up until the output makes sense.

The warning everyone meets: SettingWithCopyWarning

Sooner or later you will filter a DataFrame and then try to change the result.

# WRONG: is "south" a view of df or a copy? pandas cannot promise either
south = df[df['region'] == 'south']
south['revenue'] = 0

SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.

The warning means pandas cannot tell whether you meant to change df or only south, and the outcome may differ from what you expect. Say which one you mean:

# RIGHT, to change the original:
df.loc[df['region'] == 'south', 'revenue'] = 0

# RIGHT, to work on an independent copy:
south = df[df['region'] == 'south'].copy()
south['revenue'] = 0

pandas 2.0 offers an opt-in mode called copy-on-write that removes the ambiguity. With it, every result behaves as an independent copy, and the warning disappears.

pd.options.mode.copy_on_write = True

The mode is not the default in 2.0, so existing code keeps its old behavior. Turning it on for new projects is reasonable, and it makes the rule simple: to change a DataFrame, operate on that DataFrame directly.

pandas 2.0 also adds optional Arrow-backed data types, which you request with pd.read_csv(path, dtype_backend='pyarrow') after installing the pyarrow package. They handle missing values without turning integers into floats, and they store strings more compactly. They are new, so try them on a copy of your pipeline first.

How real systems use pandas

• Scheduled reports. A job reads exports or query results, cleans them, aggregates, and writes a CSV or Excel file. The script replaces a manual spreadsheet routine.
• Data cleaning before loading. Pipelines use pandas to fix types, drop duplicates, and validate ranges before inserting rows into a database.
• Exploration in notebooks. Analysts load a sample, inspect distributions, and try transformations interactively, then move the working code into a script.
• Explicit types on load. Production code passes dtype and parse_dates to read_csv, so a malformed file fails loudly instead of producing an object column.
• Aggregation pushed to the database when possible. If the data lives in SQL, teams let the database group and filter, and bring only the summary into pandas.

A mistake I have seen in production is a nightly pricing job that used iterrows() to apply a discount rule to each of roughly 400,000 rows. It ran for about twenty minutes, and people assumed that was the price of using pandas. Rewriting the rule as a boolean mask with .loc assignments brought the same step down to well under a second. Nobody had measured it, because it had always been slow.

Choosing the right pandas operation: a decision framework

1. Do you want certain rows? Build a boolean mask and use df[mask] or df.loc[mask].
2. Do you want a new value for every row? Write an expression on whole columns, or use assign.
3. Do you want one value per category? Use groupby with agg.
4. Do you want to change some rows in place? Use df.loc[mask, 'column'] = value.
5. Do you want columns from another table? Use merge on a shared key.
6. Are you about to write a loop? Look for a .str, .dt, or arithmetic method first. Use apply only when no column operation exists.

When NOT to use pandas

• Data larger than memory. pandas loads everything into RAM, and intermediate steps make copies. For files bigger than your memory, use a database, DuckDB, or Polars, or process the file in chunks.
• Simple row-by-row file processing. If you only need to read a CSV and write some rows elsewhere, the standard library’s csv module does it with no dependency and constant memory.
• Inside a request handler for a few values. Importing pandas and building a DataFrame to add three numbers adds startup time and overhead. Use plain Python.

Common mistakes

• Looping over rows. iterrows() and row-wise apply run Python code per row. Jobs take minutes instead of milliseconds.
• Forgetting to assign the result. df.dropna() alone returns a new DataFrame and discards it. The original still has the missing rows.
• Using and and or on Series. Python’s keywords cannot compare element by element and raise ValueError. Use & and | with parentheses.
• Chained assignment on a filtered result. Changing a slice may or may not affect the original. Use .loc on the original, or call .copy().
• Ignoring dtypes after loading. A number column that loaded as object sorts as text and breaks arithmetic. Check df.dtypes first.
• Confusing labels with positions. After filtering, index labels have gaps. Code that assumes df.loc[0] is the first row fails or returns the wrong one.

Key takeaways

• A DataFrame is a table of typed columns. Inspect shape, dtypes, and isna().sum() before anything else.
• Filter with boolean masks, combined using &, |, and parentheses.
• Use .loc for labels and conditions, and .iloc for positions.
• Compute on whole columns. A loop over rows is almost always the wrong tool.
• Summarize with groupby and named aggregation, then cross-check against a total.
• Chain operations for readable pipelines, and assign results back.
• Change data with df.loc[mask, column] = value to avoid SettingWithCopyWarning.

FAQ

What is pandas used for in Python?

pandas is used to load, clean, transform, and summarize tabular data, such as CSV files, spreadsheets, and database query results. Its main object is the DataFrame.

What is a DataFrame in pandas?

A DataFrame is a two-dimensional table with labeled rows and columns. Each column is a Series that holds values of one data type.

What is the difference between loc and iloc in pandas?

loc selects by index labels and boolean conditions. iloc selects by integer position. They return different rows once the index is no longer a simple 0 to n sequence.

How does groupby work in pandas?

groupby splits the rows into groups by the values of one or more columns, applies a calculation such as sum or count to each group, and combines the results into a new table.

Why is my pandas code slow?

The usual cause is a Python loop over rows, such as iterrows() or row-wise apply. Replace it with operations on whole columns, which run in compiled code.

Think in columns, and check against a total

pandas rewards one change of habit: describe the result for the whole column and let the library do the looping. Add the practice of verifying a few rows and one total by hand, and you will trust your numbers. The rest of the API is variations on select, transform, and group.

Rule of thumb: if your pandas code contains a for loop over rows, there is almost certainly a one-line version that runs a thousand times faster.