Data Handling using Pandas — Study Notes

1. Introduction

Data handling is the process of collecting, cleaning, transforming, and summarising data so that it can be analysed and understood. In the earlier Pandas chapters we learned to create Series and DataFrames and to apply basic operations such as sorting, grouping, and merging. Data Handling using Pandas takes that foundation and applies it to realistic data problems: inspecting a dataset, computing new columns from existing ones, removing duplicates, reshaping tables, and writing complete data-handling programs that read a file, process it, and produce a report.

The chapter is organised around the operations a working data analyst performs most often. We begin with the tools for understanding a new dataset, then move through column computation, user-defined functions, duplicate detection, and table reshaping with pivot tables. We also revisit filtering and slicing in greater depth, since precise row and column selection is the heart of practical data handling, and we end with a complete case study that ties every step together.

For the examination, this chapter is the bridge between the theory of Pandas and the practical file work that carries significant marks. Programs here combine reading a CSV, cleaning the data, computing derived columns, grouping the results, and printing a readable summary. Practising these complete workflows, rather than isolated commands, is the best preparation for both the practical examination and the case-based questions in the theory paper.

2. Inspecting a Dataset

Before any analysis, a dataset must be inspected to understand its structure, size, and quality.

import pandas as pd
df = pd.read_csv("students.csv")
print(df.info())
print(df.describe())
print(df.shape)
print(df.columns)
print(df.isnull().sum())

• df.info() prints the number of rows, column names, non-null counts, and data types.
• df.describe() summarises the numerical columns.
• df.isnull().sum() gives the count of missing values in each column.

For categorical columns, useful inspection tools are unique() and value_counts():

print(df["City"].unique())
print(df["City"].value_counts())

unique() lists the distinct values, and value_counts() counts how often each distinct value occurs.

3. Computing New Columns

New columns are created from arithmetic on existing ones, and the operations are applied to every row automatically:

import pandas as pd
df = pd.DataFrame({"Maths": [90, 85, 78], "Science": [88, 92, 80]})
df["Total"] = df["Maths"] + df["Science"]
df["Average"] = df["Total"] / 2
print(df)

The result adds a Total column holding the row-wise sums and an Average column holding the row-wise means. Conditional columns can also be created:

df["Result"] = ["Pass" if m >= 40 else "Fail" for m in df["Average"]]

4. Applying Functions with apply()

The apply() method runs a function across each column (axis=0) or each row (axis=1) of a DataFrame:

import pandas as pd
df = pd.DataFrame({"Maths": [90, 85, 78], "Science": [88, 92, 80]})
print(df.apply(sum))
print(df.apply(lambda r: r["Maths"] + r["Science"], axis=1))

• df.apply(sum) sums each column.
• df.apply(func, axis=1) applies func to every row, where the function receives the row as a Series.
• Series.apply(func) applies a function to each element of a Series.

A user-defined function can be defined first and then passed to apply:

def grade(m):
    if m >= 90:
        return "A"
    elif m >= 75:
        return "B"
    else:
        return "C"

df["Grade"] = df["Maths"].apply(grade)

5. Handling Duplicate Values

Real data frequently contains duplicate rows. The duplicated() method flags duplicates and drop_duplicates() removes them:

import pandas as pd
df = pd.DataFrame({"Roll": [1, 2, 2, 3], "Name": ["A", "B", "B", "C"]})
print(df.duplicated())
print(df.drop_duplicates())

• df.duplicated() returns a boolean Series marking each duplicated row after the first.
• df.drop_duplicates() removes the duplicate rows, keeping the first occurrence.
• subset restricts duplicate checking to chosen columns: df.drop_duplicates(subset=["Name"]).

6. Filtering Rows and Columns with loc and iloc

Precise selection uses loc for labels and iloc for positions, and both support conditional filtering:

print(df.loc[df["Marks"] > 80])
print(df.loc[df["City"] == "Delhi", ["Name", "Marks"]])
print(df.iloc[1:4, 0:2])

• df.loc[condition] returns all rows satisfying the condition.
• df.loc[rows, columns] selects rows and columns by label in one step.
• df.iloc[r1:r2, c1:c2] selects a block by integer positions, with the end excluded.

Logical combinations with & (AND), | (OR), and ~ (NOT) build complex filters:

print(df.loc[(df["Marks"] > 70) & (df["City"] == "Delhi")])

Parentheses around each condition are mandatory when combining filters.

7. Pivoting and Reshaping Data

7.1 pivot_table

pivot_table() summarises data by crossing two categorical columns and aggregating a value column:

import pandas as pd
df = pd.DataFrame({
    "City": ["Delhi", "Delhi", "Mumbai", "Mumbai"],
    "Year": [2023, 2024, 2023, 2024],
    "Sales": [100, 120, 150, 170]
})
print(df.pivot_table(values="Sales", index="City", columns="Year", aggfunc="sum"))

The result is a small table with one row per city and one column per year, each cell holding the summed sales. The default aggregation is the mean, and aggfunc can be changed to sum, count, max, min, or a list of functions.

7.2 Reshaping with stack and unstack

stack() converts a DataFrame into a Series with a multi-level index, and unstack() reverses it:

s = df.groupby(["City", "Year"])["Sales"].sum()
print(s)
print(s.unstack())

stack and unstack are used to move between the long and wide forms of data, which is important when preparing data for charts or reports.

8. Combining DataFrames in Detail

8.1 Concatenation

import pandas as pd
df1 = pd.DataFrame({"Roll": [1, 2], "Name": ["A", "B"]})
df2 = pd.DataFrame({"Roll": [3, 4], "Name": ["C", "D"]})
print(pd.concat([df1, df2], ignore_index=True))

Vertical concatenation appends rows; axis=1 appends columns instead.

8.2 Merging on a Key

marks = pd.DataFrame({"Roll": [1, 2, 3], "Marks": [88, 92, 78]})
city = pd.DataFrame({"Roll": [1, 2, 3], "City": ["Delhi", "Mumbai", "Pune"]})
merged = pd.merge(marks, city, on="Roll")
print(merged)

The how parameter controls whether the join is inner, left, right, or outer, exactly as in SQL.

9. Complete Data Handling Workflow

A typical program combines all the techniques of this chapter:

import pandas as pd

df = pd.read_csv("students.csv")
df = df.dropna()
df["Total"] = df["Maths"] + df["Science"] + df["English"]
df = df.drop_duplicates(subset=["Roll"])

summary = df.groupby("City")["Total"].agg(["count", "mean"])
print(summary)
print(df.loc[df["Total"] > 200, ["Name", "Total"]])
df.to_csv("report.csv", index=False)

The program reads the data, removes missing rows, computes a new column, drops duplicates, groups and aggregates the results, filters the top students, and exports a clean report file.

Quick Revision Tables

Table 1: Data Inspection Methods

Table 2: Reshaping and Combining Functions

Mind Map

Important Diagrams (SVG)

Diagram 1: Data Handling Pipeline

Common Mistakes

1. Using dropna() on a dataset where the empty cells are meaningful, thereby losing useful rows.
2. Forgetting that duplicated rows are flagged only after the first occurrence, so a duplicate check can miss earlier copies.
3. Writing df["Total"] = df["Maths"] + df["Science"] + df["English"] when one subject column contains NaN, because the whole row becomes NaN.
4. Combining multiple conditions without parentheses in loc, producing a confusing "truth value is ambiguous" error.
5. Using df.apply(func) without axis=1 and expecting row-wise results; the default is column-wise.
6. Assuming pivot_table averages by default; the default is the mean, but it is easy to forget to set aggfunc explicitly.
7. Confusing stack with unstack: stack moves columns into the index, while unstack does the reverse.
8. Calling pd.concat without ignore_index=True and getting repeated index labels.
9. Trying to apply arithmetic between columns of different lengths after a merge, causing alignment errors.

Exam Tips

1. Practise the complete workflow program, as a similar program is very likely to appear in the practical examination.
2. Memorise the default axis values: apply defaults to columns (axis=0), and dropna defaults to rows.
3. Remember that value_counts() is the fastest way to answer "how many times does each value occur" questions.
4. Know that combining filters requires & or | with parentheses around every condition.
5. Be ready to write a pivot_table call and to state what its default aggregation function is.
6. Learn the pair stack/unstack as the tools for converting between long and wide formats.
7. In practicals, print each intermediate result with print(df.head()) so the examiner can verify the step-by-step logic.

Conclusion

Data Handling using Pandas brings together everything needed to work with realistic datasets. Inspection methods such as info(), describe(), unique(), and value_counts() reveal what the data contains; arithmetic between columns and the apply() method create derived values; duplicated() and drop_duplicates() remove redundant rows; pivot_table(), stack(), and unstack() reshape tables; and concat() and merge() combine sources. Combined into the standard pipeline of read, clean, compute, analyse, present, and export, these operations turn raw files into meaningful reports. This toolkit directly supports the practical file work that follows, where complete programs of exactly this kind are written, executed, and documented.