Lesson 6: Handling Missing Values – Detection & Strategies

1. Why Missing Values Matter

Missing data (NaN) is common in real datasets — sensors fail, surveys skip questions.

ML models usually cannot handle NaN → must fix before training.

Common causes: data entry errors, non-response, sensor failure.

2. Detecting Missing Values

df.isnull().sum()          # count NaN per column
df.isna().sum()             # same as isnull()
df.notnull().sum()          # non-missing count
df[df["age"].isnull()]      # rows with missing age

Visual: import seaborn as sns; sns.heatmap(df.isnull())

3. Strategies to Handle Missing Data

• Drop: df.dropna() — remove rows/columns with NaN
• Fill: df.fillna(value) — e.g. 0, mean, median
• Forward/back fill: df.fillna(method="ffill")
• Interpolate: df.interpolate() — linear guess
• Impute with model: advanced (e.g. KNNImputer)

Rule: never just fill with 0 without thinking — can distort data.