Handling Missing Data in Python for Real-World Applications

In the world of data, missing values are inevitable. Whether you’re working with user inputs or legacy datasets, handling missing data effectively is crucial for robust analysis. This blog covers practical strategies to handle missing data.

Why Missing Data Matters

Missing data can distort analysis, lead to inaccuracies in predictions, and even cause system failures.

Example Scenario:

• You’re analyzing customer feedback. Missing values in rating and feedback columns can skew insights and lead to incorrect conclusions.

️ Methods to Handle Missing Data

1. Identifying Missing Values

Pandas provides tools to identify missing data:

<span>import</span> <span>pandas</span> <span>as</span> <span>pd</span>
<span># Load dataset </span><span>df</span> <span>=</span> <span>pd</span><span>.</span><span>read_csv</span><span>(</span><span>'</span><span>customer_feedback.csv</span><span>'</span><span>)</span>
<span># Check for missing values </span><span>print</span><span>(</span><span>df</span><span>.</span><span>isnull</span><span>().</span><span>sum</span><span>())</span>  <span># This reveals the number of missing entries in each column. </span>
<span>import</span> <span>pandas</span> <span>as</span> <span>pd</span>

<span># Load dataset </span><span>df</span> <span>=</span> <span>pd</span><span>.</span><span>read_csv</span><span>(</span><span>'</span><span>customer_feedback.csv</span><span>'</span><span>)</span>

<span># Check for missing values </span><span>print</span><span>(</span><span>df</span><span>.</span><span>isnull</span><span>().</span><span>sum</span><span>())</span>  <span># This reveals the number of missing entries in each column. </span>
import pandas as pd

# Load dataset df = pd.read_csv('customer_feedback.csv')

# Check for missing values print(df.isnull().sum())  # This reveals the number of missing entries in each column. 

Enter fullscreen mode Exit fullscreen mode

2. Removing Missing Data

If missing values are minimal and non-critical, you can drop them:

<span># Drop rows with missing values </span><span>df_cleaned</span> <span>=</span> <span>df</span><span>.</span><span>dropna</span><span>()</span>
<span># Drop columns with missing values </span><span>df_cleaned</span> <span>=</span> <span>df</span><span>.</span><span>dropna</span><span>(</span><span>axis</span><span>=</span><span>1</span><span>)</span>
<span># Drop rows with missing values </span><span>df_cleaned</span> <span>=</span> <span>df</span><span>.</span><span>dropna</span><span>()</span>

<span># Drop columns with missing values </span><span>df_cleaned</span> <span>=</span> <span>df</span><span>.</span><span>dropna</span><span>(</span><span>axis</span><span>=</span><span>1</span><span>)</span>
# Drop rows with missing values df_cleaned = df.dropna()

# Drop columns with missing values df_cleaned = df.dropna(axis=1)

Enter fullscreen mode Exit fullscreen mode

3. Imputing Missing Values

a) Replace with Default Values

<span># Replace categorical missing values </span><span>df</span><span>[</span><span>'</span><span>Feedback</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>'</span><span>No Feedback</span><span>'</span><span>,</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
<span># Replace categorical missing values </span><span>df</span><span>[</span><span>'</span><span>Feedback</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>'</span><span>No Feedback</span><span>'</span><span>,</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
# Replace categorical missing values df['Feedback'].fillna('No Feedback', inplace=True)

Enter fullscreen mode Exit fullscreen mode

b) Use Statistical Measures

<span># Replace missing ratings with column mean </span><span>df</span><span>[</span><span>'</span><span>Rating</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>df</span><span>[</span><span>'</span><span>Rating</span><span>'</span><span>].</span><span>mean</span><span>(),</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
<span># Replace missing ratings with column mean </span><span>df</span><span>[</span><span>'</span><span>Rating</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>df</span><span>[</span><span>'</span><span>Rating</span><span>'</span><span>].</span><span>mean</span><span>(),</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
# Replace missing ratings with column mean df['Rating'].fillna(df['Rating'].mean(), inplace=True)

Enter fullscreen mode Exit fullscreen mode

c) Forward/Backward Fill

<span># Forward fill </span><span>df</span><span>[</span><span>'</span><span>Sales</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>method</span><span>=</span><span>'</span><span>ffill</span><span>'</span><span>,</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
<span># Backward fill </span><span>df</span><span>[</span><span>'</span><span>Sales</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>method</span><span>=</span><span>'</span><span>bfill</span><span>'</span><span>,</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
<span># Forward fill </span><span>df</span><span>[</span><span>'</span><span>Sales</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>method</span><span>=</span><span>'</span><span>ffill</span><span>'</span><span>,</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>

<span># Backward fill </span><span>df</span><span>[</span><span>'</span><span>Sales</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>method</span><span>=</span><span>'</span><span>bfill</span><span>'</span><span>,</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
# Forward fill df['Sales'].fillna(method='ffill', inplace=True)

# Backward fill df['Sales'].fillna(method='bfill', inplace=True)

Enter fullscreen mode Exit fullscreen mode

4. Advanced Techniques

a) Interpolation

<span># Estimate missing values using interpolation </span><span>df</span><span>[</span><span>'</span><span>Sales</span><span>'</span><span>]</span> <span>=</span> <span>df</span><span>[</span><span>'</span><span>Sales</span><span>'</span><span>].</span><span>interpolate</span><span>()</span>
<span># Estimate missing values using interpolation </span><span>df</span><span>[</span><span>'</span><span>Sales</span><span>'</span><span>]</span> <span>=</span> <span>df</span><span>[</span><span>'</span><span>Sales</span><span>'</span><span>].</span><span>interpolate</span><span>()</span>
# Estimate missing values using interpolation df['Sales'] = df['Sales'].interpolate()

Enter fullscreen mode Exit fullscreen mode

b) Machine Learning Models

<span>from</span> <span>sklearn.impute</span> <span>import</span> <span>SimpleImputer</span>
<span># Use predictive models for missing data </span><span>imputer</span> <span>=</span> <span>SimpleImputer</span><span>(</span><span>strategy</span><span>=</span><span>'</span><span>mean</span><span>'</span><span>)</span>
<span>df</span><span>[</span><span>'</span><span>Sales</span><span>'</span><span>]</span> <span>=</span> <span>imputer</span><span>.</span><span>fit_transform</span><span>(</span><span>df</span><span>[[</span><span>'</span><span>Sales</span><span>'</span><span>]])</span>
<span>from</span> <span>sklearn.impute</span> <span>import</span> <span>SimpleImputer</span>

<span># Use predictive models for missing data </span><span>imputer</span> <span>=</span> <span>SimpleImputer</span><span>(</span><span>strategy</span><span>=</span><span>'</span><span>mean</span><span>'</span><span>)</span>
<span>df</span><span>[</span><span>'</span><span>Sales</span><span>'</span><span>]</span> <span>=</span> <span>imputer</span><span>.</span><span>fit_transform</span><span>(</span><span>df</span><span>[[</span><span>'</span><span>Sales</span><span>'</span><span>]])</span>
from sklearn.impute import SimpleImputer

# Use predictive models for missing data imputer = SimpleImputer(strategy='mean')
df['Sales'] = imputer.fit_transform(df[['Sales']])

Enter fullscreen mode Exit fullscreen mode

Real-World Example

Handling missing values in an e-commerce dataset:

<span>import</span> <span>pandas</span> <span>as</span> <span>pd</span>
<span># Load dataset </span><span>df</span> <span>=</span> <span>pd</span><span>.</span><span>read_csv</span><span>(</span><span>'</span><span>ecommerce_data.csv</span><span>'</span><span>)</span>
<span># Identify missing data </span><span>print</span><span>(</span><span>"</span><span>Missing Data:</span><span>\n</span><span>"</span><span>,</span> <span>df</span><span>.</span><span>isnull</span><span>().</span><span>sum</span><span>())</span>
<span># Fill missing values </span><span>df</span><span>[</span><span>'</span><span>Product_Price</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>df</span><span>[</span><span>'</span><span>Product_Price</span><span>'</span><span>].</span><span>median</span><span>(),</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
<span>df</span><span>[</span><span>'</span><span>Product_Category</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>'</span><span>Unknown</span><span>'</span><span>,</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
<span># Drop rows with missing 'Customer_ID' </span><span>df</span><span>.</span><span>dropna</span><span>(</span><span>subset</span><span>=</span><span>[</span><span>'</span><span>Customer_ID</span><span>'</span><span>],</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
<span># Verify cleaning </span><span>print</span><span>(</span><span>"</span><span>Cleaned Data:</span><span>\n</span><span>"</span><span>,</span> <span>df</span><span>.</span><span>isnull</span><span>().</span><span>sum</span><span>())</span>
<span>import</span> <span>pandas</span> <span>as</span> <span>pd</span>

<span># Load dataset </span><span>df</span> <span>=</span> <span>pd</span><span>.</span><span>read_csv</span><span>(</span><span>'</span><span>ecommerce_data.csv</span><span>'</span><span>)</span>

<span># Identify missing data </span><span>print</span><span>(</span><span>"</span><span>Missing Data:</span><span>\n</span><span>"</span><span>,</span> <span>df</span><span>.</span><span>isnull</span><span>().</span><span>sum</span><span>())</span>

<span># Fill missing values </span><span>df</span><span>[</span><span>'</span><span>Product_Price</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>df</span><span>[</span><span>'</span><span>Product_Price</span><span>'</span><span>].</span><span>median</span><span>(),</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>
<span>df</span><span>[</span><span>'</span><span>Product_Category</span><span>'</span><span>].</span><span>fillna</span><span>(</span><span>'</span><span>Unknown</span><span>'</span><span>,</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>

<span># Drop rows with missing 'Customer_ID' </span><span>df</span><span>.</span><span>dropna</span><span>(</span><span>subset</span><span>=</span><span>[</span><span>'</span><span>Customer_ID</span><span>'</span><span>],</span> <span>inplace</span><span>=</span><span>True</span><span>)</span>

<span># Verify cleaning </span><span>print</span><span>(</span><span>"</span><span>Cleaned Data:</span><span>\n</span><span>"</span><span>,</span> <span>df</span><span>.</span><span>isnull</span><span>().</span><span>sum</span><span>())</span>
import pandas as pd

# Load dataset df = pd.read_csv('ecommerce_data.csv')

# Identify missing data print("Missing Data:\n", df.isnull().sum())

# Fill missing values df['Product_Price'].fillna(df['Product_Price'].median(), inplace=True)
df['Product_Category'].fillna('Unknown', inplace=True)

# Drop rows with missing 'Customer_ID' df.dropna(subset=['Customer_ID'], inplace=True)

# Verify cleaning print("Cleaned Data:\n", df.isnull().sum())

Enter fullscreen mode Exit fullscreen mode

Key Takeaways

• Understand the Context: Always analyze why data is missing before deciding on a method.
• Be Consistent: Use consistent strategies across datasets.
• Document Changes: Maintain transparency by documenting your methods.

Final Thoughts

Handling missing data is both an art and a science. By applying the right techniques, you can ensure clean datasets for accurate analysis and robust machine learning.

Reach me at: harrypeacock1234@gmail.com

Visit my GitHub: Harry-Ship-It
View my Fivver: https://www.fiverr.com/s/jj5lqmZ

原文链接： Handling Missing Data in Python for Real-World Applications