One of the biggest challenges we face when working with real-world Data is dealing with messy data. There is a famous saying in computer science:
"Garbage In, Garbage Out (GIGO)" — if we put in wrong or dirty data, the result we get will definitely be wrong!
In this article, we will learn step by step how to clean dirty data using the Pandas library.
70% to 80% of the time in a Data Science Project is spent not on collecting data, but on cleaning and preparing it for analysis (Data Preprocessing).
Typically, the following problems may occur in the datasets we receive:
Missing Values: Some information or values are missing (Null/NaN).
Duplicates: The same data is entered repeatedly.
Incorrect Data Types: Numbers are written as strings or in incorrect formats.
Outliers: Values that vary abnormally.
In this article, we will learn step by step how to clean dirty data using the Pandas library.
Handling Missing Values
It is common for cells in a database to have missing values. Pandas can handle this very easily.
Detecting Missing Data (isnull())
First, let's see how much data is missing from our DataFrame:
import pandas as pd
import numpy as np
# Let's create a sample database.
data = {'Student': ['Amal', 'Bimal', 'Chathuri', 'Danuka'],
'Marks': [85, np.nan, 78, 90],
'Age': [17, 18, np.nan, 17]}
df = pd.DataFrame(data)
print(df).
# View the number of missing values in each decision
print(df.isnull().sum())
Dropping vs Filling (How to decide?)
When faced with missing data, we can use two methods:
Dropping (dropna()): Completely removes rows without a value. (This is only suitable if there is a large amount of data).
Filling (fillna()): Replaces the blanks with an average value (Mean, Median) or another appropriate value.
# 1. Null Removing all rows with values
df_cleaned = df.dropna()
# 2. Filling in the blanks with the average value (Mean) in the Marks decision
mean_marks = df['Marks'].mean()
df['Marks'] = df['Marks'].fillna(mean_marks)
Removing Duplicates
Sometimes the same information may be included in the dataframe twice. This can affect our analysis results.
Checking for duplicate data: df.duplicated()
Removing duplicates: df.drop_duplicates()
# Check for duplicate rows
print(df.duplicated())
# Remove duplicate rows and create a new DataFrame
df = df.drop_duplicates()
Fixing Data Types
In some cases, data that should be in numbers (for example, phone numbers or numbers) may be recognized by the computer as Text (String). Then we cannot perform mathematical operations on it. Use astype() to correct the data type.
#convert Age determination to integer Type
df['Age'] = df['Age'].astype(int)
#checking that the data type is correct
print(df.dtypes)
Today we learned how to deal with dirty data, fill in missing values, remove duplicates, and correct data types using Python. Now we have a clean, reliable database!
Next steps: Now we will analyze this cleaned data further (Exploratory Data Analysis - EDA) to extract meaningful information from it, and look forward to our next article on how to do Data Visualization using Matplotlib and Seaborn!
⧭If you want to try out these codes in practice, try them out using Google Colab or a Jupyter Notebook.
If you have any problems, comment below!
#DataCleaning#DataPreprocessing#DataScience#PythonProgramming#Pandas#DataAnalysis#DataWrangling#MachineLearning#BigData#CodingInPython
0 Comments