It's a big advantage if you already have knowledge of Python and SQL! With knowledge of Lists, Dictionaries, Loops and Functions in Python, NumPy and Pandas can be learned very quickly with guidance.
It will be very easy for you to understand how SQL concepts like SELECT,
WHERE, GROUP BY, JOIN are implemented in Python with Pandas.
Let's learn Data
Manipulation with NumPy and Pandas.
Data Manipulation with NumPy (Numerical Python)
NumPy is used to quickly calculate large numerical data and manipulate
Matrices/Arrays. NumPy Arrays use less memory than standard Python Lists and
perform calculations much faster.
1. Creating
Arrays
import numpy as np
data = np.array([10, 20, 30, 40, 50])
print(data)
matrix = np.array([
[1, 2, 3],
[4, 5, 6]
])
print(matrix)
1. Vectorized
Operations (Simultaneous Calculations Without Loops)
While
adding 10 to
each value in a Python List requires a for loop, in NumPy you can add directly:
# increase all values by 10
data_plus_10 = data + 10
print(data_plus_10) # Output: [20 30 40 50 60]
# Multiply all values by 2
data_doubled = data * 2
print(data_doubled) # Output: [20 40 60 80 100]
2. Basic
Statistics
numbers = np.array([12, 45, 67, 23, 89, 34])
print("Mean:", np.mean(numbers))
print("Median:", np.median(numbers))
print("Max:", np.max(numbers))
print("Min:", np.min(numbers))
print("Std Dev:", np.std(numbers))
3. Conditional
Filtering (Boolean Masking)
# Filtering only values greater than 50
filtered_data = numbers[numbers > 50]
print(filtered_data) # Output: [67 89]
Tabular Data Processing using Pandas
Pandas is used to manipulate Tabular Data (data
with rows and columns, such as SQL Tables or Excel Sheets).
Pandas has 2 main parts:
Series: A
single column (1D Data).
DataFrame: An entire table (2D Data - Rows & Columns).
1. Creating
a Data Frame
import pandas as pd
# Creating a DataFrame from a
Dictionary
dataset = {
'Name': ['Amal', 'Nimal', 'Kamal', 'Sunil'],
'Age': [24, 30, 22, 35],
'City': ['Colombo', 'Kandy', 'Galle', 'Colombo'],
'Salary': [60000, 85000, 45000, 95000]
}
df = pd.DataFrame(dataset)
print(df)
output
2. Inspecting
Data
print(df.head(2)) # Display the
first 2 rows
print(df.info()) # View data types
and non-null counts
print(df.describe()) # View summary statistics for numeric columns
1. Pandas
vs SQL (Pandas will help you learn familiar SQL concepts)
|
Task |
SQL Query |
Pandas Code |
|
Selecting the
Columns |
SELECT Name, Salary
FROM df; |
df[['Name',
'Salary']] |
|
Row Filtering |
SELECT * FROM df
WHERE Age > 25; |
df[df['Age'] >
25] |
|
Sorting |
SELECT * FROM df
ORDER BY Salary DESC; |
df.sort_values(by='Salary',
ascending=False) |
|
Grouping &
Aggregation |
SELECT City,
AVG(Salary) FROM df GROUP BY City; |
df.groupby('City')['Salary'].mean() |
|
Finding missing
values |
SELECT * FROM df
WHERE Salary IS NULL; |
df[df['Salary'].isna()] |
1. Representation
and use of CSV files
# Reading a CSV file
df = pd.read_csv('data.csv')
# Saving the cleaned data to
another CSV file
df.to_csv('cleaned_data.csv', index=False)
--------------------------------------------------------------------------------
Exercise:
Question:
I have a NumPy Array containing the marks of 8 students as follows:
marks = np.array([45, 78, 32, 90, 65, 54, 22, 81])
Take only the marks of the students who scored more than 50 (Pass) into
a separate Array.
Find the number of students who scored less than 35 (Fail).
Answer
import numpy as np
marks = np.array([45, 78, 32, 90, 65, 54, 22, 81])
# Filtering values above 50 marks
passed_marks = marks[marks >= 50]
print("Pass Marks:",
passed_marks)
# Number of students with scores below 35 (sum() counts True values)
failed_count = np.sum(marks < 35)
print("Number of failed students:", failed_count)
#python data manipulation,
#numpy vs pandas, data cleaning with pandas,
# exploratory data analysis python,
#pandas dataframe tutorial,
#numpy array operations,
#data science for beginners
previous link:
https://e-learnict.blogspot.com/2026/08/data-science-101-complete-beginners.html
0 Comments