Data Science |
Data
Science It
is the field of examining raw data obtained from various sources and obtaining
the information and knowledge (insights) necessary to make business or
practical decisions using mathematical, numerical, and computational methods.
Main fields Sum of 3:
- Computer Science and Programming (Computer
Science & Programming):Python, SQL, Data Structures.
- Mathematics and Statistics (Mathematics
& Statistics):Linear Algebra, Probability, Descriptive Statistics.
- Knowledge of the scope (Domain Knowledge):The
ability to identify a problem in a business, health, or other field.
Main types
of data (Types of Data)
Data
encountered in data science can be divided into two main types:
I. By Statistical Nature:
Quantitative data (Quantitative / Numerical):
Discrete
(countable): Discrete data are countable, discrete values. These are
usually whole numbers and there cannot be any decimal places between two
values.
Main feature:Obtained
by counting (Counted).
Examples:
The number of children in a class (e.g.25, 30 — cannot have 25.5 children).
Number of vehicles in a household (e.g.1, 2, 3).
Continuous
(measurable):Continuous data is data that can take any value within a
certain range. These can be measured precisely as decimal or fractional
numbers.
Main feature:Obtained
by measuring using measuring instruments (Measured).
Examples:
A person's height (e.g.165.4 cm).
Body weight (eg.62.75 kg).
The temperature in the classroom (e.g.26.8°C).
The time it takes to complete a race (e.g. seconds)10.45).
Qualitative data (Qualitative / Categorical):
Nominal
(unordered): Nominal data is data that exists as names or categories,
without any order or ranking.
Main feature:You cannot
set an order between categories as higher/lower or first/second.
Examples:
Blood types (A, B, AB, O)
The color of a vehicle (red),
blue, black)
Nationality (Sri Lankan),
Indian, Japanese)
Male / Female (Gender)
Ordinal
(with an order): Ordinal data is data that exists in categories with a
clear order or ranking.
Main feature:Even if
there is a sequence, the difference between those values cannot be measured
precisely.
Examples:
Examination pass (A, B, C,
S, F)
Places in a competition (1st
place, 2nd place, 3rd place)
Satisfaction (very good),
good, moderate, weak)
Clothing sizes (Small,
Medium, Large, XL)
II. By Organizational
Structure:
Structured Data:
Data is organized into tables with rows and columns according to a pre-defined
model.
Main feature:Easily
stored in Relational Databases (RDBMS), search and analysis are very easy.
Technical uses:SQL
Databases (MySQL, PostgreSQL), Excel Sheets.
Examples:
A student information table (ID,
Name, Age, Marks).
Bank account details and transaction records (Transaction history).
Semi-Structured
Data: Although not fully tabulated, data contains tags or organizational
markers to distinguish and identify the data.
Main feature:Although
there is no specific table structure, there is a hierarchy or schema of the
data.
Technical uses:XML,
JSON, CSV, NoSQL Databases (MongoDB).
Examples:
JSON File:{"Student": {"Name": "Amal",
"City": "Colombo"}}
Email:Although the Subject, Sender, Date, etc. of an email are structured, the
Body section is a free text.
Unstructured
Data: Data that does not have any pre-defined format or organizational
pattern. More than 80% of the world's data falls into this category.
Main feature:Cannot be
stored in traditional databases; requires Machine Learning or AI technology to
analyze.
Technical uses:Data
Lakes, NoSQL Stores.
Examples:
Photos (Images), Videos and
Audio files.
PDF documents, Text Documents and Social Media
Posts (Facebook, Twitter).
Data Science Lifecycle
Key steps
in conducting a data science project6:
- Identifying the problem (Business
Understanding):Clearly state the problem or issue that needs to be
resolved.
- Data collection (Data Collection):Obtaining
data through Databases, APIs, and Web Scraping.
- Data cleaning (Data Cleaning &
Preprocessing):Creating missing values, removing incorrect data.
- Data exploration (Exploratory Data
Analysis (EDA):Identifying patterns in data through graphs and
numerical values.
- Modeling (Machine Learning Modeling):Training
an algorithm to predict the future or classify.
- Presentation of results (Deployment &
Communication):Presenting final conclusions through a dashboard or
report.
Tools and
software needed to learn from scratch
Programming Language:
Python (the easiest and most widely used language to learn).
Data
Libraries:
NumPy - For
mathematical and matrix calculations.
Pandas - for
manipulating DataFrames and tabular data.
Matplotlib and
Seaborn - Data Visualization (to draw charts).
Database:SQL
(to query data from Relational Databases).
Environment:
Jupyter Notebook, Google Colab or VS Code.
#Data Science lifecycle
# Quantitative vs Qualitative Data
#Discrete and Continuous Data
#Nominal and Ordinal Data
#Structured, Semi-Structured, Unstructured Data
0 Comments