Unit 2 · Data Preprocessing

2.1 Why Preprocessing Matters

Real-world data is incomplete, noisy, and inconsistent. Data preprocessing transforms raw data into a quality format suitable for mining. It is the most time-consuming step, consuming 60-80% of a data mining project's effort.

Data Quality Dimensions:

Accuracy: Values are correct and reflect reality.
Completeness: All required data is present.
Consistency: No contradictory data across sources.
Timeliness: Data is up-to-date and relevant.

Zoom: 100%

flowchart LR RAW[(📥 Raw Data)] --> CLEAN[🧹 Data Cleaning
Handle missing values
Remove noise & outliers] CLEAN --> INT[🔗 Data Integration
Merge multiple sources
Resolve conflicts] INT --> TRANS[⚙️ Data Transformation
Normalization
Aggregation
Encoding] TRANS --> RED[📉 Data Reduction
Dimensionality reduction
Feature selection] RED --> MINED[(✅ Mined Data)] style RAW fill:#ede9fe,stroke:#7c3aed,stroke-width:3px,color:#1e1b4b style MINED fill:#dcfce7,stroke:#22c55e,stroke-width:3px,color:#1e1b4b style CLEAN fill:#dbeafe,stroke:#3b82f6,stroke-width:3px,color:#1e1b4b style INT fill:#fef3c7,stroke:#f59e0b,stroke-width:3px,color:#1e1b4b style TRANS fill:#fce7f3,stroke:#ec4899,stroke-width:3px,color:#1e1b4b style RED fill:#f3e8ff,stroke:#8b5cf6,stroke-width:3px,color:#1e1b4b

2.2 Data Cleaning

Data cleaning handles missing values, noise, and inconsistencies in the dataset.

Handling Missing Values

🗑️ Delete Tuples Remove rows with missing values (works if missing data is small, <5%).

✍️ Fill Manually Use domain knowledge to fill values (time-consuming).

📊 Mean/Median/Mode Replace with statistical average (fast but may reduce variance).

📈 Regression Predict missing values using regression models.

🔍 KNN Imputation Use k-nearest neighbors to estimate missing values.

🎲 Multiple Imputation Create multiple plausible values and average results.

Missing Data Types:

MCAR (Missing Completely at Random): Missingness has no pattern — deletion is safe.
MAR (Missing at Random): Missingness depends on observed data — imputation works.
MNAR (Missing Not at Random): Missingness depends on the missing value itself — most problematic.

Noise Handling

Binning: Sort data into bins and replace with bin means/medians.
Regression: Smooth data by fitting a regression function.
Clustering: Detect and smooth outliers by grouping similar data points.

2.3 Data Integration

Data integration combines data from multiple heterogeneous sources into a unified view.

Challenges in Data Integration

📋 Schema Integration Matching attributes across databases (e.g., "First Name" vs "Given Name").

🔍 Entity Resolution Identifying the same real-world entity across sources (e.g., "J. Smith" = "John Smith").

🔄 Redundancy Detection Correlated attributes can be detected using correlation analysis.

⚠️ Data Value Conflicts Same entity may have different values in different sources (currency, units).

Correlation Analysis for Numerical Data:

Covariance: Measures how two variables change together. Positive = same direction, Negative = opposite.
Pearson Correlation Coefficient: Normalized covariance between -1 and +1. Values near ±1 indicate strong linear relationship.

Chi-Square Test for Categorical Data:

Tests whether two categorical attributes are independent. A high chi-square value indicates the attributes are dependent (redundant).

2.4 Data Transformation

Data transformation converts data into forms suitable for mining.

Zoom: 100%

flowchart TD A[⚙️ Data Transformation] --> B[📊 Normalization] A --> C[📦 Aggregation] A --> D[🔺 Generalization] A --> D2[🏷️ Encoding] B --> B1[Min-Max
v' = (v-min)/(max-min)] B --> B2[Z-Score
v' = (v-mean)/std] B --> B3[Decimal Scaling
v' = v / 10^d] C --> C1[Sum] C --> C2[Average] C --> C3[Count] D --> D1[Hierarchy
Day → Month → Year] D2 --> E1[One-Hot
Red=100, Blue=010] D2 --> E2[Label
Low=0, Med=1, High=2] style A fill:#ede9fe,stroke:#7c3aed,stroke-width:3px,color:#1e1b4b style B fill:#dbeafe,stroke:#3b82f6,stroke-width:3px,color:#1e1b4b style C fill:#fef3c7,stroke:#f59e0b,stroke-width:3px,color:#1e1b4b style D fill:#d1fae5,stroke:#22c55e,stroke-width:3px,color:#1e1b4b style D2 fill:#fce7f3,stroke:#ec4899,stroke-width:3px,color:#1e1b4b style B1 fill:#e0e7ff,stroke:#6366f1,stroke-width:2px,color:#1e1b4b style B2 fill:#e0e7ff,stroke:#6366f1,stroke-width:2px,color:#1e1b4b style B3 fill:#e0e7ff,stroke:#6366f1,stroke-width:2px,color:#1e1b4b style C1 fill:#fef9c3,stroke:#eab308,stroke-width:2px,color:#1e1b4b style C2 fill:#fef9c3,stroke:#eab308,stroke-width:2px,color:#1e1b4b style C3 fill:#fef9c3,stroke:#eab308,stroke-width:2px,color:#1e1b4b style D1 fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#1e1b4b style E1 fill:#fce7f3,stroke:#db2777,stroke-width:2px,color:#1e1b4b style E2 fill:#fce7f3,stroke:#db2777,stroke-width:2px,color:#1e1b4b

Normalization Techniques

Min-Max Normalization: Maps values to [0,1] range. Formula: v' = (v - min) / (max - min).
Z-Score Normalization: Standardizes to mean=0, std=1. Formula: v' = (v - mean) / standard_deviation.
Decimal Scaling: Normalizes by moving the decimal point. Formula: v' = v / 10^d where d is the number of digits.

Encoding Categorical Data

One-Hot Encoding: Creates binary columns for each category (Red → [1,0,0]).
Label Encoding: Assigns integer values (Low=0, Medium=1, High=2).
Binary Encoding: Uses binary representation of category indices.

2.5 Data Reduction

Data reduction reduces data volume while maintaining mining integrity.

Zoom: 100%

flowchart TD A[📉 Data Reduction] --> B[📐 Dimensionality
Reduce attributes] A --> C[🔢 Numerosity
Reduce data points] A --> D[📦 Data Compression
Encode compactly] B --> B1[PCA
Principal Component Analysis] B --> B2[Feature Selection
Select relevant features] B --> B3[Feature Extraction
Create new features] C --> C1[Sampling
Random, stratified, systematic] C --> C2[Data Cube Aggregation] D --> D1[Lossless
Original recoverable] D --> D2[Lossy
Approximate reconstruction] style A fill:#ede9fe,stroke:#7c3aed,stroke-width:3px,color:#1e1b4b style B fill:#dbeafe,stroke:#3b82f6,stroke-width:3px,color:#1e1b4b style C fill:#fef3c7,stroke:#f59e0b,stroke-width:3px,color:#1e1b4b style D fill:#fce7f3,stroke:#ec4899,stroke-width:3px,color:#1e1b4b style B1 fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px,color:#1e1b4b style B2 fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px,color:#1e1b4b style B3 fill:#f3e8ff,stroke:#8b5cf6,stroke-width:2px,color:#1e1b4b style C1 fill:#fef9c3,stroke:#eab308,stroke-width:2px,color:#1e1b4b style C2 fill:#fef9c3,stroke:#eab308,stroke-width:2px,color:#1e1b4b style D1 fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#1e1b4b style D2 fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#1e1b4b

Dimensionality Reduction

📐 PCA Projects data onto directions of maximum variance. Reduces dimensions while preserving most information.

✂️ Feature Selection Selects a subset of original features. Methods: filter (correlation), wrapper (model-based), embedded (LASSO).

🔧 Feature Extraction Creates new features from existing ones (e.g., polynomial features, interactions).

Numerosity Reduction

Sampling: Select a subset of data points. Types: random, stratified (proportional representation), systematic.
Data Cube Aggregation: Pre-compute aggregate values for multi-dimensional analysis.

Unit II · Data Preprocessing

2.1 Why Preprocessing Matters

2.2 Data Cleaning

Handling Missing Values

Noise Handling

2.3 Data Integration

Challenges in Data Integration

2.4 Data Transformation

Normalization Techniques

Encoding Categorical Data

2.5 Data Reduction

Dimensionality Reduction

Numerosity Reduction

Textbooks & Resources