Case 1
Healthcare Data Quality
Healthcare Electronic Medical Records (EMR)
Electronic Medical Records often have 20-40% missing values. A hospital discovered that patients with missing blood pressure data had 3x higher mortality — the missingness itself was informative.
- Missing data types: MCAR, MAR, MNAR — each requires different handling strategies.
- Simple deletion can introduce severe bias — only safe for MCAR data under 5%.
- Multiple imputation and KNN imputation are preferred for clinical data.
- Domain knowledge is critical: a missing lab result may indicate the test was not ordered because the patient was too unstable.
Focus Question:
Why is simply deleting rows with missing values problematic in healthcare data?
A. It takes too long to process
B. It can introduce bias and loss of informative patterns
C. It reduces the file size too much
D. It changes the data types
Tip: The pattern of missing data is often more informative than the missing values themselves. In healthcare, missing lab tests can indicate disease severity.
Case 2
Anomaly Detection in Financial Data
Credit Card Fraud Detection
Visa processes 150 billion transactions annually. Their system analyzes 500+ attributes per transaction, flagging anomalies. False positive rates dropped from 50% to under 2% after advanced preprocessing.
- Class imbalance: 99.9% legitimate vs 0.1% fraud — accuracy is misleading.
- Feature engineering: Velocity (transaction frequency), location deviation, amount anomalies.
- SMOTE (Synthetic Minority Over-sampling Technique): Addresses class imbalance by generating synthetic fraud samples.
- Real-time constraint: Preprocessing must complete in under 100ms per transaction.
Focus Question:
What preprocessing challenge is most critical in fraud detection?
A. Missing values in transaction records
B. Class imbalance — fraud cases are extremely rare
C. Duplicate records in the database
D. Wrong data types in amount fields
Tip: When 99.9% of data is "normal," accuracy is misleading — use precision, recall, and F1-score to evaluate fraud detection models.
Case 3
Combining Heterogeneous Data Sources
US Census Data Integration
The US Census Bureau integrates surveys, administrative records, and satellite imagery. Schema integration challenges include matching "First Name" with "Given Name" across 330 million records.
- Schema integration: Maps attributes from different databases to a unified schema (e.g., "DOB" = "Date of Birth" = "Birth Date").
- Entity resolution: Identifies the same person across multiple sources using probabilistic matching.
- Record linkage: Uses algorithms to link records that refer to the same entity without a unique identifier.
- Scale: Must handle hundreds of millions of records with high accuracy.
Focus Question:
What is matching different attribute names across databases called?
A. Data cleaning
B. Schema integration
C. Feature extraction
D. Normalization
Tip: Schema integration aligns structure (column names, types); entity resolution identifies the same real-world entity across sources.