Skip to content

PREPROCESSING_GUIDE.md

Latest commit

 

History

History
125 lines (94 loc) · 3.88 KB

PREPROCESSING_GUIDE.md

File metadata and controls

125 lines (94 loc) · 3.88 KB
Raw

Data Preprocessing Guide

What We Did to the Data

AEFI Data (covid_aefi_data.csv)

1. DROPPED COLUMNS

  • CASE_ID - Just an ID, doesn't help with patterns
  • VACCINATION_DATE - Converted to numeric first, then dropped
  • NOTES - Text description, can't use in ML
  • APPROVED_DATE - Not useful for pattern detection

2. CREATED NEW COLUMNS

  • days_since_start - Converted vaccination date to days (0, 1, 2, 3...) so model can use it

3. ENCODED CATEGORICAL COLUMNS (Text → Numbers)

VACCINE (3 types) → 2 columns:

  • Original: COVAXIN, COVISHIELD, SPUTNIK_V
  • New: VACCINE_COVISHIELD (0 or 1), VACCINE_SPUTNIK_V (0 or 1)
  • Note: COVAXIN is baseline (both = 0)

GENDER (2 types) → 1 column:

  • Original: M, F
  • New: GENDER_M (1 = Male, 0 = Female)

DIAGNOSIS (14 types) → 13 columns:

  • Original: ALLERGIC_REACTION, ANAPHYLAXIS, BELL_PALSY, FATIGUE, FEVER, etc.
  • New: DIAGNOSIS_ANAPHYLAXIS, DIAGNOSIS_BELL_PALSY, DIAGNOSIS_FATIGUE, etc.
  • Each = 0 or 1 (1 means patient had that diagnosis)
  • ALLERGIC_REACTION is baseline

OUTCOME (5 types) → 4 columns:

  • Original: DEATH, HOSPITALIZED_RECOVERED, ONGOING_TREATMENT, RECOVERED_AT_HOME, RECOVERED_WITH_SEQUELAE
  • New: OUTCOME_HOSPITALIZED_RECOVERED, OUTCOME_ONGOING_TREATMENT, etc.
  • Each = 0 or 1
  • DEATH is baseline (all = 0 means death)

STATE (15 states) → 14 columns:

  • Original: AndhraPradesh, Bihar, Delhi, Gujarat, Haryana, etc.
  • New: STATE_Bihar, STATE_Delhi, STATE_Gujarat, etc.
  • Each = 0 or 1 (1 means event happened in that state)
  • AndhraPradesh is baseline

CLASSIFICATION (5 types) → 4 columns:

  • Original: A1, A4, B1, B2, C
  • New: CLASSIFICATION_A4, CLASSIFICATION_B1, CLASSIFICATION_B2, CLASSIFICATION_C
  • Each = 0 or 1
  • A1 is baseline

HOSPITALIZED (2 types) → 1 column:

  • Original: YES, NO
  • New: HOSPITALIZED_YES (1 = Yes, 0 = No)

4. KEPT AS-IS (Already Numeric)

  • DOSE_NUMBER - 1, 2, or 3
  • AGE - Patient age
  • SYMPTOM_ONSET_DAYS - Days after vaccination
  • DAYS_HOSPITALIZED - Days in hospital

RESULT:

  • Original: 600 rows × 15 columns
  • After Preprocessing: 600 rows × 44 columns
  • After Scaling: Same size, but values normalized (mean=0, std=1)

Trust Score Data (covid_trust_scores.csv)

1. DROPPED COLUMNS

  • severity - Text label (VERY_SAFE, etc.), derived from trust_score already

2. ENCODED CATEGORICAL COLUMNS

vaccine (3 types) → 2 columns:

  • Original: COVAXIN, COVISHIELD, SPUTNIK_V
  • New: vaccine_COVISHIELD, vaccine_SPUTNIK_V
  • COVAXIN is baseline

3. KEPT AS-IS (Already Numeric)

  • trust_score
  • total_aefi_reported
  • causal_cases
  • deaths
  • hospitalized
  • total_doses_administered
  • death_rate_per_100k
  • causal_rate_per_100k
  • safe_percentage

RESULT:

  • Original: 3 rows × 11 columns
  • After Preprocessing: 3 rows × 11 columns
  • After Scaling: Same size, values normalized

Files Created

  1. covid_aefi_processed.csv - Encoded features, not scaled (good for reading)
  2. covid_trust_scores_processed.csv - Encoded features, not scaled
  3. covid_aefi_scaled.npy - Scaled data ready for ML model (numpy array)
  4. covid_trust_scores_scaled.npy - Scaled data ready for ML model

Why One-Hot Encoding?

When we have text categories like "COVAXIN", "COVISHIELD", "SPUTNIK_V", we can't directly use them in ML models.

One-Hot Encoding converts each category into binary columns:

  • COVAXIN → [0, 0] (both new columns = 0)
  • COVISHIELD → [1, 0] (first column = 1)
  • SPUTNIK_V → [0, 1] (second column = 1)

This way, the model can mathematically work with the data!

Next Steps for Your Friend

  1. Load the scaled data: np.load('covid_aefi_scaled.npy')
  2. Split using data_split() function
  3. Train Isolation Forest on training data
  4. Predict anomaly scores on test data
  5. Higher anomaly scores = suspicious patterns = lower trust!