DatasetDNA is an automated dataset health profiler for machine learning. It analyzes your CSV, detects real data-quality problems, separates them from statistical signals, identifies targets, calculates a 0–100 health score, and provides actionable recommendations.
🛡️ Your data stays on your machine. No cloud upload. No notebook required. No configuration required.
pip install datasetdna
datasetdna your_data.csvdatasetdna data.csv --target churn
datasetdna data.csv --htmlDatasetDNA makes a strict distinction between data quality and statistical signals.
| 🧬 Data Quality | 📊 Statistical Signals |
|---|---|
| Affects Health Score | Does not affect Health Score |
| Missing values | Strong correlations |
| Duplicate rows | Skewness |
| Target imbalance | High cardinality |
| Impossible values | Statistical outliers |
For example, petal_length ↔ petal_width = 0.96 is not automatically a problem. A statistical pattern can simply describe a real property of the data.
A statistical pattern is not automatically a data-quality defect.
🔍 What DatasetDNA Checks
Row count, column count, memory usage, missing cells, and duplicate rows.
Numeric, categorical, boolean, date, ID-like, and empty columns.
Detects normal and disguised missing values such as NaN, N/A, null, NULL, ?, -, and unknown.
Detects duplicate rows and their proportion.
Measures unique values and identifies unusually high-cardinality columns.
Mean, median, standard deviation, minimum, maximum, and skewness.
Category counts, percentages, associations, and binary variables.
Statistical outliers with domain-aware rules where appropriate.
Strong numerical relationships using Pearson correlation.
Classification and regression detection, class counts, distributions, and imbalance ratios for classification.
Identifies columns containing multiple underlying Python value types, such as numbers mixed with strings.
Detects multiple representations of the same category, such as Male, male, and M.
🧠 Semantic Type Detection
DatasetDNA goes beyond df.describe() and tries to understand what columns represent.
customer_id → ID
age → Numeric
income → Numeric
city → Categorical
joined_at → Date
churn → Target
It also handles messy values such as $55,000, 1,200, N/A, null, ?, and -.
ID-like columns can be excluded from statistical analysis where they would otherwise create misleading relationships.
❤️ Health Score
Every dataset receives a 0–100 Health Score.
100 ───────────────── Excellent
80 ────────────────── Good
60 ────────────────── Fair
40 ────────────────── Poor
0 ────────────────── Critical
- Missing values
- Duplicate rows
- Target class imbalance
- Impossible/domain-invalid values
- Correlation
- Skewness
- Cardinality
- Statistical outliers
Mixed-type and category-consistency findings are surfaced as recommendations and do not directly penalize the health score.
The score focuses on genuine data-quality concerns rather than penalizing valid statistical structure.
datasetdna examples/real.csv╭────────────── 🧬 Dataset Health ──────────────╮
│ 🟠 Health Score │
│ │
│ 70 / 100 │
│ Fair │
│ │
│ 6 quality issue(s) · 14 statistical signal(s) │
╰───────────────────────────────────────────────╯
🧬 DATA QUALITY
MEDIUM Column 'age' contains 10.0% missing values.
MEDIUM Dataset contains 5.0% duplicate rows.
HIGH Age-like column 'age' contains values outside 0-120.
📊 STATISTICAL SIGNALS
🔎 HIGH Strong correlation detected between age and salary.
🔎 MEDIUM Column 'customer_id' has very high cardinality.
🎯 Target Detection
Explicit target:
datasetdna data.csv --target churnAutomatic target detection checks:
target → label → churn
For classification, DatasetDNA reports class count, distribution, percentages, and imbalance ratio.
For regression, continuous targets are identified as regression targets and classification-specific metrics are not applied.
💡 Recommendations
DatasetDNA generates recommendations from detected dataset conditions.
🟡 MEDIUM
Target 'churn' has an imbalance ratio of 4:1.
Recommendation:
Use stratified splitting and monitor
class-aware evaluation metrics.
Recommendations also flag mixed Python value types and inconsistent categorical representations so they can be standardized before model training.
Recommendations guide investigation rather than blindly applying one universal fix.
Profile a pandas DataFrame directly in memory:
import pandas as pd
import datasetdna as dna
df = pd.DataFrame({
"age": [20, 21, 22, 23],
"income": [100, 200, 300, 400],
"churn": [0, 0, 1, 1],
})
result = dna.profile(df)Specify a target:
result = dna.profile(df, target="churn")Generate an HTML report:
result = dna.profile(
df,
target="churn",
html=True,
output="datasetdna_report.html",
)Access results:
print(result["health"])
print(result["target"])
print(result["recommendations"])The result contains:
overview
schema
mixed_types
category_consistency
missing
duplicates
cardinality
numerical
categorical
outliers
correlations
target
health
recommendations
The CLI and Python API use the same underlying profiling engine.
DatasetDNA
│
┌────────────┴────────────┐
│ │
CLI Python API
│ │
└────────────┬────────────┘
▼
Profiling Engine
│
┌───────────────┼────────────────┐
▼ ▼ ▼
Profilers Health Score Recommendations
│
├── Overview
├── Schema
├── Mixed Types
├── Category Consistency
├── Missing
├── Duplicates
├── Cardinality
├── Numerical
├── Categorical
├── Outliers
├── Correlations
└── Target
│
▼
Console / HTML Report
pip install datasetdnaTo install the current published release explicitly:
pip install datasetdna==1.1.0git clone https://github.com/KAVYA-29-ai/DatasetDNA.git
cd DatasetDNA
pip install -e .Requires Python 3.10+.
Automated coverage includes profilers, semantic types, target detection, health scoring, recommendations, CLI behavior, HTML reporting, Python API behavior, mixed-type detection, category consistency, and edge cases.
137 tests passed
Run the suite:
pytest -qMost ML failures aren't model failures. They're data failures:
❌ Hidden missing values
❌ Duplicate rows
❌ Numeric values stored as strings
❌ Impossible values
❌ ID columns treated as features
❌ Severe target imbalance
❌ Mixed Python value types
❌ Inconsistent category representations
❌ Statistical patterns mistaken for data problems
DatasetDNA is designed as a pre-flight check for machine learning datasets.
Raw Dataset
↓
DatasetDNA
↓
Understand
↓
Fix
↓
Validate
↓
Train
- Semantic type detection
- Missingness analysis
- Duplicate detection
- Cardinality analysis
- Numerical analysis
- Categorical analysis
- Outlier detection
- Correlation analysis
- Target analysis
- Two-tier health scoring
- Actionable recommendations
- Expanded validation
- Automatic target detection
- Large-file handling
- Deeper test coverage
- Mixed-type detection
- Category consistency detection
- Interactive Plotly charts
- Exportable HTML reports
- Dataset health visualization
- Target visualization
- Category consistency section
- PyPI package
- CLI
- Python API
- Stable
1.0.0release -
1.1.0release
- Expanded integrations
- More edge-case coverage
- API extensions
- ML pipeline integrations
Issues and pull requests are welcome.
If DatasetDNA crashes, detects the wrong type, misses a real issue, or produces unexpected output, please open an issue with:
- Dataset shape
- Column names
- Relevant data types
- Expected behavior
- Actual behavior
Please don't upload private or sensitive dataset contents.
DatasetDNA is released under the MIT License.
Copyright © Kavya Rajput
Profile better. Fix earlier. Train smarter.