Repository navigation
Getting Started
This quick start guide will help you run your first PopulationSim synthesis in about 30 minutes.
Before you begin, ensure you have:
✅ Windows 10/11 or Linux environment
✅ Python 3.9, 3.10, 3.11, or 3.12 installed
✅ Access to SANDAG's UDM staging database (for control data)
✅ ODBC Driver 17 for SQL Server (if using database features)
✅ ~5GB free disk space for outputs
git clone https://github.com/SANDAG/Population-Sim.git
cd Population-Sim
git checkout new_populationsim# Install uv package manager (if not already installed)
pip install uv
# Create virtual environment and install dependencies
uv venv
.venv\Scripts\Activate.ps1
uv pip install -e .# Check if populationsim can be imported
python -c "import populationsim; print('PopulationSim imported successfully')"
# Check installed version
uv pip show populationsim
# Should show: Version: 0.10.0Open config.yml and verify the settings:
version: "15.01"
seed_data: "pums_2017_2021"
years:
- 2022
- 2026
# ... other years# Run for a single year (recommended for first run)
python main.pyThis will:
- Create seed data (one-time, ~15 minutes)
- Generate controls from UDM database
- Run population synthesis for all configured years
- Create ABM output files
- Optionally export generated outputs to Azure Data Lake (requires
datalake.envand the matching SAS-token environment variable)
flowchart TD
A[main.py starts] --> B{Seed data exists?}
B -->|No| C[Create seed data<br/>~15 minutes]
B -->|Yes| D[Load config.yml]
C --> D
D --> E[Connect to UDM database]
E --> F[For each year...]
F --> G[Generate controls from UDM]
G --> H[Run PopulationSim IPF]
H --> I[Generate group quarters]
I --> J[Combine outputs]
J --> K[Create ABM files]
K --> L{sql.load_to_database?}
L -->|Yes| M[Export CSV outputs to Azure Data Lake]
L -->|No| N[Complete!]
M --> N
style C fill:#ffe1e1
style H fill:#e1f5ff
style K fill:#e1ffe1
After a successful run, the files are organized by year:
output/
└── 2022/
├── synthetic_households.csv
├── synthetic_persons.csv
├── mgra15_based_input_2022.csv
├── final_summary_*.csv
├── *_controls.csv
├── reporting_controls.csv
└── benchmark_comparisons.csv
benchmark_comparisons.csv contains report-only benchmark targets and results when benchmark definitions are configured. It is also included in the Azure export when enabled. See Output Files for its schema and availability.
Launch the interactive validation dashboard:
uv run -- streamlit run report/report.pyThis opens a web interface at http://localhost:8501 showing:
- Control vs. result comparisons
- Geographic distributions
- Demographic breakdowns
- Quality metrics
Solution:
# Ensure virtual environment is activated
.venv\Scripts\Activate.ps1
# Reinstall in editable mode
uv pip install -e .Solution: Install ODBC Driver 17:
# Download from Microsoft:
# https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server
# Then verify in config that connection string is correctSolution: Seed data creation failed. Delete and recreate:
Remove-Item -Recurse populationsim\data\seed_*.csv
python -c "from python.build_seed_data import main; main()"Causes:
- First run creates seed data (~15 minutes - this is normal)
- Running all 7 years takes ~2-3 hours total
- Database queries may be slow depending on network
Solution: Run one year first to verify:
# In config.yml, comment out other years:
years:
- 2022
# - 2026
# - 2029
# ... etc| Task | Time | Notes |
|---|---|---|
| Seed data creation | ~15 min | One-time operation |
| Control generation | ~5 min | Per year |
| PopulationSim synthesis | ~60-75 min | Per year, 22 parallel processes |
| Group quarters | ~2 min | Per year |
| ABM output creation | ~3 min | Per year |
| Total (1 year) | ~80 min | After seed data exists |
| Total (all 7 years) | ~8-10 hours | Including seed data |
✅ You've completed your first run! Now you can:
- Understand the outputs - Learn about file formats and specifications
- Run validation - Use the Streamlit dashboard
- Customize configuration - Adjust settings for your needs
- Deep dive into methodology - Understand the IPF algorithm
- Prepare for production - Full workflow with all options
graph LR
A[Getting Started<br/>✓ You are here] --> B[Output Files]
B --> C[Validation & QA]
C --> D[Configuration]
D --> E[Data Preparation]
E --> F[Algorithm Details]
F --> G[Production Use]
style A fill:#90EE90
Next recommended page: Output Files - Understanding what you just created
Need help? Check the Troubleshooting guide or FAQ