Repository navigation
Configuration Reference
Complete reference for all configuration files and settings.
PopulationSim uses three main configuration files:
| File | Location | Version Controlled | Purpose |
|---|---|---|---|
| config.yml | Repository root | ✓ Yes | Run configuration, SQL paths, years |
| secrets.yml | Repository root | ✗ NO (local only) | Database credentials |
| settings.yaml | populationsim/configs/ | ✓ Yes | PopulationSim algorithm settings |
| controls.csv | populationsim/configs/ | ✓ Yes | Control variable definitions |
Location: config.yml (repository root)
Purpose: Main run configuration (version-controlled)
version: "v1.0.3-prerelease"
seed_data: "ACS PUMS 5 year 2021"
comments: "No Comments"
sql:
seed_households: "sql/seed_households.sql"
seed_persons: "sql/seed_persons.sql"
mgra_controls: "sql/mgra_controls.sql"
region_controls: "sql/region_controls.sql"
mgrabase: "sql/mgrabase.sql"
load_to_database: True
datalake:
env: dev
economic_controls: "data/Economic Team Region Controls.csv"
synthesis_runs:
- name: gq_mil
configs: [configs_gq_mil, configs_common]
data: data
output: output_gq_mil
num_processes: 1
- name: gq_col
configs: [configs_gq_col, configs_common]
data: data
output: output_gq_col
num_processes: 1
- name: gq_oth
configs: [configs_gq_oth, configs_common]
data: data
output: output_gq_oth
num_processes: 1
- name: household
configs: [configs_mp, configs, configs_common]
data: data
output: output
num_processes: 22
years:
- 2022
- 2026
- 2029
- 2032
- 2035
- 2040
- 2050Metadata:
-
version- Run version identifier (for tracking) -
seed_data- Documentation of seed data vintage -
comments- Optional notes about this run
SQL Queries:
-
seed_households- Path to household seed extraction query -
seed_persons- Path to person seed extraction query -
mgra_controls- Path to MGRA control generation query -
region_controls- Path to regional control query -
mgrabase- Path to ABM land use file query -
load_to_database- Boolean: Despite its legacy name, controls the optional Azure Data Lake export inmain.py-
False- Keep generated outputs as local CSV files -
True- Export the run's CSV outputs as Parquet to Azure Data Lake
-
Data Lake export:
-
datalake.env- Azure environment:devorprod; required whensql.load_to_databaseisTrue - The exporter requires the matching SAS-token environment variable. See Running PopulationSim.
Data Sources:
-
economic_controls- Path to Economics Team forecast CSV
Synthesis Runs:
-
synthesis_runs- List of PopulationSim executions for each year- Each run is a separate PopulationSim invocation
- Runs execute sequentially in listed order
-
Run Fields:
-
name- Identifier (gq_mil, gq_col, gq_oth, household) -
configs- List of config directories (relative topopulationsim/)- Layered in order (later overrides earlier)
-
configs_commonshould be last for shared settings
-
data- Data directory path (relative topopulationsim/) -
output- Output directory path (relative topopulationsim/) -
num_processes- Number of parallel processes-
1for GQ runs (small populations) -
22for household run (one per PUMA)
-
-
- Standard Setup: 3 GQ runs + 1 household run
Run Configuration:
-
years- List of forecast years to process sequentially
Single Year Testing:
years:
- 2026 # Test with just one yearRun Households Only (Skip GQ for testing):
synthesis_runs:
- name: household
configs: [configs_mp, configs, configs_common]
data: data
output: output
num_processes: 22Single-Process Household Run (Low-Memory System):
synthesis_runs:
- name: gq_mil
configs: [configs_gq_mil, configs_common]
data: data
output: output_gq_mil
num_processes: 1
- name: gq_col
configs: [configs_gq_col, configs_common]
data: data
output: output_gq_col
num_processes: 1
- name: gq_oth
configs: [configs_gq_oth, configs_common]
data: data
output: output_gq_oth
num_processes: 1
- name: household
configs: [configs, configs_common] # configs_mp removed
data: data
output: output
num_processes: 1 # Single processOptional Azure Data Lake Export:
sql:
load_to_database: True # Legacy key name; enables the data lake export
datalake:
env: devAlternative Seed Data:
seed_data: "ACS PUMS 5 year 2016-2020"
sql:
seed_households: "sql/seed_households_2020.sql"
seed_persons: "sql/seed_persons_2020.sql"Location: secrets.yml (repository root)
Purpose: Database credentials (NOT version-controlled)
sql:
server: "<SQLInstanceName>"
schema: "<[database].[schema]>"
output_database: "<SQLoutputDatabaseName>"-
server- SQL Server instance name- Format:
ServerNameorServerName\InstanceName - Examples:
SANDAG-SQL01,localhost
- Format:
-
schema- UDM staging schema- Format:
[database_name].[schema_name] - Must use square brackets
- Example:
[udm_staging].[sr15]
- Format:
-
output_database- Database used by the Streamlit report's database source- Not used by
main.pyto export run outputs - Example:
PopulationSim_Production
- Not used by
sql:
server: "SANDAG-SQL01"
schema: "[udm_staging].[sr15]"
output_database: "PopulationSim_Production"✓ Never commit to Git - Check with git check-ignore secrets.yml
✓ Use Windows Authentication - Avoid hardcoded passwords
✓ Read-only permissions - For source databases
✓ Separate output database - For write operations
Location: populationsim/configs/settings.yaml
Purpose: PopulationSim algorithm configuration
# Integerization
INTEGERIZE_WITH_BACKSTOPPED_CONTROLS: True
USE_SIMUL_INTEGERIZER: True
SUB_BALANCE_WITH_FLOAT_SEED_WEIGHTS: False
# Expansion limits
max_expansion_factor: 30
min_expansion_factor: 0.01
# Convergence
max_iterations: 5000
absolute_convergence: 0.0001
relative_convergence: 0.0001
# Geography
geographies: [region, PUMA, mgra]
seed_geography: PUMAIntegerization:
-
INTEGERIZE_WITH_BACKSTOPPED_CONTROLS: True- Ensure critical controls exactly met -
USE_SIMUL_INTEGERIZER: True- Simultaneous integerization (better quality) -
SUB_BALANCE_WITH_FLOAT_SEED_WEIGHTS: False- Use integer weights for sub-balancing
Expansion Constraints:
-
max_expansion_factor: 30- Maximum weight multiplier (household can be replicated max 30 times) -
min_expansion_factor: 0.01- Minimum weight (effectively allow dropping households)
Convergence:
-
max_iterations: 5000- Maximum IPF iterations before failure -
absolute_convergence: 0.0001- Stop when all controls within this absolute difference -
relative_convergence: 0.0001- Stop when all controls within this relative difference
Geography:
-
geographies: [region, PUMA, mgra]- Three-level hierarchy -
seed_geography: PUMA- Seed data identified by PUMA
Convergence Issues:
- Increase
max_iterationsto 10000 - Loosen
absolute_convergenceto 0.001
Too Much Household Replication:
- Decrease
max_expansion_factorto 20 or 15
Balance Quality Issues:
- Increase
max_expansion_factorto 40 or 50 - Review control importance weights
Location: populationsim/configs_mp/settings.yaml
Purpose: Multiprocessing configuration
inherit_settings: True # Inherit from base settings.yaml
multiprocess: True
num_processes: 22 # One per PUMA
slice_geography: PUMA
multiprocess_steps:
- name: mp_sub_balancing_mgra
begin: sub_balancing.geography=mgra
num_processes: 2211 cores available:
num_processes: 11Runtime: ~2 hours (vs. 40 min with 22)
Less than 64 GB RAM:
num_processes: 10 # or lowerRuntime: ~3 hours
Debugging (disable parallelization):
multiprocess: FalseRuntime: ~13 hours (sequential)
Location: populationsim/configs/controls.csv
Purpose: Define all control variables
target,geography,seed_table,importance,expression
Total_HH,mgra,households,1000000000,(households.WGTP > 0) & (households.WGTP < np.inf)
HHSize_1,mgra,households,250000,households.NP == 1
HHSize_2,mgra,households,250000,households.NP == 2
Male_0_4,mgra,persons,1000,((persons.SEX == 1) & (persons.AGEP >= 0) & (persons.AGEP <= 4))-
target- Unique control name (e.g., HHSize_1) -
geography- Geographic level (region, PUMA, or mgra) -
seed_table- Which seed table to query (households or persons) -
importance- Weight for balancing (higher = more important) -
expression- Pandas boolean expression to identify matching records
| Category | Count | Importance Range | Geography |
|---|---|---|---|
| Total Households | 1 | 1,000,000,000 | mgra |
| Household Size | 4 | 250,000 | mgra |
| Income | 10 | 10,000 | mgra |
| Workers | 4 | 5,000 | mgra |
| Children | 3 | 5,000 | mgra |
| Age/Sex | 42 | 1,000 | mgra |
| Race/Ethnicity | 6 | 1,000 | mgra |
| Employment | 18 | 100,000,000 | region |
| Labor Force | 4 | 1,000,000 | region |
- 1,000,000,000 - Total_HH (must match exactly)
- 100,000,000 - Employment totals (critical regional controls)
- 1,000,000 - Labor force (important but allow some flex)
- 250,000 - Household size (high priority)
- 10,000 - Income (medium priority)
- 5,000 - Workers, children (medium priority)
- 1,000 - Age, sex, race (lower priority, more flexible)
Example: Households with seniors (65+)
HHSenior,mgra,households,5000,(households.persons.AGEP >= 65).any()Requirements:
- Add row to
controls.csv - Add corresponding column to
mgra_controls.csv - Test expression with seed data
- Verify control totals sum correctly
import yaml
import os
import pandas as pd
# Load configurations
with open('config.yml') as f:
config = yaml.safe_load(f)
assert os.path.exists('secrets.yml'), "secrets.yml not found!"
with open('secrets.yml') as f:
secrets = yaml.safe_load(f)
# Validate config.yml
assert 'sql' in config
assert 'years' in config
assert len(config['years']) > 0
# Validate secrets.yml
assert 'sql' in secrets
assert 'server' in secrets['sql']
assert 'schema' in secrets['sql']
# Check SQL file paths exist
for key, path in config['sql'].items():
if key != 'load_to_database':
assert os.path.exists(path), f"SQL file not found: {path}"
# Check economic controls file
assert os.path.exists(config['economic_controls'])
# Validate PopulationSim configs
assert os.path.exists('populationsim/configs/settings.yaml')
assert os.path.exists('populationsim/configs/controls.csv')
# Load and validate controls.csv
controls = pd.read_csv('populationsim/configs/controls.csv')
required_cols = ['target', 'geography', 'seed_table', 'importance', 'expression']
assert all(col in controls.columns for col in required_cols)
print("✓ All configuration files validated!")
print(f"✓ Years: {config['years']}")
print(f"✓ SQL Server: {secrets['sql']['server']}")
print(f"✓ Azure Data Lake export enabled: {config['sql']['load_to_database']}")
print(f"✓ Controls: {len(controls)}")Before running PopulationSim:
✓ config.yml
-
versionupdated for this run -
yearslist correct - SQL file paths valid
-
load_to_databaseset appropriately
✓ secrets.yml
- File created (not in Git)
-
servercorrect -
schemamatches UDM staging -
output_databaseexists (if loading)
✓ settings.yaml
-
max_expansion_factorappropriate (default: 30) -
max_iterationssufficient (default: 5000) -
geographieslist correct
✓ configs_mp/settings.yaml
-
num_processesmatches available cores -
multiprocess: True(unless debugging)
✓ controls.csv
- 56 controls defined
- No duplicate
targetnames - Expressions valid pandas syntax
- Importance weights reasonable
✅ Configuration complete? → Running PopulationSim
✅ Need data info? → Data Preparation
❓ Something not working? → Troubleshooting
For detailed technical documentation: See SANDAG_PopulationSim_Documentation.md Section 4.2