DataCenter COMPASS 🧭# DataCenter COMPASS 🧭

**A clean, production-ready system for analyzing data center projects in U.S. interconnection queues.**A production-ready, AI-powered web scraper for extracting data center project intelligence from news articles.

---## Overview

📖 OverviewDataCenter COMPASS automatically discovers and extracts structured data about data center projects from news articles across the web. It uses advanced AI and robust web scraping techniques to handle real-world challenges like SSL certificates, bot detection, and diverse article formats.

DataCenter COMPASS identifies and enriches data center projects from ISO/RTO interconnection queues using:## Key Features

• Raw data collection from all major U.S. ISOs/RTOs

• Heuristic-based detection using capacity, naming, and location signals✅ AI-Powered Extraction: OpenAI GPT-4 intelligently extracts multiple projects per article

• LLM-powered enrichment for deep research on candidates✅ SSL Certificate Bypass: Handles problematic SSL certificates automatically

• Transparent, layered architecture for reproducible results✅ Multi-Project Detection: Finds multiple data center projects in single articles

✅ Production Ready: Robust error handling and fallback systems

🚀 Quick Start✅ Multiple Output Formats: Clean JSON and CSV exports

✅ Bot Evasion: Advanced techniques to bypass website blocking

1. Install Dependencies✅ Diverse Source Support: Works with construction news, tech blogs, local news, industry publications

🧪 (Experimental) Grid Queue Intelligence: Optional complementary script to pull ISO interconnection queue data (starting with PJM) using the open-source gridstatus library to analyze power availability timelines.

# Create and activate virtual environment## Quick Start

python3 -m venv venv

source venv/bin/activate  # On Windows: venv\Scripts\activate### 1. Install Dependencies



# Install requirements```bash

pip install -r requirements.txtpip install -r requirements.txt

```python -m spacy download en_core_web_sm

2. Configure Environment

2. Set OpenAI API Key (Recommended)

Create a .env file in the root directory:

```bashexport OPENAI_API_KEY="your-openai-api-key"

# Required for PJM API access```

PJM_API_KEY=your_pjm_api_key_here

*Note: Without OpenAI API key, the system will use NLP fallback (lower accuracy)*

# Required for LLM enrichment (optional)

AZURE_OPENAI_ENDPOINT=your_azure_endpoint### 3. Extract Data from Articles

AZURE_OPENAI_API_KEY=your_azure_key

AZURE_OPENAI_DEPLOYMENT=your_deployment_name**Single Article:**

AZURE_OPENAI_API_VERSION=2024-02-15-preview```bash

```python datacenter_scraper.py "https://www.constructiondive.com/news/amazon-data-centers-mississippi/706228/"

Get a PJM API key: https://dataminer2.pjm.com/

Multiple Articles:

3. Run the Pipeline```bash

python datacenter_scraper.py \

# Fetch raw data from all ISOs  "https://sanantonioreport.org/san-antonio-vantage-data-centers-microsoft-west-side/" \

python -m queue_pipeline_v2.run fetch --isos ALL  "https://www.constructiondive.com/news/google-plans-2-new-data-centers-ohio-columbus-lancaster/649470/"

Normalize to canonical schema

python -m queue_pipeline_v2.run normalize## Output

Compute data center signalsThe scraper generates two files:

python -m queue_pipeline_v2.run signals

📄 datacenter_projects.json

Curate top 30 candidates```json

python -m queue_pipeline_v2.run curate --profile balanced --target 30{

  "total_projects": 5,

### 4. Generate LLM Batches (Optional)  "projects": [

    {

```python      "company_name": "Amazon Web Services",

from queue_pipeline_v2.v2_batch_llm import batch_llm      "project_name": null,

from pathlib import Path      "location_city": "Madison County",

      "location_state": "MS",

batch_llm(      "investment_amount": "$10 billion",

    curate_dir=Path('queue_pipeline_v2/curate'),      "capacity_mw": null,

    out_dir=Path('queue_pipeline_v2/batches'),      "size_sqft": null,

    batch_size=10,      "timeline": "partially open by 2027",

    limit=30      "project_status": "announced",

)      "article_url": "https://www.constructiondive.com/news/amazon-data-centers-mississippi/706228/",

```      "article_title": "",

      "extraction_confidence": "high"

---    }

  ]

## 📁 Project Structure}

DataCenterCOMPASS/### 📊 `datacenter_projects.csv`

│Clean tabular format ready for analysis in Excel, Google Sheets, or data analysis tools.

├── queue_pipeline_v2/          # ⭐ Main pipeline (USE THIS)

│   ├── v2_raw.py               # Fetch raw ISO queues## Data Schema

│   ├── v2_normalize.py         # Normalize to canonical schema

│   ├── v2_signals.py           # Compute DC detection signalsEach extracted project includes:

│   ├── v2_curate.py            # Curate high-quality candidates

│   ├── v2_batch_llm.py         # Generate LLM enrichment batches| Field | Description | Example |

│   ├── run.py                  # Unified CLI interface|-------|-------------|---------|

│   ├── README.md               # Detailed documentation| `company_name` | Company developing the data center | "Amazon Web Services" |

│   │| `project_name` | Specific project name (if mentioned) | "Prime Data Center Campus" |

│   ├── raw/                    # Raw ISO data (generated)| `location_city` | City location | "Madison County" |

│   ├── normalize/              # Normalized data (generated)| `location_state` | State (abbreviated) | "MS" |

│   ├── signals/                # Signal-enriched data (generated)| `investment_amount` | Investment amount | "$10 billion" |

│   ├── curate/                 # Curated candidates (generated)| `capacity_mw` | Power capacity in megawatts | "15" |

│   └── batches/                # LLM batches (generated)| `size_sqft` | Size in square feet | "85000" |

│| `timeline` | Construction timeline | "partially open by 2027" |

├── data/                       # Data storage| `project_status` | Project status | "announced", "under construction", "operational" |

│   ├── queue_json/             # Archival ISO queue snapshots| `extraction_confidence` | AI confidence level | "high", "medium", "low" |

│   │   └── raw_isos_YYYY-MM-DD/  # Date-stamped raw data| `article_url` | Source article URL | Original URL |

│   └── reference/              # Reference data (cities, states, brands)

│## Performance

├── config/                     # Configuration files

│   ├── brands.json             # Known DC operator brands**Tested Performance (Production Validation):**

│   ├── cities.json             # DC-heavy cities- ✅ **100% Success Rate** across 12 diverse news sources

│   └── states.json             # State mappings- ✅ **38 Projects Extracted** from 12 articles (4.2 projects/article average)

│- ✅ **8.1 seconds** average processing time per URL

├── prompts/                    # LLM prompt templates- ✅ **100% Data Completeness** for essential fields (company, location, status)

│   ├── deep_research_prompt.txt

│   └── deep_research_enhanced_prompt.txt**Supported Sources:**

│- Construction Industry News (ConstructionDive, etc.)

├── archive/                    # Deprecated/old code- Data Center Industry Publications (DataCenterDynamics, etc.)

│   ├── queue_pipeline_v1_2025-10-07/  # Old pipeline (archived)- Local News Outlets

│   ├── root_scripts_2025-10-07/       # Old root scripts (archived)- Tech Company Engineering Blogs

│   ├── deprecated/             # Legacy experimental code- Investment and Infrastructure News

│   └── legacy_queue_refactor_2025-09-26/  # Historical refactors

│## System Requirements

├── .env                        # Environment variables (not in git)

├── .env.example                # Environment template- Python 3.8+

├── .gitignore                  # Git ignore rules- Internet connection

├── requirements.txt            # Python dependencies- OpenAI API key (recommended)

└── README.md                   # This file- ~100MB disk space

Dependencies

---

• openai>=1.0.0 - AI extraction

🎯 Core Workflows- spacy>=3.8.0 - NLP fallback

• newspaper3k>=0.2.8 - Article extraction

Workflow 1: Fetch All ISO Queue Data- beautifulsoup4>=4.12.0 - HTML parsing

• requests>=2.31.0 - HTTP requests

Collect raw interconnection queue data from all ISOs and save for reference:- fake-useragent>=1.4.0 - Bot evasion

• gridstatus>=0.30.1,<0.32 - (Experimental) ISO interconnection queue data (PJM, expandable)

# Fetch to date-stamped folder### PJM Interconnection Queue (Now Included by Default)

python -m queue_pipeline_v2.run fetch \

  --isos ALL \The ingestion pipeline under `queue_pipeline/ingestion/fetch_queue.py` now ALWAYS attempts to include **PJM** interconnection queue data using the public fetch implemented in the `gridstatus` library. An API key is **not required** for the PJM queue itself (older code previously skipped PJM when `PJM_API_KEY` was absent). If you provide `PJM_API_KEY`, it will simply be passed through (useful only for other authenticated PJM endpoints not yet used here).

  --out-dir data/queue_json/raw_isos_$(date +%Y-%m-%d) \

  --insecureBasic one‑liner to fetch & score all ISOs including PJM (year ≥ 2020):

```bash

# Result: ~23,000+ projects from 7 ISOspython -m queue_pipeline.ingestion.fetch_queue --iso ALL --queue-year-min 2020 --out queue_pipeline/qp_outputs/raw_candidates.jsonl

Output: data/queue_json/raw_isos_YYYY-MM-DD/#### Focusing on PJM Load / Data Center Candidates

• PJM.raw.jsonl (9,251 projects)

• MISO.raw.jsonl (3,549 projects)Add the new --pjm-load-only flag to restrict PJM rows to those heuristically classified as load (resource_type LOAD or LARGE_LOAD, or rows with a structured load indicator):

• SPP.raw.jsonl (2,798 projects)```bash

• CAISO.raw.jsonl (2,278 projects)python -m queue_pipeline.ingestion.fetch_queue --iso PJM --queue-year-min 2020 --pjm-load-only --recall-bias --out queue_pipeline/qp_outputs/pjm_load_candidates.jsonl

• NYISO.raw.jsonl (2,120 projects)```

• ERCOT.raw.jsonl (1,847 projects)

• ISONE.raw.jsonl (1,751 projects)Combine with other ISOs while filtering only PJM rows:

### Workflow 2: Identify Data Center Candidatespython -m queue_pipeline.ingestion.fetch_queue --iso ALL --queue-year-min 2020 --pjm-load-only --recall-bias --limit 300

Run the full pipeline to find and curate data center candidates:

When to Use --pjm-load-only

# Step 1: Fetch (to default pipeline location)

python -m queue_pipeline_v2.run fetch --isos ALL#### Output Signals (Relevant for PJM Load Filtering)

The filter relies on:

# Step 2: Normalize- `resource_type` in {`LOAD`, `LARGE_LOAD`}

python -m queue_pipeline_v2.run normalize- `structured_load_flag` (derived from structured PJM columns containing load / data center indicators)



# Step 3: Compute signalsRows outside those criteria are excluded only for PJM when the flag is set; other ISOs are unaffected.

python -m queue_pipeline_v2.run signals

> Tip: Pair with `--prune-nonload` (if/when ported to the orchestrator) for a fully load-focused multi-ISO dataset.

# Step 4: Curate (adjust --target for more/fewer candidates)

python -m queue_pipeline_v2.run curate --profile balanced --target 30

```## Error Handling



**Output:** `queue_pipeline_v2/curate/`The system includes comprehensive error handling:

- `full_enrichment.jsonl` - Complete candidate data- **SSL Certificate Issues**: Automatic bypass

- `minimal_review.jsonl` - Minimal fields for quick review- **Bot Detection**: Multiple user agent strategies

- **Network Timeouts**: Configurable timeouts and retries

### Workflow 3: Focus on Single ISO (PJM)- **Content Extraction Failures**: Multiple parsing strategies

- **AI API Failures**: Automatic NLP fallback

```bash

# Fetch only PJM## Production Deployment

python -m queue_pipeline_v2.run fetch --isos PJM

This system is **production-ready** and includes:

# Run remaining steps as normal- Comprehensive error handling and logging

python -m queue_pipeline_v2.run normalize- SSL certificate bypass for problematic sites

python -m queue_pipeline_v2.run signals- Multiple fallback strategies

python -m queue_pipeline_v2.run curate --profile balanced --target 30- Clean, structured output

```- Automatic timestamped backups



### Workflow 4: Prepare for LLM Enrichment## License



```pythonMIT License - See LICENSE file for details.

# Generate batches for manual or automated LLM enrichment

python << 'EOF'## Support

from queue_pipeline_v2.v2_batch_llm import batch_llm

from pathlib import PathFor issues or questions, please check the existing documentation or create an issue in the repository.



batch_llm(---

    curate_dir=Path('queue_pipeline_v2/curate'),## 🧪 (Archived) Unified Data Center Interconnection Queue Pipeline (`dc_queue`)

    out_dir=Path('queue_pipeline_v2/batches'),

    batch_size=10,  # Projects per batchThe original experimental `dc_queue/` package has been fully archived under `archive/legacy_queue_refactor_2025-09-26/` and removed from the active codebase. A fresh, simplified, and modular implementation now lives under `queue_pipeline/` and should be used for all future ingestion, preparation, curation, batching, validation, and summarization tasks.

    limit=30,       # Total projects to batch

    use_full=False  # Use minimal_review.jsonlIf you are looking for the old scripts referenced below (e.g. `dc_queue.prepare_llm_batches`, `make_llm_batches`, `profile_prepared`, `quick_grid.py`), they now exist only in the archive for historical comparison. Commands in this README referencing `dc_queue` are deprecated; migrate to the orchestrator:

)

EOF```

```python -m queue_pipeline.qp ingest --iso ALL --year-min 2020 --limit 150 --recall-bias

Output: queue_pipeline_v2/batches/

• llm_batch_0001.jsonl (10 projects)Additional stages (prepare, curate, batch, validate, summarize) will be added to qp as they are ported.

• llm_batch_0002.jsonl (10 projects)

• llm_batch_0003.jsonl (10 projects)### Why a New Pipeline?

• BATCH_SUMMARY.md (documentation)Heterogeneous ISO schemas, evolving service codes, and the need to combine structural + textual + contextual signals (geo clusters, developer roles) required a layered, extensible architecture instead of ad‑hoc heuristics.

---### Architecture Overview

## 🔧 Configurationdc_queue/

  adapters/        # ISO-specific raw -> canonical mapping functions

### Curation Profiles  core/            # Normalization primitives

  enrichment/      # Cluster & developer role feature augmentation

Control the balance between recall and precision:  config/          # JSON config (geo clusters, developer roles, service mapping)

  run.py           # Orchestrated pipeline CLI (preferred entrypoint)

| Profile | Description | Use Case |```

|---------|-------------|----------|

| `recall` | Maximizes candidates found | Initial exploration |### 🔍 LLM / RAG Integration Guide (Context & Retrieval)

| `balanced` | **Default** - Good mix | Most use cases |

| `precision` | Minimizes false positives | High-confidence only |The pipeline now supports rich LLM-oriented context generation with two optional flags:

| `exploratory` | Experimental signals | Research & development |

| Flag | Purpose | Output Columns Added |

```bash|------|---------|----------------------|

# Example: High recall (more candidates)| `--llm-context` | Adds verbose, labeled context & search query fields | `llm_context`, `llm_search_query` |

python -m queue_pipeline_v2.run curate --profile recall --target 50| `--llm-compact` | Compresses labels & replaces long phrases (token efficient) | Shortened `llm_context` |

When --llm-context is used without explicit JSON options, the pipeline auto-enables JSON Lines export with a curated column subset suitable for ingestion into a vector store or downstream RAG index.

Environment Variables

Example (verbose):

| Variable | Required | Description |```

|----------|----------|-------------|python -m dc_queue.run --iso ALL --recall-bias --year-min 2020 --llm-context --limit 500

| PJM_API_KEY | Yes | PJM API access key |```

| AZURE_OPENAI_* | No | For LLM enrichment |

Compact mode:

---```

python -m dc_queue.run --iso ALL --recall-bias --llm-context --llm-compact --limit 500

📊 Data Schema```

Canonical Fields (After Normalization)#### llm_context Composition (Verbose Mode)

Pipe-delimited segments capturing high-signal attributes:

| Field | Type | Description |```

|-------|------|-------------|ISO=SPP | ID=GEN-2025-SR9 | State=OK | County='Fort Gibson' | ReqType='Energy Resource' | CapMW=410.0 | Submitted=2025-05-31 00:00:00 | Scores S=0.55 T=0.00 E=0.00 Total=0.55 | Tier=LOW

| iso | string | ISO/RTO identifier (PJM, MISO, etc.) |```

| iso_native_id | string | Original queue ID from ISO |Segments (if available):

| project_name | string | Project name |ISO, ID, Project, Entity, State, County, ReqType, CapMW, LoadMW, Submitted, ProposedISD, Scores (S/T/E/Total), Tier, Status, Stage

| capacity_mw | float | Capacity in megawatts |

| state | string | U.S. state code |Suppressed automatically:

| county | string | County name |- Empty / null-like fields (e.g., project_name == None)

| submitted_dt | datetime | Queue submission date |- ProposedISD if datetime missing / NaT

| proposed_in_service_dt | datetime | Proposed online date |

| status | string | Project status |#### Compact Mode Differences (--llm-compact)

• Removes pipe separators (|) and joins tokens with spaces

Signal Fields (After Signal Computation)- Shortens labels: Submitted→Sub, ProposedISD→PISD

• Maps long request types back to short codes (Energy Resource→ER, Network Resource→NR)

| Field | Description |- Keeps score component abbreviations (S, T, E) and Tier

|-------|-------------|

| dc_signal_strength | weak/medium/strong |Lifecycle tokens (Status=<class> and Stage=<funnel>) are appended when available; they help downstream retrieval distinguish active vs withdrawn vs operational rows.

| candidate_confidence | 0.0 - 1.0 score |

| signals.capacity_flag | True if capacity matches DC profile |Example compact:

| signals.name_flag | True if name suggests DC |```

| signals.brand_flag | True if known DC operator |ISO=SPP ID=GEN-2025-SR9 State=OK County='Fort Gibson' ReqType='ER' CapMW=410.0 Sub=2025-05-31 00:00:00 S=0.55 T=0.00 E=0.00 Total=0.55 Tier=LOW

---

#### `llm_search_query` Field

## 🧪 Testing & ValidationConstructed to seed external news / document search or retrieval expansion.

Token sources (conditionally added):

### Quick Validation- Entity (developer/operator)

- Project name (if present)

```bash- County + State or State alone

# Check raw data- ISO

wc -l queue_pipeline_v2/raw/*.jsonl- Year token (from `submitted_dt`)

- Capacity bucket ("50 MW" if ≥50, "100 MW" if ≥100)

# Check normalized- Select lifecycle token (only if `withdrawn`, `canceled`, or `in_service`) to bias search toward status-confirming evidence

wc -l queue_pipeline_v2/normalize/normalized.jsonl

Example:

# View top candidates```

head -n 5 queue_pipeline_v2/curate/full_enrichment.jsonl | python -m json.toolFort Gibson OK SPP 2025 100 MW

Example: Count by ISOYou can feed this directly into web/news/API search, then attempt to align returned article metadata against the structured queue row.

import jsonFor RAG, prefer storing BOTH:

from collections import Counter1. `llm_context` (as the retrievable text) — high signal density

2. A structured JSON object (original fields) — for programmatic filters (e.g., capacity_mw ≥ 100)

with open('queue_pipeline_v2/signals/signals.jsonl') as f:

    isos = [json.loads(line)['iso'] for line in f]Recommended JSONL ingestion schema:

```jsonl

print(Counter(isos)){"id": "SPP:GEN-2025-SR9", "text": "ISO=SPP ... Tier=LOW", "iso": "SPP", "iso_native_id": "GEN-2025-SR9", "state": "OK", "county": "Fort Gibson", "capacity_mw": 410.0, "dc_score": 0.55, "dc_tier": "LOW"}

---#### Retrieval Prompting Pattern

After retrieving top-k rows given a user/article snippet, instruct the model to cross-check:

📚 Additional Documentation```

You are linking news articles to interconnection queue entries. Given:

• Pipeline Details: queue_pipeline_v2/README.mdArticle snippet: "A 400 MW data center load proposed in eastern Oklahoma..."

• ISO Data Reference: data/queue_json/QUICK_REFERENCE.mdCandidate rows:

• Batch Summary: queue_pipeline_v2/batches/BATCH_SUMMARY.md1. ISO=SPP ID=GEN-2025-SR9 State=OK County='Fort Gibson' CapMW=410.0 ...

• Old System: archive/queue_pipeline_v1_2025-10-07/ARCHIVED_README.md2. ISO=SPP ID=GEN-2025-SR6 State=OK County='Minco' CapMW=200.0 ...

• Scraper Examples: archive/root_scripts_2025-10-07/datacenter_scraper.pyReturn the most likely match with reasoning (or NONE if mismatch).

---

### 🚀 MVP: From Raw Queue Candidates to Enriched DC Insights

## 🗄️ Archive Policy

This lightweight flow turns `pipeline_candidates.jsonl` into an analytics-ready enriched dataset with model-confirmed data center classifications, temporal metrics, and summary stats.

Old/deprecated code is preserved in `archive/` with documentation:

- **Don't delete** - Historical reference value#### 1. Prepare LLM Candidates

- **Don't modify** - Frozen snapshotsDEPRECATED (old prepare step shown for archival reference only):

- **Don't use** - Active development uses `queue_pipeline_v2/````bash

# OLD (archived) -- do not use

---python -m dc_queue.prepare_llm_batches --year 2025

🤝 ContributingNew equivalent (pending full port) will become:

When adding new features:python -m queue_pipeline.qp prepare --year 2025

1. Use queue_pipeline_v2/ - Don't create new top-level scripts

2. Follow layered architecture - raw → normalize → signals → curate##### Heuristic Enhancements (v2)

3. Add tests - Validate with real data samplesThe prepare step now adds richer pre-LLM discriminators:

4. Document - Update READMEs and add examples

5. Preserve provenance - Keep original field names in raw layer| Field | Meaning |

|-------|---------|

---| dc_heuristic_score | 0..3 additive score: regex DC terms + brand + structural (≥300MW in key states) minus 0.5 if generation only |

| dc_likelihood_tier | HIGH (≥2), MED (≥1), LOW (>0), NONE (≤0) |

📝 License| brand_hit_flag | Brand token from config/brands.json present |

| generation_flag | Contains generation tokens (solar, wind, battery, etc.) |

Internal research tool - not for public distribution.| structural_dc_flag | Large-capacity structural inference |

| negative_reason | GENERATION_ONLY or NO_DC_SIGNAL for NONE tier rows |

---| priority_bucket | New ordering: withdrawn_dc_high, dc_high, dc_mid, withdrawn, high_value, mid_value, other |

🆘 TroubleshootingQuick distribution profiling:

Deprecated old profiling command (archived):

SSL Certificate Errors```bash

OLD (archived)

Add --insecure flag to fetch command:python -m dc_queue.profile_prepared --prepared dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl

bash

python -m queue_pipeline_v2.run fetch --isos ALL --insecureNew command (after tool port) will be:

python -m queue_pipeline.qp profile

### PJM API Key Not Working```

Outputs tier mix, score histogram, brand × generation table, withdrawn rate by tier.

1. Check `.env` file has `PJM_API_KEY=your_key`

2. Verify key at https://dataminer2.pjm.com/#### 2. Create Batches (choose round_robin for early diversity)

3. Test with: `python -c "import os; from dotenv import load_dotenv; load_dotenv(); print(os.getenv('PJM_API_KEY'))"`Deprecated batching (archived):

```bash

### No Candidates Found# OLD (archived)

python -m dc_queue.make_llm_batches --batch-size 40 --mix-mode round_robin

- Try `--profile recall` for higher sensitivity```

- Check `signals/signals.jsonl` for signal distributionsFuture replacement:

- Verify raw data was fetched: `ls queue_pipeline_v2/raw/````bash

python -m queue_pipeline.qp batch --batch-size 40 --mix-mode round_robin --ensure-likely-dc 12

### Empty Batch Files```

Send each `*_prompt_strict.txt` to the model, capture only lines between BEGIN_JSONL / END_JSONL into `llm_results/batchXXXX.jsonl`.

- Ensure curate step produced results: `wc -l queue_pipeline_v2/curate/*.jsonl`

- Check `--limit` parameter in batch_llm()#### 3. Concatenate Model Output

- Verify `minimal_review.jsonl` exists```bash

cat llm_results/batch*.jsonl > llm_results/full_all.jsonl

---```



## 🎯 Roadmap#### 4. Validate (heuristic precision/recall sanity check)

```bash

- [ ] Add automated LLM enrichment workflowpython -m dc_queue.validate_llm_results \

- [ ] Create visualization dashboard for candidates  --prepared dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl \

- [ ] Add historical trend analysis  --llm-output llm_results/full_all.jsonl \

- [ ] Expand to international ISOs/TSOs  --dc-threshold 0.75

- [ ] Add project clustering by location/operator```



---#### 5. Export Classification-Only (clean minimal schema)

```bash

**Last Updated:** October 7, 2025  python -m dc_queue.export_classification_only \

**Version:** 2.0 (Clean Architecture)  --llm-output llm_results/full_all.jsonl \

  --out llm_results/full_all_classification_only.jsonl
```

#### 6. Merge Enrichment (adds queue_days, lifecycle, evidence counts)
```bash
python -m dc_queue.merge_llm_enrichment \
  --prepared dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl \
  --llm-output llm_results/full_all.jsonl \
  --out dc_queue_pipeline_outputs/enriched_candidates.jsonl \
  --dc-threshold 0.75
```

#### 7. Summarize & Generate Charts
```bash
python -m dc_queue.summarize_enriched \
  --enriched dc_queue_pipeline_outputs/enriched_candidates.jsonl \
  --summary-out dc_queue_pipeline_outputs/enriched_summary.json \
  --charts-dir figures/enriched_mvp --html
```
Optional: `pip install plotly` for HTML chart generation.

#### 8. Visual / Dashboard Inputs
Key derived fields after merge:
| Field | Meaning |
|-------|---------|
| `dc_confirmed_flag` | Model-high-confidence DC load classification |
| `queue_days` | Days since submission (approx latency so far) |
| `lead_time_days` | (If proposed date exists) planned construction window |
| `lifecycle_stage` | `active_dc_candidate`, `withdrawn_candidate`, `generic_generation_candidate`, `ambiguous_candidate` |
| `evidence_snippet_count` | Quick proxy for support robustness |

#### 9. MVP Storytelling Angles
Use `enriched_summary.json` + charts to answer:
* How many MW of confirmed data center load are active vs withdrawn?
* Average days_in_queue for confirmed DC projects vs generic generation.
* Which ISOs or states show longer queue latency (extend summarizer grouping if needed).
* Withdrawn share and (once root causes appear) leading withdrawal reasons.

#### 10. Extending to Dashboards
Feed `enriched_candidates.jsonl` into:
* A lightweight Streamlit or Dash prototype
* A BI tool (convert to Parquet/CSV) filtering `dc_confirmed_flag==True`

---
Future enhancements (see CLEAN_WORKFLOW.md) include timeline phase segmentation, geographic heatmaps, and confidence calibration plots.

### Clean Restart (Mini 30-Row Diversified Test)
Archive/remove prior artifacts and generate a fresh curated mini set with new heuristics:
```bash
python -m dc_queue.reset_pipeline_artifacts --archive
python -m dc_queue.fetch_queue_candidates --iso ALL --year-min 2020 --recall-bias --out dc_queue_pipeline_outputs/pipeline_candidates.jsonl
python -m dc_queue.prepare_llm_batches --candidates-json dc_queue_pipeline_outputs/pipeline_candidates.jsonl --output dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl --year 2025 --config-dir config
python -m dc_queue.profile_prepared --prepared dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl
python -m dc_queue.curate_mini_dataset --prepared dc_queue_pipeline_outputs/prepared_llm_candidates.jsonl --total 30 --min-high 4 --min-med 4
python -m dc_queue.make_llm_batches --prepared dc_queue_pipeline_outputs/mini_curated_30.jsonl --out-dir dc_queue_pipeline_outputs/llm_batches_mini --batch-size 15 --mix-mode round_robin --ensure-likely-dc 8 --strict-prompts --download-friendly
```
Then send the two prompt files for classification, validate, merge, and summarize as before.

#### Post-Retrieval Heuristic Filters (Optional)
Apply lightweight rules before model invocation to reduce token usage:
- Drop rows where `abs(capacity_mw - mentioned_capacity) / mentioned_capacity > 0.6`
- Prefer same state; penalize county mismatch unless only one candidate remains
- Boost recency (lower `queue_age_days` if available)

#### Potential Enhancements (Future Flags)
- `--llm-verify` (planned): run an LLM sanity pass to produce structured justification & false-positive reduction
- `--llm-match <article_jsonl>`: batch entity/timeline linking between articles and queue rows
- Optional capacity bucket customization (e.g. 25/75/150 MW)

#### Custom Capacity Buckets
To customize buckets now, you can fork and adjust the logic in `_mk_search` (capacity threshold section) inside `dc_queue/run.py`.

#### Including Additional Fields
If future adapters retain richer raw fields (e.g., substation name, phase status), extend `_mk_context` to append them; stable label prefixes keep embeddings robust.

#### Best Practices Summary
| Goal | Recommendation |
|------|----------------|
| Minimize tokens | Use `--llm-compact`, drop low-signal numeric columns from JSONL |
| Max recall | Enable `--recall-bias`, ingest all candidates (Tier LOW + MEDIUM + HIGH) |
| Precision triage | Filter to `dc_tier != 'REJECT'` and capacity ≥ threshold |
| Temporal focus | Use `--year-min 2020` (or relevant recent cutoff) |
| External linking | Use `llm_search_query` to drive article/API search |

---

Canonical Fields (partial):
`iso, iso_native_id, project_name, entity, request_type, request_category, capacity_mw, state, submitted_dt, proposed_in_service_dt, queue_age_days, in_service_lead_days, queue_status_raw, queue_status_class, status_stage, withdrawn_flag, canceled_flag, in_service_flag, dc_score, dc_flag, dc_flags_explain, dc_tier`

### Core Stages
1. Fetch: multi-ISO pull via `gridstatus` (PJM requires `PJM_API_KEY`).
2. Adapter Normalize: ISO-specific column extraction → canonical schema.
3. Enrich: add `cluster_weight`, `developer_category`, role bonuses/penalties.
4. Temporal: derive `queue_age_days`, `in_service_lead_days` (timezone-safe UTC).
5. Classify: advanced scoring (structural + textual + enrichment) with optional recall bias.
6. Tier: map score to `HIGH | MEDIUM | LOW | REJECT` for triage.
7. Status Lifecycle: normalize raw queue status to `queue_status_class`, flags (`withdrawn_flag`, `canceled_flag`, etc.), and funnel stage (`status_stage`).
8. Output: full CSV, candidate CSV, summary JSON.

### Lifecycle / Status Normalization
Added module `status.py` derives canonical lifecycle taxonomy:

| Column | Meaning |
|--------|---------|
| `queue_status_raw` | Raw text from ISO sheet (verbatim) |
| `queue_status_class` | Canonical bucket {active, withdrawn, canceled, suspended, in_service, unknown} |
| `status_stage` | Funnel stage {early, study, permitting, construction, operational, dead, unknown} |
| `withdrawn_flag` | True if withdrawn / removed |
| `canceled_flag` | Explicitly canceled / terminated |
| `suspended_flag` | Temporarily on hold |
| `in_service_flag` | Commercial / operational |
| `active_flag` | Still progressing (non-withdrawn/canceled/suspended) |
| `opposition_flag` | Placeholder (future: external signals / news NLP) |
| `status_notes` | Short derived explanation |

These fields enable attrition rate analytics (e.g., withdrawn vs total by county) and guide enrichment queries.

### LLM-Ready Preset (`--llm-ready`)
One-command bundling for downstream enrichment:
```
python -m dc_queue.run --iso ALL --year-min 2020 --llm-ready --recall-bias
```
Enables: compact LLM context, generation noise suppression, capacity floor (≥40 MW), lifecycle columns, curated JSONL export, and search-friendly query strings.

### Run (Recommended)
Recall-biased (max capture for later LLM pruning):
```bash
python -m dc_queue.run --iso ALL --limit 500 --recall-bias
```
Higher precision mode:
```bash
python -m dc_queue.run --iso ALL --limit 500
```

Outputs (default `dc_queue_pipeline_outputs/` unless overridden):
| File | Description |
|------|-------------|
| `pipeline_full.csv` | All processed rows with enrichment + scoring + tiers |
| `pipeline_candidates.csv` | Subset where `dc_flag == True` |
| `pipeline_summary.json` | Aggregated statistics & threshold metadata |

Summary JSON Keys:
`isos_processed, rows_total, candidates, candidate_pct, recall_bias, threshold, tiers_distribution, capacity_stats`

### Research-Oriented Signals & Slices

These new fields enable fast, programmatic branching of analysis (model training vs strategic scanning vs root-cause research):

| Field | Type | Meaning |
|-------|------|---------|
| `generation_noise_flag` | bool | Heuristic: row text/name looks like pure generation (solar / wind / storage) |
| `generation_override_flag` | bool | True if generation-looking row also contains strong DC override terms (kept as candidate) |
| `opposition_flag` | bool | Raw status / notes contain opposition / denial / protest keywords |
| `opposition_terms` | str | Comma-joined matched opposition keywords (debugging / transparency) |
| `capacity_band` | str | Capacity bucket used for attrition aggregation (0-50, 50-100, 100-200, 200-500, 500+) |
| `attrition_rate` | float | Laplace-smoothed withdrawn/canceled rate for (state, capacity_band) |
| `attrition_risk_tier` | str | {very_low, low, moderate, high, very_high, unknown} derived from `attrition_rate` thresholds |
| `status_reason` | str? | Heuristic reason extraction (economic, transmission, permitting, environmental, withdrawn_unspecified) |

#### How to Use the New Signals (Programmatic Filters)

Common research slices can be expressed directly (same logic the optional slice exporter uses):

1. Filter out likely false positives while keeping them for modeling (negative examples):
```
generation_noise_flag == True and generation_override_flag == False
```
2. Prioritize stable projects (lower local attrition risk + still active):
```
attrition_risk_tier in ['very_low','low'] and queue_status_class == 'active'
```
3. Investigate potential local pushback:
```
opposition_flag == True
```
4. Funnel targeted withdrawn root-cause follow-up (permits / transmission constraints):
```
withdrawn_flag == True and status_reason in ('permitting','transmission')
```

#### Generation Noise Handling (Suppress vs Tag)
Two mutually exclusive modes:
* `--suppress-generation-noise`: Drop rows classified as pure generation (fast precision boost, but loses negatives for model training).
* `--tag-generation-noise`: Keep rows and set `generation_noise_flag`; downstream filtering can exclude them while preserving data for future classifier refinement. If both flags supplied, tagging wins (no drops).

Override Behavior: If a row matches generation heuristics BUT also contains strong DC override terms (brand + data center token), `generation_override_flag` becomes True (kept as candidate regardless of suppression) and the row is not dropped.

#### Opposition & Attrition
`opposition_flag` / `opposition_terms` are simple keyword scans (e.g., "opposition", "protest", "denial", "lawsuit", "concern"). They are intentionally recall-biased. Future versions may externalize the term list into config and add NLP sentiment classification.

`attrition_rate` is computed with Laplace smoothing per (state, capacity_band): `(withdrawn+canceled + 1) / (total + 2)`. Tiers:
* `very_high` ≥ 0.60
* `high` ≥ 0.45
* `moderate` ≥ 0.30
* `low` ≥ 0.15
* `very_low` < 0.15
* `unknown` insufficient data / missing bucket

Use low/very_low + active for stable pipeline emphasis; investigate high/very_high clusters for systemic risk or permitting friction.

#### Status Reason Heuristic
`--infer-status-reasons` enables a lightweight pattern pass over `queue_status_raw` (lowercased) mapping to provisional buckets:
* economic → contains economics / cost / financial
* transmission → contains transmission / upgrade
* permitting → contains permit / zoning
* environmental → contains environment / wetland
* withdrawn_unspecified → contains withdraw (no clearer anchor)

These are placeholders for a future LLM-backed enrichment pass; absence of a match leaves `status_reason` null.

#### Automated Slice Export
Enable `--export-research-slices` to auto-write canonical subsets in `dc_queue_pipeline_outputs/slices/` (CSV + JSONL):
| Slice Filename Stem | Filter Logic |
|---------------------|--------------|
| `slice_generation_noise_negative` | generation_noise_flag == True and generation_override_flag == False |
| `slice_stable_active_low_attrition` | attrition_risk_tier in (very_low, low) and queue_status_class == 'active' |
| `slice_opposition` | opposition_flag == True |
| `slice_withdrawn_root_cause` | withdrawn_flag == True and status_reason in ('permitting','transmission') |

Each slice export respects `--json-cols` (if set) and also writes `slices_summary.json` with counts. Warning printed if `--limit` was applied (slices may be truncated).

Example combined run (recall-biased, LLM context, slices):
```bash
python -m dc_queue.run --iso ALL --year-min 2020 --recall-bias \
  --llm-context --llm-compact --tag-generation-noise \
  --infer-status-reasons --export-research-slices --limit 500
```

Resulting directories:
```
dc_queue_pipeline_outputs/
  pipeline_full.csv
  pipeline_candidates.csv
  pipeline_summary.json
  pipeline_full.jsonl (if JSON export enabled)
  slices/
    slice_generation_noise_negative.csv
    slice_stable_active_low_attrition.csv
    slice_opposition.csv
    slice_withdrawn_root_cause.csv
    slices_summary.json
```

Use these slices directly to seed downstream investigations (e.g., permitting bottleneck study, local opposition heatmap, negative training set curation).

### Scoring Highlights
Signals (illustrative — see `dc_load_classifier.py` for specifics):
- Structural: request/service type patterns, absence of generation fuel, capacity thresholds.
- Textual: strong vs weak DC terms, company/brand names, geographic DC cluster hits.
- Enrichment: `cluster_weight` scaling, `developer_category` bonus (hyperscale > colo > generic), penalties for generation-only hints.

Recall Bias Mode lowers the candidate threshold (e.g., 0.55 → 0.45) to minimize false negatives at the cost of more manual/LLM review.

### Migration from `dc_queue_filter.py`
`dc_queue_filter.py` is now a thin wrapper delegating to `dc_queue.run` and will be removed in a future cleanup.
Update automation to call:
```bash
python -m dc_queue.run --iso ALL --recall-bias --limit 500
```

### Configuration Files
| File | Purpose |
|------|---------|
| `config/cluster_locations.json` | Geo clusters with weights (e.g., Loudoun, Prince William, Santa Clara) |
| `config/developer_roles.json` | Categorizes developers (hyperscale, colocation, infrastructure) |
| `config/service_type_mapping.json` | (Planned) Maps ISO-specific service codes → canonical `request_category` |

Tuning: edit JSON, re-run pipeline; changes applied without code modification.

### Temporal Safety
All datetime computations forced to UTC-aware to avoid tz-naive subtraction errors (`queue_age_days` stability fix implemented 2025-09-26).

### Roadmap (Next Iterations)
1. Integrate `service_type_mapping.json` into adapters for deterministic `request_category` classification.
2. Add structured confidence breakdown (separate structural vs textual vs enrichment score components).
3. Introduce optional LLM validation layer (entity verification & generation disambiguation) behind `--llm-verify`.
4. Add caching & incremental diff mode (only new/changed queue rows).
5. Calibration notebook: precision/recall tuning with manually labeled sample.

### Quick Diff vs Legacy Tools
| Aspect | Old (`dc_queue_filter.py`) | New (`dc_queue.run`) |
|--------|---------------------------|-----------------------|
| Enrichment | Minimal | Geo clusters + developer roles + temporal |
| Tiers | No | Yes (`dc_tier`) |
| Temporal Features | None | `queue_age_days`, `in_service_lead_days` |
| Config JSON | Limited | Centralized + extensible |
| Wrapper Support | N/A | Legacy script delegates |
| Future LLM Hook | Manual | Planned flag + modular insertion |

### Troubleshooting
- Missing PJM data: ensure `PJM_API_KEY` exported (`echo $PJM_API_KEY`).
- SSL warnings: pass `--secure` to enforce verification, or omit for convenience (insecure mode prints a warning).
- Extremely high candidate rate (>85%): adjust threshold in classifier or remove recall bias.
- Empty outputs: network/API errors or schema shift; re-run with `--debug` and inspect intermediate CSV.

### Contributing
Open issues with: ISO name, sample row (sanitized), desired canonical mapping, and rationale for new signals.

---

## 🧪 Experimental: Interconnection Queue Data (Enhanced Tooling)

Two complementary utilities now support early-stage analysis of ISO interconnection queues for potential large data center (load) signals.

### 1. `quick_grid.py` (Fast CSV Slice)
Focused on rapid filtering & optional heuristic / config-driven classification.

Key Flags:
- `--classify` – adds dc_likelihood / dc_candidate columns (uses `dc_classifier` if present)
- `--candidates-only` – retain only rows flagged dc_candidate (implies `--classify`)
- `--export-json-meta` – writes per-ISO lightweight schema + 10-row sample to `data/queue_json/`

Example:
```bash
python quick_grid.py --iso ALL --limit 100 --classify --export-json-meta
```
Outputs:
- `quick_queue_filtered.csv` – full filtered set
- `quick_queue_filtered_slim.csv` – subset of key columns (when `--classify`)
- `data/queue_json/<iso>_queue_meta.json` – schema + sample snapshot

### 2. `export_iso_queue_json.py` (Schema & Raw JSON Export)
Purpose-built for deeper schema discovery and downstream normalization planning.

Flags:
- `--iso ALL` or comma list
- `--limit N` – limit normalized CSV rows (does not change raw schema)
- `--sample N` – number of sample rows in meta JSON (default 15)
- `--no-raw` – skip large raw JSON file (only meta + normalized)
- `--secure` – enable SSL verification (default disabled for convenience)

Example:
```bash
python export_iso_queue_json.py --iso PJM,SPP,NYISO --limit 250 --sample 20
```
Artifacts (written to `data/queue_json_full/`):
- `<iso>_meta.json` – field list, dtype map, sample rows (datetime-safe)
- `<iso>_raw.json` – full raw queue dump (omit with `--no-raw`)
- `<iso>_normalized.csv` – light canonical slice (`capacity_mw`, `state`, `submitted_dt`, etc.)

### Config‑Driven Classifier (`dc_classifier.py`)
Moves beyond embedded keyword lists. Loads brand & geo lists from `config/brands.json` and `config/cities.json` (with fallbacks). Scoring is additive + transparent:

| Component | Weight |
|-----------|--------|
| Strong DC term (e.g., "data center", "hyperscale") | 0.55 |
| Weak infra term (e.g., "campus", "facility") | 0.30 |
| Brand hit (AWS, Microsoft, Equinix, etc.) | 0.25 |
| Geo cluster hit (Ashburn, Loudoun, etc.) | 0.18 |
| Generation penalty (no strong DC term) | -0.50 |

Candidate condition: score ≥ 0.50 and not generation-only.

### Known Limitations (Active Work Area)
- Raw queue schemas differ significantly by ISO (naming, presence of load vs generation flags).
- Load vs generation determination still heuristic; needs structural signals (e.g., service type codes) integrated.
- No current disambiguation of utility-owned generation vs. dedicated campus load proxies.

### Planned Next Steps (Roadmap)
1. Field Frequency Analyzer – aggregate column presence across all ISOs (meta JSON driven).
2. Canonical Normalization Module (`queue_normalizer.py`) – ISO adapters mapping raw → canonical keys.
3. Enhanced Classifier – incorporate structural signals (request type, absence of fuel code) before textual.
4. Labeled Sample Set – manual annotation of ~100 rows for precision/recall tuning.
5. Enrichment – join county-level DC cluster index & operator category list.

### NEW: Advanced Data Center Queue Filter

`dc_queue_filter.py` performs a multi-stage pipeline:
1. Fetch raw queues (multi-ISO) via `gridstatus`.
2. Normalize into canonical fields (`queue_normalizer.py`).
3. Apply advanced recall-biased classifier (`dc_load_classifier.py`).
4. Emit full classified CSV, slim candidate CSV, and JSON summary.

Run (precision mode):
```bash
python dc_queue_filter.py --iso ALL --limit 500
```

Run (recall-biased mode – recommended for downstream LLM pruning):
```bash
python dc_queue_filter.py --iso ALL --limit 500 --recall-bias
```

Outputs (default `dc_queue_outputs/`):
- `dc_queue_classified_full.csv` – All normalized rows with `dc_score` & components.
- `dc_queue_candidates.csv` – Subset where `dc_flag` is True.
- `dc_queue_summary.json` – High-level stats (candidate percentage, capacity distribution, thresholds).

Classifier Signal Layers:
- Structural: service/request type contains load/network/service, absence of fuel with capacity, large MW thresholds (≥50 / ≥100).
- Textual: strong/weak DC phrases, brand hits, geo cluster hits.
- Penalty: explicit generation fuel without a load service anchor.

Bias Strategy:
- `--recall-bias` lowers threshold (0.55 → 0.45) to intentionally over-capture for later enrichment filtering (LLM + external sources).

Future Enhancements (post-normalization):
- Service type code dictionaries per ISO for deterministic load tagging.
- County-level DC cluster scoring & fiber/power corridor overlays.
- Historical conversion modeling (submitted → in-service lag) to weight recency.
- LLM verification pipeline: verify entity categories & weed out generation-only false positives.

### Contributing Feedback
If you have internal mappings (e.g., service type → load), open an issue or drop a structured snippet so we can incorporate into the normalization layer.

---

## 🧪 Experimental: Grid / Interconnection Queue Explorer

This repository now includes an optional complementary module (`grid_status_explorer.py`) that pulls ISO interconnection queue data (initial focus: **PJM**) using the open-source [`gridstatus`](https://github.com/kmax12/gridstatus) library. This data can help contextualize:

- Regional power capacity pipeline vs. hyperscale expansion
- Queue backlog / project aging as a proxy for grid constraints
- Technology mix (renewables, storage) potentially supporting future data center loads

### Why Separate?
The grid queue explorer is intentionally decoupled from the article-based project intelligence until alignment is established on how to correlate the datasets (e.g., geographic joins, power availability scoring, timeline merging). No assumptions are currently made about linkage.

### Install (if not already):
```bash
pip install gridstatus
```

### Run (Fetch PJM Queue):
```bash
python grid_status_explorer.py --iso PJM --limit 500
```
Options:
- `--iso` (default: PJM) – placeholder for future expansion
- `--limit` – truncate rows for a fast exploratory run
- `--output` – directory for artifacts (default: `grid_outputs`)

### Outputs
The script writes timestamped files to `grid_outputs/` (created if missing):

| File Type | Pattern | Description |
|-----------|---------|-------------|
| Raw CSV | `interconnection_queue_raw_<timestamp>.csv` | Full queue dataset from the ISO (unmodified) |
| Enriched CSV | `interconnection_queue_enriched_<timestamp>.csv` | Light temporal + capacity normalization subset |
| Summary JSON | `interconnection_queue_summary_<timestamp>.json` | Aggregated stats (year distribution, MW stats) |

### Example Summary Snippet
```json
{
  "generated_at": "2025-09-25T14:12:03Z",
  "iso": "PJM",
  "records": 8921,
  "year_distribution": {"2023": 812, "2024": 765},
  "capacity_stats": {"count": 8700, "sum_mw": 524000.5, "median_mw": 110.0, "p90_mw": 450.0, "max_mw": 1380.0}
}
```
(*Values illustrative; actual numbers depend on current queue data.)

### Roadmap Ideas (Not Yet Implemented)
- Geographic alignment to project locations (state/county normalization)
- Power availability score blending with article-derived timelines
- Filtering for high-MW renewable + storage near key hyperscale regions
- Temporal forecasting: interconnection to in-service conversion lag

If you want to prioritize any of these, open an issue with context.

---

### 🔐 PJM API Key (Required for PJM Queue)
Recent versions of `gridstatus` require a PJM API key for accessing certain PJM endpoints (including Queue Point data). You must have Account Manager API access provisioned for your user or a system account.

Steps:
1. Ensure your PJM Account Manager profile is active (you have a username like `bpulluta`).
2. Request API access (Account Manager API + Queue Point read scope). Your company account managers must approve.
3. Once the key is issued, set it locally:
  ```bash
  export PJM_API_KEY=YOUR_KEY_VALUE
  # or create .env (copy pjm.env.example) and add PJM_API_KEY=YOUR_KEY_VALUE
  ```
4. Run the explorer:
  ```bash
  python grid_status_explorer.py --filter-states VA,TX --min-mw 100 --parquet --limit 1000
  ```

Optional: supply inline without exporting
```bash
python grid_status_explorer.py --pjm-api-key YOUR_KEY_VALUE --min-mw 150 --filter-states VA,TX
```

Security Notes:
- Do NOT commit your real key (.env is git-ignored if you add it to .gitignore)
- Rotate keys per PJM security policies
- For multi-user environments, use a system account with least privilege

If the script exits with: `PJM API key missing or invalid` – confirm environment variable is exported in the *same* shell session (`echo $PJM_API_KEY`).

---

## ⚡ Fast Start: Minimal PJM Queue Pull

The former lightweight helper script `quick_grid.py` has been retired and archived. Use the ingestion stage of the new pipeline instead:
```bash
python -m queue_pipeline.qp ingest --iso ALL --year-min 2020 --limit 150 --recall-bias
```

### 1. Provide PJM API Key
Either export it for the session:
```bash
export PJM_API_KEY=YOUR_KEY_VALUE
```
Or create a `.env` file in the repo root:
```
PJM_API_KEY=YOUR_KEY_VALUE
```
(`quick_grid.py` will auto-load it if `python-dotenv` is installed; uncomment `python-dotenv` in `requirements.txt` if needed.)

### 2. Run a Simple Filter
```bash
python -m queue_pipeline.qp ingest --iso ALL --year-min 2020 --limit 500
```

### 3. Inspect Output
`quick_queue_filtered.csv` (overwritten each run) will contain only rows matching your filters (if those columns exist in the source feed).

### Script Behavior
- Auto-detects likely capacity (MW) column and queue submission date
- Adds normalized columns (`capacity_mw_norm`, `state_norm`, `year` when discoverable)
- Prints a concise summary (row counts, MW totals, years)
- Produces a single CSV: `quick_queue_filtered.csv`

Use the richer `grid_status_explorer.py` if you need caching, enrichment artifacts, summaries, or Parquet output.

### Handling Corporate / Self-Signed Certificates
If you hit SSL errors (certificate verify failed), you can retry once with:
```bash
python -m queue_pipeline.qp ingest --iso ALL --year-min 2020 --limit 500 --insecure
```
Warning: This disables SSL verification for the run. Use only if you trust the network environment.

---
## ⚡ ERCOT Interconnection MVP (Queue Load Focus)

An MVP script demonstrates ERCOT-specific large flexible load (LFL) style metrics using existing queue ingestion heuristics.

### Script
`queue_pipeline/tools/ercot_mvp.py`

### Metrics Produced
1. Total requested MW for ERCOT (sum of `capacity_mw` for ERCOT rows)
2. Heuristic LFL subset ("LFL Approx"): rows where
   - `iso == ERCOT`
   - `resource_type` in {`DATA_CENTER_CANDIDATE`, `LARGE_LOAD`}
   - Not `generation_flag` and not `transmission_flag`
   - Capacity ≥ `--min-lfl-capacity` (default 50 MW)
3. Average proposed in-service year across LFL Approx subset (if dates present)

### Run (use existing raw candidates first)
```bash
python queue_pipeline/tools/ercot_mvp.py --min-lfl-capacity 50
```

### Force Fresh ERCOT Fetch
```bash
python queue_pipeline/tools/ercot_mvp.py --fetch --queue-year-min 2020 --min-lfl-capacity 50
```

Example output:
```
=== ERCOT MVP Summary ===
Projects (all ERCOT): 14
Total Requested MW (approx): 1260.3
LFL Approx Projects: 0
LFL Approx Total MW: 0
Avg Proposed In-Service Year (LFL subset): N/A
Summary JSON -> queue_pipeline/qp_outputs/ercot_mvp_summary.json
```

If LFL counts are unexpectedly zero, lower threshold:
```bash
python queue_pipeline/tools/ercot_mvp.py --min-lfl-capacity 20
```
Or broaden heuristics during fetch (when using --fetch):
```bash
python queue_pipeline/tools/ercot_mvp.py --fetch --structural-broad --recall-experimental --min-lfl-capacity 40
```

### Output JSON
`queue_pipeline/qp_outputs/ercot_mvp_summary.json`:
```json
{
  "ercot_total_projects": 14,
  "ercot_total_requested_mw": 1260.3,
  "ercot_lfl_projects": 0,
  "ercot_lfl_total_mw": 0.0,
  "ercot_lfl_avg_in_service_year": null,
  "params": {"queue_year_min": 2020, "min_lfl_capacity": 50.0}
}
```

### Limitations
- LFL semantics approximated; no official ERCOT LLI classification parsed yet.
- Proposed in-service dates often sparse; average may be noisy.
- Does not yet segment by corridor (Houston / DFW / West Texas) or filter hydrogen/crypto explicitly.

### Roadmap Enhancements
1. Parse ERCOT Long-Term Load Interconnection (LLI) PDF/XLS directly for authoritative LFL labels.
2. Add classifier differentiating AI / hyperscale vs crypto vs hydrogen.
3. Add geographic clustering + transmission corridor MW aggregation.
4. Track delta vs previous month (time-series trend of load-side queue accumulation).
5. Integrate with batching/curation stage for targeted article retrieval about largest pending loads.

This MVP validates that the pipeline can isolate potential data center / large flexible load signals and produce investor/developer‑relevant early analytics for ERCOT.