Migration Guide - Hierarchical Data Priority

Date: 2025-11-22
Version: 2.0
Breaking Changes: YES

🚨 IMPORTANT CHANGES

The StarMaps repository now uses a hierarchical data priority system with ESO spectroscopy as the PRIMARY data source for SSZ validation.

Key Change:

OLD: GAIA-only (51% SSZ validation)
NEW: ESO primary (97.9% SSZ validation)

⚠️ BREAKING API CHANGES

1. Data Source Hierarchy

# OLD WAY (51% validation - NOT RECOMMENDED):
manager = CatalogManager()
stars = manager.fetch_nearby(distance_pc=100)
# Uses GAIA - only 51% for SSZ validation!

# NEW WAY (97.9% validation - RECOMMENDED):
manager = CatalogManager()
eso_data = manager.fetch_primary('sgr_a_stars')
# Uses ESO - 97.9% validation!

2. Purpose-Specific Methods

# For SSZ validation (HIGH priority):
eso_data = manager.fetch_primary('sgr_a_stars')  # 97.9%

# For infrared studies:
akari_map, wcs = manager.fetch_ir_map('G79.29+0.46', 'N60')

# For multi-frequency analysis:
m87_spectrum = manager.fetch_multifreq('M87')

# For positions ONLY (LOW priority for SSZ):
gaia_stars = manager.fetch_nearby(distance_pc=100)  # 51% for SSZ

📊 DATA HIERARCHY

Priority Levels:

1. PRIMARY (97.9%)     → ESO spectroscopy
   Use for: SSZ validation tests
   Method: fetch_primary()
   
2. IR DATA             → AKARI diffuse maps
   Use for: Nebula studies, temperature mapping
   Method: fetch_ir_map()
   
3. MULTI-FREQ          → NED continuum
   Use for: Jacobian tests, SEDs
   Method: fetch_multifreq()
   
4. AUXILIARY (51%)     → GAIA/SIMBAD
   Use for: Positions, astrometry ONLY
   Method: fetch_nearby(), fetch_named()

🔄 MIGRATION EXAMPLES

Example 1: SSZ Validation Script

BEFORE (❌ Wrong):

from ssz_starmaps.catalogs import CatalogManager
from ssz_starmaps.transform import transform_catalog

# OLD: Using GAIA (only 51% validation)
manager = CatalogManager()
stars = manager.fetch_nearby(distance_pc=100, max_stars=50)

# Apply SSZ transformation
ssz_stars = transform_catalog(stars)

# Validate (will only get ~51% success)
# ❌ NOT RECOMMENDED for validation!

AFTER (✅ Correct):

from ssz_starmaps.catalogs import CatalogManager
from ssz_starmaps.transform import transform_catalog

# NEW: Using ESO (97.9% validation)
manager = CatalogManager()
eso_data = manager.fetch_primary('sgr_a_stars')

# Apply SSZ transformation
ssz_data = transform_catalog(eso_data)

# Validate (will get ~97.9% success)
# ✅ RECOMMENDED for validation!

Example 2: Position-Only Queries

BEFORE (✅ Still works):

# For positions/astrometry (no change needed)
manager = CatalogManager()
stars = manager.fetch_nearby(distance_pc=100)
# This is FINE if you only need positions!

AFTER (✅ Same, but with clarity):

# For positions/astrometry (explicitly documented now)
manager = CatalogManager()
stars = manager.fetch_nearby(distance_pc=100)
# [!] Use for positions ONLY, NOT for SSZ validation!

Example 3: Multi-Source Workflow

NEW (✅ Recommended pattern):

manager = CatalogManager()

# Step 1: PRIMARY data for validation
eso_data = manager.fetch_primary('sgr_a_stars')  # 97.9%

# Step 2: IR data for temperature mapping
akari_map, wcs = manager.fetch_ir_map('G79.29+0.46', 'N60')

# Step 3: Multi-frequency for Jacobian tests
m87_spectrum = manager.fetch_multifreq('M87')

# Step 4: GAIA positions for spatial context
gaia_stars = manager.fetch_nearby(distance_pc=100)  # Positions only!

# Use the RIGHT data for the RIGHT purpose!

🎯 MIGRATION CHECKLIST

For Validation Scripts:

Replace fetch_nearby() with fetch_primary() for SSZ tests
Update validation expectations (51% → 97.9%)
Add comments explaining data source choice
Test with new ESO data
Update documentation

For Position Queries:

Keep using fetch_nearby() (no change)
Add comment: "For positions only"
Clarify in documentation
No code changes needed

For Multi-Source Analysis:

Identify data purpose (validation vs. positions vs. IR)
Use appropriate fetch method
Document data source choices
Test workflow

📝 CODE PATTERNS

Pattern 1: Check Data Availability

manager = CatalogManager()

# Check what's available
hierarchy = manager.get_data_hierarchy()

if hierarchy['primary']['available']:
    # Use ESO
    data = manager.fetch_primary('sgr_a_stars')
else:
    # Fallback to GAIA (positions only)
    data = manager.fetch_nearby(distance_pc=100)

Pattern 2: Print Usage Guide

manager = CatalogManager()

# Show user which data to use
manager.print_data_guide()

# Output:
# PRIMARY (97.9%): ESO spectroscopy
# Use fetch_primary() for SSZ validation
# ...

Pattern 3: Purpose-Specific Fetching

def get_validation_data():
    """For SSZ validation - use PRIMARY."""
    manager = CatalogManager()
    return manager.fetch_primary('sgr_a_stars')

def get_position_data():
    """For positions - use AUXILIARY."""
    manager = CatalogManager()
    return manager.fetch_nearby(distance_pc=100)

def get_ir_data():
    """For temperature mapping - use IR."""
    manager = CatalogManager()
    return manager.fetch_ir_map('G79.29+0.46', 'N60')

⚠️ COMMON PITFALLS

Pitfall 1: Using GAIA for SSZ Validation

# ❌ WRONG:
stars = manager.fetch_nearby(distance_pc=100)
validate_ssz(stars)  # Only 51% success!

# ✅ CORRECT:
eso_data = manager.fetch_primary('sgr_a_stars')
validate_ssz(eso_data)  # 97.9% success!

Pitfall 2: Not Checking Availability

# ❌ WRONG:
eso_data = manager.fetch_primary('sgr_a_stars')
# Might fail if Mass-Projection repo not accessible

# ✅ CORRECT:
hierarchy = manager.get_data_hierarchy()
if hierarchy['primary']['available']:
    eso_data = manager.fetch_primary('sgr_a_stars')
else:
    print("ESO data not available")
    print("Check Mass-Projection repo connection")

Pitfall 3: Mixing Data Sources

# ❌ WRONG:
eso_data = manager.fetch_primary('sgr_a_stars')
gaia_data = manager.fetch_nearby(distance_pc=100)
combined = pd.concat([eso_data, gaia_data])  # Different schemas!

# ✅ CORRECT:
# Use ESO for validation
eso_data = manager.fetch_primary('sgr_a_stars')
validate_ssz(eso_data)

# Use GAIA for positions separately
gaia_data = manager.fetch_nearby(distance_pc=100)
plot_positions(gaia_data)

🔍 BACKWARD COMPATIBILITY

What Still Works:

✅ fetch_nearby() - Works, but document purpose
✅ fetch_named_star() - No changes
✅ fetch_famous_stars() - No changes
✅ fetch_interesting_region() - No changes
✅ transform_catalog() - Works with all data

What's New:

🆕 fetch_primary() - ESO spectroscopy
🆕 fetch_ir_map() - AKARI infrared
🆕 fetch_multifreq() - NED multi-frequency
🆕 get_data_hierarchy() - Info method
🆕 print_data_guide() - Usage guide

📚 EXAMPLES

See the examples/ directory for complete examples:

examples/
├── example_eso_primary.py      - PRIMARY data usage
├── example_g79_workflow.py     - Multi-source workflow
└── example_m87_multifreq.py    - Multi-frequency analysis

Run examples:

python examples/example_eso_primary.py
python examples/example_g79_workflow.py
python examples/example_m87_multifreq.py

🐛 TROUBLESHOOTING

Problem 1: "ESO module not available"

Cause: astroquery not installed or Mass-Projection repo not accessible

Solution:

pip install astroquery

Or use included data:

eso_data = manager.fetch_primary('sgr_a_stars', use_included=True)

Problem 2: "AKARI module not available"

Cause: AKARI data files not downloaded

Solution: See akari_fetch.py for data acquisition instructions

Problem 3: "NED module not available"

Cause: astroquery not installed or no internet connection