Merge main into cellpack-analysis

[mogres]

Summary

Merges main into the long-lived cellpack-analysis fork branch, bringing main's
infrastructure (dependency upgrades, recipe-JSON server support, dedup hashing,
requirements.txt deprecation, etc.) into the fork without changing
downstream-observable packing behavior. The downstream cellpack-analysis
repo pins cellpack to f8b5500f (current cellpack-analysis HEAD), so the
goal was to preserve gradient math and output file naming exactly.

Base is cellpack-analysis (not main): this updates the fork.

Conflict resolutions (favoring downstream behavior)

• Gradient.py — kept the analysis behavior (probability-mixing
  combine_gradients, use_real_distances + decay validation). Left
  byte-identical to f8b5500f.
• validation/recipe_models.py — kept the analysis validation layout
  (gradient defaults in interface_objects/default_values.py). Fixed a
  silent-but-broken auto-merge by redirecting two main-authored imports of
  DEFAULT_GRADIENT_MODE_SETTINGS from recipe_models to interface_objects
  (a naive merge ImportErrors at startup otherwise).
• loaders/recipe_loader.py — took main's refactor (recipe-JSON input,
  _normalize_gradients); dropped the now-redundant analysis GradientData block.
• loaders/config_loader.py — kept the analysis parallel →
  number_of_processes rename; restored upload_results /
  open_results_in_browser defaults to True to match the downstream baseline;
  added a backward-compat shim mapping legacy parallel.
• writers/__init__.py — kept the analysis result-file naming (downstream
  globs results_*.simularium); layered in main's dedup_hash plumbing.
• deps — bioio (analysis) + requirements.txt deprecation (main)
  auto-merged cleanly.

Also: pack.py logs the resolved cellpack package path at DEBUG level (helps
confirm which cellpack build a run used).

Verification

• 93 cellpack non-AWS tests pass; imports clean; package compiles.
• Real-data integration packing (cell 743916, local meshes/grids) produces
  bit-identical packed positions vs the downstream pin f8b5500f for random,
  single-gradient, and multi-gradient combine recipes.
• Full downstream run_packing_workflow (generation + packing, random + gradient
  rules) runs green against this branch.
• main vs this branch: identical for random and single-gradient packings;
  differs only for multi-gradient combination — the intended divergence
  (probability mixing vs main's max-normalization). This branch matches
  f8b5500f exactly on that case.

Notes / follow-ups

• Re-pin the downstream cellpack-analysis repo's cellpack.rev only after this lands.
• Consider running uv lock before final merge.
• A complementary end-to-end (generation+packing) slow test was drafted in the
  downstream cellpack-analysis repo (not part of this PR).

🤖 Generated with Claude Code

[github-actions]

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Ingredient name	Encapsulating radius	Average number packed
ext_A	25	236.0

Packing image

Distance analysis

Expected minimum distance: 50.00
Actual minimum distance: 50.01

Ingredient key	Pairwise distance distribution
ext_A

[mogres]

We need to evaluate whether to keep maintaining this fork as a separate entity or if it makes sense to integrate into the current version of cellpack. Right now, this fork is used in mesoscope/cellpack-analysis as a pinned commit.

[Copilot]

Pull request overview

This PR merges main into the long-lived cellpack-analysis branch, bringing in upstream infrastructure changes (uv/pyproject-based deps, recipe-JSON server support, dedup hashing, docs updates, and related refactors) while aiming to preserve downstream packing behavior (notably multi-gradient combination behavior).

Changes:

• Deprecate exported requirements/**/requirements.txt in favor of pyproject.toml/uv, and update docs + Docker build accordingly.
• Add server-side recipe submission via JSON body + dedup hashing to avoid redundant packings, and plumb dedup identifiers through result upload paths.
• Expand recipe/config/docs schema support (description, randomness_seed) and add/adjust tests to cover the new loader + hashing behaviors.

Reviewed changes

Copilot reviewed 70 out of 77 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
requirements/windows/requirements.txt	Removed autogenerated platform requirements export (uv/pyproject flow).
requirements/macos/requirements.txt	Removed autogenerated platform requirements export (uv/pyproject flow).
requirements/linux/requirements.txt	Removed autogenerated platform requirements export (uv/pyproject flow).
README.md	Updates install instructions and expands recipe usage/docs pointers.
pyproject.toml	Adds dependencies/extras and removes requirements from packaged sources; tweaks Black config.
examples/recipes/v2/spheres_in_a_box.json	Adds recipe metadata (description).
examples/recipes/v2/peroxisome.json	Adds recipe metadata (description).
examples/recipes/v2/one_sphere.json	Updates example bounding box + radius values.
examples/recipes/v2/golgi_endosome.json	Adds recipe metadata (description).
examples/recipes/v2/er_peroxisome.json	Adds recipe metadata (description).
examples/recipes/v2/endosome.json	Adds recipe metadata (description).
examples/recipes/v2/BloodPlasma-no-HIV_fv2.1.json	Adds a new v2.1 example recipe.
examples/packing-configs/upload_and_open_result.json	Adds example config enabling upload + browser open.
examples/client-data/example_editable_fields.json	Adds example “editable fields” schema for Studio integration.
docs/STUDIO_SITE.md	Adds documentation for Studio site and publishing workflow.
docs/RECIPE_SCHEMA.md	Documents new top-level recipe fields (description, randomness_seed).
docs/INSTALLATION.md	Updates clone instructions to mesoscope/cellpack.
docs/index.md	Adjusts README include settings for docs build.
docs/DOCKER.md	Updates Docker docs to focus on ECS/server usage.
docs/CONTRIBUTING.md	Updates contributing instructions to use uv sync.
docs/CONFIG_SCHEMA.md	Updates config schema docs (but currently mismatches actual defaults).
docker/server.py	Switches packing endpoint to accept JSON body; adds dedup hashing + job status updates.
docker/entrypoint-batch.sh	Removes legacy batch entrypoint script.
docker/Dockerfile.ecs	Switches Docker build to uv sync with a server dependency group.
docker/Dockerfile.batch	Removes legacy batch Dockerfile.
cellpack/tests/test_recipe_version_migration.py	Updates RecipeLoader call sites for new signature.
cellpack/tests/test_recipe_loader.py	Adds tests for gradient normalization defaults + imports default settings.
cellpack/tests/test_partners.py	Minor formatting tweak (blank line).
cellpack/tests/test_pack_cli.py	Adds integration tests for pack() with recipe dict input + default config behavior.
cellpack/tests/test_gradient_data.py	Minor formatting tweak (blank line).
cellpack/tests/test_db_uploader.py	Extends uploader tests (description + job status updates).
cellpack/tests/test_db_recipe_loader.py	Extends DB recipe loader test data (description).
cellpack/tests/test_data_doc.py	Adds hash stability tests across key/list ordering rules.
cellpack/tests/test_aws_handler.py	Fixes tests to avoid writing into CWD by using tmp_path.
cellpack/tests/recipes/v2/test_peroxisome_combined_gradient.json	Adds a combined-gradient fixture recipe.
cellpack/tests/packing-configs/test_gradient_mixing.json	Adds a config fixture for gradient mixing tests.
cellpack/bin/upload.py	Extends upload CLI for Studio metadata + result upload + description propagation.
cellpack/bin/simularium_converter.py	Changes camera Z heuristic for converted trajectories.
cellpack/bin/pack.py	Allows recipe dict input; plumbs dedup hash into env + upload workflow.
cellpack/bin/cleanup_tasks.py	Removes Firebase cleanup task script.
cellpack/autopack/writers/init.py	Threads dedup hash into upload/open behavior.
cellpack/autopack/validation/recipe_models.py	Adds description/randomness_seed, allows float priority, removes gradient-combination validator.
cellpack/autopack/upy/simularium/simularium_helper.py	Refactors result upload/open; simplifies AWS upload; updates camera heuristic.
cellpack/autopack/upy/hostHelper.py	Minor formatting tweak (blank line).
cellpack/autopack/transformation.py	Minor formatting tweak (blank line).
cellpack/autopack/Serializable.py	Minor formatting tweak (blank line).
cellpack/autopack/MeshStore.py	Minor tuple unpacking style changes.
cellpack/autopack/loaders/recipe_loader.py	Accepts dict recipes; normalizes gradients + saves serializable snapshot.
cellpack/autopack/loaders/migrate_v1_to_v2.py	Normalizes v1 color values to 0–1 range when needed.
cellpack/autopack/loaders/config_loader.py	Adds legacy key migration (parallel → number_of_processes).
cellpack/autopack/IOutils.py	Minor formatting tweak (blank line).
cellpack/autopack/interface_objects/default_values.py	Removes results from firebase collection allow-list.
cellpack/autopack/interface_objects/database_ids.py	Adjusts AWS handler factory signature defaults.
cellpack/autopack/interface_objects/init.py	Minor formatting tweak (blank line).
cellpack/autopack/ingredient/Ingredient.py	Minor tuple unpacking style changes.
cellpack/autopack/GeometryTools.py	Minor formatting tweak (blank line).
cellpack/autopack/FirebaseHandler.py	Defaults to staging DB in non-interactive environments.
cellpack/autopack/Environment.py	Adds dedup_hash attribute to env.
cellpack/autopack/DBRecipeHandler.py	Adds dedup-hash normalization + job status + output upload workflow changes; removes DBMaintenance.
cellpack/autopack/BaseGrid.py	Minor formatting tweak (blank line).
cellpack/autopack/AWSHandler.py	Switches credential warning from print() to logging.
cellpack/autopack/Analysis.py	Minor formatting tweak (blank line).
cellpack/autopack/init.py	Adds AWS to server URL map; improves remote-db error message.
.github/workflows/make-requirements.yml	Removes workflow that generated platform requirements exports.
.github/workflows/cleanup-firebase.yml	Removes Firebase cleanup workflow.
.github/workflows/cleanup-aws.yml	Updates GitHub Actions versions.
.github/workflows/ci.yml	Updates GitHub Actions versions (checkout/codecov).
.github/workflows/cache-dependencies.yml	Updates GitHub Actions versions (checkout).
.github/workflows/build-docs.yml	Updates GitHub Actions versions (checkout/pages).
.github/workflows/analyze.yml	Updates action versions and switches PR commenting to gh CLI.
.github/workflows/add-issue-to-project.yml	Updates actions/add-to-project version.
.github/dependabot.yml	Broadens actions grouping and adds uv ecosystem updates.
.github/actions/uv/action.yml	Updates uv setup action version.
.github/actions/dependencies/action.yml	Updates uv setup action version.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.