Feature/civitai metadata

.coveragerc

@@ -0,0 +1,16 @@
+[run]
+branch = True
+# Point to the actual package directory (relative to repo root)
+source = saveimage_unimeta
+omit =
+    */tests/*
+    */__init__.py
+    saveimage_unimeta/defs/ext/generated_user_rules.py
+
+[report]
+show_missing = True
+skip_covered = True
+fail_under = 35
+exclude_lines =
+    pragma: no cover
+    if __name__ == .__main__.

.github/copilot-instructions.md

@@ -0,0 +1,66 @@
+# AI Assistant Project Instructions
+Authoritative onboarding for `ComfyUI_SaveImageWithMetaDataUniversal`. This repo is a Python custom-node pack (~220 tracked files excluding local-only `.venv`/outputs) targeting the ComfyUI runtime; edits usually run inside ComfyUI but must stay testable via pure Python.
+
+## Overview & Tech Stack
+- Purpose: `saveimage_unimeta/nodes/save_image.py` saves images with rich Automatic1111/Civitai-compatible metadata, hashes, workflow JSON, and filename tokens while degrading gracefully on JPEG limits.
+- Languages & tooling: Python 3.10+ (CI runs 3.10–3.13), Pillow/Numpy/Piexif, ComfyUI APIs (`folder_paths`, `nodes`, `execution`). Browser assets under `web/` use HTML/JavaScript for optional UI helpers. Linting via Ruff (`ruff.toml`), testing via Pytest (`tests/`), coverage tracked by `.coveragerc`.
+- Dependencies live in `requirements.txt` (runtime) and `requirements-test.txt` (adds pytest, ruff, coverage). Installing editable dev extras: `pip install -e .[dev]` (see `pyproject.toml`).
+
+## Repository Layout (edit here before searching)
+- Root: `saveimage_unimeta/` (core package), `tests/` (unit suites hitting public APIs), `docs/` (JPEG fallback, workflow compression, futures), `example_workflows/`, `user_rules/` (shipped examples), `.github/workflows/*.yml` (CI/publish), `web/` (static JS/CSS helpers referenced by ComfyUI UI), `ruff.toml`, `.pre-commit-config.yaml`.
+- `saveimage_unimeta/defs/`: canonical capture metadata definitions, sampler heuristics, combo helpers, plus `ext/` for shipped extensions. Never bypass these when adding fields.
+- `saveimage_unimeta/defs/validators.py`: prompt validators trace upstream from sampler inputs. Standard dual-path conditioning nodes and guiders are routed generically by matching positive/negative-style input names; keep explicit overrides in `defs/samplers.py` only for non-standard guider input names.
+- `saveimage_unimeta/nodes/`: save node + supporting UI tooling (scanner, rule writers, extra metadata nodes, test stubs). Each node must keep tooltip text ≤140 chars (see `pyproject` metadata for copy text).
+- `saveimage_unimeta/capture.py`: merges defs + `user_rules` + `FORCED_INCLUDE_CLASSES`, normalizes prompts, ensures `Metadata generator version` is always last. Relies on `saveimage_unimeta/hook.py` to read the active ComfyUI prompt cache (tests shim this when `METADATA_TEST_MODE=1`).
+- `saveimage_unimeta/trace.py`: BFS graph traversal + sampler selection heuristics (exact match in `defs/samplers.py`, else `MetaField.SAMPLER_NAME` or `{STEPS, CFG_SCALE}` hint). `Trace.filter_inputs_by_trace_tree` guarantees deterministic ordering upstream of capture.
+- `saveimage_unimeta/utils/`: hashing primitives (`hash.py`), LoRA/embed utilities, logging helpers (`color.py`), workflow redaction (`redaction.py`, bounded secret/path sanitization), and filename safety (`pathsafety.py`). Always use these helpers—no ad-hoc hashing/log formatting, prefer `pathresolve` for filesystem work, and route metadata/redaction and filename sanitization through the shared helpers.
+- `web/`: static TypeScript/JavaScript snippets for optional UI affordances (see `web/js/`); keep them aligned with node parameter expectations when changing UI-visible behavior.
+
+## Data Flow & Runtime Contracts
+1. `Trace.trace` builds a distance map from the save node back through executed nodes; `sampler_selection_method` (UI) controls farthest/nearest/explicit traversal.
+2. `Capture.gen_pnginfo_dict` / `.gen_parameters_str` iterate that ordering, apply rule merges, sanitize prompts, and append deterministic fields (metadata generator version last).
+3. `saveimage_unimeta/nodes/save_image.py` writes PNGInfo or EXIF/WebP metadata, attempts JPEG EXIF up to `max_jpeg_exif_kb` (≤64 KB enforced). `_last_fallback_stages` mirrors whichever fallback stage fired. Before embedding, the `sanitize_metadata` toggle (default on) passes `prompt`/`extra_pnginfo` through `utils/redaction.py` (fail-open: falls back to raw on `MetadataSanitizationError`), and the expanded filename template is always passed through `utils/pathsafety.py` (`sanitize_filename`).
+4. JPEG fallback follows the multi-stage pipeline documented in `.github/instructions/python.instructions.md`; `_last_fallback_stages` mirrors whichever stage triggered so downstream tests can assert the markers.
+5. Hashing: `saveimage_unimeta/defs/formatters.py` (via helpers in `saveimage_unimeta/utils/hash.py`) caches full SHA256 hashes in `.sha256` sidecars which are truncated to 10 chars when written to metadata; `METADATA_FORCE_REHASH=1` invalidates caches. Hash log verbosity is controlled via `METADATA_HASH_LOG_MODE` and `METADATA_HASH_LOG_PROPAGATE`.
+
+## Runtime Integration & Environment Flags
+- Runs embedded in ComfyUI; expect access to `folder_paths`, sampler nodes, and numerous other `comfy.` imports, and PIL. When unit testing, `saveimage_unimeta/piexif_alias.py` and `hook.py` provide safe stubs—only extend the stub surface actually required.
+- Runtime feature flags (read at execution time, no restart needed) include: `METADATA_TEST_MODE`, `METADATA_NO_HASH_DETAIL`, `METADATA_NO_LORA_SUMMARY`, `METADATA_FORCE_REHASH`, `METADATA_HASH_LOG_MODE`, `METADATA_HASH_LOG_PROPAGATE`, `METADATA_DUMP_LORA_INDEX`, `METADATA_DUMP_CHECKPOINT_INDEX`, `METADATA_DUMP_UNET_INDEX`, `METADATA_ENABLE_TEST_NODES`, `METADATA_DEBUG_PROMPTS`. UI checkbox `include_lora_summary` overrides the env flag.
+- JPEG/env documentation source of truth: `docs/JPEG_METADATA_FALLBACK.md`, `docs/WORKFLOW_COMPRESSION_DESIGN.md`, `docs/FUTURE_AND_PROTOTYPES.md`. Update both docs + this file when behavior changes.
+
+## Build, Lint, Test (validated locally and mirrored by CI)
+1. **Bootstrap** (from repo root, Python ≥3.10):
+	```cmd
+	python -m venv .venv
+	.venv\Scripts\activate
+	python -m pip install --upgrade pip
+	pip install -e .[dev]
+	```
+	(Alternatively, `pip install -r requirements.txt -r requirements-test.txt`.)
+2. **Lint**: `ruff check .` (configured by `ruff.toml`; CI fails on lint). Optional: `pre-commit run --all-files` (hooks defined in `.pre-commit-config.yaml`).
+3. **Unit tests**: `pytest -q` (Pytest auto-discovers under `tests/`). Set `METADATA_TEST_MODE=1` to match CI matrix determinism. Coverage is gathered in CI via `coverage run -m pytest -q`; run locally when touching core pipeline.
+4. **Workflow/CLI tests**: optional but recommended before shipping metadata format changes. Use `python tests/tools/run_dev_workflows.py --comfyui-path "<your ComfyUI root>" [--workflow-dir ...]` (see `tests/comfyui_cli_tests/DEV_WORKFLOW_TESTING.md` + `ignore/DEV_WORKFLOW_TESTING.md`). Always ensure workflows are in API format and clean outputs with `--temp-dir` or manual deletion.
+5. **Integration sanity**: when touching JPEG fallback, temporarily set `max_jpeg_exif_kb=8` via the node UI or JSON to coerce fallback coverage; inspect `_last_fallback_stages` and resulting metadata strings to confirm markers append exactly once.
+6. **CI awareness**: `.github/workflows/unimeta-ci.yml` is the primary matrix workflow (Ruff + Pytest across Python 3.10–3.13 with both metadata modes, plus lint-only jobs). `.github/workflows/ci.yml` remains from the old template and still targets the `main` branch—confirm no external automation depends on it before deleting or updating the triggers. `publish_action.yml` handles release packaging. Match local tooling to avoid failures.
+
+## Rules, Scanner & User Overrides
+- `saveimage_unimeta/nodes/scanner.py` inspects installed node classes and suggests capture rules. `rules_save.py` + `rules_view.py` manage JSON/Python persistence by writing the JSON blobs under `saveimage_unimeta/user_rules/`. `saveimage_unimeta/nodes/rules_writer.py` consumes those JSON files to regenerate `saveimage_unimeta/defs/ext/generated_user_rules.py`. After editing `defs/captures.py`, re-run scanner + saver so automated tests (`tests/test_generated_user_rules.py`) stay in sync. When users trigger node-driven backups, the `saveimage_unimeta/user_rules/backups/` folder may contain timestamped copies of the JSON files and the generated module.
+- `rules_writer.py` stamps a `RULES_VERSION` constant into generated modules; `defs.load_user_definitions` caches it as `LOADED_RULES_VERSION`. `save_image.py` logs a one-time warning when the saved rules are missing or outdated—ask users to re-run `Metadata Rule Scanner` + `Save Custom Metadata Rules` or execute `example_workflows/refresh-rules.json` after updates.
+- `Metadata Force Include` node feeds `FORCED_INCLUDE_CLASSES` used during capture merge; keep manual overrides inside `saveimage_unimeta/user_rules/` so merges remain deterministic.
+- Manual capture additions must update: `defs/captures.py`, `_build_minimal_parameters` (only if the field must survive minimal fallback), docs (README + `docs/...`), targeted tests (e.g., `tests/test_capture_core.py`, `tests/test_guidance_and_exif_fallback.py`).
+
+## Conventions & Safety Nets
+- `.github/instructions/python.instructions.md` is the authoritative source for coding conventions, logging patterns, runtime import guards, metadata ordering, JPEG fallback behavior, filename token safety, helper usage, UI override precedence, sanitization rules, and artifact locations. Follow it whenever editing `.py` files.
+- `.github/instructions/comfy.instructions.md` documents the ComfyUI manifest contract (`__init__.py` exports, `NODE_CLASS_MAPPINGS`, `NODE_DISPLAY_NAME_MAPPINGS`) and protocol expectations (`CATEGORY`, `RETURN_TYPES`, `INPUT_TYPES`, `FUNCTION`, tuple returns). Reference it when adding or modifying nodes.
+
+## Integration Resources & Troubleshooting
+- **Docs**: `docs/JPEG_METADATA_FALLBACK.md`, `docs/MIGRATIONS.md`, `docs/V3_SCHEMA_MIGRATION.md` (for future migration to V3; no specific timeline for implementing this yet), `docs/WAN22_SUPPORT.md`, `docs/FUTURE_AND_PROTOTYPES.md` (historical context). Keep them synchronized with behavior changes.
+- **Workflow samples**: `example_workflows/*.json` showcase Force Include, extra metadata, LoRA stacks, WAN/FLUX flows. Use them to reproduce bugs quickly.
+- **Testing aids**: `saveimage_unimeta/nodes/testing_stubs.py` exposes lightweight sampler nodes when `METADATA_ENABLE_TEST_NODES=1`; `tests/` contains stub fixtures demonstrating how to patch ComfyUI APIs.
+- **Troubleshooting tips**: enable `METADATA_DEBUG_PROMPTS=1` to log prompt aliasing, drop `max_jpeg_exif_kb` to 8 to hit fallback paths, set `METADATA_NO_HASH_DETAIL=1` or `METADATA_NO_LORA_SUMMARY=1` to verify UI overrides, and use `METADATA_DUMP_LORA_INDEX`, `METADATA_DUMP_CHECKPOINT_INDEX`, or `METADATA_DUMP_UNET_INDEX` to dump the first-built indexes for diagnostics. Hash mismatches? delete `.sha256` sidecars or set `METADATA_FORCE_REHASH=1`.
+
+## Working Style & Search Discipline
+- Start from this file: it summarizes architecture, commands, and directory hotspots—search the codebase only if something here is missing or inaccurate. When in doubt, inspect `saveimage_unimeta/` modules referenced above before global greps.
+- Keep diffs surgical: modify only the modules relevant to your change, maintain doc parity (README + docs + this file), and update/extend tests covering the touched behavior. CI enforces Ruff + Pytest; aim to replicate locally before pushing.
+- Document new env flags, workflow parameters, or fallback behaviors immediately here and in the README/doc section they affect. Avoid conflicting guidance—the coding agent will obey the strictest rule present.
+- Trust these instructions. Only run exploratory searches if the required information isn’t covered or appears outdated, and if you discover drift, update this file as part of your change.

.github/instructions/comfy.instructions.md

@@ -0,0 +1,89 @@
+---
+description: "ComfyUI protocol guarantees and manifest/registration contract."
+applyTo: "**/*.py"
+---
+
+# GitHub Copilot Instructions for `ComfyUI_SaveImageWithMetaDataUniversal`
+
+## How This File Fits With Other Instructions
+
+- Use `.github/copilot-instructions.md` for the full repo tour (layout, data flow, build/test commands, env flags).
+- Use `.github/instructions/python.instructions.md` for every Python-convention question (ruff expectations, logging, metadata ordering, JPEG fallback, helper usage). That file is the authority on code style and testing rules.
+- This document focuses solely on **ComfyUI protocol guarantees** and the **manifest/registration contract** that every node must respect.
+
+## Architecture Layers (ComfyUI View)
+
+1. **Registration Layer (`__init__.py`)** – Treat this as the manifest. It should primarily import concrete node classes, declare `NODE_CLASS_MAPPINGS`, `NODE_DISPLAY_NAME_MAPPINGS`, optional `WEB_DIRECTORY`, and export the curated `__all__`. Controlled side effects (e.g., gating test nodes on `METADATA_ENABLE_TEST_NODES` or logging a one-time startup banner) are acceptable, but keep them lightweight and well-documented.
+2. **Logic Layer (`saveimage_unimeta/`)** – All node behavior, utilities, capture/trace helpers, and metadata plumbing live inside this package. New functionality must be implemented here and tested via `pytest` as described in the Python instructions.
+3. **Presentation Layer (`web/`)** – Houses any optional ComfyUI frontend extensions. Keep the JS aligned with node INPUT_TYPES and update the manifest’s `WEB_DIRECTORY` when shipping UI assets.
+
+## ComfyUI Protocol Requirements
+
+When writing or modifying nodes, enforce these protocol rules to stay compatible with ComfyUI and downstream forks:
+
+- Define the static protocol fields on every node class: `CATEGORY`, `RETURN_TYPES`, `FUNCTION`, and, when applicable, `OUTPUT_NODE` or `OUTPUT_IS_LIST`.
+- Implement `@classmethod INPUT_TYPES(cls)` returning a dict with `"required"`, `"optional"`, and `"hidden"` keys. Ensure magic strings (e.g., `"IMAGE"`, `"STRING"`, `"UNIMETA_METADATA"`) match the runtime’s expectations.
+- The method referenced by `FUNCTION` must accept parameters that line up exactly with `INPUT_TYPES` and always return a tuple. Saver nodes typically `return ()` to signal “work complete, no tensors produced.”
+- Hidden inputs for prompt/workflow/state (`"PROMPT"`, `"EXTRA_PNGINFO"`, etc.) must be threaded through to capture utilities rather than reinventing workflow parsing.
+
+## Registration Checklist
+
+Whenever a new node class is introduced under `saveimage_unimeta/nodes/`:
+
+1. Import the class inside the root `__init__.py` without triggering runtime-only dependencies during import.
+2. Add the class to `NODE_CLASS_MAPPINGS` using the ComfyUI identifier as the key.
+3. Provide a concise (<140 char) entry in `NODE_DISPLAY_NAME_MAPPINGS` so UI menus look polished.
+4. Append the class name to `__all__` to keep static analyzers in sync.
+5. If the node requires web assets, expose them via `WEB_DIRECTORY` and keep the folder paths stable for ComfyUI.
+
+## Gold-Standard Node Outline
+
+New nodes should resemble the structure below. It highlights the ComfyUI-specific plumbing this document governs; read the Python instructions for broader style and metadata rules.
+
+```python
+import torch
+from comfy.cli_args import args
+
+
+class SaveImageWithMetaDataUniversal:
+    """Canonical saver node registered via NODE_CLASS_MAPPINGS."""
+
+    CATEGORY = "image/metadata"
+    RETURN_TYPES: tuple[str, ...] = ()
+    FUNCTION = "save_images"
+
+    @classmethod
+    def INPUT_TYPES(cls):
+        return {
+            "required": {
+                "images": ("IMAGE",),
+                "filename_prefix": ("STRING", {"default": "ComfyUI"}),
+            },
+            "optional": {
+                "extra_metadata": ("UNIMETA_METADATA",),
+                "metadata_rules": ("UNIMETA_RULES",),
+            },
+            "hidden": {
+                "prompt": "PROMPT",
+                "extra_pnginfo": "EXTRA_PNGINFO",
+            },
+        }
+
+    def save_images(
+        self,
+        images: torch.Tensor,
+        filename_prefix: str,
+        extra_metadata: dict | None = None,
+        metadata_rules: dict | None = None,
+        prompt: dict | None = None,
+        extra_pnginfo: dict | None = None,
+    ) -> tuple:
+        if args.dont_save_previews:
+            return ()
+
+        # Business logic defined in saveimage_unimeta/nodes/save_image.py
+
+        return ()
+```
+
+Use this outline to validate that protocol attributes, INPUT_TYPES, and return signatures remain aligned whenever you extend the saver or add supporting nodes.