Feature/dedup wrapper v2

.github/workflows/build-and-publish.yaml

@@ -3,10 +3,10 @@ name: Build and Publish to PyPI
 on:
   push:
     tags:
-      - 'v*'
+      - "v[0-9]+.[0-9]+.[0-9]+"
 
 jobs:
-  build:
+  test:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
@@ -19,9 +19,35 @@ jobs:
       - name: Install Poetry
         run: pipx install poetry
 
-      - name: Set version from tag
-        if: startsWith(github.ref, 'refs/tags/v')
-        run: poetry version "${GITHUB_REF_NAME#v}"
+      - name: Check tag matches pyproject version
+        run: |
+          PYPROJECT_VERSION="$(poetry version -s)"
+          TAG_VERSION="${GITHUB_REF_NAME#v}"
+          if [ "$PYPROJECT_VERSION" != "$TAG_VERSION" ]; then
+            echo "Tag v$TAG_VERSION does not match pyproject version $PYPROJECT_VERSION" >&2
+            exit 1
+          fi
+
+      - name: Install dependencies
+        run: poetry install --all-extras
+
+      - name: Run tests
+        run: poetry run pytest
+
+  build:
+    runs-on: ubuntu-latest
+    needs:
+      - test
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+          cache: 'pip'
+
+      - name: Install Poetry
+        run: pipx install poetry
 
       - name: Build distributions
         run: poetry build
@@ -35,7 +61,7 @@ jobs:
   publish-to-test-pypi:
     environment:
       name: testpypi
-      url: https://pypi.org/project/epoch8-smartfeed/
+      url: https://test.pypi.org/project/epoch8-smartfeed/
     permissions:
       id-token: write
     runs-on: ubuntu-latest
@@ -54,6 +80,7 @@ jobs:
   publish-to-pypi:
     environment:
       name: pypi
+      url: https://pypi.org/project/epoch8-smartfeed/
     permissions:
       id-token: write
     runs-on: ubuntu-latest

.github/workflows/lint.yaml

@@ -5,16 +5,17 @@ on:
     paths:
       - ".github/workflows/lint.yaml"
       - "smartfeed/**"
+      - "tests/**"
       - "pyproject.toml"
+      - "poetry.lock"
+  pull_request:
 
 jobs:
   lint:
     runs-on: ubuntu-latest
     strategy:
       matrix:
         python-version:
-          # - "3.8"
-          # - "3.9"
           - "3.10"
 
     steps:
@@ -37,10 +38,16 @@ jobs:
         run: |
           poetry install --all-extras
 
-      - name: Lint with mypy
+      - name: Lint with ruff
         run: |
-          poetry run mypy smartfeed
+          pip install ruff==0.15.20
+          ruff check smartfeed tests
+
+      - name: Type-check with pyright
+        run: |
+          pip install pyright==1.1.409
+          pyright --pythonpath "$(which python)" smartfeed tests
 
       - name: Lint with black
         run: |
-          poetry run black --check smartfeed
+          poetry run black --check smartfeed tests

.github/workflows/tests.yaml

@@ -5,26 +5,23 @@ on:
     paths:
       - ".github/workflows/tests.yaml"
       - "smartfeed/**"
+      - "tests/**"
       - "pyproject.toml"
+      - "poetry.lock"
+  pull_request:
 
 jobs:
   test:
     runs-on: ubuntu-latest
     strategy:
       matrix:
         python-version:
-          # - "3.8"
           - "3.9"
           - "3.10"
+          - "3.11"
           - "3.12"
           - "3.13"
 
-    services:
-      redis:
-        image: redis
-        ports:
-          - 6379:6379
-
     steps:
       - uses: actions/checkout@v3
 

.gitignore

@@ -8,7 +8,7 @@ __pycache__/
 local_settings.py
 db.sqlite3
 media
-
+frontend/
 ### Linux ###
 *~
 
@@ -210,4 +210,4 @@ ENV/
 /site
 
 # Poetry Lock
-*.lock
+*.lock

ARCHITECTURE.md

@@ -0,0 +1,161 @@
+# SmartFeed Architecture
+
+## Overview
+
+SmartFeed builds a paginated feed from multiple async data sources using a tree config.
+
+```
+FeedManager.get_feed(session_id, limit, cursor)
+  -> executor.run(root_node, ctx, limit, cursor)
+    -> tree of nodes execute in parallel
+  -> stamp smartfeed_position on results
+  -> return FeedResult
+```
+
+## Node Types
+
+Three types of nodes:
+
+```
+SubFeed (leaf)      -- calls async method from methods_dict
+Wrapper (pipeline)  -- fetch -> dedup -> rerank -> cache -> paginate
+Mixer (coordinator) -- runs children in parallel, assembles results
+```
+
+## Execution Flow
+
+```
+                    FeedManager
+                        |
+                    executor.run()
+                        |
+              +---------+---------+
+              |                   |
+         node.execute()    node.build_mix_plan()
+         (SubFeed,Wrapper)      (Mixers)
+                                  |
+                          asyncio.gather(children)
+                                  |
+                           plan.assemble()
+```
+
+### executor.py
+
+Module-level functions (no class):
+
+- `run(node, ctx, limit, cursor)` -- dispatches on `isinstance(node, MixerNode)`: mixers build a MixPlan, leaf/pipeline nodes execute()
+- `_execute_mix(plan, ctx, cursor)` -- runs MixPlan children in parallel via `asyncio.gather`
+
+### Wrapper Pipeline
+
+```
+fetch child (session_size or limit)
+  -> dedup (priority arbitration, seen-set from Redis)
+  -> rerank (call methods_dict callable)
+  -> cache (write to Redis with TTL)
+  -> paginate (slice by page from cursor)
+```
+
+Two paths:
+- **Cached** (`self.cache` set): warm = read cache + paginate. Cold = full pipeline.
+- **Passthrough** (no cache): fetch + dedup (with Redis seen-set) + rerank per page.
+
+### MixPlan
+
+Mixers return `MixPlan(children, assemble)`:
+
+```python
+MixChild(node_id="mix_0", node=subfeed_a, demand=8)   # ask for 8 items
+MixChild(node_id="mix_1", node=subfeed_b, demand=12)  # ask for 12 items
+```
+
+Executor runs all children via `asyncio.gather`, passes results to `assemble(buffers, child_cursors)`.
+
+## Key Models
+
+### Config (Pydantic v2)
+
+```
+FeedConfig(version, feed: FeedNode)
+FeedNode = Union[SubFeed, Wrapper, MergerPercentage, MergerPositional, ...]
+```
+
+Discriminated union on `type` field. Forward refs rebuilt at import time.
+
+### Runtime
+
+```
+ExecutionContext(session_id, methods_dict, redis)
+FeedResult(data: list, next_page: dict, has_next_page: bool)
+```
+
+### Output
+
+Items are plain dicts; each carries a `_smartfeed_debug_info` dict bundle
+(source, smartfeed_position, rerank_position when reranked -- all 0-based).
+
+## Redis State
+
+```
+sf:{session_id}:{cache_key or node_id}:{config_hash}           -- cached session batch (Redis LIST of orjson items)
+sf:{session_id}:{cache_key or node_id}:{config_hash}:meta      -- metadata (gen, child_cursor, child_has_next)
+sf:{session_id}:{cache_key or node_id}:{config_hash}:coldlock  -- cold-build lock (SETNX, ttl 10s)
+sf:{session_id}:{node_id}:{config_hash}:seen                   -- session-scoped dedup seen-set (Redis SET)
+sf:{session_id}:{cache_key}:{child_hash}:{segment}       -- shared base segment (blob, per continuation window)
+sf:{session_id}:{cache_key}:{child_hash}:{segment}:meta  -- shared segment meta (child cursor / has_next)
+sf:{session_id}:{cache_key}:{child_hash}:{segment}:lock  -- shared segment cold-build lock (SETNX, ttl 10s)
+```
+
+Warm page reads are one MULTI/EXEC pipeline: GET meta + LLEN + LRANGE of the page
+window + EXPIRE on data/meta/seen -- an atomic snapshot that transfers only the
+requested window, never the whole batch.
+
+TTL = inactivity timeout. Refreshed on every access via pipeline EXPIRE.
+Cursor for a cached wrapper is `{node_id: {offset, gen}}` (absolute offset).
+
+## Dedup
+
+Two dedup paths, unified `_dedup(data, seen=None)` method. Both fetch exactly the
+outstanding deficit and refill until the target is filled or the child is exhausted
+(no over-fetch, so no item loss; no early give-up, so no short pages), and both carry
+a session-scoped Redis seen-set so dedup holds across the whole scroll:
+
+1. **Cached path** (`_fetch_and_dedup` / `_cold_build`): seen-set carried across cold
+   rebuilds, reset on a fresh scroll (empty cursor).
+2. **Passthrough path** (`_passthrough`): seen-set persisted across pages; SADD after each page.
+
+Priority arbitration: higher `dedup_priority` wins even when the higher-priority copy is
+not first-seen; equal priority = first-seen.
+
+## Config Hash
+
+`BaseNode.config_hash()` = md5(json.dumps(model_dump(), sort_keys=True, default=str))[:8],
+memoized per node instance (config is immutable after validation).
+
+Used in Redis keys. Any config change = different hash = fresh cache.
+
+## Generation ID
+
+Cached wrapper stamps `gen` (random nonce) in cursor and `:meta`. Stale gen = rebuild from scratch.
+
+## Shared Cache
+
+Two wrappers with same `cache_key` share base data. Each applies its own rerank. RedisLock prevents duplicate fetches.
+
+## File Structure
+
+```
+smartfeed/
+  models/
+    base.py         -- BaseNode, MixerNode, FeedResult, config_hash
+    subfeed.py      -- SubFeed (leaf)
+    wrapper.py      -- Wrapper (cache + dedup + rerank pipeline)
+    mixers.py       -- Percentage, Positional, Append, Distribute, Gradient
+    __init__.py     -- FeedNode union, FeedConfig, exports
+  execution/
+    executor.py     -- run(), _execute_mix() (module-level functions)
+    context.py      -- ExecutionContext
+    plans.py        -- MixPlan, MixChild
+    redis_lock.py   -- RedisLock (SETNX async context manager)
+  manager.py        -- FeedManager (public entry point)
+```

CHANGELOG.md

@@ -1,7 +1,61 @@
-# 0.2.0 (2025-11-25)
+# Changelog
+
+## 1.0.0 (2026-07-07)
+
+SmartFeed v2: complete rewrite. Changes below are relative to the last published release, **0.2.0**. (The pre-rewrite internal working tree contained additional intermediate modules — never part of a published release — that are not listed here.) See [MIGRATION.md](MIGRATION.md) for step-by-step upgrade instructions.
+
+### New Features
+
+- **Wrapper** with optional pipeline stages: cache, dedup, rerank
+- **Session-scoped dedup contract**: within one continuous scroll a user sees each item at most once, no matter how far they scroll; an empty cursor starts a fresh scroll
+- **Cross-page dedup** via Redis seen-set (SET with TTL)
+- **Deficit-based refill**: dedup returns full pages without discarding unique items (both cached and passthrough paths)
+- **Rerank callable** registered in `methods_dict`, referenced by `method_name` in config
+- **Shared cache** via `cache_key` for A/B testing with different rerankers
+- **Generation ID** for stale cursor detection
+- **TTL touch** (inactivity timeout) on every cache access
+- **`raise_error`** on WrapperRerank: fail-hard or fail-soft mode
+- **`_smartfeed_debug_info`** stamped automatically: source, positions, per-wrapper data (plain dicts, all positions 0-based)
+- **`config_hash`** for automatic cache invalidation on config changes
+- **Config validation at construction**: duplicate `subfeed_id`/`node_id` anywhere in the tree are rejected (they share one cursor/Redis namespace); gradient `step`/`size_to_step` must be > 0
+- **Input validation**: `get_feed` rejects non-positive `limit`, empty `session_id`, non-dict `cursor`; malformed per-node cursor contents (tampered `offset`/`gen`/`page`) raise `ValueError` naming the node
+- **Session batch stored as a Redis LIST**: warm page reads transfer and parse only the requested window (single atomic pipeline: meta + LLEN + LRANGE + TTL refresh)
+- **PEP 561 `py.typed` marker**: annotations are visible to downstream type checkers
+
+### Breaking Changes
+
+- Models moved from `smartfeed.schemas` to `smartfeed.models` (`FeedManager` keeps its name)
+- `FeedManager.get_data(user_id, limit, next_page, **params)` -> `FeedManager.get_feed(session_id, limit, cursor)`
+- `user_id` -> `session_id` in ExecutionContext
+- Cursors are plain dicts, not `FeedResultNextPage` Pydantic models
+- Config: `merger_view_session` -> unified `wrapper` node with optional `cache` / `dedup` / `rerank` stages (`session_live_time` -> `cache.session_ttl`)
+- Config: `merger_id` -> `node_id` on merger and wrapper nodes (subfeed `subfeed_id` is unchanged)
+- Config: merger-level `shuffle` removed; only `SubFeed.shuffle` remains
+- Config: `MergerPositional.start` / `end` / `step` removed; `positions` are now page-relative (were absolute across pages)
+- Config: top-level `version` is no longer required or read (unknown top-level keys are ignored, so configs carrying it still parse)
+- `FeedResultClient`, `FeedResultNextPage` removed (use `FeedResult` and plain-dict cursors)
+- Pydantic v2 required (v1 support removed)
+
+### Removed (from the 0.2.0 public surface)
+
+- `smartfeed.schemas` module (models now live in `smartfeed.models`)
+- `MergerViewSession` node (replaced by `Wrapper(cache=..., dedup=...)`)
+- merger-level `shuffle`; `MergerPositional.start` / `end` / `step`
+- `FeedResultClient`, `FeedResultNextPage`
+- `examples/`
+
+### Stats (v2 vs the pre-rewrite internal working tree)
+
+| Metric | pre-rewrite | v2 |
+|--------|-------------|----|
+| Python files | 24 | 11 |
+| Lines of code | 2838 | ~1650 |
+| Test files | 26 | 43 |
+
+## 0.2.0 (2025-11-25)
 
 * Bump dependency versions
 
-# 0.1.0 (2024-12-26)
+## 0.1.0 (2024-12-26)
 
 * Initial build and publish to PyPI

LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2023 Epoch8
+   Copyright 2023-2026 Epoch8
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.