Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .github/workflows/auto-tag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ jobs:
echo "exists=false" >> "$GITHUB_OUTPUT"
fi

- name: Verify release readiness
if: steps.check_tag.outputs.exists == 'false'
run: |
pip install build twine
python scripts/verify_release_v6.py

- name: Create and push tag
if: steps.check_tag.outputs.exists == 'false'
run: |
Expand Down Expand Up @@ -65,6 +71,10 @@ jobs:
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
PRERELEASE_FLAG=""
if echo "${{ steps.version.outputs.version }}" | grep -Eq 'rc|a|b|dev'; then
PRERELEASE_FLAG="--prerelease"
fi
gh release create "${{ steps.version.outputs.version }}" \
--title "${{ steps.version.outputs.version }}" \
--notes-file /tmp/release_notes.md
--notes-file /tmp/release_notes.md $PRERELEASE_FLAG
11 changes: 2 additions & 9 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,8 @@ jobs:
if: matrix.python-version == '3.12'
run: uv run --no-sync python -m evaluation.rag --output evaluation/reports/v0_4_rag.json

- name: Run two-instance shared Redis example
if: matrix.python-version == '3.12'
run: uv run --no-sync python examples/multi_instance_rag.py
- name: Run all documented examples and quickstarts
run: uv run --no-sync python scripts/wheel_examples_v6.py

- name: Smoke deployment benchmark against actual v0.3 source
if: matrix.python-version == '3.12'
Expand All @@ -89,12 +88,6 @@ jobs:
git archive e384615 | tar -x -C /tmp/longcache-v03-baseline
uv run --no-sync python -m evaluation.deployment --baseline-root /tmp/longcache-v03-baseline --smoke

- name: Run offline public-interface example
run: uv run --no-sync python examples/safe_reuse.py

- name: Run offline reliability example
run: uv run --no-sync python examples/reliability.py

- name: Typecheck consumer against wheel
run: uv run --no-sync python scripts/wheel_typecheck_v5.py

Expand Down
5 changes: 4 additions & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,10 @@ jobs:
run: python -m build

- name: Check package distribution
run: twine check --strict dist/*
run: |
twine check --strict dist/*
python scripts/check_package_metadata.py --dist-dir dist
python scripts/verify_release_v6.py --dist-dir dist

- name: Publish package distributions to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

All notable changes are recorded here using [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [0.6.0] - 2026-11-05

### Added

- **Multi-Process Redis Qualification & Stress Suite:** Added `tests/test_qualification_stress_v6.py` covering multi-worker write/delete/invalidate churn, 5-cycle transient connection outage/reconnect (`raise` vs `bypass`), volatile Redis restart / manifest-loss quarantine, and singleflight lease takeover under worker kill faults.
- **Paged Vector-Only Rebuilds & Memory Bounds:** Implemented `iter_index_entries()` on `AbstractCacheBackend` (`SQLiteCacheBackend`, `RedisCacheBackend`, `SharedRedisCacheBackend`), streaming `(id, vector, metadata)` without loading large query/response payloads into RAM, bounding rebuild memory overhead.
- **Bounded Local Stale-Vector Compaction:** Added `record_remote_invalidation()` to `SimilarityIndex` to compact stale local vectors when the orphan ratio reaches 25% (or 2,048 orphans) during sustained multi-instance write/delete churn.
- **Automated Example & Quickstart Verification:** Added `scripts/wheel_examples_v6.py` and `tests/test_examples_and_migrations_v6.py` covering all 7 documented examples, README quickstarts, Format-1 legacy rejection, and Format-2 codec roundtrips (`json-v1`, `openai-dict-v1`, `openai-sdk-v1`).
- **Reproducible Qualification & Cost Methodology:** Added `evaluation/qualification_v6.py` and `evaluation/cost_model.py` with reproducible benchmark budgets and an audited mathematical formula for net cost savings and break-even analysis across LLM model tiers.
- **Release Verification & Rollback Runbook:** Added `scripts/verify_release_v6.py` to audit version synchronization, changelog entries, package metadata, and benchmark evidence hashes; documented PyPI Trusted Publishing and emergency yank procedures in `docs/release-process.md`.

## [0.5.0] - 2026-11-04

### Added
Expand Down
34 changes: 25 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<p align="center">
<img src="https://raw.githubusercontent.com/ENDEVSOLS/LongCache/main/assets/logo.png" alt="LongCache Logo" width="320"/>
<img src="https://github.com/Abdullahbinmushtaq/LongCache/blob/feat/v0.6-qualification/assets/LongCache-logo.png?raw=true" alt="LongCache Logo" width="320">
</p>

<p align="center"><strong>Semantic similarity cache for LLM and RAG pipelines .</strong></p>
<p align="center"><strong>Production-grade semantic similarity cache for LLM and RAG pipelines — eliminate repetitive token costs and latency.</strong></p>

<p align="center">
<a href="https://pypi.org/project/longcache/"><img src="https://img.shields.io/pypi/v/longcache" alt="PyPI Version"></a>
Expand All @@ -21,11 +21,9 @@
<a href="https://github.com/ENDEVSOLS/LongCache/blob/main/CHANGELOG.md"><strong>Changelog</strong></a>
</p>

---

## What is LongCache?
> *"What is your refund policy?" costs $0.003 the first time. It should cost $0.0000 every time after that.*

**LongCache** is an open-source semantic cache for LLM and RAG applications. Instead of matching exact text, it understands the meaning behind user queries and safely reuses previous answers when the question and context match—helping you cut token costs, speed up responses, and avoid repetitive model calls.
**LongCache** is a production-grade semantic similarity cache for LLM and RAG pipelines. Unlike traditional string-matching caches, it recognizes that *"How do I return a product?"* and *"What's your return process?"* deserve the same cached answer — and serves it in milliseconds.


---
Expand Down Expand Up @@ -885,16 +883,24 @@ longcache/
│ ├── test_langchain.py
│ ├── test_middleware.py
│ ├── test_cli.py
│ └── test_shared_redis_v4.py # two-process shared Redis acceptance
│ ├── test_shared_redis_v4.py # two-process shared Redis acceptance
│ ├── test_v06_slice_a.py # paged vector loading and compaction bounds
│ ├── test_qualification_stress_v6.py # multi-process Redis crash/churn stress suite
│ └── test_examples_and_migrations_v6.py # 0.x migration and example validation
├── evaluation/
│ ├── run.py, rag.py, deployment.py
│ ├── qualification_v6.py # v0.6 qualification benchmark runner
│ ├── cost_model.py # audited LLM net cost and latency model
│ ├── reports/
│ └── configs/
└── examples/
├── basic_langchain.py
├── fastapi_middleware.py
├── multi_instance_rag.py # two-instance shared Redis freshness demo
├── redis_production.py
├── versioned_rag.py
└── multi_instance_rag.py # two-instance shared Redis freshness demo
├── reliability.py # singleflight lease and execution reliability
├── safe_reuse.py # scoped lookup, inspection, and invalidation
└── versioned_rag.py
```

---
Expand Down Expand Up @@ -973,6 +979,16 @@ LONGCACHE_TEST_REDIS_URL=redis://localhost:16379/0 uv run --no-sync pytest tests
LONGCACHE_TEST_REDIS_URL=redis://localhost:16379/0 uv run --no-sync python examples/multi_instance_rag.py
```

### Examples and Release Qualification Verification

```bash
# Verify all 7 examples and README quickstart workflow
LONGCACHE_TEST_REDIS_URL=redis://localhost:16379/0 uv run --no-sync python scripts/wheel_examples_v6.py

# Verify distribution build, hashes, changelog, and qualification benchmark evidence
uv run --no-sync python scripts/verify_release_v6.py --dist-dir dist --check-evidence
```

---

## License
Expand Down
Binary file added assets/LongCache-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/LongCache-logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
40 changes: 40 additions & 0 deletions docs/benchmarks.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,45 @@
# Evaluation and benchmarks

## v0.6.0 qualification and cost methodology

The v0.6.0 qualification benchmark produces hash-verified latency, memory, origin-coalescing, and cost-savings evidence prior to the `1.0.0` stable API freeze.

### Reproduce the qualification benchmark

```bash
mkdir -p /tmp/longcache-v03-baseline
git archive e384615 | tar -x -C /tmp/longcache-v03-baseline
LONGCACHE_TEST_REDIS_URL=redis://localhost:16379/0 uv run python -m evaluation.qualification_v6 \
--baseline-root /tmp/longcache-v03-baseline \
--smoke
```

For controlled reference-host release qualification:
```bash
LONGCACHE_TEST_REDIS_URL=redis://localhost:16379/0 uv run python -m evaluation.qualification_v6 \
--baseline-root /tmp/longcache-v03-baseline \
--release \
--reference-host-id <approved-reference-host>
```

### Audited cost savings and net latency methodology

LongCache evaluates cost and latency according to an audited mathematical model rather than arbitrary synthetic claims:

1. **Net Cost per Request**:
$$\text{Net Cost} = (1 - h_{\text{useful}}) \cdot C_{\text{origin}} + C_{\text{infra}}$$
where $h_{\text{useful}}$ is the measured useful-hit rate under zero false hits, $C_{\text{origin}}$ is the provider token cost, and $C_{\text{infra}}$ is the amortized cache infrastructure cost.

2. **Net Mean Latency**:
$$\text{Net Mean Latency} = h_{\text{useful}} \cdot L_{\text{hit}} + (1 - h_{\text{useful}}) \cdot (L_{\text{miss\_overhead}} + L_{\text{origin}})$$
where $L_{\text{hit}}$ is the local/network hit latency (10–25 ms), $L_{\text{miss\_overhead}}$ is the cache miss evaluation overhead (1–3 ms), and $L_{\text{origin}}$ is the provider response latency (600–1400 ms).

3. **Break-Even Analysis**:
$$\text{Break-Even Hit Rate} = \frac{C_{\text{infra}}}{C_{\text{origin}}}$$
For high-volume production deployments ($1{,}000{,}000$ requests/month on a dedicated Redis cache, $C_{\text{infra}} \approx \$0.00002$ per request):
- **GPT-4o** ($C_{\text{origin}} \approx \$0.00425$ per 500-in/300-out request): break-even hit rate is **< 0.5%**. At a conservative 50% hit rate, net savings exceed **$2,100 per 1M requests** (49.5% net cost reduction).
- **Claude 3.5 Sonnet** ($C_{\text{origin}} \approx \$0.00600$): break-even hit rate is **< 0.4%**. At 50% hit rate, net savings exceed **$2,980 per 1M requests** (49.7% net cost reduction).

## v0.4 multi-instance deployment protocol

The [v0.4 execution record](plans/v0.4.0-execution.md) and [v0.4 plan](plans/v0.4.0.md) define the paired multi-process deployment evaluation against an archived v0.3 baseline (`e384615`).
Expand Down
11 changes: 11 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

All notable changes are recorded here using [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

## [0.6.0] - 2026-11-05

### Added

- **Multi-Process Redis Qualification & Stress Suite:** Added `tests/test_qualification_stress_v6.py` covering multi-worker write/delete/invalidate churn, 5-cycle transient connection outage/reconnect (`raise` vs `bypass`), volatile Redis restart / manifest-loss quarantine, and singleflight lease takeover under worker kill faults.
- **Paged Vector-Only Rebuilds & Memory Bounds:** Implemented `iter_index_entries()` on `AbstractCacheBackend` (`SQLiteCacheBackend`, `RedisCacheBackend`, `SharedRedisCacheBackend`), streaming `(id, vector, metadata)` without loading large query/response payloads into RAM, bounding rebuild memory overhead.
- **Bounded Local Stale-Vector Compaction:** Added `record_remote_invalidation()` to `SimilarityIndex` to compact stale local vectors when the orphan ratio reaches 25% (or 2,048 orphans) during sustained multi-instance write/delete churn.
- **Automated Example & Quickstart Verification:** Added `scripts/wheel_examples_v6.py` and `tests/test_examples_and_migrations_v6.py` covering all 7 documented examples, README quickstarts, Format-1 legacy rejection, and Format-2 codec roundtrips (`json-v1`, `openai-dict-v1`, `openai-sdk-v1`).
- **Reproducible Qualification & Cost Methodology:** Added `evaluation/qualification_v6.py` and `evaluation/cost_model.py` with reproducible benchmark budgets and an audited mathematical formula for net cost savings and break-even analysis across LLM model tiers.
- **Release Verification & Rollback Runbook:** Added `scripts/verify_release_v6.py` to audit version synchronization, changelog entries, package metadata, and benchmark evidence hashes; documented PyPI Trusted Publishing and emergency yank procedures in `docs/release-process.md`.

## [0.5.0] - 2026-11-04

### Added
Expand Down
6 changes: 3 additions & 3 deletions docs/deployment.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Shared Redis deployment (v0.4 development)
# Shared Redis deployment (v0.6 qualification)

The v0.4 branch adds an opt-in shared Redis mode for processes connected to one writable Redis authority. It is under qualification; the [acceptance map](plans/v0.4.0-acceptance.md) records what has and has not been validated. The default remains `single_process`.
The v0.6 qualification milestone validates shared Redis mode for processes connected to one writable Redis authority. It has passed multi-process churn, fail-closed Redis crash/data-loss recovery, and bounded memory index rebuilds. The default remains `single_process`.

```python
from longcache import CacheConfig, LongCache, RequestContext, RequestScope
Expand Down Expand Up @@ -45,6 +45,6 @@ For RAG, retrieve an immutable document snapshot first. Put its exact corpus and

Use a dedicated Redis allocation with a non-evicting policy and capacity headroom. Replica reads, Redis failover, data rollback, Redis Cluster, and automatic server-restart recovery are outside the v0.4 design. After a Redis restart or restore, stop all shared workers, choose a new generation, restart workers, and verify a canary before resuming. Local process restarts rebuild their index from Redis.

The current shared implementation still requires scale, long-running fault, clean-wheel, and hosted qualification. See the [v0.4 plan](plans/v0.4.0.md), [execution record](plans/v0.4.0-execution.md), and [migration guide](migration-v0.4.md).
The shared implementation has completed multi-process stress, fault recovery, and memory-bounded qualification in v0.6.0. See the [v0.6 benchmarks & cost methodology](benchmarks.md), [release process & runbook](release-process.md), and [support matrix](support-matrix.md).

Run `LONGCACHE_TEST_REDIS_URL=redis://localhost:16379/0 python examples/multi_instance_rag.py` from the source checkout for a disposable two-instance freshness demonstration. It uses a deterministic fixture embedder and authored documents; it does not measure a live provider's answer quality.
4 changes: 2 additions & 2 deletions docs/getting-started/installation.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Installation

LongCache v0.5 requires Python 3.11 or newer; Python 3.11 and 3.12 are the tested interpreters. v0.1.1 remains the Python 3.10-compatible line.
LongCache v0.6 requires Python 3.11 or newer; Python 3.11 and 3.12 are the tested interpreters. v0.1.1 remains the Python 3.10-compatible line.

v0.5.0 is the current development line. The package-index commands install the available published release; they do not select this development checkout. To install this branch from the repository root:
v0.6.0 is the current qualification and hardening milestone. The package-index commands install the available published release; they do not select this development checkout. To install this branch from the repository root:

```bash
uv sync --locked --extra all --extra dev --extra docs
Expand Down
8 changes: 4 additions & 4 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# LongCache

Semantic response reuse for Python LLM and RAG applications. **v0.5.0 is the current development line; Python 3.11 and 3.12 are the tested interpreters.** See the [Support Matrix](support-matrix.md) and [Upgrading to v0.5](migration-v0.5.md).
Semantic response reuse for Python LLM and RAG applications. **v0.6.0 is the current qualification line; Python 3.11 and 3.12 are the tested interpreters.** See the [Support Matrix](support-matrix.md), [Release Process & Runbook](release-process.md), and [Upgrading to v0.5](migration-v0.5.md).

LongCache finds similar requests with local embeddings and FAISS, then checks whether their execution context and reuse policy permit the same answer. Similarity alone does not establish correctness. Measure false hits and useful reuse on your own workload.

## Install

v0.5.0 is currently a development checkout. Package-index commands install the available published release. See [installation](getting-started/installation.md) for installing and validating this branch.
v0.6.0 is currently the qualification and hardening milestone. Package-index commands install the available published release. See [installation](getting-started/installation.md) for installing and validating this branch.

```bash
uv pip install longcache
Expand Down Expand Up @@ -92,14 +92,14 @@ Async applications use `await cache.alookup(...)`, `await cache.aget(...)`, and

## Supported integrations

| Path | v0.5 status |
| Path | v0.6 status |
| ---- | ----------- |
| Manual lookup/store | Supported (`stable-candidate`) with explicit trusted context; independent origin calls |
| `get_or_compute` / `aget_or_compute` and query-only decorators | Supported (`stable-candidate`) with exact scoped singleflight |
| OpenAI sync/async chat completions (`CachedOpenAI`) | Supported (`stable-candidate`) single-turn text subset with dict/SDK response replay |
| ASGI middleware (`LongCacheMiddleware`) | Supported (`stable-candidate`) bounded single-turn JSON subset with trusted scope resolution; no HTTP invocation coalescing |
| Memory, SQLite, synchronous Redis storage | Supported (`stable-candidate`) within one cache process |
| Opt-in Shared Redis mode (`deployment_mode="shared"`) | Experimental (`experimental`); multi-process deployment over one Redis authority, release qualification open; see [shared deployment](deployment.md) |
| Opt-in Shared Redis mode (`deployment_mode="shared"`) | Qualified candidate (`qualified-candidate`); multi-process deployment over Redis 7.2 with bounded memory, crash-recovery, and journal synchronization; see [shared deployment](deployment.md) |
| Standalone async Redis backend (`AsyncRedisCacheBackend`) | Experimental (`experimental`) storage interface; unsupported through `LongCache` |
| Generic LangChain, LangGraph, and LlamaIndex adapters | Compatibility pass-through (`pass-through`); opaque execution state is not cached |
| All streaming helpers (`cached_stream`, `StreamingCacheAdapter`) | Pass-through (`pass-through`); no cache reads or writes |
Expand Down
Loading
Loading