Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 28 additions & 2 deletions benchmark/gdpval/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,26 @@
# GDPVal Comparison Benchmark
# /benchmark/gdpval
Description: GDPVal cold/warm/Reflexio comparison across host-agent adapters.

## Main Entry Points


- **`run_benchmark.py`** — phase orchestration CLI
- **`config.py`** — paths and run settings
- **`adapters/`** — OpenSpace and Hermes execution
- **`memory/reflexio_bridge.py`** — publish/retrieve bridge
- **`memory/injection.py`** — learning injection
- **`evaluation.py`** — shared artifact evaluator
- **`report.py`** — per-task and aggregate comparison

## Purpose


Separate a host's native learning from Reflexio's marginal effect using matching warm-state snapshots.

## Architecture Pattern


P1 produces the shared host snapshot; P2 and P3 fork it, and P3 additionally retrieves Reflexio learning. Both arms use the same evaluator and task set.

Run the GDPVal dataset through two host agents (OpenSpace, Hermes) in a
three-phase cold → warm → warm+reflexio protocol to measure:
Expand All @@ -11,6 +33,7 @@ total (2 hosts × 3 phases). The headline we care about is `mean(P2 − P3)`.

## Prerequisites


1. Clone the dependency repos. By default `config.py` looks under `~/repos/`:
```bash
mkdir -p ~/repos
Expand Down Expand Up @@ -46,6 +69,7 @@ total (2 hosts × 3 phases). The headline we care about is `mean(P2 − P3)`.

## Running


```bash
uv run python -m benchmark.gdpval.run_benchmark \
--hosts openspace,hermes \
Expand All @@ -66,6 +90,7 @@ Verification ladder:

## Output


```
output/<run_name>/
config.json
Expand All @@ -87,7 +112,8 @@ output/<run_name>/
comparison.md # headline deltas
```

## Design notes
## Requirements / Problems to Avoid


- **Host isolation.** OpenSpace writes SkillStore state to `$OPENSPACE_ROOT/.openspace/`;
Hermes writes MEMORY.md/skills to `$HERMES_HOME` (default `~/.hermes`). The
Expand Down
33 changes: 26 additions & 7 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,35 @@
# Reflexio API Documentation Site
# /docs
Description: Interactive Next.js API explorer for the OSS server and Python SDK.

Interactive API reference and documentation for the Reflexio platform. Built with Next.js.
## Main Entry Points

**Hosted docs:** https://www.reflexio.ai/docs

## What's Inside
- **Method pages**: `app/[group]/[method]/page.tsx` — grouped API/SDK method reference.
- **Registry**: `lib/methods/registry.ts`, `lib/methods/` — method definitions and examples by domain.
- **Execution**: `lib/execution/api-executor.ts`, `code-generator.ts`, `code-parser.ts` — request execution and example conversion (all under `lib/execution/`).
- **Configuration UI**: `app/configure/page.tsx`, `lib/config-schema.ts`.
- **Shared rendering**: `components/`, `app/layout.tsx`, `app/providers.tsx`.
- **Backend routing**: `next.config.ts`, `lib/constants.ts`.

- API reference for all REST endpoints (interactions, profiles, playbooks, config, search)
- Interactive API explorer for testing requests against a running server
- Schema documentation generated from the Reflexio backend
## Purpose


Help developers inspect method contracts and run examples against a configured Reflexio backend. The public authored documentation is hosted at [Reflexio docs](https://www.reflexio.ai/docs).

## Architecture Pattern


App Router pages render the method registry; execution helpers translate examples into backend requests. Keep registry definitions, generated code, and the shared SDK/API schemas aligned when changing a method.

## Requirements / Problems to Avoid


- **API execution uses the configured server**; examples can mutate connected data.
- **Use service-start output for ports** when running the full stack; standalone Next.js development has a different default.

## Development Setup


```bash
cd docs
npm install
Expand All @@ -22,6 +40,7 @@ The site runs on **port 3000** by default. When started via `run_services.sh` fr

## Build


```bash
npm run build
```
285 changes: 73 additions & 212 deletions reflexio/README.md

Large diffs are not rendered by default.

35 changes: 33 additions & 2 deletions reflexio/benchmarks/retrieval_latency/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,25 @@
# Retrieval Latency Benchmark
# /reflexio/benchmarks/retrieval_latency
Description: Retrieval timing across corpus sizes, storage backends, and in-process service/ASGI layers.

## Main Entry Points


- **`bench.py`** — CLI and timing loop
- **`backends.py`** — storage setup
- **`seed.py`** — corpus generation
- **`scenarios.py`** — query scenarios
- **`embed_cache.py`** — query-vector cache
- **`report.py`** — statistics and baseline comparison

## Purpose


Measure retrieval cost while separating storage/library work from framework overhead.

## Architecture Pattern


Seed deterministic document vectors, cache real query embeddings, warm each cell, and record repeated timings for matched backend/layer/entity combinations.

> Part of the [Reflexio Code Map](../../README.md). Related to the server's [Unified Search Service](../../server/README.md#unified-search-service) and [Storage](../../server/README.md#storage) components.

Expand All @@ -7,6 +28,7 @@ and unified cross-entity search across storage backends and corpus sizes.

## What it measures


Four retrieval types × two layers × N storage backends × K corpus sizes.

**Layers:**
Expand All @@ -29,6 +51,7 @@ gracefully otherwise).

## Controlling embedder cost


Query embeddings are pre-cached on disk at
`~/.cache/reflexio-benchmarks/embeddings-<model>.json`. First run populates
the cache via the real embedding API (requires `OPENAI_API_KEY` or the
Expand All @@ -45,6 +68,7 @@ retrieval.

## Usage


```bash
# Default sweep: sizes 100/1000/10000, sqlite + supabase if available,
# both layers, all four retrieval types, 50 trials + 5 warmup per cell.
Expand All @@ -62,6 +86,7 @@ uv run python -m reflexio.benchmarks.retrieval_latency.bench \

## Output


Each run writes `results.json` and `report.md` to
`reflexio/benchmarks/retrieval_latency/results/<timestamp>/` — next to the
script itself, so reports travel with the code that produced them (override
Expand All @@ -74,7 +99,11 @@ one per retrieval type, rows grouped by `(backend, layer)`. Cell format:
`p50 / p95 (mean)` in milliseconds. When `--baseline` is passed, a ΔP95
column flags cells where p95 has grown by 20% or more (`⚠`).

## Interpreting the numbers
## Requirements / Problems to Avoid


### Interpreting the numbers


Sanity checks to run on any report:

Expand All @@ -87,6 +116,7 @@ Sanity checks to run on any report:

## Pytest smoke test


`tests/benchmarks/test_retrieval_latency_smoke.py` runs a tiny version of
this benchmark at `N=50, trials=10, sqlite + service only` and asserts that
p95 has not regressed past 3× the committed baseline. It's marked
Expand All @@ -105,5 +135,6 @@ Then commit the new `baseline.json`.

## See Also


- [Code Map (root README)](../../README.md) -- high-level overview of all Reflexio components
- [Server README](../../server/README.md) -- backend architecture including search and storage
Loading
Loading