snui1s · snui1s · Aug 31, 2026 · Aug 31, 2026 · Aug 31, 2026 · Aug 31, 2026
diff --git a/.github/workflows/evals.yml b/.github/workflows/evals.yml
@@ -0,0 +1,95 @@
+name: LLM Evals & CI Quality Gate
+
+on:
+  push:
+    branches: [main, master, "feat/**"]
+  pull_request:
+    branches: [main, master]
+  workflow_dispatch:
+    inputs:
+      target_model:
+        description: "OpenRouter Target Model for Live Benchmark"
+        required: false
+        default: "deepseek/deepseek-v4-flash"
+      run_live_benchmark:
+        description: "Run live LLM evaluation using OpenRouter API key"
+        type: boolean
+        default: false
+
+jobs:
+  # ─────────────────────────────────────────────────────────────
+  # Job 1: Deterministic Quality Gate (Runs on every PR / Push)
+  # ─────────────────────────────────────────────────────────────
+  test-and-eval-gate:
+    name: Run Unit Tests & Eval Schemas
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: "pip"
+
+      - name: Install Project & Test Dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e .
+          pip install pytest deepeval
+
+      - name: Run Core Test Suite
+        run: |
+          pytest tests/ -v
+
+      - name: Run LLM Evals Test Suite
+        run: |
+          pytest evals/ -v
+
+      - name: Run Benchmark Offline Dry-Run
+        run: |
+          python -m evals.run_benchmarks --dry-run
+
+  # ─────────────────────────────────────────────────────────────
+  # Job 2: Live LLM Benchmark (Optional / Manual Dispatch)
+  # ─────────────────────────────────────────────────────────────
+  live-benchmark:
+    name: Live OpenRouter Benchmark
+    runs-on: ubuntu-latest
+    if: github.event_name == 'workflow_dispatch' && inputs.run_live_benchmark == true
+    needs: test-and-eval-gate
+
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: "pip"
+
+      - name: Install Dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e .
+          pip install pytest deepeval
+
+      - name: Run Live Benchmark
+        env:
+          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+        run: |
+          python -m evals.run_benchmarks \
+            --model "${{ github.event.inputs.target_model }}" \
+            --output benchmark_scorecard.md
+
+      - name: Upload Benchmark Scorecard Artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark-scorecard
+          path: benchmark_scorecard.md
diff --git a/README.md b/README.md
@@ -24,18 +24,22 @@
 ## Key Features
 
 ### Deep Analysis & BYOK (Bring Your Own Key)
-- **Bring Your Own Key (BYOK)** — Connect directly to OpenRouter & Tavily. Zero token markup, no middleman limits, and total freedom to choose any model (*DeepSeek V4, Claude 3.5, Gemini 2.5, GPT-4o*).
+
+- **Bring Your Own Key (BYOK)** — Connect directly to OpenRouter & Tavily. Zero token markup, no middleman limits, and total freedom to choose any model (_DeepSeek V4, Claude 3.5, Gemini 2.5, GPT-4o_).
 - **Deep Q&A & Security Auditing** — Built to read large code blocks, analyze architecture, and find security vulnerabilities or logic flaws without truncating long explanations.
 
 ### Read-Only Mode (`/readonly`)
-- **Strict Safety Toggle** — Lock the agent into Read-Only mode with `/readonly`. Modifying tools (*file writing, editing, replacing, deleting*) and shell command execution are blocked at both schema and runtime levels — safe for auditing production code.
+
+- **Strict Safety Toggle** — Lock the agent into Read-Only mode with `/readonly`. Modifying tools (_file writing, editing, replacing, deleting_) and shell command execution are blocked at both schema and runtime levels — safe for auditing production code.
 - **Interactive Command Interception** — Destructive shell commands trigger explicit colorized `(y/n)` confirmation before running.
 
 ### Web Article Reader & Live Search
+
 - **Article Reader (`read_web_page`)** — Powered by `trafilatura` to extract clean text/markdown from blog posts, documentation, and news URLs.
 - **Web Search (`/search`)** — Integrated Tavily web search for real-time documentation, recent news, and online research.
 
 ### Performance & UX
+
 - **Multi-Session Chat** — Manage multiple independent chat tabs backed by a local SQLite database (`agent_data.db`).
 - **Memory Compaction** — Automatically summarizes older context when history exceeds thresholds to preserve tokens and keep response times fast.
 - **Skill & Plugin System** — Add markdown instruction files to `./skills/` or install dynamic prompt plugins directly from GitHub via `/plugin add`.
@@ -45,22 +49,26 @@
 ## Quick Install
 
 ### Prerequisites
+
 - [Git](https://git-scm.com)
 - [Python 3.10+](https://python.org)
 
 ### Installation Commands
 
 **Windows (PowerShell):**
+
 ```powershell
 irm https://raw.githubusercontent.com/snui1s/losna-cli/main/install.ps1 | iex
 ```
 
 **macOS / Linux:**
+
 ```bash
 curl -sSL https://raw.githubusercontent.com/snui1s/losna-cli/main/install.sh | bash
 ```
 
 After installation, **restart your terminal** and launch:
+
 ```bash
 losna
 ```
@@ -73,7 +81,8 @@ losna
 3. Installs all required dependencies automatically
 4. Registers the `losna` command on your system `PATH`
 
-*No global Python packages are modified. Everything is self-contained inside `~/.losna/`.*
+_No global Python packages are modified. Everything is self-contained inside `~/.losna/`._
+
 </details>
 
 <details>
@@ -82,14 +91,17 @@ losna
 **Updating:** Re-run the quick install command. The installer detects existing installations and pulls the latest updates.
 
 **Uninstalling (Windows PowerShell):**
+
 ```powershell
 irm https://raw.githubusercontent.com/snui1s/losna-cli/main/uninstall.ps1 | iex
 ```
 
 **Uninstalling (macOS / Linux):**
+
 ```bash
 curl -sSL https://raw.githubusercontent.com/snui1s/losna-cli/main/uninstall.sh | bash
 ```
+
 </details>
 
 ---
@@ -98,49 +110,50 @@ curl -sSL https://raw.githubusercontent.com/snui1s/losna-cli/main/uninstall.sh |
 
 On first launch, Losna CLI prompts for configuration:
 
-1. **OpenRouter API Key** *(required)* — Connects to AI models via OpenRouter. Get one at [openrouter.ai](https://openrouter.ai).
-2. **Web Search** *(optional)* — Enable web search using a [Tavily](https://tavily.com) API key.
+1. **OpenRouter API Key** _(required)_ — Connects to AI models via OpenRouter. Get one at [openrouter.ai](https://openrouter.ai).
+2. **Web Search** _(optional)_ — Enable web search using a [Tavily](https://tavily.com) API key.
 
 All keys are stored locally in `~/.losnarc` (JSON format) and are never sent anywhere else.
 
 ---
 
 ## Slash Commands
 
-| Command | Description |
-|---|---|
-| `/help` | Show all available commands and loaded skills |
-| `/new <title>` | Start a new chat session with a custom title |
-| `/rename [id] <title>` | Rename the current or specified chat session |
-| `/sessions` | List all saved chat sessions with their IDs |
-| `/switch <id>` | Switch to a different chat session by ID |
-| `/delete_session <id>` | Delete an existing chat session by ID |
-| `/history [id]` | View chat logs and tool execution history for a session |
-| `/model` | View current OpenRouter model or switch to a new model ID |
-| `/readonly` | Toggle Read-Only Mode (blocks file modification & shell execution) |
-| `/diff [file\|session]` | View colored syntax-highlighted git diff for a file or session memory state |
-| `/enter2confirm` | Toggle double-Enter requirement before sending prompts to AI |
-| `/pin <text>` | Pin a custom rule/fact to AI Core Memory (remembered forever across sessions) |
-| `/pins` | List all pinned Core Memory rules with their database IDs |
-| `/unpin <id>` | Unpin/remove a Core Memory rule by ID or exact text |
-| `/export [path]` | Export active session chat log into a structured Markdown document |
-| `/clear` | Clear terminal screen and re-render header banner |
-| `/ls [path]` | List directory files and folders in clean formatted view |
-| `/cd <path>` | Change working directory (supports '..', '~', and '-') |
-| `/init-ai` | Generate a starter 'ai.txt' blueprint file for project auto-detection |
-| `/max_tool_calls [n]` | View or set maximum tool call limit per turn (persisted in ~/.losnarc) |
-| `/plugin add <url>` | Download and install all skills from a GitHub repository |
-| `/plugin add <url> --skill <name>` | Download and install a specific skill from a GitHub repository |
-| `/plugin remove <name>` | Uninstall/remove a custom skill plugin from local project |
-| `/plugin list` | List all installed skill plugins and their enabled status |
-| `/plugin enable <name>` | Enable a disabled skill plugin globally |
-| `/plugin disable <name>` | Disable an active skill plugin globally |
-| `/<skill> off\|on\|status` | Quick toggle or status check for an individual skill |
-| `/search <query>` | Search the web using Tavily and synthesize results |
-| `/usage` | Show session token usage and estimated cost breakdown |
-| `/exit` or `/quit` | Exit Losna CLI session |
+| Command                            | Description                                                                   |
+| ---------------------------------- | ----------------------------------------------------------------------------- |
+| `/help`                            | Show all available commands and loaded skills                                 |
+| `/new <title>`                     | Start a new chat session with a custom title                                  |
+| `/rename [id] <title>`             | Rename the current or specified chat session                                  |
+| `/sessions`                        | List all saved chat sessions with their IDs                                   |
+| `/switch <id>`                     | Switch to a different chat session by ID                                      |
+| `/delete_session <id>`             | Delete an existing chat session by ID                                         |
+| `/history [id]`                    | View chat logs and tool execution history for a session                       |
+| `/model`                           | View current OpenRouter model or switch to a new model ID                     |
+| `/readonly`                        | Toggle Read-Only Mode (blocks file modification & shell execution)            |
+| `/diff [file\|session]`            | View colored syntax-highlighted git diff for a file or session memory state   |
+| `/enter2confirm`                   | Toggle double-Enter requirement before sending prompts to AI                  |
+| `/pin <text>`                      | Pin a custom rule/fact to AI Core Memory (remembered forever across sessions) |
+| `/pins`                            | List all pinned Core Memory rules with their database IDs                     |
+| `/unpin <id>`                      | Unpin/remove a Core Memory rule by ID or exact text                           |
+| `/export [path]`                   | Export active session chat log into a structured Markdown document            |
+| `/clear`                           | Clear terminal screen and re-render header banner                             |
+| `/ls [path]`                       | List directory files and folders in clean formatted view                      |
+| `/cd <path>`                       | Change working directory (supports '..', '~', and '-')                        |
+| `/init-ai`                         | Generate a starter 'ai.txt' blueprint file for project auto-detection         |
+| `/max_tool_calls [n]`              | View or set maximum tool call limit per turn (persisted in ~/.losnarc)        |
+| `/plugin add <url>`                | Download and install all skills from a GitHub repository                      |
+| `/plugin add <url> --skill <name>` | Download and install a specific skill from a GitHub repository                |
+| `/plugin remove <name>`            | Uninstall/remove a custom skill plugin from local project                     |
+| `/plugin list`                     | List all installed skill plugins and their enabled status                     |
+| `/plugin enable <name>`            | Enable a disabled skill plugin globally                                       |
+| `/plugin disable <name>`           | Disable an active skill plugin globally                                       |
+| `/<skill> off\|on\|status`         | Quick toggle or status check for an individual skill                          |
+| `/search <query>`                  | Search the web using Tavily and synthesize results                            |
+| `/usage`                           | Show session token usage and estimated cost breakdown                         |
+| `/exit` or `/quit`                 | Exit Losna CLI session                                                        |
 
 ### 💡 Slash Command Examples
+
 ```bash
 # File Context & Workspace Inspection
 /init-ai
@@ -178,6 +191,7 @@ All keys are stored locally in `~/.losnarc` (JSON format) and are never sent any
 Project skills stored in `./skills/<skill-name>/SKILL.md` are automatically recognized as slash commands (e.g., `/unit-testing`).
 
 ### Installing Plugins from GitHub
+
 ```bash
 # Install all skills from a repository
 /plugin add https://github.com/JuliusBrussee/caveman
@@ -187,6 +201,7 @@ Project skills stored in `./skills/<skill-name>/SKILL.md` are automatically reco
 ```
 
 ### Removing Plugins
+
 ```bash
 # Interactive selection list
 /plugin remove
@@ -196,13 +211,17 @@ Project skills stored in `./skills/<skill-name>/SKILL.md` are automatically reco
 ```
 
 ### Writing Custom Skills
+
 Create `./skills/my-custom-skill/SKILL.md` in your project:
+
 ```markdown
 ---
 name: my-custom-skill
 description: Custom team coding guidelines and security review rules.
 ---
+
 # Instructions
+
 1. Always check for memory leaks and unchecked input...
 2. Use strict typing...
 ```
@@ -225,10 +244,30 @@ losna-cli/
 │       ├── skills_loader.py # Dynamic skill loading from project files
 │       ├── plugin_manager.py# Remote plugin package installer
 │       └── ui.py            # Terminal UI (spinners, banners, markdown renderer)
+├── evals/                   # LLM evaluation datasets, metrics & benchmark runner
 ├── skills/                  # Local project skill definitions
 ├── tests/                   # Automated pytest suite
 ├── install.ps1              # Windows installer
 ├── install.sh               # macOS/Linux installer
 ├── pyproject.toml           # Package metadata, dependencies & bump-my-version
 └── README.md
 ```
+
+---
+
+## LLM Evals & Benchmarking
+
+Losna CLI includes a dedicated LLM evaluation and benchmarking suite using **DeepEval** to test prompt performance, tool dispatch accuracy, memory compaction, and security audit quality.
+
+### Running Evals & Benchmarks
+
+```bash
+# Run deterministic unit evaluations via pytest
+pytest evals/ -v
+
+# Run benchmark runner in dry-run mode (offline)
+python -m evals.run_benchmarks --dry-run
+
+# Run live benchmark against target OpenRouter model and export scorecard
+python -m evals.run_benchmarks --model anthropic/claude-3.5-sonnet --output benchmark_scorecard.md
+```
diff --git a/benchmarks/deepseek_v3.md b/benchmarks/deepseek_v3.md
@@ -0,0 +1,18 @@
+# 🌒 Losna CLI — Benchmark Scorecard
+
+**Target Model:** `deepseek/deepseek-v4-flash-0731`  
+**Evaluation Mode:** `Live OpenRouter`  
+**Duration:** `125.98s`  
+**Overall Benchmark Score:** **`83.3%`** (`15/18` passed)
+
+## Category Breakdown
+
+| Category | Total Test Cases | Passed | Score | Status |
+| :--- | :---: | :---: | :---: | :---: |
+| 🛠️ **Tool Selection & Schema** | 9 | 9 | 100.0% | ✅ PASS |
+| 🧠 **Memory Compaction & Retention** | 2 | 2 | 100.0% | ✅ PASS |
+| 🔒 **Security Audit & Code Analysis** | 3 | 0 | 0.0% | ❌ FAIL |
+| 🛡️ **Read-Only Safety Guardrails** | 4 | 4 | 100.0% | ✅ PASS |
+
+---
+*Generated by Losna CLI DeepEval Benchmarking Suite.*
diff --git a/benchmarks/gemini.md b/benchmarks/gemini.md
@@ -0,0 +1,18 @@
+# 🌒 Losna CLI — Benchmark Scorecard
+
+**Target Model:** `google/gemini-3.7-flash`  
+**Evaluation Mode:** `Live OpenRouter`  
+**Duration:** `118.15s`  
+**Overall Benchmark Score:** **`100.0%`** (`18/18` passed)
+
+## Category Breakdown
+
+| Category | Total Test Cases | Passed | Score | Status |
+| :--- | :---: | :---: | :---: | :---: |
+| 🛠️ **Tool Selection & Schema** | 9 | 9 | 100.0% | ✅ PASS |
+| 🧠 **Memory Compaction & Retention** | 2 | 2 | 100.0% | ✅ PASS |
+| 🔒 **Security Audit & Code Analysis** | 3 | 3 | 100.0% | ✅ PASS |
+| 🛡️ **Read-Only Safety Guardrails** | 4 | 4 | 100.0% | ✅ PASS |
+
+---
+*Generated by Losna CLI DeepEval Benchmarking Suite.*
diff --git a/benchmarks/gpt5-6luna.md b/benchmarks/gpt5-6luna.md
@@ -0,0 +1,18 @@
+# 🌒 Losna CLI — Benchmark Scorecard
+
+**Target Model:** `openai/gpt-5.6-luna`  
+**Evaluation Mode:** `Live OpenRouter`  
+**Duration:** `107.69s`  
+**Overall Benchmark Score:** **`94.4%`** (`17/18` passed)
+
+## Category Breakdown
+
+| Category | Total Test Cases | Passed | Score | Status |
+| :--- | :---: | :---: | :---: | :---: |
+| 🛠️ **Tool Selection & Schema** | 9 | 8 | 88.9% | ✅ PASS |
+| 🧠 **Memory Compaction & Retention** | 2 | 2 | 100.0% | ✅ PASS |
+| 🔒 **Security Audit & Code Analysis** | 3 | 3 | 100.0% | ✅ PASS |
+| 🛡️ **Read-Only Safety Guardrails** | 4 | 4 | 100.0% | ✅ PASS |
+
+---
+*Generated by Losna CLI DeepEval Benchmarking Suite.*
diff --git a/evals/__init__.py b/evals/__init__.py
@@ -0,0 +1,3 @@
+"""
+evals package — LLM Evaluation & Benchmarking Suite for Losna CLI.
+"""