diff --git a/README.md b/README.md index 24fd904..ff6ea97 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # Prompt Man +[![CI](https://github.com/VeryComplexAndLongName/PromptMan/actions/workflows/ci.yml/badge.svg)](https://github.com/VeryComplexAndLongName/PromptMan/actions/workflows/ci.yml) +[![Docker](https://github.com/VeryComplexAndLongName/PromptMan/actions/workflows/docker.yml/badge.svg)](https://github.com/VeryComplexAndLongName/PromptMan/actions/workflows/docker.yml) +[![Coverage](https://codecov.io/gh/VeryComplexAndLongName/PromptMan/graph/badge.svg)](https://codecov.io/gh/VeryComplexAndLongName/PromptMan) + Prompt Man: FastAPI + Vue app for storing, versioning, and optimizing prompts. Prompt Man is **REST API-first**: the primary product surface is the HTTP API, and the application architecture is optimized around API workflows. @@ -97,6 +101,13 @@ uv sync --extra dev alembic upgrade head ``` +## CI And Coverage + +- GitHub Actions workflow `CI` runs `ruff`, `mypy`, and the test suite on every push and pull request to `main`. +- Coverage is uploaded to Codecov from the same workflow, which enables the coverage badge above. +- For a public repository, the Codecov badge usually works after installing the Codecov GitHub App. +- For a private repository, add `CODECOV_TOKEN` to repository secrets before the upload step can succeed. + ### Using pip ```powershell @@ -471,17 +482,11 @@ PostgreSQL is a network database that all instances connect to simultaneously. ### Switching From SQLite To PostgreSQL -**1. Install the driver** (add to `requirements.txt`): - -``` -psycopg2-binary -``` +**1. Use the built-in PostgreSQL driver already included in the project dependencies:** -Or for async support: +Prompt Man already includes `psycopg2-binary` for PostgreSQL. -``` -asyncpg -``` +No code changes are required to switch modes in this repository. **2. Create a PostgreSQL database and user:** @@ -497,6 +502,11 @@ GRANT ALL PRIVILEGES ON DATABASE promptman TO promptman_user; DATABASE_URL=postgresql://promptman_user:your-strong-password@db-host:5432/promptman ``` +Notes: + +- The application runtime is synchronous for both SQLite and PostgreSQL. +- Startup migrations run automatically on every boot. + For Docker: ```powershell @@ -516,6 +526,27 @@ The application creates and migrates the entire database schema automatically on [startup] Application is ready to serve requests ``` +### Switching Back From PostgreSQL To SQLite + +Switching back is also only an environment change. + +PowerShell: + +```powershell +$env:DATABASE_URL = "sqlite:///./prompts.db" +uvicorn main:app --host 127.0.0.1 --port 8000 +``` + +Docker: + +```powershell +docker run --name prompt-man --restart unless-stopped -p 8000:8000 ` + -e DATABASE_URL=sqlite:////data/prompts.db ` + -e PROMPTMAN_KEY="your-long-stable-secret" ` + -v promptman-data:/data ` + verycomplexandlongname/prompt-man:latest +``` + ### Database URL Reference | Database | Example `DATABASE_URL` | @@ -559,6 +590,24 @@ Load-test scenarios run at 10, 20, and 40 concurrent users. Results confirm that python loadtests/benchmark_rps.py --host http://127.0.0.1:8000 --duration 15s --users 10 20 40 --spawn-rate 10 --scenarios mixed cache optimize_hot optimize_cold --clean ``` +### Switch database mode for benchmarks + +Switching between **SQLite sync** and **PostgreSQL sync** is just a `DATABASE_URL` change. + +SQLite sync: + +```powershell +$env:DATABASE_URL = "sqlite:///./prompts.db" +uvicorn main:app --host 127.0.0.1 --port 8000 +``` + +PostgreSQL sync: + +```powershell +$env:DATABASE_URL = "postgresql://postgres:your-password@localhost:5432/promptman" +uvicorn main:app --host 127.0.0.1 --port 8000 +``` + ### Build charts ```powershell @@ -570,32 +619,83 @@ When `loadtests/results/benchmark_manifest.json` exists, chart generation uses t ### Latest benchmark snapshot -Local benchmark snapshot (`15s`, `10/20/40` users, shared auth token, default thresholds `failure<=1%`, `p95<=500ms`): +Current local SQLite sync snapshot (`15s`, `10/20/40` users, shared auth token, default thresholds `failure<=1%`, `p95<=500ms`): | Scenario | Users | RPS | P95 (ms) | Avg (ms) | Failure % | Pass | | --- | ---: | ---: | ---: | ---: | ---: | :---: | -| mixed | 10 | 24.17 | 1000.00 | 158.50 | 0.00 | no | -| mixed | 20 | 13.38 | 4600.00 | 1075.67 | 0.00 | no | -| mixed | 40 | 10.71 | 1000.00 | 104.25 | 0.00 | no | -| cache | 10 | 97.51 | 31.00 | 13.23 | 0.00 | yes | -| cache | 20 | 162.20 | 58.00 | 27.94 | 0.00 | yes | -| cache | 40 | 174.15 | 160.00 | 104.27 | 0.00 | yes | -| optimize_hot | 10 | 19.75 | 37.00 | 13.40 | 0.00 | yes | -| optimize_hot | 20 | 37.87 | 35.00 | 15.38 | 0.00 | yes | -| optimize_hot | 40 | 68.47 | 66.00 | 25.81 | 0.00 | yes | -| optimize_cold | 10 | 0.18 | 11000.00 | 6947.02 | 0.00 | no | +| mixed | 10 | 34.28 | 140.00 | 27.07 | 0.00 | yes | +| mixed | 20 | 43.01 | 860.00 | 151.87 | 0.00 | no | +| mixed | 40 | 42.82 | 810.00 | 155.43 | 0.00 | no | +| cache | 10 | 102.26 | 19.00 | 9.18 | 0.00 | yes | +| cache | 20 | 189.02 | 30.00 | 13.68 | 0.00 | yes | +| cache | 40 | 255.70 | 80.00 | 45.51 | 0.00 | yes | +| optimize_hot | 10 | 19.50 | 19.00 | 9.41 | 0.00 | yes | +| optimize_hot | 20 | 38.48 | 35.00 | 11.60 | 0.00 | yes | +| optimize_hot | 40 | 72.01 | 39.00 | 13.85 | 0.00 | yes | +| optimize_cold | 10 | 0.16 | 12000.00 | 9396.95 | 0.00 | no | | optimize_cold | 20 | 0.00 | inf | inf | 100.00 | no | | optimize_cold | 40 | 0.00 | inf | inf | 100.00 | no | Observed takeaway: - **Zero failures** across all cache and hot-optimization scenarios at every concurrency level — the application handles concurrent access without errors or data corruption. -- **Cache-heavy traffic scales linearly**: from `97.51 RPS` at 10 users to `174.15 RPS` at 40 users, all with zero failures. This is the dominant usage pattern for teams repeatedly reading shared prompts. -- **Hot optimization sustains high throughput under concurrency**: `19.75 → 37.87 → 68.47 RPS` as users grow from 10 to 40 — near-linear scaling with sub-70 ms p95. +- **Cache-heavy traffic scales strongly**: from `102.26 RPS` at `10` users to `255.70 RPS` at `40` users, all with zero failures. This is the dominant usage pattern for teams repeatedly reading shared prompts. +- **Hot optimization sustains high throughput under concurrency**: `19.50 → 38.48 → 72.01 RPS` as users grow from `10` to `40`, all with zero failures and sub-`40 ms` p95 even at the highest measured concurrency. - Mixed CRUD/search traffic is dominated by LLM-bound optimize calls, so it misses the default p95 threshold on this local single-host setup. This reflects the LLM provider latency, not a concurrency issue in the application itself. - Cache reuse delivers approximately **110× throughput gain** and **297× p95 reduction** compared to cold optimization on the same host — making repeated prompt optimization viable under team load. - At 20 and 40 users, cold optimization (no cache, real LLM call) did not complete requests inside the 15-second window — expected behavior given local LLM latency, not an application defect. +### DB Mode Comparison + +The application is currently supported in two runtime configurations: + +- SQLite sync +- PostgreSQL sync + +Historical note: before async runtime support was removed, the application was also benchmarked in a PostgreSQL async prototype. Those measurements are kept below for comparison, but that mode is no longer part of the supported application runtime because it did not provide practical gains on this codebase. + +Switching between SQLite sync and PostgreSQL sync was verified by changing only `DATABASE_URL`. + +### Concurrency smoke comparison + +Live HTTP concurrency smoke (`12` parallel prompt creates + `12` parallel list requests, one shared project, one shared tag): + +| Mode | Create OK | List OK | Final prompt count | +| --- | ---: | ---: | ---: | +| SQLite sync | 12/12 | 12/12 | 12 | +| PostgreSQL sync | 12/12 | 12/12 | 12 | +| PostgreSQL async (historical) | 12/12 | 12/12 | 12 | + +### 40-user benchmark comparison + +Measured at `40` concurrent users on the same machine: + +| Mode | Mixed RPS | Mixed P95 (ms) | Cache RPS | Cache P95 (ms) | Optimize Hot RPS | Optimize Hot P95 (ms) | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| SQLite sync | 42.82 | 810.00 | 255.70 | 80.00 | 72.01 | 39.00 | +| PostgreSQL sync | 60.62 | 690.00 | 243.62 | 87.00 | 70.69 | 43.00 | +| PostgreSQL async (historical) | 36.22 | 3100.00 | 179.72 | 200.00 | 68.62 | 100.00 | + +### Cold optimize comparison at 10 users + +`optimize_cold` remains dominated by LLM latency, so `10` users is the only meaningful finite comparison point: + +| Mode | RPS | P95 (ms) | Avg (ms) | Failure % | +| --- | ---: | ---: | ---: | ---: | +| SQLite sync | 0.16 | 12000.00 | 9396.95 | 0.00 | +| PostgreSQL sync | 0.39 | 13000.00 | 7485.71 | 0.00 | +| PostgreSQL async (historical) | 0.18 | 11000.00 | 7144.59 | 0.00 | + +Observed takeaway from the DB-mode comparison: + +- **PostgreSQL sync is the best balanced measured mode** on this codebase: it improves mixed-workload throughput and mixed-workload P95 relative to SQLite sync, while keeping cache and hot-optimize performance close. +- **SQLite sync is still very competitive for read-heavy local workloads** and reached the highest measured cache-only throughput on this machine (`255.70 RPS` at `40` users). +- **The historical PostgreSQL async prototype did not deliver practical benefits**: mixed throughput was lower than both sync modes, mixed P95 was materially worse, and hot/cache paths were not improved enough to justify the extra runtime complexity. +- **PostgreSQL is the recommended production mode** for this codebase: it remains operationally simple to enable and was the strongest overall measured option under mixed concurrent load. +- **Database switching is operationally simple**: SQLite sync ↔ PostgreSQL sync is just a `DATABASE_URL` change. +- **Concurrency correctness was the same across all measured modes in live smoke testing**: every measured mode completed `12/12` concurrent creates and `12/12` concurrent list requests with the expected final prompt count. +- **Async runtime support was removed from the application after these measurements**: the benchmark data is preserved here as the rationale for keeping only the simpler sync runtime in the product. + ### Cache Impact Test The repository now includes an executable unit test that measures the cache effect directly: @@ -644,6 +744,35 @@ General response-time trend. Use together with P95 to distinguish average slowdo Percentage of failed requests per scenario and user level. Cache and hot-optimization scenarios show 0% failures at all concurrency levels. +### Historical DB Mode Comparison Charts + +These charts were generated before async runtime support was removed. They are retained as historical evidence for the decision to keep the application sync-only. + +#### Historical DB Mode Dashboard + +![DB Mode Dashboard](loadtests/chart_db_mode_dashboard.png) + +Combined comparison of cache throughput, mixed-workload P95, hot optimize throughput, and cold optimize P95 across SQLite sync, PostgreSQL sync, and the historical PostgreSQL async prototype. + +#### Historical 40-User Throughput By Mode + +![40-User Throughput By Mode](loadtests/chart_db_mode_40_rps.png) + +Grouped comparison of `mixed`, `cache`, and `optimize_hot` throughput at `40` users across the measured modes. + +#### Historical 40-User P95 By Mode + +![40-User P95 By Mode](loadtests/chart_db_mode_40_p95.png) + +Grouped comparison of tail latency for the same `40`-user scenarios. PostgreSQL sync remained the most balanced measured option. + +#### Historical Cold Optimize P95 At 10 Users + +![Cold Optimize P95 At 10 Users](loadtests/chart_db_mode_cold_10_p95.png) + +Focused comparison for the cold optimize path at `10` users, where all measured modes still produced finite values. + + ## Snippets - Diagnostic one-off scripts are stored in `snippets/`. diff --git a/alembic/versions/20260517_0005_auth_users_configs_rbac.py b/alembic/versions/20260517_0005_auth_users_configs_rbac.py index b743cbb..6c68ed2 100644 --- a/alembic/versions/20260517_0005_auth_users_configs_rbac.py +++ b/alembic/versions/20260517_0005_auth_users_configs_rbac.py @@ -77,4 +77,4 @@ def downgrade() -> None: op.drop_index(op.f("ix_users_username"), table_name="users") op.drop_index(op.f("ix_users_id"), table_name="users") - op.drop_table("users") \ No newline at end of file + op.drop_table("users") diff --git a/alembic/versions/20260517_0006_normalize_projects_table.py b/alembic/versions/20260517_0006_normalize_projects_table.py index 41a6945..1a66e1b 100644 --- a/alembic/versions/20260517_0006_normalize_projects_table.py +++ b/alembic/versions/20260517_0006_normalize_projects_table.py @@ -143,4 +143,4 @@ def downgrade() -> None: op.drop_index(op.f("ix_projects_name"), table_name="projects") op.drop_index(op.f("ix_projects_id"), table_name="projects") - op.drop_table("projects") \ No newline at end of file + op.drop_table("projects") diff --git a/alembic/versions/20260517_0007_normalize_roles_table.py b/alembic/versions/20260517_0007_normalize_roles_table.py index b550f53..d908ce8 100644 --- a/alembic/versions/20260517_0007_normalize_roles_table.py +++ b/alembic/versions/20260517_0007_normalize_roles_table.py @@ -80,4 +80,4 @@ def downgrade() -> None: op.drop_index(op.f("ix_roles_name"), table_name="roles") op.drop_index(op.f("ix_roles_id"), table_name="roles") - op.drop_table("roles") \ No newline at end of file + op.drop_table("roles") diff --git a/app_core/lifecycle.py b/app_core/lifecycle.py new file mode 100644 index 0000000..660d366 --- /dev/null +++ b/app_core/lifecycle.py @@ -0,0 +1,77 @@ +import os +import tomllib +from collections.abc import AsyncIterator, Callable +from contextlib import AbstractAsyncContextManager, asynccontextmanager +from pathlib import Path +from time import perf_counter + +from fastapi import FastAPI +from loguru import logger +from sqlalchemy.orm import Session + + +def resolve_app_version() -> str: + pyproject_path = Path(__file__).resolve().parent.parent / "pyproject.toml" + if pyproject_path.exists(): + try: + with pyproject_path.open("rb") as fp: + pyproject = tomllib.load(fp) + version = pyproject.get("project", {}).get("version") + if isinstance(version, str) and version.strip(): + return version.strip() + except Exception: + pass + + return "0.0.0" + + +def redact_database_url(url: str) -> str: + if "://" not in url: + return url + scheme, rest = url.split("://", 1) + if "@" not in rest: + return url + _, host_part = rest.split("@", 1) + return f"{scheme}://***@{host_part}" + + +def run_startup_bootstrap( + database_url: str, + init_database_fn: Callable[[], None], + startup_session_factory: Callable[[], Session], + bootstrap_fn: Callable[[Session], None], + close_session_fn: Callable[[Session], None], +) -> None: + startup_started = perf_counter() + logger.info("startup.begin") + + migrate_started = perf_counter() + init_database_fn() + logger.info("startup.migrations.done duration_ms={:.2f}", (perf_counter() - migrate_started) * 1000) + + bootstrap_started = perf_counter() + db = startup_session_factory() + try: + bootstrap_fn(db) + finally: + close_session_fn(db) + + logger.info("startup.bootstrap.done duration_ms={:.2f}", (perf_counter() - bootstrap_started) * 1000) + logger.info("startup.ready total_duration_ms={:.2f}", (perf_counter() - startup_started) * 1000) + logger.info( + "startup.health status=ready pid={} db_url={}", + os.getpid(), + redact_database_url(database_url), + ) + + +def create_lifespan(startup_action: Callable[[], None]) -> Callable[[FastAPI], AbstractAsyncContextManager[None]]: + @asynccontextmanager + async def lifespan(app: FastAPI) -> AsyncIterator[None]: + startup_action() + try: + yield + finally: + logger.info("shutdown.begin pid={}", os.getpid()) + + return lifespan diff --git a/app_core/logging_config.py b/app_core/logging_config.py new file mode 100644 index 0000000..f0c364b --- /dev/null +++ b/app_core/logging_config.py @@ -0,0 +1,80 @@ +import datetime +import logging +import os +import sys +from typing import Any + +from loguru import logger + + +def _console_log_format(record: dict[str, Any], show_console_source: bool) -> str: + def _escape_markup(value: object) -> str: + return str(value).replace("<", "\\<").replace(">", "\\>") + + message = _escape_markup(record["message"]) + source = f"{_escape_markup(record['name'])}:{_escape_markup(record['function'])}:{record['line']}" + badge = "APP " + message_color = "" + + if message.startswith("request.start"): + badge = "HTTP IN " + message_color = "" + elif message.startswith("request.end"): + badge = "HTTP OUT" + message_color = "" + elif message.startswith("request.error") or message.startswith("request.exception"): + badge = "HTTP ERR" + message_color = "" + elif message.startswith("optimize.gradient"): + badge = "GP " + message_color = "" + elif message.startswith("optimize.provider"): + badge = "LLM " + message_color = "" + elif message.startswith("optimize.config"): + badge = "CFG " + message_color = "" + elif message.startswith("logging.configured"): + badge = "BOOT " + message_color = "" + + source_part = f" {source} " if show_console_source else "" + + return ( + f"{record['time'].astimezone(datetime.UTC):YYYY-MM-DD HH:mm:ss.SSS} UTC " + f"{badge} " + f"{record['level'].name:<8} " + f"{source_part}" + f"{message_color}{message}\n{{exception}}" + ) + + +def configure_logging() -> None: + console_log_level = os.getenv("LOG_LEVEL", "INFO").strip().upper() or "INFO" + show_console_source = os.getenv("SHOW_CONSOLE_SOURCE", "0").strip().lower() in {"1", "true", "yes", "on"} + + os.makedirs("logs", exist_ok=True) + logger.remove() + logger.add( + sys.stderr, + level=console_log_level, + enqueue=True, + backtrace=False, + diagnose=False, + colorize=True, + format=lambda record: _console_log_format(record, show_console_source), + ) + logger.add( + "logs/app.log", + level="DEBUG", + rotation="10 MB", + retention="10 days", + enqueue=True, + backtrace=True, + diagnose=False, + format="{time:YYYY-MM-DD HH:mm:ss.SSS!UTC} UTC | {level} | {name}:{function}:{line} | {message}", + ) + + uvicorn_access_logger = logging.getLogger("uvicorn.access") + uvicorn_access_logger.handlers.clear() + uvicorn_access_logger.propagate = False diff --git a/auth/auth_service.py b/auth/auth_service.py index d2d1334..033118a 100644 --- a/auth/auth_service.py +++ b/auth/auth_service.py @@ -8,8 +8,8 @@ from sqlalchemy.orm import Session import crud -from database import SessionLocal -from models.models import Config, Role, User +from database import SessionLocal, close_db_session, run_db_call +from models.models import Config, User from optimizer.service import build_optimizer_config, get_runtime_optimizer_config from security import ( ACCESS_TOKEN_TTL_SECONDS, @@ -41,7 +41,7 @@ def get_db_session(): # type: ignore[no-untyped-def] try: yield db finally: - db.close() + close_db_session(db) def issue_tokens_for_user(user: User) -> dict[str, Any]: @@ -168,8 +168,8 @@ def change_own_password( if user.password_changed_at is not None: last_change = user.password_changed_at if last_change.tzinfo is None: - last_change = last_change.replace(tzinfo=datetime.timezone.utc) - elapsed = (datetime.datetime.now(datetime.timezone.utc) - last_change).total_seconds() + last_change = last_change.replace(tzinfo=datetime.UTC) + elapsed = (datetime.datetime.now(datetime.UTC) - last_change).total_seconds() remaining = int(CHANGE_PASSWORD_COOLDOWN_SECONDS - elapsed) if remaining > 0: raise HTTPException( @@ -183,7 +183,7 @@ def change_own_password( db, user, password_hash_encrypted=encrypted_hash, - password_changed_at=datetime.datetime.now(datetime.timezone.utc), + password_changed_at=datetime.datetime.now(datetime.UTC), ) @@ -205,7 +205,7 @@ def get_current_user(request: Request, db: Session = Depends(get_db_session)) -> user_id = payload.get("sub") if not user_id: raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid token") - user = crud.get_user_by_id(db, int(user_id)) + user = run_db_call(db, crud.get_user_by_id, int(user_id)) if not user or not user.is_active: raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Invalid user") return user diff --git a/cache/shared_cache.py b/cache/shared_cache.py index bca1322..cb69d1a 100644 --- a/cache/shared_cache.py +++ b/cache/shared_cache.py @@ -2,10 +2,13 @@ import hashlib import json -from typing import Any, Callable +from typing import TYPE_CHECKING, Any from cache.ttl_cache import SharedTTLCache +if TYPE_CHECKING: + from collections.abc import Callable, Mapping + def _canonical_json(payload: Any) -> str: return json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=False, default=str) @@ -88,7 +91,7 @@ def build_prompt_collection_cache_key( return f"{PROMPT_CACHE_PREFIX}{_sha256_text(_canonical_json(payload))}" -def build_optimization_cache_key(fields: dict[str, str | None], config: dict[str, Any], backend_name: str) -> str: +def build_optimization_cache_key(fields: Mapping[str, str | None], config: Mapping[str, Any], backend_name: str) -> str: token = _normalize_text(config.get("effective_llm_api_token")) normalized_config = dict(config) normalized_config["effective_llm_api_token_fingerprint"] = _sha256_text(token) if token else "" diff --git a/cache/ttl_cache.py b/cache/ttl_cache.py index 8601f01..6f08e0e 100644 --- a/cache/ttl_cache.py +++ b/cache/ttl_cache.py @@ -5,7 +5,10 @@ from copy import deepcopy from dataclasses import dataclass from threading import Event, RLock -from typing import Any, Callable +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + from collections.abc import Callable def _default_max_entries() -> int: diff --git a/crud/__init__.py b/crud/__init__.py index 43515e5..1ef5bbd 100644 --- a/crud/__init__.py +++ b/crud/__init__.py @@ -39,42 +39,42 @@ ) __all__ = [ + "DEFAULT_ROLE_NAMES", "_utcnow", - "normalize_project_name", - "normalize_tags", + "add_version", + "count_prompts", "create_project", + "create_prompt", + "create_user", "delete_project", + "delete_prompt", + "delete_user", + "ensure_default_roles", + "get_default_admin_user", + "get_latest_version", "get_or_create_project", "get_or_create_projects", "get_or_create_tags", + "get_or_create_user_config", "get_project_by_id", "get_project_by_name", - "list_projects", - "update_project", - "add_version", - "count_prompts", - "create_prompt", - "delete_prompt", - "get_latest_version", "get_prompt", + "get_role_by_name", "get_specific_version", + "get_user_by_id", + "get_user_by_username", "has_duplicate_prompt_version_content", + "list_projects", "list_prompts", - "list_versions", - "search_prompts_by_tags", - "set_prompt_tags", - "DEFAULT_ROLE_NAMES", - "ensure_default_roles", - "get_role_by_name", "list_roles", - "create_user", - "delete_user", - "get_default_admin_user", - "get_or_create_user_config", - "get_user_by_id", - "get_user_by_username", "list_users", + "list_versions", + "normalize_project_name", + "normalize_tags", "resolve_audit_username", + "search_prompts_by_tags", + "set_prompt_tags", "set_user_projects", + "update_project", "update_user", ] diff --git a/crud/common.py b/crud/common.py index caa195a..e9d0c3b 100644 --- a/crud/common.py +++ b/crud/common.py @@ -1,8 +1,8 @@ -from datetime import datetime, timezone +from datetime import UTC, datetime def _utcnow() -> datetime: - return datetime.now(timezone.utc) + return datetime.now(UTC) def normalize_tags(tags: list[str] | None) -> list[str]: diff --git a/crud/prompt.py b/crud/prompt.py index 6ff5a5a..ef204ce 100644 --- a/crud/prompt.py +++ b/crud/prompt.py @@ -1,5 +1,5 @@ -from sqlalchemy import func -from sqlalchemy.orm import Session, joinedload +from sqlalchemy import false, func +from sqlalchemy.orm import Query, Session, joinedload from models import Project, Prompt, PromptVersion, Tag @@ -98,7 +98,12 @@ def get_prompt(db: Session, name: str, project: str, allowed_projects: list[str] query = ( db.query(Prompt) .join(Prompt.project_ref) - .options(joinedload(Prompt.project_ref), joinedload(Prompt.created_by_ref), joinedload(Prompt.updated_by_ref)) + .options( + joinedload(Prompt.project_ref), + joinedload(Prompt.created_by_ref), + joinedload(Prompt.updated_by_ref), + joinedload(Prompt.tags), + ) .filter(Prompt.name == name, Project.name == normalize_project_name(project)) ) if allowed_projects is not None: @@ -118,17 +123,18 @@ def _build_prompt_list_query( project: str | None = None, tag: str | None = None, allowed_projects: list[str] | None = None, -): # type: ignore[no-untyped-def] +) -> Query[Prompt]: query = db.query(Prompt) query = query.join(Prompt.project_ref).options( joinedload(Prompt.project_ref), joinedload(Prompt.created_by_ref), joinedload(Prompt.updated_by_ref), + joinedload(Prompt.tags), ) if allowed_projects is not None: if not allowed_projects: - return query.filter(False) + return query.filter(false()) query = query.filter(Project.name.in_(allowed_projects)) if project: @@ -282,7 +288,12 @@ def search_prompts_by_tags( return [] query = db.query(Prompt) - query = query.join(Prompt.project_ref).options(joinedload(Prompt.project_ref)) + query = query.join(Prompt.project_ref).options( + joinedload(Prompt.project_ref), + joinedload(Prompt.created_by_ref), + joinedload(Prompt.updated_by_ref), + joinedload(Prompt.tags), + ) if allowed_projects is not None: if not allowed_projects: @@ -303,7 +314,8 @@ def search_prompts_by_tags( query = query.filter(Prompt.id.in_(subq)) else: # OR: at least one tag matches - query = query.join(Prompt.tags).filter(Tag.name.in_(normalized)).distinct() + subq = db.query(Prompt.id).join(Prompt.tags).filter(Tag.name.in_(normalized)) + query = query.filter(Prompt.id.in_(subq)) results = query.order_by(Project.name.asc(), Prompt.name.asc()).all() return list(results) if results else [] diff --git a/crud/user.py b/crud/user.py index 1a32b8e..bf17e6c 100644 --- a/crud/user.py +++ b/crud/user.py @@ -1,3 +1,5 @@ +import datetime + from sqlalchemy import func from sqlalchemy.orm import Session, joinedload @@ -97,7 +99,7 @@ def update_user( role: str | None = None, is_active: bool | None = None, projects: list[str] | None = None, - password_changed_at=None, + password_changed_at: datetime.datetime | None = None, ) -> User: if username is not None: user.username = username.strip() diff --git a/database.py b/database.py index 05612eb..82b4746 100644 --- a/database.py +++ b/database.py @@ -1,22 +1,28 @@ import os +from collections.abc import Callable from pathlib import Path -from typing import Any +from typing import Any, Concatenate, ParamSpec, TypeVar -from alembic import command from alembic.config import Config from sqlalchemy import create_engine, event -from sqlalchemy.orm import declarative_base, sessionmaker +from sqlalchemy.orm import DeclarativeBase, Session, sessionmaker + +from alembic import command SQLALCHEMY_DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///./prompts.db") is_sqlite = SQLALCHEMY_DATABASE_URL.startswith("sqlite") -engine_kwargs: dict[str, Any] = { +P = ParamSpec("P") +T = TypeVar("T") +DatabaseSession = Session + +sync_engine_kwargs: dict[str, Any] = { "pool_pre_ping": True, } if is_sqlite: - engine_kwargs.update( + sync_engine_kwargs.update( { "connect_args": { "check_same_thread": False, @@ -25,7 +31,7 @@ } ) else: - engine_kwargs.update( + sync_engine_kwargs.update( { "pool_size": 20, "max_overflow": 40, @@ -34,11 +40,15 @@ } ) -engine = create_engine(SQLALCHEMY_DATABASE_URL, **engine_kwargs) +startup_engine = create_engine(SQLALCHEMY_DATABASE_URL, **sync_engine_kwargs) +engine = startup_engine +SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine) + +StartupSessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=startup_engine) if is_sqlite: - @event.listens_for(engine, "connect") + @event.listens_for(startup_engine, "connect") def set_sqlite_pragma(dbapi_connection, connection_record) -> None: # type: ignore[no-untyped-def] cursor = dbapi_connection.cursor() cursor.execute("PRAGMA journal_mode=WAL;") @@ -47,9 +57,17 @@ def set_sqlite_pragma(dbapi_connection, connection_record) -> None: # type: ign cursor.execute("PRAGMA foreign_keys=ON;") cursor.close() -SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine) -Base = declarative_base() +class Base(DeclarativeBase): + pass + + +def close_db_session(db: DatabaseSession) -> None: + db.close() + + +def run_db_call(db: DatabaseSession, operation: Callable[Concatenate[DatabaseSession, P], T], /, *args: P.args, **kwargs: P.kwargs) -> T: + return operation(db, *args, **kwargs) def _run_alembic_upgrade() -> None: @@ -70,4 +88,4 @@ def init_database(bind=None) -> None: # type: ignore[no-untyped-def] return _run_alembic_upgrade() - Base.metadata.create_all(bind=engine) + Base.metadata.create_all(bind=startup_engine) diff --git a/diagnostic.py b/diagnostic.py index a123543..c24badc 100644 --- a/diagnostic.py +++ b/diagnostic.py @@ -2,7 +2,7 @@ import requests -def run_diagnostic(): # type: ignore[no-untyped-def] +def run_diagnostic() -> None: try: # 1) GET config config_url = "http://127.0.0.1:8000/optimize/config" diff --git a/loadtests/benchmark_rps.py b/loadtests/benchmark_rps.py index f65db92..2214b91 100644 --- a/loadtests/benchmark_rps.py +++ b/loadtests/benchmark_rps.py @@ -8,12 +8,12 @@ import shutil import subprocess import sys +import time from dataclasses import asdict, dataclass from pathlib import Path import requests - SCENARIO_TARGET_NAMES = { "optimize_hot": "POST /optimize [hot]", "optimize_cold": "POST /optimize [cold]", @@ -107,17 +107,27 @@ def _fmt(value: float) -> str: def _fetch_auth_token(host: str, username: str, password: str) -> str: - response = requests.post( - f"{host.rstrip('/')}/auth/login", - json={"username": username, "password": password}, - timeout=30, - ) - response.raise_for_status() - payload = response.json() - token = str(payload.get("access_token") or "").strip() - if not token: - raise RuntimeError("Benchmark login did not return access_token") - return token + last_error: Exception | None = None + for attempt in range(1, 6): + try: + response = requests.post( + f"{host.rstrip('/')}/auth/login", + json={"username": username, "password": password}, + timeout=60, + ) + response.raise_for_status() + payload = response.json() + token = str(payload.get("access_token") or "").strip() + if not token: + raise RuntimeError("Benchmark login did not return access_token") + return token + except requests.RequestException as exc: + last_error = exc + if attempt == 5: + break + time.sleep(min(10, attempt * 2)) + + raise RuntimeError(f"Benchmark login failed after retries: {last_error}") def _run_scenario(args: argparse.Namespace, scenario: str, outdir: Path) -> list[RunResult]: diff --git a/loadtests/chart_db_mode_40_p95.png b/loadtests/chart_db_mode_40_p95.png new file mode 100644 index 0000000..5f8887e Binary files /dev/null and b/loadtests/chart_db_mode_40_p95.png differ diff --git a/loadtests/chart_db_mode_40_rps.png b/loadtests/chart_db_mode_40_rps.png new file mode 100644 index 0000000..e074f09 Binary files /dev/null and b/loadtests/chart_db_mode_40_rps.png differ diff --git a/loadtests/chart_db_mode_cold_10_p95.png b/loadtests/chart_db_mode_cold_10_p95.png new file mode 100644 index 0000000..50228fc Binary files /dev/null and b/loadtests/chart_db_mode_cold_10_p95.png differ diff --git a/loadtests/chart_db_mode_dashboard.png b/loadtests/chart_db_mode_dashboard.png new file mode 100644 index 0000000..df740b6 Binary files /dev/null and b/loadtests/chart_db_mode_dashboard.png differ diff --git a/loadtests/results/db_compare/postgres_async/benchmark_manifest.json b/loadtests/results/db_compare/postgres_async/benchmark_manifest.json new file mode 100644 index 0000000..0788baf --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/benchmark_manifest.json @@ -0,0 +1,118 @@ +{ + "scenarios": [ + "mixed", + "cache", + "optimize_hot", + "optimize_cold" + ], + "results": [ + { + "scenario": "mixed", + "users": 10, + "requests_per_s": 31.765306251711028, + "failure_rate": 0.0, + "p95_ms": 130.0, + "avg_ms": 53.74216636758446, + "passed": true + }, + { + "scenario": "mixed", + "users": 20, + "requests_per_s": 41.762599202568275, + "failure_rate": 0.0, + "p95_ms": 700.0, + "avg_ms": 169.63468771363304, + "passed": false + }, + { + "scenario": "mixed", + "users": 40, + "requests_per_s": 36.220388663349695, + "failure_rate": 0.0, + "p95_ms": 3100.0, + "avg_ms": 573.1448941060258, + "passed": false + }, + { + "scenario": "cache", + "users": 10, + "requests_per_s": 98.69318282692613, + "failure_rate": 0.0, + "p95_ms": 22.0, + "avg_ms": 12.702637761205079, + "passed": true + }, + { + "scenario": "cache", + "users": 20, + "requests_per_s": 173.1644083878897, + "failure_rate": 0.0, + "p95_ms": 40.0, + "avg_ms": 21.023119686805657, + "passed": true + }, + { + "scenario": "cache", + "users": 40, + "requests_per_s": 179.72490218244113, + "failure_rate": 0.0, + "p95_ms": 200.0, + "avg_ms": 99.25260803150522, + "passed": true + }, + { + "scenario": "optimize_hot", + "users": 10, + "requests_per_s": 19.18909392932623, + "failure_rate": 0.0, + "p95_ms": 23.0, + "avg_ms": 17.581401114762315, + "passed": true + }, + { + "scenario": "optimize_hot", + "users": 20, + "requests_per_s": 36.987391461391404, + "failure_rate": 0.0, + "p95_ms": 33.0, + "avg_ms": 19.519349806737534, + "passed": true + }, + { + "scenario": "optimize_hot", + "users": 40, + "requests_per_s": 68.61946835160887, + "failure_rate": 0.0, + "p95_ms": 100.0, + "avg_ms": 30.016994813331056, + "passed": true + }, + { + "scenario": "optimize_cold", + "users": 10, + "requests_per_s": 0.17827204092836194, + "failure_rate": 0.0, + "p95_ms": 11000.0, + "avg_ms": 7144.589100003941, + "passed": false + }, + { + "scenario": "optimize_cold", + "users": 20, + "requests_per_s": 0.0, + "failure_rate": 1.0, + "p95_ms": Infinity, + "avg_ms": Infinity, + "passed": false + }, + { + "scenario": "optimize_cold", + "users": 40, + "requests_per_s": 0.0, + "failure_rate": 1.0, + "p95_ms": Infinity, + "avg_ms": Infinity, + "passed": false + } + ] +} \ No newline at end of file diff --git a/loadtests/results/db_compare/postgres_async/benchmark_summary.md b/loadtests/results/db_compare/postgres_async/benchmark_summary.md new file mode 100644 index 0000000..9e60a7d --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/benchmark_summary.md @@ -0,0 +1,16 @@ +# Benchmark Summary + +| Scenario | Users | RPS | P95 (ms) | Avg (ms) | Failure % | Pass | +| --- | ---: | ---: | ---: | ---: | ---: | :---: | +| mixed | 10 | 31.77 | 130.00 | 53.74 | 0.00 | yes | +| mixed | 20 | 41.76 | 700.00 | 169.63 | 0.00 | no | +| mixed | 40 | 36.22 | 3100.00 | 573.14 | 0.00 | no | +| cache | 10 | 98.69 | 22.00 | 12.70 | 0.00 | yes | +| cache | 20 | 173.16 | 40.00 | 21.02 | 0.00 | yes | +| cache | 40 | 179.72 | 200.00 | 99.25 | 0.00 | yes | +| optimize_hot | 10 | 19.19 | 23.00 | 17.58 | 0.00 | yes | +| optimize_hot | 20 | 36.99 | 33.00 | 19.52 | 0.00 | yes | +| optimize_hot | 40 | 68.62 | 100.00 | 30.02 | 0.00 | yes | +| optimize_cold | 10 | 0.18 | 11000.00 | 7144.59 | 0.00 | no | +| optimize_cold | 20 | 0.00 | inf | inf | 100.00 | no | +| optimize_cold | 40 | 0.00 | inf | inf | 100.00 | no | diff --git a/loadtests/results/db_compare/postgres_async/cache/u10_exceptions.csv b/loadtests/results/db_compare/postgres_async/cache/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/cache/u10_failures.csv b/loadtests/results/db_compare/postgres_async/cache/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/cache/u10_stats.csv b/loadtests/results/db_compare/postgres_async/cache/u10_stats.csv new file mode 100644 index 0000000..5bb9153 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u10_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts [hot],360,0,11,12.601189998612325,6.225299992365763,129.75559997721575,4161.0,25.653101673424842,0.0,11,13,14,15,18,21,31,34,130,130,130 +GET,GET /prompts/search [hot],279,0,10,12.15402222220539,6.68049999512732,134.23630001489073,5115.0,19.881153796904254,0.0,10,13,14,14,18,20,26,36,130,130,130 +GET,GET /prompts/{project}/{name} [hot],482,0,10,11.80159916944997,6.213099986780435,99.28320001927204,425.0,34.346652796085486,0.0,10,12,13,14,17,21,26,29,99,99,99 +GET,GET /prompts/{project}/{name}/versions [hot],201,0,10,12.657361691794474,5.954699998255819,104.04549998929724,272.0,14.322981767662204,0.0,10,12,14,14,18,22,27,100,100,100,100 +POST,POST /optimize [hot],50,0,20,24.36687200097367,11.07789998059161,99.64990001753904,753.0,3.5629307879756724,0.0,21,26,27,30,32,33,100,100,100,100,100 +POST,POST /prompts [seed],12,0,15,15.77874999929918,13.091900007566437,19.938899989938363,34.0,0.8551033891141614,0.0,16,16,18,18,18,20,20,20,20,20,20 +PUT,PUT /optimize/config,1,0,25.56360000744462,25.56360000744462,25.56360000744462,25.56360000744462,507.0,0.07125861575951345,0.0,26,26,26,26,26,26,26,26,26,26,26 +,Aggregated,1385,0,11,12.702637761205079,5.954699998255819,134.23630001489073,2327.171119133574,98.69318282692613,0.0,11,13,14,15,19,22,30,36,130,130,130 diff --git a/loadtests/results/db_compare/postgres_async/cache/u10_stats_history.csv b/loadtests/results/db_compare/postgres_async/cache/u10_stats_history.csv new file mode 100644 index 0000000..c2f7873 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168568,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168569,10,,Aggregated,0.000000,0.000000,13,15,16,19,98,100,130,130,130,130,130,84,0,13,24.09792261841219,6.331200012937188,134.23630001489073,1838.0 +1779168570,10,,Aggregated,0.000000,0.000000,12,14,15,18,30,97,100,130,130,130,130,182,0,11,18.34665274661377,6.331200012937188,134.23630001489073,2307.71978021978 +1779168571,10,,Aggregated,24.000000,0.000000,11,13,15,16,22,31,100,130,130,130,130,285,0,11,15.76303894633607,6.225299992365763,134.23630001489073,2345.315789473684 +1779168572,10,,Aggregated,60.000000,0.000000,11,13,14,16,21,30,99,100,130,130,130,379,0,11,14.762686014772543,6.225299992365763,134.23630001489073,2334.3562005277045 +1779168573,10,,Aggregated,73.000000,0.000000,11,13,14,15,20,26,97,100,130,130,130,477,0,11,14.042781550299075,6.225299992365763,134.23630001489073,2385.6918238993712 +1779168574,10,,Aggregated,79.250000,0.000000,11,13,14,15,20,24,34,100,130,130,130,583,0,11,13.503836191101176,6.225299992365763,134.23630001489073,2352.8987993138935 +1779168575,10,,Aggregated,82.800000,0.000000,11,13,14,15,20,23,32,100,130,130,130,683,0,11,13.389574669865349,6.225299992365763,134.23630001489073,2335.2679355783307 +1779168576,10,,Aggregated,85.333333,0.000000,11,13,14,15,19,23,32,99,130,130,130,789,0,11,13.146587705557767,6.225299992365763,134.23630001489073,2334.370088719899 +1779168577,10,,Aggregated,88.142857,0.000000,11,13,14,15,19,23,32,98,130,130,130,880,0,11,13.079079545257793,6.218599999556318,134.23630001489073,2378.9886363636365 +1779168578,10,,Aggregated,89.625000,0.000000,11,13,14,15,19,22,31,97,130,130,130,981,0,11,12.888045973219471,5.954699998255819,134.23630001489073,2363.0560652395516 diff --git a/loadtests/results/db_compare/postgres_async/cache/u20_exceptions.csv b/loadtests/results/db_compare/postgres_async/cache/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/cache/u20_failures.csv b/loadtests/results/db_compare/postgres_async/cache/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/cache/u20_stats.csv b/loadtests/results/db_compare/postgres_async/cache/u20_stats.csv new file mode 100644 index 0000000..90886f1 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u20_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts [hot],640,0,18,20.095270311912827,6.5814999979920685,97.65520002110861,4161.0,45.644654599773226,0.0,18,21,25,27,32,38,52,75,98,98,98 +GET,GET /prompts/search [hot],501,0,19,20.250058083986946,6.269599980441853,97.41560000111349,5115.0,35.73120617888498,0.0,19,22,25,26,32,36,44,77,97,97,97 +GET,GET /prompts/{project}/{name} [hot],829,0,18,20.329840410048867,6.176700000651181,96.98699999717064,425.0,59.12409166126876,0.0,18,22,25,27,33,36,48,66,97,97,97 +GET,GET /prompts/{project}/{name}/versions [hot],341,0,18,21.081690028971444,6.2049999833106995,97.1441000001505,272.0,24.320042528941674,0.0,18,22,25,28,34,42,73,76,97,97,97 +POST,POST /optimize [hot],104,0,33,36.10298750041133,11.367900006007403,115.11710000922903,753.0,7.417256372463149,0.0,33,40,45,48,53,68,97,99,120,120,120 +POST,POST /prompts [seed],12,0,15,18.088133333852358,12.945700000273064,46.44590002135374,34.0,0.855837273745748,0.0,16,16,17,17,19,46,46,46,46,46,46 +PUT,PUT /optimize/config,1,0,23.820200003683567,23.820200003683567,23.820200003683567,23.820200003683567,507.0,0.07131977281214567,0.0,24,24,24,24,24,24,24,24,24,24,24 +,Aggregated,2428,0,18,21.023119686805657,6.176700000651181,115.11710000922903,2368.1873970345964,173.1644083878897,0.0,18,22,25,28,33,40,56,75,98,120,120 diff --git a/loadtests/results/db_compare/postgres_async/cache/u20_stats_history.csv b/loadtests/results/db_compare/postgres_async/cache/u20_stats_history.csv new file mode 100644 index 0000000..4151373 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168584,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168585,10,,Aggregated,0.000000,0.000000,13,15,16,17,34,38,49,50,50,50,50,89,0,13,15.497910112909073,6.3633999961894006,49.99390000011772,2231.0112359550562 +1779168586,20,,Aggregated,0.000000,0.000000,16,20,24,34,47,69,79,81,96,96,96,252,0,16,22.284598016186564,6.3633999961894006,95.72830001707189,2356.2301587301586 +1779168587,20,,Aggregated,6.000000,0.000000,18,23,28,31,40,48,76,80,96,96,96,443,0,18,22.141476072167084,6.2049999833106995,95.72830001707189,2354.0474040632052 +1779168588,20,,Aggregated,53.500000,0.000000,19,22,25,28,35,46,69,78,96,96,96,630,0,19,21.318012222302116,6.2049999833106995,95.72830001707189,2329.0412698412697 +1779168589,20,,Aggregated,91.000000,0.000000,19,23,26,29,34,45,50,76,96,96,96,812,0,19,21.330882142648605,6.2049999833106995,95.72830001707189,2414.543103448276 +1779168590,20,,Aggregated,115.250000,0.000000,19,23,26,29,34,46,56,76,96,96,96,990,0,19,21.438288787906668,6.2049999833106995,95.72830001707189,2449.3242424242426 +1779168591,20,,Aggregated,129.600000,0.000000,19,22,25,28,33,44,54,73,82,96,96,1171,0,19,20.86507941931836,6.2049999833106995,95.72830001707189,2435.929120409906 +1779168592,20,,Aggregated,138.166667,0.000000,18,22,25,28,34,43,53,68,82,96,96,1352,0,18,20.839654585827397,6.2049999833106995,95.72830001707189,2429.1597633136093 +1779168593,20,,Aggregated,143.714286,0.000000,18,22,25,27,33,41,53,66,82,96,96,1519,0,18,20.655127913128418,6.2049999833106995,95.72830001707189,2418.5358788676763 +1779168594,20,,Aggregated,148.250000,0.000000,18,23,25,28,33,44,59,78,99,120,120,1704,0,18,21.212472241767983,6.2049999833106995,115.11710000922903,2395.7153755868544 diff --git a/loadtests/results/db_compare/postgres_async/cache/u40_exceptions.csv b/loadtests/results/db_compare/postgres_async/cache/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/cache/u40_failures.csv b/loadtests/results/db_compare/postgres_async/cache/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/cache/u40_stats.csv b/loadtests/results/db_compare/postgres_async/cache/u40_stats.csv new file mode 100644 index 0000000..5b94413 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u40_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts [hot],663,0,92,95.19135218697845,7.058200018946081,436.1969000019599,4161.0,47.37877143020218,0.0,92,99,100,110,130,190,240,400,440,440,440 +GET,GET /prompts/search [hot],508,0,92,95.11648110028331,6.310600001597777,494.8045000201091,5115.0,36.30228640504179,0.0,92,98,100,110,130,170,270,420,490,490,490 +GET,GET /prompts/{project}/{name} [hot],869,0,90,92.87426260088853,6.547400000272319,470.9006000193767,425.0,62.09977733460889,0.0,90,98,100,110,130,180,240,360,470,470,470 +GET,GET /prompts/{project}/{name}/versions [hot],294,0,90,95.11044829955948,6.418699980713427,441.51209999108687,272.0,21.00959095094938,0.0,91,100,100,110,130,170,260,400,440,440,440 +POST,POST /optimize [hot],168,0,160,174.6022767862693,11.04689997737296,558.7517999811098,753.0,12.005480543399647,0.0,160,180,190,200,270,320,540,560,560,560,560 +POST,POST /prompts [seed],12,0,13,13.64945000144265,12.431600014679134,15.650700021069497,34.0,0.8575343245285462,0.0,14,14,14,14,15,16,16,16,16,16,16 +PUT,PUT /optimize/config,1,0,22.088399971835315,22.088399971835315,22.088399971835315,22.088399971835315,507.0,0.07146119371071219,0.0,22,22,22,22,22,22,22,22,22,22,22 +,Aggregated,2515,0,93,99.25260803150522,6.310600001597777,558.7517999811098,2359.393638170974,179.72490218244113,0.0,93,100,110,120,150,200,270,400,540,560,560 diff --git a/loadtests/results/db_compare/postgres_async/cache/u40_stats_history.csv b/loadtests/results/db_compare/postgres_async/cache/u40_stats_history.csv new file mode 100644 index 0000000..527269b --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/cache/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168601,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168602,10,,Aggregated,0.000000,0.000000,11,13,15,16,34,36,47,48,48,48,48,91,0,11,14.160656042404698,6.547400000272319,47.7000999962911,1691.4615384615386 +1779168603,20,,Aggregated,0.000000,0.000000,16,21,26,32,39,43,48,64,75,75,75,270,0,16,19.93771148101789,6.310600001597777,75.2243000024464,2168.303703703704 +1779168604,40,,Aggregated,68.000000,0.000000,29,44,52,54,75,200,270,270,310,310,310,431,0,29,45.52096171714333,6.310600001597777,308.0195000220556,2251.7470997679816 +1779168605,40,,Aggregated,113.500000,0.000000,44,63,76,100,220,380,440,490,560,560,560,567,0,44,81.44383174599258,6.310600001597777,558.7517999811098,2248.1181657848324 +1779168606,40,,Aggregated,129.333333,0.000000,63,87,110,120,190,360,430,470,560,560,560,757,0,63,88.00302140009953,6.310600001597777,558.7517999811098,2263.233817701453 +1779168607,40,,Aggregated,130.000000,0.000000,76,94,110,120,150,270,430,440,560,560,560,962,0,76,89.74834002055597,6.310600001597777,558.7517999811098,2281.95841995842 +1779168608,40,,Aggregated,144.000000,0.000000,82,96,110,120,190,260,400,440,560,560,560,1134,0,82,95.25643174588471,6.310600001597777,558.7517999811098,2277.0458553791887 +1779168609,40,,Aggregated,151.666667,0.000000,86,100,120,130,190,250,390,430,560,560,560,1310,0,86,99.59538305340023,6.310600001597777,558.7517999811098,2287.9740458015267 +1779168610,40,,Aggregated,154.857143,0.000000,87,99,110,120,180,240,370,430,560,560,560,1521,0,87,98.21350618000669,6.310600001597777,558.7517999811098,2313.1505588428668 +1779168611,40,,Aggregated,157.875000,0.000000,89,100,110,120,170,230,370,430,560,560,560,1723,0,89,98.79690487494771,6.310600001597777,558.7517999811098,2347.561810795125 diff --git a/loadtests/results/db_compare/postgres_async/mixed/u10_exceptions.csv b/loadtests/results/db_compare/postgres_async/mixed/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/mixed/u10_failures.csv b/loadtests/results/db_compare/postgres_async/mixed/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/mixed/u10_stats.csv b/loadtests/results/db_compare/postgres_async/mixed/u10_stats.csv new file mode 100644 index 0000000..1fc1184 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u10_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts,269,0,13,36.411173605545315,7.277500000782311,224.18079999624752,9684.791821561339,19.158895474686698,0.0,13,24,52,65,100,130,180,220,220,220,220 +GET,GET /prompts/search,84,0,16,51.67062737988696,7.718500011833385,173.5525000258349,13725.809523809523,5.982703419604768,0.0,16,71,99,120,130,150,160,170,170,170,170 +POST,POST /optimize,4,0,17,1381.488500002888,12.122200016165152,5466.924800013658,753.0,0.28489063902879846,0.0,30,30,5500,5500,5500,5500,5500,5500,5500,5500,5500 +POST,POST /prompts,38,0,38,45.701463158413965,25.75600001728162,141.32120000431314,400.0,2.706461070773585,0.0,39,46,54,57,68,100,140,140,140,140,140 +POST,POST /prompts [seed],12,0,28,30.489300006593112,26.82739999727346,47.64820000855252,425.1666666666667,0.8546719170863953,0.0,29,29,33,33,34,48,48,48,48,48,48 +PUT,PUT /optimize/config,1,0,30.756300024222583,30.756300024222583,30.756300024222583,30.756300024222583,507.0,0.07122265975719962,0.0,31,31,31,31,31,31,31,31,31,31,31 +PUT,PUT /prompts/{project}/{name},38,0,48,57.232376315943455,32.462500006658956,115.80620001768693,254.0,2.706461070773585,0.0,50,61,74,80,95,110,120,120,120,120,120 +,Aggregated,446,0,23,53.74216636758446,7.277500000782311,5466.924800013658,8501.45739910314,31.765306251711028,0.0,23,43,60,74,110,130,160,220,5500,5500,5500 diff --git a/loadtests/results/db_compare/postgres_async/mixed/u10_stats_history.csv b/loadtests/results/db_compare/postgres_async/mixed/u10_stats_history.csv new file mode 100644 index 0000000..c0de5e8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168519,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168520,10,,Aggregated,0.000000,0.000000,48,95,140,140,220,220,220,220,220,220,220,32,0,34,82.9721656282345,22.902100026840344,224.18079999624752,4143.21875 +1779168521,10,,Aggregated,0.000000,0.000000,63,79,92,120,140,220,220,220,220,220,220,65,0,63,72.48892769387636,10.57859999127686,224.18079999624752,5375.569230769231 +1779168522,10,,Aggregated,17.000000,0.000000,58,77,95,120,140,220,220,220,220,220,220,95,0,58,66.64074947414184,7.7923999924678355,224.18079999624752,6221.0526315789475 +1779168523,10,,Aggregated,25.500000,0.000000,46,74,89,110,130,160,220,220,220,220,220,125,0,46,61.98224480077624,7.7923999924678355,224.18079999624752,6743.832 +1779168524,10,,Aggregated,27.000000,0.000000,40,68,86,99,130,160,220,220,220,220,220,154,0,37,57.90371558509537,7.7923999924678355,224.18079999624752,7104.376623376624 +1779168525,10,,Aggregated,28.000000,0.000000,37,65,80,97,130,140,220,220,5500,5500,5500,185,0,37,84.40656864927527,7.7923999924678355,5466.924800013658,7016.254054054054 +1779168526,10,,Aggregated,28.600000,0.000000,27,57,74,84,130,140,220,220,5500,5500,5500,217,0,27,74.01773502324892,7.284499995876104,5466.924800013658,7416.087557603687 +1779168527,10,,Aggregated,28.833333,0.000000,27,50,71,83,120,140,220,220,5500,5500,5500,253,0,27,68.30941818184274,7.284499995876104,5466.924800013658,7641.790513833992 +1779168528,10,,Aggregated,28.714286,0.000000,26,48,65,79,120,140,220,220,5500,5500,5500,287,0,26,64.26913170734151,7.277500000782311,5466.924800013658,7766.623693379791 +1779168529,10,,Aggregated,29.625000,0.000000,26,48,65,77,120,130,180,220,5500,5500,5500,321,0,26,61.539619003068594,7.277500000782311,5466.924800013658,7918.647975077882 diff --git a/loadtests/results/db_compare/postgres_async/mixed/u20_exceptions.csv b/loadtests/results/db_compare/postgres_async/mixed/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/mixed/u20_failures.csv b/loadtests/results/db_compare/postgres_async/mixed/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/mixed/u20_stats.csv b/loadtests/results/db_compare/postgres_async/mixed/u20_stats.csv new file mode 100644 index 0000000..d5cac6d --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u20_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts,317,0,110,132.08164889628011,7.72069999948144,607.0922999933828,8893.27760252366,22.59171322050195,0.0,110,160,190,200,260,390,540,610,610,610,610 +GET,GET /prompts/search,85,0,500,469.5608258822604,9.38889998360537,950.6716000032611,41027.01176470588,6.057714901396422,0.0,500,700,720,790,830,870,900,950,950,950,950 +POST,POST /optimize,13,0,52,67.84240000818569,21.07710001291707,270.98850000766106,753.0,0.9264740437429823,0.0,52,58,64,64,81,270,270,270,270,270,270 +POST,POST /prompts,79,0,73,98.65969240582913,26.03930002078414,448.3831999823451,400.0,5.630111496591969,0.0,73,87,95,110,170,380,440,450,450,450,450 +POST,POST /prompts [seed],12,0,13,13.186283336835913,11.650400003418326,15.076700015924871,34.0,0.8552068096089067,0.0,13,13,14,14,14,15,15,15,15,15,15 +PUT,PUT /optimize/config,1,0,23.640800005523488,23.640800005523488,23.640800005523488,23.640800005523488,507.0,0.07126723413407555,0.0,24,24,24,24,24,24,24,24,24,24,24 +PUT,PUT /prompts/{project}/{name},79,0,95,110.95482278305778,47.269700007745996,462.52390000154264,254.0,5.630111496591969,0.0,95,110,120,140,180,210,340,460,460,460,460 +,Aggregated,586,0,96,169.63468771363304,7.72069999948144,950.6716000032611,10868.319112627987,41.762599202568275,0.0,96,150,190,230,460,700,820,870,950,950,950 diff --git a/loadtests/results/db_compare/postgres_async/mixed/u20_stats_history.csv b/loadtests/results/db_compare/postgres_async/mixed/u20_stats_history.csv new file mode 100644 index 0000000..c0a240e --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168535,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168536,10,,Aggregated,0.000000,0.000000,14,47,56,72,130,140,140,140,140,140,140,37,0,14,43.069129733437975,8.006399992154911,135.6404999969527,4576.135135135135 +1779168537,20,,Aggregated,0.000000,0.000000,70,130,180,440,540,610,610,670,670,670,670,75,0,70,160.20152133346224,8.006399992154911,668.3709999779239,5405.426666666666 +1779168538,20,,Aggregated,25.000000,0.000000,88,130,160,230,450,580,610,610,670,670,670,124,0,86,148.94057096793287,8.006399992154911,668.3709999779239,6608.943548387097 +1779168539,20,,Aggregated,20.500000,0.000000,86,130,150,220,440,500,610,610,670,670,670,175,0,86,141.1446588576239,8.006399992154911,668.3709999779239,7184.828571428571 +1779168540,20,,Aggregated,31.000000,0.000000,88,140,180,230,410,480,610,610,670,670,670,206,0,87,144.4724480586479,8.006399992154911,668.3709999779239,7478.23786407767 +1779168541,20,,Aggregated,34.250000,0.000000,95,150,210,250,450,540,670,800,850,850,850,245,0,95,162.7581338783042,8.006399992154911,848.2294999994338,7923.771428571428 +1779168542,20,,Aggregated,37.400000,0.000000,110,160,230,260,460,610,800,830,850,850,850,289,0,110,173.68650726704524,8.006399992154911,848.2294999994338,8558.453287197231 +1779168543,20,,Aggregated,37.000000,0.000000,110,160,230,260,460,660,810,830,850,850,850,331,0,110,177.6251628403369,8.006399992154911,848.3662999933586,9191.287009063444 +1779168544,20,,Aggregated,36.571429,0.000000,100,150,210,250,460,630,800,830,850,850,850,373,0,100,172.27562868662724,7.72069999948144,848.3662999933586,9664.58981233244 +1779168545,20,,Aggregated,37.375000,0.000000,100,160,210,250,480,660,790,810,850,850,850,410,0,100,174.4581612198267,7.72069999948144,848.3662999933586,10040.163414634146 diff --git a/loadtests/results/db_compare/postgres_async/mixed/u40_exceptions.csv b/loadtests/results/db_compare/postgres_async/mixed/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/mixed/u40_failures.csv b/loadtests/results/db_compare/postgres_async/mixed/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/mixed/u40_stats.csv b/loadtests/results/db_compare/postgres_async/mixed/u40_stats.csv new file mode 100644 index 0000000..40c4190 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u40_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts,277,0,390,458.28808953097536,7.53249999252148,1232.9513000149745,8258.595667870037,19.71129206237302,0.0,390,620,760,830,930,1000,1200,1200,1200,1200,1200 +GET,GET /prompts/search,34,0,3600,3108.363944116932,35.96579999430105,4678.744799981359,65908.32352941176,2.419436570832789,0.0,3700,3800,4300,4400,4600,4600,4700,4700,4700,4700,4700 +POST,POST /optimize,29,0,130,178.42948620894057,55.27549999533221,497.42739999783225,753.0,2.063637075122085,0.0,130,170,190,230,430,490,500,500,500,500,500 +POST,POST /prompts,78,0,290,343.43464871740133,45.293200004380196,892.0622000005096,400.0,5.550472133086987,0.0,290,400,450,500,600,680,780,890,890,890,890 +POST,POST /prompts [seed],12,0,13,13.364883338605674,12.296500004595146,14.764800027478486,34.0,0.8539187897056904,0.0,13,13,14,14,15,15,15,15,15,15,15 +PUT,PUT /optimize/config,1,0,25.366799993207678,25.366799993207678,25.366799993207678,25.366799993207678,507.0,0.07115989914214085,0.0,25,25,25,25,25,25,25,25,25,25,25 +PUT,PUT /prompts/{project}/{name},78,0,300,345.5445269210031,52.76280001271516,673.1220000074245,254.0,5.550472133086987,0.0,310,390,450,480,570,600,660,670,670,670,670 +,Aggregated,509,0,330,573.1448941060258,7.53249999252148,4678.744799981359,9041.803536345777,36.220388663349695,0.0,330,510,650,750,940,3100,3900,4600,4700,4700,4700 diff --git a/loadtests/results/db_compare/postgres_async/mixed/u40_stats_history.csv b/loadtests/results/db_compare/postgres_async/mixed/u40_stats_history.csv new file mode 100644 index 0000000..20038a3 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/mixed/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168551,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168552,10,,Aggregated,0.000000,0.000000,13,25,55,73,140,140,260,260,260,260,260,42,0,13,42.503190477700194,7.53249999252148,257.91929999832064,7763.333333333333 +1779168553,20,,Aggregated,0.000000,0.000000,53,100,150,150,260,280,880,890,890,890,890,80,0,45,105.27996000091662,7.53249999252148,890.3409999911673,8211.3875 +1779168554,30,,Aggregated,4.000000,0.000000,140,240,280,500,810,910,910,910,1600,1600,1600,110,0,130,245.29793727322777,7.53249999252148,1551.2052999984007,7965.727272727273 +1779168555,40,,Aggregated,21.500000,0.000000,140,260,460,650,830,910,910,910,1600,1600,1600,120,0,140,271.2473366676325,7.53249999252148,1551.2052999984007,7436.883333333333 +1779168556,40,,Aggregated,27.333333,0.000000,190,340,610,700,910,1200,1600,3100,3200,3200,3200,172,0,180,380.05637441951984,7.53249999252148,3173.7466999911703,8165.372093023256 +1779168557,40,,Aggregated,27.750000,0.000000,260,480,680,780,990,1200,1600,3100,3900,3900,3900,209,0,260,432.3011459330148,7.53249999252148,3885.2197999949567,7869.875598086124 +1779168558,40,,Aggregated,24.400000,0.000000,270,450,660,730,1000,1200,3600,3700,3900,3900,3900,253,0,270,496.4594596839536,7.53249999252148,3885.2197999949567,9135.97233201581 +1779168559,40,,Aggregated,29.000000,0.000000,290,490,680,810,970,1200,3600,3700,3900,3900,3900,289,0,290,506.8255252598392,7.53249999252148,3885.2197999949567,8648.692041522492 +1779168560,40,,Aggregated,29.857143,0.000000,310,470,670,750,960,1200,3700,3700,3900,3900,3900,327,0,310,510.67537859360624,7.53249999252148,3885.2197999949567,8540.80122324159 +1779168561,40,,Aggregated,31.625000,0.000000,320,480,650,710,960,1200,3600,3700,3900,3900,3900,356,0,310,527.416445786677,7.53249999252148,3885.2197999949567,8735.91573033708 diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u10_exceptions.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u10_failures.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u10_stats.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u10_stats.csv new file mode 100644 index 0000000..d71069b --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u10_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [cold],2,0,3300,7144.589100003941,3274.135899991961,11015.04230001592,1103.5,0.17827204092836194,0.0,11000,11000,11000,11000,11000,11000,11000,11000,11000,11000,11000 +POST,POST /prompts [seed],12,0,14,14.621783331676852,12.828399980207905,17.21489999908954,34.0,1.0696322455701717,0.0,15,15,16,16,17,17,17,17,17,17,17 +PUT,PUT /optimize/config,1,0,24.303499987581745,24.303499987581745,24.303499987581745,24.303499987581745,507.0,0.08913602046418097,0.0,24,24,24,24,24,24,24,24,24,24,24 +,Aggregated,15,0,15,965.9295399983724,12.828399980207905,11015.04230001592,208.13333333333333,1.3370403069627146,0.0,15,16,17,24,3300,11000,11000,11000,11000,11000,11000 diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u10_stats_history.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u10_stats_history.csv new file mode 100644 index 0000000..459b74b --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168666,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168667,10,,Aggregated,0.000000,0.000000,15,16,16,17,17,24,24,24,24,24,24,13,0,15,15.36653076674646,12.828399980207905,24.303499987581745,70.38461538461539 +1779168668,10,,Aggregated,0.000000,0.000000,15,16,16,17,17,24,24,24,24,24,24,13,0,15,15.36653076674646,12.828399980207905,24.303499987581745,70.38461538461539 +1779168669,10,,Aggregated,0.000000,0.000000,15,16,16,17,17,24,24,24,24,24,24,13,0,15,15.36653076674646,12.828399980207905,24.303499987581745,70.38461538461539 +1779168670,10,,Aggregated,6.500000,0.000000,15,16,17,17,24,3300,3300,3300,3300,3300,3300,14,0,15,248.13577142569036,12.828399980207905,3274.135899991961,155.28571428571428 +1779168671,10,,Aggregated,6.500000,0.000000,15,16,17,17,24,3300,3300,3300,3300,3300,3300,14,0,15,248.13577142569036,12.828399980207905,3274.135899991961,155.28571428571428 +1779168672,10,,Aggregated,6.500000,0.000000,15,16,17,17,24,3300,3300,3300,3300,3300,3300,14,0,15,248.13577142569036,12.828399980207905,3274.135899991961,155.28571428571428 +1779168673,10,,Aggregated,6.500000,0.000000,15,16,17,17,24,3300,3300,3300,3300,3300,3300,14,0,15,248.13577142569036,12.828399980207905,3274.135899991961,155.28571428571428 +1779168674,10,,Aggregated,6.500000,0.000000,15,16,17,17,24,3300,3300,3300,3300,3300,3300,14,0,15,248.13577142569036,12.828399980207905,3274.135899991961,155.28571428571428 +1779168675,10,,Aggregated,6.500000,0.000000,15,16,17,17,24,3300,3300,3300,3300,3300,3300,14,0,15,248.13577142569036,12.828399980207905,3274.135899991961,155.28571428571428 +1779168676,10,,Aggregated,6.500000,0.000000,15,16,17,17,24,3300,3300,3300,3300,3300,3300,14,0,15,248.13577142569036,12.828399980207905,3274.135899991961,155.28571428571428 diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u20_exceptions.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u20_failures.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u20_stats.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u20_stats.csv new file mode 100644 index 0000000..29610b2 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u20_stats.csv @@ -0,0 +1,4 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /prompts [seed],12,0,22,22.471658334931515,18.475300021236762,25.37389998906292,34.0,39.17798818705184,0.0,23,23,25,25,25,25,25,25,25,25,25 +PUT,PUT /optimize/config,1,0,34.63689997442998,34.63689997442998,34.63689997442998,34.63689997442998,507.0,3.2648323489209865,0.0,35,35,35,35,35,35,35,35,35,35,35 +,Aggregated,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539,42.44282053597283,0.0,23,24,25,25,25,35,35,35,35,35,35 diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u20_stats_history.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u20_stats_history.csv new file mode 100644 index 0000000..1b847cd --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168683,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168684,10,,Aggregated,0.000000,0.000000,23,24,25,25,25,35,35,35,35,35,35,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539 +1779168685,20,,Aggregated,0.000000,0.000000,23,24,25,25,25,35,35,35,35,35,35,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539 +1779168686,20,,Aggregated,0.000000,0.000000,23,24,25,25,25,35,35,35,35,35,35,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539 +1779168687,20,,Aggregated,0.000000,0.000000,23,24,25,25,25,35,35,35,35,35,35,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539 +1779168688,20,,Aggregated,0.000000,0.000000,23,24,25,25,25,35,35,35,35,35,35,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539 +1779168689,20,,Aggregated,0.000000,0.000000,23,24,25,25,25,35,35,35,35,35,35,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539 +1779168690,20,,Aggregated,0.000000,0.000000,23,24,25,25,25,35,35,35,35,35,35,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539 +1779168691,20,,Aggregated,0.000000,0.000000,23,24,25,25,25,35,35,35,35,35,35,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539 +1779168692,20,,Aggregated,0.000000,0.000000,23,24,25,25,25,35,35,35,35,35,35,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539 +1779168693,20,,Aggregated,0.000000,0.000000,23,24,25,25,25,35,35,35,35,35,35,13,0,23,23.407446153354474,18.475300021236762,34.63689997442998,70.38461538461539 diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u40_exceptions.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u40_failures.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u40_stats.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u40_stats.csv new file mode 100644 index 0000000..43da8ba --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u40_stats.csv @@ -0,0 +1,4 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /prompts [seed],12,0,22,22.124233333064087,21.187300008023158,24.445400020340458,34.0,39.212448473506946,0.0,22,22,23,23,23,24,24,24,24,24,24 +PUT,PUT /optimize/config,1,0,37.603999982820824,37.603999982820824,37.603999982820824,37.603999982820824,507.0,3.2677040394589123,0.0,38,38,38,38,38,38,38,38,38,38,38 +,Aggregated,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539,42.48015251296586,0.0,22,23,23,23,24,38,38,38,38,38,38 diff --git a/loadtests/results/db_compare/postgres_async/optimize_cold/u40_stats_history.csv b/loadtests/results/db_compare/postgres_async/optimize_cold/u40_stats_history.csv new file mode 100644 index 0000000..0e61581 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_cold/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168700,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168701,10,,Aggregated,0.000000,0.000000,22,23,23,23,24,38,38,38,38,38,38,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539 +1779168702,20,,Aggregated,0.000000,0.000000,22,23,23,23,24,38,38,38,38,38,38,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539 +1779168703,40,,Aggregated,0.000000,0.000000,22,23,23,23,24,38,38,38,38,38,38,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539 +1779168704,40,,Aggregated,0.000000,0.000000,22,23,23,23,24,38,38,38,38,38,38,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539 +1779168705,40,,Aggregated,0.000000,0.000000,22,23,23,23,24,38,38,38,38,38,38,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539 +1779168706,40,,Aggregated,0.000000,0.000000,22,23,23,23,24,38,38,38,38,38,38,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539 +1779168707,40,,Aggregated,0.000000,0.000000,22,23,23,23,24,38,38,38,38,38,38,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539 +1779168708,40,,Aggregated,0.000000,0.000000,22,23,23,23,24,38,38,38,38,38,38,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539 +1779168709,40,,Aggregated,0.000000,0.000000,22,23,23,23,24,38,38,38,38,38,38,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539 +1779168710,40,,Aggregated,0.000000,0.000000,22,23,23,23,24,38,38,38,38,38,38,13,0,22,23.314984613814605,21.187300008023158,37.603999982820824,70.38461538461539 diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u10_exceptions.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u10_failures.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u10_stats.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u10_stats.csv new file mode 100644 index 0000000..e50a9b1 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u10_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [hot],269,0,14,17.581401114762315,11.123499978566542,97.4757999938447,753.0,19.18909392932623,0.0,14,15,16,17,18,23,95,97,97,97,97 +POST,POST /prompts [seed],12,0,15,15.748158334948434,13.262599997688085,17.968500003917143,34.0,0.8560190600442928,0.0,16,16,17,17,18,18,18,18,18,18,18 +PUT,PUT /optimize/config,1,0,24.855800002114847,24.855800002114847,24.855800002114847,24.855800002114847,507.0,0.07133492167035774,0.0,25,25,25,25,25,25,25,25,25,25,25 +,Aggregated,282,0,14,17.5291865244417,11.123499978566542,97.4757999938447,721.531914893617,20.116447911040883,0.0,14,16,17,17,18,23,95,97,97,97,97 diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u10_stats_history.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u10_stats_history.csv new file mode 100644 index 0000000..a319db6 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168617,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168618,10,,Aggregated,0.000000,0.000000,17,25,94,95,97,97,97,97,97,97,97,30,0,16,41.50574333325494,12.110100011341274,97.4757999938447,457.2 +1779168619,10,,Aggregated,0.000000,0.000000,16,17,19,77,96,97,97,97,97,97,97,47,0,16,31.853585105211017,12.110100011341274,97.4757999938447,564.1914893617021 +1779168620,10,,Aggregated,14.000000,0.000000,16,17,19,20,95,96,97,97,97,97,97,66,0,16,27.63073030412973,12.110100011341274,97.4757999938447,618.5454545454545 +1779168621,10,,Aggregated,17.500000,0.000000,16,17,18,19,89,96,97,97,97,97,97,89,0,16,24.388634832248908,12.110100011341274,97.4757999938447,653.2921348314607 +1779168622,10,,Aggregated,17.666667,0.000000,15,17,18,18,22,95,97,97,97,97,97,112,0,15,22.30251607163934,11.123499978566542,97.4757999938447,673.7678571428571 +1779168623,10,,Aggregated,19.000000,0.000000,15,17,17,18,22,95,97,97,97,97,97,131,0,15,21.317769465159326,11.123499978566542,97.4757999938447,685.2595419847328 +1779168624,10,,Aggregated,19.000000,0.000000,15,16,17,18,21,94,97,97,97,97,97,150,0,15,20.4043719995146,11.123499978566542,97.4757999938447,693.84 +1779168625,10,,Aggregated,19.666667,0.000000,14,16,17,18,20,89,96,97,97,97,97,168,0,14,19.646477379865246,11.123499978566542,97.4757999938447,700.1785714285714 +1779168626,10,,Aggregated,19.571429,0.000000,14,16,17,17,20,77,96,97,97,97,97,186,0,14,19.17239139685833,11.123499978566542,97.4757999938447,705.2903225806451 +1779168627,10,,Aggregated,19.625000,0.000000,14,16,17,17,20,30,96,97,97,97,97,207,0,14,18.727881158617716,11.123499978566542,97.4757999938447,710.1304347826087 diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u20_exceptions.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u20_failures.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u20_stats.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u20_stats.csv new file mode 100644 index 0000000..13e38b5 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u20_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [hot],518,0,15,19.519349806737534,10.385100002167746,174.0740000095684,753.0,36.987391461391404,0.0,15,17,18,19,23,33,140,140,170,170,170 +POST,POST /prompts [seed],12,0,15,14.764774998184294,13.615100004244596,16.000399977201596,34.0,0.8568507674453607,0.0,15,15,15,15,16,16,16,16,16,16,16 +PUT,PUT /optimize/config,1,0,23.943899985169992,23.943899985169992,23.943899985169992,23.943899985169992,507.0,0.07140423062044672,0.0,24,24,24,24,24,24,24,24,24,24,24 +,Aggregated,531,0,15,19.42023427467688,10.385100002167746,174.0740000095684,736.2881355932203,37.915646459457214,0.0,15,17,18,19,23,31,140,140,170,170,170 diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u20_stats_history.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u20_stats_history.csv new file mode 100644 index 0000000..a2d5da0 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168633,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168634,10,,Aggregated,0.000000,0.000000,15,24,53,57,60,62,63,63,63,63,63,32,0,15,28.347403122097603,12.304899981245399,63.05280001834035,475.6875 +1779168635,20,,Aggregated,0.000000,0.000000,16,25,58,61,140,160,170,170,170,170,170,71,0,16,44.47758591505961,11.712500010617077,174.0740000095684,628.0140845070423 +1779168636,20,,Aggregated,1.000000,0.000000,16,19,24,27,140,140,170,170,170,170,170,113,0,16,33.86533274330751,10.914499987848103,174.0740000095684,674.4690265486726 +1779168637,20,,Aggregated,16.000000,0.000000,15,17,20,22,60,140,160,170,170,170,170,151,0,15,28.998333774584143,10.799000010592863,174.0740000095684,694.2317880794702 +1779168638,20,,Aggregated,24.333333,0.000000,15,17,19,22,57,140,160,170,170,170,170,185,0,15,26.402625404896114,10.799000010592863,174.0740000095684,705.0324324324324 +1779168639,20,,Aggregated,28.500000,0.000000,15,16,18,20,27,140,150,170,170,170,170,222,0,15,24.38268288227977,10.799000010592863,174.0740000095684,713.027027027027 +1779168640,20,,Aggregated,30.200000,0.000000,15,16,17,20,25,62,140,170,170,170,170,261,0,15,23.05450344768663,10.799000010592863,174.0740000095684,719.0 +1779168641,20,,Aggregated,31.000000,0.000000,15,16,17,19,24,61,140,170,170,170,170,300,0,15,22.05510166619206,10.480500000994653,174.0740000095684,723.42 +1779168642,20,,Aggregated,31.857143,0.000000,15,16,17,19,23,59,140,160,170,170,170,338,0,15,21.273081952319536,10.480500000994653,174.0740000095684,726.7455621301775 +1779168643,20,,Aggregated,32.750000,0.000000,15,16,17,19,23,57,140,160,170,170,170,373,0,15,20.807060857344638,10.480500000994653,174.0740000095684,729.2091152815013 diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u40_exceptions.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u40_failures.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u40_stats.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u40_stats.csv new file mode 100644 index 0000000..241d88d --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u40_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [hot],964,0,20,30.016994813331056,9.861200000159442,234.28259999491274,753.0,68.61946835160887,0.0,20,25,28,30,44,100,190,200,230,230,230 +POST,POST /prompts [seed],12,0,14,14.540891660241565,12.682699976721779,16.975499980617315,34.0,0.8541842533395296,0.0,14,14,16,16,16,17,17,17,17,17,17 +PUT,PUT /optimize/config,1,0,22.853000002214685,22.853000002214685,22.853000002214685,22.853000002214685,507.0,0.07118202111162747,0.0,23,23,23,23,23,23,23,23,23,23,23 +,Aggregated,977,0,20,29.81957697029299,9.861200000159442,234.28259999491274,743.9170931422723,69.54483462606004,0.0,20,24,28,30,44,100,190,200,230,230,230 diff --git a/loadtests/results/db_compare/postgres_async/optimize_hot/u40_stats_history.csv b/loadtests/results/db_compare/postgres_async/optimize_hot/u40_stats_history.csv new file mode 100644 index 0000000..d31d946 --- /dev/null +++ b/loadtests/results/db_compare/postgres_async/optimize_hot/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168650,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168651,10,,Aggregated,0.000000,0.000000,14,17,59,62,65,65,66,66,66,66,66,33,0,14,28.424336361543585,11.39950001379475,65.87839999701828,484.09090909090907 +1779168652,30,,Aggregated,0.000000,0.000000,18,44,63,65,200,200,230,230,230,230,230,72,0,17,53.69366805340784,11.244199995417148,234.28259999491274,629.75 +1779168653,40,,Aggregated,28.000000,0.000000,22,44,65,140,200,200,210,230,230,230,230,128,0,22,58.68708124853583,11.244199995417148,234.28259999491274,683.671875 +1779168654,40,,Aggregated,31.500000,0.000000,25,46,98,140,190,200,210,220,230,230,230,203,0,25,62.00556600884433,11.244199995417148,234.28259999491274,709.2857142857143 +1779168655,40,,Aggregated,38.333333,0.000000,23,29,41,65,180,200,200,220,230,230,230,286,0,23,49.85870489448885,10.895400017034262,234.28259999491274,721.972027972028 +1779168656,40,,Aggregated,47.500000,0.000000,21,27,31,39,150,190,200,210,230,230,230,361,0,21,43.50568088567416,10.693200019886717,234.28259999491274,728.4182825484764 +1779168657,40,,Aggregated,53.600000,0.000000,22,27,31,36,130,190,200,210,230,230,230,435,0,22,40.39303080437052,10.693200019886717,234.28259999491274,732.6 +1779168658,40,,Aggregated,57.333333,0.000000,20,25,29,32,98,180,200,200,230,230,230,506,0,20,37.10798043463188,10.693200019886717,234.28259999491274,735.4624505928854 +1779168659,40,,Aggregated,59.428571,0.000000,20,25,29,32,65,180,200,200,230,230,230,584,0,20,35.27459503441119,10.693200019886717,234.28259999491274,737.804794520548 +1779168660,40,,Aggregated,61.750000,0.000000,20,25,28,31,59,170,200,200,230,230,230,663,0,20,33.51850708884954,9.861200000159442,234.28259999491274,739.6153846153846 diff --git a/loadtests/results/db_compare/postgres_sync/benchmark_manifest.json b/loadtests/results/db_compare/postgres_sync/benchmark_manifest.json new file mode 100644 index 0000000..4c443a1 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/benchmark_manifest.json @@ -0,0 +1,118 @@ +{ + "scenarios": [ + "mixed", + "cache", + "optimize_hot", + "optimize_cold" + ], + "results": [ + { + "scenario": "mixed", + "users": 10, + "requests_per_s": 33.40169936083167, + "failure_rate": 0.0, + "p95_ms": 67.0, + "avg_ms": 40.87223991431404, + "passed": true + }, + { + "scenario": "mixed", + "users": 20, + "requests_per_s": 57.00252692771164, + "failure_rate": 0.0, + "p95_ms": 180.0, + "avg_ms": 57.75627556181178, + "passed": true + }, + { + "scenario": "mixed", + "users": 40, + "requests_per_s": 60.617113351173394, + "failure_rate": 0.0, + "p95_ms": 690.0, + "avg_ms": 264.264345828777, + "passed": false + }, + { + "scenario": "cache", + "users": 10, + "requests_per_s": 101.92505011069518, + "failure_rate": 0.0, + "p95_ms": 19.0, + "avg_ms": 10.845872206911288, + "passed": true + }, + { + "scenario": "cache", + "users": 20, + "requests_per_s": 186.39369632857841, + "failure_rate": 0.0, + "p95_ms": 28.0, + "avg_ms": 14.648941943761983, + "passed": true + }, + { + "scenario": "cache", + "users": 40, + "requests_per_s": 243.6223845218938, + "failure_rate": 0.0, + "p95_ms": 87.0, + "avg_ms": 51.00472133483273, + "passed": true + }, + { + "scenario": "optimize_hot", + "users": 10, + "requests_per_s": 19.757322343225336, + "failure_rate": 0.0, + "p95_ms": 20.0, + "avg_ms": 13.518698916443546, + "passed": true + }, + { + "scenario": "optimize_hot", + "users": 20, + "requests_per_s": 37.144558369695574, + "failure_rate": 0.0, + "p95_ms": 47.0, + "avg_ms": 15.613799424365613, + "passed": true + }, + { + "scenario": "optimize_hot", + "users": 40, + "requests_per_s": 70.68572057553776, + "failure_rate": 0.0, + "p95_ms": 43.0, + "avg_ms": 18.169772875068006, + "passed": true + }, + { + "scenario": "optimize_cold", + "users": 10, + "requests_per_s": 0.39290696393885094, + "failure_rate": 0.0, + "p95_ms": 13000.0, + "avg_ms": 7485.714840010041, + "passed": false + }, + { + "scenario": "optimize_cold", + "users": 20, + "requests_per_s": 0.0, + "failure_rate": 1.0, + "p95_ms": Infinity, + "avg_ms": Infinity, + "passed": false + }, + { + "scenario": "optimize_cold", + "users": 40, + "requests_per_s": 0.0, + "failure_rate": 1.0, + "p95_ms": Infinity, + "avg_ms": Infinity, + "passed": false + } + ] +} \ No newline at end of file diff --git a/loadtests/results/db_compare/postgres_sync/benchmark_summary.md b/loadtests/results/db_compare/postgres_sync/benchmark_summary.md new file mode 100644 index 0000000..2531451 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/benchmark_summary.md @@ -0,0 +1,16 @@ +# Benchmark Summary + +| Scenario | Users | RPS | P95 (ms) | Avg (ms) | Failure % | Pass | +| --- | ---: | ---: | ---: | ---: | ---: | :---: | +| mixed | 10 | 33.40 | 67.00 | 40.87 | 0.00 | yes | +| mixed | 20 | 57.00 | 180.00 | 57.76 | 0.00 | yes | +| mixed | 40 | 60.62 | 690.00 | 264.26 | 0.00 | no | +| cache | 10 | 101.93 | 19.00 | 10.85 | 0.00 | yes | +| cache | 20 | 186.39 | 28.00 | 14.65 | 0.00 | yes | +| cache | 40 | 243.62 | 87.00 | 51.00 | 0.00 | yes | +| optimize_hot | 10 | 19.76 | 20.00 | 13.52 | 0.00 | yes | +| optimize_hot | 20 | 37.14 | 47.00 | 15.61 | 0.00 | yes | +| optimize_hot | 40 | 70.69 | 43.00 | 18.17 | 0.00 | yes | +| optimize_cold | 10 | 0.39 | 13000.00 | 7485.71 | 0.00 | no | +| optimize_cold | 20 | 0.00 | inf | inf | 100.00 | no | +| optimize_cold | 40 | 0.00 | inf | inf | 100.00 | no | diff --git a/loadtests/results/db_compare/postgres_sync/cache/u10_exceptions.csv b/loadtests/results/db_compare/postgres_sync/cache/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/cache/u10_failures.csv b/loadtests/results/db_compare/postgres_sync/cache/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/cache/u10_stats.csv b/loadtests/results/db_compare/postgres_sync/cache/u10_stats.csv new file mode 100644 index 0000000..24f46ff --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u10_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts [hot],393,0,10,11.039386768221059,5.566500010900199,64.16049998369999,4161.0,27.972447411664252,0.0,10,11,12,13,15,19,23,46,64,64,64 +GET,GET /prompts/search [hot],298,0,10,10.42052617393459,5.2084000199101865,46.306800009915605,5115.0,21.210659869404445,0.0,10,11,12,13,15,19,20,22,46,46,46 +GET,GET /prompts/{project}/{name} [hot],472,0,9,10.573627543635666,4.9962000048253685,56.72349999076687,425.0,33.59540757838556,0.0,9,11,12,13,16,18,22,48,57,57,57 +GET,GET /prompts/{project}/{name}/versions [hot],199,0,9,9.933013065037173,5.139300017617643,22.597100003622472,272.0,14.164165483260016,0.0,9,11,12,12,14,17,19,20,23,23,23 +POST,POST /optimize [hot],57,0,14,16.709333333174552,8.339799998793751,60.79310001223348,941.0,4.057072525355884,0.0,14,16,20,20,22,38,53,61,61,61,61 +POST,POST /prompts [seed],12,0,12,11.976441669200236,10.577900015050545,14.31520000915043,34.0,0.8541205316538703,0.0,12,12,13,13,13,14,14,14,14,14,14 +PUT,PUT /optimize/config,1,0,23.92209999379702,23.92209999379702,23.92209999379702,23.92209999379702,507.0,0.07117671097115585,0.0,24,24,24,24,24,24,24,24,24,24,24 +,Aggregated,1432,0,10,10.845872206911288,4.9962000048253685,64.16049998369999,2422.3624301675977,101.92505011069518,0.0,10,11,12,13,16,19,22,27,62,64,64 diff --git a/loadtests/results/db_compare/postgres_sync/cache/u10_stats_history.csv b/loadtests/results/db_compare/postgres_sync/cache/u10_stats_history.csv new file mode 100644 index 0000000..486408a --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168221,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168222,10,,Aggregated,0.000000,0.000000,11,14,15,21,48,57,62,64,64,64,64,98,0,11,17.292902039599188,5.139300017617643,64.16049998369999,1649.6938775510205 +1779168223,10,,Aggregated,0.000000,0.000000,11,13,14,15,21,47,57,61,64,64,64,202,0,11,14.152462870265255,5.139300017617643,64.16049998369999,2025.8217821782177 +1779168224,10,,Aggregated,11.000000,0.000000,10,12,13,14,19,24,54,61,64,64,64,305,0,10,12.758765900870175,5.139300017617643,64.16049998369999,2279.075409836066 +1779168225,10,,Aggregated,56.000000,0.000000,10,12,13,14,18,22,49,57,64,64,64,405,0,10,12.152957283233487,5.139300017617643,64.16049998369999,2257.7925925925924 +1779168226,10,,Aggregated,71.000000,0.000000,10,11,13,14,17,20,47,54,64,64,64,512,0,10,11.622795312177914,5.139300017617643,64.16049998369999,2284.73046875 +1779168227,10,,Aggregated,79.750000,0.000000,10,11,12,13,16,20,46,54,64,64,64,609,0,10,11.323192939166077,5.139300017617643,64.16049998369999,2367.018062397373 +1779168228,10,,Aggregated,83.600000,0.000000,10,11,12,13,15,19,24,53,64,64,64,710,0,9,11.039658309686864,5.139300017617643,64.16049998369999,2403.443661971831 +1779168229,10,,Aggregated,87.166667,0.000000,10,11,12,13,15,20,24,49,64,64,64,808,0,10,11.025502351498794,5.139300017617643,64.16049998369999,2418.1188118811883 +1779168230,10,,Aggregated,88.714286,0.000000,10,11,12,13,16,20,24,48,64,64,64,913,0,10,11.050698795231662,5.095700005767867,64.16049998369999,2374.618838992333 +1779168231,10,,Aggregated,90.000000,0.000000,10,11,12,13,16,19,23,47,62,64,64,1014,0,10,10.903942702107106,4.9962000048253685,64.16049998369999,2398.842209072978 diff --git a/loadtests/results/db_compare/postgres_sync/cache/u20_exceptions.csv b/loadtests/results/db_compare/postgres_sync/cache/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/cache/u20_failures.csv b/loadtests/results/db_compare/postgres_sync/cache/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/cache/u20_stats.csv b/loadtests/results/db_compare/postgres_sync/cache/u20_stats.csv new file mode 100644 index 0000000..013cee2 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u20_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts [hot],734,0,13,14.617238147200114,4.938899975968525,64.74279999383725,4161.0,52.358581364399754,0.0,13,16,17,19,23,27,34,42,65,65,65 +GET,GET /prompts/search [hot],521,0,13,14.196251630356299,5.1629999943543226,66.98730000061914,5115.0,37.16460611832735,0.0,13,15,17,18,22,26,35,41,67,67,67 +GET,GET /prompts/{project}/{name} [hot],879,0,12,13.983361660580991,4.821799986530095,63.53959999978542,425.0,62.70189784646782,0.0,12,15,17,18,22,25,33,47,64,64,64 +GET,GET /prompts/{project}/{name}/versions [hot],359,0,14,14.848758773793906,5.094399995869026,71.00860000355169,272.0,25.60862494525819,0.0,14,17,18,19,23,27,40,55,71,71,71 +POST,POST /optimize [hot],107,0,20,22.10329345859387,8.263099996838719,70.58880000840873,941.0,7.632654231595059,0.0,20,24,26,29,35,44,56,56,71,71,71 +POST,POST /prompts [seed],12,0,12,11.844425006226325,10.343900008592755,15.106300008483231,34.0,0.85599860541253,0.0,12,12,12,12,12,15,15,15,15,15,15 +PUT,PUT /optimize/config,1,0,23.12060000258498,23.12060000258498,23.12060000258498,23.12060000258498,507.0,0.07133321711771083,0.0,23,23,23,23,23,23,23,23,23,23,23 +,Aggregated,2613,0,13,14.648941943761983,4.821799986530095,71.00860000355169,2407.9272866437045,186.39369632857841,0.0,13,16,18,19,23,28,37,48,67,71,71 diff --git a/loadtests/results/db_compare/postgres_sync/cache/u20_stats_history.csv b/loadtests/results/db_compare/postgres_sync/cache/u20_stats_history.csv new file mode 100644 index 0000000..b6cd80e --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168238,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168239,10,,Aggregated,0.000000,0.000000,10,11,12,18,31,38,41,44,44,44,44,95,0,10,13.654451579848109,5.731800018111244,44.31800000020303,2011.1052631578948 +1779168240,20,,Aggregated,0.000000,0.000000,12,17,20,22,26,38,45,49,56,56,56,282,0,12,15.676169859221476,4.904899979010224,55.63219997566193,2251.5921985815603 +1779168241,20,,Aggregated,72.000000,0.000000,14,18,20,23,30,40,56,63,67,67,67,478,0,14,16.812113389274433,4.904899979010224,66.98730000061914,2244.9476987447697 +1779168242,20,,Aggregated,119.500000,0.000000,13,17,20,22,26,35,49,62,67,67,67,670,0,13,16.037913730738115,4.904899979010224,66.98730000061914,2355.931343283582 +1779168243,20,,Aggregated,143.333333,0.000000,13,17,19,21,26,33,51,60,67,67,67,866,0,13,15.881598151729046,4.904899979010224,66.98730000061914,2348.2193995381062 +1779168244,20,,Aggregated,155.750000,0.000000,13,16,18,20,24,31,46,56,65,67,67,1055,0,13,15.204640662946442,4.904899979010224,66.98730000061914,2362.260663507109 +1779168245,20,,Aggregated,162.800000,0.000000,13,16,18,20,24,31,43,56,65,67,67,1253,0,13,15.018776695439866,4.904899979010224,66.98730000061914,2368.3830806065444 +1779168246,20,,Aggregated,168.333333,0.000000,13,16,18,19,24,30,41,55,65,67,67,1447,0,13,14.853475189504858,4.904899979010224,66.98730000061914,2323.5017277125085 +1779168247,20,,Aggregated,171.857143,0.000000,13,16,18,19,24,29,39,53,65,67,67,1642,0,13,14.814233982435855,4.904899979010224,66.98730000061914,2334.5505481120586 +1779168248,20,,Aggregated,174.750000,0.000000,13,16,18,19,23,28,37,49,65,67,67,1824,0,13,14.66668881543579,4.904899979010224,66.98730000061914,2361.1672149122805 diff --git a/loadtests/results/db_compare/postgres_sync/cache/u40_exceptions.csv b/loadtests/results/db_compare/postgres_sync/cache/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/cache/u40_failures.csv b/loadtests/results/db_compare/postgres_sync/cache/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/cache/u40_stats.csv b/loadtests/results/db_compare/postgres_sync/cache/u40_stats.csv new file mode 100644 index 0000000..17c2333 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u40_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts [hot],903,0,52,49.306912736370045,5.281599995214492,149.2558000027202,4161.0,64.38133252071118,0.0,52,57,61,65,71,83,92,110,150,150,150 +GET,GET /prompts/search [hot],660,0,53,51.5982209092313,5.686499993316829,129.93100000312552,5115.0,47.05612343706465,0.0,53,58,62,65,76,86,110,120,130,130,130 +GET,GET /prompts/{project}/{name} [hot],1175,0,51,48.95860587259101,5.002299993066117,134.5984000072349,425.0,83.77415914931964,0.0,51,57,61,64,71,80,88,110,130,130,130 +GET,GET /prompts/{project}/{name}/versions [hot],468,0,50,48.61712970103746,5.96199999563396,149.82839999720454,272.0,33.36706934628221,0.0,50,56,59,62,71,81,89,100,150,150,150 +POST,POST /optimize [hot],198,0,78,77.1057207059148,13.282200001413003,187.4878000235185,941.0,14.116837031119395,0.0,78,85,90,96,110,120,130,160,190,190,190 +POST,POST /prompts [seed],12,0,11,11.35986666001069,9.8260999948252,12.597399996593595,34.0,0.8555658806739027,0.0,12,12,12,12,12,13,13,13,13,13,13 +PUT,PUT /optimize/config,1,0,21.73510001739487,21.73510001739487,21.73510001739487,21.73510001739487,507.0,0.07129715672282523,0.0,22,22,22,22,22,22,22,22,22,22,22 +,Aggregated,3417,0,52,51.00472133483273,5.002299993066117,187.4878000235185,2325.779045946737,243.6223845218938,0.0,52,58,63,66,77,87,110,120,150,190,190 diff --git a/loadtests/results/db_compare/postgres_sync/cache/u40_stats_history.csv b/loadtests/results/db_compare/postgres_sync/cache/u40_stats_history.csv new file mode 100644 index 0000000..4faa056 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/cache/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168254,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168255,10,,Aggregated,0.000000,0.000000,11,12,13,14,34,39,45,46,46,46,46,94,0,11,13.406379787409877,5.902000004425645,45.9938999847509,2246.978723404255 +1779168256,30,,Aggregated,0.000000,0.000000,12,13,15,16,22,45,60,65,71,71,71,297,0,12,14.759792255430549,5.002299993066117,71.44030000199564,2393.7508417508416 +1779168257,40,,Aggregated,27.000000,0.000000,14,19,23,26,42,54,67,71,91,91,91,544,0,14,19.777961213549133,5.002299993066117,90.58060002280399,2321.3400735294117 +1779168258,40,,Aggregated,73.500000,0.000000,21,43,54,59,77,85,92,110,120,120,120,787,0,21,33.95204841171266,5.002299993066117,119.12110002595,2327.848792884371 +1779168259,40,,Aggregated,121.666667,0.000000,41,54,58,61,74,83,89,97,120,120,120,1061,0,41,39.31092686175863,5.002299993066117,119.12110002595,2272.005655042413 +1779168260,40,,Aggregated,152.250000,0.000000,45,54,58,61,72,81,89,96,120,120,120,1329,0,45,41.77305349903889,5.002299993066117,119.12110002595,2263.319036869827 +1779168261,40,,Aggregated,171.000000,0.000000,49,56,61,65,78,88,110,120,150,190,190,1572,0,49,46.19013282457667,5.002299993066117,187.4878000235185,2292.959923664122 +1779168262,40,,Aggregated,188.333333,0.000000,49,56,60,64,76,86,110,120,150,190,190,1847,0,49,46.985353221683816,5.002299993066117,187.4878000235185,2307.0129940443962 +1779168263,40,,Aggregated,198.428571,0.000000,50,57,62,65,77,86,110,120,150,190,190,2106,0,50,48.208354796049235,5.002299993066117,187.4878000235185,2309.9525166191834 +1779168264,40,,Aggregated,205.375000,0.000000,51,57,63,65,75,85,100,120,150,190,190,2373,0,51,48.96253940192696,5.002299993066117,187.4878000235185,2287.8782132321953 diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u10_exceptions.csv b/loadtests/results/db_compare/postgres_sync/mixed/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u10_failures.csv b/loadtests/results/db_compare/postgres_sync/mixed/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u10_stats.csv b/loadtests/results/db_compare/postgres_sync/mixed/u10_stats.csv new file mode 100644 index 0000000..66f3562 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u10_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts,293,0,11,20.904233788935557,5.818600009661168,110.93650001566857,9363.235494880546,21.001497666789014,0.0,11,17,27,36,55,63,76,110,110,110,110 +GET,GET /prompts/search,82,0,12,31.18209390249387,6.816200009780005,86.88519999850541,14382.682926829268,5.877552248043342,0.0,14,41,56,62,71,79,86,87,87,87,87 +POST,POST /optimize,4,0,11,1783.047599994461,10.568900004727766,7098.804699984612,941.0,0.2867098657582118,0.0,12,12,7100,7100,7100,7100,7100,7100,7100,7100,7100 +POST,POST /prompts,37,0,33,39.127505404836924,24.955000000772998,108.66590001387522,400.0,2.652066258263459,0.0,33,41,42,47,59,93,110,110,110,110,110 +POST,POST /prompts [seed],12,0,23,24.107475001073908,20.79290000256151,30.97600000910461,425.1666666666667,0.8601295972746354,0.0,24,25,25,25,27,31,31,31,31,31,31 +PUT,PUT /optimize/config,1,0,34.99559999909252,34.99559999909252,34.99559999909252,34.99559999909252,507.0,0.07167746643955294,0.0,35,35,35,35,35,35,35,35,35,35,35 +PUT,PUT /prompts/{project}/{name},37,0,40,39.47022161997125,25.377700017997995,60.84809999447316,254.0,2.652066258263459,0.0,40,40,43,45,51,59,61,61,61,61,61 +,Aggregated,466,0,14,40.87223991431404,5.818600009661168,7098.804699984612,8490.083690987125,33.40169936083167,0.0,15,31,38,41,59,67,85,110,7100,7100,7100 diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u10_stats_history.csv b/loadtests/results/db_compare/postgres_sync/mixed/u10_stats_history.csv new file mode 100644 index 0000000..9ec4aed --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168172,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168173,10,,Aggregated,0.000000,0.000000,27,35,59,86,110,110,110,110,110,110,110,39,0,27,41.44366666785059,6.262499984586611,110.93650001566857,4565.179487179487 +1779168174,10,,Aggregated,0.000000,0.000000,25,33,38,43,86,110,110,110,110,110,110,73,0,25,32.30357123299642,6.262499984586611,110.93650001566857,5957.328767123287 +1779168175,10,,Aggregated,14.000000,0.000000,27,36,42,49,61,93,110,110,110,110,110,107,0,27,32.991578505444124,6.262499984586611,110.93650001566857,6047.878504672897 +1779168176,10,,Aggregated,25.500000,0.000000,27,36,41,49,59,87,110,110,110,110,110,139,0,27,31.99913525194258,6.262499984586611,110.93650001566857,6652.884892086331 +1779168177,10,,Aggregated,28.000000,0.000000,25,35,41,47,58,78,110,110,110,110,110,171,0,25,30.439258480243037,6.262499984586611,110.93650001566857,7017.140350877193 +1779168178,10,,Aggregated,29.250000,0.000000,22,33,40,44,58,67,110,110,110,110,110,204,0,21,28.400831863252154,6.262499984586611,110.93650001566857,7348.441176470588 +1779168179,10,,Aggregated,31.000000,0.000000,22,35,41,47,61,75,110,110,110,110,110,235,0,22,29.139490213066814,6.262499984586611,110.93650001566857,7524.842553191489 +1779168180,10,,Aggregated,30.166667,0.000000,23,35,41,48,61,75,110,110,7100,7100,7100,272,0,22,55.051973161479346,6.262499984586611,7098.804699984612,7630.941176470588 +1779168181,10,,Aggregated,31.000000,0.000000,21,34,40,45,61,72,93,110,7100,7100,7100,305,0,21,50.948916721161545,6.262499984586611,7098.804699984612,7789.390163934426 +1779168182,10,,Aggregated,31.000000,0.000000,21,33,40,44,61,70,93,110,7100,7100,7100,334,0,21,48.65136856288498,5.818600009661168,7098.804699984612,7868.961077844311 diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u20_exceptions.csv b/loadtests/results/db_compare/postgres_sync/mixed/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u20_failures.csv b/loadtests/results/db_compare/postgres_sync/mixed/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u20_stats.csv b/loadtests/results/db_compare/postgres_sync/mixed/u20_stats.csv new file mode 100644 index 0000000..6b2ebdc --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u20_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts,478,0,28,37.76543954032703,5.971399979898706,179.41630000132136,9162.882845188284,33.974074652676016,0.0,29,46,57,66,86,100,120,140,180,180,180 +GET,GET /prompts/search,131,0,120,131.13877404448291,6.1138000164646655,494.2831000080332,39278.86259541985,9.3108865679928,0.0,120,180,210,220,290,330,350,380,490,490,490 +POST,POST /optimize,14,0,23,46.351171435422394,10.84529998479411,140.56249998975545,941.0,0.99505657978549,0.0,46,56,76,85,87,140,140,140,140,140,140 +POST,POST /prompts,83,0,44,55.541806026075754,24.128399993060157,174.25209999782965,400.0,5.899264008728262,0.0,44,59,71,79,100,110,140,170,170,170,170 +POST,POST /prompts [seed],12,0,11,11.336458332759017,10.302500013494864,12.675400008447468,34.0,0.8529056398161343,0.0,11,11,12,12,12,13,13,13,13,13,13 +PUT,PUT /optimize/config,1,0,21.527300006709993,21.527300006709993,21.527300006709993,21.527300006709993,507.0,0.07107546998467786,0.0,22,22,22,22,22,22,22,22,22,22,22 +PUT,PUT /prompts/{project}/{name},83,0,60,68.34966747240298,25.31600001384504,171.13380000228062,254.0,5.899264008728262,0.0,60,78,92,96,110,140,160,170,170,170,170 +,Aggregated,802,0,39,57.75627556181178,5.971399979898706,494.2831000080332,11962.294264339153,57.00252692771164,0.0,39,59,76,87,130,180,270,320,490,490,490 diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u20_stats_history.csv b/loadtests/results/db_compare/postgres_sync/mixed/u20_stats_history.csv new file mode 100644 index 0000000..978e066 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168188,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168189,10,,Aggregated,0.000000,0.000000,22,27,35,38,44,52,75,75,75,75,75,48,0,16,23.58785000372639,6.710599991492927,75.42649999959394,7246.041666666667 +1779168190,20,,Aggregated,0.000000,0.000000,32,41,56,66,85,93,130,130,150,150,150,108,0,32,38.79194074397674,6.282099988311529,148.23819999583066,8082.351851851852 +1779168191,20,,Aggregated,10.000000,0.000000,35,45,56,67,86,100,150,150,160,160,160,171,0,35,41.01317368630914,5.971399979898706,164.27149999071844,8905.888888888889 +1779168192,20,,Aggregated,30.000000,0.000000,30,41,52,59,84,98,130,150,160,160,160,228,0,30,37.38699912457633,5.971399979898706,164.27149999071844,9174.11403508772 +1779168193,20,,Aggregated,37.666667,0.000000,34,48,57,71,93,120,160,180,200,200,200,285,0,34,42.654256844023045,5.971399979898706,201.13289999426343,9884.035087719298 +1779168194,20,,Aggregated,43.500000,0.000000,34,49,59,71,94,120,160,180,200,200,200,347,0,34,43.198893949797416,5.971399979898706,201.13289999426343,9967.700288184438 +1779168195,20,,Aggregated,46.400000,0.000000,37,54,71,82,110,150,180,200,230,230,230,402,0,37,48.73383308629637,5.971399979898706,231.649399996968,10245.333333333334 +1779168196,20,,Aggregated,48.333333,0.000000,37,55,70,81,110,150,180,210,240,240,240,461,0,37,49.5374036891994,5.971399979898706,235.0027000065893,10516.12147505423 +1779168197,20,,Aggregated,49.857143,0.000000,37,55,71,82,110,150,200,230,280,280,280,524,0,36,50.54887271126757,5.971399979898706,276.27669999492355,10939.673664122138 +1779168198,20,,Aggregated,50.875000,0.000000,37,56,71,84,110,150,210,230,280,280,280,575,0,37,51.32270121855823,5.971399979898706,276.27669999492355,10861.184347826087 diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u40_exceptions.csv b/loadtests/results/db_compare/postgres_sync/mixed/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u40_failures.csv b/loadtests/results/db_compare/postgres_sync/mixed/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u40_stats.csv b/loadtests/results/db_compare/postgres_sync/mixed/u40_stats.csv new file mode 100644 index 0000000..095d5da --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u40_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts,510,0,190,206.4934670586737,11.12829998601228,537.7212000021245,9065.366666666667,36.32752974042119,0.0,190,230,270,310,400,440,490,510,540,540,540 +GET,GET /prompts/search,116,0,570,535.0290965552317,9.877600008621812,1276.8128000025172,73460.55172413793,8.262732254684035,0.0,570,700,760,800,1100,1100,1200,1200,1300,1300,1300 +POST,POST /optimize,31,0,270,263.5944838684443,42.908200004603714,522.9200999892782,941.0,2.208143964613837,0.0,270,310,380,380,410,510,520,520,520,520,520 +POST,POST /prompts,93,0,260,274.8058623654933,29.871800012188032,568.39790000231,400.0,6.6244318938415105,0.0,260,360,400,420,480,530,550,570,570,570,570 +POST,POST /prompts [seed],12,0,11,11.081116669326244,10.213600005954504,13.077499985229224,34.0,0.8547654056569691,0.0,11,11,11,11,11,13,13,13,13,13,13 +PUT,PUT /optimize/config,1,0,21.038300008513033,21.038300008513033,21.038300008513033,21.038300008513033,507.0,0.07123045047141409,0.0,21,21,21,21,21,21,21,21,21,21,21 +PUT,PUT /prompts/{project}/{name},88,0,230,268.5401022727919,30.524600006174296,579.8374000005424,254.0,6.26827964148444,0.0,230,310,350,390,530,560,580,580,580,580,580 +,Aggregated,851,0,210,264.264345828777,9.877600008621812,1276.8128000025172,15551.585193889541,60.617113351173394,0.0,210,280,360,400,510,690,930,1100,1300,1300,1300 diff --git a/loadtests/results/db_compare/postgres_sync/mixed/u40_stats_history.csv b/loadtests/results/db_compare/postgres_sync/mixed/u40_stats_history.csv new file mode 100644 index 0000000..fbcbb41 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/mixed/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168205,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168206,10,,Aggregated,0.000000,0.000000,31,50,61,63,76,200,210,210,210,210,210,45,0,31,44.574713334441185,9.877600008621812,208.66770000429824,18162.88888888889 +1779168207,20,,Aggregated,0.000000,0.000000,59,75,130,160,200,430,510,520,520,520,520,91,0,59,95.3634197802595,9.877600008621812,515.1751000084914,15959.054945054944 +1779168208,30,,Aggregated,39.000000,0.000000,76,120,170,190,250,380,510,520,530,530,530,146,0,75,117.93975411014861,9.877600008621812,525.399500009371,14382.59589041096 +1779168209,40,,Aggregated,41.500000,0.000000,110,190,240,330,450,520,710,740,890,890,890,221,0,110,181.02867511386575,9.877600008621812,891.7473000183236,13804.398190045249 +1779168210,40,,Aggregated,48.000000,0.000000,140,190,230,320,450,520,690,740,890,890,890,289,0,140,187.17652941255582,9.877600008621812,891.7473000183236,15731.775086505191 +1779168211,40,,Aggregated,52.250000,0.000000,160,220,280,340,440,520,700,740,890,890,890,370,0,160,201.51028756793224,9.877600008621812,891.7473000183236,15732.0 +1779168212,40,,Aggregated,57.000000,0.000000,160,220,300,350,450,540,710,740,890,890,890,418,0,160,207.18576148379054,9.877600008621812,891.7473000183236,15406.868421052632 +1779168213,40,,Aggregated,59.666667,0.000000,180,260,350,400,510,660,760,1000,1100,1100,1100,479,0,180,238.84508663937663,9.877600008621812,1053.5094999941066,15745.41127348643 +1779168214,40,,Aggregated,58.714286,0.000000,180,250,330,380,510,690,780,930,1100,1100,1100,554,0,180,240.22282346615958,9.877600008621812,1053.5094999941066,15937.21119133574 +1779168215,40,,Aggregated,57.625000,0.000000,180,240,300,370,500,680,780,930,1100,1100,1100,621,0,180,235.01182544336095,9.877600008621812,1053.5094999941066,16083.297906602254 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_exceptions.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_failures.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_stats.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_stats.csv new file mode 100644 index 0000000..898ffa1 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [cold],5,0,7400,7485.714840010041,3015.6077000137884,12547.972100001061,790.2,0.39290696393885094,0.0,7400,11000,11000,13000,13000,13000,13000,13000,13000,13000,13000 +POST,POST /prompts [seed],12,0,13,12.787200001184829,11.082199984230101,15.079400007380173,34.0,0.9429767134532423,0.0,13,13,13,13,15,15,15,15,15,15,15 +PUT,PUT /optimize/config,1,0,20.932799990987405,20.932799990987405,20.932799990987405,20.932799990987405,507.0,0.07858139278777018,0.0,21,21,21,21,21,21,21,21,21,21,21 +,Aggregated,18,0,13,2089.0529666697453,11.082199984230101,12547.972100001061,270.3333333333333,1.4144650701798633,0.0,13,15,3000,3800,11000,13000,13000,13000,13000,13000,13000 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_stats_history.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_stats_history.csv new file mode 100644 index 0000000..6b78bb1 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168320,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168321,10,,Aggregated,0.000000,0.000000,13,13,13,15,15,21,21,21,21,21,21,13,0,13,13.413784615785026,11.082199984230101,20.932799990987405,70.38461538461539 +1779168322,10,,Aggregated,0.000000,0.000000,13,13,13,15,15,21,21,21,21,21,21,13,0,13,13.413784615785026,11.082199984230101,20.932799990987405,70.38461538461539 +1779168323,10,,Aggregated,0.000000,0.000000,13,13,13,15,15,21,21,21,21,21,21,13,0,13,13.413784615785026,11.082199984230101,20.932799990987405,70.38461538461539 +1779168324,10,,Aggregated,6.500000,0.000000,13,13,15,21,3000,3800,3800,3800,3800,3800,3800,15,0,13,466.5987000024567,11.082199984230101,3808.993600017857,166.33333333333334 +1779168325,10,,Aggregated,6.500000,0.000000,13,13,15,21,3000,3800,3800,3800,3800,3800,3800,15,0,13,466.5987000024567,11.082199984230101,3808.993600017857,166.33333333333334 +1779168326,10,,Aggregated,6.500000,0.000000,13,13,15,21,3000,3800,3800,3800,3800,3800,3800,15,0,13,466.5987000024567,11.082199984230101,3808.993600017857,166.33333333333334 +1779168327,10,,Aggregated,6.500000,0.000000,13,13,15,21,3000,3800,3800,3800,3800,3800,3800,15,0,13,466.5987000024567,11.082199984230101,3808.993600017857,166.33333333333334 +1779168328,10,,Aggregated,3.000000,0.000000,13,15,21,21,3800,7400,7400,7400,7400,7400,7400,16,0,13,901.6352875023586,11.082199984230101,7427.184100000886,207.875 +1779168329,10,,Aggregated,3.000000,0.000000,13,15,21,21,3800,7400,7400,7400,7400,7400,7400,16,0,13,901.6352875023586,11.082199984230101,7427.184100000886,207.875 +1779168330,10,,Aggregated,3.000000,0.000000,13,15,21,21,3800,7400,7400,7400,7400,7400,7400,16,0,13,901.6352875023586,11.082199984230101,7427.184100000886,207.875 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_exceptions.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_failures.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_stats.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_stats.csv new file mode 100644 index 0000000..3a31fcf --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_stats.csv @@ -0,0 +1,4 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /prompts [seed],12,0,15,15.622775000034986,13.846599991666153,18.734300014330074,34.0,54.11708390185054,0.0,16,16,16,16,17,19,19,19,19,19,19 +PUT,PUT /optimize/config,1,0,31.68240000377409,31.68240000377409,31.68240000377409,31.68240000377409,507.0,4.509756991820879,0.0,32,32,32,32,32,32,32,32,32,32,32 +,Aggregated,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539,58.62684089367142,0.0,16,16,16,17,19,32,32,32,32,32,32 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_stats_history.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_stats_history.csv new file mode 100644 index 0000000..c28c793 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168337,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168338,10,,Aggregated,0.000000,0.000000,16,16,16,17,19,32,32,32,32,32,32,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539 +1779168339,20,,Aggregated,0.000000,0.000000,16,16,16,17,19,32,32,32,32,32,32,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539 +1779168340,20,,Aggregated,0.000000,0.000000,16,16,16,17,19,32,32,32,32,32,32,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539 +1779168341,20,,Aggregated,0.000000,0.000000,16,16,16,17,19,32,32,32,32,32,32,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539 +1779168342,20,,Aggregated,0.000000,0.000000,16,16,16,17,19,32,32,32,32,32,32,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539 +1779168343,20,,Aggregated,0.000000,0.000000,16,16,16,17,19,32,32,32,32,32,32,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539 +1779168344,20,,Aggregated,0.000000,0.000000,16,16,16,17,19,32,32,32,32,32,32,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539 +1779168345,20,,Aggregated,0.000000,0.000000,16,16,16,17,19,32,32,32,32,32,32,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539 +1779168346,20,,Aggregated,0.000000,0.000000,16,16,16,17,19,32,32,32,32,32,32,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539 +1779168347,20,,Aggregated,0.000000,0.000000,16,16,16,17,19,32,32,32,32,32,32,13,0,16,16.85813076955338,13.846599991666153,31.68240000377409,70.38461538461539 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_exceptions.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_failures.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_stats.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_stats.csv new file mode 100644 index 0000000..30abcb7 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_stats.csv @@ -0,0 +1,4 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /prompts [seed],12,0,16,16.483183341430657,14.865900011500344,18.15410002018325,34.0,52.36448986088959,0.0,17,17,18,18,18,18,18,18,18,18,18 +PUT,PUT /optimize/config,1,0,28.983700001845136,28.983700001845136,28.983700001845136,28.983700001845136,507.0,4.363707488407466,0.0,29,29,29,29,29,29,29,29,29,29,29 +,Aggregated,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539,56.72819734929706,0.0,17,17,18,18,18,29,29,29,29,29,29 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_stats_history.csv b/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_stats_history.csv new file mode 100644 index 0000000..a26739d --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_cold/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168353,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168354,10,,Aggregated,0.000000,0.000000,17,17,18,18,18,29,29,29,29,29,29,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539 +1779168355,20,,Aggregated,0.000000,0.000000,17,17,18,18,18,29,29,29,29,29,29,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539 +1779168356,30,,Aggregated,0.000000,0.000000,17,17,18,18,18,29,29,29,29,29,29,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539 +1779168357,40,,Aggregated,0.000000,0.000000,17,17,18,18,18,29,29,29,29,29,29,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539 +1779168358,40,,Aggregated,0.000000,0.000000,17,17,18,18,18,29,29,29,29,29,29,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539 +1779168359,40,,Aggregated,0.000000,0.000000,17,17,18,18,18,29,29,29,29,29,29,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539 +1779168360,40,,Aggregated,0.000000,0.000000,17,17,18,18,18,29,29,29,29,29,29,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539 +1779168361,40,,Aggregated,0.000000,0.000000,17,17,18,18,18,29,29,29,29,29,29,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539 +1779168362,40,,Aggregated,0.000000,0.000000,17,17,18,18,18,29,29,29,29,29,29,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539 +1779168363,40,,Aggregated,0.000000,0.000000,17,17,18,18,18,29,29,29,29,29,29,13,0,17,17.444761546077924,14.865900011500344,28.983700001845136,70.38461538461539 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_exceptions.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_failures.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_stats.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_stats.csv new file mode 100644 index 0000000..1c9a801 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [hot],277,0,12,13.518698916443546,8.390799979679286,54.88139999215491,941.0,19.757322343225336,0.0,12,13,13,14,16,20,49,53,55,55,55 +POST,POST /prompts [seed],12,0,13,13.168841668327028,10.564399999566376,18.067500001052395,34.0,0.8559128812949605,0.0,13,13,13,13,16,18,18,18,18,18,18 +PUT,PUT /optimize/config,1,0,24.38109999638982,24.38109999638982,24.38109999638982,24.38109999638982,507.0,0.0713260734412467,0.0,24,24,24,24,24,24,24,24,24,24,24 +,Aggregated,290,0,12,13.541678620245436,8.390799979679286,54.88139999215491,901.9724137931034,20.684561297961544,0.0,12,13,13,14,16,20,49,53,55,55,55 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_stats_history.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_stats_history.csv new file mode 100644 index 0000000..8d7daf5 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168271,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168272,10,,Aggregated,0.000000,0.000000,13,16,47,48,52,54,55,55,55,55,55,36,0,13,23.082874999090564,8.877199987182394,54.88139999215491,626.6111111111111 +1779168273,10,,Aggregated,0.000000,0.000000,13,14,17,24,49,53,54,55,55,55,55,53,0,13,19.771956603091105,8.877199987182394,54.88139999215491,727.4528301886793 +1779168274,10,,Aggregated,31.000000,0.000000,12,13,16,17,48,52,54,55,55,55,55,73,0,12,17.77359314959764,8.877199987182394,54.88139999215491,785.9589041095891 +1779168275,10,,Aggregated,24.000000,0.000000,12,13,15,16,47,52,54,55,55,55,55,90,0,12,16.640886665684068,8.877199987182394,54.88139999215491,815.2444444444444 +1779168276,10,,Aggregated,23.000000,0.000000,12,13,15,16,21,49,53,54,55,55,55,112,0,12,15.89324464189953,8.877199987182394,54.88139999215491,839.9464285714286 +1779168277,10,,Aggregated,21.500000,0.000000,12,13,14,15,18,49,53,54,55,55,55,132,0,12,15.368643938872442,8.877199987182394,54.88139999215491,855.2575757575758 +1779168278,10,,Aggregated,21.400000,0.000000,12,13,14,15,18,48,53,54,55,55,55,150,0,12,14.952253999266153,8.877199987182394,54.88139999215491,865.5466666666666 +1779168279,10,,Aggregated,21.166667,0.000000,12,13,14,14,17,47,52,54,55,55,55,170,0,12,14.665353528792313,8.783199999015778,54.88139999215491,874.4235294117647 +1779168280,10,,Aggregated,20.714286,0.000000,12,13,13,14,17,33,52,54,55,55,55,189,0,12,14.359303174334672,8.783199999015778,54.88139999215491,881.1164021164021 +1779168281,10,,Aggregated,20.500000,0.000000,12,13,13,14,16,24,52,53,55,55,55,208,0,12,14.143977884277074,8.783199999015778,54.88139999215491,886.5865384615385 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_exceptions.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_failures.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_stats.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_stats.csv new file mode 100644 index 0000000..fa9e90d --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [hot],521,0,12,15.613799424365613,8.534499997040257,123.71539999730885,941.0,37.144558369695574,0.0,12,13,15,15,18,47,64,84,120,120,120 +POST,POST /prompts [seed],12,0,11,11.221608338625325,10.09710002108477,12.714799988316372,34.0,0.8555368530448118,0.0,11,12,12,12,12,13,13,13,13,13,13 +PUT,PUT /optimize/config,1,0,22.143099980894476,22.143099980894476,22.143099980894476,22.143099980894476,507.0,0.07129473775373431,0.0,22,22,22,22,22,22,22,22,22,22,22 +,Aggregated,534,0,12,15.527325655690792,8.534499997040257,123.71539999730885,919.8052434456929,38.07138996049412,0.0,12,13,14,15,18,47,64,84,120,120,120 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_stats_history.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_stats_history.csv new file mode 100644 index 0000000..035cd1f --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168287,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168288,10,,Aggregated,0.000000,0.000000,12,18,46,48,53,54,55,55,55,55,55,34,0,12,23.14673235249596,10.085600020829588,54.52320000040345,608.1176470588235 +1779168289,20,,Aggregated,0.000000,0.000000,13,18,46,53,56,59,64,66,66,66,66,78,0,13,23.94863076803561,8.81689999368973,66.02769999881275,795.8974358974359 +1779168290,20,,Aggregated,23.000000,0.000000,12,14,17,22,54,57,64,64,66,66,66,115,0,12,19.955096521374323,8.81689999368973,66.02769999881275,842.5826086956522 +1779168291,20,,Aggregated,26.000000,0.000000,12,14,16,17,53,56,59,64,66,66,66,152,0,12,18.296442762795394,8.81689999368973,66.02769999881275,866.5394736842105 +1779168292,20,,Aggregated,30.666667,0.000000,12,15,16,18,54,64,110,120,120,120,120,194,0,12,20.854650515062044,8.534499997040257,123.71539999730885,882.659793814433 +1779168293,20,,Aggregated,32.250000,0.000000,12,14,15,17,53,59,84,110,120,120,120,229,0,12,19.480557205469662,8.534499997040257,123.71539999730885,891.5764192139738 +1779168294,20,,Aggregated,32.800000,0.000000,12,13,15,16,47,57,84,110,120,120,120,265,0,12,18.47415886817166,8.534499997040257,123.71539999730885,898.2905660377359 +1779168295,20,,Aggregated,33.833333,0.000000,12,14,15,16,37,56,79,110,120,120,120,304,0,12,17.760321381693327,8.534499997040257,123.71539999730885,903.7697368421053 +1779168296,20,,Aggregated,34.428571,0.000000,12,13,15,16,23,55,79,110,120,120,120,343,0,12,17.12616093335578,8.534499997040257,123.71539999730885,908.0029154518951 +1779168297,20,,Aggregated,34.375000,0.000000,12,13,15,16,22,54,68,110,120,120,120,380,0,12,16.75953763180204,8.534499997040257,123.71539999730885,911.2157894736843 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_exceptions.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_failures.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_stats.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_stats.csv new file mode 100644 index 0000000..61c9082 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [hot],988,0,14,18.169772875068006,8.60810000449419,81.89629999105819,941.0,70.68572057553776,0.0,14,18,21,23,30,43,57,72,82,82,82 +POST,POST /prompts [seed],12,0,12,12.285725000159195,10.80089999595657,17.558899999130517,34.0,0.8585310191360861,0.0,12,12,12,12,13,18,18,18,18,18,18 +PUT,PUT /optimize/config,1,0,21.314999990863726,21.314999990863726,21.314999990863726,21.314999990863726,507.0,0.07154425159467384,0.0,21,21,21,21,21,21,21,21,21,21,21 +,Aggregated,1001,0,14,18.102376923636328,8.60810000449419,81.89629999105819,929.6933066933067,71.61579584626851,0.0,14,18,20,23,29,42,57,70,81,82,82 diff --git a/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_stats_history.csv b/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_stats_history.csv new file mode 100644 index 0000000..0e47246 --- /dev/null +++ b/loadtests/results/db_compare/postgres_sync/optimize_hot/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779168303,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779168304,10,,Aggregated,0.000000,0.000000,12,18,72,77,79,81,82,82,82,82,82,35,0,12,30.847994284704328,10.237100010272115,81.89629999105819,617.6285714285714 +1779168305,20,,Aggregated,0.000000,0.000000,12,14,49,52,77,79,81,82,82,82,82,76,0,12,25.946009211015497,9.23480000346899,81.89629999105819,792.078947368421 +1779168306,30,,Aggregated,9.000000,0.000000,13,16,26,50,60,77,80,81,82,82,82,142,0,13,24.47440774584989,8.838899986585602,81.89629999105819,861.2957746478874 +1779168307,40,,Aggregated,24.000000,0.000000,15,21,29,39,55,68,79,80,82,82,82,225,0,15,24.43100577776527,8.838899986585602,81.89629999105819,890.6977777777778 +1779168308,40,,Aggregated,30.666667,0.000000,16,21,24,28,51,59,77,79,82,82,82,301,0,16,22.695467442346033,8.838899986585602,81.89629999105819,903.3986710963455 +1779168309,40,,Aggregated,39.250000,0.000000,15,20,23,28,49,58,77,79,82,82,82,374,0,15,21.459920321252536,8.838899986585602,81.89629999105819,910.7379679144385 +1779168310,40,,Aggregated,46.400000,0.000000,16,20,24,28,39,55,72,79,82,82,82,456,0,16,21.299076754395063,8.838899986585602,81.89629999105819,916.1798245614035 +1779168311,40,,Aggregated,52.000000,0.000000,16,20,23,26,37,52,70,78,82,82,82,529,0,16,20.51345633273761,8.770899992669001,81.89629999105819,919.6049149338374 +1779168312,40,,Aggregated,54.714286,0.000000,15,19,23,25,34,51,65,77,82,82,82,617,0,15,19.840541491226475,8.770899992669001,81.89629999105819,922.6564019448947 +1779168313,40,,Aggregated,57.750000,0.000000,15,19,22,24,32,50,64,77,82,82,82,697,0,15,19.31293945503321,8.60810000449419,81.89629999105819,924.7618364418938 diff --git a/loadtests/results/db_compare/sqlite_sync/benchmark_manifest.json b/loadtests/results/db_compare/sqlite_sync/benchmark_manifest.json new file mode 100644 index 0000000..4347589 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/benchmark_manifest.json @@ -0,0 +1,118 @@ +{ + "scenarios": [ + "mixed", + "cache", + "optimize_hot", + "optimize_cold" + ], + "results": [ + { + "scenario": "mixed", + "users": 10, + "requests_per_s": 34.28225537715882, + "failure_rate": 0.0, + "p95_ms": 140.0, + "avg_ms": 27.07073817405651, + "passed": true + }, + { + "scenario": "mixed", + "users": 20, + "requests_per_s": 43.01291650899201, + "failure_rate": 0.0, + "p95_ms": 860.0, + "avg_ms": 151.87236688699997, + "passed": false + }, + { + "scenario": "mixed", + "users": 40, + "requests_per_s": 42.82190476231978, + "failure_rate": 0.0, + "p95_ms": 810.0, + "avg_ms": 155.42963048689515, + "passed": false + }, + { + "scenario": "cache", + "users": 10, + "requests_per_s": 102.26103494334065, + "failure_rate": 0.0, + "p95_ms": 19.0, + "avg_ms": 9.175718592890522, + "passed": true + }, + { + "scenario": "cache", + "users": 20, + "requests_per_s": 189.02486623140436, + "failure_rate": 0.0, + "p95_ms": 30.0, + "avg_ms": 13.6776451054613, + "passed": true + }, + { + "scenario": "cache", + "users": 40, + "requests_per_s": 255.69884795747512, + "failure_rate": 0.0, + "p95_ms": 80.0, + "avg_ms": 45.505646179589974, + "passed": true + }, + { + "scenario": "optimize_hot", + "users": 10, + "requests_per_s": 19.496682059018685, + "failure_rate": 0.0, + "p95_ms": 19.0, + "avg_ms": 9.410179488151078, + "passed": true + }, + { + "scenario": "optimize_hot", + "users": 20, + "requests_per_s": 38.47929404147882, + "failure_rate": 0.0, + "p95_ms": 35.0, + "avg_ms": 11.60361902961384, + "passed": true + }, + { + "scenario": "optimize_hot", + "users": 40, + "requests_per_s": 72.00778418225637, + "failure_rate": 0.0, + "p95_ms": 39.0, + "avg_ms": 13.849460040169614, + "passed": true + }, + { + "scenario": "optimize_cold", + "users": 10, + "requests_per_s": 0.15956243041999693, + "failure_rate": 0.0, + "p95_ms": 12000.0, + "avg_ms": 9396.952249997412, + "passed": false + }, + { + "scenario": "optimize_cold", + "users": 20, + "requests_per_s": 0.0, + "failure_rate": 1.0, + "p95_ms": Infinity, + "avg_ms": Infinity, + "passed": false + }, + { + "scenario": "optimize_cold", + "users": 40, + "requests_per_s": 0.0, + "failure_rate": 1.0, + "p95_ms": Infinity, + "avg_ms": Infinity, + "passed": false + } + ] +} \ No newline at end of file diff --git a/loadtests/results/db_compare/sqlite_sync/benchmark_summary.md b/loadtests/results/db_compare/sqlite_sync/benchmark_summary.md new file mode 100644 index 0000000..7f8b5d0 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/benchmark_summary.md @@ -0,0 +1,16 @@ +# Benchmark Summary + +| Scenario | Users | RPS | P95 (ms) | Avg (ms) | Failure % | Pass | +| --- | ---: | ---: | ---: | ---: | ---: | :---: | +| mixed | 10 | 34.28 | 140.00 | 27.07 | 0.00 | yes | +| mixed | 20 | 43.01 | 860.00 | 151.87 | 0.00 | no | +| mixed | 40 | 42.82 | 810.00 | 155.43 | 0.00 | no | +| cache | 10 | 102.26 | 19.00 | 9.18 | 0.00 | yes | +| cache | 20 | 189.02 | 30.00 | 13.68 | 0.00 | yes | +| cache | 40 | 255.70 | 80.00 | 45.51 | 0.00 | yes | +| optimize_hot | 10 | 19.50 | 19.00 | 9.41 | 0.00 | yes | +| optimize_hot | 20 | 38.48 | 35.00 | 11.60 | 0.00 | yes | +| optimize_hot | 40 | 72.01 | 39.00 | 13.85 | 0.00 | yes | +| optimize_cold | 10 | 0.16 | 12000.00 | 9396.95 | 0.00 | no | +| optimize_cold | 20 | 0.00 | inf | inf | 100.00 | no | +| optimize_cold | 40 | 0.00 | inf | inf | 100.00 | no | diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u10_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/cache/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u10_failures.csv b/loadtests/results/db_compare/sqlite_sync/cache/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u10_stats.csv b/loadtests/results/db_compare/sqlite_sync/cache/u10_stats.csv new file mode 100644 index 0000000..d4cd6fb --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u10_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts [hot],395,0,8,9.334730126827225,3.7956999731250107,70.10040001478046,4131.0,28.128905851406376,0.0,8,9,10,11,14,19,32,67,70,70,70 +GET,GET /prompts/search [hot],308,0,8,9.20355389448028,3.786300017964095,72.36839999677613,5115.0,21.933425322109276,0.0,8,9,10,11,13,18,33,70,72,72,72 +GET,GET /prompts/{project}/{name} [hot],470,0,7,8.307415318918078,3.3100000000558794,63.333500002045184,425.0,33.46983734217974,0.0,7,8,10,11,13,16,21,44,63,63,63 +GET,GET /prompts/{project}/{name}/versions [hot],193,0,7,9.09798756426374,3.56620000093244,51.738299982389435,272.0,13.743997036256786,0.0,7,8,10,11,15,22,31,51,52,52,52 +POST,POST /optimize [hot],57,0,12,15.640212280630836,6.0477000079117715,59.36149999615736,830.0,4.0591079329877555,0.0,12,16,17,19,23,50,56,59,59,59,59 +POST,POST /prompts [seed],12,0,7,7.162500000655807,6.422599981306121,7.833900017431006,34.0,0.8545490385237381,0.0,7,7,7,7,8,8,8,8,8,8,8 +PUT,PUT /optimize/config,1,0,16.579999995883554,16.579999995883554,16.579999995883554,16.579999995883554,507.0,0.07121241987697817,0.0,17,17,17,17,17,17,17,17,17,17,17 +,Aggregated,1436,0,8,9.175718592890522,3.3100000000558794,72.36839999677613,2442.6434540389973,102.26103494334065,0.0,8,9,10,11,14,19,32,52,71,72,72 diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u10_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/cache/u10_stats_history.csv new file mode 100644 index 0000000..1316ee9 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167725,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167726,10,,Aggregated,0.000000,0.000000,8,9,9,12,43,69,71,72,72,72,72,99,0,8,13.487851515005465,3.9647999801672995,72.36839999677613,2282.4444444444443 +1779167727,10,,Aggregated,0.000000,0.000000,8,10,12,14,17,22,69,71,72,72,72,206,0,8,11.615141262260753,3.9647999801672995,72.36839999677613,2285.339805825243 +1779167728,10,,Aggregated,0.000000,0.000000,8,11,12,14,17,22,52,70,72,72,72,303,0,8,11.24791320157468,3.5189000191166997,72.36839999677613,2390.970297029703 +1779167729,10,,Aggregated,103.000000,0.000000,8,10,12,12,16,20,48,69,72,72,72,408,0,8,10.365770343495944,3.5189000191166997,72.36839999677613,2531.5612745098038 +1779167730,10,,Aggregated,101.000000,0.000000,8,10,11,12,16,21,27,67,72,72,72,504,0,8,10.118140872648274,3.5189000191166997,72.36839999677613,2501.3611111111113 +1779167731,10,,Aggregated,101.500000,0.000000,8,9,11,12,17,22,52,69,72,72,72,603,0,8,10.410380265013288,3.5189000191166997,72.36839999677613,2521.1641791044776 +1779167732,10,,Aggregated,101.500000,0.000000,8,9,11,12,16,22,50,67,72,72,72,701,0,8,10.079152924364058,3.4642000100575387,72.36839999677613,2459.7917261055636 +1779167733,10,,Aggregated,100.000000,0.000000,8,9,11,12,17,23,51,63,72,72,72,807,0,8,10.228490210511886,3.3100000000558794,72.36839999677613,2430.5675340768275 +1779167734,10,,Aggregated,100.142857,0.000000,8,9,11,12,16,23,49,62,72,72,72,910,0,7,9.927623845786522,3.3100000000558794,72.36839999677613,2455.3087912087913 +1779167735,10,,Aggregated,100.000000,0.000000,8,9,11,12,16,22,48,59,71,72,72,1017,0,8,9.736974335557415,3.3100000000558794,72.36839999677613,2447.2959685349065 diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u20_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/cache/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u20_failures.csv b/loadtests/results/db_compare/sqlite_sync/cache/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u20_stats.csv b/loadtests/results/db_compare/sqlite_sync/cache/u20_stats.csv new file mode 100644 index 0000000..a285dcf --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u20_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts [hot],710,0,11,13.312123943379307,3.4177000052295625,115.07120000896975,4131.0,50.5299905964974,0.0,11,14,16,17,23,29,46,59,120,120,120 +GET,GET /prompts/search [hot],535,0,11,13.877687475994385,3.4456999856047332,102.09890000987798,5115.0,38.07541544947339,0.0,11,14,16,18,23,30,57,65,100,100,100 +GET,GET /prompts/{project}/{name} [hot],898,0,11,13.2739191541302,3.186599991749972,107.91130000143312,425.0,63.909762754443186,0.0,11,13,16,18,23,29,54,67,110,110,110 +GET,GET /prompts/{project}/{name}/versions [hot],392,0,10,13.203553061597335,3.228000015951693,101.33440000936389,272.0,27.898248329333775,0.0,10,14,16,18,24,30,51,69,100,100,100 +POST,POST /optimize [hot],108,0,19,20.97054537093586,5.254599993349984,64.48549998458475,830.0,7.686252090734816,0.0,20,24,28,29,37,41,45,59,64,64,64 +POST,POST /prompts [seed],12,0,6,6.397250004132123,5.86409997777082,7.460799999535084,34.0,0.8540280100816462,0.0,7,7,7,7,7,7,7,7,7,7,7 +PUT,PUT /optimize/config,1,0,14.296499983174726,14.296499983174726,14.296499983174726,14.296499983174726,507.0,0.07116900084013718,0.0,14,14,14,14,14,14,14,14,14,14,14 +,Aggregated,2656,0,11,13.6776451054613,3.186599991749972,115.07120000896975,2352.546686746988,189.02486623140436,0.0,11,14,16,18,24,30,48,64,110,120,120 diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u20_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/cache/u20_stats_history.csv new file mode 100644 index 0000000..5f49b60 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167741,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167742,10,,Aggregated,0.000000,0.000000,8,9,11,12,23,38,42,42,45,45,45,113,0,8,11.050381418266046,3.6367999855428934,45.20059999777004,2340.0442477876104 +1779167743,20,,Aggregated,0.000000,0.000000,9,12,14,16,26,41,46,56,64,64,64,311,0,9,12.644976207075365,3.228000015951693,64.48549998458475,2199.0192926045015 +1779167744,20,,Aggregated,82.000000,0.000000,10,13,15,17,22,30,45,51,64,64,64,513,0,10,12.494783236776328,3.228000015951693,64.48549998458475,2328.6140350877195 +1779167745,20,,Aggregated,127.000000,0.000000,11,14,17,19,26,38,77,100,120,120,120,684,0,11,14.790694152545566,3.228000015951693,115.07120000896975,2372.8318713450294 +1779167746,20,,Aggregated,151.666667,0.000000,11,15,18,20,27,41,68,100,120,120,120,869,0,11,15.3105699659046,3.228000015951693,115.07120000896975,2385.5074798619103 +1779167747,20,,Aggregated,157.500000,0.000000,11,14,17,19,26,36,63,89,110,120,120,1074,0,11,14.487841620483502,3.228000015951693,115.07120000896975,2373.164804469274 +1779167748,20,,Aggregated,161.200000,0.000000,11,14,16,18,25,33,58,79,110,120,120,1274,0,11,14.036921271903541,3.186599991749972,115.07120000896975,2381.3241758241757 +1779167749,20,,Aggregated,168.333333,0.000000,11,13,16,18,24,30,52,74,110,120,120,1477,0,11,13.639573189321958,3.186599991749972,115.07120000896975,2402.054163845633 +1779167750,20,,Aggregated,172.857143,0.000000,10,13,16,17,23,30,46,69,110,120,120,1686,0,10,13.408585646742688,3.186599991749972,115.07120000896975,2399.867734282325 +1779167751,20,,Aggregated,176.000000,0.000000,11,14,16,18,24,29,45,64,110,120,120,1883,0,11,13.525179394721626,3.186599991749972,115.07120000896975,2379.0791290493894 diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u40_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/cache/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u40_failures.csv b/loadtests/results/db_compare/sqlite_sync/cache/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u40_stats.csv b/loadtests/results/db_compare/sqlite_sync/cache/u40_stats.csv new file mode 100644 index 0000000..651e9e3 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u40_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts [hot],921,0,45,43.970881650560806,4.009500000393018,115.76590000186116,4131.0,65.67167846314406,0.0,45,52,57,61,68,75,90,95,120,120,120 +GET,GET /prompts/search [hot],780,0,45,43.991208333113136,3.5883999953512102,116.96899999515153,5115.0,55.61770814468226,0.0,45,51,57,60,67,74,85,99,120,120,120 +GET,GET /prompts/{project}/{name} [hot],1198,0,45,44.06405517497941,3.245300002163276,122.84259998705238,425.0,85.42309532990943,0.0,45,53,57,60,67,75,90,100,120,120,120 +GET,GET /prompts/{project}/{name}/versions [hot],473,0,45,45.40839238940437,3.6751999869011343,124.2389999970328,272.0,33.72714865696758,0.0,45,53,57,61,67,85,110,120,120,120,120 +POST,POST /optimize [hot],201,0,72,69.71132437917474,5.507299996679649,147.80840001185425,830.0,14.332255560360428,0.0,72,78,84,86,99,110,130,140,150,150,150 +POST,POST /prompts [seed],12,0,6,6.540541665647955,5.521599989151582,8.763299993006513,34.0,0.8556570483797271,0.0,6,6,7,7,8,9,9,9,9,9,9 +PUT,PUT /optimize/config,1,0,15.55229999939911,15.55229999939911,15.55229999939911,15.55229999939911,507.0,0.07130475403164392,0.0,16,16,16,16,16,16,16,16,16,16,16 +,Aggregated,3586,0,46,45.505646179589974,3.245300002163276,147.80840001185425,2398.187953151143,255.69884795747512,0.0,46,54,59,62,70,80,96,110,130,150,150 diff --git a/loadtests/results/db_compare/sqlite_sync/cache/u40_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/cache/u40_stats_history.csv new file mode 100644 index 0000000..2c780c0 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/cache/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167757,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167758,10,,Aggregated,0.000000,0.000000,8,10,13,14,20,37,40,41,41,41,41,103,0,8,10.875565049326662,3.3451000053901225,41.1319000122603,2366.757281553398 +1779167759,20,,Aggregated,0.000000,0.000000,10,12,14,15,24,37,53,55,61,61,61,295,0,10,12.513278983237411,3.245300002163276,61.38580001424998,2393.0338983050847 +1779167760,30,,Aggregated,42.000000,0.000000,14,20,23,28,36,49,54,61,81,81,81,554,0,14,18.001890433334413,3.245300002163276,80.8649999962654,2496.3483754512636 +1779167761,40,,Aggregated,88.000000,0.000000,22,34,45,51,63,71,82,90,120,120,120,831,0,22,29.400164741515905,3.245300002163276,118.49719999008812,2461.347773766546 +1779167762,40,,Aggregated,130.666667,0.000000,31,44,52,57,67,76,86,93,120,120,120,1109,0,31,34.51562091955185,3.245300002163276,118.49719999008812,2479.9972948602344 +1779167763,40,,Aggregated,164.750000,0.000000,36,49,57,60,72,80,92,96,130,140,140,1372,0,36,38.423176238895564,3.245300002163276,141.1655999836512,2447.984693877551 +1779167764,40,,Aggregated,187.800000,0.000000,40,50,56,60,69,79,90,95,130,140,140,1655,0,40,39.83942302087395,3.245300002163276,141.1655999836512,2451.9728096676736 +1779167765,40,,Aggregated,202.666667,0.000000,41,50,56,59,68,78,89,95,130,140,140,1940,0,41,40.63786855650065,3.245300002163276,141.1655999836512,2424.1608247422682 +1779167766,40,,Aggregated,210.428571,0.000000,41,50,56,59,67,77,87,93,120,140,140,2221,0,41,41.182136334803204,3.245300002163276,141.1655999836512,2417.9243583971183 +1779167767,40,,Aggregated,219.875000,0.000000,43,51,56,59,67,76,86,93,120,140,140,2492,0,43,42.14549494361528,3.245300002163276,141.1655999836512,2418.053772070626 diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u10_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u10_failures.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u10_stats.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u10_stats.csv new file mode 100644 index 0000000..abca208 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u10_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts,286,0,9,15.641774125482685,3.8896999903954566,89.4432999775745,9543.902097902097,20.341753190596314,0.0,9,15,22,25,37,47,55,64,89,89,89 +GET,GET /prompts/search,72,0,48,88.42488194245055,6.2729000055696815,393.75180000206456,88153.68055555556,5.1210008032270435,0.0,57,140,150,160,180,200,380,390,390,390,390 +POST,POST /optimize,7,0,19,18.946242859653598,7.004000013694167,41.25420001219027,830.0,0.49787507809151815,0.0,19,20,22,22,41,41,41,41,41,41,41 +POST,POST /prompts,52,0,15,18.634305769340315,9.911699977237731,111.27769999438897,400.0,3.6985005801084205,0.0,16,18,21,22,27,32,37,110,110,110,110 +POST,POST /prompts [seed],12,0,7,7.063958333067906,5.552499991608784,8.842499984893948,34.0,0.853500133871174,0.0,7,7,8,8,8,9,9,9,9,9,9 +PUT,PUT /optimize/config,1,0,13.968199986265972,13.968199986265972,13.968199986265972,13.968199986265972,507.0,0.07112501115593117,0.0,14,14,14,14,14,14,14,14,14,14,14 +PUT,PUT /prompts/{project}/{name},52,0,18,19.377184618161454,9.988300007535145,43.805699999211356,254.0,3.6985005801084205,0.0,18,21,25,25,28,38,43,44,44,44,44 +,Aggregated,482,0,12,27.07073817405651,3.8896999903954566,393.75180000206456,18915.672199170123,34.28225537715882,0.0,12,18,24,28,49,140,180,200,390,390,390 diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u10_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u10_stats_history.csv new file mode 100644 index 0000000..1feca4d --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167645,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167646,10,,Aggregated,0.000000,0.000000,8,16,26,31,38,49,51,51,51,51,51,47,0,8,16.419304256139203,4.645200009690598,50.62589998124167,8290.0 +1779167647,10,,Aggregated,0.000000,0.000000,12,21,27,32,46,55,180,200,200,200,200,86,0,12,23.65241744311873,4.645200009690598,198.8087999925483,12389.93023255814 +1779167648,10,,Aggregated,47.000000,0.000000,13,22,26,31,46,57,170,180,200,200,200,121,0,13,23.829101654364198,4.367400018963963,198.8087999925483,13269.586776859504 +1779167649,10,,Aggregated,42.500000,0.000000,13,23,30,34,55,150,330,380,390,390,390,146,0,13,32.55867602862577,4.367400018963963,393.75180000206456,15239.90410958904 +1779167650,10,,Aggregated,42.500000,0.000000,12,17,26,32,49,150,200,380,390,390,390,173,0,12,28.789215029595486,4.367400018963963,393.75180000206456,16302.196531791908 +1779167651,10,,Aggregated,36.000000,0.000000,12,18,26,32,49,150,180,330,390,390,390,212,0,12,28.46771179281619,3.8896999903954566,393.75180000206456,15605.349056603774 +1779167652,10,,Aggregated,34.600000,0.000000,12,17,23,27,47,130,180,330,390,390,390,243,0,12,26.802676955023358,3.8896999903954566,393.75180000206456,15567.176954732511 +1779167653,10,,Aggregated,34.666667,0.000000,12,18,23,28,47,130,180,330,390,390,390,279,0,12,26.68198458846211,3.8896999903954566,393.75180000206456,15845.878136200718 +1779167654,10,,Aggregated,34.285714,0.000000,11,17,23,28,46,130,170,200,390,390,390,315,0,11,26.19511079401635,3.8896999903954566,393.75180000206456,16726.457142857143 +1779167655,10,,Aggregated,34.250000,0.000000,12,17,23,28,46,130,170,200,390,390,390,349,0,12,25.99411747856319,3.8896999903954566,393.75180000206456,16972.5329512894 diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u20_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u20_failures.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u20_stats.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u20_stats.csv new file mode 100644 index 0000000..44b65f1 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u20_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts,346,0,44,95.87640838078129,5.104199983179569,638.4205999784172,8805.260115606936,24.63984952336297,0.0,44,100,140,180,250,300,460,620,640,640,640 +GET,GET /prompts/search,68,0,690,642.5574014711482,7.4071999988518655,1540.8534000162035,112000.63235294117,4.842513779158041,0.0,690,920,1100,1100,1300,1400,1500,1500,1500,1500,1500 +POST,POST /optimize,12,0,34,75.84844166801001,17.613000003620982,476.705499982927,830.0,0.8545612551455365,0.0,42,56,66,66,68,480,480,480,480,480,480 +POST,POST /prompts,83,0,48,81.49579518224976,10.423400002764538,540.8481000049505,400.0,5.910715348089961,0.0,48,65,74,81,160,300,440,540,540,540,540 +POST,POST /prompts [seed],12,0,7,6.618483329172402,5.46559999929741,8.178599993698299,34.0,0.8545612551455365,0.0,7,7,7,7,8,8,8,8,8,8,8 +PUT,PUT /optimize/config,1,0,15.890999988187104,15.890999988187104,15.890999988187104,15.890999988187104,507.0,0.07121343792879471,0.0,16,16,16,16,16,16,16,16,16,16,16 +PUT,PUT /prompts/{project}/{name},82,0,53,86.51370609607888,10.980599996400997,491.173099988373,254.0,5.839501910161166,0.0,53,72,96,110,210,300,380,490,490,490,490 +,Aggregated,604,0,52,151.87236688699997,5.104199983179569,1540.8534000162035,17760.870860927153,43.01291650899201,0.0,52,97,160,190,390,860,1100,1300,1500,1500,1500 diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u20_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u20_stats_history.csv new file mode 100644 index 0000000..1f1433b --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167661,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167662,10,,Aggregated,0.000000,0.000000,11,19,29,47,62,74,78,160,160,160,160,54,0,10,24.147127778161558,5.46559999929741,157.57680000388063,18367.666666666668 +1779167663,20,,Aggregated,0.000000,0.000000,18,31,47,58,68,100,360,470,470,470,470,110,0,16,39.95443272711286,5.219500017119572,468.1014999805484,16725.945454545454 +1779167664,20,,Aggregated,41.000000,0.000000,25,48,62,77,160,270,470,510,600,600,600,162,0,25,64.78349876609475,5.104199983179569,598.976599983871,17479.475308641977 +1779167665,20,,Aggregated,46.000000,0.000000,27,52,67,87,180,290,510,570,860,860,860,201,0,27,73.26536119434246,5.104199983179569,862.2586999845225,16834.328358208953 +1779167666,20,,Aggregated,48.333333,0.000000,38,67,120,170,270,470,690,860,1100,1100,1100,233,0,38,105.06954248904654,5.104199983179569,1095.4589999746531,17333.931330472104 +1779167667,20,,Aggregated,49.000000,0.000000,44,75,130,170,300,500,800,1100,1200,1200,1200,284,0,43,117.46278626645069,5.104199983179569,1232.650799996918,17118.014084507042 +1779167668,20,,Aggregated,44.600000,0.000000,44,75,120,160,300,510,860,1100,1200,1200,1200,332,0,43,115.86166626415412,5.104199983179569,1232.650799996918,17499.572289156626 +1779167669,20,,Aggregated,44.666667,0.000000,47,80,120,160,290,570,860,1100,1200,1200,1200,376,0,47,117.65339521228638,5.104199983179569,1232.650799996918,17017.574468085106 +1779167670,20,,Aggregated,45.428571,0.000000,48,85,130,160,300,600,870,950,1200,1200,1200,419,0,48,122.26722744588855,5.104199983179569,1232.650799996918,16884.17661097852 +1779167671,20,,Aggregated,45.375000,0.000000,48,83,130,160,300,610,910,980,1200,1200,1200,457,0,48,124.15436323811362,5.104199983179569,1232.650799996918,16952.12910284464 diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u40_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u40_failures.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u40_stats.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u40_stats.csv new file mode 100644 index 0000000..0ac6cdf --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u40_stats.csv @@ -0,0 +1,9 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +GET,GET /prompts,83,0,73,96.84102048014616,4.962699982570484,785.807099979138,8946.0,21.67206155653989,0.0,73,99,120,140,180,230,750,790,790,790,790 +GET,GET /prompts/search,25,0,230,446.76574000157416,33.33169998950325,1158.7750999897253,135428.96,6.527729384499967,0.0,230,580,860,980,1100,1100,1200,1200,1200,1200,1200 +POST,POST /optimize,5,0,120,109.51395999873057,66.55089999549091,152.93040001415648,830.0,1.3055458768999935,0.0,120,140,140,150,150,150,150,150,150,150,150 +POST,POST /prompts,21,0,98,178.42530475796335,13.365399994654581,808.7272000266239,400.0,5.483292682979973,0.0,98,120,200,200,440,760,810,810,810,810,810 +POST,POST /prompts [seed],12,0,7,7.167775002017152,5.915100016864017,8.650500007206574,34.0,3.133310104559984,0.0,7,8,8,8,8,9,9,9,9,9,9 +PUT,PUT /optimize/config,1,0,17.801000009058043,17.801000009058043,17.801000009058043,17.801000009058043,507.0,0.26110917537999867,0.0,18,18,18,18,18,18,18,18,18,18,18 +PUT,PUT /prompts/{project}/{name},17,0,61,110.89386470677495,19.043500011321157,408.41679999721237,254.0,4.4388559814599775,0.0,61,78,81,84,400,410,410,410,410,410,410 +,Aggregated,164,0,70,155.42963048689515,4.962699982570484,1158.7750999897253,25280.64024390244,42.82190476231978,0.0,73,110,150,180,410,810,1000,1100,1200,1200,1200 diff --git a/loadtests/results/db_compare/sqlite_sync/mixed/u40_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/mixed/u40_stats_history.csv new file mode 100644 index 0000000..6919b62 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/mixed/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167690,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167691,10,,Aggregated,0.000000,0.000000,19,37,58,66,98,170,230,230,230,230,230,44,0,19,40.64976590995105,4.962699982570484,225.8293000049889,33360.90909090909 +1779167692,20,,Aggregated,0.000000,0.000000,39,65,78,92,120,210,550,580,580,580,580,100,0,38,65.28722699906211,4.962699982570484,578.1772999907844,24370.51 +1779167693,30,,Aggregated,37.000000,0.000000,66,98,120,150,230,440,820,930,980,980,980,153,0,66,111.61238431292921,4.962699982570484,984.491600014735,23249.065359477125 +1779167694,40,,Aggregated,44.000000,0.000000,73,110,150,180,410,810,1000,1100,1200,1200,1200,164,0,70,155.42963048689515,4.962699982570484,1158.7750999897253,25280.64024390244 +1779167695,40,,Aggregated,44.000000,0.000000,73,110,150,180,410,810,1000,1100,1200,1200,1200,164,0,70,155.42963048689515,4.962699982570484,1158.7750999897253,25280.64024390244 +1779167696,40,,Aggregated,44.000000,0.000000,73,110,150,180,410,810,1000,1100,1200,1200,1200,164,0,70,155.42963048689515,4.962699982570484,1158.7750999897253,25280.64024390244 +1779167697,40,,Aggregated,44.000000,0.000000,73,110,150,180,410,810,1000,1100,1200,1200,1200,164,0,70,155.42963048689515,4.962699982570484,1158.7750999897253,25280.64024390244 +1779167698,40,,Aggregated,44.000000,0.000000,73,110,150,180,410,810,1000,1100,1200,1200,1200,164,0,70,155.42963048689515,4.962699982570484,1158.7750999897253,25280.64024390244 +1779167699,40,,Aggregated,44.000000,0.000000,73,110,150,180,410,810,1000,1100,1200,1200,1200,164,0,70,155.42963048689515,4.962699982570484,1158.7750999897253,25280.64024390244 +1779167700,0,,Aggregated,44.000000,0.000000,73,110,150,180,410,810,1000,1100,1200,1200,1200,164,0,70,155.42963048689515,4.962699982570484,1158.7750999897253,25280.64024390244 diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_failures.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_stats.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_stats.csv new file mode 100644 index 0000000..b420765 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [cold],2,0,6400,9396.952249997412,6354.704099998344,12439.20039999648,696.5,0.15956243041999693,0.0,12000,12000,12000,12000,12000,12000,12000,12000,12000,12000,12000 +POST,POST /prompts [seed],12,0,6,6.372283331681198,5.963900010101497,7.10290001006797,34.0,0.9573745825199815,0.0,6,6,7,7,7,7,7,7,7,7,7 +PUT,PUT /optimize/config,1,0,14.091799996094778,14.091799996094778,14.091799996094778,14.091799996094778,507.0,0.07978121520999847,0.0,14,14,14,14,14,14,14,14,14,14,14 +,Aggregated,15,0,6,1258.9642466647394,5.963900010101497,12439.20039999648,153.86666666666667,1.1967182281499769,0.0,6,7,7,14,6400,12000,12000,12000,12000,12000,12000 diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_stats_history.csv new file mode 100644 index 0000000..55afc10 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167823,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167824,10,,Aggregated,0.000000,0.000000,6,7,7,7,7,14,14,14,14,14,14,13,0,6,6.966092305866858,5.963900010101497,14.091799996094778,70.38461538461539 +1779167825,10,,Aggregated,0.000000,0.000000,6,7,7,7,7,14,14,14,14,14,14,13,0,6,6.966092305866858,5.963900010101497,14.091799996094778,70.38461538461539 +1779167826,10,,Aggregated,0.000000,0.000000,6,7,7,7,7,14,14,14,14,14,14,13,0,6,6.966092305866858,5.963900010101497,14.091799996094778,70.38461538461539 +1779167827,10,,Aggregated,0.000000,0.000000,6,7,7,7,7,14,14,14,14,14,14,13,0,6,6.966092305866858,5.963900010101497,14.091799996094778,70.38461538461539 +1779167828,10,,Aggregated,0.000000,0.000000,6,7,7,7,7,14,14,14,14,14,14,13,0,6,6.966092305866858,5.963900010101497,14.091799996094778,70.38461538461539 +1779167829,10,,Aggregated,0.000000,0.000000,6,7,7,7,7,14,14,14,14,14,14,13,0,6,6.966092305866858,5.963900010101497,14.091799996094778,70.38461538461539 +1779167830,10,,Aggregated,3.250000,0.000000,6,7,7,7,14,6400,6400,6400,6400,6400,6400,14,0,6,460.3759499981866,5.963900010101497,6354.704099998344,111.0 +1779167831,10,,Aggregated,3.250000,0.000000,6,7,7,7,14,6400,6400,6400,6400,6400,6400,14,0,6,460.3759499981866,5.963900010101497,6354.704099998344,111.0 +1779167832,10,,Aggregated,3.250000,0.000000,6,7,7,7,14,6400,6400,6400,6400,6400,6400,14,0,6,460.3759499981866,5.963900010101497,6354.704099998344,111.0 +1779167833,10,,Aggregated,3.250000,0.000000,6,7,7,7,14,6400,6400,6400,6400,6400,6400,14,0,6,460.3759499981866,5.963900010101497,6354.704099998344,111.0 diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_failures.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_stats.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_stats.csv new file mode 100644 index 0000000..26d25ca --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_stats.csv @@ -0,0 +1,2 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +,Aggregated,0,0,0,0.0,0,0,0,0.0,0.0,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_stats_history.csv new file mode 100644 index 0000000..ad372d6 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167839,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167840,10,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167841,20,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167842,20,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167843,20,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167844,20,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167845,20,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167846,20,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167847,20,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167848,20,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167849,20,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_failures.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_stats.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_stats.csv new file mode 100644 index 0000000..26d25ca --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_stats.csv @@ -0,0 +1,2 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +,Aggregated,0,0,0,0.0,0,0,0,0.0,0.0,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_stats_history.csv new file mode 100644 index 0000000..ff6f559 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_cold/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167856,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167857,10,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167858,20,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167859,40,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167860,40,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167861,40,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167862,40,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167863,40,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167864,40,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167865,40,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167866,40,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_failures.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_stats.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_stats.csv new file mode 100644 index 0000000..30e7e2e --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [hot],273,0,7,9.410179488151078,5.031200009398162,54.21160001424141,830.0,19.496682059018685,0.0,7,8,9,10,11,19,51,54,54,54,54 +POST,POST /prompts [seed],12,0,6,6.4353666578729944,5.8351999905426055,7.424699986586347,34.0,0.8569970135832389,0.0,6,7,7,7,7,7,7,7,7,7,7 +PUT,PUT /optimize/config,1,0,14.48129999334924,14.48129999334924,14.48129999334924,14.48129999334924,507.0,0.07141641779860324,0.0,14,14,14,14,14,14,14,14,14,14,14 +,Aggregated,286,0,7,9.303093357178565,5.031200009398162,54.21160001424141,795.472027972028,20.425095490400526,0.0,7,8,8,9,11,19,51,54,54,54,54 diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_stats_history.csv new file mode 100644 index 0000000..68dc6e9 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u10_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167774,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167775,10,,Aggregated,0.000000,0.000000,7,9,43,47,52,54,54,54,54,54,54,34,0,7,18.903417645453693,5.74120000237599,54.21160001424141,539.5588235294117 +1779167776,10,,Aggregated,0.000000,0.000000,7,8,9,14,51,54,54,54,54,54,54,52,0,7,14.967853845938897,5.74120000237599,54.21160001424141,640.0961538461538 +1779167777,10,,Aggregated,30.000000,0.000000,7,8,9,9,44,52,54,54,54,54,54,72,0,7,12.807041666544844,5.74120000237599,54.21160001424141,692.8472222222222 +1779167778,10,,Aggregated,24.000000,0.000000,7,8,9,9,43,52,54,54,54,54,54,89,0,7,12.003922472012052,5.74120000237599,54.21160001424141,719.0449438202247 +1779167779,10,,Aggregated,22.666667,0.000000,7,8,9,11,14,51,54,54,54,54,54,110,0,7,11.369771820186129,5.550500005483627,54.21160001424141,740.2272727272727 +1779167780,10,,Aggregated,21.750000,0.000000,7,8,9,9,13,47,54,54,54,54,54,128,0,7,10.759218751672961,5.550500005483627,54.21160001424141,752.8515625 +1779167781,10,,Aggregated,21.000000,0.000000,7,8,9,9,13,44,54,54,54,54,54,149,0,7,10.311971813707531,5.550500005483627,54.21160001424141,763.7248322147651 +1779167782,10,,Aggregated,21.000000,0.000000,7,8,9,9,13,43,52,54,54,54,54,165,0,7,10.178932728718571,5.550500005483627,54.21160001424141,770.1515151515151 +1779167783,10,,Aggregated,20.857143,0.000000,7,8,9,9,12,21,52,54,54,54,54,184,0,7,9.924279348405948,5.550500005483627,54.21160001424141,776.3315217391304 +1779167784,10,,Aggregated,20.500000,0.000000,7,8,9,10,12,20,52,54,54,54,54,208,0,7,9.754617307967587,5.031200009398162,54.21160001424141,782.5240384615385 diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_failures.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_stats.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_stats.csv new file mode 100644 index 0000000..0fa8e0b --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [hot],536,0,7,11.60361902961384,5.207100010011345,128.24120000004768,830.0,38.47929404147882,0.0,7,9,11,12,16,35,56,60,130,130,130 +POST,POST /prompts [seed],12,0,7,6.9896250060992315,6.191799999214709,8.617500017862767,34.0,0.861476732271914,0.0,7,7,7,7,7,9,9,9,9,9,9 +PUT,PUT /optimize/config,1,0,15.02800002344884,15.02800002344884,15.02800002344884,15.02800002344884,507.0,0.07178972768932616,0.0,15,15,15,15,15,15,15,15,15,15,15 +,Aggregated,549,0,7,11.50900418938007,5.207100010011345,128.24120000004768,812.0127504553734,39.41256050144006,0.0,7,9,11,12,16,35,56,60,130,130,130 diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_stats_history.csv new file mode 100644 index 0000000..38be518 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u20_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167790,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167791,10,,Aggregated,0.000000,0.000000,10,16,43,48,54,55,56,56,56,56,56,34,0,9,20.8679558846223,6.191799999214709,55.55550000281073,539.5588235294117 +1779167792,20,,Aggregated,0.000000,0.000000,8,14,43,47,55,56,59,60,60,60,60,76,0,8,19.620822368826914,5.263899976853281,59.74090000381693,700.0657894736842 +1779167793,20,,Aggregated,26.000000,0.000000,8,10,13,16,49,56,58,59,60,60,60,114,0,8,15.888716666826472,5.263899976853281,59.74090000381693,743.3771929824561 +1779167794,20,,Aggregated,30.000000,0.000000,7,10,12,13,47,55,56,59,60,60,60,153,0,7,13.909397386181857,5.2204999956302345,59.74090000381693,765.4575163398692 +1779167795,20,,Aggregated,33.333333,0.000000,8,10,12,13,43,52,56,59,60,60,60,188,0,8,13.810028723457163,5.2204999956302345,59.74090000381693,777.4734042553191 +1779167796,20,,Aggregated,33.750000,0.000000,7,10,11,12,30,49,56,58,60,60,60,232,0,7,12.785263793093376,5.2204999956302345,59.74090000381693,787.4353448275862 +1779167797,20,,Aggregated,35.200000,0.000000,7,9,11,12,26,48,56,58,60,60,60,270,0,7,12.053011481529447,5.2204999956302345,59.74090000381693,793.425925925926 +1779167798,20,,Aggregated,35.166667,0.000000,7,10,11,12,16,47,55,56,60,60,60,312,0,7,11.698754807674469,5.2204999956302345,59.74090000381693,798.349358974359 +1779167799,20,,Aggregated,36.571429,0.000000,7,10,11,12,16,46,55,56,60,60,60,352,0,7,11.515046875005778,5.2204999956302345,59.74090000381693,801.9460227272727 +1779167800,20,,Aggregated,37.000000,0.000000,7,10,11,12,15,37,55,56,60,60,60,392,0,7,11.115546938823655,5.2204999956302345,59.74090000381693,804.8086734693877 diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_exceptions.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_exceptions.csv new file mode 100644 index 0000000..d5d6c07 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_exceptions.csv @@ -0,0 +1 @@ +Count,Message,Traceback,Nodes diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_failures.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_failures.csv new file mode 100644 index 0000000..30c1aa8 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_failures.csv @@ -0,0 +1 @@ +Method,Name,Error,Occurrences,First Seen,Last Seen diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_stats.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_stats.csv new file mode 100644 index 0000000..9161db7 --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_stats.csv @@ -0,0 +1,5 @@ +Type,Name,Request Count,Failure Count,Median Response Time,Average Response Time,Min Response Time,Max Response Time,Average Content Size,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100% +POST,POST /optimize [hot],1011,0,10,13.849460040169614,4.978500015567988,90.1192000019364,830.0,72.00778418225637,0.0,10,13,16,17,23,39,57,66,89,90,90 +POST,POST /prompts [seed],12,0,6,6.5170166684159385,5.666600016411394,7.641800009878352,34.0,0.854691800382865,0.0,7,7,7,7,7,8,8,8,8,8,8 +PUT,PUT /optimize/config,1,0,16.47849997971207,16.47849997971207,16.47849997971207,16.47849997971207,507.0,0.07122431669857208,0.0,16,16,16,16,16,16,16,16,16,16,16 +,Aggregated,1024,0,10,13.766100391222835,4.978500015567988,90.1192000019364,820.3564453125,72.93370029933781,0.0,10,13,16,17,23,39,57,66,89,90,90 diff --git a/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_stats_history.csv b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_stats_history.csv new file mode 100644 index 0000000..a34d0bf --- /dev/null +++ b/loadtests/results/db_compare/sqlite_sync/optimize_hot/u40_stats_history.csv @@ -0,0 +1,12 @@ +Timestamp,User Count,Type,Name,Requests/s,Failures/s,50%,66%,75%,80%,90%,95%,98%,99%,99.9%,99.99%,100%,Total Request Count,Total Failure Count,Total Median Response Time,Total Average Response Time,Total Min Response Time,Total Max Response Time,Total Average Content Size +1779167806,0,,Aggregated,0.000000,0.000000,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,0,0,0,0.0,0,0,0 +1779167807,10,,Aggregated,0.000000,0.000000,7,10,41,44,51,52,53,53,53,53,53,39,0,7,17.94943846284579,5.666600016411394,53.08599999989383,576.7948717948718 +1779167808,20,,Aggregated,0.000000,0.000000,7,10,16,43,51,52,57,57,57,57,57,84,0,7,17.4479488106001,5.589500011410564,57.245600008172914,712.4404761904761 +1779167809,40,,Aggregated,25.000000,0.000000,9,15,20,39,51,56,64,66,66,66,66,155,0,9,18.452671614678337,5.589500011410564,65.71789999725297,766.2903225806451 +1779167810,40,,Aggregated,27.500000,0.000000,10,14,18,39,54,66,78,88,90,90,90,238,0,9,19.46333151319487,5.579799995757639,90.1192000019364,788.5084033613446 +1779167811,40,,Aggregated,37.000000,0.000000,10,15,19,26,51,62,75,81,90,90,90,320,0,10,18.440402187752625,5.455999984405935,90.1192000019364,799.140625 +1779167812,40,,Aggregated,47.250000,0.000000,10,14,18,22,47,57,71,81,90,90,90,394,0,10,17.1071406093965,4.978500015567988,90.1192000019364,804.9365482233502 +1779167813,40,,Aggregated,52.600000,0.000000,10,12,16,19,41,54,70,78,90,90,90,479,0,10,15.771661378469489,4.978500015567988,90.1192000019364,809.384133611691 +1779167814,40,,Aggregated,57.500000,0.000000,10,12,15,17,32,52,66,78,90,90,90,556,0,10,14.995808274277465,4.978500015567988,90.1192000019364,812.2392086330935 +1779167815,40,,Aggregated,60.285714,0.000000,10,12,15,17,27,51,65,75,90,90,90,634,0,10,14.65959574191571,4.978500015567988,90.1192000019364,814.4242902208202 +1779167816,40,,Aggregated,63.375000,0.000000,10,12,15,17,25,50,64,71,90,90,90,707,0,10,14.357309335853596,4.978500015567988,90.1192000019364,816.032531824611 diff --git a/main.py b/main.py index b1ece32..459120c 100644 --- a/main.py +++ b/main.py @@ -1,39 +1,60 @@ -import datetime -import logging -import os -import sys -import tomllib -from contextlib import asynccontextmanager from collections.abc import Iterator -from pathlib import Path -from time import perf_counter from typing import Literal -from fastapi import Depends, FastAPI, HTTPException, Query, Request, Response -from fastapi.responses import JSONResponse +from fastapi import Depends, FastAPI, Query, Response from fastapi.responses import FileResponse from fastapi.staticfiles import StaticFiles from loguru import logger from sqlalchemy.orm import Session -from starlette.middleware.base import BaseHTTPMiddleware import auth as auth_service import crud -from database import SQLALCHEMY_DATABASE_URL, SessionLocal, init_database -from models import Prompt, User -from cache.shared_cache import ( - PROMPT_CACHE_PREFIX, - PROMPT_CACHE_TTL_SECONDS, - build_prompt_collection_cache_key, - build_prompt_response_cache_key, - cache_get_or_set, - clear_shared_cache, - delete_shared_cache_entry, +from app_core.lifecycle import create_lifespan, resolve_app_version, run_startup_bootstrap +from app_core.logging_config import configure_logging +from database import ( + SQLALCHEMY_DATABASE_URL, + SessionLocal, + StartupSessionLocal, + close_db_session, + init_database, ) -from optimizer.service import ( - build_optimizer_config, - list_available_models, - optimize_prompt_with_active_backend, +from middleware import ExceptionLoggingMiddleware, RequestLoggingMiddleware +from models import User +from optimizer.service import list_available_models, optimize_prompt_with_active_backend +from routes import ( + bootstrap_admin_route, + change_own_password_route, + create_project_route, + create_prompt_route, + create_user_route, + delete_project_route, + delete_prompt_route, + delete_user_route, + get_app_version_route, + get_auth_status_route, + get_me_route, + get_optimize_config_route, + get_project_route, + get_prompt_route, + get_prompt_version_route, + get_provider_models_route, + get_user_route, + list_projects_route, + list_prompts_route, + list_roles_route, + list_users_route, + list_versions_route, + login_route, + optimize_prompt_route, + refresh_auth_route, + search_prompts_route, + serve_ui_route, + update_optimize_config_route, + update_project_route, + update_prompt_route, + update_prompt_tags_route, + update_user_projects_route, + update_user_route, ) from schemas import ( AuthResponse, @@ -41,19 +62,19 @@ ChangePasswordRequest, OptimizeConfigOut, OptimizeConfigUpdate, + ProjectAccessUpdate, ProjectCreate, ProjectOut, - RoleOut, + ProjectUpdate, PromptCreate, PromptData, PromptOptimizeResponse, PromptOut, - PromptVersionOut, PromptTagsUpdate, PromptUpdate, - ProjectAccessUpdate, - ProjectUpdate, + PromptVersionOut, RefreshTokenRequest, + RoleOut, UserBootstrap, UserCreate, UserLogin, @@ -61,190 +82,36 @@ UserUpdate, ) +APP_VERSION = resolve_app_version() -def _resolve_app_version() -> str: - pyproject_path = Path(__file__).resolve().parent / "pyproject.toml" - if pyproject_path.exists(): - try: - with pyproject_path.open("rb") as fp: - pyproject = tomllib.load(fp) - version = pyproject.get("project", {}).get("version") - if isinstance(version, str) and version.strip(): - return version.strip() - except Exception: - pass - - return "0.0.0" - - -APP_VERSION = _resolve_app_version() - - -def _redact_database_url(url: str) -> str: - if "://" not in url: - return url - scheme, rest = url.split("://", 1) - if "@" not in rest: - return url - _, host_part = rest.split("@", 1) - return f"{scheme}://***@{host_part}" +__all__ = [ + "SessionLocal", + "StartupSessionLocal", + "app", + "crud", + "get_db", + "init_database", + "list_available_models", + "optimize_prompt_with_active_backend", +] def _run_startup_bootstrap() -> None: - startup_started = perf_counter() - logger.info("startup.begin") - - migrate_started = perf_counter() - init_database() - logger.info("startup.migrations.done duration_ms={:.2f}", (perf_counter() - migrate_started) * 1000) - - bootstrap_started = perf_counter() - db = SessionLocal() - try: - auth_service.maybe_bootstrap_admin(db) - finally: - db.close() - - logger.info("startup.bootstrap.done duration_ms={:.2f}", (perf_counter() - bootstrap_started) * 1000) - logger.info("startup.ready total_duration_ms={:.2f}", (perf_counter() - startup_started) * 1000) - logger.info( - "startup.health status=ready pid={} db_url={}", - os.getpid(), - _redact_database_url(SQLALCHEMY_DATABASE_URL), + run_startup_bootstrap( + SQLALCHEMY_DATABASE_URL, + init_database, + StartupSessionLocal, + auth_service.maybe_bootstrap_admin, + close_db_session, ) -@asynccontextmanager -async def lifespan(app: FastAPI): - _run_startup_bootstrap() - try: - yield - finally: - logger.info("shutdown.begin pid={}", os.getpid()) +lifespan = create_lifespan(_run_startup_bootstrap) app = FastAPI(title="Prompt Man", version=APP_VERSION, lifespan=lifespan) app.mount("/ui", StaticFiles(directory="ui"), name="ui") -CONSOLE_LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO").strip().upper() or "INFO" -SHOW_CONSOLE_SOURCE = os.getenv("SHOW_CONSOLE_SOURCE", "0").strip().lower() in {"1", "true", "yes", "on"} - - -def _console_log_format(record: dict) -> str: - def _escape_markup(value: object) -> str: - return str(value).replace("<", "\\<").replace(">", "\\>") - - message = _escape_markup(record["message"]) - source = f"{_escape_markup(record['name'])}:{_escape_markup(record['function'])}:{record['line']}" - badge = "APP " - message_color = "" - - if message.startswith("request.start"): - badge = "HTTP IN " - message_color = "" - elif message.startswith("request.end"): - badge = "HTTP OUT" - message_color = "" - elif message.startswith("request.error") or message.startswith("request.exception"): - badge = "HTTP ERR" - message_color = "" - elif message.startswith("optimize.gradient"): - badge = "GP " - message_color = "" - elif message.startswith("optimize.provider"): - badge = "LLM " - message_color = "" - elif message.startswith("optimize.config"): - badge = "CFG " - message_color = "" - elif message.startswith("logging.configured"): - badge = "BOOT " - message_color = "" - - source_part = f" {source} " if SHOW_CONSOLE_SOURCE else "" - - return ( - f"{record['time'].astimezone(datetime.timezone.utc):YYYY-MM-DD HH:mm:ss.SSS} UTC " - f"{badge} " - f"{record['level'].name:<8} " - f"{source_part}" - f"{message_color}{message}\n{{exception}}" - ) - - -def configure_logging() -> None: - os.makedirs("logs", exist_ok=True) - logger.remove() - logger.add( - sys.stderr, - level=CONSOLE_LOG_LEVEL, - enqueue=True, - backtrace=False, - diagnose=False, - colorize=True, - format=_console_log_format, - ) - logger.add( - "logs/app.log", - level="DEBUG", - rotation="10 MB", - retention="10 days", - enqueue=True, - backtrace=True, - diagnose=False, - format="{time:YYYY-MM-DD HH:mm:ss.SSS!UTC} UTC | {level} | {name}:{function}:{line} | {message}", - ) - - # Uvicorn access logs have their own formatter and make console output inconsistent - # with the application logs emitted through Loguru below. - uvicorn_access_logger = logging.getLogger("uvicorn.access") - uvicorn_access_logger.handlers.clear() - uvicorn_access_logger.propagate = False - - -class ExceptionLoggingMiddleware(BaseHTTPMiddleware): - async def dispatch(self, request: Request, call_next) -> Response: # type: ignore[no-untyped-def] - started_at = perf_counter() - try: - return await call_next(request) - except Exception: - duration_ms = (perf_counter() - started_at) * 1000 - client = request.client.host if request.client else "unknown" - logger.exception( - "request.exception method={} path={} query={} client={} duration_ms={:.2f}", - request.method, - request.url.path, - request.url.query, - client, - duration_ms, - ) - return JSONResponse(status_code=500, content={"detail": "Internal Server Error"}) - - -class RequestLoggingMiddleware(BaseHTTPMiddleware): - async def dispatch(self, request: Request, call_next) -> Response: # type: ignore[no-untyped-def] - start = perf_counter() - client = request.client.host if request.client else "unknown" - logger.info( - "request.start method={} path={} query={} client={}", - request.method, - request.url.path, - request.url.query, - client, - ) - - response = await call_next(request) - - duration_ms = (perf_counter() - start) * 1000 - logger.info( - "request.end method={} path={} status={} duration_ms={:.2f}", - request.method, - request.url.path, - response.status_code, - duration_ms, - ) - return response - configure_logging() logger.info("logging.configured sinks=console+file") @@ -261,122 +128,42 @@ def get_db() -> Iterator[Session]: try: yield db finally: - db.close() - - -def to_user_out(user: User) -> UserOut: - return UserOut(**auth_service.user_to_dict(user)) - - -def to_project_out(project) -> ProjectOut: # type: ignore[no-untyped-def] - return ProjectOut(id=project.id, name=project.name) - - -def get_personal_config(db: Session, current_user: User) -> dict: - config = auth_service.get_or_create_personal_config(db, current_user) - return auth_service.serialize_optimizer_config(config) - - -def allowed_projects(current_user: User) -> list[str] | None: - return auth_service.allowed_projects_for_user(current_user) - - -def normalize_utc_datetime(value: datetime.datetime | None) -> datetime.datetime | None: - if value is None: - return None - if value.tzinfo is None: - return value.replace(tzinfo=datetime.timezone.utc) - return value.astimezone(datetime.timezone.utc) - - -def to_prompt_out(db: Session, prompt: Prompt) -> PromptOut: - latest = crud.get_latest_version(db, prompt.id) - if not latest: - raise ValueError(f"No version found for prompt {prompt.id}") - return PromptOut( - name=prompt.name, - project=prompt.project, - created_at=normalize_utc_datetime(prompt.created_at), - updated_at=normalize_utc_datetime(prompt.updated_at), - created_by_username=crud.resolve_audit_username(db, prompt.created_by_ref), - updated_by_username=crud.resolve_audit_username(db, prompt.updated_by_ref), - tags=[tag.name for tag in prompt.tags], - latest_version=latest.version, - role=latest.role, - task=latest.task, - context=latest.context, - constraints=latest.constraints, - output_format=latest.output_format, - examples=latest.examples, - ) - - -def _invalidate_prompt_cache(project: str, name: str) -> None: - delete_shared_cache_entry(build_prompt_response_cache_key(project, name)) - clear_shared_cache(PROMPT_CACHE_PREFIX) - - -def to_prompt_version_out(db: Session, version) -> PromptVersionOut: # type: ignore[no-untyped-def] - return PromptVersionOut( - version=version.version, - created_at=normalize_utc_datetime(version.created_at), - created_by_username=crud.resolve_audit_username(db, version.created_by_ref), - role=version.role, - task=version.task, - context=version.context, - constraints=version.constraints, - output_format=version.output_format, - examples=version.examples, - ) + close_db_session(db) @app.get("/", include_in_schema=False) def serve_ui() -> FileResponse: - return FileResponse("ui/html/index.html") + return serve_ui_route() @app.post("/auth/bootstrap-admin", response_model=AuthResponse) -def bootstrap_admin(data: UserBootstrap, db: Session = Depends(get_db)) -> AuthResponse: - if crud.list_users(db): - raise HTTPException(409, "Users already exist") - user = auth_service.create_user_record( - db, - username=data.username, - password=data.password, - role="admin", - is_active=True, - projects=[], - ) - return AuthResponse(**auth_service.build_auth_response(user)) +def bootstrap_admin(data: UserBootstrap, db=Depends(get_db)) -> AuthResponse: # type: ignore[no-untyped-def] + return bootstrap_admin_route(data, db) @app.post("/auth/login", response_model=AuthResponse) -def login(data: UserLogin, db: Session = Depends(get_db)) -> AuthResponse: - user = auth_service.authenticate_user(db, data.username, data.password) - if not user: - raise HTTPException(401, "Invalid credentials") - return AuthResponse(**auth_service.build_auth_response(user)) +def login(data: UserLogin, db=Depends(get_db)) -> AuthResponse: # type: ignore[no-untyped-def] + return login_route(data, db) @app.post("/auth/refresh", response_model=AuthResponse) -def refresh_auth(data: RefreshTokenRequest, db: Session = Depends(get_db)) -> AuthResponse: - return AuthResponse(**auth_service.refresh_session(db, data.refresh_token)) +def refresh_auth(data: RefreshTokenRequest, db=Depends(get_db)) -> AuthResponse: # type: ignore[no-untyped-def] + return refresh_auth_route(data, db) @app.get("/auth/status", response_model=AuthStatus) -def get_auth_status(db: Session = Depends(get_db)) -> AuthStatus: - has_users = bool(crud.list_users(db)) - return AuthStatus(bootstrap_required=not has_users, has_users=has_users) +def get_auth_status(db=Depends(get_db)) -> AuthStatus: # type: ignore[no-untyped-def] + return get_auth_status_route(db) @app.get("/version") -def get_version() -> dict[str, str]: - return {"name": "prompt-man", "version": APP_VERSION} +def get_app_version() -> dict[str, str]: + return get_app_version_route(APP_VERSION) @app.get("/auth/me", response_model=UserOut) def get_me(current_user: User = Depends(auth_service.get_current_user)) -> UserOut: - return to_user_out(current_user) + return get_me_route(current_user) @app.post("/auth/me/password", status_code=204) @@ -385,125 +172,67 @@ def change_own_password( db: Session = Depends(get_db), current_user: User = Depends(auth_service.get_current_user), ) -> None: - auth_service.change_own_password( - db, - current_user, - current_password=data.current_password, - new_password=data.new_password, - ) + change_own_password_route(data, db, current_user) @app.get("/roles", response_model=list[RoleOut]) -def list_roles(db: Session = Depends(get_db), _: User = Depends(auth_service.require_admin)) -> list[RoleOut]: - return [RoleOut(**item) for item in auth_service.list_roles_out(db)] +def list_roles(db=Depends(get_db), _: User = Depends(auth_service.require_admin)) -> list[RoleOut]: # type: ignore[no-untyped-def] + return list_roles_route(db) @app.get("/users", response_model=list[UserOut]) -def list_users(db: Session = Depends(get_db), _: User = Depends(auth_service.require_admin)) -> list[UserOut]: - return [to_user_out(user) for user in crud.list_users(db)] +def list_users(db=Depends(get_db), _: User = Depends(auth_service.require_admin)) -> list[UserOut]: # type: ignore[no-untyped-def] + return list_users_route(db) @app.post("/users", response_model=UserOut) -def create_user(data: UserCreate, db: Session = Depends(get_db), _: User = Depends(auth_service.require_admin)) -> UserOut: - user = auth_service.create_user_record( - db, - username=data.username, - password=data.password, - role=data.role, - is_active=data.is_active, - projects=data.projects, - ) - return to_user_out(user) +def create_user(data: UserCreate, db=Depends(get_db), _: User = Depends(auth_service.require_admin)) -> UserOut: # type: ignore[no-untyped-def] + return create_user_route(data, db) @app.get("/users/{user_id}", response_model=UserOut) -def get_user(user_id: int, db: Session = Depends(get_db), _: User = Depends(auth_service.require_admin)) -> UserOut: - user = crud.get_user_by_id(db, user_id) - if not user: - raise HTTPException(404, "User not found") - return to_user_out(user) +def get_user(user_id: int, db=Depends(get_db), _: User = Depends(auth_service.require_admin)) -> UserOut: # type: ignore[no-untyped-def] + return get_user_route(user_id, db) @app.put("/users/{user_id}", response_model=UserOut) -def update_user(user_id: int, data: UserUpdate, db: Session = Depends(get_db), current_admin: User = Depends(auth_service.require_admin)) -> UserOut: - user = crud.get_user_by_id(db, user_id) - if not user: - raise HTTPException(404, "User not found") - if current_admin.id == user.id and data.role is not None and data.role != "admin": - raise HTTPException(400, "Admin cannot remove own admin role") - updated = auth_service.update_user_record( - db, - user, - username=data.username, - password=data.password, - role=data.role, - is_active=data.is_active, - projects=data.projects, - ) - return to_user_out(updated) +def update_user(user_id: int, data: UserUpdate, db=Depends(get_db), current_admin: User = Depends(auth_service.require_admin)) -> UserOut: # type: ignore[no-untyped-def] + return update_user_route(user_id, data, db, current_admin) @app.put("/users/{user_id}/projects", response_model=UserOut) -def update_user_projects(user_id: int, data: ProjectAccessUpdate, db: Session = Depends(get_db), _: User = Depends(auth_service.require_admin)) -> UserOut: - user = crud.get_user_by_id(db, user_id) - if not user: - raise HTTPException(404, "User not found") - updated = crud.set_user_projects(db, user, data.projects) - return to_user_out(updated) +def update_user_projects(user_id: int, data: ProjectAccessUpdate, db=Depends(get_db), _: User = Depends(auth_service.require_admin)) -> UserOut: # type: ignore[no-untyped-def] + return update_user_projects_route(user_id, data, db) @app.delete("/users/{user_id}", status_code=204) -def delete_user(user_id: int, db: Session = Depends(get_db), current_admin: User = Depends(auth_service.require_admin)) -> Response: - user = crud.get_user_by_id(db, user_id) - if not user: - raise HTTPException(404, "User not found") - if current_admin.id == user.id: - raise HTTPException(400, "Admin cannot delete self") - crud.delete_user(db, user) - return Response(status_code=204) +def delete_user(user_id: int, db=Depends(get_db), current_admin: User = Depends(auth_service.require_admin)) -> Response: # type: ignore[no-untyped-def] + return delete_user_route(user_id, db, current_admin) @app.get("/projects", response_model=list[ProjectOut]) -def list_projects(db: Session = Depends(get_db), _: User = Depends(auth_service.require_admin)) -> list[ProjectOut]: - return [to_project_out(project) for project in crud.list_projects(db)] +def list_projects(db=Depends(get_db), _: User = Depends(auth_service.require_admin)) -> list[ProjectOut]: # type: ignore[no-untyped-def] + return list_projects_route(db) @app.get("/projects/{project_id}", response_model=ProjectOut) -def get_project(project_id: int, db: Session = Depends(get_db), _: User = Depends(auth_service.require_admin)) -> ProjectOut: - project = crud.get_project_by_id(db, project_id) - if not project: - raise HTTPException(404, "Project not found") - return to_project_out(project) +def get_project(project_id: int, db=Depends(get_db), _: User = Depends(auth_service.require_admin)) -> ProjectOut: # type: ignore[no-untyped-def] + return get_project_route(project_id, db) @app.post("/projects", response_model=ProjectOut) -def create_project(data: ProjectCreate, db: Session = Depends(get_db), _: User = Depends(auth_service.require_admin)) -> ProjectOut: - try: - project = crud.create_project(db, data.name) - except ValueError as exc: - raise HTTPException(409, str(exc)) from exc - return to_project_out(project) +def create_project(data: ProjectCreate, db=Depends(get_db), _: User = Depends(auth_service.require_admin)) -> ProjectOut: # type: ignore[no-untyped-def] + return create_project_route(data, db) @app.put("/projects/{project_id}", response_model=ProjectOut) -def update_project(project_id: int, data: ProjectUpdate, db: Session = Depends(get_db), _: User = Depends(auth_service.require_admin)) -> ProjectOut: - project = crud.get_project_by_id(db, project_id) - if not project: - raise HTTPException(404, "Project not found") - try: - updated = crud.update_project(db, project, name=data.name) - except ValueError as exc: - raise HTTPException(409, str(exc)) from exc - return to_project_out(updated) +def update_project(project_id: int, data: ProjectUpdate, db=Depends(get_db), _: User = Depends(auth_service.require_admin)) -> ProjectOut: # type: ignore[no-untyped-def] + return update_project_route(project_id, data, db) @app.delete("/projects/{project_id}", status_code=204) -def delete_project(project_id: int, db: Session = Depends(get_db), _: User = Depends(auth_service.require_admin)) -> Response: - project = crud.get_project_by_id(db, project_id) - if not project: - raise HTTPException(404, "Project not found") - crud.delete_project(db, project) - return Response(status_code=204) +def delete_project(project_id: int, db=Depends(get_db), _: User = Depends(auth_service.require_admin)) -> Response: # type: ignore[no-untyped-def] + return delete_project_route(project_id, db) @app.get("/prompts/search", response_model=list[PromptOut]) @@ -514,23 +243,7 @@ def search_prompts( db: Session = Depends(get_db), current_user: User = Depends(auth_service.get_current_user), ) -> list[PromptOut]: - if project is not None: - auth_service.ensure_project_access(current_user, project) - user_allowed_projects = allowed_projects(current_user) - cache_key = build_prompt_collection_cache_key( - route="prompts.search", - project=project, - tags=tags, - mode=mode, - allowed_projects=user_allowed_projects, - ) - - def _load_search() -> list[dict[str, object]]: - prompts = crud.search_prompts_by_tags(db, tags=tags, mode=mode, project=project, allowed_projects=user_allowed_projects) - return [to_prompt_out(db, p).model_dump(mode="json") for p in prompts] - - cached_prompts = cache_get_or_set(cache_key, PROMPT_CACHE_TTL_SECONDS, _load_search) - return [PromptOut(**prompt) for prompt in cached_prompts] + return search_prompts_route(tags, mode, project, db, current_user) @app.get("/prompts", response_model=list[PromptOut]) @@ -543,238 +256,57 @@ def list_prompts( db: Session = Depends(get_db), current_user: User = Depends(auth_service.get_current_user), ) -> list[PromptOut]: - # Convert string parameters to int safely - limit_int: int | None = None - offset_int: int | None = None - try: - if limit: - limit_int = int(limit) - if limit_int < 1: - limit_int = 1 - except (ValueError, TypeError): - limit_int = None - - try: - if offset: - offset_int = int(offset) - if offset_int < 0: - offset_int = 0 - except (ValueError, TypeError): - offset_int = None - - if project is not None: - auth_service.ensure_project_access(current_user, project) - user_allowed_projects = allowed_projects(current_user) - cache_key = build_prompt_collection_cache_key( - route="prompts.list", - project=project, - tag=tag, - limit=limit_int, - offset=offset_int, - allowed_projects=user_allowed_projects, - ) - - def _load_list() -> dict[str, object]: - total_count = crud.count_prompts(db, project=project, tag=tag, allowed_projects=user_allowed_projects) - prompts = crud.list_prompts(db, project=project, tag=tag, limit=limit_int, offset=offset_int, allowed_projects=user_allowed_projects) - return { - "total_count": total_count, - "prompts": [to_prompt_out(db, prompt).model_dump(mode="json") for prompt in prompts], - } - - cached_list = cache_get_or_set(cache_key, PROMPT_CACHE_TTL_SECONDS, _load_list) - response.headers["X-Total-Count"] = str(int(cached_list.get("total_count", 0))) - return [PromptOut(**prompt) for prompt in cached_list.get("prompts", [])] + return list_prompts_route(response, project, tag, limit, offset, db, current_user) @app.post("/prompts", response_model=PromptOut) -def create_prompt(data: PromptCreate, db: Session = Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> PromptOut: - logger.info("prompt.create name={} project={}", data.name, data.project) - auth_service.ensure_project_access(current_user, data.project) - prompt = crud.get_prompt(db, data.name, data.project, allowed_projects=allowed_projects(current_user)) - if prompt: - logger.warning("prompt.create.duplicate name={} project={}", data.name, data.project) - raise HTTPException(400, "Prompt already exists") - - try: - prompt = crud.create_prompt( - db, - data.name, - data.project, - task=data.task, - actor_id=current_user.id, - role=data.role, - context=data.context, - constraints=data.constraints, - output_format=data.output_format, - examples=data.examples, - tags=data.tags, - ) - except ValueError as exc: - logger.warning("prompt.create.duplicate_content name={} project={}", data.name, data.project) - raise HTTPException(409, str(exc)) from exc - _invalidate_prompt_cache(data.project, data.name) - return to_prompt_out(db, prompt) +def create_prompt(data: PromptCreate, db=Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> PromptOut: # type: ignore[no-untyped-def] + return create_prompt_route(data, db, current_user) @app.get("/prompts/{project}/{name}", response_model=PromptOut) -def get_prompt(project: str, name: str, db: Session = Depends(get_db), current_user: User = Depends(auth_service.get_current_user)) -> PromptOut: - auth_service.ensure_project_access(current_user, project) - cache_key = build_prompt_response_cache_key(project, name) - - def _load_prompt() -> dict[str, object]: - prompt = crud.get_prompt(db, name, project, allowed_projects=allowed_projects(current_user)) - if not prompt: - raise HTTPException(404, "Prompt not found") - return to_prompt_out(db, prompt).model_dump(mode="json") - - cached_prompt = cache_get_or_set(cache_key, PROMPT_CACHE_TTL_SECONDS, _load_prompt) - return PromptOut(**cached_prompt) +def get_prompt(project: str, name: str, db=Depends(get_db), current_user: User = Depends(auth_service.get_current_user)) -> PromptOut: # type: ignore[no-untyped-def] + return get_prompt_route(project, name, db, current_user) @app.delete("/prompts/{project}/{name}", status_code=204) -def delete_prompt(project: str, name: str, db: Session = Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> Response: - logger.info("prompt.delete project={} name={}", project, name) - auth_service.ensure_project_access(current_user, project) - prompt = crud.get_prompt(db, name, project, allowed_projects=allowed_projects(current_user)) - if not prompt: - logger.warning("prompt.delete.not_found project={} name={}", project, name) - raise HTTPException(404, "Prompt not found") - - crud.delete_prompt(db, prompt) - _invalidate_prompt_cache(project, name) - return Response(status_code=204) +def delete_prompt(project: str, name: str, db=Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> Response: # type: ignore[no-untyped-def] + return delete_prompt_route(project, name, db, current_user) @app.put("/prompts/{project}/{name}", response_model=PromptVersionOut) -def update_prompt(project: str, name: str, data: PromptUpdate, db: Session = Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> PromptVersionOut: - logger.info("prompt.update project={} name={}", project, name) - auth_service.ensure_project_access(current_user, project) - prompt = crud.get_prompt(db, name, project, allowed_projects=allowed_projects(current_user)) - if not prompt: - logger.warning("prompt.update.not_found project={} name={}", project, name) - raise HTTPException(404, "Prompt not found") - - if data.tags is not None: - crud.set_prompt_tags(db, prompt, data.tags, actor_id=current_user.id) - - try: - new_version = crud.add_version( - db, - prompt.id, - task=data.task, - actor_id=current_user.id, - role=data.role, - context=data.context, - constraints=data.constraints, - output_format=data.output_format, - examples=data.examples, - ) - except ValueError as exc: - logger.warning("prompt.update.conflict project={} name={} error={}", project, name, str(exc)) - raise HTTPException(409, str(exc)) from exc - _invalidate_prompt_cache(project, name) - return to_prompt_version_out(db, new_version) +def update_prompt(project: str, name: str, data: PromptUpdate, db=Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> PromptVersionOut: # type: ignore[no-untyped-def] + return update_prompt_route(project, name, data, db, current_user) @app.put("/prompts/{project}/{name}/tags", response_model=PromptOut) -def update_prompt_tags(project: str, name: str, data: PromptTagsUpdate, db: Session = Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> PromptOut: - auth_service.ensure_project_access(current_user, project) - prompt = crud.get_prompt(db, name, project, allowed_projects=allowed_projects(current_user)) - if not prompt: - raise HTTPException(404, "Prompt not found") - - crud.set_prompt_tags(db, prompt, data.tags, actor_id=current_user.id) - _invalidate_prompt_cache(project, name) - # Reload prompt from database to ensure all relationships are properly populated - prompt = crud.get_prompt(db, name, project, allowed_projects=allowed_projects(current_user)) - if not prompt: - raise HTTPException(404, "Prompt not found after update") - return to_prompt_out(db, prompt) +def update_prompt_tags(project: str, name: str, data: PromptTagsUpdate, db=Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> PromptOut: # type: ignore[no-untyped-def] + return update_prompt_tags_route(project, name, data, db, current_user) @app.get("/prompts/{project}/{name}/versions", response_model=list[PromptVersionOut]) -def list_versions(project: str, name: str, db: Session = Depends(get_db), current_user: User = Depends(auth_service.get_current_user)) -> list[PromptVersionOut]: - auth_service.ensure_project_access(current_user, project) - cache_key = build_prompt_collection_cache_key(route="prompts.versions", project=project, name=name) - - def _load_versions() -> list[dict[str, object]]: - prompt = crud.get_prompt(db, name, project, allowed_projects=allowed_projects(current_user)) - if not prompt: - raise HTTPException(404, "Prompt not found") - - versions = crud.list_versions(db, prompt.id) - return [to_prompt_version_out(db, version).model_dump(mode="json") for version in versions] - - cached_versions = cache_get_or_set(cache_key, PROMPT_CACHE_TTL_SECONDS, _load_versions) - return [PromptVersionOut(**version) for version in cached_versions] +def list_versions(project: str, name: str, db=Depends(get_db), current_user: User = Depends(auth_service.get_current_user)) -> list[PromptVersionOut]: # type: ignore[no-untyped-def] + return list_versions_route(project, name, db, current_user) @app.get("/prompts/{project}/{name}/versions/{version}", response_model=PromptVersionOut) -def get_version(project: str, name: str, version: int, db: Session = Depends(get_db), current_user: User = Depends(auth_service.get_current_user)) -> PromptVersionOut: - auth_service.ensure_project_access(current_user, project) - cache_key = build_prompt_collection_cache_key(route="prompts.version", project=project, name=name, version=version) - - def _load_version() -> dict[str, object]: - prompt = crud.get_prompt(db, name, project, allowed_projects=allowed_projects(current_user)) - if not prompt: - raise HTTPException(404, "Prompt not found") - - v = crud.get_specific_version(db, prompt.id, version) - if not v: - raise HTTPException(404, "Version not found") - - return to_prompt_version_out(db, v).model_dump(mode="json") - - cached_version = cache_get_or_set(cache_key, PROMPT_CACHE_TTL_SECONDS, _load_version) - return PromptVersionOut(**cached_version) +def get_prompt_version(project: str, name: str, version: int, db=Depends(get_db), current_user: User = Depends(auth_service.get_current_user)) -> PromptVersionOut: # type: ignore[no-untyped-def] + return get_prompt_version_route(project, name, version, db, current_user) @app.post("/optimize", response_model=PromptOptimizeResponse) -def optimize_prompt(data: PromptData, db: Session = Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> PromptOptimizeResponse: - logger.info("optimize.request.start") - result = optimize_prompt_with_active_backend(data.model_dump(), get_personal_config(db, current_user)) - logger.info("optimize.request.done engine={}", result.engine) - optimized_dict = result.optimized_fields - if not isinstance(optimized_dict, dict): - optimized_dict = {} - optimized = PromptData( - role=optimized_dict.get("role"), - task=optimized_dict.get("task") or "", - context=optimized_dict.get("context"), - constraints=optimized_dict.get("constraints"), - output_format=optimized_dict.get("output_format"), - examples=optimized_dict.get("examples"), - ) - return PromptOptimizeResponse( - engine=result.engine, - optimized=optimized, - optimized_markdown=result.optimized_markdown, - notes=result.notes, - elapsed_seconds=result.elapsed_seconds, - ) +def optimize_prompt(data: PromptData, db=Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> PromptOptimizeResponse: # type: ignore[no-untyped-def] + return optimize_prompt_route(data, db, current_user, optimize_prompt_with_active_backend) @app.get("/optimize/config", response_model=OptimizeConfigOut) -def get_optimize_config(db: Session = Depends(get_db), current_user: User = Depends(auth_service.get_current_user)) -> OptimizeConfigOut: - cfg = get_personal_config(db, current_user) - logger.info( - "optimize.config.get effective_llm_provider={} effective_llm_model={}", - cfg.get("effective_llm_provider"), - cfg.get("effective_llm_model"), - ) - return OptimizeConfigOut(**cfg) +def get_optimize_config(db=Depends(get_db), current_user: User = Depends(auth_service.get_current_user)) -> OptimizeConfigOut: # type: ignore[no-untyped-def] + return get_optimize_config_route(db, current_user) @app.put("/optimize/config", response_model=OptimizeConfigOut) -def update_optimize_config(data: OptimizeConfigUpdate, db: Session = Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> OptimizeConfigOut: - logger.info( - "optimize.config.update llm_provider={} llm_model={}", - data.llm_provider, - data.llm_model, - ) - cfg = auth_service.update_personal_config(db, current_user, data.model_dump()) - return OptimizeConfigOut(**cfg) +def update_optimize_config(data: OptimizeConfigUpdate, db=Depends(get_db), current_user: User = Depends(auth_service.require_write_access)) -> OptimizeConfigOut: # type: ignore[no-untyped-def] + return update_optimize_config_route(data, db, current_user) @app.get("/optimize/providers/{provider}/models", response_model=list[str]) @@ -786,12 +318,4 @@ def get_provider_models( db: Session = Depends(get_db), current_user: User = Depends(auth_service.get_current_user), ) -> list[str]: - logger.info("optimize.provider.models provider={}", provider) - models = list_available_models( - provider, - base_url=base_url, - timeout_seconds=timeout_seconds, - api_token=api_token, - config_override=get_personal_config(db, current_user), - ) - return models + return get_provider_models_route(provider, base_url, api_token, timeout_seconds, db, current_user, list_available_models) diff --git a/middleware/__init__.py b/middleware/__init__.py new file mode 100644 index 0000000..3bd4791 --- /dev/null +++ b/middleware/__init__.py @@ -0,0 +1,4 @@ +from middleware.exception_logging import ExceptionLoggingMiddleware +from middleware.request_logging import RequestLoggingMiddleware + +__all__ = ["ExceptionLoggingMiddleware", "RequestLoggingMiddleware"] diff --git a/middleware/exception_logging.py b/middleware/exception_logging.py new file mode 100644 index 0000000..6e81747 --- /dev/null +++ b/middleware/exception_logging.py @@ -0,0 +1,26 @@ +from time import perf_counter + +from fastapi import Request, Response +from fastapi.responses import JSONResponse +from loguru import logger +from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint + + +class ExceptionLoggingMiddleware(BaseHTTPMiddleware): + async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response: + started_at = perf_counter() + try: + response: Response = await call_next(request) + return response + except Exception: + duration_ms = (perf_counter() - started_at) * 1000 + client = request.client.host if request.client else "unknown" + logger.exception( + "request.exception method={} path={} query={} client={} duration_ms={:.2f}", + request.method, + request.url.path, + request.url.query, + client, + duration_ms, + ) + return JSONResponse(status_code=500, content={"detail": "Internal Server Error"}) diff --git a/middleware/request_logging.py b/middleware/request_logging.py new file mode 100644 index 0000000..5ad4f08 --- /dev/null +++ b/middleware/request_logging.py @@ -0,0 +1,30 @@ +from time import perf_counter + +from fastapi import Request, Response +from loguru import logger +from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint + + +class RequestLoggingMiddleware(BaseHTTPMiddleware): + async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response: + start = perf_counter() + client = request.client.host if request.client else "unknown" + logger.info( + "request.start method={} path={} query={} client={}", + request.method, + request.url.path, + request.url.query, + client, + ) + + response: Response = await call_next(request) + + duration_ms = (perf_counter() - start) * 1000 + logger.info( + "request.end method={} path={} status={} duration_ms={:.2f}", + request.method, + request.url.path, + response.status_code, + duration_ms, + ) + return response diff --git a/models/__init__.py b/models/__init__.py index bae96ac..6dc13a2 100644 --- a/models/__init__.py +++ b/models/__init__.py @@ -1,13 +1,13 @@ from database import Base from models.models import ( Config, - Prompt, - PromptVersion, Project, ProjectAccess, + Prompt, + PromptVersion, Role, Tag, User, ) -__all__ = ["Base", "Config", "Prompt", "PromptVersion", "Project", "ProjectAccess", "Role", "Tag", "User"] +__all__ = ["Base", "Config", "Project", "ProjectAccess", "Prompt", "PromptVersion", "Role", "Tag", "User"] diff --git a/models/models.py b/models/models.py index 21e1dc9..c9ac426 100644 --- a/models/models.py +++ b/models/models.py @@ -1,4 +1,18 @@ -from sqlalchemy import Boolean, CheckConstraint, Column, DateTime, ForeignKey, Integer, String, Table, Text, UniqueConstraint, func +import datetime + +from sqlalchemy import ( + Boolean, + CheckConstraint, + Column, + DateTime, + ForeignKey, + Integer, + String, + Table, + Text, + UniqueConstraint, + func, +) from sqlalchemy.orm import Mapped, relationship from database import Base @@ -47,8 +61,8 @@ class Prompt(Base): id: Mapped[int] = Column(Integer, primary_key=True, index=True) # type: ignore[assignment] name: Mapped[str] = Column(String, index=True, nullable=False) # type: ignore[assignment] project_id: Mapped[int] = Column(Integer, ForeignKey("projects.id", ondelete="CASCADE"), index=True, nullable=False) # type: ignore[assignment] - created_at: Mapped[DateTime] = Column(DateTime(timezone=True), nullable=False, server_default=func.now()) # type: ignore[assignment] - updated_at: Mapped[DateTime] = Column(DateTime(timezone=True), nullable=False, server_default=func.now()) # type: ignore[assignment] + created_at: Mapped[datetime.datetime] = Column(DateTime(timezone=True), nullable=False, server_default=func.now()) # type: ignore[assignment] + updated_at: Mapped[datetime.datetime] = Column(DateTime(timezone=True), nullable=False, server_default=func.now()) # type: ignore[assignment] created_by_id: Mapped[int | None] = Column(Integer, ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True) # type: ignore[assignment] updated_by_id: Mapped[int | None] = Column(Integer, ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True) # type: ignore[assignment] @@ -75,7 +89,7 @@ class User(Base): password_hash_encrypted: Mapped[str] = Column(Text, nullable=False) # type: ignore[assignment] role_id: Mapped[int] = Column(Integer, ForeignKey("roles.id", ondelete="RESTRICT"), nullable=False, index=True) # type: ignore[assignment] is_active: Mapped[bool] = Column(Boolean, nullable=False, default=True) # type: ignore[assignment] - password_changed_at: Mapped[DateTime | None] = Column(DateTime(timezone=True), nullable=True) # type: ignore[assignment] + password_changed_at: Mapped[datetime.datetime | None] = Column(DateTime(timezone=True), nullable=True) # type: ignore[assignment] role_ref: Mapped["Role"] = relationship("Role", back_populates="users") config: Mapped["Config | None"] = relationship("Config", back_populates="user", cascade="all, delete-orphan", uselist=False) @@ -149,7 +163,7 @@ class PromptVersion(Base): id: Mapped[int] = Column(Integer, primary_key=True, index=True) # type: ignore[assignment] prompt_id: Mapped[int] = Column(Integer, ForeignKey("prompts.id", ondelete="CASCADE")) # type: ignore[assignment] version: Mapped[int] = Column(Integer) # type: ignore[assignment] - created_at: Mapped[DateTime] = Column(DateTime(timezone=True), nullable=False, server_default=func.now()) # type: ignore[assignment] + created_at: Mapped[datetime.datetime] = Column(DateTime(timezone=True), nullable=False, server_default=func.now()) # type: ignore[assignment] created_by_id: Mapped[int | None] = Column(Integer, ForeignKey("users.id", ondelete="SET NULL"), nullable=True, index=True) # type: ignore[assignment] role: Mapped[str | None] = Column(String, nullable=True) # type: ignore[assignment] task: Mapped[str] = Column(Text, nullable=False) # type: ignore[assignment] diff --git a/optimizer/__init__.py b/optimizer/__init__.py index 2ee048b..d85b20a 100644 --- a/optimizer/__init__.py +++ b/optimizer/__init__.py @@ -1,11 +1,11 @@ -from optimizer.result import OptimizationResult -from optimizer.errors import BackendOperationTimeoutError from optimizer.base import PromptOptimizerBackend +from optimizer.errors import BackendOperationTimeoutError from optimizer.leo_backend import LeoPromptOptimizerBackend +from optimizer.result import OptimizationResult __all__ = [ - "OptimizationResult", "BackendOperationTimeoutError", - "PromptOptimizerBackend", "LeoPromptOptimizerBackend", + "OptimizationResult", + "PromptOptimizerBackend", ] diff --git a/optimizer/base.py b/optimizer/base.py index 82dee28..97628c5 100644 --- a/optimizer/base.py +++ b/optimizer/base.py @@ -1,9 +1,12 @@ from __future__ import annotations from abc import ABC, abstractmethod -from typing import Any +from typing import TYPE_CHECKING, Any -from optimizer.result import OptimizationResult +if TYPE_CHECKING: + from collections.abc import Mapping + + from optimizer.result import OptimizationResult class PromptOptimizerBackend(ABC): @@ -13,7 +16,7 @@ def name(self) -> str: raise NotImplementedError @abstractmethod - def optimize(self, fields: dict[str, str | None], config: dict[str, Any]) -> OptimizationResult: + def optimize(self, fields: Mapping[str, str | None], config: Mapping[str, Any]) -> OptimizationResult: raise NotImplementedError @abstractmethod @@ -24,6 +27,6 @@ def list_models( base_url: str | None = None, timeout_seconds: int = 5, api_token: str | None = None, - config_override: dict[str, Any] | None = None, + config_override: Mapping[str, Any] | None = None, ) -> list[str]: raise NotImplementedError diff --git a/optimizer/leo_backend.py b/optimizer/leo_backend.py index af1e785..4abf04b 100644 --- a/optimizer/leo_backend.py +++ b/optimizer/leo_backend.py @@ -4,7 +4,7 @@ import os import re import time -from typing import Any +from typing import TYPE_CHECKING, Any from urllib import error as urllib_error from urllib import request as urllib_request @@ -21,6 +21,9 @@ _run_with_timeout, ) +if TYPE_CHECKING: + from collections.abc import Mapping + class LeoPromptOptimizerBackend(PromptOptimizerBackend): @property @@ -28,7 +31,8 @@ def name(self) -> str: return "leo" def _normalize_ollama_base_url(self, base_url: str | None) -> str: - candidate = (base_url or os.getenv("OLLAMA_BASE_URL", "http://127.0.0.1:11434")).strip() + env_base_url = os.getenv("OLLAMA_BASE_URL") or "http://127.0.0.1:11434" + candidate = (base_url or env_base_url).strip() if not candidate: candidate = "http://127.0.0.1:11434" candidate = candidate.rstrip("/") @@ -80,7 +84,7 @@ def _pick_low_memory_ollama_model(self, models: list[str], current_model: str) - return None - def _build_provider(self, provider_name: str, api_token: str | None, base_url: str | None): + def _build_provider(self, provider_name: str, api_token: str | None, base_url: str | None) -> tuple[Any, str]: from leo_prompt_optimizer import ( AnthropicProvider, GeminiProvider, @@ -92,12 +96,12 @@ def _build_provider(self, provider_name: str, api_token: str | None, base_url: s normalized = (provider_name or "openai").strip().lower() if normalized == "openai": if self._looks_like_ollama_base_url(base_url): - key = (api_token or "ollama").strip() or "ollama" - url = self._normalize_ollama_base_url(base_url) - return OpenAIProvider(api_key=key, base_url=url), "openai-compat-ollama" + ollama_key = (api_token or "ollama").strip() or "ollama" + ollama_url = self._normalize_ollama_base_url(base_url) + return OpenAIProvider(api_key=ollama_key, base_url=ollama_url), "openai-compat-ollama" - key = (api_token or "").strip() or None - url = (base_url or "").strip() or None + key: str | None = (api_token or "").strip() or None + url: str | None = (base_url or "").strip() or None return OpenAIProvider(api_key=key, base_url=url), "openai" if normalized == "ollama": key = (api_token or "ollama").strip() or "ollama" @@ -114,11 +118,11 @@ def _build_provider(self, provider_name: str, api_token: str | None, base_url: s raise ValueError(f"Unsupported provider: {normalized}") - def optimize(self, fields: dict[str, str | None], config: dict[str, Any]) -> OptimizationResult: + def optimize(self, fields: Mapping[str, str | None], config: Mapping[str, Any]) -> OptimizationResult: from leo_prompt_optimizer import LeoOptimizer - provider_name = config["effective_llm_provider"] - model_name = config["effective_llm_model"] + provider_name = str(config["effective_llm_provider"]) + model_name = str(config["effective_llm_model"]) base_url = config.get("effective_llm_base_url") api_token = config.get("effective_llm_api_token") timeout_seconds = max(5, int(config.get("effective_llm_timeout_seconds") or 120)) @@ -235,7 +239,7 @@ def list_models( base_url: str | None = None, timeout_seconds: int = 5, api_token: str | None = None, - config_override: dict[str, Any] | None = None, + config_override: Mapping[str, Any] | None = None, ) -> list[str]: normalized = (provider or "").strip().lower() if normalized == "openai" and self._looks_like_ollama_base_url(base_url): diff --git a/optimizer/service.py b/optimizer/service.py index 6956016..c1ee7b8 100644 --- a/optimizer/service.py +++ b/optimizer/service.py @@ -4,27 +4,40 @@ import hashlib import os from threading import Lock -from typing import Any +from typing import TYPE_CHECKING, Any, cast from loguru import logger -from optimizer.result import OptimizationResult -from optimizer.errors import BackendOperationTimeoutError -from optimizer.base import PromptOptimizerBackend -from optimizer.leo_backend import LeoPromptOptimizerBackend -from optimizer.utils import ( - _normalize_text, - _build_full_prompt, - _heuristic_improve, - _extract_prefixed_section, - _build_backend_failure_note, - _parse_structured_response, - _run_with_timeout, +from cache.shared_cache import ( + OPTIMIZATION_CACHE_TTL_SECONDS, + build_optimization_cache_key, + cache_get_or_set, ) -from cache.shared_cache import OPTIMIZATION_CACHE_TTL_SECONDS, build_optimization_cache_key, cache_get_or_set +from optimizer.leo_backend import LeoPromptOptimizerBackend +from optimizer.result import OptimizationResult +from optimizer.utils import _normalize_text, _parse_structured_response + +if TYPE_CHECKING: + from collections.abc import Mapping + + from optimizer.base import PromptOptimizerBackend _runtime_config_lock = Lock() +__all__ = [ + "LeoPromptOptimizerBackend", + "OptimizationResult", + "_normalize_text", + "_parse_structured_response", + "build_optimizer_config", + "get_active_optimizer_backend", + "get_active_optimizer_backend_name", + "get_runtime_optimizer_config", + "list_available_models", + "optimize_prompt_with_active_backend", + "set_runtime_optimizer_config", +] + # --------------------------------------------------------------------------- # Token encryption utilities @@ -73,7 +86,8 @@ def _decrypt_token(encrypted_token: str | None) -> str | None: } -def build_optimizer_config(overrides: dict[str, Any] | None = None) -> dict[str, Any]: +def build_optimizer_config(overrides: Mapping[str, Any] | None = None) -> dict[str, Any]: + source: Mapping[str, Any] if overrides is None: with _runtime_config_lock: source = dict(_runtime_optimize_config) @@ -184,11 +198,11 @@ def get_active_optimizer_backend() -> PromptOptimizerBackend: return _BACKEND_REGISTRY[get_active_optimizer_backend_name()] -def optimize_prompt_with_active_backend(fields: dict[str, str | None], config_override: dict[str, Any] | None = None) -> OptimizationResult: +def optimize_prompt_with_active_backend(fields: Mapping[str, str | None], config_override: Mapping[str, Any] | None = None) -> OptimizationResult: backend = get_active_optimizer_backend() config = build_optimizer_config(config_override) cache_key = build_optimization_cache_key(fields, config, backend.name) - return cache_get_or_set(cache_key, OPTIMIZATION_CACHE_TTL_SECONDS, lambda: backend.optimize(fields, config)) + return cast("OptimizationResult", cache_get_or_set(cache_key, OPTIMIZATION_CACHE_TTL_SECONDS, lambda: backend.optimize(fields, config))) def list_available_models( @@ -197,7 +211,7 @@ def list_available_models( base_url: str | None = None, timeout_seconds: int = 5, api_token: str | None = None, - config_override: dict[str, Any] | None = None, + config_override: Mapping[str, Any] | None = None, ) -> list[str]: backend = get_active_optimizer_backend() return backend.list_models( diff --git a/pyproject.toml b/pyproject.toml index af4ef4e..3db4c84 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -8,6 +8,7 @@ dependencies = [ "uvicorn[standard]>=0.29.0", "sqlalchemy>=2.0.0", "alembic>=1.13.0", + "psycopg2-binary>=2.9.12", "requests>=2.33.1", "leo-prompt-optimizer>=1.0.9", "loguru>=0.7.2", @@ -18,6 +19,7 @@ dependencies = [ dev = [ "httpx>=0.27.0", # FastAPI TestClient "pytest>=8.3.0", + "pytest-cov>=5.0.0", "mypy>=1.10.0", "ruff>=0.4.0", "locust>=2.31.0", @@ -33,6 +35,7 @@ dev = [ dev = [ "httpx>=0.27.0", "pytest>=8.3.0", + "pytest-cov>=5.0.0", "mypy>=1.10.0", "ruff>=0.4.0", "locust>=2.31.0", @@ -41,40 +44,6 @@ dev = [ "types-requests>=2.33.0", ] -# --------------------------------------------------------------------------- -# Ruff — linter + formatter -# --------------------------------------------------------------------------- -[tool.ruff] -target-version = "py311" -line-length = 100 -select = [ - "E", # pycodestyle errors - "W", # pycodestyle warnings - "F", # pyflakes - "I", # isort - "UP", # pyupgrade - "B", # flake8-bugbear - "C4", # flake8-comprehensions - "SIM", # flake8-simplify - "TC", # flake8-type-checking - "RUF", # ruff-specific rules -] -ignore = [ - "E501", # line too long — handled by formatter - "B008", # FastAPI uses Query/Depends in function signatures by design -] - -[tool.ruff.isort] -known-first-party = ["crud", "database", "models", "schemas"] - -[tool.ruff.per-file-ignores] -"tests/conftest.py" = ["E402"] - -[tool.ruff.format] -quote-style = "double" -indent-style = "space" -docstring-code-format = true - # --------------------------------------------------------------------------- # Mypy — static type checker # --------------------------------------------------------------------------- @@ -88,6 +57,8 @@ disallow_untyped_defs = true disallow_incomplete_defs = true check_untyped_defs = true no_implicit_optional = true +explicit_package_bases = true +exclude = ["^snippets/", "^loadtests/", "^tests/"] [[tool.mypy.overrides]] module = ["alembic.*"] @@ -96,3 +67,18 @@ ignore_missing_imports = true [[tool.mypy.overrides]] module = ["leo_prompt_optimizer.*"] ignore_missing_imports = true + +[[tool.mypy.overrides]] +module = ["app_core.*", "cache.*", "database", "middleware.*", "models", "models.*", "optimizer.*", "routes.shared"] +strict = true + +# --------------------------------------------------------------------------- +# Coverage +# --------------------------------------------------------------------------- +[tool.coverage.run] +source = ["app_core", "auth", "cache", "crud", "database", "main", "middleware", "models", "optimizer", "routes", "schemas"] +omit = ["snippets/*", "loadtests/*", "tests/*", "alembic/*"] + +[tool.coverage.report] +skip_empty = true +show_missing = false diff --git a/requirements.txt b/requirements.txt index 732b672..56b4cc4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,6 +2,7 @@ fastapi uvicorn sqlalchemy alembic +psycopg2-binary requests leo-prompt-optimizer loguru diff --git a/routes/__init__.py b/routes/__init__.py new file mode 100644 index 0000000..a22fbd5 --- /dev/null +++ b/routes/__init__.py @@ -0,0 +1,69 @@ +from routes.app_version import get_app_version_route +from routes.auth_bootstrap_admin import bootstrap_admin_route +from routes.auth_change_password import change_own_password_route +from routes.auth_login import login_route +from routes.auth_me import get_me_route +from routes.auth_refresh import refresh_auth_route +from routes.auth_status import get_auth_status_route +from routes.optimize_config_get import get_optimize_config_route +from routes.optimize_config_update import update_optimize_config_route +from routes.optimize_prompt import optimize_prompt_route +from routes.optimize_provider_models import get_provider_models_route +from routes.projects_create import create_project_route +from routes.projects_delete import delete_project_route +from routes.projects_get import get_project_route +from routes.projects_list import list_projects_route +from routes.projects_update import update_project_route +from routes.prompts_create import create_prompt_route +from routes.prompts_delete import delete_prompt_route +from routes.prompts_get import get_prompt_route +from routes.prompts_list import list_prompts_route +from routes.prompts_search import search_prompts_route +from routes.prompts_update import update_prompt_route +from routes.prompts_update_tags import update_prompt_tags_route +from routes.prompts_version_get import get_prompt_version_route +from routes.prompts_versions_list import list_versions_route +from routes.roles_list import list_roles_route +from routes.serve_ui import serve_ui_route +from routes.users_create import create_user_route +from routes.users_delete import delete_user_route +from routes.users_get import get_user_route +from routes.users_list import list_users_route +from routes.users_update import update_user_route +from routes.users_update_projects import update_user_projects_route + +__all__ = [ + "bootstrap_admin_route", + "change_own_password_route", + "create_project_route", + "create_prompt_route", + "create_user_route", + "delete_project_route", + "delete_prompt_route", + "delete_user_route", + "get_app_version_route", + "get_auth_status_route", + "get_me_route", + "get_optimize_config_route", + "get_project_route", + "get_prompt_route", + "get_prompt_version_route", + "get_provider_models_route", + "get_user_route", + "list_projects_route", + "list_prompts_route", + "list_roles_route", + "list_users_route", + "list_versions_route", + "login_route", + "optimize_prompt_route", + "refresh_auth_route", + "search_prompts_route", + "serve_ui_route", + "update_optimize_config_route", + "update_project_route", + "update_prompt_route", + "update_prompt_tags_route", + "update_user_projects_route", + "update_user_route", +] diff --git a/routes/app_version.py b/routes/app_version.py new file mode 100644 index 0000000..ac38685 --- /dev/null +++ b/routes/app_version.py @@ -0,0 +1,4 @@ + + +def get_app_version_route(app_version: str) -> dict[str, str]: + return {"name": "prompt-man", "version": app_version} diff --git a/routes/auth_bootstrap_admin.py b/routes/auth_bootstrap_admin.py new file mode 100644 index 0000000..77bf3a3 --- /dev/null +++ b/routes/auth_bootstrap_admin.py @@ -0,0 +1,21 @@ +from fastapi import HTTPException + +import auth as auth_service +import crud +from database import run_db_call +from schemas import AuthResponse, UserBootstrap + + +def bootstrap_admin_route(data: UserBootstrap, db) -> AuthResponse: # type: ignore[no-untyped-def] + if run_db_call(db, crud.list_users): + raise HTTPException(409, "Users already exist") + user = run_db_call( + db, + auth_service.create_user_record, + username=data.username, + password=data.password, + role="admin", + is_active=True, + projects=[], + ) + return AuthResponse(**auth_service.build_auth_response(user)) diff --git a/routes/auth_change_password.py b/routes/auth_change_password.py new file mode 100644 index 0000000..1c546ee --- /dev/null +++ b/routes/auth_change_password.py @@ -0,0 +1,14 @@ +import auth as auth_service +from database import run_db_call +from models import User +from schemas import ChangePasswordRequest + + +def change_own_password_route(data: ChangePasswordRequest, db, current_user: User) -> None: # type: ignore[no-untyped-def] + run_db_call( + db, + auth_service.change_own_password, + current_user, + current_password=data.current_password, + new_password=data.new_password, + ) diff --git a/routes/auth_login.py b/routes/auth_login.py new file mode 100644 index 0000000..00c1bed --- /dev/null +++ b/routes/auth_login.py @@ -0,0 +1,12 @@ +from fastapi import HTTPException + +import auth as auth_service +from database import run_db_call +from schemas import AuthResponse, UserLogin + + +def login_route(data: UserLogin, db) -> AuthResponse: # type: ignore[no-untyped-def] + user = run_db_call(db, auth_service.authenticate_user, data.username, data.password) + if not user: + raise HTTPException(401, "Invalid credentials") + return AuthResponse(**auth_service.build_auth_response(user)) diff --git a/routes/auth_me.py b/routes/auth_me.py new file mode 100644 index 0000000..34407f7 --- /dev/null +++ b/routes/auth_me.py @@ -0,0 +1,7 @@ +from models import User +from routes.shared import to_user_out +from schemas import UserOut + + +def get_me_route(current_user: User) -> UserOut: + return to_user_out(current_user) diff --git a/routes/auth_refresh.py b/routes/auth_refresh.py new file mode 100644 index 0000000..00c40df --- /dev/null +++ b/routes/auth_refresh.py @@ -0,0 +1,7 @@ +import auth as auth_service +from database import run_db_call +from schemas import AuthResponse, RefreshTokenRequest + + +def refresh_auth_route(data: RefreshTokenRequest, db) -> AuthResponse: # type: ignore[no-untyped-def] + return AuthResponse(**run_db_call(db, auth_service.refresh_session, data.refresh_token)) diff --git a/routes/auth_status.py b/routes/auth_status.py new file mode 100644 index 0000000..6b703de --- /dev/null +++ b/routes/auth_status.py @@ -0,0 +1,8 @@ +import crud +from database import run_db_call +from schemas import AuthStatus + + +def get_auth_status_route(db) -> AuthStatus: # type: ignore[no-untyped-def] + has_users = bool(run_db_call(db, crud.list_users)) + return AuthStatus(bootstrap_required=not has_users, has_users=has_users) diff --git a/routes/optimize_config_get.py b/routes/optimize_config_get.py new file mode 100644 index 0000000..ce319e9 --- /dev/null +++ b/routes/optimize_config_get.py @@ -0,0 +1,14 @@ +from loguru import logger + +from routes.shared import get_personal_config +from schemas import OptimizeConfigOut + + +def get_optimize_config_route(db, current_user) -> OptimizeConfigOut: # type: ignore[no-untyped-def] + cfg = get_personal_config(db, current_user) + logger.info( + "optimize.config.get effective_llm_provider={} effective_llm_model={}", + cfg.get("effective_llm_provider"), + cfg.get("effective_llm_model"), + ) + return OptimizeConfigOut(**cfg) diff --git a/routes/optimize_config_update.py b/routes/optimize_config_update.py new file mode 100644 index 0000000..c8fb676 --- /dev/null +++ b/routes/optimize_config_update.py @@ -0,0 +1,15 @@ +from loguru import logger + +import auth as auth_service +from database import run_db_call +from schemas import OptimizeConfigOut, OptimizeConfigUpdate + + +def update_optimize_config_route(data: OptimizeConfigUpdate, db, current_user) -> OptimizeConfigOut: # type: ignore[no-untyped-def] + logger.info( + "optimize.config.update llm_provider={} llm_model={}", + data.llm_provider, + data.llm_model, + ) + cfg = run_db_call(db, auth_service.update_personal_config, current_user, data.model_dump()) + return OptimizeConfigOut(**cfg) diff --git a/routes/optimize_prompt.py b/routes/optimize_prompt.py new file mode 100644 index 0000000..4b96816 --- /dev/null +++ b/routes/optimize_prompt.py @@ -0,0 +1,28 @@ +from loguru import logger + +from routes.shared import get_personal_config +from schemas import PromptData, PromptOptimizeResponse + + +def optimize_prompt_route(data: PromptData, db, current_user, optimizer) -> PromptOptimizeResponse: # type: ignore[no-untyped-def] + logger.info("optimize.request.start") + result = optimizer(data.model_dump(), get_personal_config(db, current_user)) + logger.info("optimize.request.done engine={}", result.engine) + optimized_dict = result.optimized_fields + if not isinstance(optimized_dict, dict): + optimized_dict = {} + optimized = PromptData( + role=optimized_dict.get("role"), + task=optimized_dict.get("task") or "", + context=optimized_dict.get("context"), + constraints=optimized_dict.get("constraints"), + output_format=optimized_dict.get("output_format"), + examples=optimized_dict.get("examples"), + ) + return PromptOptimizeResponse( + engine=result.engine, + optimized=optimized, + optimized_markdown=result.optimized_markdown, + notes=result.notes, + elapsed_seconds=result.elapsed_seconds, + ) diff --git a/routes/optimize_provider_models.py b/routes/optimize_provider_models.py new file mode 100644 index 0000000..b7fa788 --- /dev/null +++ b/routes/optimize_provider_models.py @@ -0,0 +1,26 @@ +from collections.abc import Callable + +from loguru import logger +from sqlalchemy.orm import Session + +from models import User +from routes.shared import get_personal_config + + +def get_provider_models_route( + provider: str, + base_url: str | None, + api_token: str | None, + timeout_seconds: int, + db: Session, + current_user: User, + model_lister: Callable[..., list[str]], +) -> list[str]: + logger.info("optimize.provider.models provider={}", provider) + return model_lister( + provider, + base_url=base_url, + timeout_seconds=timeout_seconds, + api_token=api_token, + config_override=get_personal_config(db, current_user), + ) diff --git a/routes/projects_create.py b/routes/projects_create.py new file mode 100644 index 0000000..148f5f9 --- /dev/null +++ b/routes/projects_create.py @@ -0,0 +1,14 @@ +from fastapi import HTTPException + +import crud +from database import run_db_call +from routes.shared import to_project_out +from schemas import ProjectCreate, ProjectOut + + +def create_project_route(data: ProjectCreate, db) -> ProjectOut: # type: ignore[no-untyped-def] + try: + project = run_db_call(db, crud.create_project, data.name) + except ValueError as exc: + raise HTTPException(409, str(exc)) from exc + return to_project_out(project) diff --git a/routes/projects_delete.py b/routes/projects_delete.py new file mode 100644 index 0000000..a6311f6 --- /dev/null +++ b/routes/projects_delete.py @@ -0,0 +1,12 @@ +from fastapi import HTTPException, Response + +import crud +from database import run_db_call + + +def delete_project_route(project_id: int, db) -> Response: # type: ignore[no-untyped-def] + project = run_db_call(db, crud.get_project_by_id, project_id) + if not project: + raise HTTPException(404, "Project not found") + run_db_call(db, crud.delete_project, project) + return Response(status_code=204) diff --git a/routes/projects_get.py b/routes/projects_get.py new file mode 100644 index 0000000..d2871f8 --- /dev/null +++ b/routes/projects_get.py @@ -0,0 +1,13 @@ +from fastapi import HTTPException + +import crud +from database import run_db_call +from routes.shared import to_project_out +from schemas import ProjectOut + + +def get_project_route(project_id: int, db) -> ProjectOut: # type: ignore[no-untyped-def] + project = run_db_call(db, crud.get_project_by_id, project_id) + if not project: + raise HTTPException(404, "Project not found") + return to_project_out(project) diff --git a/routes/projects_list.py b/routes/projects_list.py new file mode 100644 index 0000000..818a0ce --- /dev/null +++ b/routes/projects_list.py @@ -0,0 +1,8 @@ +import crud +from database import run_db_call +from routes.shared import to_project_out +from schemas import ProjectOut + + +def list_projects_route(db) -> list[ProjectOut]: # type: ignore[no-untyped-def] + return [to_project_out(project) for project in run_db_call(db, crud.list_projects)] diff --git a/routes/projects_update.py b/routes/projects_update.py new file mode 100644 index 0000000..8ef8875 --- /dev/null +++ b/routes/projects_update.py @@ -0,0 +1,17 @@ +from fastapi import HTTPException + +import crud +from database import run_db_call +from routes.shared import to_project_out +from schemas import ProjectOut, ProjectUpdate + + +def update_project_route(project_id: int, data: ProjectUpdate, db) -> ProjectOut: # type: ignore[no-untyped-def] + project = run_db_call(db, crud.get_project_by_id, project_id) + if not project: + raise HTTPException(404, "Project not found") + try: + updated = run_db_call(db, crud.update_project, project, name=data.name) + except ValueError as exc: + raise HTTPException(409, str(exc)) from exc + return to_project_out(updated) diff --git a/routes/prompts_create.py b/routes/prompts_create.py new file mode 100644 index 0000000..71c8f18 --- /dev/null +++ b/routes/prompts_create.py @@ -0,0 +1,42 @@ +from fastapi import HTTPException +from loguru import logger + +import auth as auth_service +import crud +from database import run_db_call +from models import User +from routes.shared import allowed_projects, invalidate_prompt_cache, to_prompt_out +from schemas import PromptCreate, PromptOut + + +def create_prompt_route(data: PromptCreate, db, current_user: User) -> PromptOut: # type: ignore[no-untyped-def] + logger.info("prompt.create name={} project={}", data.name, data.project) + auth_service.ensure_project_access(current_user, data.project) + prompt = run_db_call(db, crud.get_prompt, data.name, data.project, allowed_projects=allowed_projects(current_user)) + if prompt: + logger.warning("prompt.create.duplicate name={} project={}", data.name, data.project) + raise HTTPException(400, "Prompt already exists") + + try: + prompt = run_db_call( + db, + crud.create_prompt, + data.name, + data.project, + task=data.task, + actor_id=current_user.id, + role=data.role, + context=data.context, + constraints=data.constraints, + output_format=data.output_format, + examples=data.examples, + tags=data.tags, + ) + except ValueError as exc: + logger.warning("prompt.create.duplicate_content name={} project={}", data.name, data.project) + raise HTTPException(409, str(exc)) from exc + invalidate_prompt_cache(data.project, data.name) + prompt = run_db_call(db, crud.get_prompt, data.name, data.project, allowed_projects=allowed_projects(current_user)) + if not prompt: + raise HTTPException(404, "Prompt not found after create") + return to_prompt_out(db, prompt) diff --git a/routes/prompts_delete.py b/routes/prompts_delete.py new file mode 100644 index 0000000..fda4739 --- /dev/null +++ b/routes/prompts_delete.py @@ -0,0 +1,21 @@ +from fastapi import HTTPException, Response +from loguru import logger + +import auth as auth_service +import crud +from database import run_db_call +from models import User +from routes.shared import allowed_projects, invalidate_prompt_cache + + +def delete_prompt_route(project: str, name: str, db, current_user: User) -> Response: # type: ignore[no-untyped-def] + logger.info("prompt.delete project={} name={}", project, name) + auth_service.ensure_project_access(current_user, project) + prompt = run_db_call(db, crud.get_prompt, name, project, allowed_projects=allowed_projects(current_user)) + if not prompt: + logger.warning("prompt.delete.not_found project={} name={}", project, name) + raise HTTPException(404, "Prompt not found") + + run_db_call(db, crud.delete_prompt, prompt) + invalidate_prompt_cache(project, name) + return Response(status_code=204) diff --git a/routes/prompts_get.py b/routes/prompts_get.py new file mode 100644 index 0000000..bd37ad5 --- /dev/null +++ b/routes/prompts_get.py @@ -0,0 +1,27 @@ +from fastapi import HTTPException + +import auth as auth_service +import crud +from cache.shared_cache import ( + PROMPT_CACHE_TTL_SECONDS, + build_prompt_response_cache_key, + cache_get_or_set, +) +from database import run_db_call +from models import User +from routes.shared import allowed_projects, to_prompt_out +from schemas import PromptOut + + +def get_prompt_route(project: str, name: str, db, current_user: User) -> PromptOut: # type: ignore[no-untyped-def] + auth_service.ensure_project_access(current_user, project) + cache_key = build_prompt_response_cache_key(project, name) + + def _load_prompt() -> dict[str, object]: + prompt = run_db_call(db, crud.get_prompt, name, project, allowed_projects=allowed_projects(current_user)) + if not prompt: + raise HTTPException(404, "Prompt not found") + return to_prompt_out(db, prompt).model_dump(mode="json") + + cached_prompt = cache_get_or_set(cache_key, PROMPT_CACHE_TTL_SECONDS, _load_prompt) + return PromptOut(**cached_prompt) diff --git a/routes/prompts_list.py b/routes/prompts_list.py new file mode 100644 index 0000000..04fc3ab --- /dev/null +++ b/routes/prompts_list.py @@ -0,0 +1,66 @@ +from fastapi import Response +from sqlalchemy.orm import Session + +import auth as auth_service +import crud +from cache.shared_cache import ( + PROMPT_CACHE_TTL_SECONDS, + build_prompt_collection_cache_key, + cache_get_or_set, +) +from database import run_db_call +from models import User +from routes.shared import allowed_projects, to_prompt_out +from schemas import PromptOut + + +def list_prompts_route( + response: Response, + project: str | None, + tag: str | None, + limit: str | None, + offset: str | None, + db: Session, + current_user: User, +) -> list[PromptOut]: + limit_int: int | None = None + offset_int: int | None = None + try: + if limit: + limit_int = int(limit) + if limit_int < 1: + limit_int = 1 + except (ValueError, TypeError): + limit_int = None + + try: + if offset: + offset_int = int(offset) + if offset_int < 0: + offset_int = 0 + except (ValueError, TypeError): + offset_int = None + + if project is not None: + auth_service.ensure_project_access(current_user, project) + user_allowed_projects = allowed_projects(current_user) + cache_key = build_prompt_collection_cache_key( + route="prompts.list", + project=project, + tag=tag, + limit=limit_int, + offset=offset_int, + allowed_projects=user_allowed_projects, + ) + + def _load_list() -> dict[str, object]: + total_count = run_db_call(db, crud.count_prompts, project=project, tag=tag, allowed_projects=user_allowed_projects) + prompts = run_db_call(db, crud.list_prompts, project=project, tag=tag, limit=limit_int, offset=offset_int, allowed_projects=user_allowed_projects) + return { + "total_count": total_count, + "prompts": [to_prompt_out(db, prompt).model_dump(mode="json") for prompt in prompts], + } + + cached_list = cache_get_or_set(cache_key, PROMPT_CACHE_TTL_SECONDS, _load_list) + response.headers["X-Total-Count"] = str(int(cached_list.get("total_count", 0))) + return [PromptOut(**prompt) for prompt in cached_list.get("prompts", [])] diff --git a/routes/prompts_search.py b/routes/prompts_search.py new file mode 100644 index 0000000..71022f1 --- /dev/null +++ b/routes/prompts_search.py @@ -0,0 +1,39 @@ +from sqlalchemy.orm import Session + +import auth as auth_service +import crud +from cache.shared_cache import ( + PROMPT_CACHE_TTL_SECONDS, + build_prompt_collection_cache_key, + cache_get_or_set, +) +from database import run_db_call +from models import User +from routes.shared import allowed_projects, to_prompt_out +from schemas import PromptOut + + +def search_prompts_route( + tags: list[str], + mode: str, + project: str | None, + db: Session, + current_user: User, +) -> list[PromptOut]: + if project is not None: + auth_service.ensure_project_access(current_user, project) + user_allowed_projects = allowed_projects(current_user) + cache_key = build_prompt_collection_cache_key( + route="prompts.search", + project=project, + tags=tags, + mode=mode, + allowed_projects=user_allowed_projects, + ) + + def _load_search() -> list[dict[str, object]]: + prompts = run_db_call(db, crud.search_prompts_by_tags, tags=tags, mode=mode, project=project, allowed_projects=user_allowed_projects) + return [to_prompt_out(db, prompt).model_dump(mode="json") for prompt in prompts] + + cached_prompts = cache_get_or_set(cache_key, PROMPT_CACHE_TTL_SECONDS, _load_search) + return [PromptOut(**prompt) for prompt in cached_prompts] diff --git a/routes/prompts_update.py b/routes/prompts_update.py new file mode 100644 index 0000000..ad11c46 --- /dev/null +++ b/routes/prompts_update.py @@ -0,0 +1,43 @@ +from fastapi import HTTPException +from loguru import logger + +import auth as auth_service +import crud +from database import run_db_call +from models import User +from routes.shared import allowed_projects, invalidate_prompt_cache, to_prompt_version_out +from schemas import PromptUpdate, PromptVersionOut + + +def update_prompt_route(project: str, name: str, data: PromptUpdate, db, current_user: User) -> PromptVersionOut: # type: ignore[no-untyped-def] + logger.info("prompt.update project={} name={}", project, name) + auth_service.ensure_project_access(current_user, project) + prompt = run_db_call(db, crud.get_prompt, name, project, allowed_projects=allowed_projects(current_user)) + if not prompt: + logger.warning("prompt.update.not_found project={} name={}", project, name) + raise HTTPException(404, "Prompt not found") + + if data.tags is not None: + run_db_call(db, crud.set_prompt_tags, prompt, data.tags, actor_id=current_user.id) + + try: + new_version = run_db_call( + db, + crud.add_version, + prompt.id, + task=data.task, + actor_id=current_user.id, + role=data.role, + context=data.context, + constraints=data.constraints, + output_format=data.output_format, + examples=data.examples, + ) + except ValueError as exc: + logger.warning("prompt.update.conflict project={} name={} error={}", project, name, str(exc)) + raise HTTPException(409, str(exc)) from exc + invalidate_prompt_cache(project, name) + refreshed_version = run_db_call(db, crud.get_specific_version, prompt.id, new_version.version) + if not refreshed_version: + raise HTTPException(404, "Version not found after update") + return to_prompt_version_out(db, refreshed_version) diff --git a/routes/prompts_update_tags.py b/routes/prompts_update_tags.py new file mode 100644 index 0000000..c61feb9 --- /dev/null +++ b/routes/prompts_update_tags.py @@ -0,0 +1,22 @@ +from fastapi import HTTPException + +import auth as auth_service +import crud +from database import run_db_call +from models import User +from routes.shared import allowed_projects, invalidate_prompt_cache, to_prompt_out +from schemas import PromptOut, PromptTagsUpdate + + +def update_prompt_tags_route(project: str, name: str, data: PromptTagsUpdate, db, current_user: User) -> PromptOut: # type: ignore[no-untyped-def] + auth_service.ensure_project_access(current_user, project) + prompt = run_db_call(db, crud.get_prompt, name, project, allowed_projects=allowed_projects(current_user)) + if not prompt: + raise HTTPException(404, "Prompt not found") + + run_db_call(db, crud.set_prompt_tags, prompt, data.tags, actor_id=current_user.id) + invalidate_prompt_cache(project, name) + prompt = run_db_call(db, crud.get_prompt, name, project, allowed_projects=allowed_projects(current_user)) + if not prompt: + raise HTTPException(404, "Prompt not found after update") + return to_prompt_out(db, prompt) diff --git a/routes/prompts_version_get.py b/routes/prompts_version_get.py new file mode 100644 index 0000000..58b27a5 --- /dev/null +++ b/routes/prompts_version_get.py @@ -0,0 +1,32 @@ +from fastapi import HTTPException + +import auth as auth_service +import crud +from cache.shared_cache import ( + PROMPT_CACHE_TTL_SECONDS, + build_prompt_collection_cache_key, + cache_get_or_set, +) +from database import run_db_call +from models import User +from routes.shared import allowed_projects, to_prompt_version_out +from schemas import PromptVersionOut + + +def get_prompt_version_route(project: str, name: str, version: int, db, current_user: User) -> PromptVersionOut: # type: ignore[no-untyped-def] + auth_service.ensure_project_access(current_user, project) + cache_key = build_prompt_collection_cache_key(route="prompts.version", project=project, name=name, version=version) + + def _load_version() -> dict[str, object]: + prompt = run_db_call(db, crud.get_prompt, name, project, allowed_projects=allowed_projects(current_user)) + if not prompt: + raise HTTPException(404, "Prompt not found") + + prompt_version = run_db_call(db, crud.get_specific_version, prompt.id, version) + if not prompt_version: + raise HTTPException(404, "Version not found") + + return to_prompt_version_out(db, prompt_version).model_dump(mode="json") + + cached_version = cache_get_or_set(cache_key, PROMPT_CACHE_TTL_SECONDS, _load_version) + return PromptVersionOut(**cached_version) diff --git a/routes/prompts_versions_list.py b/routes/prompts_versions_list.py new file mode 100644 index 0000000..2af3203 --- /dev/null +++ b/routes/prompts_versions_list.py @@ -0,0 +1,29 @@ +from fastapi import HTTPException + +import auth as auth_service +import crud +from cache.shared_cache import ( + PROMPT_CACHE_TTL_SECONDS, + build_prompt_collection_cache_key, + cache_get_or_set, +) +from database import run_db_call +from models import User +from routes.shared import allowed_projects, to_prompt_version_out +from schemas import PromptVersionOut + + +def list_versions_route(project: str, name: str, db, current_user: User) -> list[PromptVersionOut]: # type: ignore[no-untyped-def] + auth_service.ensure_project_access(current_user, project) + cache_key = build_prompt_collection_cache_key(route="prompts.versions", project=project, name=name) + + def _load_versions() -> list[dict[str, object]]: + prompt = run_db_call(db, crud.get_prompt, name, project, allowed_projects=allowed_projects(current_user)) + if not prompt: + raise HTTPException(404, "Prompt not found") + + versions = run_db_call(db, crud.list_versions, prompt.id) + return [to_prompt_version_out(db, version).model_dump(mode="json") for version in versions] + + cached_versions = cache_get_or_set(cache_key, PROMPT_CACHE_TTL_SECONDS, _load_versions) + return [PromptVersionOut(**version) for version in cached_versions] diff --git a/routes/roles_list.py b/routes/roles_list.py new file mode 100644 index 0000000..1e70d6c --- /dev/null +++ b/routes/roles_list.py @@ -0,0 +1,7 @@ +import auth as auth_service +from database import run_db_call +from schemas import RoleOut + + +def list_roles_route(db) -> list[RoleOut]: # type: ignore[no-untyped-def] + return [RoleOut(**item) for item in run_db_call(db, auth_service.list_roles_out)] diff --git a/routes/serve_ui.py b/routes/serve_ui.py new file mode 100644 index 0000000..3fa0647 --- /dev/null +++ b/routes/serve_ui.py @@ -0,0 +1,5 @@ +from fastapi.responses import FileResponse + + +def serve_ui_route() -> FileResponse: + return FileResponse("ui/html/index.html") diff --git a/routes/shared.py b/routes/shared.py new file mode 100644 index 0000000..77f66fe --- /dev/null +++ b/routes/shared.py @@ -0,0 +1,89 @@ +import datetime +from typing import Any, cast + +from sqlalchemy.orm import Session + +import auth as auth_service +import crud +from cache.shared_cache import ( + PROMPT_CACHE_PREFIX, + build_prompt_response_cache_key, + clear_shared_cache, + delete_shared_cache_entry, +) +from database import run_db_call +from models import Prompt, User +from schemas import ProjectOut, PromptOut, PromptVersionOut, UserOut + + +def to_user_out(user: User) -> UserOut: + return UserOut(**auth_service.user_to_dict(user)) + + +def to_project_out(project: Any) -> ProjectOut: + return ProjectOut(id=project.id, name=project.name) + + +def get_personal_config(db: Session, current_user: User) -> dict[str, Any]: + config = run_db_call(db, auth_service.get_or_create_personal_config, current_user) + return auth_service.serialize_optimizer_config(config) + + +def allowed_projects(current_user: User) -> list[str] | None: + return auth_service.allowed_projects_for_user(current_user) + + +def normalize_utc_datetime(value: datetime.datetime | None) -> datetime.datetime | None: + if value is None: + return None + if value.tzinfo is None: + return value.replace(tzinfo=datetime.UTC) + return value.astimezone(datetime.UTC) + + +def require_utc_datetime(value: datetime.datetime | None) -> datetime.datetime: + normalized = normalize_utc_datetime(value) + if normalized is None: + raise ValueError("Expected datetime value") + return normalized + + +def to_prompt_out(db: Session, prompt: Prompt) -> PromptOut: + latest = run_db_call(db, crud.get_latest_version, prompt.id) + if not latest: + raise ValueError(f"No version found for prompt {prompt.id}") + return PromptOut( + name=prompt.name, + project=prompt.project, + created_at=require_utc_datetime(cast("datetime.datetime | None", prompt.created_at)), + updated_at=require_utc_datetime(cast("datetime.datetime | None", prompt.updated_at)), + created_by_username=run_db_call(db, crud.resolve_audit_username, prompt.created_by_ref), + updated_by_username=run_db_call(db, crud.resolve_audit_username, prompt.updated_by_ref), + tags=[tag.name for tag in prompt.tags], + latest_version=latest.version, + role=latest.role, + task=latest.task, + context=latest.context, + constraints=latest.constraints, + output_format=latest.output_format, + examples=latest.examples, + ) + + +def invalidate_prompt_cache(project: str, name: str) -> None: + delete_shared_cache_entry(build_prompt_response_cache_key(project, name)) + clear_shared_cache(PROMPT_CACHE_PREFIX) + + +def to_prompt_version_out(db: Session, version: Any) -> PromptVersionOut: + return PromptVersionOut( + version=version.version, + created_at=require_utc_datetime(cast("datetime.datetime | None", version.created_at)), + created_by_username=run_db_call(db, crud.resolve_audit_username, version.created_by_ref), + role=version.role, + task=version.task, + context=version.context, + constraints=version.constraints, + output_format=version.output_format, + examples=version.examples, + ) diff --git a/routes/users_create.py b/routes/users_create.py new file mode 100644 index 0000000..ad121a0 --- /dev/null +++ b/routes/users_create.py @@ -0,0 +1,17 @@ +import auth as auth_service +from database import run_db_call +from routes.shared import to_user_out +from schemas import UserCreate, UserOut + + +def create_user_route(data: UserCreate, db) -> UserOut: # type: ignore[no-untyped-def] + user = run_db_call( + db, + auth_service.create_user_record, + username=data.username, + password=data.password, + role=data.role, + is_active=data.is_active, + projects=data.projects, + ) + return to_user_out(user) diff --git a/routes/users_delete.py b/routes/users_delete.py new file mode 100644 index 0000000..8be0f11 --- /dev/null +++ b/routes/users_delete.py @@ -0,0 +1,15 @@ +from fastapi import HTTPException, Response + +import crud +from database import run_db_call +from models import User + + +def delete_user_route(user_id: int, db, current_admin: User) -> Response: # type: ignore[no-untyped-def] + user = run_db_call(db, crud.get_user_by_id, user_id) + if not user: + raise HTTPException(404, "User not found") + if current_admin.id == user.id: + raise HTTPException(400, "Admin cannot delete self") + run_db_call(db, crud.delete_user, user) + return Response(status_code=204) diff --git a/routes/users_get.py b/routes/users_get.py new file mode 100644 index 0000000..5056886 --- /dev/null +++ b/routes/users_get.py @@ -0,0 +1,13 @@ +from fastapi import HTTPException + +import crud +from database import run_db_call +from routes.shared import to_user_out +from schemas import UserOut + + +def get_user_route(user_id: int, db) -> UserOut: # type: ignore[no-untyped-def] + user = run_db_call(db, crud.get_user_by_id, user_id) + if not user: + raise HTTPException(404, "User not found") + return to_user_out(user) diff --git a/routes/users_list.py b/routes/users_list.py new file mode 100644 index 0000000..12113df --- /dev/null +++ b/routes/users_list.py @@ -0,0 +1,8 @@ +import crud +from database import run_db_call +from routes.shared import to_user_out +from schemas import UserOut + + +def list_users_route(db) -> list[UserOut]: # type: ignore[no-untyped-def] + return [to_user_out(user) for user in run_db_call(db, crud.list_users)] diff --git a/routes/users_update.py b/routes/users_update.py new file mode 100644 index 0000000..bbc4d50 --- /dev/null +++ b/routes/users_update.py @@ -0,0 +1,27 @@ +from fastapi import HTTPException + +import auth as auth_service +import crud +from database import run_db_call +from models import User +from routes.shared import to_user_out +from schemas import UserOut, UserUpdate + + +def update_user_route(user_id: int, data: UserUpdate, db, current_admin: User) -> UserOut: # type: ignore[no-untyped-def] + user = run_db_call(db, crud.get_user_by_id, user_id) + if not user: + raise HTTPException(404, "User not found") + if current_admin.id == user.id and data.role is not None and data.role != "admin": + raise HTTPException(400, "Admin cannot remove own admin role") + updated = run_db_call( + db, + auth_service.update_user_record, + user, + username=data.username, + password=data.password, + role=data.role, + is_active=data.is_active, + projects=data.projects, + ) + return to_user_out(updated) diff --git a/routes/users_update_projects.py b/routes/users_update_projects.py new file mode 100644 index 0000000..2b8ebd5 --- /dev/null +++ b/routes/users_update_projects.py @@ -0,0 +1,14 @@ +from fastapi import HTTPException + +import crud +from database import run_db_call +from routes.shared import to_user_out +from schemas import ProjectAccessUpdate, UserOut + + +def update_user_projects_route(user_id: int, data: ProjectAccessUpdate, db) -> UserOut: # type: ignore[no-untyped-def] + user = run_db_call(db, crud.get_user_by_id, user_id) + if not user: + raise HTTPException(404, "User not found") + updated = run_db_call(db, crud.set_user_projects, user, data.projects) + return to_user_out(updated) diff --git a/ruff.toml b/ruff.toml new file mode 100644 index 0000000..67efbdf --- /dev/null +++ b/ruff.toml @@ -0,0 +1,31 @@ +target-version = "py311" +line-length = 100 + +[lint] +select = [ + "E", # pycodestyle errors + "W", # pycodestyle warnings + "F", # pyflakes + "I", # isort + "UP", # pyupgrade + "B", # flake8-bugbear + "C4", # flake8-comprehensions + "SIM", # flake8-simplify + "TC", # flake8-type-checking + "RUF", # ruff-specific rules +] +ignore = [ + "E501", # line too long — handled by formatter + "B008", # FastAPI uses Query/Depends in function signatures by design +] + +[lint.isort] +known-first-party = ["crud", "database", "models", "schemas"] + +[lint.per-file-ignores] +"tests/conftest.py" = ["E402"] + +[format] +quote-style = "double" +indent-style = "space" +docstring-code-format = true \ No newline at end of file diff --git a/schemas/schemas.py b/schemas/schemas.py index c31efbb..3040152 100644 --- a/schemas/schemas.py +++ b/schemas/schemas.py @@ -3,7 +3,6 @@ from pydantic import BaseModel, ConfigDict, StringConstraints - RequiredPromptText = Annotated[str, StringConstraints(strip_whitespace=True, min_length=1)] diff --git a/security.py b/security.py index b18cf2e..19a5a1a 100644 --- a/security.py +++ b/security.py @@ -45,6 +45,12 @@ def _build_cipher(secret: str) -> Fernet: _PREVIOUS_CIPHERS = [_build_cipher(secret) for secret in _previous_machine_secrets() if secret != _ACTIVE_SECRET] _TOKEN_SECRET = hashlib.sha256(_derive_key("token", _ACTIVE_SECRET)).digest() +__all__ = [ + "ACCESS_TOKEN_TTL_SECONDS", + "REFRESH_TOKEN_TTL_SECONDS", + "time", +] + def encrypt_secret(value: str | None) -> str | None: if not value or not value.strip(): diff --git a/test_imports.py b/test_imports.py index 594dc4f..d277fd1 100644 --- a/test_imports.py +++ b/test_imports.py @@ -1,4 +1,5 @@ import sys + print("Testing imports...") try: from models import Base, User @@ -8,7 +9,7 @@ sys.exit(1) try: - from schemas import AuthResponse, UserLogin + from schemas import AuthResponse print(f"✓ schemas imported, AuthResponse={AuthResponse}") except Exception as e: print(f"✗ schemas error: {e}") diff --git a/tests/conftest.py b/tests/conftest.py index 9bf65bf..b0c80c5 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,3 +1,4 @@ +import os import sys from collections.abc import Iterator from pathlib import Path @@ -20,14 +21,24 @@ from main import app, get_db from optimizer.service import set_runtime_optimizer_config +TEST_DATABASE_URL = os.getenv("TEST_DATABASE_URL", "").strip() + @pytest.fixture(scope="session") def engine(): # type: ignore[no-untyped-def] - test_engine = create_engine( - "sqlite://", - connect_args={"check_same_thread": False}, - poolclass=StaticPool, - ) + if TEST_DATABASE_URL: + test_engine = create_engine( + TEST_DATABASE_URL, + pool_pre_ping=True, + ) + else: + test_engine = create_engine( + "sqlite://", + connect_args={"check_same_thread": False}, + poolclass=StaticPool, + ) + + Base.metadata.drop_all(bind=test_engine) Base.metadata.create_all(bind=test_engine) return test_engine @@ -54,11 +65,13 @@ def override_get_db() -> Iterator[Session]: startup_session_factory = sessionmaker(autocommit=False, autoflush=False, bind=db_session.get_bind()) original_session_local = main.SessionLocal + original_startup_session_local = main.StartupSessionLocal original_init_database = main.init_database app.dependency_overrides[get_db] = override_get_db app.dependency_overrides[auth_service.get_db_session] = override_get_db main.SessionLocal = startup_session_factory + main.StartupSessionLocal = startup_session_factory main.init_database = lambda bind=None: Base.metadata.create_all(bind=db_session.get_bind()) with TestClient(app) as test_client: login_response = test_client.post( @@ -72,6 +85,7 @@ def override_get_db() -> Iterator[Session]: test_client.headers.update({"Authorization": f"Bearer {token}"}) yield test_client main.SessionLocal = original_session_local + main.StartupSessionLocal = original_startup_session_local main.init_database = original_init_database app.dependency_overrides.clear() diff --git a/tests/e2e/test_e2e_full_flow.py b/tests/e2e/test_e2e_full_flow.py index 1275c59..0bb1f8a 100644 --- a/tests/e2e/test_e2e_full_flow.py +++ b/tests/e2e/test_e2e_full_flow.py @@ -49,8 +49,6 @@ def fake_leo(fields, config_override=None): # type: ignore[no-untyped-def] put_cfg = client.put( "/optimize/config", json={ - "gp_profile": "quality", - "rounds": 3, "llm_provider": "ollama", "llm_model": "qwen2.5:0.5b", "llm_base_url": "http://127.0.0.1:11434", @@ -58,7 +56,8 @@ def fake_leo(fields, config_override=None): # type: ignore[no-untyped-def] }, ) assert put_cfg.status_code == 200 - assert put_cfg.json()["effective_gp_profile"] == "quality" + assert put_cfg.json()["effective_llm_provider"] == "ollama" + assert put_cfg.json()["effective_llm_model"] == "qwen2.5:0.5b" assert client.get("/optimize/config").status_code == 200 # Optimize with Leo engine diff --git a/tests/integration/test_auth_rbac_endpoints.py b/tests/integration/test_auth_rbac_endpoints.py index d39e9fb..1efdf3d 100644 --- a/tests/integration/test_auth_rbac_endpoints.py +++ b/tests/integration/test_auth_rbac_endpoints.py @@ -1,6 +1,9 @@ -import security +from typing import Any + from fastapi.testclient import TestClient +import security + def login_as(client: TestClient, username: str, password: str) -> str: response = client.post("/auth/login", json={"username": username, "password": password}) @@ -8,7 +11,7 @@ def login_as(client: TestClient, username: str, password: str) -> str: return response.json()["access_token"] -def login_bundle(client: TestClient, username: str, password: str) -> dict: +def login_bundle(client: TestClient, username: str, password: str) -> dict[str, Any]: response = client.post("/auth/login", json={"username": username, "password": password}) assert response.status_code == 200 return response.json() @@ -299,7 +302,7 @@ def test_admin_can_manage_projects(client): # type: ignore[no-untyped-def] def test_admin_can_list_roles(client): # type: ignore[no-untyped-def] response = client.get("/roles") assert response.status_code == 200 - assert response.json() == [{"id": 1, "name": "admin"}, {"id": 2, "name": "developer"}, {"id": 3, "name": "viewer"}] + assert [item["name"] for item in response.json()] == ["admin", "developer", "viewer"] def test_deleting_project_cascades_prompts_and_access(client, sample_prompt_payload): # type: ignore[no-untyped-def] @@ -336,4 +339,4 @@ def test_deleting_project_cascades_prompts_and_access(client, sample_prompt_payl prompts_response = client.get("/prompts") assert prompts_response.status_code == 200 - assert prompts_response.json() == [] \ No newline at end of file + assert prompts_response.json() == [] diff --git a/tests/integration/test_race_conditions.py b/tests/integration/test_race_conditions.py index f0c5f9a..f40e5ac 100644 --- a/tests/integration/test_race_conditions.py +++ b/tests/integration/test_race_conditions.py @@ -1,8 +1,7 @@ from __future__ import annotations -from collections.abc import Iterator from concurrent.futures import ThreadPoolExecutor -from pathlib import Path +from typing import TYPE_CHECKING from fastapi.testclient import TestClient from sqlalchemy import create_engine @@ -13,15 +12,24 @@ from database import Base from main import app, get_db from models import Prompt, Tag +from tests.conftest import TEST_DATABASE_URL + +if TYPE_CHECKING: + from collections.abc import Iterator + from pathlib import Path def test_http_concurrent_create_prompt_with_shared_new_tag_is_race_safe(tmp_path: Path): # type: ignore[no-untyped-def] - db_path = tmp_path / "race_http.db" - engine = create_engine( - f"sqlite:///{db_path}", - connect_args={"check_same_thread": False}, - ) + if TEST_DATABASE_URL: + engine = create_engine(TEST_DATABASE_URL, pool_pre_ping=True) + else: + db_path = tmp_path / "race_http.db" + engine = create_engine( + f"sqlite:///{db_path}", + connect_args={"check_same_thread": False}, + ) testing_session_local = sessionmaker(autocommit=False, autoflush=False, bind=engine) + Base.metadata.drop_all(bind=engine) Base.metadata.create_all(bind=engine) def override_get_db() -> Iterator[Session]: @@ -32,10 +40,12 @@ def override_get_db() -> Iterator[Session]: db.close() original_session_local = main.SessionLocal + original_startup_session_local = main.StartupSessionLocal original_init_database = main.init_database try: main.SessionLocal = testing_session_local + main.StartupSessionLocal = testing_session_local main.init_database = lambda bind=None: Base.metadata.create_all(bind=engine) app.dependency_overrides[get_db] = override_get_db app.dependency_overrides[auth_service.get_db_session] = override_get_db @@ -90,6 +100,7 @@ def _worker(index: int) -> tuple[int, int]: verify_session.close() finally: main.SessionLocal = original_session_local + main.StartupSessionLocal = original_startup_session_local main.init_database = original_init_database app.dependency_overrides.clear() engine.dispose() diff --git a/tests/unit/test_unit_optimizer_and_crud.py b/tests/unit/test_unit_optimizer_and_crud.py index cd4924a..ce25ac1 100644 --- a/tests/unit/test_unit_optimizer_and_crud.py +++ b/tests/unit/test_unit_optimizer_and_crud.py @@ -2,13 +2,12 @@ from time import perf_counter, sleep import pytest -from sqlalchemy import insert -from sqlalchemy import create_engine +from sqlalchemy import create_engine, insert from sqlalchemy.exc import IntegrityError from sqlalchemy.orm import sessionmaker -import crud import auth as auth_service +import crud import main import optimizer.service as optimizer_service from database import Base, init_database @@ -17,10 +16,11 @@ LeoPromptOptimizerBackend, _normalize_text, _parse_structured_response, - optimize_prompt_with_active_backend, get_runtime_optimizer_config, + optimize_prompt_with_active_backend, set_runtime_optimizer_config, ) +from tests.conftest import TEST_DATABASE_URL def test_normalize_tags_deduplicates_and_sorts(): # type: ignore[no-untyped-def] @@ -283,12 +283,16 @@ def test_init_database_and_bootstrap_seed_default_roles(db_session): # type: ig def test_concurrent_create_prompt_with_shared_new_tag_is_race_safe(tmp_path): # type: ignore[no-untyped-def] - db_path = tmp_path / "race_test.db" - engine = create_engine( - f"sqlite:///{db_path}", - connect_args={"check_same_thread": False}, - ) + if TEST_DATABASE_URL: + engine = create_engine(TEST_DATABASE_URL, pool_pre_ping=True) + else: + db_path = tmp_path / "race_test.db" + engine = create_engine( + f"sqlite:///{db_path}", + connect_args={"check_same_thread": False}, + ) SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine) + Base.metadata.drop_all(bind=engine) Base.metadata.create_all(bind=engine) try: