psyb0t · Marinski · Sep 12, 2026
diff --git a/.gitignore b/.gitignore
@@ -73,6 +73,7 @@ fuckyeah.md
 .anustimes/
 .brutal_reviews/
 .pr-reviews/
+.deep-qa/
 
 # Local orchestration scaffolding — never committed
 .plan-and-delegate/

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,6 +8,32 @@ The project follows [Semantic Versioning](https://semver.org/): patch = bug fixe
 
 ## [Unreleased]
 
+### Fixed
+
+- **Broker symbol suffixes are no longer appended blindly** (`mt5api/backtest/handler.py` `_normalize_symbol`, new `mt5api/symbol_cache.py`). `symbol_suffix` used to be appended to every `[Tester].Symbol` that did not already end with it, but brokers rarely suffix their whole book — Eightcap Global carries 56 suffixed FX pairs (`EURUSD.i`) alongside 785 bare metals, indices and crypto (`XAUUSD`, `BTCUSD`, `ASX200`), so every non-FX backtest there asked the tester for a symbol that does not exist and came back empty. The suffix is now skipped when the broker's symbol list has the bare name and lacks the suffixed one. Brokers that carry both forms (BlackBull lists `AUDUSD` *and* `AUDUSDp`) still get the suffix, so `symbol_suffix: p` keeps meaning "use the prime variant".
+
+  A `mode: backtest` terminal never attaches the MT5 SDK and `Bases/<server>/symbols/*.dat` is encrypted, so there is no way to enumerate symbols at INI-build time. `GET /symbols` (unfiltered only) now persists what it saw to `<terminal>/mt5api-symbols.json`, and the INI builder reads that back. With no cache the old append-always behaviour is used unchanged, so this cannot regress a terminal that has never been primed. Tests: `tests/test_symbol_suffix_remap.py`.
+
+  The cache has a finite trust window, enforced inside `load()` itself: past `symbol_cache_max_age` (default 7 days; `SYMBOL_CACHE_MAX_AGE` in the environment), or with a missing/malformed `updated` stamp, it counts as absent and the append-always fallback applies. Without that, the cache was authoritative forever — a broker moving a symbol between bare and suffixed stayed wrongly normalized until someone happened to call `GET /symbols`.
+
+- **The documented cache-priming request no longer wedges the backtest terminal.** `GET /symbols` is `@with_mt5`, so calling it on a `mode: backtest` terminal — which never attaches the SDK at startup — fell through `ensure_initialized()` into a full `mt5.initialize()`, spawning `terminal64.exe` and holding the tester's single-instance data-dir lock for the rest of that terminal's life. Every backtest submitted to it afterward exited clean with an empty report. `GET /symbols` is now refused with 409 on a `mode: backtest` terminal before any SDK call. The new `POST /symbols/import` (`mt5api/handlers/symbols.py`) is the safe replacement: it writes a caller-supplied symbol list straight to `symbol_cache` and never calls `mt5.*` or acquires the MT5 lock, so it is the priming path for a broker/account that has no live terminal to call `GET /symbols` on. Test: `tests/test_symbol_import.py` asserts against the real Flask app that neither route reaches `ensure_initialized()` when `MODE == "backtest"`.
+
+  That refusal is also no longer queued behind the global MT5 lock. `@with_mt5` enters `session()` and blocks on `_mt5_lock` for up to `SESSION_ACQUIRE_TIMEOUT` *before* the handler body runs, so a backtest terminal already sitting behind a stuck SDK request answered with a 503 a minute later instead of the 409 — a refusal that needs no SDK at all. `list_symbols` now runs the `MODE` check undecorated and delegates the live path to a `@with_mt5` `_list_symbols_live`, so only the branch that touches the SDK waits for it. Tests hold `_mt5_lock` from another thread and assert the backtest request still returns 409 promptly without acquiring it, and that the live listing still does wait.
+
+  `config/config.yaml.example` documented the old, wedging flow — prime and refresh with `GET /<broker>/<account>/symbols` — which a `mode: backtest` reader now cannot follow at all. It carries the `POST /symbols/import` workflow instead, with a concrete request body and where to source the list (a live terminal on the same broker/account, or the broker's own documentation). Covered behaviorally rather than by asserting on the file's text: the test walks the documented steps against the real app and asserts the outcome the comment promises — that the INI builder stops appending the suffix to a symbol the broker carries bare.
+
+- **`POST /symbols/import` now bounds what it will accept** (`mt5api/handlers/symbols.py`, `mt5api/config.py`). The endpoint had no input limits at all: one `symbols` item of 2,097,153 bytes was accepted with a `200` and persisted into `<terminal>/mt5api-symbols.json`, a file the backtest INI builder reads and parses on every run inside a fixed-disk Windows VM. Three per-request caps now apply, each configurable at the top level of `config/config.yaml` or under the same name uppercased in the environment, and each clamped to a minimum of 1 rather than raising — config.py is imported by the whole API, so a typo in one endpoint's tuning value must not stop trading, nor disable the only offline path a `mode: backtest` terminal has for priming its cache.
+
+  `symbol_import_max_body_bytes` (default 2 MiB) is checked against the declared `Content-Length` **before** `request.get_json()` runs; parsing first would already have paid the memory cost the cap exists to prevent, so the placement is the point and is covered by a test that sends an over-cap `Content-Length` with a body that is not valid JSON — only an ordering where the gate runs first can answer `413`. A body with no declared length but a transfer encoding cannot be bounded up front and is refused with `411` rather than waved past the cap; a request with no body at all carries no transfer encoding and keeps the `400` it has always had. `symbol_import_max_symbols` (default 20000, ~20x the widest broker book seen in this fleet) is counted on the raw array before deduplication, because the cap's job is to bound the strip/dedupe/sort pass. `symbol_import_max_symbol_length` (default 64, double MT5's own 31-character symbol-name limit) is measured on the *normalized* name, so whitespace padding can neither fail a legal name nor smuggle an illegal one through. Count and length violations return `400`; nothing is written to the cache when a request is refused. Documented in `config/config.yaml.example` and `docs/market-data.md`; tests in `tests/test_symbol_import.py` cover the max and max-plus-one boundary of each cap, the pre-parse ordering, and the reported 2 MB payload at shipped defaults.
+
+- **Every request body is bounded now, not just `POST /symbols/import`'s** (`mt5api/server.py`, `mt5api/config.py`). Capping the one endpoint that was reported left six others parsing whatever arrived — `POST /orders`, `PUT /orders/<id>`, `PUT` and `DELETE /positions/<id>`, `POST /symbols/<symbol>/rates/ta`, `POST /backtest/build-ini` and `/backtest/build-set` — and a per-endpoint check only bounds the endpoint someone remembered, so the next route added is the next hole. The cap now lives once in the `before_request` hook every route already passes through: `max_request_body_bytes` (4 MiB) for anything that is not a file upload, `max_upload_body_bytes` (25 MiB) for multipart, since `POST /backtest` carries a compiled `.ex5` plus its `.set` and `.ini`. The upload cap matches the `client_max_body_size` nginx already enforces in front of the API, so reaching the port directly answers the same as coming through the proxy. Both are refused with 413 from the declared `Content-Length` before the body is parsed, and an endpoint with a tighter cap of its own still reports that one, because it is checked inside the handler. Tests in `tests/test_request_body_cap.py` cover every body-reading route, the pre-parse ordering on each, both caps' boundaries, and that the endpoint-specific cap still wins.
+
+- **The unified `endpoints` MCP tool is tested through the protocol now, not by parsing its source.** `tests/test_mcp_tool_parity.py` asserted the route catalog by AST-parsing `_ROUTE_CATALOG` out of `mcpunifier/mcp_server.py` — an implementation-source assertion that stays green while MCP tool registration, transport or response serialization breaks. It is replaced by black-box tests in `tests/integration/test_mcpunifier.py`, which boot the shipped unifier image, call the public `endpoints` tool over MCP, and compare the routes it returns against the real `mt5api.server.app.url_map`. Demonstrated by renaming the tool's response key: the old test still passed, the new ones fail. The comparison is equality in both directions — a missing entry hides a real route from agents, a surplus entry sends them at a 404 — with Werkzeug's converter prefix stripped from the Flask side (`<int:ticket>` against the catalog's `<ticket>`), since the converter is a routing-layer detail with no presence on the wire while the parameter name is what an agent reads. `requirements-test.txt` grows `flask`, `flask-compress` and `psutil` so the host-side integration run can import the real router instead of a hand-copied list.
+
+- **Terminal journals are now bounded by size as well as age** (`scripts/rotate-logs.sh`). The retention window added in v4.13.1 deletes dated journals once they are `RETAIN_DAYS` old, which cannot help with the actual hazard: a high-frequency grid strategy logs every order placement, modification and cancellation, so one backtest writes tens of gigabytes into *today's* journal — the disk fills a week before that file is even eligible for the age pass. Any journal still inside the retention window is now truncated once it exceeds `MAX_LOG_BYTES` (default 2 GiB), skipping anything written within `IDLE_MINUTES` (default 30) so a running backtest never loses its own diagnostics. Truncated in place rather than deleted, because `terminal64.exe` holds the journal open and unlinking the inode would leave it writing to a deleted file with the space unreclaimed until it exited. Both knobs are set on the `log-rotator` service in both the generated and single-VM Compose files alongside the existing retention window, and validated at startup like `RETAIN_DAYS`. Tests in `tests/test_rotate_logs.py` cover each journal location, inode preservation, the active-run exemption, files under the cap, and startup validation.
+
+  Sizing uses `stat` rather than `wc -c`: busybox `wc` reads the whole file to count bytes — roughly 18s per 3 GB journal in the `alpine:3.20` image the sidecar runs in, repeated every `INTERVAL` on precisely the files the cap exists for. The mtime is preserved across the truncation, so a just-emptied journal cannot outrank the journal a running job is writing in `_tail_dir_log`'s newest-by-mtime selection and blank out `GET /backtest/<id>/tail`.
+
 ## [v4.13.1]: 2026-09-10
 
 ### Changed

diff --git a/config/config.yaml.example b/config/config.yaml.example
@@ -66,6 +66,85 @@ accounts:
 #   See docs/multi-vm-setup.md for multi-VM setup.
 # symbol_suffix: optional broker-specific suffix appended to [Tester].Symbol
 #   when missing. Examples: "p", ".p", "-mini". Use "" for no suffix.
+#   Most brokers suffix only part of their book — Eightcap Global has 56
+#   suffixed FX pairs (EURUSD.i) and 785 bare metals/indices/crypto (XAUUSD,
+#   ASX200) — so the suffix is skipped for any symbol the broker carries bare
+#   and does NOT carry suffixed. That check needs a symbol list, which a
+#   backtest-mode terminal cannot fetch (no SDK attached), so it reads the
+#   cache at <terminal>/mt5api-symbols.json. Until that cache exists the
+#   suffix is appended unconditionally, as it always was.
+#
+#   Priming the cache, by mode:
+#     mode: live      — GET /<broker>/<account>/symbols writes it as a side
+#                       effect of the unfiltered listing. Nothing else to do.
+#     mode: backtest  — GET /symbols is REFUSED with 409 there: it would
+#                       attach the SDK, spawn terminal64.exe and hold the
+#                       tester's single-instance data-dir lock, leaving the
+#                       next backtest run with an empty report. Prime it with
+#                       POST /symbols/import instead, which touches no SDK:
+#
+#     curl -X POST -H "Authorization: Bearer $MT5_API_TOKEN" \
+#       -H 'Content-Type: application/json' \
+#       -d '{"symbols": ["EURUSD", "GBPUSD", "XAUUSD"]}' \
+#       http://127.0.0.1:8888/<broker>/<account>/symbols/import
+#
+#   Source that list from a live terminal on the SAME broker/account —
+#   `curl $MT5_API_URL/symbols` against it returns exactly the JSON array
+#   this endpoint accepts under the "symbols" key — or from the broker's own
+#   symbol documentation.
+#
+#   The cache is trusted for symbol_cache_max_age (default 7d, also
+#   SYMBOL_CACHE_MAX_AGE in the environment); past that it counts as absent
+#   again — refresh it the same way you primed it — so a broker moving a
+#   symbol between bare and suffixed cannot be papered over by a years-old
+#   list.
+#
+# symbol_cache_max_age is read at the TOP LEVEL of this file, not per
+# terminal — it applies to every terminal. Uncomment it here, at the same
+# indentation as `terminals:` below; setting it inside a terminal entry is
+# silently ignored.
+# symbol_cache_max_age: "7d"
+
+# Per-request caps on POST /symbols/import. Also read at the TOP LEVEL, and
+# also settable in the environment under the uppercase name. Each is clamped to
+# a minimum of 1 rather than raising, so a typo here cannot take the only
+# priming path a backtest terminal has offline.
+#
+# symbol_import_max_body_bytes (default 2097152, 2 MiB) — an oversized body is
+#   refused with 413 from its declared Content-Length, before any of it is
+#   parsed. A request with no Content-Length at all (chunked) is refused with
+#   411. Sized to admit the largest legitimate payload, which is the count cap
+#   times the length cap plus JSON quoting (~1.4 MB).
+# symbol_import_max_symbols (default 20000) — maximum entries in the 'symbols'
+#   array, counted before deduplication; over the cap is a 400. No MT5 broker
+#   publishes anything close: the widest books seen here are low thousands.
+# symbol_import_max_symbol_length (default 64) — maximum characters per symbol,
+#   measured on the NORMALIZED (whitespace-stripped) name; over the cap is a
+#   400. MT5 itself caps a symbol name at 31 characters, so this is double the
+#   platform's own limit.
+# symbol_import_max_body_bytes: 2097152
+# symbol_import_max_symbols: 20000
+# symbol_import_max_symbol_length: 64
+
+# Global per-request body caps, applied to EVERY route in one place
+# (server.py's before_request hook) rather than endpoint by endpoint. A
+# per-endpoint cap only bounds the endpoint someone remembered to add one to;
+# this bounds the rest, including routes added later. An endpoint with a
+# tighter cap of its own — symbol_import_max_body_bytes above — still reports
+# that one, because it is checked inside the handler.
+#
+# Both are refused with 413 from the declared Content-Length, before the body
+# is parsed, and both take the uppercase name in the environment.
+#
+# max_request_body_bytes (default 4194304, 4 MiB) — everything that is not a
+#   file upload. Well above the largest legitimate JSON this API takes.
+# max_upload_body_bytes (default 26214400, 25 MiB) — multipart, i.e.
+#   POST /backtest, which carries a compiled .ex5 plus its .set and .ini.
+#   Matched to the client_max_body_size nginx already enforces in front of
+#   this API, so reaching the port directly gives the same answer as coming
+#   through the proxy.
+# max_request_body_bytes: 4194304
+# max_upload_body_bytes: 26214400
 terminals:
   - broker: ftmo
     account: tenkchallenge

diff --git a/docker-compose.yml.example b/docker-compose.yml.example
@@ -87,6 +87,8 @@ services:
       LOG_DIR: /logs
       TERMINALS_DIR: /terminals
       RETAIN_DAYS: "7"
+      MAX_LOG_BYTES: "2147483648"
+      IDLE_MINUTES: "30"
       INTERVAL: "3600"
     volumes:
       - ./data/shared/logs:/logs

diff --git a/docker-compose.yml.j2 b/docker-compose.yml.j2
@@ -116,6 +116,8 @@ services:
       LOG_DIR: /logs
       TERMINALS_DIR: /terminals
       RETAIN_DAYS: "7"
+      MAX_LOG_BYTES: "2147483648"
+      IDLE_MINUTES: "30"
       INTERVAL: "3600"
     volumes:
       - /data/mt5-shared/logs:/logs

diff --git a/docs/market-data.md b/docs/market-data.md
@@ -15,7 +15,8 @@ Find symbols, inspect the contract details that brokers love making weird, pull
 
 | Method | Endpoint                 | Description                               |
 | ------ | ------------------------ | ----------------------------------------- |
-| GET    | `/symbols`               | List symbols (`?group=*USD*`)             |
+| GET    | `/symbols`               | List symbols (`?group=*USD*`). **Refused with 409 on a `mode: backtest` terminal** — it never attaches the SDK; see below. |
+| POST   | `/symbols/import`        | Prime the symbol cache from a JSON body without touching the SDK — the safe path on a `mode: backtest` terminal. |
 | GET    | `/symbols/:symbol`       | Symbol details                            |
 | GET    | `/symbols/:symbol/tick`  | Latest tick                               |
 | GET    | `/symbols/:symbol/rates` | OHLCV candles (`?timeframe=H1&count=100`, `?timeframe=H1&from=<unix>&count=-100`, or `?timeframe=H1&from=<unix>&to=<unix>`) |
@@ -28,6 +29,44 @@ Find symbols, inspect the contract details that brokers love making weird, pull
 ["EURUSD", "GBPUSD", "ADAUSD", "BTCUSD", "..."]
 ```
 
+On success (unfiltered, i.e. no `?group=`) this also persists the list to that
+terminal's symbol cache, which the backtest INI builder reads to decide
+whether a broker's `symbol_suffix` applies (see [Backtesting](backtesting.md)).
+`GET /symbols` calls the MT5 SDK, so on a `mode: backtest` terminal — which
+never attaches the SDK — it is refused with 409 instead of triggering a full
+`mt5.initialize()` that would spawn `terminal64.exe` and hold the tester's
+single-instance lock, leaving the next backtest run with an empty report.
+
+**POST `/symbols/import`** — prime a terminal's symbol cache directly, with no
+MT5 SDK call at all. This is the only supported way to prime a `mode: backtest`
+terminal: get the list from a live terminal on the same broker/account (`GET
+/symbols` there) or from the broker's own documentation, then post it here.
+
+```bash
+curl -X POST -H "Authorization: Bearer $MT5_API_TOKEN" \
+  -H 'Content-Type: application/json' \
+  -d '{"symbols": ["EURUSD", "GBPUSD", "XAUUSD"]}' \
+  "$MT5_API_URL/symbols/import"
+```
+
+```json
+{"imported": 3}
+```
+
+The body is bounded on three axes, each configurable at the top level of
+`config/config.yaml` or under the same name uppercased in the environment:
+
+| Setting | Default | Enforced |
+| ------- | ------- | -------- |
+| `symbol_import_max_body_bytes` | `2097152` (2 MiB) | `413` from the declared `Content-Length`, before the body is parsed. A request with no `Content-Length` at all (chunked) is refused with `411`. |
+| `symbol_import_max_symbols` | `20000` | `400` — entries in `symbols`, counted before deduplication. |
+| `symbol_import_max_symbol_length` | `64` | `400` — characters per symbol, measured on the whitespace-stripped name. |
+
+The defaults sit far above any real broker book — the widest seen in this fleet
+is 841 symbols, and MT5 itself caps a symbol name at 31 characters — so they
+only fire on input that was never going to be a usable symbol list. Nothing is
+written to the cache when a request is refused.
+
 **GET `/symbols/:symbol`** — full symbol info:
 
 ```json