diff --git a/.agents/skills/mt5-httpapi/SKILL.md b/.agents/skills/mt5-httpapi/SKILL.md index 92ce88b7..2dec246d 100644 --- a/.agents/skills/mt5-httpapi/SKILL.md +++ b/.agents/skills/mt5-httpapi/SKILL.md @@ -324,6 +324,56 @@ curl -H "Authorization: Bearer $MT5_API_TOKEN" "$MT5_API_URL/history/deals?from= Deal fields: `type` (0=buy, 1=sell), `entry` (0=opening, 1=closing), `profit` (0 for entries, realized P&L for exits). +### Chart Deployments + +Deploy Expert Advisors to charts over HTTP — no RDP, no terminal restart. +Stage `.ex5` + `.set` files, declare deployments, and a resident loader EA +inside the terminal reconciles charts to match. The API holds desired state; +the loader reports observed truth. A deployment only flips to `running` once +the loader confirms the expert is live on a chart. + +**Setup:** automatic via `[StartUp] Expert=` on boot. No manual attach needed. +Disable per terminal with `chartctl: false`. + +Endpoint reference: + +| Method | Endpoint | Description | +| ------ | -------- | ----------- | +| `POST` / `GET` / `DELETE` | `/experts` `/experts/` | Stage, list, remove EA `.ex5` | +| `POST` / `GET` | `/sets` `/sets/` | Stage, list, inspect `.set` (returns parsed inputs) | +| `POST` / `GET` | `/deployments` | Create or list deployments | +| `GET` / `PATCH` / `DELETE` | `/deployments/` | Inspect, pause/resume, change set, delete | +| `POST` | `/deployments/reconcile` | Force immediate reconcile | +| `GET` | `/charts` | Live chart/EA inventory | +| `GET` | `/loader` | Loader EA status and version | +| `POST` | `/charts//screenshot` | Capture chart as PNG | +| `POST` | `/charts//close` | Close a chart by id | + +```bash +# Stage artifacts +curl -F "expert=@HappyGoldScalp.ex5" "$MT5_API_URL/experts" +curl -F "set=@gold-m5.set" "$MT5_API_URL/sets" + +# Deploy +curl -X POST "$MT5_API_URL/deployments" \ + -H "Authorization: Bearer $MT5_API_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"expert":"HappyGoldScalp.ex5","set":"gold-m5.set","symbol":"XAUUSD","timeframe":"M5"}' + +# Check status +curl -H "Authorization: Bearer $MT5_API_TOKEN" "$MT5_API_URL/deployments" + +# Pause / resume / delete +curl -X PATCH "$MT5_API_URL/deployments/dep_a1b2c3" \ + -H "Authorization: Bearer $MT5_API_TOKEN" \ + -d '{"enabled":false}' +curl -X DELETE "$MT5_API_URL/deployments/dep_a1b2c3" \ + -H "Authorization: Bearer $MT5_API_TOKEN" +``` + +Full protocol: [`docs/chart-control-protocol.md`](docs/chart-control-protocol.md). +WebRequest allowlist provisioning: `GET/PUT /webrequest`, `POST /webrequest/apply`. + ### Backtest Run MT5 Strategy Tester via the API. Two-stage workflow: build the INI from a diff --git a/.gitignore b/.gitignore index 373b2473..5299c1d5 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,9 @@ mt5installers/* assets/experts/* !assets/experts/.gitkeep !assets/experts/MT5SystemWarmup.mq5 +!assets/experts/MT5ChartLoader.mq5 +!assets/experts/include +!assets/experts/include/ChartControl.mqh assets/sets/* !assets/sets/.gitkeep config/config.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index a144eb44..f1b0145e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,40 @@ The project follows [Semantic Versioning](https://semver.org/): patch = bug fixe ## [Unreleased] +Remote EA deployment: attach Expert Advisors to charts with set files over the +HTTP API, no RDP and no terminal restart. + +### Added + +- **Chart Deployments feature** (`mt5api/chartctl/`, `mt5api/handlers/chartctl.py`). Stage `.ex5`/`.set` artifacts, declare deployments (expert + set + symbol + timeframe) as desired state, and a resident loader EA reconciles the terminal's charts to it. New endpoints: `POST/GET/DELETE /experts`, `POST/GET /sets` + `GET /sets/`, `POST/GET /deployments` + `GET/PATCH/DELETE /deployments/`, `POST /deployments/reconcile`, `GET /charts`, `GET /loader`, `POST /charts//screenshot`. +- **Chart Control Protocol v1** — a file-based contract in `MQL5\Files\chartctl\` (`desired.json` / `observed.json` / command channel). Documented in `docs/chart-control-protocol.md`. Deployments only report `running` once the loader confirms the expert is live on a chart; drift and failures surface in `observed.json`. +- **Reference loader EA** `assets/experts/MT5ChartLoader.mq5` plus the portable include `assets/experts/include/ChartControl.mqh`, so an existing resident EA (e.g. an account tracker) can adopt the protocol with three calls instead of running a second EA. Single-loader mutex via a terminal GlobalVariable makes co-existence safe. +- Config block `chartctl:` in `config.yaml` (enable flag, reconcile hint, staleness window, command timeout, upload cap). Live-mode terminals only; per-terminal `chartctl: false` override. +- **Zero-touch loader bootstrap** — `scripts/compile-chartctl-loader.bat` auto-compiles the loader in every broker base on boot and propagates the `.ex5` to existing terminal instances; `config_helper.py write_ini` adds a `[StartUp] Expert=Advisors\MT5ChartLoader` section (honoring `symbol_suffix`) to live chartctl-enabled terminals, so the loader attaches itself at terminal launch. No RDP or manual attach anywhere in the deploy path. Duplicate loaders from re-fired `[StartUp]` lines self-close via the mutex. +- Tests: `tests/test_chartctl_units.py`, `tests/test_chartctl_endpoints.py`, and a Python `tests/chartctl_fake_loader.py` that plays the EA side of the protocol so the full endpoint suite runs on Linux with no MT5. +- **WebRequest allowlist provisioning** (`GET`/`PUT /webrequest`, `POST /webrequest/apply`; `mt5api/chartctl/webrequest.py`, `mt5api/chartctl/autoit_webrequest.py`, `mt5api/handlers/webrequest.py`). Set the terminal's `WebRequest()` allowed-URL list over the API instead of the Options dialog. A dedicated call (not a deployment field) since it's rarely needed. Two apply paths, chosen at runtime: + - **Windows VM (default here):** the allowlist is *not* stored in `common.ini` on this terminal build — it lives in the machine-bound `MQL5\experts.dat` and MT5 drops it on every restart. So it's set the way a user would: a bundled portable AutoIt interpreter (`assets/autoit/AutoIt3_x64.exe` + `set_webrequest.au3`) drives Tools → Options → Expert Advisors and types the URLs in. Takes effect immediately in-session (verified: a probe EA's `WebRequest()` returns HTTP 200 right after). Since MT5 forgets it on restart, `main.py` re-applies the persisted list automatically ~25 s after each terminal (re)start, surviving the periodic auto-reboot. + - **Bare metal:** where `common.ini` *is* the store, falls back to encoding the list into `common.ini` (`scripts/webrequest_allowlist_codec.py`, format reverse-engineered from `terminal64.exe`, verified byte-identical against 18 real broker blobs) + a terminal restart. + + Desired list persisted per terminal (`Config/webrequest.json`); first use migrates from the terminal's existing list, preserving manually-configured URLs. GUI applies are serialized host-wide by a named Windows kernel mutex (`Global\mt5_httpapi_webrequest_autoit`; crash-safe via `WAIT_ABANDONED`) since desktop focus is shared across terminals, and each terminal's boot re-apply is staggered by its port — so many terminals per VM can provision URLs without keystroke collisions. `inspect_options.au3`/`selftest.au3` ship as GUI-automation diagnostics (`POST /webrequest/apply?script=`). Tests: `tests/test_webrequest.py`. Live-mode chartctl terminals only. + +- `POST /charts//close` + a `close_chart` loader command — close any chart by id, including charts the loader cannot attribute to a deployment. The loader refuses to close its own chart. + +### Fixed + +- **WebRequest AutoIt apply drove the wrong terminal window (`options_not_found` / silent wrong-terminal writes).** The scripts matched the MT5 main window by title substring (the login), but `WinList()` also returns hidden windows and cloned terminals of the same account have byte-identical titles — so the apply could target the wrong terminal or fail to activate one at all. The API now resolves its terminal64.exe PID by executable path (WMI) and passes it to the scripts, which match only visible windows owned by that PID and verify the Options dialog belongs to it too, with activation retries. Legacy 3-arg script invocation still works (title fallback). + +- **Loader v1.0.2 — duplicate-chart leak across terminal restarts.** The `chartctl:` chart-comment stamp does not survive MT5's profile save/restore cycle, so every terminal restart (including the periodic auto-reboot) left the loader unable to recognize its own chart and it opened a fresh duplicate — accumulating until the terminal's chart cap. Reconcile now first **adopts** an unowned chart already running the deployment's exact expert + symbol + timeframe before opening a new one; comment stamps are verified by read-back (`ChartSetString` is async); a failed attach closes the chart it opened (previously an expert-less chart leaked per attempt) and backs off for 60 s; and errors are tracked per deployment instead of in a single shared slot (one deployment's failure no longer masks another's). + +### Notes + +- All chartctl handlers are **lock-free** — pure file I/O against the terminal data dir — so they never queue behind the process-wide MT5 SDK lock. +- Additive and **opt-in**: `chartctl.enabled` defaults to `false`. With the block + absent the API behaves exactly as before and no `[StartUp]` section is written. + Enabling it auto-attaches the loader EA to every live terminal, which is a + fleet-wide behaviour change and must never arrive by upgrade. + + ## [v4.12.2] — 2026-08-20 ### Fixed diff --git a/Dockerfile.test b/Dockerfile.test index f4f7445d..e7dcdfdd 100644 --- a/Dockerfile.test +++ b/Dockerfile.test @@ -25,7 +25,7 @@ COPY tests ./tests # docker-compose.yml.j2 is here because the compose-generation test renders the # REAL template — a stub would assert nothing about what actually ships. COPY requirements-api.txt requirements-mcpunifier.txt docker-compose.yml.example docker-compose.yml.j2 run.sh ./ -COPY scripts/config_helper.py scripts/start.bat scripts/check_health.py scripts/healthcheck.sh scripts/verify_binaries.py scripts/wickworks-healthcheck.py scripts/recreate-vm.sh ./scripts/ +COPY scripts/config_helper.py scripts/start.bat scripts/check_health.py scripts/healthcheck.sh scripts/verify_binaries.py scripts/wickworks-healthcheck.py scripts/recreate-vm.sh scripts/webrequest_allowlist_codec.py ./scripts/ COPY assets/binaries.lock.json ./assets/ ENV PYTHONPATH=/app diff --git a/README.md b/README.md index 7e70747d..4d4a624b 100644 --- a/README.md +++ b/README.md @@ -129,6 +129,7 @@ The old README became a massive wall of API shit, so the details now live in sep | Copy working curl and Go examples instead of guessing | [Clients and examples](docs/clients-and-examples.md) | | Operate the bastard: Make targets, ports, remote access, concurrency, and logs | [Operations](docs/operations.md) | | Split terminals across several Windows VMs or NUMA nodes | [Multi-VM setup](docs/multi-vm-setup.md) | +| Deploy EAs onto charts over HTTP instead of clicking through the Navigator | [Chart Deployments](docs/chart-deployments.md) | ## API at a glance diff --git a/assets/autoit/AutoIt3_x64.exe b/assets/autoit/AutoIt3_x64.exe new file mode 100644 index 00000000..43d81f5a Binary files /dev/null and b/assets/autoit/AutoIt3_x64.exe differ diff --git a/assets/autoit/EULA.htm b/assets/autoit/EULA.htm new file mode 100644 index 00000000..fb6130f4 --- /dev/null +++ b/assets/autoit/EULA.htm @@ -0,0 +1,49 @@ + + + + License + + + + +

Software License

+

AutoIt

+

Author : Jonathan Bennett and the AutoIt Team
+ WWW : https://www.autoitscript.com/site/autoit/
+ Email : support at autoitscript dot com
+ ________________________________________________________

+

END-USER LICENSE AGREEMENT FOR THIS SOFTWARE

+

This End-User License Agreement ("EULA") is a legal agreement between you (either an individual or a single entity) and the mentioned author of this Software for the software product identified above, which includes computer software and may include + associated media, printed materials, and "online" or electronic documentation ("SOFTWARE PRODUCT"). By installing, copying, or otherwise using the SOFTWARE PRODUCT, you agree to be bound by the terms of this EULA. If you do not agree to the terms of + this EULA, do not install or use the SOFTWARE PRODUCT.

+

 

+

SOFTWARE PRODUCT LICENSE

+

The SOFTWARE PRODUCT is protected by copyright laws and international copyright treaties, as well as other intellectual property laws and treaties. The SOFTWARE PRODUCT is licensed, not sold.

+

The definition of SOFTWARE PRODUCT does not includes any files generated by the SOFTWARE PRODUCT, such as compiled script files in the form of standalone executables.

+

1. GRANT OF LICENSE

+

This EULA grants you the following rights:

+

Installation and Use. You may install and use an unlimited number of copies of the SOFTWARE PRODUCT.

+

Reproduction and Distribution. You may reproduce and distribute an unlimited number of copies of the SOFTWARE PRODUCT either in whole or in part; each copy should include all copyright and trademark notices, and shall be accompanied by a copy of + this EULA. Copies of the SOFTWARE PRODUCT may be distributed as a standalone product or included with your own product.

+

Commercial Use. You may use the SOFTWARE PRODUCT for commercial purposes. You may sell for profit and freely distribute scripts and/or compiled scripts that were created with the SOFTWARE PRODUCT.

+

Reverse engineering. You may not reverse engineer or disassemble the SOFTWARE PRODUCT.

+

2. COPYRIGHT

+

All title and copyrights in and to the SOFTWARE PRODUCT (including but not limited to any images, photographs, animations, video, audio, music, text, and "applets" incorporated into the SOFTWARE PRODUCT), the accompanying printed materials, and any + copies of the SOFTWARE PRODUCT are owned by the Author of this Software. The SOFTWARE PRODUCT is protected by copyright laws and international treaty provisions. Therefore, you must treat the SOFTWARE PRODUCT like any other copyrighted material.

+

 

+

MISCELLANEOUS

+

If you acquired this product in the United Kingdom, this EULA is governed by the laws of the United Kingdom. If this product was acquired outside the United Kingdom, then local law may apply.

+

Should you have any questions concerning this EULA, or if you desire to contact the author of this Software for any reason, please contact him/her at the email address mentioned at the top of this EULA.

+

 

+

LIMITED WARRANTY

+

1. NO WARRANTIES

+

The Author of this Software expressly disclaims any warranty for the SOFTWARE PRODUCT. The SOFTWARE PRODUCT and any related documentation is provided "as is" without warranty of any kind, either express or implied, including, without limitation, the + implied warranties or merchantability, fitness for a particular purpose, or non-infringement. The entire risk arising out of use or performance of the SOFTWARE PRODUCT remains with you.

+

2. NO LIABILITY FOR DAMAGES

+

In no event shall the author of this Software be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or any other pecuniary loss) arising out of + the use of or inability to use this product, even if the Author of this Software has been advised of the possibility of such damages. Because some states/jurisdictions do not allow the exclusion or limitation of liability for consequential or + incidental damages, the above limitation may not apply to you.

+

 

+

[END OF LICENSE]

+ + diff --git a/assets/autoit/NOTICE.txt b/assets/autoit/NOTICE.txt new file mode 100644 index 00000000..87024c46 --- /dev/null +++ b/assets/autoit/NOTICE.txt @@ -0,0 +1,37 @@ +AutoIt — bundled copy, copyright and trademark notices +====================================================== + +This directory redistributes one unmodified component of AutoIt v3: + + AutoIt3_x64.exe version 3.3.18.0 + sha256: 5d69a932a077fee044b193c28e84564143f5c7e51079ab48e88fef74ab0b77b7 + byte-identical to the copy in AutoIt's official portable archive + (https://www.autoitscript.com/files/autoit3/autoit-v3.zip) + +Copyright notice (from the binary's own version resource): + + © 1999-2025 Jonathan Bennett & AutoIt Team + +Trademark notice (from https://www.autoitscript.com/site/): + + AutoIt is a trademark of AutoIt Consulting Ltd. + +Author : Jonathan Bennett and the AutoIt Team +WWW : https://www.autoitscript.com/site/autoit/ +Email : support at autoitscript dot com + +License +------- + +AutoIt is redistributed here under its End-User License Agreement, which +permits reproduction and distribution of copies "either in whole or in part" +provided that "each copy should include all copyright and trademark notices, +and shall be accompanied by a copy of this EULA". + +The unmodified official EULA accompanies this copy as EULA.htm in this +directory, as published by the AutoIt Team in the AutoIt v3 documentation +(https://www.autoitscript.com/autoit3/docs/license.htm, retrieved 2026-08-27, +matching the release the bundled binary ships in). + +The .au3 scripts in this directory are NOT part of AutoIt. They are this +repository's own automation, covered by this repository's license. diff --git a/assets/autoit/inspect_options.au3 b/assets/autoit/inspect_options.au3 new file mode 100644 index 00000000..83802798 --- /dev/null +++ b/assets/autoit/inspect_options.au3 @@ -0,0 +1,124 @@ +; inspect_options.au3 +; Non-destructive inspector for MT5's Tools->Options dialog. NO #includes +; (an undefined include function pops a blocking error dialog). Finds the MT5 +; window containing (the login), opens Options (Ctrl+O), walks +; every tab, and logs the VISIBLE controls (ClassNN, pos, text) so we can +; identify the Expert Advisors tab index + the WebRequest checkbox/list, then +; cancels with Esc. + +Opt("WinTitleMatchMode", 2) +Opt("SendKeyDelay", 15) + +Global $gLog = -1 + +Func LogW($s) + If $gLog <> -1 Then FileWrite($gLog, $s & @CRLF) +EndFunc + +Func DumpVisible($hWin, $tag) + LogW("--- " & $tag & " ---") + Local $cl = WinGetClassList($hWin) + Local $arr = StringSplit(StringStripCR($cl), @LF) + Local $seen = "|" + For $i = 1 To $arr[0] + Local $c = $arr[$i] + If $c = "" Then ContinueLoop + If StringInStr($seen, "|" & $c & "|") > 0 Then ContinueLoop + $seen &= $c & "|" + For $n = 1 To 80 + Local $ctrl = $c & $n + Local $pos = ControlGetPos($hWin, "", $ctrl) + If @error Then ExitLoop + Local $vis = ControlCommand($hWin, "", $ctrl, "IsVisible", "") + If $vis = 1 Then + Local $txt = ControlGetText($hWin, "", $ctrl) + LogW(" " & $ctrl _ + & " xywh=" & $pos[0] & "," & $pos[1] & "," & $pos[2] & "," & $pos[3] _ + & " text='" & StringLeft(StringReplace($txt, @CRLF, " "), 70) & "'") + EndIf + Next + Next +EndFunc + +; ---- args ---- +; 3 args: match, pid, logpath. 2 args (legacy): match, logpath. +If $CmdLine[0] < 2 Then Exit 10 +Global $match = $CmdLine[1] +Global $pid = 0 +Global $logpath +If $CmdLine[0] >= 3 Then + $pid = Int($CmdLine[2]) + $logpath = $CmdLine[3] +Else + $logpath = $CmdLine[2] +EndIf + +$gLog = FileOpen($logpath, 2) +If $gLog = -1 Then Exit 11 +LogW("=== inspect_options match=" & $match & " pid=" & $pid & " ===") + +; ---- find + activate MT5 (visible windows only; by owner pid when known — +; same-login terminal clones share the exact same window title) ---- +Local $wl = WinList() +Local $hMT5 = 0 +For $i = 1 To $wl[0][0] + Local $h = $wl[$i][1] + If $wl[$i][0] = "" Then ContinueLoop + If BitAND(WinGetState($h), 2) = 0 Then ContinueLoop + If $pid > 0 And WinGetProcess($h) <> $pid Then ContinueLoop + If StringInStr($wl[$i][0], $match) > 0 Then $hMT5 = $h +Next +If $hMT5 = 0 Then + LogW("RESULT=FAIL reason=mt5_window_not_found") + FileClose($gLog) + Exit 2 +EndIf +WinActivate($hMT5) +Sleep(600) +LogW("mt5='" & WinGetTitle($hMT5) & "' win_pid=" & WinGetProcess($hMT5)) + +; ---- open Options ---- +Send("^o") +Local $hOpt = WinWait("Options", "", 10) +If $hOpt <> 0 And $pid > 0 And WinGetProcess($hOpt) <> $pid Then $hOpt = 0 +If $hOpt = 0 Then + LogW("RESULT=FAIL reason=options_not_found") + FileClose($gLog) + Exit 3 +EndIf +WinActivate($hOpt) +Sleep(300) +LogW("options='" & WinGetTitle($hOpt) & "'") +LogW("classlist=" & StringReplace(WinGetClassList($hOpt), @LF, " | ")) + +; ---- find tab control, walk tabs ---- +Local $tabClass = "" +Local $cls = StringSplit(StringStripCR(WinGetClassList($hOpt)), @LF) +For $i = 1 To $cls[0] + If StringInStr($cls[$i], "SysTabControl32") > 0 Then + $tabClass = $cls[$i] + ExitLoop + EndIf +Next +LogW("tabClass='" & $tabClass & "'") + +If $tabClass <> "" Then + Local $cnt = ControlCommand($hOpt, "", $tabClass, "GetItemCount", "") + LogW("tab_count=" & $cnt) + If $cnt >= 1 Then + For $t = 1 To $cnt + ControlCommand($hOpt, "", $tabClass, "CurrentTab", $t) + Sleep(250) + DumpVisible($hOpt, "tab#" & $t) + Next + Else + DumpVisible($hOpt, "single") + EndIf +Else + DumpVisible($hOpt, "no-tabctl") +EndIf + +LogW("RESULT=OK") +Send("{ESC}") +FileClose($gLog) +Exit 0 diff --git a/assets/autoit/selftest.au3 b/assets/autoit/selftest.au3 new file mode 100644 index 00000000..f544dc13 --- /dev/null +++ b/assets/autoit/selftest.au3 @@ -0,0 +1,60 @@ +; selftest.au3 [] +; Minimal, NO external includes. Confirms AutoIt runs at all, reports whether +; it's elevated, and whether it can find + drive the (elevated) MT5 window. +; With 3 args the second is the terminal64.exe pid (0 = unknown) — same-login +; terminal clones share the exact same window title, so only the pid pins one. + +Opt("WinTitleMatchMode", 2) +Opt("SendKeyDelay", 15) + +Local $match = ($CmdLine[0] >= 1) ? $CmdLine[1] : "?" +Local $pid = 0 +Local $logpath = @ScriptDir & "\selftest.log" +If $CmdLine[0] >= 3 Then + $pid = Int($CmdLine[2]) + $logpath = $CmdLine[3] +ElseIf $CmdLine[0] = 2 Then + $logpath = $CmdLine[2] +EndIf + +Local $h = FileOpen($logpath, 2) +If $h = -1 Then Exit 11 +FileWrite($h, "started args=" & $CmdLine[0] & " match=" & $match & " pid=" & $pid & @CRLF) +FileWrite($h, "IsAdmin=" & IsAdmin() & @CRLF) + +Local $wl = WinList() +FileWrite($h, "windows_total=" & $wl[0][0] & @CRLF) +Local $hMT5 = 0 +For $i = 1 To $wl[0][0] + If $wl[$i][0] <> "" And StringInStr($wl[$i][0], $match) > 0 Then + FileWrite($h, " match_win='" & $wl[$i][0] & "' visible=" _ + & (BitAND(WinGetState($wl[$i][1]), 2) > 0 ? 1 : 0) _ + & " pid=" & WinGetProcess($wl[$i][1]) & @CRLF) + If BitAND(WinGetState($wl[$i][1]), 2) = 0 Then ContinueLoop + If $pid > 0 And WinGetProcess($wl[$i][1]) <> $pid Then ContinueLoop + $hMT5 = $wl[$i][1] + EndIf +Next +If $hMT5 = 0 Then + FileWrite($h, "RESULT=FAIL reason=no_mt5_window" & @CRLF) + FileClose($h) + Exit 2 +EndIf + +WinActivate($hMT5) +Sleep(600) +FileWrite($h, "active_title='" & WinGetTitle("[ACTIVE]") & "'" & @CRLF) + +Send("^o") +Local $hOpt = WinWait("Options", "", 8) +If $hOpt <> 0 And $pid > 0 And WinGetProcess($hOpt) <> $pid Then $hOpt = 0 +If $hOpt = 0 Then + FileWrite($h, "RESULT=FAIL reason=options_did_not_open (input blocked? UIPI/elevation)" & @CRLF) + FileClose($h) + Exit 3 +EndIf +FileWrite($h, "options_opened title='" & WinGetTitle($hOpt) & "'" & @CRLF) +Send("{ESC}") +FileWrite($h, "RESULT=OK" & @CRLF) +FileClose($h) +Exit 0 diff --git a/assets/autoit/set_webrequest.au3 b/assets/autoit/set_webrequest.au3 new file mode 100644 index 00000000..8d97db32 --- /dev/null +++ b/assets/autoit/set_webrequest.au3 @@ -0,0 +1,232 @@ +; set_webrequest.au3 [] +; Adds the URLs in (one per line) to MT5's Tools->Options->Expert +; Advisors "Allow WebRequest for listed URL" list, for the terminal whose +; window title contains (the login). NO #includes (an undefined +; include function pops a blocking error dialog). +; +; (optional, 0 = unknown) is the terminal64.exe process id. Cloned +; terminals of the same account have IDENTICAL window titles, so the login is +; ambiguous — the pid is the only thing that pins the right terminal. When +; given, both the main window and the Options dialog are matched by owner pid. +; +; MT5's URL list is a SysListView32 with a greyed "add new URL like ..." row at +; the bottom; double-clicking it opens an inline edit. We type the URL + Enter, +; which commits and produces a fresh add-row below. Then OK. + +Opt("WinTitleMatchMode", 2) +Opt("SendKeyDelay", 20) +Opt("SendKeyDownDelay", 5) +Opt("MouseCoordMode", 2) ; coords relative to the control for ControlClick + +Global $gLog = -1 +Func LogW($s) + If $gLog <> -1 Then FileWrite($gLog, $s & @CRLF) +EndFunc + +; Scan Button1..25 for the first VISIBLE control whose text contains $needle. +Func FindButtonByText($hWin, $needle) + For $n = 1 To 25 + Local $ctrl = "Button" & $n + ControlGetPos($hWin, "", $ctrl) + If @error Then ContinueLoop + If ControlCommand($hWin, "", $ctrl, "IsVisible", "") <> 1 Then ContinueLoop + If StringInStr(ControlGetText($hWin, "", $ctrl), $needle) > 0 Then Return $ctrl + Next + Return "" +EndFunc + +Func DumpControls($hWin, $tag) + LogW(" [" & $tag & "] classlist=" & StringReplace(WinGetClassList($hWin), @LF, " | ")) +EndFunc + +; Log the current listview rows (so we can verify what MT5 actually holds). +Func DumpList($hWin, $tag) + Local $cnt = ControlListView($hWin, "", "SysListView321", "GetItemCount") + Local $s = "" + For $r = 0 To $cnt - 1 + $s &= "[" & ControlListView($hWin, "", "SysListView321", "GetText", $r, 0) & "] " + Next + LogW(" list(" & $tag & ") count=" & $cnt & " items=" & $s) + Return $cnt +EndFunc + +; Visible top-level window for this terminal: by owner pid when known +; (title as tie-break among the pid's windows), else by title substring. +; WinList() alone is NOT enough — it returns hidden windows, and same-login +; terminal clones share the exact same title. +Func FindMainWindow($match, $pid) + Local $wl = WinList() + Local $best = 0 + For $i = 1 To $wl[0][0] + Local $h = $wl[$i][1] + Local $title = $wl[$i][0] + If $title = "" Then ContinueLoop + If BitAND(WinGetState($h), 2) = 0 Then ContinueLoop ; visible only + If $pid > 0 Then + If WinGetProcess($h) <> $pid Then ContinueLoop + If StringInStr($title, $match) > 0 Then Return $h + If $best = 0 Then $best = $h + Else + If StringInStr($title, $match) > 0 Then $best = $h + EndIf + Next + Return $best +EndFunc + +; Focus the terminal and open Tools->Options (Ctrl+O), retrying, and only +; accept an Options window owned by our pid (another terminal's dialog, or +; any window with "Options" in its title, must not be driven). +Func OpenOptions($hMT5, $pid) + For $try = 1 To 3 + WinActivate($hMT5) + If WinWaitActive($hMT5, "", 3) = 0 Then + LogW(" activate attempt " & $try & " failed (active='" & WinGetTitle("[ACTIVE]") & "')") + ContinueLoop + EndIf + Send("^o") + Local $t = TimerInit() + While TimerDiff($t) < 5000 + Local $hOpt = WinWait("Options", "", 1) + If $hOpt <> 0 Then + If $pid = 0 Or WinGetProcess($hOpt) = $pid Then Return $hOpt + LogW(" ignoring foreign Options window (pid=" & WinGetProcess($hOpt) & ")") + EndIf + WEnd + LogW(" ctrl+o attempt " & $try & ": no Options dialog") + Next + Return 0 +EndFunc + +; ---- args ---- +; 4 args: match, pid, urlfile, logpath. 3 args (legacy caller): match, +; urlfile, logpath with pid unknown. +If $CmdLine[0] < 3 Then Exit 10 +Global $match = $CmdLine[1] +Global $pid = 0 +Global $urlfile, $logpath +If $CmdLine[0] >= 4 Then + $pid = Int($CmdLine[2]) + $urlfile = $CmdLine[3] + $logpath = $CmdLine[4] +Else + $urlfile = $CmdLine[2] + $logpath = $CmdLine[3] +EndIf + +$gLog = FileOpen($logpath, 2) +If $gLog = -1 Then Exit 11 +LogW("=== set_webrequest match=" & $match & " pid=" & $pid & " ===") + +; ---- read urls ---- +Global $raw = FileRead($urlfile) +Global $urls = StringSplit(StringStripCR($raw), @LF) +Local $n_urls = 0 +For $i = 1 To $urls[0] + If StringStripWS($urls[$i], 3) <> "" Then $n_urls += 1 +Next +LogW("urls_in_file=" & $n_urls) + +; ---- find + activate MT5 ---- +Local $hMT5 = FindMainWindow($match, $pid) +If $hMT5 = 0 Then + LogW("RESULT=FAIL reason=mt5_window_not_found") + FileClose($gLog) + Exit 2 +EndIf +LogW("mt5='" & WinGetTitle($hMT5) & "' win_pid=" & WinGetProcess($hMT5)) + +; ---- open Options ---- +Local $hOpt = OpenOptions($hMT5, $pid) +If $hOpt = 0 Then + LogW("RESULT=FAIL reason=options_not_found") + FileClose($gLog) + Exit 3 +EndIf +WinActivate($hOpt) +Sleep(400) + +; ---- ensure the Expert Advisors tab is active (WebRequest checkbox present) ---- +Local $cb = FindButtonByText($hOpt, "WebRequest") +Local $tries = 0 +While $cb = "" And $tries < 12 + Send("^{TAB}") ; cycle property-sheet tabs + Sleep(250) + $cb = FindButtonByText($hOpt, "WebRequest") + $tries += 1 +WEnd +If $cb = "" Then + LogW("RESULT=FAIL reason=webrequest_checkbox_not_found") + Send("{ESC}") + FileClose($gLog) + Exit 4 +EndIf +LogW("checkbox=" & $cb & " text='" & ControlGetText($hOpt, "", $cb) & "'") + +; ---- ensure the checkbox is checked (real click triggers MT5's enable logic) ---- +If ControlCommand($hOpt, "", $cb, "IsChecked", "") <> 1 Then + ControlClick($hOpt, "", $cb) + Sleep(300) + LogW("checkbox now checked=" & ControlCommand($hOpt, "", $cb, "IsChecked", "")) +Else + LogW("checkbox already checked") +EndIf + +; ---- the URL list ---- +Local $lp = ControlGetPos($hOpt, "", "SysListView321") +If @error Then + LogW("RESULT=FAIL reason=listview_not_found") + Send("{ESC}") + FileClose($gLog) + Exit 5 +EndIf +LogW("list xywh=" & $lp[0] & "," & $lp[1] & "," & $lp[2] & "," & $lp[3]) +Local $rowH = 17 + +; ---- clear existing entries (a PUT sets the full list) ---- +; Select the first data row and press Delete, repeatedly. The greyed "add new +; URL" row can't be deleted, so extra iterations are harmless no-ops. +DumpList($hOpt, "before-clear") +For $k = 1 To 40 + Local $before = ControlListView($hOpt, "", "SysListView321", "GetItemCount") + If $before <= 0 Then ExitLoop + ControlClick($hOpt, "", "SysListView321", "left", 1, Int($lp[2] / 2), 10) + Sleep(60) + Send("{DELETE}") + Sleep(90) + If ControlListView($hOpt, "", "SysListView321", "GetItemCount") >= $before Then ExitLoop +Next +DumpList($hOpt, "after-clear") + +; ---- add each url ---- +Local $added = 0 +Local $idx = 0 +For $i = 1 To $urls[0] + Local $u = StringStripWS($urls[$i], 3) + If $u = "" Then ContinueLoop + ; add-row is the last row: relative Y grows by rowH per existing entry + Local $ry = 10 + ($idx * $rowH) + If $ry > $lp[3] - 4 Then $ry = $lp[3] - 8 ; clamp into the control + ControlClick($hOpt, "", "SysListView321", "left", 2, Int($lp[2] / 2), $ry) + Sleep(250) + If $i = 1 Then DumpControls($hOpt, "after-first-dblclick") + ; type the URL into whatever inline edit appeared, then commit + Send("^a") ; select any placeholder text + Send($u, 1) ; raw send (URLs contain / : . which are literal in raw mode) + Sleep(120) + Send("{ENTER}") + Sleep(300) + LogW("typed[" & $idx & "] ry=" & $ry & " url=" & $u) + $added += 1 + $idx += 1 +Next +DumpList($hOpt, "final") + +; ---- confirm with OK ---- +Local $ok = FindButtonByText($hOpt, "OK") +If $ok = "" Then $ok = "Button8" +ControlClick($hOpt, "", $ok) +Sleep(400) + +LogW("RESULT=OK added=" & $added) +FileClose($gLog) +Exit 0 diff --git a/assets/binaries.lock.json b/assets/binaries.lock.json index aacddfdc..092a711a 100644 --- a/assets/binaries.lock.json +++ b/assets/binaries.lock.json @@ -21,6 +21,17 @@ "source": "https://www.sordum.org/downloads/?power-run=", "signature": "malformed", "note": "REPACKED, NOT PRISTINE. Arrived vendored inside the defender-remover toolkit rather than from Sordum directly. Its certificate directory is not a well-formed WIN_CERTIFICATE (declared length 776284822, revision 0xc496, type 14951 against the required 0x200/2) and 512 bytes trail it, so the Authenticode signature cannot validate. Its hash matches no Sordum release: official v1.6.0.0 x64 is da77bc401ef0d7b8e23be3a9387660172aea176cd9d1248034130811d29942c9 (934464 bytes) and current v1.9.0.0 x64 is b305126c2881073a53073bd1782d0649764d198562e02fc02395fc07637dcec5. Not known to be malicious — repacking is normal for that toolkit — but it is unverifiable. Replacing it with an official Sordum download would let this entry move to signature=valid." + }, + { + "path": "assets/autoit/AutoIt3_x64.exe", + "sha256": "5d69a932a077fee044b193c28e84564143f5c7e51079ab48e88fef74ab0b77b7", + "size": 1107552, + "product": "AutoIt v3 Script (portable interpreter)", + "version": "3.3.18.0", + "vendor": "AutoIt Team", + "source": "https://www.autoitscript.com/site/autoit/downloads/", + "signature": "valid", + "note": "Bundled because this terminal build keeps the WebRequest allowlist in machine-bound MQL5\\\\experts.dat rather than common.ini, so the list can only be set by driving Tools > Options > Expert Advisors. Only AutoIt3_x64.exe is vendored - no installer, no Aut2Exe, no UDFs - and it runs only the .au3 scripts in this repo. Authenticode signature present and validating. AutoIt publishes no per-file sha256, but the file has since been verified BYTE-IDENTICAL (same sha256 as above) to AutoIt3_x64.exe inside the official portable archive at autoitscript.com/files/autoit3/autoit-v3.zip - independently by the upstream reviewer and by this repo's maintainer - in addition to the validating Authenticode signature. Redistribution terms: see EULA.htm and NOTICE.txt beside the binary (the EULA requires both to accompany every copy)." } ] } diff --git a/assets/experts/MT5ChartLoader.mq5 b/assets/experts/MT5ChartLoader.mq5 new file mode 100644 index 00000000..b8c109b9 --- /dev/null +++ b/assets/experts/MT5ChartLoader.mq5 @@ -0,0 +1,62 @@ +//+------------------------------------------------------------------+ +//| MT5ChartLoader.mq5 | +//| Reference chart-deployment loader for mt5-httpapi chartctl. | +//| | +//| Attach this to ANY single chart in the terminal (it doesn't | +//| matter which symbol/timeframe — it manages other charts, not | +//| its own). It reconciles the terminal to the desired deployments | +//| written by the API and reports live state back. | +//| | +//| It never trades. It only opens/closes charts and applies | +//| templates. Safe to run on a live account. | +//| | +//| This EA is intentionally thin: all logic lives in the portable | +//| include ChartControl.mqh so you can drop the same capability | +//| into your own resident EA (e.g. an account tracker) instead of | +//| running this standalone. See docs/chart-control-protocol.md. | +//+------------------------------------------------------------------+ +#property copyright "mt5-httpapi" +#property link "https://github.com/psyb0t/mt5-httpapi" +#property version "1.00" +#property strict + +#include + +input int InpLoopSeconds = 1; // reconcile timer period (seconds) + +CChartControl ctl; + +//+------------------------------------------------------------------+ +int OnInit() +{ + // true: if another loader already owns the terminal, close this chart + // and vanish (mt5start.ini [StartUp] re-attaches us on every launch; + // this keeps that idempotent instead of accumulating loader charts). + if(!ctl.Init(true)) + return INIT_FAILED; + EventSetTimer(InpLoopSeconds < 1 ? 1 : InpLoopSeconds); + Comment("MT5ChartLoader active — chartctl loader\n", + ctl.IsOwner() ? "role: OWNER" : "role: passive (another loader owns the mutex)"); + return INIT_SUCCEEDED; +} + +//+------------------------------------------------------------------+ +void OnTimer() +{ + ctl.Tick(); +} + +//+------------------------------------------------------------------+ +void OnDeinit(const int reason) +{ + EventKillTimer(); + ctl.Deinit(); + Comment(""); +} + +//+------------------------------------------------------------------+ +//| No trading. OnTick is intentionally empty; the loader is timer- | +//| driven so it works on weekends and on disconnected symbols. | +//+------------------------------------------------------------------+ +void OnTick() {} +//+------------------------------------------------------------------+ diff --git a/assets/experts/include/ChartControl.mqh b/assets/experts/include/ChartControl.mqh new file mode 100644 index 00000000..a2df55c0 --- /dev/null +++ b/assets/experts/include/ChartControl.mqh @@ -0,0 +1,787 @@ +//+------------------------------------------------------------------+ +//| ChartControl.mqh | +//| Chart Control Protocol v1 — reference implementation | +//| | +//| Drop this into a resident EA to make it the terminal's chart | +//| deployment loader. It reconciles the terminal's charts to the | +//| desired state written by mt5-httpapi's chartctl endpoints. | +//| | +//| Usage inside your EA: | +//| #include | +//| CChartControl ctl; | +//| int OnInit(){ if(!ctl.Init()) return INIT_FAILED; | +//| EventSetTimer(1); return INIT_SUCCEEDED; } | +//| void OnTimer(){ ctl.Tick(); } | +//| void OnDeinit(const int r){ ctl.Deinit(); } | +//| | +//| Contract & file formats: docs/chart-control-protocol.md | +//+------------------------------------------------------------------+ +#property strict + +#define CHARTCTL_PROTOCOL 1 +#define CHARTCTL_VERSION "1.0.2" +#define CHARTCTL_DIR "chartctl" // under MQL5\Files\ +#define CHARTCTL_MUTEX_GV "chartctl_loader_owner" +#define CHARTCTL_ID_INPUT "__chartctl_id" + +//--- one desired deployment +struct ChartCtlDeployment +{ + string id; + string expert; // expert short name (CHART_EXPERT_NAME match target) + string templ; // template path for ChartApplyTemplate; leading \ = relative to \MQL5 (e.g. \Files\chartctl\dep_x.tpl) + string symbol; + string timeframe; + bool enabled; +}; + +//--- what we observed on one chart +struct ChartCtlChart +{ + long chart_id; + string symbol; + string timeframe; + string expert; + bool expert_enabled; + string deployment_id; // parsed from the chart's __chartctl_id if present +}; + +//+------------------------------------------------------------------+ +class CChartControl +{ +private: + bool m_owner; // did we win the single-loader mutex? + long m_applied_revision; // last desired revision we reconciled + long m_last_revision_seen; + datetime m_started; + // Per-deployment error slots (parallel arrays keyed by deployment id). + // A single shared slot let one deployment's error mask another's. + string m_err_ids[]; + string m_err_codes[]; + string m_err_details[]; + datetime m_err_times[]; // drives the failed-attach retry cooldown + + //--- file helpers ------------------------------------------------- + bool ReadFile(const string relpath, string &out); + bool WriteFileAtomic(const string relpath, const string content); + void DeleteFileSafe(const string relpath); + + //--- json (minimal, tailored to our own compact output) ---------- + string JsonStr(const string key, const string s, const string json); + long JsonNum(const string key, const string json); + bool ExtractDeployments(const string json, ChartCtlDeployment &out[]); + string JsonEscape(const string s); + + //--- reconcile ---------------------------------------------------- + void ScanCharts(ChartCtlChart &out[]); + long FindChartFor(const string dep_id, ChartCtlChart &charts[]); + long FindAdoptableChart(const ChartCtlDeployment &dep, ChartCtlChart &charts[]); + bool StampChart(const long cid, const string dep_id); + bool AttachDeployment(const ChartCtlDeployment &dep); + void DetachChart(const long chart_id); + ENUM_TIMEFRAMES TF(const string s); + void RecordError(const string id, const string code, const string detail); + void ClearError(const string id); + bool InRetryCooldown(const string id); + + //--- observed + command output ----------------------------------- + void WriteObserved(ChartCtlDeployment &desired[], ChartCtlChart &charts[]); + void HandleCommand(); + +public: + CChartControl(void); + // close_own_chart_on_duplicate: standalone loaders pass true so a + // second copy (e.g. re-fired by mt5start.ini [StartUp] on every + // launch) closes its own chart and vanishes instead of idling. + // EAs that embed this module MUST leave it false — closing the chart + // would kill the host EA (e.g. an account tracker) too. + bool Init(const bool close_own_chart_on_duplicate=false); + void Tick(void); + void Deinit(void); + bool IsOwner(void) const { return m_owner; } +}; + +//+------------------------------------------------------------------+ +CChartControl::CChartControl(void) +{ + m_owner = false; + m_applied_revision = -1; + m_last_revision_seen = -1; + m_started = 0; +} + +//+------------------------------------------------------------------+ +//| Claim the single-loader mutex via a terminal GlobalVariable. | +//+------------------------------------------------------------------+ +bool CChartControl::Init(const bool close_own_chart_on_duplicate) +{ + m_started = TimeCurrent(); + + // If another loader already holds the mutex and is fresh, step aside. + if(GlobalVariableCheck(CHARTCTL_MUTEX_GV)) + { + datetime held = (datetime)GlobalVariableGet(CHARTCTL_MUTEX_GV); + // Treat a mutex touched within 120s as a live owner. + if(TimeCurrent() - held < 120) + { + m_owner = false; + if(close_own_chart_on_duplicate) + { + // Standalone duplicate (e.g. [StartUp] re-fired on relaunch): + // remove ourselves entirely so charts never accumulate. + Print("ChartControl: live owner exists; closing own chart."); + ChartClose(ChartID()); + return true; // unloading anyway; don't fail the host + } + Print("ChartControl: another loader owns the mutex; standing down."); + return true; // do NOT fail the host EA — just stay passive + } + } + GlobalVariableSet(CHARTCTL_MUTEX_GV, (double)TimeCurrent()); + GlobalVariableTemp(CHARTCTL_MUTEX_GV); // auto-clears if terminal exits + m_owner = true; + PrintFormat("ChartControl v%s active (owner). dir=MQL5\\Files\\%s", + CHARTCTL_VERSION, CHARTCTL_DIR); + return true; +} + +//+------------------------------------------------------------------+ +void CChartControl::Tick(void) +{ + if(!m_owner) + { + // Passive mode: try to reclaim if the previous owner is gone. + if(!GlobalVariableCheck(CHARTCTL_MUTEX_GV)) + Init(); + return; + } + + // Refresh mutex heartbeat. + GlobalVariableSet(CHARTCTL_MUTEX_GV, (double)TimeCurrent()); + + // Always answer commands (screenshot etc.), even without desired change. + HandleCommand(); + + string desired_json; + ChartCtlDeployment desired[]; + if(ReadFile(CHARTCTL_DIR + "\\desired.json", desired_json)) + { + long rev = JsonNum("revision", desired_json); + ExtractDeployments(desired_json, desired); + + // Reconcile every pass (cheap) — attach missing, detach orphans. + ChartCtlChart charts[]; + ScanCharts(charts); + + // 1) Attach / repair enabled deployments. + for(int i = 0; i < ArraySize(desired); i++) + { + if(!desired[i].enabled) + continue; + long cid = FindChartFor(desired[i].id, charts); + if(cid >= 0) + continue; + // Adopt before opening: an unowned chart already running this + // exact expert/symbol/timeframe is almost certainly a previous + // incarnation of this deployment whose comment stamp was lost + // (comments do NOT reliably survive terminal restarts). Claiming + // it instead of opening a fresh chart is what stops duplicates + // from accumulating one-per-reboot. + cid = FindAdoptableChart(desired[i], charts); + if(cid >= 0) + { + if(StampChart(cid, desired[i].id)) + { + ClearError(desired[i].id); + PrintFormat("ChartControl: adopted chart %I64d for %s (%s %s)", + cid, desired[i].id, desired[i].symbol, + desired[i].timeframe); + } + else + RecordError(desired[i].id, "STAMP_FAILED", + "adoption stamp on chart " + + IntegerToString(cid) + " did not read back"); + continue; + } + if(InRetryCooldown(desired[i].id)) + continue; // recent failure — don't hammer ChartOpen every pass + AttachDeployment(desired[i]); + } + + // 2) Detach charts we own whose deployment is gone or disabled. + ScanCharts(charts); // rescan after possible attaches + for(int c = 0; c < ArraySize(charts); c++) + { + if(charts[c].deployment_id == "") + continue; // not ours — never touch + bool wanted = false; + for(int d = 0; d < ArraySize(desired); d++) + if(desired[d].id == charts[c].deployment_id && desired[d].enabled) + { wanted = true; break; } + if(!wanted) + DetachChart(charts[c].chart_id); + } + + m_applied_revision = rev; + ScanCharts(charts); + WriteObserved(desired, charts); + } + else + { + // No desired file yet — still publish liveness + inventory. + ChartCtlChart charts[]; + ScanCharts(charts); + ChartCtlDeployment none[]; + WriteObserved(none, charts); + } +} + +//+------------------------------------------------------------------+ +void CChartControl::Deinit(void) +{ + if(m_owner && GlobalVariableCheck(CHARTCTL_MUTEX_GV)) + GlobalVariableDel(CHARTCTL_MUTEX_GV); +} + +//+------------------------------------------------------------------+ +//| Attach: select symbol, open chart, apply template, verify. | +//+------------------------------------------------------------------+ +bool CChartControl::AttachDeployment(const ChartCtlDeployment &dep) +{ + if(!SymbolSelect(dep.symbol, true)) + { + RecordError(dep.id, "SYMBOL_NOT_FOUND", + "SymbolSelect failed for " + dep.symbol); + return false; + } + + long cid = ChartOpen(dep.symbol, TF(dep.timeframe)); + if(cid == 0) + { + RecordError(dep.id, "CHART_OPEN_FAILED", + "ChartOpen failed err=" + IntegerToString(GetLastError())); + return false; + } + + if(!ChartApplyTemplate(cid, dep.templ)) + { + RecordError(dep.id, "TEMPLATE_APPLY_FAILED", + "ChartApplyTemplate(" + dep.templ + ") err=" + + IntegerToString(GetLastError())); + ChartClose(cid); + return false; + } + + // Verify the expert actually attached within ~10s. + for(int i = 0; i < 40; i++) + { + ChartRedraw(cid); + // CHART_EXPERT_NAME is NULL (not "") when no expert is attached, + // and NULL != "" is true in MQL5 — test length, or the very first + // iteration false-passes and an expert-less chart reports running. + string en = ChartGetString(cid, CHART_EXPERT_NAME); + if(StringLen(en) > 0) + { + if(!StampChart(cid, dep.id)) + { + // Without the stamp we could never re-identify the chart and + // would open a duplicate next pass — better to fail visibly. + RecordError(dep.id, "STAMP_FAILED", + "expert attached but CHART_COMMENT stamp did not " + "read back; closing chart"); + ChartClose(cid); + return false; + } + ClearError(dep.id); + PrintFormat("ChartControl: attached %s on %s %s (chart %I64d)", + en, dep.symbol, dep.timeframe, cid); + return true; + } + Sleep(250); + } + + // Leaving the chart open here leaks an expert-less chart per pass (the + // expert may still load later, but then adoption reclaims a closed-and- + // reopened one just as well). Close what we opened. + ChartClose(cid); + RecordError(dep.id, "EXPERT_NOT_ATTACHED", + "template applied but CHART_EXPERT_NAME empty after 10s; " + "GetLastError=" + IntegerToString(GetLastError())); + return false; +} + +//+------------------------------------------------------------------+ +//| Stamp attribution into the chart comment and verify it stuck. | +//| ChartSetString is asynchronous — the write is only queued — so | +//| read it back (with retries) before trusting it. | +//+------------------------------------------------------------------+ +bool CChartControl::StampChart(const long cid, const string dep_id) +{ + string want = "chartctl:" + dep_id; + for(int i = 0; i < 12; i++) + { + ChartSetString(cid, CHART_COMMENT, want); + ChartRedraw(cid); + Sleep(250); + if(ChartGetString(cid, CHART_COMMENT) == want) + return true; + } + PrintFormat("ChartControl: CHART_COMMENT stamp failed on chart %I64d (%s)", + cid, dep_id); + return false; +} + +//+------------------------------------------------------------------+ +//| An unowned chart matching a deployment's expert+symbol+timeframe | +//| (a prior incarnation whose stamp was lost, or a verify-timeout | +//| chart whose expert loaded late). | +//+------------------------------------------------------------------+ +long CChartControl::FindAdoptableChart(const ChartCtlDeployment &dep, + ChartCtlChart &charts[]) +{ + for(int i = 0; i < ArraySize(charts); i++) + { + if(charts[i].deployment_id != "") + continue; // owned by another deployment + if(!charts[i].expert_enabled) + continue; + if(charts[i].expert != dep.expert) + continue; + if(charts[i].symbol != dep.symbol) + continue; + if(charts[i].timeframe != "PERIOD_" + dep.timeframe) + continue; + return charts[i].chart_id; + } + return -1; +} + +//+------------------------------------------------------------------+ +void CChartControl::DetachChart(const long chart_id) +{ + PrintFormat("ChartControl: detaching chart %I64d", chart_id); + ChartClose(chart_id); +} + +//+------------------------------------------------------------------+ +//| Enumerate all open charts and classify ownership by the | +//| __chartctl_id we baked into each deployment template. | +//+------------------------------------------------------------------+ +void CChartControl::ScanCharts(ChartCtlChart &out[]) +{ + ArrayResize(out, 0); + long cid = ChartFirst(); + int guard = 0; + while(cid >= 0 && guard < 1000) + { + guard++; + ChartCtlChart c; + c.chart_id = cid; + c.symbol = ChartSymbol(cid); + c.timeframe = EnumToString(ChartPeriod(cid)); + c.expert = ChartGetString(cid, CHART_EXPERT_NAME); + c.expert_enabled = (c.expert != ""); + c.deployment_id = ""; // attribution below + + // Attribution is by the chart comment we set at attach time + // (ChartSetString CHART_COMMENT = "chartctl:"). We cannot read + // a foreign expert's inputs from MQL5, which is why the comment — + // not the template's __chartctl_id input — is the marker. The + // comment does NOT reliably survive a terminal restart (observed + // live 2026-07-16: one duplicate chart accumulated per reboot), so + // reconcile also adopts unowned exact-match charts (see + // FindAdoptableChart) instead of trusting this alone. + string cmt = ChartGetString(cid, CHART_COMMENT); + int p = StringFind(cmt, "chartctl:"); + if(p >= 0) + c.deployment_id = StringSubstr(cmt, p + 9); + + int n = ArraySize(out); + ArrayResize(out, n + 1); + out[n] = c; + + cid = ChartNext(cid); + } +} + +//+------------------------------------------------------------------+ +long CChartControl::FindChartFor(const string dep_id, ChartCtlChart &charts[]) +{ + for(int i = 0; i < ArraySize(charts); i++) + if(charts[i].deployment_id == dep_id && charts[i].expert_enabled) + return charts[i].chart_id; + return -1; +} + +//+------------------------------------------------------------------+ +//| Command channel: one-shot ops that produce artifacts. | +//+------------------------------------------------------------------+ +void CChartControl::HandleCommand(void) +{ + string body; + if(!ReadFile(CHARTCTL_DIR + "\\command.json", body)) + return; + + string cmd_id = JsonStr("command_id", "", body); + string action = JsonStr("action", "", body); + if(cmd_id == "") + { + DeleteFileSafe(CHARTCTL_DIR + "\\command.json"); + return; + } + + string result = "{"; + result += "\"command_id\":\"" + JsonEscape(cmd_id) + "\","; + + if(action == "screenshot") + { + long cid = JsonNum("chart_id", body); + int w = (int)JsonNum("width", body); if(w <= 0) w = 1280; + int h = (int)JsonNum("height", body); if(h <= 0) h = 720; + string fname = "shots\\" + cmd_id + ".png"; + // ChartScreenShot writes under MQL5\Files\. + if(ChartScreenShot((long)cid, CHARTCTL_DIR + "\\" + fname, w, h)) + result += "\"status\":\"ok\",\"file\":\"" + cmd_id + ".png\""; + else + result += "\"status\":\"error\",\"error_code\":\"SCREENSHOT_FAILED\"," + + "\"error_detail\":\"err=" + + IntegerToString(GetLastError()) + "\""; + } + else if(action == "reconcile") + { + m_applied_revision = -1; // force a full reconcile next pass + result += "\"status\":\"ok\""; + } + else if(action == "close_chart") + { + long cid = JsonNum("chart_id", body); + if(cid == ChartID()) + result += "\"status\":\"error\",\"error_code\":\"CLOSE_REFUSED\"," + + "\"error_detail\":\"refusing to close the loader's own chart\""; + else if(ChartClose(cid)) + result += "\"status\":\"ok\""; + else + result += "\"status\":\"error\",\"error_code\":\"CLOSE_FAILED\"," + + "\"error_detail\":\"err=" + + IntegerToString(GetLastError()) + "\""; + } + else + { + result += "\"status\":\"error\",\"error_code\":\"UNKNOWN_ACTION\"," + + "\"error_detail\":\"" + JsonEscape(action) + "\""; + } + result += "}"; + + WriteFileAtomic(CHARTCTL_DIR + "\\command_result.json", result); + DeleteFileSafe(CHARTCTL_DIR + "\\command.json"); +} + +//+------------------------------------------------------------------+ +//| Write observed.json — the API's window into terminal truth. | +//+------------------------------------------------------------------+ +void CChartControl::WriteObserved(ChartCtlDeployment &desired[], + ChartCtlChart &charts[]) +{ + string j = "{"; + j += "\"protocol\":" + IntegerToString(CHARTCTL_PROTOCOL) + ","; + j += "\"loader\":{"; + j += "\"name\":\"" + JsonEscape(MQLInfoString(MQL_PROGRAM_NAME)) + "\","; + j += "\"version\":\"" + CHARTCTL_VERSION + "\","; + j += "\"last_loop\":\"" + TimeToString(TimeGMT(), TIME_DATE|TIME_SECONDS) + "\","; + j += "\"applied_revision\":" + IntegerToString(m_applied_revision); + j += "},"; + j += "\"terminal\":{\"auto_trading\":" + + (string)(TerminalInfoInteger(TERMINAL_TRADE_ALLOWED) ? "true" : "false") + + "},"; + + // charts[] + j += "\"charts\":["; + for(int i = 0; i < ArraySize(charts); i++) + { + if(i) j += ","; + j += "{"; + j += "\"chart_id\":" + IntegerToString(charts[i].chart_id) + ","; + j += "\"symbol\":\"" + JsonEscape(charts[i].symbol) + "\","; + j += "\"timeframe\":\"" + JsonEscape(charts[i].timeframe) + "\","; + j += "\"expert\":\"" + JsonEscape(charts[i].expert) + "\","; + j += "\"expert_enabled\":" + (string)(charts[i].expert_enabled ? "true" : "false") + ","; + j += "\"deployment_id\":\"" + JsonEscape(charts[i].deployment_id) + "\""; + j += "}"; + } + j += "],"; + + // deployments[] status + j += "\"deployments\":["; + int written = 0; + for(int d = 0; d < ArraySize(desired); d++) + { + long cid = FindChartFor(desired[d].id, charts); + string status = (cid >= 0) ? "running" + : (desired[d].enabled ? "pending" : "paused"); + if(written) j += ","; + j += "{\"id\":\"" + JsonEscape(desired[d].id) + "\","; + j += "\"status\":\"" + status + "\""; + if(cid >= 0) j += ",\"chart_id\":" + IntegerToString(cid); + j += "}"; + written++; + } + j += "],"; + + // errors[] — one entry per failing deployment, cleared on its success + j += "\"errors\":["; + for(int e = 0; e < ArraySize(m_err_ids); e++) + { + if(e) j += ","; + j += "{\"id\":\"" + JsonEscape(m_err_ids[e]) + "\","; + j += "\"status\":\"failed\","; + j += "\"code\":\"" + JsonEscape(m_err_codes[e]) + "\","; + j += "\"detail\":\"" + JsonEscape(m_err_details[e]) + "\"}"; + } + j += "]"; + + j += "}"; + WriteFileAtomic(CHARTCTL_DIR + "\\observed.json", j); +} + +//+------------------------------------------------------------------+ +void CChartControl::RecordError(const string id, const string code, + const string detail) +{ + int slot = -1; + for(int i = 0; i < ArraySize(m_err_ids); i++) + if(m_err_ids[i] == id) { slot = i; break; } + if(slot < 0) + { + slot = ArraySize(m_err_ids); + ArrayResize(m_err_ids, slot + 1); + ArrayResize(m_err_codes, slot + 1); + ArrayResize(m_err_details, slot + 1); + ArrayResize(m_err_times, slot + 1); + } + m_err_ids[slot] = id; + m_err_codes[slot] = code; + m_err_details[slot] = detail; + m_err_times[slot] = TimeCurrent(); + PrintFormat("ChartControl ERROR [%s] %s: %s", id, code, detail); +} + +void CChartControl::ClearError(const string id) +{ + for(int i = 0; i < ArraySize(m_err_ids); i++) + { + if(m_err_ids[i] != id) + continue; + int last = ArraySize(m_err_ids) - 1; + m_err_ids[i] = m_err_ids[last]; + m_err_codes[i] = m_err_codes[last]; + m_err_details[i] = m_err_details[last]; + m_err_times[i] = m_err_times[last]; + ArrayResize(m_err_ids, last); + ArrayResize(m_err_codes, last); + ArrayResize(m_err_details, last); + ArrayResize(m_err_times, last); + return; + } +} + +// A deployment that just failed to attach gets a 60s cooldown so the +// loader doesn't churn ChartOpen/ChartClose on every reconcile pass. +bool CChartControl::InRetryCooldown(const string id) +{ + for(int i = 0; i < ArraySize(m_err_ids); i++) + if(m_err_ids[i] == id) + return (TimeCurrent() - m_err_times[i]) < 60; + return false; +} + +//+------------------------------------------------------------------+ +//| File helpers — everything under MQL5\Files\ (FILE_COMMON off). | +//+------------------------------------------------------------------+ +bool CChartControl::ReadFile(const string relpath, string &out) +{ + int h = FileOpen(relpath, FILE_READ | FILE_TXT | FILE_ANSI); + if(h == INVALID_HANDLE) + return false; + out = ""; + while(!FileIsEnding(h)) + out += FileReadString(h); + FileClose(h); + return true; +} + +bool CChartControl::WriteFileAtomic(const string relpath, const string content) +{ + string tmp = relpath + ".tmp"; + int h = FileOpen(tmp, FILE_WRITE | FILE_TXT | FILE_ANSI); + if(h == INVALID_HANDLE) + { + PrintFormat("ChartControl: cannot open %s for write (err=%d)", + tmp, GetLastError()); + return false; + } + FileWriteString(h, content); + FileClose(h); + // FileMove with rewrite flag = atomic-ish replace on Windows. Without + // FILE_REWRITE the move needs the pre-delete to have worked, and that + // fails with 5020 whenever the API side has the file open for a read. + if(!FileMove(tmp, 0, relpath, FILE_REWRITE)) + { + PrintFormat("ChartControl: FileMove %s->%s failed (err=%d)", + tmp, relpath, GetLastError()); + return false; + } + return true; +} + +void CChartControl::DeleteFileSafe(const string relpath) +{ + if(FileIsExist(relpath)) + FileDelete(relpath); +} + +//+------------------------------------------------------------------+ +//| Minimal JSON readers for our own compact, predictable output. | +//| NOT a general parser — only the shapes chartctl produces. | +//+------------------------------------------------------------------+ +string CChartControl::JsonStr(const string key, const string def, + const string json) +{ + string needle = "\"" + key + "\""; + int p = StringFind(json, needle); + if(p < 0) return def; + int colon = StringFind(json, ":", p + StringLen(needle)); + if(colon < 0) return def; + int q1 = StringFind(json, "\"", colon + 1); + if(q1 < 0) return def; + // find unescaped closing quote + int i = q1 + 1; + string val = ""; + while(i < StringLen(json)) + { + ushort ch = StringGetCharacter(json, i); + if(ch == '\\') + { + if(i + 1 < StringLen(json)) + val += ShortToString(StringGetCharacter(json, i + 1)); + i += 2; + continue; + } + if(ch == '"') + break; + val += ShortToString(ch); + i++; + } + return val; +} + +long CChartControl::JsonNum(const string key, const string json) +{ + string needle = "\"" + key + "\""; + int p = StringFind(json, needle); + if(p < 0) return 0; + int colon = StringFind(json, ":", p + StringLen(needle)); + if(colon < 0) return 0; + int i = colon + 1; + string num = ""; + while(i < StringLen(json)) + { + ushort ch = StringGetCharacter(json, i); + if((ch >= '0' && ch <= '9') || ch == '-') + num += ShortToString(ch); + else if(num != "") + break; + i++; + } + return (long)StringToInteger(num); +} + +//+------------------------------------------------------------------+ +//| Extract the deployments[] array from desired.json. | +//| Each object: {id,expert,template,symbol,timeframe,enabled}. | +//+------------------------------------------------------------------+ +bool CChartControl::ExtractDeployments(const string json, + ChartCtlDeployment &out[]) +{ + ArrayResize(out, 0); + int arr = StringFind(json, "\"deployments\""); + if(arr < 0) return false; + int i = StringFind(json, "[", arr); + if(i < 0) return false; + + int depth = 0; + int obj_start = -1; + for(; i < StringLen(json); i++) + { + ushort ch = StringGetCharacter(json, i); + if(ch == '{') + { + if(depth == 0) obj_start = i; + depth++; + } + else if(ch == '}') + { + depth--; + if(depth == 0 && obj_start >= 0) + { + string obj = StringSubstr(json, obj_start, i - obj_start + 1); + ChartCtlDeployment d; + d.id = JsonStr("id", "", obj); + d.expert = JsonStr("expert", "", obj); + d.templ = JsonStr("template", "", obj); + d.symbol = JsonStr("symbol", "", obj); + d.timeframe = JsonStr("timeframe", "", obj); + // enabled is a bare JSON bool; desired.json emits it compactly. + d.enabled = (StringFind(obj, "\"enabled\": false") < 0 + && StringFind(obj, "\"enabled\":false") < 0); + if(d.id != "") + { + int n = ArraySize(out); + ArrayResize(out, n + 1); + out[n] = d; + } + obj_start = -1; + } + } + else if(ch == ']' && depth == 0) + break; + } + return true; +} + +string CChartControl::JsonEscape(const string s) +{ + string out = s; + StringReplace(out, "\\", "\\\\"); + StringReplace(out, "\"", "\\\""); + StringReplace(out, "\n", " "); + StringReplace(out, "\r", " "); + return out; +} + +//+------------------------------------------------------------------+ +ENUM_TIMEFRAMES CChartControl::TF(const string s) +{ + if(s == "M1") return PERIOD_M1; + if(s == "M2") return PERIOD_M2; + if(s == "M3") return PERIOD_M3; + if(s == "M4") return PERIOD_M4; + if(s == "M5") return PERIOD_M5; + if(s == "M6") return PERIOD_M6; + if(s == "M10") return PERIOD_M10; + if(s == "M12") return PERIOD_M12; + if(s == "M15") return PERIOD_M15; + if(s == "M20") return PERIOD_M20; + if(s == "M30") return PERIOD_M30; + if(s == "H1") return PERIOD_H1; + if(s == "H2") return PERIOD_H2; + if(s == "H3") return PERIOD_H3; + if(s == "H4") return PERIOD_H4; + if(s == "H6") return PERIOD_H6; + if(s == "H8") return PERIOD_H8; + if(s == "H12") return PERIOD_H12; + if(s == "D1") return PERIOD_D1; + if(s == "W1") return PERIOD_W1; + if(s == "MN1") return PERIOD_MN1; + return PERIOD_H1; +} +//+------------------------------------------------------------------+ diff --git a/config/config.yaml.example b/config/config.yaml.example index 79bddb69..7a6a1260 100644 --- a/config/config.yaml.example +++ b/config/config.yaml.example @@ -25,6 +25,21 @@ tailscale: auth_key: "" # leave empty to disable the tailscale sidecar entirely login_server: "" # Headscale login server URL; empty = Tailscale cloud +# Chart Deployments (chartctl) — EA hosting primitives. Stage .ex5/.set +# artifacts and declare "run expert X with set Y on symbol S timeframe T" +# as desired state; a resident loader EA (assets/experts/MT5ChartLoader.mq5, +# or your own EA using assets/experts/include/ChartControl.mqh) reconciles +# the terminal's charts to it. Live-mode terminals only; backtest-mode +# terminals ignore this entirely. See docs/chart-control-protocol.md. +chartctl: + enabled: false # OPT-IN. true also auto-attaches the loader EA + # to every live terminal via [StartUp]; false = the + # chartctl endpoints return 404 and nothing changes. + reconcile_hint_interval: "5s" # written into desired.json for the loader + observed_stale_after: "60s" # observed.json older than this = loader stale + command_timeout: "30s" # wait for screenshot/one-shot loader replies + max_upload_bytes: 16777216 # 16 MB cap on .ex5/.set uploads + # Extra pip packages installed in the VM on every boot. # MetaTrader5, flask, waitress, and flask-compress are always installed. requirements: [] diff --git a/docs/chart-control-protocol.md b/docs/chart-control-protocol.md new file mode 100644 index 00000000..5df07b03 --- /dev/null +++ b/docs/chart-control-protocol.md @@ -0,0 +1,226 @@ +# Chart Control Protocol v1 + +How mt5-httpapi's **Chart Deployments** feature attaches Expert Advisors to +charts remotely, keeps them running, and reports what's actually live — +without RDP, without restarting terminals, and without any orchestrator. + +MT5 has no SDK call to attach an EA to a chart. The only programmatic path +is `ChartApplyTemplate()` from MQL5 code already running inside the +terminal. So the API and a small **resident loader EA** cooperate over a +handful of JSON files in the terminal's `MQL5\Files\chartctl\` sandbox. + +The protocol is intentionally file-based so that: + +- it needs zero WebRequest whitelist entries (works on locked-down terminals), +- it's trivially debuggable over RDP during rollout, and +- **any** EA can implement it — the bundled `MT5ChartLoader`, or your own + resident utility EA (e.g. an account tracker) that adopts + `ChartControl.mqh`. + +--- + +## Roles + +| Side | Writes | Reads | +|------|--------|-------| +| **API** (mt5-httpapi) | `desired.json`, `command.json`, generated `.tpl` files | `observed.json`, `command_result.json` | +| **Loader EA** (in terminal) | `observed.json`, `command_result.json`, screenshots | `desired.json`, `command.json` | + +The API owns *desired state*. The loader owns *observed truth*. Neither +writes the other's files. Success is defined as **observed converging on +desired**, never as "the API copied some files." + +--- + +## Files (all under `MQL5\Files\chartctl\`) + +### `desired.json` — API → loader + +```json +{ + "protocol": 1, + "revision": 42, + "updated_at": "2026-07-09T12:00:00Z", + "reconcile_interval": 5, + "deployments": [ + { + "id": "dep_a1b2c3", + "expert": "HappyGoldScalp", + "template": "\\Files\\chartctl\\dep_a1b2c3.tpl", + "symbol": "XAUUSD", + "timeframe": "M5", + "enabled": true + } + ] +} +``` + +`revision` is a monotonic counter; the loader can skip a full parse when it +hasn't changed. `template` is passed verbatim to `ChartApplyTemplate()`; +the leading backslash makes MT5 resolve it against `\MQL5` — the +only search root that doesn't depend on which EX5 hosts the loader +(paths without a leading backslash resolve relative to the calling EX5's +own folder, and the `templates\` GUI directory is never searched). The +API generates one `.tpl` per deployment, written into the +`MQL5\Files\chartctl\` protocol directory. + +### `observed.json` — loader → API (rewritten every reconcile pass, ~5s) + +```json +{ + "protocol": 1, + "loader": { "name": "MT5ChartLoader", "version": "1.0.0", + "last_loop": "2026-07-09 12:00:03", "applied_revision": 42 }, + "terminal": { "auto_trading": true }, + "charts": [ + { "chart_id": 133039117, "symbol": "XAUUSD", "timeframe": "PERIOD_M5", + "expert": "HappyGoldScalp", "expert_enabled": true, + "deployment_id": "dep_a1b2c3" } + ], + "deployments": [ + { "id": "dep_a1b2c3", "status": "running", "chart_id": 133039117 } + ], + "errors": [] +} +``` + +`applied_revision == desired.revision` **and** deployment `status: running` +is the only definition of a converged deployment. The API's +`GET /deployments` merges the two files and derives per-deployment status +(`pending → running → degraded → failed → paused`). + +### `command.json` / `command_result.json` — one-shot imperatives + +For operations that produce an artifact rather than converge state +(currently `screenshot`, `close_chart`, and a `reconcile` nudge). One +command in flight; the loader writes the result keyed by `command_id` and +deletes the command. + +`close_chart` closes an arbitrary chart by `chart_id` (from the observed +charts list) — the cleanup escape hatch for charts the loader cannot +attribute. The loader refuses (`CLOSE_REFUSED`) to close its own chart. + +--- + +## Loader responsibilities + +Every pass (timer-driven, ~1s), the owning loader: + +1. Reads `desired.json`; if `revision` changed, reconciles. +2. **Reconcile** = diff desired deployments against actual charts: + - Enabled deployment with no matching chart → first try to **adopt** an + unowned chart already running the exact expert + symbol + timeframe + (stamp it `chartctl:`); only if none exists, `SymbolSelect` → + `ChartOpen` → `ChartApplyTemplate` → verify `CHART_EXPERT_NAME` + within 10s → stamp `chartctl:` into the chart comment. Stamps + are verified by read-back (`ChartSetString` is asynchronous); a + failed attach closes the chart it opened and backs off 60s. + - A chart it owns (comment starts `chartctl:`) whose deployment is gone + or disabled → `ChartClose`. + - Charts it doesn't own are **reported but never touched** by + reconcile (an explicit `close_chart` command can close them). +3. Writes `observed.json`. +4. Answers any pending `command.json`. + +### Rules + +- The loader **never calls trade functions**. Chart lifecycle only. +- Exactly **one** loader per terminal, guarded by a terminal + `GlobalVariable` mutex (`chartctl_loader_owner`). A second loader detects + the live mutex and stays passive, reclaiming only if the owner vanishes. +- All file writes are atomic (temp + `FileMove`). +- Attribution is by the `chartctl:` chart comment the loader sets at + attach time. The comment does **not** reliably survive a terminal + restart (observed live: MT5's saved profile restores the chart and + expert but drops the comment, leaking one duplicate chart per reboot), + which is why reconcile adopts exact-match unowned charts instead of + trusting the comment alone. + +--- + +## Self-healing + +Three layers converge a terminal back to desired state after any restart +(including the optional `reboot_interval` VM reboots): + +1. **MT5 native chart restoration** brings back charts + attached experts + from the last saved profile — often everything, for free. +2. **Loader reconciliation** repairs whatever native restoration missed + (crash before profile save, chart closed by hand, failed `OnInit`). +3. **Watchdog** (`monitor.py`) logs loudly if the terminal is alive but the + loader's `last_loop` goes stale — the one state layers 1–2 can't fix + alone. + +--- + +## Implementing the protocol in your own EA + +The whole loader is a portable include. In your resident EA: + +```mql5 +#include +CChartControl ctl; + +int OnInit() { if(!ctl.Init()) return INIT_FAILED; + EventSetTimer(1); return INIT_SUCCEEDED; } +void OnTimer(){ ctl.Tick(); } +void OnDeinit(const int r){ ctl.Deinit(); } +``` + +That's it — your account tracker (or any always-on EA) becomes the loader, +so one resident EA does telemetry *and* chart deployment instead of two. The +mutex makes running both your EA and the standalone `MT5ChartLoader` +degrade safely. See `assets/experts/MT5ChartLoader.mq5` for the reference +glue and `assets/experts/include/ChartControl.mqh` for the implementation. + +--- + +## Bootstrapping the loader + +**Zero-touch (default).** Provisioning does everything — no RDP, no manual +attach, fully API/config-driven: + +1. On VM boot, `start.bat` runs `compile-chartctl-loader.bat`, which copies + `ChartControl.mqh` + `MT5ChartLoader.mq5` into every broker base + terminal, compiles with that base's MetaEditor64, and propagates the + `.ex5` into every existing terminal instance (new instances inherit it + from the base copy). +2. `config_helper.py` writes a `[StartUp] Expert=Advisors\MT5ChartLoader` + section into each terminal's generated `mt5start.ini` — so the terminal + attaches the loader itself at every launch. The startup chart symbol + honors the terminal's `symbol_suffix` (e.g. `EURUSD.r`). +3. The `[StartUp]` line re-fires on every launch (including the periodic + `reboot_interval` reboots); the GlobalVariable mutex makes this + idempotent — a duplicate loader closes its own chart and vanishes, so + charts never accumulate. + +Both steps honor the same gating as the API: live-mode terminals only, +`chartctl.enabled` globally, per-terminal `chartctl: false` opts out. + +**Manual (fallback / existing fleets).** Drag `MT5ChartLoader` onto any +chart once over RDP, or fold `ChartControl.mqh` into a resident EA you +already deploy. + +--- + +## Endpoint summary + +| Method | Path | Purpose | +|--------|------|---------| +| `POST` | `/experts` | Upload `.ex5` (multipart) | +| `GET` | `/experts` | List staged experts | +| `DELETE` | `/experts/` | Remove staged expert (refused if in use) | +| `POST` | `/sets` | Upload `.set` (returns parsed inputs) | +| `GET` | `/sets`, `/sets/` | List / inspect set files | +| `POST` | `/deployments` | Declare a deployment (EA+set+symbol+TF) | +| `GET` | `/deployments`, `/deployments/` | Desired ⋈ observed status | +| `PATCH` | `/deployments/` | Change set file or pause/resume | +| `DELETE` | `/deployments/` | Remove a deployment | +| `POST` | `/deployments/reconcile` | Force the loader to re-reconcile | +| `GET` | `/charts` | Live chart/EA inventory from the terminal | +| `GET` | `/loader` | Loader presence/version/liveness | +| `POST` | `/charts//screenshot` | PNG of a chart | +| `POST` | `/charts//close` | Close a chart (incl. unattributed ones) | + +All routes sit behind the terminal's existing per-account route prefix and +bearer-token auth. Nothing here touches the MT5 SDK lock. diff --git a/docs/chart-deployments.md b/docs/chart-deployments.md new file mode 100644 index 00000000..dc5f30a6 --- /dev/null +++ b/docs/chart-deployments.md @@ -0,0 +1,223 @@ +# Chart Deployments — remote EA deployment over HTTP + +Attach Expert Advisors to charts with set files over the HTTP API. No RDP, no +terminal restart, and no human clicking through the Navigator. + +## Contents + +- [Quick start](#quick-start) +- [`POST /deployments`](#post-deployments) +- [`GET /loader`](#get-loader) +- [`GET /charts`](#get-charts) +- [Screenshots](#post-chartschart_idscreenshot) and [closing charts](#post-chartschart_idclose) +- [WebRequest allowlist](#webrequest-allowlist) +- [Chart Control Protocol](chart-control-protocol.md) — the file contract the loader speaks + + +> 📹 **Video walkthrough:** *Coming soon — a full walkthrough of staging, deploying, and +> debugging EAs via the chartctl API.* + +Attach Expert Advisors to charts with set files over the HTTP API — no RDP, +no terminal restart. Stage an `.ex5` + `.set`, declare a deployment (expert + +set + symbol + timeframe), and a resident loader EA inside the terminal +reconciles the terminal's actual charts to match. The API holds *desired +state*; the loader reports *observed truth* back, so a deployment only reads +`running` once the expert is confirmed live on a chart. + +Any client — a dashboard, a script, an AI agent — drives it over plain REST. +Full protocol contract and file formats: [`docs/chart-control-protocol.md`](docs/chart-control-protocol.md). + +| Method | Endpoint | Description | +| ------------------------------------------------ | ------------------------------------- | -------------------------------------------------------- | +| `POST` / `GET` / `DELETE` | `/experts` `/experts/` | Stage, list, remove EA `.ex5` files | +| `POST` / `GET` | `/sets` `/sets/` | Stage, list, inspect `.set` parameter files (parsed) | +| `POST` / `GET` | `/deployments` | Create or list deployments | +| `GET` / `PATCH` / `DELETE` | `/deployments/` | Inspect, pause/resume/change set, tear down a deployment | +| `POST` | `/deployments/reconcile` | Force an immediate reconcile cycle (otherwise periodic) | +| `GET` | `/charts` | Live chart/EA inventory from inside the terminal | +| `GET` | `/loader` | Loader EA status — alive, version, chart open count | +| `POST` | `/charts//screenshot` | Capture a chart PNG from inside the terminal | +| `POST` | `/charts//close` | Close a chart by id (any chart, including leaks) | + +**Opt-in.** `chartctl.enabled` defaults to `false`; set it to `true` in +`config.yaml` to switch the feature on. Live-mode terminals only, and any single +terminal can stay clear of it with `chartctl: false`. + +That default is deliberate. Enabling this writes a `[StartUp]` section into every +live terminal's INI, so the loader EA attaches itself at launch — fleet-wide +behaviour that must be asked for, never inherited from an upgrade. With the block +absent nothing changes and the endpoints return 404. + +**Once enabled, setup is none.** On boot, provisioning auto-compiles the bundled loader EA +(`assets/experts/MT5ChartLoader.mq5`) in every broker base and wires a +`[StartUp] Expert=` line into each live terminal's `mt5start.ini`, so the +loader attaches itself at launch — the whole path is API/config-driven, no +RDP. Already have a resident utility EA on every terminal? Adopt the protocol +into it with three calls instead of running a second EA — see +`assets/experts/include/ChartControl.mqh`. The standalone loader steps aside +automatically (single-loader mutex via terminal GlobalVariable). + +### Quick start + +```bash +# 0. Set your base URL + auth token +export MT5_API_URL=http://localhost:8888/yourbroker/yourlogin +export MT5_API_TOKEN=$(grep api_token config/config.yaml | awk -F'"' '{print $2}') + +# 1. Stage an expert (.ex5) and a set file (.set) +curl -F "expert=@HappyGoldScalp.ex5" "$MT5_API_URL/experts" +curl -F "set=@gold-m5.set" "$MT5_API_URL/sets" # returns parsed inputs + +# 2. Deploy — declare desired state +curl -X POST "$MT5_API_URL/deployments" \ + -H "Authorization: Bearer $MT5_API_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"expert":"HappyGoldScalp.ex5","set":"gold-m5.set","symbol":"XAUUSD","timeframe":"M5"}' +# -> {"id":"dep_a1b2c3","status":"pending"} + +# 3. Verify — status flips to "running" once the loader confirms attach +curl -H "Authorization: Bearer $MT5_API_TOKEN" "$MT5_API_URL/deployments" + +# 4. Change the set file in place (no restart), pause, or tear down +curl -X PATCH "$MT5_API_URL/deployments/dep_a1b2c3" \ + -H "Authorization: Bearer $MT5_API_TOKEN" \ + -d '{"set":"gold-m5-v2.set"}' +curl -X PATCH "$MT5_API_URL/deployments/dep_a1b2c3" \ + -H "Authorization: Bearer $MT5_API_TOKEN" \ + -d '{"enabled":false}' +curl -X DELETE "$MT5_API_URL/deployments/dep_a1b2c3" \ + -H "Authorization: Bearer $MT5_API_TOKEN" +``` + +### `POST /deployments` + +| Field | Required | Description | +| ----------- | -------- | -------------------------------------------- | +| `expert` | yes | `.ex5` filename (previously staged via `POST /experts`) | +| `set` | yes | `.set` filename (previously staged via `POST /sets`) | +| `symbol` | yes | e.g. `EURUSD` | +| `timeframe` | yes | `M1` `M5` `M15` `M30` `H1` `H4` `D1` `W1` `MN` | + +Example response: + +```json +{ + "id": "dep_a1b2c3", + "expert": "HappyGoldScalp.ex5", + "set": "gold-m5.set", + "symbol": "XAUUSD", + "timeframe": "M5", + "enabled": true, + "status": "pending", + "revision": 1, + "created_at": "2026-07-27T12:00:00Z", + "updated_at": "2026-07-27T12:00:00Z" +} +``` + +Deployment lifecycle: `pending` → `running` (loader confirmed) → `degraded` +(loader sees an error) → `failed` (unrecoverable) or `paused` (disabled by +user). + +### `GET /loader` + +Returns the resident loader EA's status: + +```json +{ + "alive": true, + "version": "1.0.2", + "charts_open": 3, + "observed_revision": 5, + "desired_revision": 5, + "in_sync": true +} +``` + +If `alive` is `false`, the loader hasn't started yet — check that +`chartctl.enabled` is on and the terminal was restarted after provisioning. +The first boot compile log lives at `logs/compile-chartctl-loader.log` inside +the VM. + +### `GET /charts` + +Lists every chart known to the terminal, annotated with which deployment (if +any) the loader attributes it to: + +```json +{ + "charts": [ + {"id": 123, "symbol": "XAUUSD", "timeframe": "M5", + "expert": "HappyGoldScalp.ex5", "deployment_id": "dep_a1b2c3"}, + {"id": 456, "symbol": "EURUSD", "timeframe": "H1", + "expert": "", "deployment_id": null} + ] +} +``` + +Charts with no deployment (e.g. leftover duplicates) can be closed with +`POST /charts//close`. + +### `POST /charts//screenshot` + +Captures a PNG of the chart from inside the terminal. Returns the raw binary +(`image/png`). No query-string auth — put the token in the header. + +```bash +curl -H "Authorization: Bearer $MT5_API_TOKEN" \ + "$MT5_API_URL/charts/123/screenshot" -o xauusd-m5.png +``` + +### `POST /charts//close` + +Sends a `close_chart` command to the loader. The loader refuses to close its +own chart (returns `CLOSE_REFUSED`). + +```json +// Response +{"command_id":"cmd_xxx","status":"accepted"} +``` + +Poll the deployment status to confirm the chart was recreated if expected. + +## WebRequest allowlist + +EAs that call `WebRequest()` need their target hosts in the terminal's +allowlist (Tools → Options → Expert Advisors → *Allow WebRequest for*). +It's a dedicated call rather than something on the deploy hot path (most +deployments need no URLs): + +| Method | Endpoint | Description | +| ------ | ------------------------- | ------------------------------------- | +| `GET` | `/webrequest` | Current effective allowlist | +| `PUT` | `/webrequest` | Replace or add to the allowlist | +| `POST` | `/webrequest/apply` | Re-apply current desired list now | + +```bash +curl -H "Authorization: Bearer $MT5_API_TOKEN" "$MT5_API_URL/webrequest" +curl -X PUT "$MT5_API_URL/webrequest" \ + -H "Authorization: Bearer $MT5_API_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"add":["https://api.telegram.org"]}' # or {"urls":[...]} to replace +# -> {"success":true,"urls":[...],"applied_via":"autoit:OK"} +curl -X POST "$MT5_API_URL/webrequest/apply" \ + -H "Authorization: Bearer $MT5_API_TOKEN" # re-apply current list now +``` + +Inside the Windows VM the allowlist is **not** stored in `common.ini` — it +lives in the machine-bound `MQL5\experts.dat` and MT5 drops it on every +restart. So the list is applied the way a user would: a bundled AutoIt +interpreter (`assets/autoit/`; unmodified official binary, redistributed +with its EULA and notices — see `assets/autoit/NOTICE.txt`) drives +Tools → Options → Expert Advisors and types the URLs in. This takes effect immediately in-session (no restart), and +because MT5 forgets it on restart, the API re-applies the persisted list +automatically ~25 s after each terminal (re)start, so it survives the periodic +auto-reboot. On a bare-metal terminal where `common.ini` *is* the store, it +falls back to writing `common.ini` + restarting. + +The desired list is persisted per terminal (`Config/webrequest.json`); the +first call migrates whatever the terminal already has, so manually configured +URLs are preserved. GUI applies are serialized across every terminal on a host +by a named Windows kernel mutex (crash-safe — a dead holder is auto-released), +and each terminal's boot re-apply is staggered by its port, so many terminals +per VM can safely provision WebRequest URLs without their keystrokes colliding. diff --git a/docs/rest-api.md b/docs/rest-api.md index de935f80..2efea264 100644 --- a/docs/rest-api.md +++ b/docs/rest-api.md @@ -12,6 +12,7 @@ One HTTP surface for routing, auth, health, terminal control, account state, and - [Market data](market-data.md) - [Trading and history](trading-and-history.md) - [Backtesting](backtesting.md) +- [Chart Deployments](chart-deployments.md) ## API diff --git a/examples/chartctl/deploy.sh b/examples/chartctl/deploy.sh new file mode 100755 index 00000000..a7966893 --- /dev/null +++ b/examples/chartctl/deploy.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Chart Deployments — example workflow +# Stage an expert + set, deploy to a chart, verify, then tear down. +# +# Usage: +# export MT5_API_URL=http://localhost:8888/yourbroker/yourlogin +# export MT5_API_TOKEN=your-api-token +# ./deploy.sh +# +# Example: +# ./deploy.sh ~/EAs/MyScalp.ex5 ~/EAs/eurusd-m5.set EURUSD M5 + +MT5_API_URL="${MT5_API_URL:?Set MT5_API_URL}" +MT5_API_TOKEN="${MT5_API_TOKEN:-}" +AUTH=() +if [ -n "$MT5_API_TOKEN" ]; then + AUTH=(-H "Authorization: Bearer $MT5_API_TOKEN") +fi + +EXPERT_PATH="${1:?Usage: deploy.sh }" +SET_PATH="${2:?}" +SYMBOL="${3:?}" +TIMEFRAME="${4:?}" +EXPERT_NAME=$(basename "$EXPERT_PATH") +SET_NAME=$(basename "$SET_PATH") + +echo "=== Step 1: Stage expert ===" +curl -sS "${AUTH[@]}" -F "expert=@$EXPERT_PATH" "$MT5_API_URL/experts" | head -c 200 +echo + +echo "=== Step 2: Stage set file ===" +curl -sS "${AUTH[@]}" -F "set=@$SET_PATH" "$MT5_API_URL/sets" | head -c 200 +echo + +echo "=== Step 3: Create deployment ===" +DEPLOY_JSON=$(curl -sS -X POST "${AUTH[@]}" \ + -H "Content-Type: application/json" \ + "$MT5_API_URL/deployments" \ + -d "{\"expert\":\"$EXPERT_NAME\",\"set\":\"$SET_NAME\",\"symbol\":\"$SYMBOL\",\"timeframe\":\"$TIMEFRAME\"}") +echo "$DEPLOY_JSON" | head -c 300 +echo +DEPLOY_ID=$(echo "$DEPLOY_JSON" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])" 2>/dev/null || echo "") + +if [ -z "$DEPLOY_ID" ]; then + echo "Failed to extract deployment id from response." + exit 1 +fi + +echo "=== Step 4: Poll until running (up to 60s) ===" +for i in $(seq 1 12); do + sleep 5 + STATUS=$(curl -sS "${AUTH[@]}" "$MT5_API_URL/deployments/$DEPLOY_ID" | + python3 -c "import sys,json; print(json.load(sys.stdin).get('status','unknown'))" 2>/dev/null || echo "unknown") + echo " Attempt $i: status=$STATUS" + if [ "$STATUS" = "running" ]; then + echo "=== Deployment running! ===" + break + fi +done + +echo "=== Step 5: Verify via /charts ===" +curl -sS "${AUTH[@]}" "$MT5_API_URL/charts" | python3 -m json.tool | head -30 + +echo "=== Step 6: Take a screenshot ===" +CHART_ID=$(curl -sS "${AUTH[@]}" "$MT5_API_URL/charts" | + python3 -c "import sys,json; charts=json.load(sys.stdin).get('charts',[]); print([c['id'] for c in charts if c.get('deployment_id')=='$DEPLOY_ID'][0])" 2>/dev/null || echo "") +if [ -n "$CHART_ID" ]; then + curl -sS "${AUTH[@]}" "$MT5_API_URL/charts/$CHART_ID/screenshot" -o "chartctl-${SYMBOL}-${TIMEFRAME}.png" + echo "Screenshot saved to chartctl-${SYMBOL}-${TIMEFRAME}.png" +fi + +echo "=== Done ===" +echo "Deployment $DEPLOY_ID is running. Clean up with:" +# Placeholder rather than "${AUTH[@]}": this line is printed, not run, and +# expanding the array here would echo the bearer token to the terminal (and +# into whatever captures this output). +echo " curl -X DELETE -H \"Authorization: Bearer \$MT5_API_TOKEN\" \"$MT5_API_URL/deployments/$DEPLOY_ID\"" diff --git a/mt5api/chartctl/__init__.py b/mt5api/chartctl/__init__.py new file mode 100644 index 00000000..96ea4a78 --- /dev/null +++ b/mt5api/chartctl/__init__.py @@ -0,0 +1,14 @@ +"""Chart Deployments (chartctl) — EA deployment primitives. + +Generic, orchestrator-agnostic capability: stage .ex5/.set artifacts, +declare desired deployments (expert + set + symbol + timeframe), and let a +resident loader EA reconcile the terminal's charts to that desired state +over a file protocol inside the MQL5 sandbox. + +Protocol contract: docs/chart-control-protocol.md +Reference loader: assets/experts/MT5ChartLoader.mq5 (+ ChartControl.mqh) + +Nothing in this package touches the MT5 SDK — every operation is plain +file I/O against TERMINAL_DIR, so no handler here ever queues behind the +process-wide MT5 lock. +""" diff --git a/mt5api/chartctl/autoit_webrequest.py b/mt5api/chartctl/autoit_webrequest.py new file mode 100644 index 00000000..a2da97f3 --- /dev/null +++ b/mt5api/chartctl/autoit_webrequest.py @@ -0,0 +1,240 @@ +"""Apply the WebRequest allowlist by driving MT5's Options dialog with AutoIt. + +Why not just write the file? On the dockur-VM terminal build the allowlist is +NOT stored in (or read back from) ``common.ini`` — it lives in the machine-bound +``MQL5\\experts.dat`` and does not survive a terminal restart even when set in +MT5's own Options dialog. So file injection can't provision it there. Instead we +set it the way a user would: open Tools -> Options -> Expert Advisors and add the +URLs. WebRequest then works immediately in-session (verified with a probe EA: +``ret=200`` on an allowed URL). Because MT5 drops the list on restart, this is a +re-appliable operation (a dedicated call, plus an optional boot re-apply). + +The heavy lifting is a portable AutoIt interpreter + script shipped under +``assets/autoit/`` (live-mounted into the VM). MT5 runs elevated (``-Verb RunAs``) +and Windows UIPI blocks a non-elevated process from sending it input — but the API +process itself already runs elevated here (verified: AutoIt reports ``IsAdmin=1``), +so a plain, blocking ``subprocess.run`` inherits that elevation, drives MT5 fine, +and lets us capture the exit code. A ``use_runas`` path (async ``Start-Process +-Verb RunAs`` + log polling) is kept as a fallback for non-elevated deployments. +""" +from __future__ import annotations + +import contextlib +import os +import re +import subprocess +import time + +from mt5api.config import ACCOUNT, ASSETS_DIR, BROKER, INI_FILE, INSTANCE, TERMINAL_DIR +from mt5api.logger import log + +AUTOIT_DIR = os.path.join(ASSETS_DIR, "autoit") +AUTOIT_EXE = os.path.join(AUTOIT_DIR, "AutoIt3_x64.exe") + +_LOG_NAME = "webrequest_autoit.log" +_URLS_NAME = "webrequest_apply_urls.txt" + +# Machine-wide mutex serializing GUI automation across every terminal's API +# process on this host (see _gui_lock). +_GUI_MUTEX_NAME = "Global\\mt5_httpapi_webrequest_autoit" +_GUI_LOCK_WAIT_MS = 180_000 # 3 min — comfortably longer than any real apply + + +@contextlib.contextmanager +def _gui_lock(wait_ms: int = _GUI_LOCK_WAIT_MS): + """Serialize GUI automation across all terminal API processes on this host. + + Desktop input (window focus + the keyboard) is a single shared resource, so + two AutoIt applies running at once would steal focus from each other and leak + keystrokes into the wrong terminal — even producing a RESULT=OK that typed a + URL into the wrong window. Each terminal runs its own API *process*, so an + in-process ``threading.Lock`` is not enough; we take a named Windows kernel + mutex, which is machine-wide. It is crash-safe: if a holder dies, the kernel + hands ownership to the next waiter (WAIT_ABANDONED), so there is no stale + lock. On wait-timeout we proceed anyway (URLs are needed and 3 min means + something is wedged, not merely busy). No-op off Windows.""" + if os.name != "nt": + yield True + return + import ctypes + from ctypes import wintypes + + k = ctypes.windll.kernel32 + k.CreateMutexW.restype = wintypes.HANDLE + k.CreateMutexW.argtypes = (wintypes.LPVOID, wintypes.BOOL, wintypes.LPCWSTR) + k.WaitForSingleObject.argtypes = (wintypes.HANDLE, wintypes.DWORD) + handle = k.CreateMutexW(None, False, _GUI_MUTEX_NAME) + owned = False + try: + if handle: + res = k.WaitForSingleObject(handle, wait_ms) + owned = res in (0x0, 0x80) # WAIT_OBJECT_0 / WAIT_ABANDONED + if not owned: + log.warning("WebRequest GUI lock: wait timed out (%d ms); proceeding", wait_ms) + else: + log.warning("WebRequest GUI lock: CreateMutex failed; proceeding unlocked") + yield owned + finally: + if handle: + if owned: + k.ReleaseMutex(handle) + k.CloseHandle(handle) + + +def _resolve_script(script: str) -> str: + """Only run a plain-named .au3 that actually exists in AUTOIT_DIR (the + read-only, repo-controlled mount) — no path separators, no traversal.""" + if os.path.basename(script) != script or not script.lower().endswith(".au3"): + raise ValueError(f"bad script name: {script}") + path = os.path.join(AUTOIT_DIR, script) + if not os.path.exists(path): + raise ValueError(f"script not found: {script}") + return path + + +def available() -> bool: + """True only on Windows with the AutoIt interpreter present (i.e. the VM). + The .exe ships in the repo for all platforms, but only runs under Windows, + so gate on the OS too — elsewhere the caller uses the common.ini fallback.""" + return os.name == "nt" and os.path.exists(AUTOIT_EXE) + + +def _window_match() -> str: + """A token guaranteed to appear in this terminal's window title: the login + (read lock-free from mt5start.ini). Falls back to the broker name.""" + try: + with open(INI_FILE, "r", encoding="utf-8", errors="ignore") as f: + for line in f: + if line.strip().lower().startswith("login="): + val = line.split("=", 1)[1].strip() + if val: + return val + except OSError: + pass + return BROKER + + +def _terminal_pid() -> int: + """PID of THIS terminal's terminal64.exe, resolved by executable path. + + The window title (login) is ambiguous — cloned terminals of the same + account have identical titles — so the pid is what actually pins the right + window for the AutoIt script. WMI via PowerShell, because it can read exe + paths of elevated processes. Returns 0 if not found (script falls back to + title matching).""" + if ACCOUNT: + path_filter = f"*\\{BROKER}\\{ACCOUNT}\\{INSTANCE}\\*" + else: + path_filter = f"*\\{BROKER}\\*" + ps_cmd = ( + "Get-WmiObject Win32_Process -Filter \"Name='terminal64.exe'\" " + "| Where-Object { $_.ExecutablePath -like '" + path_filter + "' } " + "| Select-Object -First 1 -ExpandProperty ProcessId" + ) + try: + result = subprocess.run( + ["powershell", "-Command", ps_cmd], + capture_output=True, text=True, timeout=30, + ) + pid = int(result.stdout.strip()) + return pid if pid > 0 else 0 + except (ValueError, subprocess.SubprocessError, OSError): + log.warning("WebRequest: terminal pid lookup failed; falling back to title match") + return 0 + + +def _config_path(name: str) -> str: + return os.path.join(TERMINAL_DIR, "Config", name) + + +def _wait_result(logpath: str, timeout: float) -> tuple[str, str]: + """Poll the AutoIt log for a ``RESULT=`` line. Returns (status, log).""" + deadline = time.time() + timeout + while time.time() < deadline: + time.sleep(1) + try: + with open(logpath, "r", encoding="utf-8", errors="ignore") as f: + txt = f.read() + except OSError: + continue + m = re.search(r"RESULT=(\w+)", txt) + if m: + return m.group(1), txt + try: + with open(logpath, "r", encoding="utf-8", errors="ignore") as f: + txt = f.read() + except OSError: + txt = "" + return "TIMEOUT", txt + + +def _run_script( + script: str, extra_args: list[str], timeout: float, use_runas: bool = False +) -> tuple[str, str]: + if not available(): + raise RuntimeError(f"AutoIt interpreter not found at {AUTOIT_EXE}") + script_path = _resolve_script(script) + + logpath = _config_path(_LOG_NAME) + try: + os.remove(logpath) + except OSError: + pass + + match = _window_match() + pid = _terminal_pid() + # AutoIt argv: match, pid, extra..., logpath + au3_args = [match, str(pid), *extra_args, logpath] + log.info("AutoIt: launching %s (match=%s, pid=%d, use_runas=%s)", + script, match, pid, use_runas) + + # Hold the machine-wide GUI mutex for the whole run so no other terminal's + # apply steals focus mid-type. + with _gui_lock(): + dbg = "" + if use_runas: + # MT5 runs elevated; UIPI blocks a non-elevated process from sending + # it input. Launch elevated (async — no exit code) and poll the log. + ps_args = ",".join("'%s'" % a for a in [script_path, *au3_args]) + ps = ( + f"Start-Process '{AUTOIT_EXE}' -ArgumentList {ps_args} " + "-Verb RunAs -WindowStyle Hidden" + ) + subprocess.Popen(["powershell", "-Command", ps]) + status, txt = _wait_result(logpath, timeout) + else: + # Direct, blocking — captures exit code + stderr for diagnostics. + cmd = [AUTOIT_EXE, script_path, *au3_args] + try: + proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) + dbg = f"[rc={proc.returncode} stderr={proc.stderr.strip()[:300]}]" + except subprocess.TimeoutExpired: + dbg = "[rc=TIMEOUT]" + try: + with open(logpath, "r", encoding="utf-8", errors="ignore") as f: + txt = f.read() + except OSError: + txt = "" + m = re.search(r"RESULT=(\w+)", txt) + status = m.group(1) if m else "NORESULT" + + txt = (txt + "\n" + dbg).strip() + log.info("AutoIt: %s -> RESULT=%s %s", script, status, dbg) + if status not in ("OK",): + log.warning("AutoIt %s log tail:\n%s", script, txt[-800:]) + return status, txt + + +def apply_urls(urls: list[str], timeout: float = 120, use_runas: bool = False) -> tuple[str, str]: + """Set the given allowlist in the running terminal via the Options dialog. + Returns (status, autoit_log). status == 'OK' on success.""" + urlfile = _config_path(_URLS_NAME) + os.makedirs(os.path.dirname(urlfile), exist_ok=True) + with open(urlfile, "w", encoding="utf-8") as f: + f.write("\n".join(urls)) + return _run_script("set_webrequest.au3", [urlfile], timeout, use_runas) + + +def run_named(script: str, timeout: float = 60, use_runas: bool = False) -> tuple[str, str]: + """Dev helper: run an arbitrary repo-shipped .au3 (e.g. the inspector).""" + return _run_script(script, [], timeout, use_runas) diff --git a/mt5api/chartctl/command.py b/mt5api/chartctl/command.py new file mode 100644 index 00000000..6a79700c --- /dev/null +++ b/mt5api/chartctl/command.py @@ -0,0 +1,78 @@ +"""Imperative one-shot command channel (screenshot, forced refresh). + +Deploy/stop are NOT commands — they are desired-state edits handled by +registry.py. This channel exists only for operations that produce a +side-effect artifact rather than converge state. + +Protocol: API writes command.json {command_id, action, ...}; loader +executes, writes command_result.json {command_id, status, ...}, deletes +command.json. API polls for the result up to a timeout. One command in +flight at a time, serialized by a lock (matching the loader's +one-command contract). +""" +import json +import os +import secrets +import threading +import time + +from mt5api.chartctl import paths +from mt5api.config import CHARTCTL_COMMAND_TIMEOUT_SECONDS + +_LOCK = threading.Lock() + + +class LoaderTimeout(TimeoutError): + pass + + +class LoaderBusy(RuntimeError): + pass + + +def run_command(action: str, payload: dict | None = None, + timeout: float | None = None) -> dict: + """Write a command, block until its result arrives or timeout.""" + timeout = timeout or CHARTCTL_COMMAND_TIMEOUT_SECONDS + if not _LOCK.acquire(blocking=False): + raise LoaderBusy("another command is in flight") + try: + paths.ensure_dirs() + if os.path.exists(paths.COMMAND_PATH): + # Stale command from a crashed request: clear it. + _try_remove(paths.COMMAND_PATH) + _try_remove(paths.COMMAND_RESULT_PATH) + + command_id = "cmd_" + secrets.token_hex(4) + body = {"command_id": command_id, "action": action} + body.update(payload or {}) + paths.atomic_write_text(paths.COMMAND_PATH, + json.dumps(body, indent=2, sort_keys=True)) + + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + result = _read_result() + if result and result.get("command_id") == command_id: + _try_remove(paths.COMMAND_RESULT_PATH) + return result + time.sleep(0.25) + _try_remove(paths.COMMAND_PATH) + raise LoaderTimeout( + f"loader did not answer '{action}' within {timeout:.0f}s") + finally: + _LOCK.release() + + +def _read_result() -> dict | None: + try: + with open(paths.COMMAND_RESULT_PATH, "r", encoding="utf-8") as handle: + return json.load(handle) + except (OSError, ValueError): + return None + + +def _try_remove(path: str) -> None: + try: + os.remove(path) + except OSError: + pass diff --git a/mt5api/chartctl/paths.py b/mt5api/chartctl/paths.py new file mode 100644 index 00000000..15995ec9 --- /dev/null +++ b/mt5api/chartctl/paths.py @@ -0,0 +1,82 @@ +"""Directory layout and filename safety for chartctl. + +All chartctl writes are confined to four roots under TERMINAL_DIR: + + MQL5/Experts/Uploaded/ staged .ex5 (shared with the backtest feature) + chartctl/sets/ staged .set parameter files + chartctl/registry.json API-side desired-state registry + MQL5/Files/chartctl/ the EA-visible protocol directory, including + the generated per-deployment .tpl files — + ChartApplyTemplate only resolves paths under + the MQL5 dir (leading backslash) or relative + to the calling EX5, so templates must live here + +Every externally supplied filename passes safe_name() — same contract as +the backtest handler's _safe_basename, factored here so both artifact +endpoints and the tpl builder share one guard with one test matrix. +""" +import os +import re + +from mt5api.config import TERMINAL_DIR, ASSETS_DIR + +EXPERTS_DIR = os.path.join(TERMINAL_DIR, "MQL5", "Experts", "Uploaded") +SETS_DIR = os.path.join(TERMINAL_DIR, "chartctl", "sets") +REGISTRY_PATH = os.path.join(TERMINAL_DIR, "chartctl", "registry.json") +PROTOCOL_DIR = os.path.join(TERMINAL_DIR, "MQL5", "Files", "chartctl") +TEMPLATES_DIR = PROTOCOL_DIR +SCREENSHOTS_DIR = os.path.join(PROTOCOL_DIR, "shots") + +HOST_EXPERTS_DIR = os.path.join(ASSETS_DIR, "experts") +HOST_SETS_DIR = os.path.join(ASSETS_DIR, "sets") + +DESIRED_PATH = os.path.join(PROTOCOL_DIR, "desired.json") +OBSERVED_PATH = os.path.join(PROTOCOL_DIR, "observed.json") +COMMAND_PATH = os.path.join(PROTOCOL_DIR, "command.json") +COMMAND_RESULT_PATH = os.path.join(PROTOCOL_DIR, "command_result.json") + +# Windows-reserved characters plus anything that could smuggle a path. +_BAD_CHARS = re.compile(r'[<>:"/\\|?*\x00-\x1f]') + + +def ensure_dirs() -> None: + for d in (EXPERTS_DIR, SETS_DIR, PROTOCOL_DIR, TEMPLATES_DIR, + SCREENSHOTS_DIR, os.path.dirname(REGISTRY_PATH)): + os.makedirs(d, exist_ok=True) + + +def safe_name(name: str, field: str, required_ext: str | None = None) -> str: + """Validate an externally supplied filename. Returns the bare name. + + Rejects: empty, path separators, traversal, drive prefixes, UNC, + Windows-reserved characters, hidden dotfiles, and (optionally) a + wrong extension. Raises ValueError with a client-facing message. + """ + name = (name or "").strip() + if not name: + raise ValueError(f"{field}: filename is required") + if name != os.path.basename(name) or name in (".", ".."): + raise ValueError(f"{field}: path components are not allowed") + if _BAD_CHARS.search(name): + raise ValueError(f"{field}: illegal characters in filename") + if name.startswith("."): + raise ValueError(f"{field}: hidden files are not allowed") + if ".." in name: + raise ValueError(f"{field}: traversal sequences are not allowed") + if required_ext and not name.lower().endswith(required_ext): + raise ValueError(f"{field}: must end in {required_ext}") + return name + + +def atomic_write_text(path: str, text: str, encoding: str = "utf-8") -> None: + tmp = f"{path}.tmp" + with open(tmp, "w", encoding=encoding, newline="\r\n") as handle: + handle.write(text) + os.replace(tmp, path) + + +def atomic_write_bytes(path: str, data: bytes) -> None: + tmp = f"{path}.tmp" + with open(tmp, "wb") as handle: + handle.write(data) + os.replace(tmp, path) diff --git a/mt5api/chartctl/registry.py b/mt5api/chartctl/registry.py new file mode 100644 index 00000000..a910724d --- /dev/null +++ b/mt5api/chartctl/registry.py @@ -0,0 +1,274 @@ +"""Deployment registry: the API-side source of truth for desired state. + +Follows the backtest jobs.py pattern — persistent JSON, in-memory +write-through cache behind a lock, survives API restarts. Every mutation +bumps a monotonic `revision` and rewrites the EA-facing desired.json. + +Status model (derived, never stored as truth): + pending declared, loader has not confirmed it yet + running observed.json shows the expert live on a chart + degraded previously running, currently missing (reconciliation active) + failed loader reported a terminal error for this deployment + paused enabled=false in desired state +""" +from __future__ import annotations + +import json +import os +import secrets +import threading +import time +from datetime import datetime, timezone + +from mt5api.chartctl import paths +from mt5api.chartctl.tpl_builder import tpl_relative_name +from mt5api.config import ( + CHARTCTL_OBSERVED_STALE_SECONDS, + CHARTCTL_RECONCILE_HINT_SECONDS, +) +from mt5api.logger import log + +_LOCK = threading.Lock() +_STATE: dict | None = None # {"revision": int, "deployments": {id: {...}}} + +PROTOCOL_VERSION = 1 + + +def _now_iso() -> str: + return datetime.now(timezone.utc).replace(microsecond=0, tzinfo=None).isoformat() + "Z" + + +def new_id() -> str: + return "dep_" + secrets.token_hex(4) + + +def _empty_state() -> dict: + return {"revision": 0, "deployments": {}} + + +def _load_locked() -> dict: + global _STATE + if _STATE is not None: + return _STATE + if os.path.exists(paths.REGISTRY_PATH): + try: + with open(paths.REGISTRY_PATH, "r", encoding="utf-8") as handle: + _STATE = json.load(handle) + except (OSError, ValueError) as exc: + log.error("chartctl registry unreadable (%s) — starting empty; " + "corrupt file preserved as .bad", exc) + try: + os.replace(paths.REGISTRY_PATH, paths.REGISTRY_PATH + ".bad") + except OSError: + pass + _STATE = _empty_state() + else: + _STATE = _empty_state() + _STATE.setdefault("revision", 0) + _STATE.setdefault("deployments", {}) + return _STATE + + +def _persist_locked(state: dict) -> None: + paths.ensure_dirs() + paths.atomic_write_text(paths.REGISTRY_PATH, + json.dumps(state, indent=2, sort_keys=True)) + _write_desired_locked(state) + + +def _write_desired_locked(state: dict) -> None: + desired = { + "protocol": PROTOCOL_VERSION, + "revision": state["revision"], + "updated_at": _now_iso(), + "reconcile_interval": CHARTCTL_RECONCILE_HINT_SECONDS, + "deployments": [ + { + "id": dep["id"], + "expert": dep["expert_name"], + "template": tpl_relative_name(dep["id"]), + "symbol": dep["symbol"], + "timeframe": dep["timeframe"], + "enabled": dep.get("enabled", True), + } + for dep in sorted(state["deployments"].values(), + key=lambda d: d["created_at"]) + ], + } + paths.atomic_write_text(paths.DESIRED_PATH, + json.dumps(desired, indent=2, sort_keys=True)) + + +def list_deployments() -> list[dict]: + with _LOCK: + state = _load_locked() + return [dict(d) for d in state["deployments"].values()] + + +def get_deployment(dep_id: str) -> dict | None: + with _LOCK: + state = _load_locked() + dep = state["deployments"].get(dep_id) + return dict(dep) if dep else None + + +def add_deployment(*, expert_file: str, expert_name: str, set_file: str | None, + symbol: str, timeframe: str, enabled: bool = True) -> dict: + with _LOCK: + state = _load_locked() + for other in state["deployments"].values(): + if (other.get("enabled", True) and enabled + and other["symbol"].upper() == symbol.upper() + and other["timeframe"].upper() == timeframe.upper()): + raise DuplicateChart( + f"enabled deployment {other['id']} already targets " + f"{symbol} {timeframe}") + dep = { + "id": new_id(), + "expert_file": expert_file, + "expert_name": expert_name, + "set_file": set_file, + "symbol": symbol, + "timeframe": timeframe, + "enabled": enabled, + "created_at": _now_iso(), + "updated_at": _now_iso(), + } + state["deployments"][dep["id"]] = dep + state["revision"] += 1 + _persist_locked(state) + log.info("chartctl deployment %s created: %s %s %s (rev=%d)", + dep["id"], expert_name, symbol, timeframe, state["revision"]) + return dict(dep) + + +def update_deployment(dep_id: str, **changes) -> dict: + with _LOCK: + state = _load_locked() + dep = state["deployments"].get(dep_id) + if dep is None: + raise KeyError(dep_id) + dep.update(changes) + dep["updated_at"] = _now_iso() + state["revision"] += 1 + _persist_locked(state) + log.info("chartctl deployment %s updated (%s) rev=%d", + dep_id, ", ".join(changes), state["revision"]) + return dict(dep) + + +def remove_deployment(dep_id: str) -> dict: + with _LOCK: + state = _load_locked() + dep = state["deployments"].pop(dep_id, None) + if dep is None: + raise KeyError(dep_id) + state["revision"] += 1 + _persist_locked(state) + log.info("chartctl deployment %s removed (rev=%d)", + dep_id, state["revision"]) + return dep + + +def expert_in_use(expert_file: str) -> bool: + with _LOCK: + state = _load_locked() + return any(d["expert_file"] == expert_file + for d in state["deployments"].values()) + + +def current_revision() -> int: + with _LOCK: + return _load_locked()["revision"] + + +def rewrite_desired() -> None: + """Force-regenerate desired.json from the registry (bumps revision so + the loader re-reads even if content is identical — used by + /deployments/reconcile).""" + with _LOCK: + state = _load_locked() + state["revision"] += 1 + _persist_locked(state) + + +class DuplicateChart(ValueError): + pass + + +# ── Observed-state reading & merge ────────────────────────────────── + +def read_observed() -> tuple[dict | None, bool]: + """Return (observed_dict_or_None, is_stale).""" + try: + with open(paths.OBSERVED_PATH, "r", encoding="utf-8") as handle: + observed = json.load(handle) + except (OSError, ValueError): + return None, True + stale = True + try: + mtime = os.path.getmtime(paths.OBSERVED_PATH) + stale = (time.time() - mtime) > CHARTCTL_OBSERVED_STALE_SECONDS + except OSError: + pass + return observed, stale + + +def loader_alive(observed: dict | None, stale: bool) -> bool: + return bool(observed) and not stale and bool(observed.get("loader")) + + +def merged_view() -> dict: + """The GET /deployments payload: desired ⋈ observed.""" + observed, stale = read_observed() + obs_by_id: dict[str, dict] = {} + err_by_id: dict[str, dict] = {} + if observed: + for entry in observed.get("deployments") or []: + if entry.get("id"): + obs_by_id[entry["id"]] = entry + for entry in observed.get("errors") or []: + if entry.get("id"): + err_by_id[entry["id"]] = entry + + applied_revision = (observed or {}).get("loader", {}).get("applied_revision") + revision = current_revision() + items = [] + all_converged = True + for dep in list_deployments(): + obs = obs_by_id.get(dep["id"]) + err = err_by_id.get(dep["id"]) + status = _derive_status(dep, obs, err, stale) + if status not in ("running", "paused"): + all_converged = False + items.append({ + "id": dep["id"], + "desired": dep, + "observed": obs, + "error": err, + "status": status, + }) + + return { + "revision": revision, + "applied_revision": applied_revision, + "converged": all_converged and applied_revision == revision and not stale, + "observed_stale": stale, + "loader": (observed or {}).get("loader"), + "deployments": items, + } + + +def _derive_status(dep: dict, obs: dict | None, err: dict | None, + stale: bool) -> str: + if not dep.get("enabled", True): + return "paused" + if err and (obs is None or obs.get("status") != "running"): + return "failed" + if obs and obs.get("status") == "running" and not stale: + return "running" + if obs and obs.get("status") == "running" and stale: + return "degraded" + if obs and obs.get("status"): + return str(obs["status"]) + return "pending" diff --git a/mt5api/chartctl/setparse.py b/mt5api/chartctl/setparse.py new file mode 100644 index 00000000..8a874083 --- /dev/null +++ b/mt5api/chartctl/setparse.py @@ -0,0 +1,63 @@ +"""Parse MT5 .set parameter files into structured inputs. + +The mirror of backtest/set_builder.py (JSON -> .set); this goes .set -> +structured list. MT5 saves .set as UTF-16-LE with BOM; hand-written ones +are often UTF-8 or ASCII. Grammar per line: + + Name=value plain input + Name=value||start||step||stop||Y input with optimization metadata + ; comment + +The optimization tail is irrelevant for live deployment — the leading +value is what the terminal applies — but we keep it so clients can +round-trip and diff files losslessly. +""" + + +def _decode(data: bytes) -> str: + # Never blind-try utf-16: the codec "succeeds" on any even-length + # ASCII input by pairing bytes into CJK garbage, which then parses to + # zero inputs — and the deployment silently runs on EA defaults. + # Decide by BOM (MT5 exports carry one), then by embedded NULs + # (BOM-less UTF-16), then plain 8-bit. + if data[:2] in (b"\xff\xfe", b"\xfe\xff"): + return data.decode("utf-16") + if b"\x00" in data: + try: + return data.decode("utf-16-le") + except (UnicodeDecodeError, UnicodeError): + pass + for enc in ("utf-8-sig", "utf-8"): + try: + return data.decode(enc) + except UnicodeDecodeError: + continue + return data.decode("latin-1") + + +def parse_set_bytes(data: bytes) -> list[dict]: + return parse_set_text(_decode(data)) + + +def parse_set_text(text: str) -> list[dict]: + """Return [{name, value, optimize?, start?, step?, stop?}, ...].""" + inputs: list[dict] = [] + for raw_line in text.splitlines(): + line = raw_line.strip().lstrip("\ufeff") + if not line or line.startswith(";") or line.startswith("#"): + continue + if "=" not in line: + continue + name, _, rhs = line.partition("=") + name = name.strip() + if not name: + continue + parts = rhs.split("||") + entry: dict = {"name": name, "value": parts[0].strip()} + if len(parts) >= 5: + entry["start"] = parts[1].strip() + entry["step"] = parts[2].strip() + entry["stop"] = parts[3].strip() + entry["optimize"] = parts[4].strip().upper() == "Y" + inputs.append(entry) + return inputs diff --git a/mt5api/chartctl/tpl_builder.py b/mt5api/chartctl/tpl_builder.py new file mode 100644 index 00000000..e0cadbe6 --- /dev/null +++ b/mt5api/chartctl/tpl_builder.py @@ -0,0 +1,100 @@ +"""Generate per-deployment MT5 chart templates (.tpl). + +A minimal template whose block carries the expert path, flags, +and the full list translated from a parsed .set file. Applying +the template to a chart attaches the expert with those inputs — the only +programmatic attach path MT5 offers (via ChartApplyTemplate from MQL5). + +Attribution: the loader sets a chart comment `chartctl:` at attach +time — that's the authoritative cross-restart marker (MT5 persists the +comment with the saved chart). We ALSO stamp the id into the template's +`description` field and a reserved __chartctl_id input purely for human +forensics / grepping raw .tpl files; the loader does not depend on being +able to read another expert's inputs (MQL5 can't), so the comment is the +real mechanism. + +Encoding: modern terminal builds save templates as UTF-16-LE with BOM +and CRLF line endings; older builds accept the same. We always emit +UTF-16-LE + BOM. Every template carries a comment header with the +generator version and the terminal build it was generated for, so a +misbehaving attach can be forensically matched to an encoding profile. + +Known-risk note (spec §8): value encoding for enums/booleans has +build-specific quirks. Values are passed through verbatim from the .set +file, which is the safest policy: the .set was produced by the same +terminal family that will consume the template. Golden-file tests pin +the exact bytes per build. +""" +from mt5api.chartctl.paths import TEMPLATES_DIR, atomic_write_bytes + +import os + +GENERATOR_VERSION = "1.0.0" + +# EA flags observed in terminal-saved templates: allow live trading + +# allow DLL confirmations off + enabled. 343 = common "enabled, algo +# trading allowed" profile seen across builds; kept as a constant so a +# build-specific override is one line. +EXPERT_FLAGS = 343 + +# Reserved input name the loader EA reads for attribution. Harmless for +# the target expert: MT5 ignores unknown inputs in templates. +ID_INPUT = "__chartctl_id" + + +def build_tpl_text(*, deployment_id: str, expert_name: str, + expert_rel_path: str, inputs: list[dict], + terminal_build: int | None = None) -> str: + """Compose template text. expert_rel_path like 'Experts\\Uploaded\\X.ex5'.""" + lines = [ + "", + f"description=chartctl:{deployment_id} gen={GENERATOR_VERSION}" + + (f" build={terminal_build}" if terminal_build else ""), + "shift=1", + "autoscroll=1", + "ohlc=1", + "one_click=0", + "", + f"name={expert_name}", + f"path={expert_rel_path}", + f"flags={EXPERT_FLAGS}", + "expertmode=1", + "", + f"{ID_INPUT}={deployment_id}", + ] + for entry in inputs: + lines.append(f"{entry['name']}={entry['value']}") + lines += [ + "", + "", + "", + "", + ] + return "\r\n".join(lines) + + +def tpl_filename(deployment_id: str) -> str: + return f"{deployment_id}.tpl" + + +def tpl_relative_name(deployment_id: str) -> str: + """The name the loader passes to ChartApplyTemplate. The leading + backslash makes MT5 resolve it against \\MQL5 — the only + host-EA-independent root ChartApplyTemplate searches (without it the + path is relative to the calling EX5's folder, which varies per host + EA and yields err 5019 file-not-found).""" + return f"\\Files\\chartctl\\{tpl_filename(deployment_id)}" + + +def write_tpl(*, deployment_id: str, expert_name: str, expert_rel_path: str, + inputs: list[dict], terminal_build: int | None = None) -> str: + text = build_tpl_text( + deployment_id=deployment_id, + expert_name=expert_name, + expert_rel_path=expert_rel_path, + inputs=inputs, + terminal_build=terminal_build, + ) + path = os.path.join(TEMPLATES_DIR, tpl_filename(deployment_id)) + atomic_write_bytes(path, b"\xff\xfe" + text.encode("utf-16-le")) + return path diff --git a/mt5api/chartctl/webrequest.py b/mt5api/chartctl/webrequest.py new file mode 100644 index 00000000..620d1364 --- /dev/null +++ b/mt5api/chartctl/webrequest.py @@ -0,0 +1,208 @@ +"""WebRequest allowlist manager for a single terminal. + +MT5's WebRequest allowed-URL list lives ONLY in ``/Config/common.ini`` +(``[Experts] WebRequest=1`` + ``WebRequestUrl=``), a UTF-16LE file that MT5 +reads at terminal startup. There is no live-reload, so applying a change requires a +terminal restart, and ``start.bat`` deletes ``common.ini`` on every boot — so the +authoritative desired state is kept in a sibling ``webrequest.json`` that survives +reboots and is re-applied to ``common.ini`` at two points: + + * boot — ``scripts/config_helper.py`` write_ini, right after start.bat deletes + common.ini and before the terminal launches; + * runtime — ``mt5client.restart_terminal``, after the terminal is killed and + before it is relaunched (MT5 rewrites common.ini on exit, so the + write must happen while the terminal is down). + +First adoption migrates from whatever the terminal currently has (decode the +existing ``WebRequestUrl=`` blob) so manually-configured URLs are preserved. + +The blob codec (fully reverse-engineered) lives in +``scripts/webrequest_allowlist_codec.py`` and is loaded here by path so it stays a +single dependency-free source of truth, importable from both the app and the boot +helper. +""" +from __future__ import annotations + +import importlib.util +import json +import os + +DESIRED_FILENAME = "webrequest.json" +_BOM = b"\xff\xfe" + +# --- load the standalone codec by path (no package coupling) --- +_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +_CODEC_PATH = os.path.join(_REPO_ROOT, "scripts", "webrequest_allowlist_codec.py") +_spec = importlib.util.spec_from_file_location("webrequest_allowlist_codec", _CODEC_PATH) +_codec = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_codec) +encode_urls = _codec.encode_urls +decode_blob = _codec.decode_blob + + +# ── paths ──────────────────────────────────────────────────────────── +def config_dir(terminal_dir: str) -> str: + return os.path.join(terminal_dir, "Config") + + +def common_ini_path(cfg_dir: str) -> str: + return os.path.join(cfg_dir, "common.ini") + + +def desired_path(cfg_dir: str) -> str: + return os.path.join(cfg_dir, DESIRED_FILENAME) + + +# ── url hygiene ────────────────────────────────────────────────────── +def clean_urls(urls) -> list[str]: + """Trim, drop blanks/dupes, keep only http(s) URLs without the ';' delimiter.""" + out: list[str] = [] + if not isinstance(urls, (list, tuple)): + return out + for u in urls: + if not isinstance(u, str): + continue + u = u.strip() + if not u or ";" in u or any(ord(c) < 0x20 for c in u): + continue + low = u.lower() + if not (low.startswith("http://") or low.startswith("https://")): + continue + if u not in out: + out.append(u) + return out + + +# ── desired-state store (survives reboots) ─────────────────────────── +def load_desired(cfg_dir: str) -> list[str] | None: + """Return the persisted URL list, or None if this terminal has none set.""" + path = desired_path(cfg_dir) + if not os.path.exists(path): + return None + try: + with open(path, "r", encoding="utf-8") as f: + data = json.load(f) + except (OSError, ValueError): + return None + urls = data.get("urls") if isinstance(data, dict) else data + return clean_urls(urls) if isinstance(urls, list) else None + + +def save_desired(cfg_dir: str, urls: list[str]) -> None: + os.makedirs(cfg_dir, exist_ok=True) + path = desired_path(cfg_dir) + tmp = path + ".tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump({"urls": clean_urls(urls)}, f, indent=2) + os.replace(tmp, path) + + +# ── common.ini (UTF-16LE + BOM + CRLF) ─────────────────────────────── +def _read_lines(path: str) -> list[str]: + if not os.path.exists(path): + return [] + try: + with open(path, "rb") as f: + raw = f.read() + except OSError: + return [] + text = raw.decode("utf-16", errors="ignore") # strips BOM + return text.replace("\r\n", "\n").replace("\r", "\n").split("\n") + + +def _write_lines(path: str, lines: list[str]) -> None: + os.makedirs(os.path.dirname(path), exist_ok=True) + text = "\r\n".join(lines) + data = _BOM + text.encode("utf-16-le") + tmp = path + ".tmp" + with open(tmp, "wb") as f: + f.write(data) + os.replace(tmp, path) + + +def read_current_urls(cfg_dir: str) -> list[str]: + """Decode the URL list from an existing common.ini (migrate-from-current).""" + blob = "" + in_experts = False + for line in _read_lines(common_ini_path(cfg_dir)): + s = line.strip() + if s.startswith("[") and s.endswith("]"): + in_experts = s.lower() == "[experts]" + continue + if in_experts and "=" in s and s.split("=", 1)[0].strip().lower() == "webrequesturl": + blob = s.split("=", 1)[1].strip() + break + if not blob: + return [] + try: + return clean_urls(decode_blob(blob)) + except Exception: + return [] + + +def effective_urls(cfg_dir: str) -> list[str]: + """Desired state if set, else whatever the terminal currently holds.""" + desired = load_desired(cfg_dir) + return desired if desired is not None else read_current_urls(cfg_dir) + + +def write_common_ini(cfg_dir: str, urls: list[str]) -> None: + """Set ``[Experts] WebRequest=1`` + ``WebRequestUrl=`` in common.ini, + preserving every other line/section. Creates the file (and [Experts]) if + absent. Encoding stays UTF-16LE+BOM+CRLF, as MT5 writes it.""" + urls = clean_urls(urls) + kv = {"WebRequest": "1", "WebRequestUrl": encode_urls(urls)} + lines = _read_lines(common_ini_path(cfg_dir)) + out: list[str] = [] + in_experts = False + written: set[str] = set() + has_experts = any(l.strip().lower() == "[experts]" for l in lines) + + def flush_missing(): + for key, val in kv.items(): + if key not in written: + out.append(f"{key}={val}") + written.add(key) + + for line in lines: + s = line.strip() + if s.startswith("[") and s.endswith("]"): + if in_experts: + flush_missing() + in_experts = s.lower() == "[experts]" + out.append(line) + continue + if in_experts and "=" in s: + key = s.split("=", 1)[0].strip() + for real in kv: + if key.lower() == real.lower(): + if real not in written: + out.append(f"{real}={kv[real]}") + written.add(real) + break + else: + out.append(line) + continue + out.append(line) + + if in_experts: + flush_missing() + if not has_experts: + out.append("[Experts]") + flush_missing() + # drop trailing blank lines then keep exactly one terminator via join + while out and out[-1] == "": + out.pop() + _write_lines(common_ini_path(cfg_dir), out) + + +def apply_from_desired(terminal_dir: str) -> int | None: + """Regenerate common.ini from the desired file. Returns URL count, or None + if there is no desired state (in which case common.ini is left untouched). + Safe to call while the terminal is stopped.""" + cfg_dir = config_dir(terminal_dir) + desired = load_desired(cfg_dir) + if desired is None: + return None + write_common_ini(cfg_dir, desired) + return len(desired) diff --git a/mt5api/config.py b/mt5api/config.py index 1338f4c5..a78a826d 100644 --- a/mt5api/config.py +++ b/mt5api/config.py @@ -280,6 +280,64 @@ def load_terminal_config(): TERMINAL_DIR = os.path.dirname(TERMINAL_PATH) INI_FILE = os.path.join(TERMINAL_DIR, "mt5start.ini") + +# Chart Deployments (chartctl) — live-mode only feature. Global default from +# config.yaml `chartctl:` block; per-terminal `chartctl: false` in terminals[] +# overrides it. Backtest-mode terminals never enable it: there is no running +# terminal64.exe to manage charts on. +_chartctl_cfg = load_yaml_config().get("chartctl") or {} +_chartctl_terminal_override = _terminal_config.get("chartctl") +# Opt-IN. Defaulting this on would auto-attach the loader EA to every live +# terminal of any install that upgraded without asking for it - a fleet-wide +# behaviour change nobody opted into. Absent chartctl block = unchanged API. +_chartctl_global_enabled = bool(_chartctl_cfg.get("enabled", False)) +CHARTCTL_ENABLED = ( + MODE == "live" + and _chartctl_global_enabled + and (_chartctl_terminal_override is not False) +) +def _chartctl_seconds(raw, default_text: str, floor: int) -> int: + """A duration from the chartctl block, never below `floor`. + + `or ` already covers absent and zero, but NOT negative: + parse_duration_to_seconds accepts "-5s" on purpose, because west-of-UTC + broker offsets need the sign. Without the floor a negative is taken + literally - a stale window that calls every observation stale, or a hint + interval that is permanently due. Both present as a broken loader rather + than as a bad setting. + + Clamped rather than refused, deliberately. This module is imported by the + whole API, so raising on an optional feature's tuning value would stop + trading and backtesting too. vm-watchdog makes the opposite call for the + opposite reason: it holds the Docker socket, so it refuses to start rather + than run on values nobody chose. + """ + parsed = parse_duration_to_seconds(str(raw or default_text)) + return max(floor, parsed or parse_duration_to_seconds(default_text)) + + +def _chartctl_bytes(raw, default: int, floor: int) -> int: + """A byte cap from the chartctl block, never below `floor`. + + A negative cap makes read(cap + 1) return nothing and every upload fail the + length check, reporting a limit of -1 bytes. + """ + return max(floor, int(raw or default)) + + +CHARTCTL_RECONCILE_HINT_SECONDS = _chartctl_seconds( + _chartctl_cfg.get("reconcile_hint_interval"), "5s", 1 +) +CHARTCTL_OBSERVED_STALE_SECONDS = _chartctl_seconds( + _chartctl_cfg.get("observed_stale_after"), "60s", 1 +) +CHARTCTL_COMMAND_TIMEOUT_SECONDS = _chartctl_seconds( + _chartctl_cfg.get("command_timeout"), "30s", 1 +) +# Floor of 1 KiB: a cap below that rejects every real .ex5 and .set. +CHARTCTL_MAX_UPLOAD_BYTES = _chartctl_bytes( + _chartctl_cfg.get("max_upload_bytes"), 16 * 1024 * 1024, 1024 +) IDENTITY = make_identity(BROKER, ACCOUNT, INSTANCE) LOG_DIR = os.path.join(BASE_DIR, "logs") FULL_LOG = os.path.join(LOG_DIR, "full.log") diff --git a/mt5api/handlers/chartctl.py b/mt5api/handlers/chartctl.py new file mode 100644 index 00000000..8dad4bc5 --- /dev/null +++ b/mt5api/handlers/chartctl.py @@ -0,0 +1,394 @@ +"""Chart Deployments REST handlers. + +Lock-free by design: nothing here touches the MT5 SDK, so these routes +never queue behind the process-wide MT5 lock (mt5client.py). Everything +is file I/O against TERMINAL_DIR plus the loader-EA file protocol. +""" +import hashlib +import os + +from flask import jsonify, request, send_file + +from mt5api.chartctl import command as cmd +from mt5api.chartctl import paths, registry +from mt5api.chartctl.setparse import parse_set_bytes +from mt5api.chartctl.tpl_builder import write_tpl +from mt5api.config import CHARTCTL_MAX_UPLOAD_BYTES +from mt5api.logger import log + +_VALID_TIMEFRAMES = frozenset({ + "M1", "M2", "M3", "M4", "M5", "M6", "M10", "M12", "M15", "M20", "M30", + "H1", "H2", "H3", "H4", "H6", "H8", "H12", "D1", "W1", "MN1", +}) + + +def _err(status: int, code: str, message: str): + return jsonify({"error": message, "code": code}), status + + +def _sha256(path: str) -> str: + digest = hashlib.sha256() + with open(path, "rb") as handle: + for chunk in iter(lambda: handle.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _list_dir(directory: str, ext: str, source: str) -> list[dict]: + items = [] + if not os.path.isdir(directory): + return items + for name in sorted(os.listdir(directory)): + if not name.lower().endswith(ext): + continue + full = os.path.join(directory, name) + if not os.path.isfile(full): + continue + stat = os.stat(full) + items.append({ + "name": name, + "size": stat.st_size, + "sha256": _sha256(full), + "modified_at": int(stat.st_mtime), + "source": source, + }) + return items + + +def _read_upload(field: str, required_ext: str) -> tuple[str, bytes]: + upload = request.files.get(field) + if upload is None or not upload.filename: + raise ValueError(f"Missing form file: {field}") + name = paths.safe_name(upload.filename, field, required_ext) + data = upload.stream.read(CHARTCTL_MAX_UPLOAD_BYTES + 1) + if len(data) > CHARTCTL_MAX_UPLOAD_BYTES: + raise ValueError( + f"{field}: file exceeds {CHARTCTL_MAX_UPLOAD_BYTES} bytes") + if not data: + raise ValueError(f"{field}: file is empty") + return name, data + + +# ── Artifacts: experts ─────────────────────────────────────────────── + +def upload_expert(): + try: + name, data = _read_upload("expert", ".ex5") + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + paths.ensure_dirs() + dest = os.path.join(paths.EXPERTS_DIR, name) + new_hash = hashlib.sha256(data).hexdigest() + overwrite = (request.args.get("overwrite", "false").lower() == "true") + if os.path.exists(dest) and not overwrite: + if _sha256(dest) == new_hash: + return jsonify({"name": name, "sha256": new_hash, + "skipped": True}) + return _err(409, "EXISTS", + f"{name} exists with different content; " + "pass ?overwrite=true to replace") + paths.atomic_write_bytes(dest, data) + log.info("chartctl expert staged: %s (%d bytes, %s)", + name, len(data), new_hash[:12]) + return jsonify({"name": name, "sha256": new_hash, "size": len(data)}), 201 + + +def list_experts(): + return jsonify({ + "experts": _list_dir(paths.EXPERTS_DIR, ".ex5", "uploaded") + + _list_dir(paths.HOST_EXPERTS_DIR, ".ex5", "host"), + }) + + +def delete_expert(name): + try: + name = paths.safe_name(name, "expert", ".ex5") + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + if os.path.exists(os.path.join(paths.HOST_EXPERTS_DIR, name)) and \ + not os.path.exists(os.path.join(paths.EXPERTS_DIR, name)): + return _err(403, "HOST_ASSET", "host-managed assets are read-only") + target = os.path.join(paths.EXPERTS_DIR, name) + if not os.path.exists(target): + return _err(404, "ARTIFACT_NOT_FOUND", f"{name} is not staged") + if registry.expert_in_use(name): + return _err(409, "IN_USE", + f"{name} is referenced by an existing deployment") + os.remove(target) + return jsonify({"deleted": name}) + + +# ── Artifacts: sets ────────────────────────────────────────────────── + +def upload_set(): + try: + name, data = _read_upload("set", ".set") + inputs = parse_set_bytes(data) + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + paths.ensure_dirs() + dest = os.path.join(paths.SETS_DIR, name) + paths.atomic_write_bytes(dest, data) + log.info("chartctl set staged: %s (%d inputs)", name, len(inputs)) + return jsonify({"name": name, + "sha256": hashlib.sha256(data).hexdigest(), + "inputs": inputs}), 201 + + +def list_sets(): + return jsonify({ + "sets": _list_dir(paths.SETS_DIR, ".set", "uploaded") + + _list_dir(paths.HOST_SETS_DIR, ".set", "host"), + }) + + +def get_set(name): + try: + name = paths.safe_name(name, "set", ".set") + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + path = _resolve_set(name) + if path is None: + return _err(404, "ARTIFACT_NOT_FOUND", f"{name} is not staged") + with open(path, "rb") as handle: + data = handle.read() + return jsonify({"name": name, "inputs": parse_set_bytes(data)}) + + +def _resolve_set(name: str) -> str | None: + for base in (paths.SETS_DIR, paths.HOST_SETS_DIR): + candidate = os.path.join(base, name) + if os.path.isfile(candidate): + return candidate + return None + + +def _resolve_expert(name: str) -> str | None: + for base in (paths.EXPERTS_DIR, paths.HOST_EXPERTS_DIR): + candidate = os.path.join(base, name) + if os.path.isfile(candidate): + return candidate + return None + + +# ── Deployments ────────────────────────────────────────────────────── + +def _terminal_build() -> int | None: + """Best-effort build stamp for the template header. Never blocks: + peeks at cached terminal info without taking the MT5 lock.""" + try: + from mt5api import mt5client + info = getattr(mt5client, "LAST_TERMINAL_INFO", None) + if info and getattr(info, "build", None): + return int(info.build) + except Exception: # noqa: BLE001 — stamp is cosmetic, never fail on it + pass + return None + + +def _materialize_tpl(dep: dict) -> None: + inputs: list[dict] = [] + if dep.get("set_file"): + set_path = _resolve_set(dep["set_file"]) + if set_path is None: + raise ValueError(f"set file {dep['set_file']} disappeared") + with open(set_path, "rb") as handle: + inputs = parse_set_bytes(handle.read()) + # Optimization tails are meaningless on a live chart. + inputs = [{"name": i["name"], "value": i["value"]} for i in inputs] + write_tpl( + deployment_id=dep["id"], + expert_name=dep["expert_name"], + expert_rel_path=f"Experts\\Uploaded\\{dep['expert_file']}", + inputs=inputs, + terminal_build=_terminal_build(), + ) + + +def create_deployment(): + body = request.get_json(silent=True) or {} + try: + expert_file = paths.safe_name(body.get("expert", ""), "expert", ".ex5") + set_file = None + if body.get("set"): + set_file = paths.safe_name(body["set"], "set", ".set") + symbol = str(body.get("symbol", "")).strip() + timeframe = str(body.get("timeframe", "")).strip().upper() + enabled = bool(body.get("enabled", True)) + if not symbol: + raise ValueError("symbol is required") + if timeframe not in _VALID_TIMEFRAMES: + raise ValueError(f"timeframe must be one of " + f"{sorted(_VALID_TIMEFRAMES)}") + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + + expert_path = _resolve_expert(expert_file) + if expert_path is None: + return _err(404, "ARTIFACT_NOT_FOUND", + f"expert {expert_file} is not staged — upload it first") + if expert_path.startswith(paths.HOST_EXPERTS_DIR): + # Host asset: mirror into Uploaded/ so the terminal can load it. + paths.ensure_dirs() + with open(expert_path, "rb") as handle: + paths.atomic_write_bytes( + os.path.join(paths.EXPERTS_DIR, expert_file), handle.read()) + if set_file and _resolve_set(set_file) is None: + return _err(404, "ARTIFACT_NOT_FOUND", + f"set {set_file} is not staged — upload it first") + + expert_name = expert_file[:-4] if expert_file.lower().endswith(".ex5") \ + else expert_file + try: + dep = registry.add_deployment( + expert_file=expert_file, expert_name=expert_name, + set_file=set_file, symbol=symbol, timeframe=timeframe, + enabled=enabled) + except registry.DuplicateChart as exc: + return _err(409, "DUPLICATE_CHART", str(exc)) + + try: + _materialize_tpl(dep) + except ValueError as exc: + registry.remove_deployment(dep["id"]) + return _err(500, "TPL_GENERATION_FAILED", str(exc)) + + return jsonify({"id": dep["id"], "status": "pending", + "deployment": dep}), 202 + + +def list_deployments(): + return jsonify(registry.merged_view()) + + +def get_deployment(dep_id): + view = registry.merged_view() + for item in view["deployments"]: + if item["id"] == dep_id: + item["revision"] = view["revision"] + item["observed_stale"] = view["observed_stale"] + return jsonify(item) + return _err(404, "NOT_FOUND", f"deployment {dep_id} does not exist") + + +def patch_deployment(dep_id): + body = request.get_json(silent=True) or {} + changes: dict = {} + try: + if "set" in body: + changes["set_file"] = ( + paths.safe_name(body["set"], "set", ".set") + if body["set"] else None) + if changes["set_file"] and _resolve_set(changes["set_file"]) is None: + return _err(404, "ARTIFACT_NOT_FOUND", + f"set {changes['set_file']} is not staged") + if "enabled" in body: + changes["enabled"] = bool(body["enabled"]) + except ValueError as exc: + return _err(400, "BAD_REQUEST", str(exc)) + if not changes: + return _err(400, "BAD_REQUEST", + "nothing to change: pass 'set' and/or 'enabled'") + try: + dep = registry.update_deployment(dep_id, **changes) + except KeyError: + return _err(404, "NOT_FOUND", f"deployment {dep_id} does not exist") + if "set_file" in changes: + try: + _materialize_tpl(dep) + except ValueError as exc: + return _err(500, "TPL_GENERATION_FAILED", str(exc)) + return jsonify({"id": dep_id, "deployment": dep}) + + +def delete_deployment(dep_id): + try: + dep = registry.remove_deployment(dep_id) + except KeyError: + return _err(404, "NOT_FOUND", f"deployment {dep_id} does not exist") + # Template file is left on disk until the loader confirms detach; the + # loader clears the chart because the deployment vanished from + # desired.json. Cleanup of orphaned .tpl files happens lazily here. + tpl = os.path.join(paths.TEMPLATES_DIR, f"{dep_id}.tpl") + try: + os.remove(tpl) + except OSError: + pass + return jsonify({"deleted": dep_id, "was": dep}) + + +def reconcile(): + registry.rewrite_desired() + return jsonify({"revision": registry.current_revision()}), 202 + + +# ── Observation ────────────────────────────────────────────────────── + +def charts(): + observed, stale = registry.read_observed() + return jsonify({ + "loader_alive": registry.loader_alive(observed, stale), + "observed_stale": stale, + "loader": (observed or {}).get("loader"), + "auto_trading": (observed or {}).get("terminal", {}).get("auto_trading"), + "charts": (observed or {}).get("charts", []), + }) + + +def loader_status(): + observed, stale = registry.read_observed() + alive = registry.loader_alive(observed, stale) + payload = { + "alive": alive, + "observed_stale": stale, + "loader": (observed or {}).get("loader"), + "desired_revision": registry.current_revision(), + "applied_revision": + (observed or {}).get("loader", {}).get("applied_revision"), + } + if not alive: + payload["hint"] = ( + "No live loader detected. Attach MT5ChartLoader (bundled under " + "assets/experts/) to any chart, or add ChartControl.mqh to your " + "own resident EA — see docs/chart-control-protocol.md.") + return jsonify(payload) + + +def close_chart(chart_id): + try: + result = cmd.run_command("close_chart", {"chart_id": int(chart_id)}) + except ValueError: + return _err(400, "BAD_REQUEST", "chart_id must be an int") + except cmd.LoaderBusy as exc: + return _err(409, "LOADER_BUSY", str(exc)) + except cmd.LoaderTimeout as exc: + return _err(504, "LOADER_TIMEOUT", str(exc)) + if result.get("status") != "ok": + return _err(502, result.get("error_code", "LOADER_ERROR"), + result.get("error_detail", "loader reported failure")) + return jsonify({"closed": int(chart_id)}) + + +def screenshot(chart_id): + try: + result = cmd.run_command("screenshot", { + "chart_id": int(chart_id), + "width": int(request.args.get("width", 1280)), + "height": int(request.args.get("height", 720)), + }) + except ValueError: + return _err(400, "BAD_REQUEST", "chart_id/width/height must be ints") + except cmd.LoaderBusy as exc: + return _err(409, "LOADER_BUSY", str(exc)) + except cmd.LoaderTimeout as exc: + return _err(504, "LOADER_TIMEOUT", str(exc)) + if result.get("status") != "ok": + return _err(502, result.get("error_code", "LOADER_ERROR"), + result.get("error_detail", "loader reported failure")) + filename = paths.safe_name(result.get("file", ""), "screenshot") + png = os.path.join(paths.SCREENSHOTS_DIR, filename) + if not os.path.isfile(png): + return _err(502, "LOADER_ERROR", + "loader reported a screenshot that does not exist") + response = send_file(png, mimetype="image/png") + return response diff --git a/mt5api/handlers/webrequest.py b/mt5api/handlers/webrequest.py new file mode 100644 index 00000000..f03bb9f4 --- /dev/null +++ b/mt5api/handlers/webrequest.py @@ -0,0 +1,100 @@ +"""WebRequest allowlist endpoints. + +GET /webrequest -> current effective allowlist for this terminal. +PUT /webrequest -> set/patch the allowlist and apply it now. +POST /webrequest/apply -> re-apply the current allowlist (boot / manual hook, + since the VM terminal drops the list on restart). + +The apply mechanism is chosen at runtime. Inside the dockur VM the allowlist is +set by driving MT5's Options dialog with AutoIt (the only thing that persists it +in-session there — see chartctl/autoit_webrequest.py). Elsewhere (a bare-metal +terminal where ``common.ini`` IS the WebRequest store) it falls back to writing +``common.ini`` and restarting the terminal. + +Dedicated call (not a deployment field): URLs are only needed by the minority of +EAs that use WebRequest. First use migrates from whatever the terminal already +has so manually-configured URLs are preserved. +""" +from flask import jsonify, request + +from mt5api.chartctl import autoit_webrequest as autoit +from mt5api.chartctl import webrequest as wr +from mt5api.config import TERMINAL_DIR +from mt5api.mt5client import restart_terminal, session + + +def _cfg_dir(): + return wr.config_dir(TERMINAL_DIR) + + +def get_webrequest(): + return jsonify({"urls": wr.effective_urls(_cfg_dir())}) + + +def _resolve_urls(body, cfg_dir): + """Return (urls, error). Full replace via 'urls', or patch via 'add'/'remove'.""" + if "urls" in body: + return wr.clean_urls(body.get("urls")), None + if "add" in body or "remove" in body: + current = wr.effective_urls(cfg_dir) # migrate-from-current on first use + remove = set(wr.clean_urls(body.get("remove", []))) + new_urls = [u for u in current if u not in remove] + for u in wr.clean_urls(body.get("add", [])): + if u not in new_urls: + new_urls.append(u) + return new_urls, None + return None, "provide 'urls', or 'add'/'remove'" + + +def _apply(urls, use_runas=False): + """Apply ``urls`` to the running terminal. Returns (ok, detail).""" + if autoit.available(): + status, _log = autoit.apply_urls(urls, use_runas=use_runas) + return status == "OK", f"autoit:{status}" + # bare-metal fallback: write common.ini from desired, then restart. + with session(): + ok = restart_terminal() + return ok, "restart" if ok else "restart-failed" + + +def _use_runas(): + return request.args.get("runas", "0") == "1" + + +def put_webrequest(): + body = request.get_json(silent=True) + if not isinstance(body, dict): + return jsonify({"success": False, "error": "JSON body required"}), 400 + + cfg_dir = _cfg_dir() + new_urls, err = _resolve_urls(body, cfg_dir) + if err: + return jsonify({"success": False, "error": err}), 400 + + wr.save_desired(cfg_dir, new_urls) + ok, detail = _apply(new_urls, _use_runas()) + if not ok: + return jsonify( + {"success": False, "error": f"apply failed ({detail})", "urls": new_urls} + ), 500 + return jsonify({"success": True, "urls": new_urls, "applied_via": detail}) + + +def apply_webrequest(): + """Re-apply the current desired allowlist (idempotent). Boot/manual hook.""" + # dev: ?script= runs a repo-shipped AutoIt script (VM only), e.g. + # inspect_options.au3. ?runas=0 launches it non-elevated for diagnostics. + dev_script = request.args.get("script") + if dev_script and autoit.available(): + status, txt = autoit.run_named(dev_script, use_runas=_use_runas()) + return jsonify({"script": dev_script, "status": status, "log": txt}) + + urls = wr.effective_urls(_cfg_dir()) + if not urls: + return jsonify({"success": True, "urls": [], "note": "nothing to apply"}) + ok, detail = _apply(urls, _use_runas()) + if not ok: + return jsonify( + {"success": False, "error": f"apply failed ({detail})", "urls": urls} + ), 500 + return jsonify({"success": True, "urls": urls, "applied_via": detail}) diff --git a/mt5api/main.py b/mt5api/main.py index ac71e0d7..994e8c33 100644 --- a/mt5api/main.py +++ b/mt5api/main.py @@ -101,6 +101,48 @@ def _gated(environ, start_response): _WSGI_APP = DispatcherMiddleware(app, {"/mcp": _mcp_auth_gate(_MCP_BRIDGE)}) +# Seconds to let the terminal GUI settle before re-applying the WebRequest +# allowlist via AutoIt on boot (see _reapply_webrequest_once). +WEBREQUEST_BOOT_DELAY = 25 +_webrequest_reapplied = threading.Event() + + +def _reapply_webrequest_once(): + """Re-apply this terminal's desired WebRequest allowlist via AutoIt after a + (re)start. MT5 on the VM drops the list every restart, so the API re-sets it + once the terminal GUI is up. No-op unless AutoIt is present (the VM) AND a + desired allowlist has been configured. Runs in a daemon thread so it never + blocks startup; guarded to run only once per process.""" + if _webrequest_reapplied.is_set(): + return + _webrequest_reapplied.set() + + def _work(): + try: + from mt5api.chartctl import autoit_webrequest as autoit + from mt5api.chartctl import webrequest as wr + from mt5api.config import TERMINAL_DIR + + if not autoit.available(): + return + urls = wr.effective_urls(wr.config_dir(TERMINAL_DIR)) + if not urls: + return + time.sleep(WEBREQUEST_BOOT_DELAY + (PORT % 10) * 3) + for attempt in range(1, 4): + status, _log = autoit.apply_urls(urls) + log.info( + "Boot WebRequest re-apply attempt %d: %d url(s) -> %s", + attempt, len(urls), status, + ) + if status == "OK": + break + time.sleep(15) + except Exception: + log.exception("Boot WebRequest re-apply failed") + + threading.Thread(target=_work, daemon=True).start() + def _background_init(): """Keep retrying MT5 init until it connects. Runs in a daemon thread. @@ -121,6 +163,7 @@ def _background_init(): connected = False if connected: log.info("MT5 connected on attempt %d.", attempt) + _reapply_webrequest_once() return log.warning("MT5 not ready, retrying in %ds...", RETRY_INTERVAL) time.sleep(RETRY_INTERVAL) @@ -204,6 +247,7 @@ def main(): if connected: log.info("MT5 connected.") + _reapply_webrequest_once() else: log.warning( "MT5 not ready yet, retrying every %ds in background...", diff --git a/mt5api/mt5client.py b/mt5api/mt5client.py index 9f5baebe..4f3c4984 100644 --- a/mt5api/mt5client.py +++ b/mt5api/mt5client.py @@ -289,12 +289,20 @@ def init_mt5(login=None, password=None, server=None): return result +# Last successful terminal_info snapshot, cached lock-free for cosmetic +# readers (e.g. chartctl template build stamp). Never authoritative. +LAST_TERMINAL_INFO = None + + def ensure_initialized(): """Probe + reconnect helper. Caller must hold the MT5 lock.""" + global LAST_TERMINAL_INFO try: info = m(mt5.terminal_info, _timeout=15) except MT5Timeout: info = None + if info is not None: + LAST_TERMINAL_INFO = info if info is None: log.warning("Terminal not responding, attempting full init...") account = get_first_account() @@ -412,6 +420,18 @@ def restart_terminal(): if not killed: log.warning("No terminal process found, launching fresh.") + # Apply the WebRequest allowlist while the terminal is down. MT5 rewrites + # common.ini on exit, so this must happen after the kill and before launch. + # No-op unless this terminal has a desired allowlist set. + try: + from mt5api.chartctl import webrequest as _wr + + applied = _wr.apply_from_desired(TERMINAL_DIR) + if applied is not None: + log.info("Applied WebRequest allowlist (%d URL(s)) to common.ini", applied) + except Exception: + log.exception("Failed to apply WebRequest allowlist; continuing restart") + today = date.today().strftime("%Y%m%d") journal_log = os.path.join(TERMINAL_DIR, "logs", f"{today}.log") offset = 0 diff --git a/mt5api/server.py b/mt5api/server.py index 1aca624a..486881ed 100644 --- a/mt5api/server.py +++ b/mt5api/server.py @@ -4,7 +4,7 @@ from flask import Flask, abort, g, request from flask_compress import Compress from mt5api.backtest import handler as backtest_handler -from mt5api.config import API_TOKEN +from mt5api.config import API_TOKEN, CHARTCTL_ENABLED from mt5api.handlers import account, history, orders, positions, symbols, terminal from mt5api.logger import log @@ -94,6 +94,39 @@ def _end_request(response): app.get("/history/orders")(history.get_orders) app.get("/history/deals")(history.get_deals) +# ── Chart Deployments (chartctl) ───────────────────────────────── +# Lock-free EA deployment primitives. Gated: live mode + config enabled. +if CHARTCTL_ENABLED: + from mt5api.handlers import chartctl + + app.post("/experts")(chartctl.upload_expert) + app.get("/experts")(chartctl.list_experts) + app.delete("/experts/")(chartctl.delete_expert) + + app.post("/sets")(chartctl.upload_set) + app.get("/sets")(chartctl.list_sets) + app.get("/sets/")(chartctl.get_set) + + app.post("/deployments")(chartctl.create_deployment) + app.get("/deployments")(chartctl.list_deployments) + app.post("/deployments/reconcile")(chartctl.reconcile) + app.get("/deployments/")(chartctl.get_deployment) + app.patch("/deployments/")(chartctl.patch_deployment) + app.delete("/deployments/")(chartctl.delete_deployment) + + app.get("/charts")(chartctl.charts) + app.get("/loader")(chartctl.loader_status) + app.post("/charts//screenshot")(chartctl.screenshot) + app.post("/charts//close")(chartctl.close_chart) + + # WebRequest allowlist — dedicated call. Applied via AutoIt (VM) or a + # common.ini rewrite + restart (bare metal). /apply re-applies on demand. + from mt5api.handlers import webrequest + + app.get("/webrequest")(webrequest.get_webrequest) + app.put("/webrequest")(webrequest.put_webrequest) + app.post("/webrequest/apply")(webrequest.apply_webrequest) + # ── Backtest ───────────────────────────────────────────────────── app.post("/backtest/build-ini")(backtest_handler.build_ini_route) app.post("/backtest/build-set")(backtest_handler.build_set_route) diff --git a/run.sh b/run.sh index dbe4b469..384f53d8 100755 --- a/run.sh +++ b/run.sh @@ -61,8 +61,10 @@ cp "${DIR}/scripts/acquire_lock.ps1" "${DIR}/data/shared/scripts/acquire_lock.ps cp "${DIR}/scripts/api_runner.bat" "${DIR}/data/shared/scripts/api_runner.bat" cp "${DIR}/scripts/compile-warmup-ea.bat" "${DIR}/data/shared/scripts/compile-warmup-ea.bat" +cp "${DIR}/scripts/compile-chartctl-loader.bat" "${DIR}/data/shared/scripts/compile-chartctl-loader.bat" cp "${DIR}/scripts/check_health.py" "${DIR}/data/shared/scripts/check_health.py" cp "${DIR}/scripts/config_helper.py" "${DIR}/data/shared/scripts/config_helper.py" +cp "${DIR}/scripts/webrequest_allowlist_codec.py" "${DIR}/data/shared/scripts/webrequest_allowlist_codec.py" cp "${DIR}/scripts/event-log-tailer.ps1" "${DIR}/data/shared/scripts/event-log-tailer.ps1" cp "${DIR}/scripts/healthcheck.sh" "${DIR}/data/shared/scripts/healthcheck.sh" diff --git a/scripts/compile-chartctl-loader.bat b/scripts/compile-chartctl-loader.bat new file mode 100644 index 00000000..cddd72cf --- /dev/null +++ b/scripts/compile-chartctl-loader.bat @@ -0,0 +1,87 @@ +@echo off +setlocal enabledelayedexpansion + +rem ════════════════════════════════════════════════════════════════ +rem compile-chartctl-loader.bat +rem +rem Zero-touch bootstrap for the chartctl loader EA: +rem 1. Copies ChartControl.mqh into MQL5\Include\ of every broker +rem base terminal and MT5ChartLoader.mq5 into MQL5\Experts\Advisors\. +rem 2. Compiles the loader with each base's MetaEditor64. +rem 3. Copies the compiled .ex5 (+ include + source) into every +rem already-provisioned terminal instance dir, so existing +rem terminals pick it up without re-provisioning. New instances +rem inherit it automatically via the base xcopy. +rem +rem Combined with the [StartUp] Expert= line that config_helper.py +rem writes into mt5start.ini for live chartctl terminals, the loader +rem attaches itself at terminal launch — no RDP, no manual step. +rem +rem Modeled on compile-warmup-ea.bat (same MetaEditor invocation). +rem ════════════════════════════════════════════════════════════════ + +set "SHARED=C:\Users\Docker\Desktop\Shared" +set "ASSETS=C:\Users\Docker\Desktop\Assets" +if not exist "%ASSETS%\experts" set "ASSETS=%SHARED%\assets" + +set "SRC_EA=%ASSETS%\experts\MT5ChartLoader.mq5" +set "SRC_INC=%ASSETS%\experts\include\ChartControl.mqh" +set "COMPILE_LOG=%SHARED%\logs\compile-chartctl-loader.log" + +if not exist "%SRC_EA%" ( + echo ERROR: source not found: %SRC_EA% + exit /b 1 +) +if not exist "%SRC_INC%" ( + echo ERROR: include not found: %SRC_INC% + exit /b 1 +) + +set FOUND=0 +set FAILED=0 + +for /d %%B in ("%SHARED%\terminals\*") do ( + if exist "%%~fB\base\MetaEditor64.exe" ( + set /a FOUND+=1 + set "BASE=%%~fB\base" + echo [%%~nB] compiling loader in base... + + if not exist "!BASE!\MQL5\Include" mkdir "!BASE!\MQL5\Include" + if not exist "!BASE!\MQL5\Experts\Advisors" mkdir "!BASE!\MQL5\Experts\Advisors" + copy /Y "%SRC_INC%" "!BASE!\MQL5\Include\ChartControl.mqh" >nul + copy /Y "%SRC_EA%" "!BASE!\MQL5\Experts\Advisors\MT5ChartLoader.mq5" >nul + + "!BASE!\MetaEditor64.exe" /compile:"!BASE!\MQL5\Experts\Advisors\MT5ChartLoader.mq5" /inc:"!BASE!\MQL5" /log:"%COMPILE_LOG%" >nul 2>&1 + + if exist "!BASE!\MQL5\Experts\Advisors\MT5ChartLoader.ex5" ( + echo [%%~nB] OK: MT5ChartLoader.ex5 + + rem Propagate into every provisioned instance of this broker: + rem terminals\\\\ layout, skip \base. + for /d %%A in ("%%~fB\*") do ( + if /i not "%%~nxA"=="base" ( + for /d %%I in ("%%~fA\*") do ( + if exist "%%~fI\terminal64.exe" ( + if not exist "%%~fI\MQL5\Include" mkdir "%%~fI\MQL5\Include" + if not exist "%%~fI\MQL5\Experts\Advisors" mkdir "%%~fI\MQL5\Experts\Advisors" + copy /Y "!BASE!\MQL5\Experts\Advisors\MT5ChartLoader.ex5" "%%~fI\MQL5\Experts\Advisors\" >nul + copy /Y "%SRC_INC%" "%%~fI\MQL5\Include\ChartControl.mqh" >nul + copy /Y "%SRC_EA%" "%%~fI\MQL5\Experts\Advisors\MT5ChartLoader.mq5" >nul + echo [%%~nB] propagated to %%~nA\%%~nI + ) + ) + ) + ) + ) else ( + set /a FAILED+=1 + echo [%%~nB] ERROR: compile produced no .ex5 — see %COMPILE_LOG% + ) + ) +) + +if %FOUND%==0 ( + echo ERROR: no broker base terminals found under %SHARED%\terminals + exit /b 1 +) +if %FAILED% gtr 0 exit /b 1 +exit /b 0 diff --git a/scripts/config_helper.py b/scripts/config_helper.py index 92c360b4..c7c51361 100644 --- a/scripts/config_helper.py +++ b/scripts/config_helper.py @@ -209,9 +209,11 @@ def main(): elif cmd == "write_ini": if len(sys.argv) < 5: - print("Usage: config_helper.py write_ini ", file=sys.stderr) + print("Usage: config_helper.py write_ini [instance] [mode]", file=sys.stderr) sys.exit(1) broker, account, outpath = sys.argv[2], sys.argv[3], sys.argv[4] + instance = sys.argv[5] if len(sys.argv) > 5 else "default" + mode = (sys.argv[6] if len(sys.argv) > 6 else "live").lower() accounts = cfg.get("accounts", {}) b = accounts.get(broker, {}) creds = b.get(account) if account else next(iter(b.values()), None) if b else None @@ -223,9 +225,53 @@ def main(): ini += "KeepPrivate=0\nAutoTrading=1\nNewsEnable=0\n" ini += "[Experts]\nAllowLiveTrading=1\nAllowDllImport=1\nEnabled=1\n" ini += "[Email]\nEnable=0\n" + + # Chart Deployments loader bootstrap: auto-attach MT5ChartLoader at + # terminal launch via [StartUp]. Gated exactly like the API's + # CHARTCTL_ENABLED: live mode + global chartctl.enabled (opt-in, + # default FALSE) + no per-terminal `chartctl: false` override. The loader's + # GlobalVariable mutex makes the re-fire on every launch idempotent + # (a duplicate closes its own chart and exits). + chartctl_cfg = cfg.get("chartctl") or {} + # Opt-IN: see mt5api/config.py. A [StartUp] expert on every live terminal + # must never arrive by upgrade. + chartctl_on = bool(chartctl_cfg.get("enabled", False)) + term_override = None + term_suffix = "" + for t in cfg.get("terminals", []): + if (t.get("broker") == broker and str(t.get("account")) == str(account) + and (t.get("instance") or "default") == instance): + term_override = t.get("chartctl") + term_suffix = t.get("symbol_suffix") or "" + break + if mode == "live" and chartctl_on and term_override is not False: + ini += "[StartUp]\n" + ini += "Expert=Advisors\\MT5ChartLoader\n" + ini += f"Symbol=EURUSD{term_suffix}\n" + ini += "Period=H1\n" + with open(outpath, "w", encoding="utf-8") as f: f.write(ini) + # WebRequest allowlist boot-seed: start.bat deletes Config/common.ini + # every boot, so re-emit it here (after the delete, before launch) from + # the persistent per-terminal desired file. No-op when this terminal has + # no allowlist set. Gated exactly like the loader [StartUp] block above. + if mode == "live" and chartctl_on and term_override is not False: + try: + sys.path.insert(0, _SHARED_DIR) + from mt5api.chartctl import webrequest as wr + cfg_dir = os.path.join(os.path.dirname(os.path.abspath(outpath)), "Config") + urls = wr.load_desired(cfg_dir) + if urls is not None: + wr.write_common_ini(cfg_dir, urls) + except Exception as exc: # non-fatal: never block terminal launch + print(f"WARN: WebRequest allowlist seed failed: {exc}", file=sys.stderr) + + elif cmd == "chartctl_enabled": + chartctl_cfg = cfg.get("chartctl") or {} + print("1" if bool(chartctl_cfg.get("enabled", False)) else "0") + elif cmd == "nginx_conf": if len(sys.argv) < 3: print("Usage: config_helper.py nginx_conf ", file=sys.stderr) diff --git a/scripts/install.bat b/scripts/install.bat index faef744b..8098fa24 100644 --- a/scripts/install.bat +++ b/scripts/install.bat @@ -12,45 +12,34 @@ set "LOCKDIR=%SHARED%\install.running" set "DEBLOAT_DONE=%SHARED%\debloat.done" mkdir "%LOGDIR%" 2>nul -:: ── Stale lock cleanup after a protocol reboot ───────────────────── -:: reboot.bat writes this flag before every reboot. /s /q is REQUIRED now -:: that acquire_lock.ps1 stamps a boot.id file inside the lock dir -- a bare -:: rmdir fails on a non-empty directory and would leave the lock behind. +:: ── Stale lock cleanup after reboot ──────────────────────────────── if exist "%SHARED%\rebooting.flag" ( del "%SHARED%\rebooting.flag" 2>nul - rmdir /s /q "%SHARED%\install.running" 2>nul + rmdir "%SHARED%\install.running" 2>nul ) -:: ── Boot-scoped lock ─────────────────────────────────────────────── -:: The flag above only covers reboots that went through reboot.bat. An abrupt -:: kill -- container OOM, `docker kill`, host power loss -- writes no flag, and -:: %LOCKDIR% lives on the host-mounted %SHARED% volume so it outlives the VM. -:: That stranded lock would block every later boot exactly the way the -:: start.bat one did. The boot stamp makes an ownerless lock self-evident. -set "LOCK_OUT=%TEMP%\mt5_install_lock_result.txt" -del "%LOCK_OUT%" 2>nul -powershell -NoProfile -ExecutionPolicy Bypass -File "%SCRIPTS%\acquire_lock.ps1" -LockDir "%LOCKDIR%" > "%LOCK_OUT%" 2>&1 -set "LOCK_EC=!errorlevel!" -if exist "%LOCK_OUT%" ( - for /f "usebackq delims=" %%A in ("%LOCK_OUT%") do call :log "lock: %%A" +:: A lock from a previous boot session is stale even without the flag — +:: hard reboots (MT5AutoReboot, docker restart, power loss) give the +:: holder no cleanup window. Boot-time stamp lives in a sibling file so +:: the lock dir stays empty and plain rmdir release still works. +set "BOOTID=" +for /f "delims=" %%B in ('powershell -NoProfile -Command "(Get-CimInstance Win32_OperatingSystem).LastBootUpTime.ToString('yyyyMMddHHmmss')" 2^>nul') do set "BOOTID=%%B" +if defined BOOTID if exist "%LOCKDIR%" ( + set "LOCK_BOOTID=" + if exist "%LOCKDIR%.bootid" set /p LOCK_BOOTID=<"%LOCKDIR%.bootid" + if not "!LOCK_BOOTID!"=="!BOOTID!" ( + call :log "Removing stale install.running lock from previous boot." + rmdir /s /q "%LOCKDIR%" 2>nul + ) ) -del "%LOCK_OUT%" 2>nul -rem Exit 10 = a live instance from THIS boot holds it. Anything else non-zero -rem means the helper itself failed; fall back to the plain mkdir lock rather -rem than concluding "held", which is what deadlocked boot when a parse error -rem in the helper made PowerShell exit 1. -if !LOCK_EC! equ 10 ( - call :log "install.bat already running this boot, exiting." + +:: ── Atomic lock ───────────────────────────────────────────────────── +if defined BOOTID echo !BOOTID!>"%LOCKDIR%.bootid" +mkdir "%LOCKDIR%" 2>nul +if !errorlevel! neq 0 ( + call :log "Another install.bat is already running, exiting." exit /b 0 ) -if !LOCK_EC! neq 0 ( - call :log "WARN acquire_lock.ps1 failed (exit !LOCK_EC!), falling back to mkdir lock" - mkdir "%LOCKDIR%" 2>nul - if !errorlevel! neq 0 ( - call :log "fallback lock held, exiting." - exit /b 0 - ) -) call :log "============================================" call :log " MT5 Setup" @@ -91,7 +80,7 @@ call :log "[1/4] Creating scheduled task and disabling UAC..." schtasks /create /tn "MT5Start" /tr "cmd /c \"%SCRIPTS%\start.bat\"" /sc onlogon /ru "Docker" /rl HIGHEST /f >nul 2>&1 if !errorlevel! neq 0 ( call :log "ERROR: Failed to create MT5Start scheduled task" - call :release_lock + rmdir "%LOCKDIR%" 2>nul exit /b 1 ) call :log " MT5Start task created." @@ -131,21 +120,21 @@ call :log "[3/4] Installing Python 3.12..." curl -L -o C:\python-installer.exe https://www.python.org/ftp/python/3.12.7/python-3.12.7-amd64.exe >> "%INSTALL_LOG%" 2>&1 if not exist C:\python-installer.exe ( call :log "ERROR: Failed to download Python installer" - call :release_lock + rmdir "%LOCKDIR%" 2>nul exit /b 1 ) C:\python-installer.exe /quiet InstallAllUsers=1 PrependPath=1 Include_pip=1 Include_test=0 if !errorlevel! neq 0 ( call :log "ERROR: Python installer failed (exit code !errorlevel!)" del C:\python-installer.exe 2>nul - call :release_lock + rmdir "%LOCKDIR%" 2>nul exit /b 1 ) timeout /t 10 /nobreak >nul del C:\python-installer.exe 2>nul if not exist "%PYDIR%\python.exe" ( call :log "ERROR: python.exe not found after installation" - call :release_lock + rmdir "%LOCKDIR%" 2>nul exit /b 1 ) for /f "delims=" %%V in ('"%PYDIR%\python.exe" --version 2^>^&1') do call :log " %%V installed." @@ -154,7 +143,7 @@ call :log " Installing MetaTrader5 pip package..." "%PYDIR%\python.exe" -m pip install --upgrade pip >> "%INSTALL_LOG%" 2>&1 if !errorlevel! neq 0 ( call :log "ERROR: pip upgrade failed" - call :release_lock + rmdir "%LOCKDIR%" 2>nul exit /b 1 ) rem MetaTrader5 5.0.5735 was built against numpy 1.x. With numpy 2.x installed, @@ -164,7 +153,7 @@ rem Pin numpy<2 until MetaQuotes ships a wheel rebuilt for numpy 2.x. "%PYDIR%\python.exe" -m pip install MetaTrader5 "numpy<2" pyyaml >> "%INSTALL_LOG%" 2>&1 if !errorlevel! neq 0 ( call :log "ERROR: Failed to install MetaTrader5 / numpy / pyyaml" - call :release_lock + rmdir "%LOCKDIR%" 2>nul exit /b 1 ) call :log " MetaTrader5 + numpy<2 + pyyaml installed." @@ -215,7 +204,7 @@ for %%F in ("%BROKERS%\mt5setup-*.exe") do ( set "BNAME=!FNAME:mt5setup-=!" call :install_one "!BNAME!" "%%~F" if !errorlevel! neq 0 ( - call :release_lock + rmdir "%LOCKDIR%" 2>nul exit /b 1 ) ) @@ -228,7 +217,7 @@ for /d %%D in ("%BROKERS%\*") do ( ) if !HAS_TERMINALS! equ 0 ( call :log "ERROR: No MT5 terminals installed and no installers found!" - call :release_lock + rmdir "%LOCKDIR%" 2>nul exit /b 1 ) @@ -254,7 +243,7 @@ call :log "[4/4] All terminals present." call :log "============================================" call :log " Setup complete!" call :log "============================================" -call :release_lock +rmdir "%LOCKDIR%" 2>nul exit /b 0 @@ -302,20 +291,11 @@ tzutil /s "UTC" >nul 2>&1 exit /b 0 -:: ══════════════════════════════════════════════════════════════════ -:release_lock -:: /s /q is REQUIRED: the lock dir contains acquire_lock.ps1's boot.id stamp, -:: so a bare `rmdir` fails on a non-empty directory and would strand the lock. -rmdir /s /q "%LOCKDIR%" 2>nul -exit /b 0 - :: ══════════════════════════════════════════════════════════════════ :do_reboot -:: Routed through reboot.bat so there is exactly ONE reboot implementation in -:: the stack. It writes rebooting.flag (which this script consumes on the next -:: boot to clear a stale lock) AND releases both lock dirs up front, so the -:: previous "lock intentionally NOT released" contract still holds either way. -call "%SCRIPTS%\reboot.bat" install-requested +echo rebooting > "%SHARED%\rebooting.flag" +shutdown /r /t 5 /f +:: lock intentionally NOT released — rebooting.flag cleans it on next boot exit /b 0 diff --git a/scripts/start.bat b/scripts/start.bat index b880b21f..9f52d26f 100644 --- a/scripts/start.bat +++ b/scripts/start.bat @@ -1,388 +1,381 @@ -@echo off -setlocal enabledelayedexpansion -set SHARED=C:\Users\Docker\Desktop\Shared -set SCRIPTS=%SHARED%\scripts -set CONFIG=%SHARED%\config -set BROKERS=%SHARED%\terminals -set LOGDIR=%SHARED%\logs -set INSTALL_LOG=%LOGDIR%\install.log -set PIP_LOG=%LOGDIR%\pip.log -set START_LOG=%LOGDIR%\start.log -set FULL_LOG=%LOGDIR%\full.log -set "PYDIR=C:\Program Files\Python312" -set "PATH=%PYDIR%;%PYDIR%\Scripts;%PATH%" -set "LOCKDIR=%SHARED%\start.running" - -mkdir "%LOGDIR%" 2>nul -rmdir "%FULL_LOG%.lock" 2>nul - -:: ── Boot-scoped lock (only one start.bat instance per boot) ────── -:: A bare `mkdir %LOCKDIR%` used to deadlock the stack permanently. %LOCKDIR% -:: lives on the host-mounted %SHARED% volume, so it survives a VM reboot; the -:: MT5AutoReboot task fires `shutdown /r /t 0 /f` with no grace period and can -:: land anywhere inside this script's run (which legitimately spans from -:: seconds to over an hour). Killed mid-run, the lock outlived the process and -:: every later boot bailed on the orphan forever. -:: -:: acquire_lock.ps1 stamps the lock with the OS boot time, so a lock from a -:: previous boot is provably ownerless and gets cleared automatically. -:: Exit 0 = acquired, exit 1 = a live instance from THIS boot holds it. -:: -:: Deliberately does NOT consume %SHARED%\rebooting.flag — install.bat (called -:: below) owns that flag for its own lock, and eating it here would break it. -:: Tempfile rather than `for /f` because errorlevel after a for/f loop is the -:: loop body's exit code, not the invoked command's -- the lock verdict would -:: be silently lost. Same reason the API-token block below uses a tempfile. -set "LOCK_OUT=%TEMP%\mt5_lock_result.txt" -del "%LOCK_OUT%" 2>nul -powershell -NoProfile -ExecutionPolicy Bypass -File "%SCRIPTS%\acquire_lock.ps1" -LockDir "%LOCKDIR%" > "%LOCK_OUT%" 2>&1 -set "LOCK_EC=!errorlevel!" -if exist "%LOCK_OUT%" ( - for /f "usebackq delims=" %%A in ("%LOCK_OUT%") do ( - echo [%date% %time%] [start] lock: %%A >> "%FULL_LOG%" - echo [%date% %time%] lock: %%A >> "%START_LOG%" - ) -) -del "%LOCK_OUT%" 2>nul -rem Exit 10 means a live instance from THIS boot holds the lock -- bail. -if !LOCK_EC! equ 10 ( - echo [%date% %time%] start.bat already running this boot, exiting. - echo [%date% %time%] start.bat already running this boot, exiting. >> "%START_LOG%" - echo [%date% %time%] [start] start.bat already running this boot, exiting. >> "%FULL_LOG%" - exit /b 0 -) -rem Any other non-zero means acquire_lock.ps1 ITSELF failed (parse error, -rem missing cmdlet, unwritable mount). Do NOT treat that as "held" -- that is -rem precisely what deadlocked the stack when a syntax error in the helper made -rem PowerShell exit 1 and every boot concluded the lock was taken. Fall back to -rem the plain mkdir lock so boot still proceeds with single-instance safety. -if !LOCK_EC! neq 0 ( - echo [%date% %time%] WARN acquire_lock.ps1 failed ^(exit !LOCK_EC!^), falling back to mkdir lock >> "%START_LOG%" - echo [%date% %time%] [start] WARN acquire_lock.ps1 failed ^(exit !LOCK_EC!^), falling back to mkdir lock >> "%FULL_LOG%" - mkdir "%LOCKDIR%" 2>nul - if !errorlevel! neq 0 ( - echo [%date% %time%] fallback lock held, exiting. >> "%START_LOG%" - echo [%date% %time%] [start] fallback lock held, exiting. >> "%FULL_LOG%" - exit /b 0 - ) -) - -call :log "%START_LOG%" "====== Boot ======" -call :log "%INSTALL_LOG%" "====== Boot ======" - -:: ── Run install ────────────────────────────────────────────────── -call :log "%START_LOG%" "Running install.bat..." -call "%SCRIPTS%\install.bat" -if !errorlevel! equ 3 ( - call :log "%START_LOG%" "Reboot scheduled by install.bat, stopping." - call :release_lock - exit /b 0 -) -if !errorlevel! neq 0 ( - call :log "%START_LOG%" "ERROR: install.bat failed (exit code !errorlevel!)" - call :release_lock - exit /b 1 -) -call :log "%START_LOG%" "install.bat done." - -:: ── Pip install ────────────────────────────────────────────────── -:: Install base deps first (pyyaml required for config_helper.py below). -:: numpy<2 pin: MetaTrader5 5.0.5735 was built against numpy 1.x and breaks -:: silently with numpy 2.x — reads still work but order_send fails immediately -:: with (-2, 'Unnamed arguments not allowed'). Drop the pin once MetaQuotes -:: ships a numpy-2-compatible wheel. -:: -:: Each pip command writes to a per-call temp file so we can detect whether -:: anything ACTUALLY got installed/upgraded (presence of "Successfully -:: installed" in pip's output). If so, the python processes already running -:: from the previous boot are stale → reboot to pick up the new libs. -set "PIP_TMP=%TEMP%\mt5-pip-%RANDOM%-%RANDOM%.txt" -set "PIP_CHANGED=0" - -call :log "%START_LOG%" "Installing pip packages..." -call :log "%PIP_LOG%" "Installing pip packages..." -rem MCP v2 removed mcp.server.fastmcp; keep this synchronized with requirements-api.txt. -"%PYDIR%\python.exe" -m pip install pyyaml MetaTrader5 "numpy<2" flask waitress flask-compress psutil "mcp==1.28.0" a2wsgi > "%PIP_TMP%" 2>&1 -set "PIP_EC=!errorlevel!" -type "%PIP_TMP%" >> "%PIP_LOG%" -findstr /C:"Successfully installed" "%PIP_TMP%" >nul 2>&1 && set "PIP_CHANGED=1" -del "%PIP_TMP%" 2>nul -if !PIP_EC! neq 0 ( - call :log "%START_LOG%" "ERROR: pip install (base) failed (exit code !PIP_EC!), aborting." - call :log "%PIP_LOG%" "ERROR: pip install (base) failed" - call :release_lock - exit /b 1 -) -:: Extra packages from config.yaml requirements list. -:: NOTE: no `usebackq` — with usebackq, single-quoted strings are LITERAL, -:: not commands. Without usebackq, ('cmd') executes the command. This is -:: the same pattern install.bat uses for the `ports` lookup. -for /f "delims=" %%R in ('"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" requirements 2^>nul') do ( - "%PYDIR%\python.exe" -m pip install "%%R" > "%PIP_TMP%" 2>&1 - type "%PIP_TMP%" >> "%PIP_LOG%" - findstr /C:"Successfully installed" "%PIP_TMP%" >nul 2>&1 && set "PIP_CHANGED=1" - del "%PIP_TMP%" 2>nul -) -call :log "%START_LOG%" "pip done." -call :log "%PIP_LOG%" "pip done." - -:: Routed through reboot.bat so the flag write + lock release happen in one -:: place. Previously this inlined its own flag+shutdown+rmdir sequence, which -:: was correct but meant three separate reboot implementations to keep in sync. -if "!PIP_CHANGED!"=="1" ( - call :log "%START_LOG%" "pip changed packages -> rebooting so api_runners pick up new libs" - call "%SCRIPTS%\reboot.bat" pip-changed - exit /b 0 -) - -:: ── Start Windows event log tailer (background) ──────────────── -:: Streams Warning/Error/Critical from System + Application logs into -:: %LOGDIR%\windows-events.log so OOM kills, BSODs, terminal64 crashes, -:: etc. show up alongside the API logs. Single-instance via lock file -:: inside the script. -call :log "%START_LOG%" "Starting Windows event log tailer..." -start "Win Event Tailer" /B powershell -NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File "%SCRIPTS%\event-log-tailer.ps1" - -:: ── Kill lingering MT5 terminals ──────────────────────────────── -call :log "%START_LOG%" "Killing lingering MT5 terminals..." -tasklist /fi "imagename eq terminal64.exe" 2>nul | find /i "terminal64.exe" >nul && ( - taskkill /f /im terminal64.exe >nul 2>&1 - timeout /t 2 /nobreak >nul -) - -:: ── Verify config.yaml exists ─────────────────────────────────── -if not exist "%CONFIG%\config.yaml" ( - call :log "%START_LOG%" "ERROR: config.yaml not found! Copy config/config.yaml.example and re-run." - call :release_lock - exit /b 1 -) - -:: ── Parse config.yaml terminals once ──────────────────────────── -set "TERM_LIST=%TEMP%\mt5_terminals.txt" -"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" terminals > "%TERM_LIST%" 2>"%TEMP%\mt5_parse_err.txt" -if !errorlevel! neq 0 ( - call :log "%START_LOG%" "ERROR: Failed to parse config.yaml:" - type "%TEMP%\mt5_parse_err.txt" >> "%START_LOG%" - del "%TERM_LIST%" 2>nul - call :release_lock - exit /b 1 -) - -:: ── Periodic auto-reboot scheduled task ───────────────────────── -:: MT5 terminals share a desktop with DWM, and DWM/VirtIO-GPU crashes -:: under sustained load wedge the SDK pipe (terminal64.exe stops -:: responding to GDI/IPC). Cheapest mitigation: hard-reboot every N -:: minutes to flush GPU/desktop state before it rots. -:: Configured via config.yaml reboot_interval (minutes). 0 = disabled. -:: Default: 30. /f on schtasks is idempotent -- overwrites existing task. -:: -:: The task calls reboot.bat, NOT `shutdown` directly. Calling shutdown here -:: was the root cause of the permanent-deadlock outage: it killed start.bat -:: mid-run without writing rebooting.flag and without releasing -:: %SHARED%\start.running, and that orphaned lock (living on the host mount) -:: blocked every subsequent boot. reboot.bat always does both. -:: -:: No inner quotes in /tr -- schtasks quote-escaping is fragile, and %SCRIPTS% -:: has no spaces (C:\Users\Docker\Desktop\Shared\scripts). -set "REBOOT_INTERVAL=30" -"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" reboot_interval > "%SHARED%\mt5_ri.tmp" 2>nul -for /f "usebackq delims=" %%V in ("%SHARED%\mt5_ri.tmp") do set "REBOOT_INTERVAL=%%V" -del "%SHARED%\mt5_ri.tmp" 2>nul -if "!REBOOT_INTERVAL!"=="0" ( - schtasks /delete /tn "MT5AutoReboot" /f >nul 2>&1 - call :log "%START_LOG%" "Auto-reboot disabled (reboot_interval=0)." -) else ( - schtasks /create /tn "MT5AutoReboot" /tr "cmd.exe /c %SCRIPTS%\reboot.bat scheduled" /sc minute /mo !REBOOT_INTERVAL! /ru "SYSTEM" /rl HIGHEST /f >nul 2>&1 - if !errorlevel! equ 0 ( - call :log "%START_LOG%" "MT5AutoReboot task ensured (every !REBOOT_INTERVAL! min)." - ) else ( - call :log "%START_LOG%" "WARN: failed to create MT5AutoReboot task (errorlevel !errorlevel!)." - ) -) - -:: ── Launch MT5 terminals ───────────────────────────────────────── -call :log "%START_LOG%" "Launching MT5 terminals..." -set TERM_COUNT=0 -for /f "usebackq delims=" %%L in ("%TERM_LIST%") do ( - call :launch_terminal %%L - if !errorlevel! neq 0 ( - call :log "%START_LOG%" "ERROR: Failed to launch terminal, aborting." - del "%TERM_LIST%" 2>nul - call :release_lock - exit /b 1 - ) - set /a TERM_COUNT+=1 -) - -if !TERM_COUNT! equ 0 ( - call :log "%START_LOG%" "ERROR: No terminals configured in config.yaml" - del "%TERM_LIST%" 2>nul - call :release_lock - exit /b 1 -) - -call :log "%START_LOG%" "Launched !TERM_COUNT! terminal(s), waiting 30s to initialize..." -timeout /t 30 /nobreak >nul - -:: ── Load API token from config.yaml (optional) ────────────────── -:: Tempfile path is more robust than `for /f`'s subshell+quoting dance — -:: any python crash, pyyaml fallback install, or stdout buffering quirk -:: showed up as "API_TOKEN empty" through the for/f path. -set "API_TOKEN=" -set "TOKEN_TMP=%TEMP%\mt5_api_token.txt" -del "%TOKEN_TMP%" 2>nul -"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" api_token > "%TOKEN_TMP%" 2>nul -if exist "%TOKEN_TMP%" set /p API_TOKEN=<"%TOKEN_TMP%" -del "%TOKEN_TMP%" 2>nul -if defined API_TOKEN ( - call :log "%START_LOG%" "API token loaded." -) else ( - call :log "%START_LOG%" "WARNING: api_token empty in config.yaml, API running without auth." -) - -:: ── Launch API processes (all background) ──────────────────────── -call :log "%START_LOG%" "Launching API processes..." -for /f "usebackq delims=" %%L in ("%TERM_LIST%") do ( - call :launch_api_bg %%L -) -del "%TERM_LIST%" 2>nul -call :release_lock -call :log "%START_LOG%" "All !TERM_COUNT! API(s) running in background." - -:: ── Foreground: status + health monitor ────────────────────────── -:status_loop -cls -echo. -echo ===================================================== -echo MT5 HTTP API RUNNING -- %DATE% %TIME% -echo ===================================================== -echo. -"%PYDIR%\python.exe" "%SCRIPTS%\check_health.py" -echo. -timeout /t 60 /nobreak >nul -goto status_loop - -:: ══════════════════════════════════════════════════════════════════ -:launch_terminal -:: %1=broker %2=account %3=instance %4=port %5=utc_offset %6=mode (live|backtest) -set "LT_BROKER=%~1" -set "LT_ACCOUNT=%~2" -set "LT_INSTANCE=%~3" -set "LT_PORT=%~4" -set "LT_MODE=%~6" -if "!LT_INSTANCE!"=="" set "LT_INSTANCE=default" -if "!LT_MODE!"=="" set "LT_MODE=live" -set "LT_BASEDIR=%BROKERS%\!LT_BROKER!\base" -set "LT_DIR=%BROKERS%\!LT_BROKER!\!LT_ACCOUNT!\!LT_INSTANCE!" - -if not exist "!LT_BASEDIR!\terminal64.exe" ( - call :log "%START_LOG%" "ERROR: No base install for !LT_BROKER! at !LT_BASEDIR!" - exit /b 1 -) - -if not exist "!LT_DIR!\terminal64.exe" ( - call :log "%START_LOG%" "Copying !LT_BROKER!\base to !LT_BROKER!\!LT_ACCOUNT!\!LT_INSTANCE!..." - xcopy "!LT_BASEDIR!\*" "!LT_DIR!\" /E /I /H /Y /Q >nul 2>&1 - if !errorlevel! neq 0 ( - call :log "%START_LOG%" "ERROR: xcopy failed for !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE!" - exit /b 1 - ) -) - -del "!LT_DIR!\Config\settings.ini" 2>nul -del "!LT_DIR!\Config\common.ini" 2>nul - -call :write_ini "!LT_DIR!" "!LT_BROKER!" "!LT_ACCOUNT!" - -rem Save journal log size before launch so we only check NEW content -for /f "delims=" %%D in ('python -c "from datetime import date;print(date.today().strftime('%%Y%%m%%d'))"') do set "LT_LOGDATE=%%D" -set "LT_LOGFILE=!LT_DIR!\logs\!LT_LOGDATE!.log" -set LT_LOGSIZE=0 -if exist "!LT_LOGFILE!" ( - for %%A in ("!LT_LOGFILE!") do set LT_LOGSIZE=%%~zA -) - -if /i "!LT_MODE!"=="backtest" ( - call :log "%START_LOG%" " !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! mode=backtest -- portable dir prepared, terminal NOT launched (tester will spawn it on demand)." - exit /b 0 -) - -call :log "%START_LOG%" "Starting terminal: !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! (port !LT_PORT!) [log offset !LT_LOGSIZE!]" -powershell -Command "Start-Process '!LT_DIR!\terminal64.exe' -ArgumentList '/portable','/config:\"!LT_DIR!\mt5start.ini\"' -Verb RunAs -WindowStyle Normal" - -rem Wait for 'started for' in journal log (for /L avoids goto inside call) -set LT_STARTED=0 -for /L %%N in (1,1,120) do ( - if !LT_STARTED! equ 0 ( - python -c "import sys;f=open(sys.argv[1],'rb');f.seek(int(sys.argv[2]));d=f.read().decode('utf-16-le',errors='ignore');f.close();sys.exit(0 if 'started for' in d else 1)" "!LT_LOGFILE!" !LT_LOGSIZE! 2>nul - if !errorlevel! equ 0 ( - set LT_STARTED=1 - ) else ( - call :log "%START_LOG%" " Waiting for !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! to start (%%N)..." - timeout /t 5 /nobreak >nul - ) - ) -) -if !LT_STARTED! equ 0 ( - call :log "%START_LOG%" "ERROR: !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! failed to start after 10 minutes" - exit /b 1 -) -call :log "%START_LOG%" " !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! started." -exit /b 0 - -:: ══════════════════════════════════════════════════════════════════ -:launch_api_bg -set "LA_BROKER=%~1" -set "LA_ACCOUNT=%~2" -set "LA_INSTANCE=%~3" -set "LA_PORT=%~4" -set "LA_OFFSET=%~5" -set "LA_MODE=%~6" -if "!LA_INSTANCE!"=="" set "LA_INSTANCE=default" -if "!LA_OFFSET!"=="" set "LA_OFFSET=0" -if "!LA_MODE!"=="" set "LA_MODE=live" - -call :log "%START_LOG%" "Starting API (bg): !LA_BROKER!/!LA_ACCOUNT!/!LA_INSTANCE! on port !LA_PORT! (utc_offset=!LA_OFFSET! mode=!LA_MODE!)" -if "!LA_INSTANCE!"=="default" ( - start "MT5 API !LA_BROKER!/!LA_ACCOUNT!" cmd /c ""%SCRIPTS%\api_runner.bat" !LA_BROKER! !LA_ACCOUNT! !LA_INSTANCE! !LA_PORT! !API_TOKEN! !LA_OFFSET! !LA_MODE!" -) else ( - start "MT5 API !LA_BROKER!/!LA_ACCOUNT!/!LA_INSTANCE!" cmd /c ""%SCRIPTS%\api_runner.bat" !LA_BROKER! !LA_ACCOUNT! !LA_INSTANCE! !LA_PORT! !API_TOKEN! !LA_OFFSET! !LA_MODE!" -) -exit /b 0 - -:: ══════════════════════════════════════════════════════════════════ -:write_ini -set "WI_DIR=%~1" -set "WI_BROKER=%~2" -set "WI_ACCOUNT=%~3" -set "WI_CFG=!WI_DIR!\mt5start.ini" -"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" write_ini "!WI_BROKER!" "!WI_ACCOUNT!" "!WI_CFG!" >> "%START_LOG%" 2>&1 -if errorlevel 1 ( - call :log "%START_LOG%" "WARNING: Could not write ini for !WI_BROKER!/!WI_ACCOUNT!, using defaults" - echo [Common]> "!WI_CFG!" - echo KeepPrivate=0>> "!WI_CFG!" - echo AutoTrading=1>> "!WI_CFG!" - echo NewsEnable=0>> "!WI_CFG!" - echo [Experts]>> "!WI_CFG!" - echo AllowLiveTrading=1>> "!WI_CFG!" - echo AllowDllImport=1>> "!WI_CFG!" - echo Enabled=1>> "!WI_CFG!" - echo [Email]>> "!WI_CFG!" - echo Enable=0>> "!WI_CFG!" -) -exit /b 0 - -:: ══════════════════════════════════════════════════════════════════ -:release_lock -:: /s /q is REQUIRED: the lock dir contains acquire_lock.ps1's boot.id stamp, -:: so a bare `rmdir` fails on a non-empty directory and would leave the lock -:: behind -- reintroducing the deadlock this whole mechanism removes. -rmdir /s /q "%LOCKDIR%" 2>nul -exit /b 0 - -:: ══════════════════════════════════════════════════════════════════ -:log -echo [%date% %time%] %~2 -echo [%date% %time%] %~2 >> "%~1" -echo [%date% %time%] [start] %~2 >> "%FULL_LOG%" -exit /b 0 +@echo off +setlocal enabledelayedexpansion +set SHARED=C:\Users\Docker\Desktop\Shared +set SCRIPTS=%SHARED%\scripts +set CONFIG=%SHARED%\config +set BROKERS=%SHARED%\terminals +set LOGDIR=%SHARED%\logs +set INSTALL_LOG=%LOGDIR%\install.log +set PIP_LOG=%LOGDIR%\pip.log +set START_LOG=%LOGDIR%\start.log +set FULL_LOG=%LOGDIR%\full.log +set "PYDIR=C:\Program Files\Python312" +set "PATH=%PYDIR%;%PYDIR%\Scripts;%PATH%" +set "LOCKDIR=%SHARED%\start.running" + +mkdir "%LOGDIR%" 2>nul +rmdir "%FULL_LOG%.lock" 2>nul + +:: ── Stale lock cleanup ──────────────────────────────────────────── +:: The lock lives on the shared mount and survives reboots, but the +:: holder can't always release it: install.bat's staged reboots leave +:: only a 5s shutdown window (missable under TCG emulation), and +:: MT5AutoReboot / docker restart / power loss leave none. A lock can +:: only legitimately be held by a start.bat from THIS boot session, so +:: it's stamped with the boot time (sibling file, so the lock dir stays +:: empty and plain rmdir release still works) and any other stamp — or +:: none — marks it stale. +set "BOOTID=" +for /f "delims=" %%B in ('powershell -NoProfile -Command "(Get-CimInstance Win32_OperatingSystem).LastBootUpTime.ToString('yyyyMMddHHmmss')" 2^>nul') do set "BOOTID=%%B" +if defined BOOTID if exist "%LOCKDIR%" ( + set "LOCK_BOOTID=" + if exist "%LOCKDIR%.bootid" set /p LOCK_BOOTID=<"%LOCKDIR%.bootid" + if not "!LOCK_BOOTID!"=="!BOOTID!" ( + echo [%date% %time%] Removing stale start.running lock from previous boot. >> "%START_LOG%" + echo [%date% %time%] [start] Removing stale start.running lock from previous boot. >> "%FULL_LOG%" + rmdir /s /q "%LOCKDIR%" 2>nul + ) +) + +:: ── Atomic lock (only one start.bat instance at a time) ────────── +if defined BOOTID echo !BOOTID!>"%LOCKDIR%.bootid" +mkdir "%LOCKDIR%" 2>nul +if !errorlevel! neq 0 ( + echo [%date% %time%] Another start.bat is already running, exiting. + echo [%date% %time%] Another start.bat is already running, exiting. >> "%START_LOG%" + echo [%date% %time%] [start] Another start.bat is already running, exiting. >> "%FULL_LOG%" + exit /b 0 +) + +call :log "%START_LOG%" "====== Boot ======" +call :log "%INSTALL_LOG%" "====== Boot ======" + +:: ── Run install ────────────────────────────────────────────────── +call :log "%START_LOG%" "Running install.bat..." +call "%SCRIPTS%\install.bat" +if !errorlevel! equ 3 ( + call :log "%START_LOG%" "Reboot scheduled by install.bat, stopping." + rmdir "%LOCKDIR%" 2>nul + exit /b 0 +) +if !errorlevel! neq 0 ( + call :log "%START_LOG%" "ERROR: install.bat failed (exit code !errorlevel!)" + rmdir "%LOCKDIR%" 2>nul + exit /b 1 +) +call :log "%START_LOG%" "install.bat done." + +:: ── Pip install ────────────────────────────────────────────────── +:: Install base deps first (pyyaml required for config_helper.py below). +:: numpy<2 pin: MetaTrader5 5.0.5735 was built against numpy 1.x and breaks +:: silently with numpy 2.x — reads still work but order_send fails immediately +:: with (-2, 'Unnamed arguments not allowed'). Drop the pin once MetaQuotes +:: ships a numpy-2-compatible wheel. +:: +:: Each pip command writes to a per-call temp file so we can detect whether +:: anything ACTUALLY got installed/upgraded (presence of "Successfully +:: installed" in pip's output). If so, the python processes already running +:: from the previous boot are stale → reboot to pick up the new libs. +set "PIP_TMP=%TEMP%\mt5-pip-%RANDOM%-%RANDOM%.txt" +set "PIP_CHANGED=0" + +call :log "%START_LOG%" "Installing pip packages..." +call :log "%PIP_LOG%" "Installing pip packages..." +rem MCP v2 removed mcp.server.fastmcp; keep this synchronized with requirements-api.txt. +"%PYDIR%\python.exe" -m pip install pyyaml MetaTrader5 "numpy<2" flask waitress flask-compress psutil "mcp==1.28.0" a2wsgi > "%PIP_TMP%" 2>&1 +set "PIP_EC=!errorlevel!" +type "%PIP_TMP%" >> "%PIP_LOG%" +findstr /C:"Successfully installed" "%PIP_TMP%" >nul 2>&1 && set "PIP_CHANGED=1" +del "%PIP_TMP%" 2>nul +if !PIP_EC! neq 0 ( + call :log "%START_LOG%" "ERROR: pip install (base) failed (exit code !PIP_EC!), aborting." + call :log "%PIP_LOG%" "ERROR: pip install (base) failed" + rmdir "%LOCKDIR%" 2>nul + exit /b 1 +) +:: Extra packages from config.yaml requirements list. +:: NOTE: no `usebackq` — with usebackq, single-quoted strings are LITERAL, +:: not commands. Without usebackq, ('cmd') executes the command. This is +:: the same pattern install.bat uses for the `ports` lookup. +for /f "delims=" %%R in ('"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" requirements 2^>nul') do ( + "%PYDIR%\python.exe" -m pip install "%%R" > "%PIP_TMP%" 2>&1 + type "%PIP_TMP%" >> "%PIP_LOG%" + findstr /C:"Successfully installed" "%PIP_TMP%" >nul 2>&1 && set "PIP_CHANGED=1" + del "%PIP_TMP%" 2>nul +) +call :log "%START_LOG%" "pip done." +call :log "%PIP_LOG%" "pip done." + +if "!PIP_CHANGED!"=="1" ( + call :log "%START_LOG%" "pip changed packages -> rebooting so api_runners pick up new libs" + call :log "%FULL_LOG%" "[start] pip changed packages -> rebooting" + echo rebooting > "%SHARED%\rebooting.flag" + shutdown /r /t 5 /f + rmdir "%LOCKDIR%" 2>nul + exit /b 0 +) + +:: ── Start Windows event log tailer (background) ──────────────── +:: Streams Warning/Error/Critical from System + Application logs into +:: %LOGDIR%\windows-events.log so OOM kills, BSODs, terminal64 crashes, +:: etc. show up alongside the API logs. Single-instance via lock file +:: inside the script. +call :log "%START_LOG%" "Starting Windows event log tailer..." +start "Win Event Tailer" /B powershell -NoProfile -ExecutionPolicy Bypass -WindowStyle Hidden -File "%SCRIPTS%\event-log-tailer.ps1" + +:: ── Kill lingering MT5 terminals ──────────────────────────────── +call :log "%START_LOG%" "Killing lingering MT5 terminals..." +tasklist /fi "imagename eq terminal64.exe" 2>nul | find /i "terminal64.exe" >nul && ( + taskkill /f /im terminal64.exe >nul 2>&1 + timeout /t 2 /nobreak >nul +) + +:: ── Verify config.yaml exists ─────────────────────────────────── +if not exist "%CONFIG%\config.yaml" ( + call :log "%START_LOG%" "ERROR: config.yaml not found! Copy config/config.yaml.example and re-run." + rmdir "%LOCKDIR%" 2>nul + exit /b 1 +) + +:: ── Parse config.yaml terminals once ──────────────────────────── +set "TERM_LIST=%TEMP%\mt5_terminals.txt" +"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" terminals > "%TERM_LIST%" 2>"%TEMP%\mt5_parse_err.txt" +if !errorlevel! neq 0 ( + call :log "%START_LOG%" "ERROR: Failed to parse config.yaml:" + type "%TEMP%\mt5_parse_err.txt" >> "%START_LOG%" + del "%TERM_LIST%" 2>nul + rmdir "%LOCKDIR%" 2>nul + exit /b 1 +) + +:: ── Periodic auto-reboot scheduled task ───────────────────────── +:: MT5 terminals share a desktop with DWM, and DWM/VirtIO-GPU crashes +:: under sustained load wedge the SDK pipe (terminal64.exe stops +:: responding to GDI/IPC). Cheapest mitigation: hard-reboot every N +:: minutes to flush GPU/desktop state before it rots. +:: Configured via config.yaml reboot_interval (minutes). 0 = disabled. +:: Default: 30. /f on schtasks is idempotent -- overwrites existing task. +set "REBOOT_INTERVAL=30" +"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" reboot_interval > "%SHARED%\mt5_ri.tmp" 2>nul +for /f "usebackq delims=" %%V in ("%SHARED%\mt5_ri.tmp") do set "REBOOT_INTERVAL=%%V" +del "%SHARED%\mt5_ri.tmp" 2>nul +if "!REBOOT_INTERVAL!"=="0" ( + schtasks /delete /tn "MT5AutoReboot" /f >nul 2>&1 + call :log "%START_LOG%" "Auto-reboot disabled (reboot_interval=0)." +) else ( + schtasks /create /tn "MT5AutoReboot" /tr "shutdown /r /t 0 /f /d p:0:0" /sc minute /mo !REBOOT_INTERVAL! /ru "SYSTEM" /rl HIGHEST /f >nul 2>&1 + if !errorlevel! equ 0 ( + call :log "%START_LOG%" "MT5AutoReboot task ensured (every !REBOOT_INTERVAL! min)." + ) else ( + call :log "%START_LOG%" "WARN: failed to create MT5AutoReboot task (errorlevel !errorlevel!)." + ) +) + +:: ── Compile chartctl loader EA (zero-touch bootstrap) ──────────── +:: Compiles MT5ChartLoader in every broker base and propagates the .ex5 +:: into existing terminal instances, so the [StartUp] Expert= line in +:: mt5start.ini can auto-attach it at launch. Skipped when chartctl is +:: disabled globally in config.yaml. Non-fatal: a compile failure only +:: means chart deployments stay unavailable until fixed. +:: Tempfile read, NOT for /f ('command') — with both python.exe and the +:: script path quoted, cmd's quote-stripping mangles the subshell command +:: and it silently outputs nothing (same failure the api_token block +:: documents; also why install.bat's ports lookup falls back to 6542). +set "CHARTCTL_ON=" +"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" chartctl_enabled > "%SHARED%\mt5_cc.tmp" 2>nul +for /f "usebackq delims=" %%C in ("%SHARED%\mt5_cc.tmp") do set "CHARTCTL_ON=%%C" +del "%SHARED%\mt5_cc.tmp" 2>nul +if "!CHARTCTL_ON!"=="1" ( + call :log "%START_LOG%" "Compiling chartctl loader EA (MT5ChartLoader)..." + call "%SCRIPTS%\compile-chartctl-loader.bat" >> "%START_LOG%" 2>&1 + if !errorlevel! neq 0 ( + call :log "%START_LOG%" "WARN: chartctl loader compile failed -- chart deployments unavailable. See logs\compile-chartctl-loader.log" + ) else ( + call :log "%START_LOG%" "chartctl loader compiled and propagated." + ) +) else ( + call :log "%START_LOG%" "chartctl disabled in config.yaml -- skipping loader compile." +) + +:: ── Launch MT5 terminals ───────────────────────────────────────── +call :log "%START_LOG%" "Launching MT5 terminals..." +set TERM_COUNT=0 +for /f "usebackq delims=" %%L in ("%TERM_LIST%") do ( + call :launch_terminal %%L + if !errorlevel! neq 0 ( + call :log "%START_LOG%" "ERROR: Failed to launch terminal, aborting." + del "%TERM_LIST%" 2>nul + rmdir "%LOCKDIR%" 2>nul + exit /b 1 + ) + set /a TERM_COUNT+=1 +) + +if !TERM_COUNT! equ 0 ( + call :log "%START_LOG%" "ERROR: No terminals configured in config.yaml" + del "%TERM_LIST%" 2>nul + rmdir "%LOCKDIR%" 2>nul + exit /b 1 +) + +call :log "%START_LOG%" "Launched !TERM_COUNT! terminal(s), waiting 30s to initialize..." +timeout /t 30 /nobreak >nul + +:: ── Load API token from config.yaml (optional) ────────────────── +:: Tempfile path is more robust than `for /f`'s subshell+quoting dance — +:: any python crash, pyyaml fallback install, or stdout buffering quirk +:: showed up as "API_TOKEN empty" through the for/f path. +set "API_TOKEN=" +set "TOKEN_TMP=%TEMP%\mt5_api_token.txt" +del "%TOKEN_TMP%" 2>nul +"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" api_token > "%TOKEN_TMP%" 2>nul +if exist "%TOKEN_TMP%" set /p API_TOKEN=<"%TOKEN_TMP%" +del "%TOKEN_TMP%" 2>nul +if defined API_TOKEN ( + call :log "%START_LOG%" "API token loaded." +) else ( + call :log "%START_LOG%" "WARNING: api_token empty in config.yaml, API running without auth." +) + +:: ── Launch API processes (all background) ──────────────────────── +call :log "%START_LOG%" "Launching API processes..." +for /f "usebackq delims=" %%L in ("%TERM_LIST%") do ( + call :launch_api_bg %%L +) +del "%TERM_LIST%" 2>nul +rmdir "%LOCKDIR%" 2>nul +call :log "%START_LOG%" "All !TERM_COUNT! API(s) running in background." + +:: ── Foreground: status + health monitor ────────────────────────── +:status_loop +cls +echo. +echo ===================================================== +echo MT5 HTTP API RUNNING -- %DATE% %TIME% +echo ===================================================== +echo. +"%PYDIR%\python.exe" "%SCRIPTS%\check_health.py" +echo. +timeout /t 60 /nobreak >nul +goto status_loop + +:: ══════════════════════════════════════════════════════════════════ +:launch_terminal +:: %1=broker %2=account %3=instance %4=port %5=utc_offset %6=mode (live|backtest) +set "LT_BROKER=%~1" +set "LT_ACCOUNT=%~2" +set "LT_INSTANCE=%~3" +set "LT_PORT=%~4" +set "LT_MODE=%~6" +if "!LT_INSTANCE!"=="" set "LT_INSTANCE=default" +if "!LT_MODE!"=="" set "LT_MODE=live" +set "LT_BASEDIR=%BROKERS%\!LT_BROKER!\base" +set "LT_DIR=%BROKERS%\!LT_BROKER!\!LT_ACCOUNT!\!LT_INSTANCE!" + +if not exist "!LT_BASEDIR!\terminal64.exe" ( + call :log "%START_LOG%" "ERROR: No base install for !LT_BROKER! at !LT_BASEDIR!" + exit /b 1 +) + +if not exist "!LT_DIR!\terminal64.exe" ( + call :log "%START_LOG%" "Copying !LT_BROKER!\base to !LT_BROKER!\!LT_ACCOUNT!\!LT_INSTANCE!..." + xcopy "!LT_BASEDIR!\*" "!LT_DIR!\" /E /I /H /Y /Q >nul 2>&1 + if !errorlevel! neq 0 ( + call :log "%START_LOG%" "ERROR: xcopy failed for !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE!" + exit /b 1 + ) +) + +del "!LT_DIR!\Config\settings.ini" 2>nul +del "!LT_DIR!\Config\common.ini" 2>nul + +call :write_ini "!LT_DIR!" "!LT_BROKER!" "!LT_ACCOUNT!" "!LT_INSTANCE!" "!LT_MODE!" + +rem Save journal log size before launch so we only check NEW content +for /f "delims=" %%D in ('python -c "from datetime import date;print(date.today().strftime('%%Y%%m%%d'))"') do set "LT_LOGDATE=%%D" +set "LT_LOGFILE=!LT_DIR!\logs\!LT_LOGDATE!.log" +set LT_LOGSIZE=0 +if exist "!LT_LOGFILE!" ( + for %%A in ("!LT_LOGFILE!") do set LT_LOGSIZE=%%~zA +) + +if /i "!LT_MODE!"=="backtest" ( + call :log "%START_LOG%" " !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! mode=backtest -- portable dir prepared, terminal NOT launched (tester will spawn it on demand)." + exit /b 0 +) + +call :log "%START_LOG%" "Starting terminal: !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! (port !LT_PORT!) [log offset !LT_LOGSIZE!]" +powershell -Command "Start-Process '!LT_DIR!\terminal64.exe' -ArgumentList '/portable','/config:\"!LT_DIR!\mt5start.ini\"' -Verb RunAs -WindowStyle Normal" + +rem Wait for 'started for' in journal log (for /L avoids goto inside call) +set LT_STARTED=0 +for /L %%N in (1,1,120) do ( + if !LT_STARTED! equ 0 ( + python -c "import sys;f=open(sys.argv[1],'rb');f.seek(int(sys.argv[2]));d=f.read().decode('utf-16-le',errors='ignore');f.close();sys.exit(0 if 'started for' in d else 1)" "!LT_LOGFILE!" !LT_LOGSIZE! 2>nul + if !errorlevel! equ 0 ( + set LT_STARTED=1 + ) else ( + call :log "%START_LOG%" " Waiting for !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! to start (%%N)..." + timeout /t 5 /nobreak >nul + ) + ) +) +if !LT_STARTED! equ 0 ( + call :log "%START_LOG%" "ERROR: !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! failed to start after 10 minutes" + exit /b 1 +) +call :log "%START_LOG%" " !LT_BROKER!/!LT_ACCOUNT!/!LT_INSTANCE! started." +exit /b 0 + +:: ══════════════════════════════════════════════════════════════════ +:launch_api_bg +set "LA_BROKER=%~1" +set "LA_ACCOUNT=%~2" +set "LA_INSTANCE=%~3" +set "LA_PORT=%~4" +set "LA_OFFSET=%~5" +set "LA_MODE=%~6" +if "!LA_INSTANCE!"=="" set "LA_INSTANCE=default" +if "!LA_OFFSET!"=="" set "LA_OFFSET=0" +if "!LA_MODE!"=="" set "LA_MODE=live" + +call :log "%START_LOG%" "Starting API (bg): !LA_BROKER!/!LA_ACCOUNT!/!LA_INSTANCE! on port !LA_PORT! (utc_offset=!LA_OFFSET! mode=!LA_MODE!)" +if "!LA_INSTANCE!"=="default" ( + start "MT5 API !LA_BROKER!/!LA_ACCOUNT!" cmd /c ""%SCRIPTS%\api_runner.bat" !LA_BROKER! !LA_ACCOUNT! !LA_INSTANCE! !LA_PORT! !API_TOKEN! !LA_OFFSET! !LA_MODE!" +) else ( + start "MT5 API !LA_BROKER!/!LA_ACCOUNT!/!LA_INSTANCE!" cmd /c ""%SCRIPTS%\api_runner.bat" !LA_BROKER! !LA_ACCOUNT! !LA_INSTANCE! !LA_PORT! !API_TOKEN! !LA_OFFSET! !LA_MODE!" +) +exit /b 0 + +:: ══════════════════════════════════════════════════════════════════ +:write_ini +set "WI_DIR=%~1" +set "WI_BROKER=%~2" +set "WI_ACCOUNT=%~3" +set "WI_INSTANCE=%~4" +set "WI_MODE=%~5" +if "!WI_INSTANCE!"=="" set "WI_INSTANCE=default" +if "!WI_MODE!"=="" set "WI_MODE=live" +set "WI_CFG=!WI_DIR!\mt5start.ini" +"%PYDIR%\python.exe" "%SCRIPTS%\config_helper.py" write_ini "!WI_BROKER!" "!WI_ACCOUNT!" "!WI_CFG!" "!WI_INSTANCE!" "!WI_MODE!" >> "%START_LOG%" 2>&1 +if errorlevel 1 ( + call :log "%START_LOG%" "WARNING: Could not write ini for !WI_BROKER!/!WI_ACCOUNT!, using defaults" + echo [Common]> "!WI_CFG!" + echo KeepPrivate=0>> "!WI_CFG!" + echo AutoTrading=1>> "!WI_CFG!" + echo NewsEnable=0>> "!WI_CFG!" + echo [Experts]>> "!WI_CFG!" + echo AllowLiveTrading=1>> "!WI_CFG!" + echo AllowDllImport=1>> "!WI_CFG!" + echo Enabled=1>> "!WI_CFG!" + echo [Email]>> "!WI_CFG!" + echo Enable=0>> "!WI_CFG!" +) +exit /b 0 + +:: ══════════════════════════════════════════════════════════════════ +:log +echo [%date% %time%] %~2 +echo [%date% %time%] %~2 >> "%~1" +echo [%date% %time%] [start] %~2 >> "%FULL_LOG%" +exit /b 0 diff --git a/scripts/webrequest_allowlist_codec.py b/scripts/webrequest_allowlist_codec.py new file mode 100644 index 00000000..4a59c7d7 --- /dev/null +++ b/scripts/webrequest_allowlist_codec.py @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +"""Codec for the MT5 ``[Experts] WebRequestUrl=`` allowlist blob (Config/common.ini). + +The WebRequest allowed-URL list (Tools -> Options -> Expert Advisors) is stored in +each terminal's ``Config/common.ini`` as ``WebRequestUrl=``. The hex is a +length-preserving encrypted blob produced by terminal64.exe. It is +NOT machine-bound: the key is fixed in the binary, so a blob generated here is +accepted verbatim by any terminal of the same build. That lets us provision the +allowlist programmatically (no RDP / no Options dialog), which is what Chart +Deployments needs so deployed EAs can call WebRequest without manual setup. + +FORMAT (fully reverse-engineered from terminal64.exe fn @0x7ff7824d4010): + ini hex string --%04X per uint16, stored LE--> ciphertext bytes (== swap16 of naive hex) + plaintext = u16le(1) + u16le(checksum) + utf16le(";".join(urls)) + where checksum = (sum of every UTF-16 code unit in the joined string) & 0xffff + cipher (decode, config-load direction) is a byte-wise CFB stream: + p[i] = ((c[i-1] + KEY[i % 16]) & 0xff) ^ c[i] with c[-1] = 0 + encode is the exact inverse (feedback taken from the ciphertext byte). + +The leading u16 is a constant 1 in every observed blob. The checksum lives at the +FRONT of the plaintext, so editing any URL character changes it and re-ciphers the +whole tail -- this is why the blob looked like a strong full-avalanche cipher in +black-box testing when it is really a single CFB pass plus a front checksum. + +Verified: round-trips byte-identically against 18 independent real broker-terminal +blobs (BlackBull, IC Markets, FP Markets, Darwinex, Ducascopy, AquaFunded, ...). +""" +from __future__ import annotations + +# 16-byte key, recovered by cryptanalysis of the CFB recurrence and confirmed +# against 50+ plaintext bytes and 18 full-blob round-trips. (The binary derives +# it at runtime via an obfuscated routine from a .rdata seed; the derived bytes +# are what matter and are reproduced here directly.) +KEY = bytes([0xe2, 0x30, 0x54, 0xb4, 0xde, 0xe5, 0xcc, 0x04, + 0x9c, 0x70, 0x8f, 0x3c, 0x6b, 0x87, 0x78, 0xf0]) + + +def _swap16(data: bytes) -> bytes: + b = bytearray(data) + for i in range(0, len(b) - 1, 2): + b[i], b[i + 1] = b[i + 1], b[i] + return bytes(b) + + +def _decode_cfb(ct: bytes) -> bytes: + out = bytearray(len(ct)) + prev = 0 + for i, c in enumerate(ct): + out[i] = ((prev + KEY[i % 16]) & 0xff) ^ c + prev = c + return bytes(out) + + +def _encode_cfb(pt: bytes) -> bytes: + out = bytearray(len(pt)) + prev = 0 + for i, p in enumerate(pt): + c = ((prev + KEY[i % 16]) & 0xff) ^ p + out[i] = c + prev = c + return bytes(out) + + +def _checksum(s: str) -> int: + # sum over UTF-16 code units (BMP chars == ord); & 0xffff + return sum(b[0] | (b[1] << 8) + for b in (s.encode("utf-16-le")[i:i + 2] + for i in range(0, len(s) * 2, 2))) & 0xffff + + +def decode_blob(hex_blob: str) -> list[str]: + """Ciphertext hex (value of ``WebRequestUrl=``) -> list of URL strings.""" + pt = _decode_cfb(_swap16(bytes.fromhex(hex_blob.strip()))) + body = pt[4:].decode("utf-16-le") + return body.split(";") if body else [] + + +def encode_urls(urls: list[str]) -> str: + """List of URL strings -> uppercase ciphertext hex for ``WebRequestUrl=``.""" + joined = ";".join(urls) + pt = (1).to_bytes(2, "little") + _checksum(joined).to_bytes(2, "little") \ + + joined.encode("utf-16-le") + return _swap16(_encode_cfb(pt)).hex().upper() + + +# --- embedded verified test vectors (real captured broker blobs) --- +_VECTORS = [ + ("13E33B7856715A56F2822DF172EBC0D0BD8DF23E89A42B2716A602C68D06506070409EEA021DA6A29" + "525874B0D86E7F7D8A8FC4898B30E0A3DCDFBBF9D166474552580CC55704642FD8D1DE13AB3CADA08D8" + "DC28AFCA0C080292FABED14A1828BA8A1B67BCD700FC69F9D094E55E2A3AAE7E17637D986B67D868511" + "562DB", + ["https://tracker.algotradingspace.com", "https://api.telegram.org"]), + ("13E33B7E56715A56F2822DF172EBC0D0BD8DF23E96B1070332C2DCA0AD263444A7774E9A324DA39FCE5" + "E783C0982D0E0CC9CF7439FBA", + ["https://aaaaaaaaaaaaaa.co"]), +] + + +def _selftest() -> None: + for blob, urls in _VECTORS: + blob = blob.replace("\n", "") + assert decode_blob(blob) == urls, decode_blob(blob) + assert encode_urls(urls) == blob, encode_urls(urls) + print("webrequest_allowlist_codec: self-test OK (%d vectors)" % len(_VECTORS)) + + +if __name__ == "__main__": + import sys + if len(sys.argv) == 1 or sys.argv[1] == "--selftest": + _selftest() + elif sys.argv[1] == "decode": + for u in decode_blob(sys.argv[2]): + print(u) + elif sys.argv[1] == "encode": + print(encode_urls(sys.argv[2:])) + else: + print("usage: webrequest_allowlist_codec.py [--selftest | decode | encode ...]") diff --git a/tests/chartctl_fake_loader.py b/tests/chartctl_fake_loader.py new file mode 100644 index 00000000..455b4cf9 --- /dev/null +++ b/tests/chartctl_fake_loader.py @@ -0,0 +1,111 @@ +"""A Python stand-in for the loader EA, implementing the terminal side of +Chart Control Protocol v1. Lets the full endpoint suite run on Linux with +no MT5, no Windows — the same trick conftest uses to stub the SDK. + +It reads desired.json and writes observed.json exactly as the MQL5 loader +would, so integration tests exercise the real registry merge + status +derivation against a realistic observed file. +""" +import json +import os + + +class FakeLoader: + def __init__(self, protocol_dir: str): + self.dir = protocol_dir + os.makedirs(self.dir, exist_ok=True) + os.makedirs(os.path.join(self.dir, "shots"), exist_ok=True) + self.auto_trading = True + self._next_chart_id = 133039100 + + # ── protocol files ─────────────────────────────────────────── + def _read(self, name): + try: + with open(os.path.join(self.dir, name), encoding="utf-8") as fh: + return json.load(fh) + except (OSError, ValueError): + return None + + def _write(self, name, obj): + path = os.path.join(self.dir, name) + tmp = path + ".tmp" + with open(tmp, "w", encoding="utf-8") as fh: + json.dump(obj, fh) + os.replace(tmp, path) + + # ── one reconcile pass ─────────────────────────────────────── + def reconcile(self, *, fail_ids=None, missing_ids=None): + """Materialize observed.json from desired.json. + + fail_ids: deployments that should report a terminal error. + missing_ids: enabled deployments the loader 'fails to attach' + (stay pending / degraded). + """ + fail_ids = set(fail_ids or []) + missing_ids = set(missing_ids or []) + desired = self._read("desired.json") or {"revision": 0, "deployments": []} + + charts, dep_status, errors = [], [], [] + for dep in desired.get("deployments", []): + did = dep["id"] + if not dep.get("enabled", True): + dep_status.append({"id": did, "status": "paused"}) + continue + if did in fail_ids: + errors.append({"id": did, "status": "failed", + "code": "EXPERT_NOT_ATTACHED", + "detail": "fake failure"}) + dep_status.append({"id": did, "status": "failed"}) + continue + if did in missing_ids: + dep_status.append({"id": did, "status": "pending"}) + continue + cid = self._next_chart_id + self._next_chart_id += 1 + charts.append({ + "chart_id": cid, "symbol": dep["symbol"], + "timeframe": dep["timeframe"], "expert": dep["expert"], + "expert_enabled": True, "deployment_id": did, + }) + dep_status.append({"id": did, "status": "running", "chart_id": cid}) + + observed = { + "protocol": 1, + "loader": {"name": "FakeLoader", "version": "1.0.0", + "last_loop": "now", + "applied_revision": desired.get("revision", 0)}, + "terminal": {"auto_trading": self.auto_trading}, + "charts": charts, + "deployments": dep_status, + "errors": errors, + } + self._write("observed.json", observed) + return observed + + # ── command channel ────────────────────────────────────────── + def handle_command(self): + cmd = self._read("command.json") + if not cmd: + return None + cid = cmd["command_id"] + result = {"command_id": cid} + if cmd.get("action") == "screenshot": + fname = f"{cid}.png" + with open(os.path.join(self.dir, "shots", fname), "wb") as fh: + fh.write(b"\x89PNG\r\n\x1a\n") # PNG magic, enough for a test + result.update({"status": "ok", "file": fname}) + elif cmd.get("action") == "close_chart": + if cmd.get("chart_id") == -1: # sentinel: unknown chart + result.update({"status": "error", "error_code": "CLOSE_FAILED", + "error_detail": "err=4101"}) + else: + result.update({"status": "ok"}) + else: + result.update({"status": "error", "error_code": "UNKNOWN_ACTION", + "error_detail": cmd.get("action", "")}) + self._write("command_result.json", result) + try: + os.remove(os.path.join(self.dir, "command.json")) + except OSError: + pass + return result diff --git a/tests/test_chartctl_endpoints.py b/tests/test_chartctl_endpoints.py new file mode 100644 index 00000000..260f6aa6 --- /dev/null +++ b/tests/test_chartctl_endpoints.py @@ -0,0 +1,305 @@ +"""End-to-end chartctl endpoint tests via Flask's test client, with the +Python FakeLoader playing the terminal side of the protocol. + +Repoints every chartctl path at a tmp dir, registers the routes on a +fresh Flask app (config gating is bypassed here — we wire handlers +directly so the suite doesn't depend on CHARTCTL_ENABLED at import). +""" +from __future__ import annotations + +import io +import json +import os + +import pytest +from flask import Flask + +from tests.chartctl_fake_loader import FakeLoader + + +@pytest.fixture +def client(monkeypatch, tmp_path): + from mt5api.chartctl import paths, registry, command + + experts = tmp_path / "experts" + sets_ = tmp_path / "sets" + proto = tmp_path / "proto" + tpls = tmp_path / "tpls" + host_e = tmp_path / "host_experts" + host_s = tmp_path / "host_sets" + for d in (experts, sets_, proto, tpls, host_e, host_s, + proto / "shots"): + d.mkdir(parents=True, exist_ok=True) + + monkeypatch.setattr(paths, "EXPERTS_DIR", str(experts)) + monkeypatch.setattr(paths, "SETS_DIR", str(sets_)) + monkeypatch.setattr(paths, "PROTOCOL_DIR", str(proto)) + monkeypatch.setattr(paths, "TEMPLATES_DIR", str(tpls)) + monkeypatch.setattr(paths, "SCREENSHOTS_DIR", str(proto / "shots")) + monkeypatch.setattr(paths, "HOST_EXPERTS_DIR", str(host_e)) + monkeypatch.setattr(paths, "HOST_SETS_DIR", str(host_s)) + monkeypatch.setattr(paths, "REGISTRY_PATH", str(tmp_path / "registry.json")) + monkeypatch.setattr(paths, "DESIRED_PATH", str(proto / "desired.json")) + monkeypatch.setattr(paths, "OBSERVED_PATH", str(proto / "observed.json")) + monkeypatch.setattr(paths, "COMMAND_PATH", str(proto / "command.json")) + monkeypatch.setattr(paths, "COMMAND_RESULT_PATH", + str(proto / "command_result.json")) + # registry caches some path constants via import — repoint those too. + monkeypatch.setattr(registry.paths, "REGISTRY_PATH", str(tmp_path / "registry.json")) + monkeypatch.setattr(registry.paths, "DESIRED_PATH", str(proto / "desired.json")) + monkeypatch.setattr(registry.paths, "OBSERVED_PATH", str(proto / "observed.json")) + monkeypatch.setattr(registry.paths, "PROTOCOL_DIR", str(proto)) + monkeypatch.setattr(registry, "_STATE", None) + # tpl_builder writes into TEMPLATES_DIR imported at module load + from mt5api.chartctl import tpl_builder + monkeypatch.setattr(tpl_builder, "TEMPLATES_DIR", str(tpls)) + # shorten command timeout so the timeout test is fast + monkeypatch.setattr(command, "CHARTCTL_COMMAND_TIMEOUT_SECONDS", 1) + + from mt5api.handlers import chartctl + app = Flask(__name__) + app.post("/experts")(chartctl.upload_expert) + app.get("/experts")(chartctl.list_experts) + app.delete("/experts/")(chartctl.delete_expert) + app.post("/sets")(chartctl.upload_set) + app.get("/sets")(chartctl.list_sets) + app.get("/sets/")(chartctl.get_set) + app.post("/deployments")(chartctl.create_deployment) + app.get("/deployments")(chartctl.list_deployments) + app.post("/deployments/reconcile")(chartctl.reconcile) + app.get("/deployments/")(chartctl.get_deployment) + app.patch("/deployments/")(chartctl.patch_deployment) + app.delete("/deployments/")(chartctl.delete_deployment) + app.get("/charts")(chartctl.charts) + app.get("/loader")(chartctl.loader_status) + app.post("/charts//screenshot")(chartctl.screenshot) + app.post("/charts//close")(chartctl.close_chart) + + c = app.test_client() + c._proto_dir = str(proto) # stash for the fake loader + return c + + +def _upload_expert(client, name="EA.ex5", content=b"MZ\x00fakeex5"): + return client.post("/experts", data={ + "expert": (io.BytesIO(content), name)}, + content_type="multipart/form-data") + + +def _upload_set(client, name="gold.set", text="Lots=0.10\nMagic=777\n"): + return client.post("/sets", data={ + "set": (io.BytesIO(text.encode("utf-16")), name)}, + content_type="multipart/form-data") + + +# ── artifacts ──────────────────────────────────────────────────────── + +def test_expert_upload_list_dedupe(client): + r = _upload_expert(client) + assert r.status_code == 201 + sha = r.get_json()["sha256"] + # re-upload identical -> skipped + r2 = _upload_expert(client) + assert r2.get_json()["skipped"] is True + lst = client.get("/experts").get_json()["experts"] + assert any(e["name"] == "EA.ex5" and e["sha256"] == sha for e in lst) + + +def test_expert_upload_conflict_on_hash_change(client): + _upload_expert(client, content=b"one") + r = _upload_expert(client, content=b"two") + assert r.status_code == 409 + assert r.get_json()["code"] == "EXISTS" + + +def test_set_upload_returns_parsed_inputs(client): + r = _upload_set(client) + assert r.status_code == 201 + inputs = r.get_json()["inputs"] + assert {"name": "Lots", "value": "0.10"} in inputs + + +def test_expert_traversal_rejected(client): + r = client.post("/experts", data={ + "expert": (io.BytesIO(b"x"), "../evil.ex5")}, + content_type="multipart/form-data") + assert r.status_code == 400 + + +# ── deployment lifecycle with fake loader ──────────────────────────── + +def test_full_deploy_verify_cycle(client): + _upload_expert(client) + _upload_set(client) + r = client.post("/deployments", json={ + "expert": "EA.ex5", "set": "gold.set", + "symbol": "XAUUSD", "timeframe": "M5"}) + assert r.status_code == 202 + dep_id = r.get_json()["id"] + + # Before the loader runs: pending, not converged. + v = client.get("/deployments").get_json() + assert v["deployments"][0]["status"] == "pending" + assert v["converged"] is False + + # A .tpl was generated. + from mt5api.chartctl import paths + assert os.path.exists(os.path.join(paths.TEMPLATES_DIR, f"{dep_id}.tpl")) + + # Loader reconciles -> running + converged. + loader = FakeLoader(client._proto_dir) + loader.reconcile() + v = client.get("/deployments").get_json() + assert v["deployments"][0]["status"] == "running" + assert v["converged"] is True + + # /charts reflects the live inventory. + charts = client.get("/charts").get_json() + assert charts["loader_alive"] is True + assert charts["charts"][0]["symbol"] == "XAUUSD" + + +def test_deploy_requires_staged_expert(client): + r = client.post("/deployments", json={ + "expert": "NOPE.ex5", "symbol": "EURUSD", "timeframe": "H1"}) + assert r.status_code == 404 + assert r.get_json()["code"] == "ARTIFACT_NOT_FOUND" + + +def test_duplicate_chart_conflict(client): + _upload_expert(client) + client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "EURUSD", "timeframe": "H1"}) + r = client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "EURUSD", "timeframe": "H1"}) + assert r.status_code == 409 + assert r.get_json()["code"] == "DUPLICATE_CHART" + + +def test_pause_then_delete(client): + _upload_expert(client) + dep_id = client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "USDJPY", "timeframe": "M30" + }).get_json()["id"] + + # pause + r = client.patch(f"/deployments/{dep_id}", json={"enabled": False}) + assert r.status_code == 200 + loader = FakeLoader(client._proto_dir) + loader.reconcile() + item = client.get(f"/deployments/{dep_id}").get_json() + assert item["status"] == "paused" + + # delete removes the tpl and the row + r = client.delete(f"/deployments/{dep_id}") + assert r.status_code == 200 + from mt5api.chartctl import paths + assert not os.path.exists(os.path.join(paths.TEMPLATES_DIR, f"{dep_id}.tpl")) + assert client.get("/deployments").get_json()["deployments"] == [] + + +def test_loader_reports_failure(client): + _upload_expert(client) + dep_id = client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "GBPUSD", "timeframe": "M15" + }).get_json()["id"] + loader = FakeLoader(client._proto_dir) + loader.reconcile(fail_ids={dep_id}) + item = client.get(f"/deployments/{dep_id}").get_json() + assert item["status"] == "failed" + assert item["error"]["code"] == "EXPERT_NOT_ATTACHED" + + +def test_patch_set_regenerates_tpl(client): + _upload_expert(client) + _upload_set(client, name="a.set", text="Lots=0.01\n") + _upload_set(client, name="b.set", text="Lots=0.99\n") + dep_id = client.post("/deployments", json={ + "expert": "EA.ex5", "set": "a.set", + "symbol": "AUDUSD", "timeframe": "H1"}).get_json()["id"] + from mt5api.chartctl import paths + tpl = os.path.join(paths.TEMPLATES_DIR, f"{dep_id}.tpl") + before = open(tpl, "rb").read() + client.patch(f"/deployments/{dep_id}", json={"set": "b.set"}) + after = open(tpl, "rb").read() + assert b"0.99" in after.decode("utf-16").encode("utf-8") or before != after + + +def test_loader_absent_hint(client): + r = client.get("/loader").get_json() + assert r["alive"] is False + assert "hint" in r + + +def test_screenshot_via_command_channel(client, monkeypatch): + _upload_expert(client) + dep_id = client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "XAUUSD", "timeframe": "M5" + }).get_json()["id"] + loader = FakeLoader(client._proto_dir) + loader.reconcile() + charts = client.get("/charts").get_json()["charts"] + chart_id = charts[0]["chart_id"] + + # Command channel is synchronous in the handler; drive the fake loader + # from a thread so it answers while the request blocks. + import threading + import time + + def answer(): + for _ in range(20): + if loader.handle_command(): + return + time.sleep(0.05) + + t = threading.Thread(target=answer) + t.start() + r = client.post(f"/charts/{chart_id}/screenshot") + t.join() + assert r.status_code == 200 + assert r.mimetype == "image/png" + + +def _run_with_fake_loader(client, loader, method, url): + """Issue a command-channel request while the fake loader answers.""" + import threading + import time + + def answer(): + for _ in range(20): + if loader.handle_command(): + return + time.sleep(0.05) + + t = threading.Thread(target=answer) + t.start() + r = getattr(client, method)(url) + t.join() + return r + + +def test_close_chart_via_command_channel(client): + _upload_expert(client) + client.post("/deployments", json={ + "expert": "EA.ex5", "symbol": "XAUUSD", "timeframe": "M5"}) + loader = FakeLoader(client._proto_dir) + loader.reconcile() + chart_id = client.get("/charts").get_json()["charts"][0]["chart_id"] + + r = _run_with_fake_loader(client, loader, "post", f"/charts/{chart_id}/close") + assert r.status_code == 200 + assert r.get_json() == {"closed": chart_id} + + +def test_close_chart_loader_failure(client): + loader = FakeLoader(client._proto_dir) + loader.reconcile() + # chart_id -1 is the fake loader's CLOSE_FAILED sentinel + r = _run_with_fake_loader(client, loader, "post", "/charts/-1/close") + assert r.status_code == 502 + assert r.get_json()["code"] == "CLOSE_FAILED" + + +def test_close_chart_bad_id(client): + r = client.post("/charts/notanint/close") + assert r.status_code == 400 diff --git a/tests/test_chartctl_units.py b/tests/test_chartctl_units.py new file mode 100644 index 00000000..3e26087c --- /dev/null +++ b/tests/test_chartctl_units.py @@ -0,0 +1,251 @@ +"""Unit tests for chartctl: path safety, set parsing, tpl generation, +registry desired-state + status derivation. + +All Linux-safe: no MT5 SDK, no Windows. The registry/paths modules are +repointed at a tmp dir per test via monkeypatch, mirroring the backtest +jobs test fixture. +""" +from __future__ import annotations + +import json +import os + +import pytest + + +# ── paths.safe_name ────────────────────────────────────────────────── + +@pytest.mark.parametrize("bad", [ + "", " ", "..", ".", "a/b.ex5", "a\\b.ex5", "../evil.ex5", + "C:\\evil.ex5", "\\\\host\\share\\x.ex5", ".hidden.ex5", + "na.ex5", 'quote".ex5', "pipe|.ex5", "null\x00.ex5", +]) +def test_safe_name_rejects(bad): + from mt5api.chartctl import paths + with pytest.raises(ValueError): + paths.safe_name(bad, "expert", ".ex5") + + +def test_safe_name_accepts_and_checks_ext(): + from mt5api.chartctl import paths + assert paths.safe_name("HappyGoldScalp.ex5", "expert", ".ex5") \ + == "HappyGoldScalp.ex5" + with pytest.raises(ValueError): + paths.safe_name("HappyGoldScalp.set", "expert", ".ex5") + + +# ── setparse ───────────────────────────────────────────────────────── + +def test_parse_plain_and_optimized_set(): + from mt5api.chartctl.setparse import parse_set_text + text = ( + "; comment line\n" + "Lots=0.10\n" + "StopLoss=50||10||5||100||Y\n" + "UseTrailing=1\n" + ) + got = parse_set_text(text) + assert got[0] == {"name": "Lots", "value": "0.10"} + assert got[1]["name"] == "StopLoss" + assert got[1]["value"] == "50" + assert got[1]["optimize"] is True + assert got[1]["start"] == "10" and got[1]["stop"] == "100" + assert got[2] == {"name": "UseTrailing", "value": "1"} + + +def test_parse_utf16_bytes(): + from mt5api.chartctl.setparse import parse_set_bytes + raw = "Lots=0.01\nMagic=12345\n".encode("utf-16") + got = parse_set_bytes(raw) + assert {"name": "Lots", "value": "0.01"} in got + assert {"name": "Magic", "value": "12345"} in got + + +def test_parse_ascii_bytes_even_length(): + # Even-length ASCII input decodes "successfully" as utf-16 garbage if + # utf-16 is blind-tried first, silently yielding zero inputs — and the + # deployment would run on EA defaults. Regression for that decode bug. + from mt5api.chartctl.setparse import parse_set_bytes + raw = b"; comment\r\nLots=0.10\r\nMagic=99\r\n" + assert len(raw) % 2 == 0 + got = parse_set_bytes(raw) + assert {"name": "Lots", "value": "0.10"} in got + assert {"name": "Magic", "value": "99"} in got + + +def test_parse_bomless_utf16_bytes(): + from mt5api.chartctl.setparse import parse_set_bytes + raw = "Lots=0.01\nMagic=12345\n".encode("utf-16-le") # no BOM + got = parse_set_bytes(raw) + assert {"name": "Lots", "value": "0.01"} in got + assert {"name": "Magic", "value": "12345"} in got + + +# ── tpl_builder ────────────────────────────────────────────────────── + +def test_tpl_text_structure(): + from mt5api.chartctl import tpl_builder + text = tpl_builder.build_tpl_text( + deployment_id="dep_abc123", + expert_name="HappyGoldScalp", + expert_rel_path="Experts\\Uploaded\\HappyGoldScalp.ex5", + inputs=[{"name": "Lots", "value": "0.10"}, + {"name": "Magic", "value": "777"}], + terminal_build=4620, + ) + assert "" in text and "" in text + assert "" in text and "" in text + assert "name=HappyGoldScalp" in text + assert "path=Experts\\Uploaded\\HappyGoldScalp.ex5" in text + assert "expertmode=1" in text + assert "__chartctl_id=dep_abc123" in text # attribution input + assert "Lots=0.10" in text and "Magic=777" in text + assert "build=4620" in text # forensic stamp + assert text.endswith("\r\n") + + +def test_tpl_written_as_utf16_with_bom(tmp_path, monkeypatch): + from mt5api.chartctl import tpl_builder, paths + monkeypatch.setattr(paths, "TEMPLATES_DIR", str(tmp_path)) + monkeypatch.setattr(tpl_builder, "TEMPLATES_DIR", str(tmp_path)) + path = tpl_builder.write_tpl( + deployment_id="dep_x", expert_name="EA", + expert_rel_path="Experts\\Uploaded\\EA.ex5", inputs=[]) + data = open(path, "rb").read() + assert data[:2] == b"\xff\xfe" # UTF-16-LE BOM + assert "name=EA" in data.decode("utf-16") + + +# ── registry ───────────────────────────────────────────────────────── + +@pytest.fixture +def reg(monkeypatch, tmp_path): + from mt5api.chartctl import registry, paths + monkeypatch.setattr(paths, "REGISTRY_PATH", str(tmp_path / "registry.json")) + monkeypatch.setattr(paths, "DESIRED_PATH", str(tmp_path / "desired.json")) + monkeypatch.setattr(paths, "OBSERVED_PATH", str(tmp_path / "observed.json")) + monkeypatch.setattr(paths, "PROTOCOL_DIR", str(tmp_path)) + monkeypatch.setattr(paths, "TEMPLATES_DIR", str(tmp_path)) + monkeypatch.setattr(registry, "_STATE", None) + return registry, tmp_path + + +def test_add_bumps_revision_and_writes_desired(reg): + registry, tmp = reg + d = registry.add_deployment( + expert_file="EA.ex5", expert_name="EA", set_file=None, + symbol="XAUUSD", timeframe="M5") + assert d["id"].startswith("dep_") + assert registry.current_revision() == 1 + desired = json.loads((tmp / "desired.json").read_text()) + assert desired["revision"] == 1 + assert desired["deployments"][0]["symbol"] == "XAUUSD" + assert desired["deployments"][0]["template"].startswith("\\Files\\chartctl\\") + + +def test_duplicate_enabled_chart_rejected(reg): + registry, _ = reg + registry.add_deployment(expert_file="A.ex5", expert_name="A", + set_file=None, symbol="EURUSD", timeframe="H1") + with pytest.raises(registry.DuplicateChart): + registry.add_deployment(expert_file="B.ex5", expert_name="B", + set_file=None, symbol="EURUSD", timeframe="H1") + + +def test_disabled_does_not_conflict(reg): + registry, _ = reg + registry.add_deployment(expert_file="A.ex5", expert_name="A", + set_file=None, symbol="EURUSD", timeframe="H1", + enabled=False) + # Same slot, enabled — must be allowed since the first is paused. + registry.add_deployment(expert_file="B.ex5", expert_name="B", + set_file=None, symbol="EURUSD", timeframe="H1") + + +def test_persistence_across_reload(reg): + registry, tmp = reg + registry.add_deployment(expert_file="EA.ex5", expert_name="EA", + set_file=None, symbol="GBPUSD", timeframe="M15") + registry._STATE = None # simulate API restart + deps = registry.list_deployments() + assert len(deps) == 1 and deps[0]["symbol"] == "GBPUSD" + + +def test_remove_bumps_and_clears(reg): + registry, _ = reg + d = registry.add_deployment(expert_file="EA.ex5", expert_name="EA", + set_file=None, symbol="USDJPY", timeframe="H4") + registry.remove_deployment(d["id"]) + assert registry.list_deployments() == [] + assert registry.current_revision() == 2 + + +def test_merged_view_status_pending_without_observed(reg): + registry, _ = reg + registry.add_deployment(expert_file="EA.ex5", expert_name="EA", + set_file=None, symbol="XAUUSD", timeframe="M5") + view = registry.merged_view() + assert view["deployments"][0]["status"] == "pending" + assert view["converged"] is False + assert view["observed_stale"] is True + + +def test_merged_view_running_when_observed_matches(reg): + registry, tmp = reg + d = registry.add_deployment(expert_file="EA.ex5", expert_name="EA", + set_file=None, symbol="XAUUSD", timeframe="M5") + observed = { + "loader": {"applied_revision": registry.current_revision(), + "last_loop": "now"}, + "terminal": {"auto_trading": True}, + "charts": [{"chart_id": 1, "expert": "EA", "deployment_id": d["id"]}], + "deployments": [{"id": d["id"], "status": "running", "chart_id": 1}], + } + (tmp / "observed.json").write_text(json.dumps(observed)) + view = registry.merged_view() + assert view["deployments"][0]["status"] == "running" + + +# ── Numeric configuration is clamped, not trusted ──────────────────────────── +# +# Same class as the vm-watchdog finding on PR #15: invalid configuration must +# not be taken literally. `or ` covers absent and zero but not +# negative — parse_duration_to_seconds accepts "-5s" on purpose, because +# west-of-UTC broker offsets need the sign. +# +# Clamped rather than refused: this module is imported by the whole API, so +# raising on an optional feature's tuning value would stop trading too. + +from mt5api.config import _chartctl_bytes, _chartctl_seconds + + +@pytest.mark.parametrize("raw", ["-5s", "-1h", "-90m"]) +def test_a_negative_duration_is_clamped_to_the_floor(raw): + """A negative stale window calls every observation stale; a negative hint + interval is permanently due. Both look like a broken loader.""" + assert _chartctl_seconds(raw, "60s", 1) >= 1 + + +@pytest.mark.parametrize("raw", [None, "", 0]) +def test_absent_and_zero_fall_back_to_the_default(raw): + assert _chartctl_seconds(raw, "60s", 1) == 60 + + +@pytest.mark.parametrize("raw,expected", [("12s", 12), ("2m", 120), ("1h", 3600)]) +def test_valid_durations_are_left_alone(raw, expected): + """The clamp must not quietly rewrite a configuration someone meant.""" + assert _chartctl_seconds(raw, "60s", 1) == expected + + +def test_a_negative_upload_cap_cannot_reject_every_file(): + """read(cap + 1) with cap = -1 reads nothing, then len(0) > -1 rejects the + upload while reporting a limit of -1 bytes.""" + assert _chartctl_bytes(-1, 16 * 1024 * 1024, 1024) >= 1024 + + +def test_absent_upload_cap_uses_the_default(): + assert _chartctl_bytes(None, 16 * 1024 * 1024, 1024) == 16 * 1024 * 1024 + + +def test_a_valid_upload_cap_is_left_alone(): + assert _chartctl_bytes(32 * 1024 * 1024, 16 * 1024 * 1024, 1024) == 32 * 1024 * 1024 diff --git a/tests/test_config_generation.py b/tests/test_config_generation.py index 9b03310f..b2243687 100644 --- a/tests/test_config_generation.py +++ b/tests/test_config_generation.py @@ -184,6 +184,65 @@ def test_live_terminal_ini_declares_no_startup_expert(tmp_path, monkeypatch): assert content.count("[Experts]") == 1 +def test_startup_expert_is_written_only_when_chartctl_is_opted_in(tmp_path, monkeypatch): + """The other half of the guard above. + + That test pins the default; this one pins that the feature still works when + asked for, so "no [StartUp]" cannot be satisfied by quietly breaking the + loader bootstrap. Chart Deployments is opt-IN: an upgrade must not start + attaching an EA to every live terminal in a fleet that never enabled it. + """ + helper = _load_config_helper_module() + path = tmp_path / "config.yaml" + path.write_text( + yaml.safe_dump({ + "api_token": "test-token", + "chartctl": {"enabled": True}, + "terminals": [{"broker": "acme", "account": "main", "port": 5001}], + }), + encoding="utf-8", + ) + outpath = tmp_path / "terminal.ini" + monkeypatch.setattr(helper, "CONFIG_PATH", str(path)) + monkeypatch.setattr( + "sys.argv", + ["config_helper.py", "write_ini", "acme", "main", str(outpath), "default", "live"], + ) + + helper.main() + + content = outpath.read_text(encoding="utf-8") + assert "[StartUp]" in content + assert "Expert=Advisors\\MT5ChartLoader" in content + + +def test_a_terminal_can_opt_out_even_when_chartctl_is_on(tmp_path, monkeypatch): + """Per-terminal `chartctl: false` has to beat the global enable, or there is + no way to keep one terminal clear of the loader.""" + helper = _load_config_helper_module() + path = tmp_path / "config.yaml" + path.write_text( + yaml.safe_dump({ + "api_token": "test-token", + "chartctl": {"enabled": True}, + "terminals": [ + {"broker": "acme", "account": "main", "port": 5001, "chartctl": False} + ], + }), + encoding="utf-8", + ) + outpath = tmp_path / "terminal.ini" + monkeypatch.setattr(helper, "CONFIG_PATH", str(path)) + monkeypatch.setattr( + "sys.argv", + ["config_helper.py", "write_ini", "acme", "main", str(outpath), "default", "live"], + ) + + helper.main() + + assert "[StartUp]" not in outpath.read_text(encoding="utf-8") + + def test_clean_start_uses_single_vm_compose_without_explicit_topology(tmp_path): repo_root = Path(__file__).resolve().parents[1] run_script = (repo_root / "run.sh").read_text(encoding="utf-8") diff --git a/tests/test_webrequest.py b/tests/test_webrequest.py new file mode 100644 index 00000000..a7648e8d --- /dev/null +++ b/tests/test_webrequest.py @@ -0,0 +1,338 @@ +"""Tests for the WebRequest allowlist codec + manager + endpoint. + +Runs entirely on Linux (no MT5/Windows): the codec is pure Python and the +manager is plain file I/O. The live apply (terminal restart writing common.ini) +is exercised via a stubbed restart_terminal. +""" +from __future__ import annotations + +import json +import os + +import pytest +from flask import Flask + +from mt5api.chartctl import webrequest as wr + +# A real captured broker blob and its plaintext URLs (verified byte-identical). +REAL_BLOB = ( + "13E33B7856715A56F2822DF172EBC0D0BD8DF23E89A42B2716A602C68D06506070409EEA021DA6A29" + "525874B0D86E7F7D8A8FC4898B30E0A3DCDFBBF9D166474552580CC55704642FD8D1DE13AB3CADA08D8" + "DC28AFCA0C080292FABED14A1828BA8A1B67BCD700FC69F9D094E55E2A3AAE7E17637D986B67D868511" + "562DB" +) +REAL_URLS = ["https://tracker.algotradingspace.com", "https://api.telegram.org"] + +_BOM = b"\xff\xfe" + + +def _write_utf16_ini(path: str, text: str) -> None: + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "wb") as f: + f.write(_BOM + text.replace("\n", "\r\n").encode("utf-16-le")) + + +# ── codec ──────────────────────────────────────────────────────────── +def test_codec_roundtrip_real_blob(): + assert wr.decode_blob(REAL_BLOB) == REAL_URLS + assert wr.encode_urls(REAL_URLS) == REAL_BLOB + + +def test_codec_roundtrip_arbitrary(): + urls = ["https://api.example.com", "http://nfs.faireconomy.media/x.xml"] + assert wr.decode_blob(wr.encode_urls(urls)) == urls + + +# ── url hygiene ────────────────────────────────────────────────────── +def test_clean_urls_filters_and_dedupes(): + got = wr.clean_urls([ + " https://a.com ", "https://a.com", "ftp://no.com", + "https://b.com;evil", "", 42, "HTTPS://C.com", + ]) + assert got == ["https://a.com", "HTTPS://C.com"] + + +# ── desired store ──────────────────────────────────────────────────── +def test_desired_store_roundtrip(tmp_path): + cfg = str(tmp_path / "Config") + assert wr.load_desired(cfg) is None + wr.save_desired(cfg, REAL_URLS) + assert wr.load_desired(cfg) == REAL_URLS + with open(wr.desired_path(cfg)) as f: + assert json.load(f)["urls"] == REAL_URLS + + +# ── migrate-from-current ───────────────────────────────────────────── +def test_read_current_urls_from_existing_common_ini(tmp_path): + cfg = str(tmp_path / "Config") + _write_utf16_ini( + wr.common_ini_path(cfg), + f"[Charts]\nProfileLast=Default\n[Experts]\nAllowDllImport=1\n" + f"WebRequest=1\nWebRequestUrl={REAL_BLOB}\n[Objects]\nShow=0\n", + ) + assert wr.read_current_urls(cfg) == REAL_URLS + + +def test_read_current_urls_absent(tmp_path): + assert wr.read_current_urls(str(tmp_path / "Config")) == [] + + +def test_effective_prefers_desired_then_migrates(tmp_path): + cfg = str(tmp_path / "Config") + _write_utf16_ini( + wr.common_ini_path(cfg), + f"[Experts]\nWebRequest=1\nWebRequestUrl={REAL_BLOB}\n", + ) + # no desired yet -> migrate from current + assert wr.effective_urls(cfg) == REAL_URLS + wr.save_desired(cfg, ["https://only.example.com"]) + assert wr.effective_urls(cfg) == ["https://only.example.com"] + + +# ── write_common_ini: preserve everything, upsert the two keys ─────── +def test_write_common_ini_preserves_other_sections(tmp_path): + cfg = str(tmp_path / "Config") + _write_utf16_ini( + wr.common_ini_path(cfg), + "[Charts]\nProfileLast=Default\n[Experts]\nAllowDllImport=1\n" + "WebRequest=0\nWebRequestUrl=DEADBEEF\n[Objects]\nShow=0\n", + ) + new = ["https://api.example.com"] + wr.write_common_ini(cfg, new) + lines = wr._read_lines(wr.common_ini_path(cfg)) + assert "[Charts]" in lines and "ProfileLast=Default" in lines + assert "[Objects]" in lines and "Show=0" in lines + assert "AllowDllImport=1" in lines # sibling key preserved + assert "WebRequest=1" in lines # flipped on + assert wr.read_current_urls(cfg) == new # blob replaced, decodes to new + # exactly one WebRequestUrl line + assert sum(l.strip().lower().startswith("webrequesturl=") for l in lines) == 1 + + +def test_write_common_ini_creates_file_and_section(tmp_path): + cfg = str(tmp_path / "Config") + wr.write_common_ini(cfg, REAL_URLS) + assert os.path.exists(wr.common_ini_path(cfg)) + with open(wr.common_ini_path(cfg), "rb") as f: + assert f.read(2) == _BOM # UTF-16LE BOM preserved + assert wr.read_current_urls(cfg) == REAL_URLS + + +def test_write_common_ini_is_utf16(tmp_path): + cfg = str(tmp_path / "Config") + wr.write_common_ini(cfg, REAL_URLS) + with open(wr.common_ini_path(cfg), "rb") as f: + raw = f.read() + assert b"\x00" in raw # wide chars + assert raw.decode("utf-16").count("[Experts]") == 1 + + +def test_apply_from_desired(tmp_path): + term = str(tmp_path / "term") + cfg = wr.config_dir(term) + assert wr.apply_from_desired(term) is None # no desired -> no-op + assert not os.path.exists(wr.common_ini_path(cfg)) + wr.save_desired(cfg, REAL_URLS) + assert wr.apply_from_desired(term) == len(REAL_URLS) + assert wr.read_current_urls(cfg) == REAL_URLS + + +# ── endpoint ───────────────────────────────────────────────────────── +@pytest.fixture +def client(monkeypatch, tmp_path): + from mt5api.handlers import webrequest as handler + + term = str(tmp_path / "term") + os.makedirs(wr.config_dir(term), exist_ok=True) + monkeypatch.setattr(handler, "TERMINAL_DIR", term) + + calls = {"restart": 0} + + def fake_restart(): + calls["restart"] += 1 + # emulate the real restart applying desired -> common.ini + wr.apply_from_desired(term) + return True + + monkeypatch.setattr(handler, "restart_terminal", fake_restart) + + # default: no AutoIt (bare-metal fallback path) unless a test flips it + from mt5api.chartctl import autoit_webrequest as autoit + monkeypatch.setattr(autoit, "available", lambda: False) + + app = Flask(__name__) + app.get("/webrequest")(handler.get_webrequest) + app.put("/webrequest")(handler.put_webrequest) + app.post("/webrequest/apply")(handler.apply_webrequest) + c = app.test_client() + c._term = term + c._calls = calls + c._autoit = autoit + return c + + +def test_get_empty(client): + r = client.get("/webrequest") + assert r.status_code == 200 and r.get_json() == {"urls": []} + + +def test_put_replace_restarts_and_applies(client): + r = client.put("/webrequest", json={"urls": REAL_URLS}) + assert r.status_code == 200 + body = r.get_json() + assert body["success"] is True and body["urls"] == REAL_URLS + assert body["applied_via"] == "restart" # bare-metal fallback path + assert client._calls["restart"] == 1 + # persisted + written to common.ini + assert wr.load_desired(wr.config_dir(client._term)) == REAL_URLS + assert wr.read_current_urls(wr.config_dir(client._term)) == REAL_URLS + # readable back through GET + assert client.get("/webrequest").get_json()["urls"] == REAL_URLS + + +def test_put_add_remove_migrates_from_current(client): + cfg = wr.config_dir(client._term) + _write_utf16_ini(wr.common_ini_path(cfg), + f"[Experts]\nWebRequest=1\nWebRequestUrl={REAL_BLOB}\n") + r = client.put("/webrequest", json={ + "add": ["https://new.example.com"], + "remove": ["https://api.telegram.org"], + }) + assert r.status_code == 200 + assert r.get_json()["urls"] == [ + "https://tracker.algotradingspace.com", "https://new.example.com", + ] + + +def test_put_uses_autoit_when_available(client, monkeypatch): + """On the VM (AutoIt present) the allowlist is applied via the Options + dialog, not a terminal restart.""" + seen = {} + + def fake_apply(urls, timeout=120, use_runas=False): + seen["urls"] = list(urls) + return "OK", "log" + + monkeypatch.setattr(client._autoit, "available", lambda: True) + monkeypatch.setattr(client._autoit, "apply_urls", fake_apply) + + r = client.put("/webrequest", json={"urls": REAL_URLS}) + assert r.status_code == 200 + body = r.get_json() + assert body["success"] is True and body["applied_via"] == "autoit:OK" + assert seen["urls"] == REAL_URLS + assert client._calls["restart"] == 0 # no restart on the AutoIt path + assert wr.load_desired(wr.config_dir(client._term)) == REAL_URLS + + +def test_put_autoit_failure_returns_500(client, monkeypatch): + monkeypatch.setattr(client._autoit, "available", lambda: True) + monkeypatch.setattr(client._autoit, "apply_urls", lambda urls, timeout=120, use_runas=False: ("FAIL", "boom")) + r = client.put("/webrequest", json={"urls": REAL_URLS}) + assert r.status_code == 500 + # desired still persisted so a later /apply or boot can retry + assert wr.load_desired(wr.config_dir(client._term)) == REAL_URLS + + +def test_apply_endpoint_reapplies_desired(client, monkeypatch): + seen = {} + monkeypatch.setattr(client._autoit, "available", lambda: True) + monkeypatch.setattr( + client._autoit, "apply_urls", + lambda urls, timeout=120, use_runas=False: (seen.update(urls=list(urls)) or ("OK", "")), + ) + wr.save_desired(wr.config_dir(client._term), REAL_URLS) + r = client.post("/webrequest/apply") + assert r.status_code == 200 + assert r.get_json()["urls"] == REAL_URLS + assert seen["urls"] == REAL_URLS + + +def test_apply_endpoint_noop_when_empty(client): + r = client.post("/webrequest/apply") + assert r.status_code == 200 + assert r.get_json()["urls"] == [] + + +def test_put_rejects_non_dict(client): + assert client.put("/webrequest", data="nope").status_code == 400 + + +def test_put_rejects_empty_body(client): + assert client.put("/webrequest", json={}).status_code == 400 + + +def test_put_restart_failure_returns_500(client, monkeypatch): + from mt5api.handlers import webrequest as handler + monkeypatch.setattr(handler, "restart_terminal", lambda: False) + r = client.put("/webrequest", json={"urls": REAL_URLS}) + assert r.status_code == 500 + # desired still persisted (next boot/restart will apply it) + assert wr.load_desired(wr.config_dir(client._term)) == REAL_URLS + + +# ── config_helper boot-seed (start.bat deletes common.ini every boot) ─ +def _load_config_helper(): + import importlib.util + root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + path = os.path.join(root, "scripts", "config_helper.py") + spec = importlib.util.spec_from_file_location("config_helper_under_test", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +def test_config_helper_write_ini_seeds_common_ini(tmp_path, monkeypatch): + ch = _load_config_helper() + cfg_yaml = tmp_path / "config.yaml" + cfg_yaml.write_text( + "chartctl:\n enabled: true\n" + "accounts:\n testbroker:\n acct1:\n" + " login: 123\n server: TestServer\n password: pw\n" + "terminals:\n - broker: testbroker\n account: acct1\n" + " instance: default\n port: 6542\n mode: live\n" + ) + monkeypatch.setattr(ch, "CONFIG_PATH", str(cfg_yaml)) + + term = tmp_path / "term" + (term / "Config").mkdir(parents=True) + outpath = term / "mt5start.ini" + # simulate a terminal that already has a desired allowlist persisted + wr.save_desired(str(term / "Config"), REAL_URLS) + + monkeypatch.setattr( + ch.sys, "argv", + ["config_helper.py", "write_ini", "testbroker", "acct1", str(outpath), + "default", "live"], + ) + ch.main() + + # mt5start.ini written with the loader StartUp block (chartctl on) + start_ini = outpath.read_text() + assert "[StartUp]" in start_ini and "MT5ChartLoader" in start_ini + # common.ini re-emitted from the desired file, decodes back to the URLs + assert wr.read_current_urls(str(term / "Config")) == REAL_URLS + + +def test_config_helper_write_ini_no_desired_no_common_ini(tmp_path, monkeypatch): + ch = _load_config_helper() + cfg_yaml = tmp_path / "config.yaml" + cfg_yaml.write_text( + "chartctl:\n enabled: true\n" + "accounts:\n testbroker:\n acct1:\n" + " login: 123\n server: TestServer\n password: pw\n" + "terminals:\n - broker: testbroker\n account: acct1\n" + " instance: default\n port: 6542\n mode: live\n" + ) + monkeypatch.setattr(ch, "CONFIG_PATH", str(cfg_yaml)) + term = tmp_path / "term" + (term / "Config").mkdir(parents=True) + outpath = term / "mt5start.ini" + monkeypatch.setattr( + ch.sys, "argv", + ["config_helper.py", "write_ini", "testbroker", "acct1", str(outpath), + "default", "live"], + ) + ch.main() + # no desired file -> boot-seed is a no-op, common.ini left absent + assert not os.path.exists(wr.common_ini_path(str(term / "Config")))