Add async dataset query jobs - #11
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces an async job-based API for long-running FinMind dataset queries, enabling callers to start a query and poll for completion using a durable, SQLite-backed handle.
Changes:
- Added
start_query_dataset_job/check_query_dataset_jobtools to support async dataset querying viahandle_id. - Implemented a minimal SQLite-backed durable job store keyed by
handle_id. - Updated tests and stdio smoke checks to validate the expanded tool surface and async job behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/finmind_mcp/tools.py |
Registers new async job tools and wires dispatch/handlers to start and poll durable jobs. |
src/finmind_mcp/jobs.py |
Adds a lightweight SQLite-backed job store for persisting job status/results by handle_id. |
tests/test_tools.py |
Expands tool definition expectations and adds async job lifecycle/durability tests. |
smoke.py |
Updates smoke test to assert the full expected tool list (now includes async job tools). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| arguments: dict[str, Any], | ||
| original_query: str, | ||
| ) -> JobRecord: | ||
| handle_id = uuid.uuid4().hex[:12] |
| This is intentionally minimal: the MCP tool returns a handle immediately, | ||
| background work updates the row, and polling can resume from another store | ||
| instance in the same process or after a process restart. |
| def _make_job_store() -> SQLiteJobStore: | ||
| """Factory hook for async handleId jobs.""" | ||
| db_path = os.environ.get("FINMIND_MCP_JOB_DB") | ||
| if not db_path: | ||
| db_path = str(Path(tempfile.gettempdir()) / "finmind_mcp_jobs.sqlite3") | ||
| return SQLiteJobStore(db_path) |
linsamtw
left a comment
linsamtw
left a comment
There was a problem hiding this comment.
感謝貢獻!非同步化長查詢的動機可以理解,不過目前的實作有幾個必須先解的問題,以及一個希望你先回答的前提問題。
前提:實際 timeout 場景
一般 /v4/data 查詢多在幾秒內回應,而 MCP host 的 tool timeout 通常是 60–120 秒;真正慢的重資料集(tick 級)本來就不走 generic dispatch。能否提供一個實際會 timeout 的具體案例(dataset + 參數 + 實測耗時)? 若沒有真實痛點,多兩個工具會稀釋 tool surface(模型選錯工具機率上升),我們傾向保持精簡。
Blockers
-
Python 3.10 會掛:
jobs.py的from datetime import UTC是 3.11+ 才有(datetime.UTCalias 於 3.11 新增),本專案requires-python >= 3.10、CI matrix 含 3.10,import 會直接 ImportError。請改用from datetime import timezone+timezone.utc。 -
背景任務可能被 GC:
asyncio.create_task(...)的回傳值沒有保留 reference,CPython 文件明確警告 task 可能中途被垃圾回收(https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task)。請保留強引用(例如 module-level set + done callback discard)。 -
重啟後的殭屍 job:SQLite 只保住 job 那一列,背景 coroutine 不會跨 process 存活。server 重啟後所有
processing的 job 會永遠輪詢不完(無 TTL、無 stale 判定、無清理),對模型比直接回 timeout 錯誤更誤導。至少需要:stale 判定(如 updated_at 超過 N 分鐘即回 FAILED)+ 過期資料清理。 -
共用 temp 路徑的安全/隱私問題:預設
tempfile.gettempdir()/finmind_mcp_jobs.sqlite3在多人共用機器上是全域共享路徑——其他本機使用者可預先佔住該檔(symlink / 權限),也能讀到查詢紀錄與付費資料結果。請改預設到 per-user 位置(如~/.cache/finmind-mcp/)並限制檔案權限。
需同步的文件(本 repo 的硬規則)
knowledge/ 是 MCP server 與 Custom GPT 共用的 single source of truth,新增工具必須同步,否則就是我們用 CI gate 在防的 drift:
knowledge/instructions.md(工具清單與使用時機)chatgpt/openapi.yaml/docs/spec.md(工具面描述)install/、README 若有提及工具數量處
其他建議
original_query參數建議拿掉:host 本來就保有對話上下文,由呼叫端把使用者問題塞進工具參數是冗餘的,也讓 schema 變複雜。handle_id直接用uuid4().hex全長即可,截 12 碼沒有必要。
先把前提問題(實測 timeout 案例)回覆後,我們再決定要不要往下走這個方向。再次感謝!
Summary
start_query_dataset_jobandcheck_query_dataset_jobfor long-running FinMind dataset callshandle_idoriginal_queryin polling responses so async results keep user contextVerification
uv run pytest -q→ 35 passeduv run python smoke.py→ SMOKE OKuv build→ sdist and wheel builtuvx twine check dist/*→ PASSEDuv run python chatgpt/build_instructions.py→ OK, instructions 7986/8000 chars