Repository navigation
Standalone host: serve without Ray (python -m tabctx.serve) - #83
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #83 +/- ##
==========================================
+ Coverage 84.78% 88.74% +3.95%
==========================================
Files 56 59 +3
Lines 5021 5108 +87
==========================================
+ Hits 4257 4533 +276
+ Misses 764 575 -189
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
The service moves to serve/service.py, host-agnostic and importable without ray; serve/app.py keeps its import path and is now only the Ray host (the deployment settings and the consistent-hash affinity router). serve/standalone.py serves the same routes from one plain uvicorn process for platforms that don't run Ray, with a /livez liveness route (503 only on a dead per-GPU worker; /readyz also 503s on transient pressure, which must not restart a replica) and a request limiter replicating max_ongoing_requests/max_queued_requests shedding. New 'standalone' extra ([serve] minus ray). A Dockerfile and docker_image.yml build and smoke-test a serving image on release tags. One subtlety: serve.ingress rewrites each route's 'self' into a Depends on the handler functions' own __signature__, then freezes the app with cloudpickle. With the handlers now importable in pristine form from serve/service.py, cloudpickle shipped them by reference and replicas lost the rewrite (a 422 demanding 'self' as a query parameter on every route, caught by the integration suite). app.py registers the service module for by-value pickling, the semantics the old single-module layout had implicitly.
c0b6085 to
1b178ff
Compare
|
Automated first-pass review by Claude (AI), not a human review. @amrit110 will follow up. Reviewed against how inference-platform#204 runs this server ( 1. Under #204's limits,
#204's probe times out at 10 s, and three failures (about 90 s) remove the endpoint, so readiness would flap under load. Options: serve 2. The image has no C compiler, so TabICL's 3. Only torch is pinned in the image. 4. The branch doesn't merge cleanly into main ( Verified matching #204's contract:
Unit tests: 602 passed, 12 skipped. |
# Conflicts: # .dockerignore # CHANGELOG.md # uv.lock
- Workers answer /readyz's report on their own thread. It waited behind fits in the shared 16-thread pool, so more than 16 calls queued on one device failed readiness on a healthy replica. - The image installs gcc, so torch.compile can build Triton's C helper instead of silently running eager kernels. - The image installs every dependency at its uv.lock version, except torch's own stack, which keeps the image's torch pin. CI checks both, and builds the image when pyproject.toml or uv.lock change.
inference-platform is moving its LLM serving to Dynamo, which cannot host tabctx (VectorInstitute/inference-platform#192). This adds a Ray-free host for the same service, so tabctx can run as a plain container behind a gateway. The Ray deployment path is unchanged.
What changed:
serve/service.py: the service, moved out of app.py verbatim; importable without ray. New:/livez,close(), and aROUTE_SPECSsnapshot of the registered routes.serve/app.py: sametabctx.serve.app:appimport path, now only the Ray host (deployment settings and the affinity router).serve/standalone.py:python -m tabctx.serveserves the same routes under uvicorn. Newstandaloneextra,[serve]minus ray. Single replica only, since the affinity router is Ray's.GET /livezischeck_healthas HTTP for liveness probes: 503 only on a dead per-GPU worker./readyzstays the readiness probe (it also 503s on transient pressure, which must not restart the process).max_ongoing_requests/max_queued_requestsshedding (TABCTX_MAX_ONGOING_REQUESTS/TABCTX_MAX_QUEUED_REQUESTS, defaults 8/16; probes exempt).Dockerfile+docker_image.yml: a serving image (python:3.12-slim, pinned torch,tabctx[standalone,tabicl,tabpfn], non-root, weights mounted not baked), pushed toghcr.io/vectorinstitute/tabctx:<version>on release tags, built and smoke-tested on PRs. Another registry is a few lines if preferred.One subtlety for review:
serve.ingressrewrites each route'sselfinto a Depends by mutating the handler functions'__signature__, then freezes the app with cloudpickle. Once the handlers became importable in pristine form from service.py, cloudpickle shipped them by reference and replicas lost the rewrite (422 demandingselfas a query parameter on every route; the integration tests caught it). app.py now registers the service module for by-value pickling, the semantics the old single-module layout had implicitly.Tested: unit suite in a ray-less environment (602 passed; new standalone tests cover fit/predict inline and multipart, tenancy, 404/422, the limiter, livez, and a fresh-interpreter no-ray import check), integration suite on a live two-replica Ray Serve cluster (42 passed, ray 2.59.0), a real uvicorn boot exercised over a socket, and the image workflow's fake-backend smoke test in CI. Not validated on a GPU; that stays with the deploying platform's rehearsal.