Skip to content

admin-memory-optimization-handoff.md

Latest commit

 

History

History
488 lines (401 loc) · 24.2 KB

admin-memory-optimization-handoff.md

File metadata and controls

488 lines (401 loc) · 24.2 KB
Raw

RoleFit Admin Memory Optimization Handoff

Document Purpose

This records the completed RoleFit Admin memory work and the deferred Phase 3 proposal. It is written for fresh coding-agent windows that do not have the investigation context. Read this file completely before changing code.

Project root: /home/ubuntu/workspace/projects/zhitu-job-match

Live Admin service: zhitu-admin.service, 127.0.0.1:3061, public URL https://zhitu-admin.6767.chat/

Student service: zhitu-student.service, 127.0.0.1:3060

Vector service: zhitu-qdrant.service, 127.0.0.1:6333

Coordination Rule

Phase 3 is deferred after the completed Phase 2 work. Do not begin it unless a measured restart condition in the Phase 3 proposal is met. If it is resumed, it must start from the completed Phase 2 commit and fresh measurements. The phases must not be implemented concurrently because both modify runtime_state.py, job_data_service.py, matching_service.py, tag_sync.py, and related tests.

Do not reset or revert unknown worktree changes. Check git status --short before editing. Preserve the public API response shapes and URLs.

Baseline and Completed Work

Original condition

  • Full Admin import: about 146 MB RSS.
  • Full runtime after loading 10,822 jobs: about 410 MB RSS.
  • Process startup high-water mark: about 604 MB RSS.
  • systemd cgroup startup peak: about 684 MB.
  • Startup duration: about 11 seconds.
  • About 312 MB was eventually swapped out, so a later Admin request could make memory appear to jump back toward 420 MB.

Phase 1 status: COMPLETE

Implementation commit: b8b46e8 (Reduce Zhitu admin startup memory)

Phase 1 purpose was to remove avoidable allocation peaks and eager dependency cost without changing the runtime architecture or API behavior.

Completed changes:

  • SQLite job loading now normalizes one row at a time instead of retaining the complete SQLite result, raw decoded jobs, and normalized jobs together.
  • Pandas and LangChain/OpenAI imports are deferred until CSV/Excel or LLM functionality is actually invoked.
  • Tag snapshot output has a source database signature manifest at dataset/db/tag_center/snapshot_manifest.json.
  • Unchanged startup data reuses existing tag/domain assets instead of rebuilding roughly 100 MB of relation files.
  • The periodic snapshot scheduler uses the independent source signature, fixing the prior mtime logic that could miss a required rebuild after a mutation.
  • Embedding cache loading and status counting stream the 196.8 MB JSON cache. The status result is memoized from startup loading, so the normalization status API no longer scans the same file again.
  • Focused regression coverage was added in tests/test_startup_memory_paths.py.

Verified Phase 1 results on 2026-08-24:

  • App import reduced from about 146 MB to about 63 MB.
  • Isolated import + full job load + runtime index reduced from about 407 MB to about 182 MB.
  • Unchanged live startup completes in about 3.7 seconds.
  • Live process high-water mark is about 235 MB.
  • Live stable RSS after normal Admin requests is about 234 MB.
  • systemd cgroup peak is about 214 MB to 215 MB.
  • VmSwap is 0.
  • /api/admin/normalization/cache reduced from about 1.7 seconds to about 2 ms after startup memoization.
  • Test result: 9 passed.

Phase 1 deliberately did not remove the complete job library or vector cache from process memory. Phase 2 addressed the job library; the vector cache and heavy in-process workflows are recorded in the deferred Phase 3 proposal.

Phase 2: Remove Full Job Objects From Resident Memory

Status

COMPLETE and deployed on 2026-08-24.

Starting commit: ad89311 (Add Zhitu memory optimization handoff)

Implementation commit: bb972a2 (Reduce Zhitu admin resident job memory)

Completed implementation

  • job_sqlite.py is now the repository for canonical payload reads, bounded bulk reads, pagination/search, statistics, and transactional create/update/delete.
  • Schema version 2 adds job_search, job_search_fts, job_tags, and job_repository_meta. The live host supports JSON1 and FTS5. Search terms of at least three characters use the FTS5 trigram posting index; shorter terms use precomputed search columns so two-character Chinese substring behavior is preserved. Scalar filters and ordering use ordinary B-tree indexes.
  • Runtime state no longer contains state.jobs_metadata. It retains ordered job IDs, ID positions, internship positions, tag frequencies, metadata lists, and compact tag-to-position postings. This is the appropriate structure at the current 10,822-job/46,792-relation scale. Roaring bitmaps or succinct tries would add deployment and mutation complexity without a measured gain; reconsider them only if posting lists grow by an order of magnitude.
  • List and detail endpoints read only the requested payloads. Matching recalls positions/IDs first and bulk-loads one bounded candidate batch, with no N+1 payload queries. Job exports read 200 payloads at a time and stream output.
  • Canonical payload, derived search row, FTS row, and tag relations mutate in one SQLite transaction. Triggers mark out-of-band writes dirty, advance a logical source revision, and make repository rebuilds and tag snapshot invalidation deterministic.
  • Tag snapshot signatures now use repository schema version, logical source revision, and job count. This avoids false invalidation when SQLite checkpoints or truncates its WAL without changing canonical data.
  • career.json remains migration-only. No Redis, URL, response schema, match formula, or Qdrant collection change was introduced.

Migration and rollback

  • The pre-migration online SQLite backup is /home/ubuntu/workspace/projects/zhitu-job-match/dataset/career_planner/backups/job_library.sqlite3.phase2-pre-migration-20260824T065701+0800.sqlite3. It is 47,247,360 bytes, contains 10,822 jobs, has no Phase 2 tables, and passed PRAGMA integrity_check.
  • The migrated database is 204,058,624 bytes. It contains 10,822 canonical rows, 10,822 search rows, 10,822 FTS rows, and 46,792 tag relations, and passed PRAGMA integrity_check.
  • The first live migration plus the required tag snapshot rebuild took 20.512 seconds to health and temporarily reached VmHWM: 474752 kB and MemoryPeak=682577920. This is a one-time/offline-style cost, not the unchanged-startup path.
  • To roll back completely, first stop zhitu-admin.service, revert commit bb972a2, restore the backup above over dataset/career_planner/job_library.sqlite3, remove only that database's -wal and -shm sidecars, then start the service. If any job mutations have occurred after this handoff, export or back them up before restoring because the Phase 2 backup predates them.

Verified Phase 2 results

  • The aged Phase 1 process immediately before deployment reported VmRSS: 168516 kB, VmHWM: 240780 kB, VmSwap: 72380 kB, MemoryCurrent=151228416, and MemoryPeak=224632832. The swap was the long-lived Phase 1 condition described above, so the documented unswapped Phase 1 stable RSS of about 234 MB remains the cleaner steady-state reference.
  • Isolated Admin import plus compact runtime index: 71432 kB maximum RSS, down from about 182 MB for import plus the full job runtime in Phase 1.
  • Unchanged systemd restart to successful health: 3.537 seconds. The app's initialization log interval was 2.317 seconds, below the 5-second target.
  • After list, summary, and representative match requests: VmRSS: 131320 kB, VmHWM: 131320 kB, RssAnon: 106708 kB, RssFile: 24612 kB, and VmSwap: 0 kB.
  • The same warm run reported MemoryCurrent=109858816, MemoryPeak=111419392, and NRestarts=0. Phase 1's documented stable RSS and process high-water marks were about 234 MB and 235 MB respectively.
  • /api/jobs?limit=20 measured 9.3-11.1 ms, versus about 20-25 ms in Phase 1. /api/admin/summary measured 14.8 ms.
  • The fixed Python/backend match fixture measured 0.818 seconds. Its lane IDs, totals, top job IDs, scores, and tiers were identical to the captured Phase 1 response; the captured Phase 1 call took 2.022 seconds.
  • Unfiltered jobs, filtered jobs, job detail, Admin summary, and Admin tags were byte/JSON equivalent to captured fixtures. Admin frequencies were equivalent after excluding the intentionally refreshed updatedAt fields.
  • Repository tests cover idempotent migration, FTS/scalar search, all sort orders, pagination, ordered bulk reads, logical snapshot signatures, transactional create/update/delete across repository reopen, and streaming export. Final result: 14 passed.
  • Local and public Admin/Student health, Qdrant health, vector-store status, and node hooker.mjs test projects zhitu-job-match all passed.

Purpose

Stop keeping all 10,822 fully nested, normalized job payloads in state.jobs_metadata. Keep only compact fields and indexes needed for fast navigation and candidate recall. Read full job payloads from SQLite only for the small set of job IDs needed by a request.

Why this was next

After Phase 1, full normalized job objects are the largest remaining resident allocation. In the isolated benchmark they account for roughly 120 MB beyond the imported application. The current SQLite table already owns canonical job payloads, but the service treats it as a startup serialization source rather than an online data store.

Completed investigation checklist

  1. Find every read and mutation of state.jobs_metadata, state.inverted_index, state.global_tag_freq, and state.metadata_cache.
  2. Classify each consumer as list/search, detail lookup, matching candidate recall, export, statistics, or mutation.
  3. Capture current response fixtures for these endpoints before refactoring: /api/jobs?limit=20, one filtered /api/jobs request, one job detail, /api/admin/summary, /api/admin/tags, /api/admin/frequencies, and the smallest successful match request available in existing logs/tests.
  4. Confirm SQLite JSON1 and FTS5 support with read-only probes. Do not assume either extension without checking.

Implemented direction

  1. Add a repository layer around job_library.sqlite3. It must provide paginated list/search, full payload by ID, bulk payload by a bounded list of IDs, count/stat queries, and transactional create/update/delete.
  2. Introduce derived search storage. Prefer an FTS5 table plus ordinary indexed scalar columns when supported. Otherwise use a normal derived table with precomputed searchable text and explicit indexes. Do not repeatedly call SQLite json_extract over every row for hot queries.
  3. Store tag-to-job relations in SQLite or a compact ID-based in-memory index. Do not retain duplicate nested job dictionaries merely to support tag recall.
  4. Replace state.jobs_metadata with a compact representation containing only fields proven necessary for hot recall, such as job ID, direction, industry, sort sequence, and compact tag IDs. Prefer integer/string IDs over copied nested dictionaries.
  5. Change list and detail routes to query the repository. Preserve all current JSON keys, pagination semantics, filters, ordering, and error behavior.
  6. Change matching so it recalls candidate IDs first, then bulk-loads only the selected full payloads. Never issue an N+1 SQLite query per candidate.
  7. Update mutation paths atomically: canonical payload, derived search row, tag relations, and snapshot invalidation must commit together or roll back.
  8. Keep career.json migration-only. Do not reintroduce it as a runtime source.
  9. Add a schema/version migration that is idempotent and safe on the existing 10,822-row database. Take a recoverable SQLite backup before the first live migration and state its exact path in the completion report.

Changed files

  • job-admin/backend/job_sqlite.py or a new focused repository module.
  • job-admin/backend/backend_app/runtime_state.py
  • job-admin/backend/backend_app/job_data_service.py
  • job-admin/backend/backend_app/search_export_service.py
  • job-admin/backend/backend_app/matching_service.py
  • tests/test_job_repository.py
  • tests/test_startup_memory_paths.py

Constraints

  • No Redis dependency.
  • Do not alter public URLs, frontend contracts, job payload schema, match score formulas, or Qdrant collections.
  • Do not load all full payloads merely to generate an Admin page.
  • Export operations may stream many rows, but must not materialize both raw and normalized copies of the entire library.
  • Do not optimize by reducing the number of jobs or fields returned by existing APIs.

Phase 2 acceptance criteria (verified)

  • All Phase 1 tests and new repository/search contract tests pass.
  • Captured endpoint fixtures remain semantically equivalent.
  • Create/update/delete survive restart and update derived indexes correctly.
  • Filtered and unfiltered job ordering matches the pre-change behavior.
  • A restart with unchanged data remains under 5 seconds.
  • Idle/warm Admin RSS target: at most 160 MB with VmSwap: 0.
  • Admin process high-water target: at most 220 MB on unchanged startup.
  • /api/jobs?limit=20 target: no slower than the Phase 1 result by more than 20 percent; Phase 1 local result was about 20-25 ms.
  • Record before/after VmRSS, VmHWM, MemoryCurrent, MemoryPeak, startup duration, and endpoint timing in the commit/report.

Phase 2 verification commands

cd /home/ubuntu/workspace/projects/zhitu-job-match
.venv/bin/pytest -q
.venv/bin/python -m compileall -q job-admin/backend career-planner/backend shared tests
git diff --check
sudo systemctl restart zhitu-admin.service
curl --noproxy '*' -fsS http://127.0.0.1:3061/api/health
curl --noproxy '*' -fsS 'http://127.0.0.1:3061/api/jobs?limit=20' >/dev/null
curl --noproxy '*' -fsS http://127.0.0.1:3061/api/admin/summary >/dev/null
pid=$(systemctl show zhitu-admin.service -p MainPID --value)
cat /proc/$pid/status | rg '^(VmHWM|VmRSS|RssAnon|RssFile|VmSwap):'
systemctl show zhitu-admin.service -p MemoryCurrent -p MemoryPeak -p NRestarts

Commit Phase 2 independently before starting Phase 3. Put the Phase 2 commit hash and measured results into this document.

Phase 3 Proposal: Remove Online Vectors and Heavy Jobs From the Web Process

Status

DEFERRED on 2026-08-24. Do not start Phase 3 only because it appears in this handoff. Phase 2 is the stopping point for the current optimization cycle.

The proposal remains below as a design reference. Reassess it from implementation commit bb972a2 and fresh production measurements only when one of the restart conditions in this section is met.

Deferral decision

Phase 2 already resolved the operational memory problem:

  • Warm Admin VmRSS is about 131.9 MB, with VmHWM at the same level and VmSwap: 0.
  • The cgroup reports about 105.4 MiB current memory and 106.8 MiB peak memory.
  • Unchanged startup takes about 3.5 seconds and remains below the existing startup and resident-memory targets.
  • The host had about 1.9 GiB available memory at the decision point.

Removing the remaining online vectors is not a transparent cleanup. The local vectors still participate in detailed match scoring and provide recall fallback when Qdrant is unavailable. Removing them would change outage behavior, add Qdrant request work, and require a fixed-query parity suite plus an explicit decision that a structured 503 is preferable to local degradation.

Heavy-worker isolation is also deferred. No normalization, portrait-build, or tag-review run artifacts were present, so there is no evidence that these batch workflows currently create recurring production pressure. The roughly 475 MB process high-water mark recorded during Phase 2 came from its one-time migration and snapshot rebuild, not normal traffic.

Restart conditions

Reopen Phase 3 when at least one condition is measured rather than anticipated:

  1. Warm Admin RSS repeatedly exceeds 160 MB or normal traffic produces swap.
  2. Host memory pressure becomes sustained, or multiple Admin Web workers are required and each process duplicates the vector cache.
  3. Normalization, portrait build, import, tag review, or snapshot rebuilds run frequently enough to leave the Web process materially above its baseline.
  4. The product accepts Qdrant as a hard matching dependency and accepts deterministic 503 behavior during vector-store outages.
  5. The tag/vector corpus grows enough that the current cache and posting lists are again a measured bottleneck.

If resumed, split the work into Phase 3A (online vector ownership and Qdrant parity) and Phase 3B (one-shot heavy worker isolation). They have different risks and do not need to ship together.

Runtime ownership after Phase 2

  • SQLite owns all complete job payloads plus scalar/FTS search data and tag relations. The Web process loads complete jobs only for bounded request candidates or explicitly heavy snapshot/normalization workflows.
  • runtime_state.py owns compact job IDs/positions, internship positions, tag-to-position postings, tag frequencies, metadata lists, and tag_vectors_cache.
  • The active embedding profile loads 4,867 vectors from the 196.8 MB JSON cache into EMBEDDING_CACHE_MEMO; 2,666 matcher tags reference vectors from that shared cache through state.tag_vectors_cache. An isolated load measured 39,870,464 bytes of vector payload and about 50.7 MB of RSS growth.
  • Tag snapshot rebuilds, normalization, import, portrait build, and tag review can still load the complete job library or heavy optional dependencies inside the Web PID. The one-time Phase 2 migration demonstrated a roughly 475 MB process high-water mark on such a full rebuild. Phase 3 worker isolation must include scheduled/on-demand tag snapshot rebuilds, not only user-launched normalization and imports.

Purpose if resumed

Make Qdrant the only online semantic vector store and isolate memory-heavy normalization, import, and bulk processing from the long-running Admin Web process. The Web service should return to a small stable baseline after each heavy operation.

Potential remaining benefit

The 196.8 MB JSON embedding cache becomes about 39.9 MB of float32 vectors in memory and remains memoized for the life of the process. Some vectors are then referenced by state.tag_vectors_cache. Qdrant already persists and searches the same online vector space, so this local online cache duplicates ownership.

Lazy imports from Phase 1 avoid loading Pandas/LangChain at startup, but the modules remain in the Web process once a user invokes CSV/Excel, portrait build, normalization, or LLM-heavy workflows. Process isolation is needed if that memory must be fully reclaimed.

Required investigation before resuming

  1. Re-run all Phase 2 measurements and inspect the new runtime state.
  2. Find every access to tag_vectors_cache, load_embedding_cache, NumPy vector matrices, and local cosine/dot-product matching.
  3. Map each vector operation to an existing Qdrant collection and payload. Verify dimensions, profile ID, model, score interpretation, and normalization.
  4. Compare local and Qdrant results on a fixed representative query set before changing production behavior. Save IDs, order, and score deltas.
  5. Inventory heavy Admin workflows and their progress/cancel/resume contracts. Decide which can safely run in a separate one-shot process without breaking frontend polling.

Vector implementation direction

  1. Add a Qdrant batch-query API in shared/qdrant_store.py; avoid one HTTP round trip per student tag or candidate.
  2. Replace local all-vector scans and matrices with bounded Qdrant queries. Preserve threshold meaning and deterministic tie-breaking.
  3. Remove active-profile startup calls that populate EMBEDDING_CACHE_MEMO/state.tag_vectors_cache. Legacy GLM behavior may stay behind its explicit profile if it is still supported and tested.
  4. Keep JSON/NPY files only as offline rebuild or migration inputs. Prefer a compact float32 NPY/memmap representation for offline tools if the JSON file is still needed operationally; do not mmap it into the Web process.
  5. Make Qdrant availability/degradation explicit. If matching cannot preserve correctness without vectors, return the existing structured 503 behavior rather than silently changing scoring.
  6. Update scripts/build_qdrant_index.py, vector docs, and tests together if the canonical rebuild format changes.

Worker isolation direction

  1. Extract normalization, large import, portrait build, and similar batch entry points into CLI-callable worker modules with explicit input/run IDs.
  2. Keep progress, logs, cancellation intent, and result files in the existing run directories so current polling APIs can remain stable.
  3. Launch each heavy run in a separately managed process. Use the repository's existing systemd conventions or a narrowly scoped subprocess supervisor; document ownership and failure recovery. Do not introduce an always-on heavy worker that merely moves the same idle memory elsewhere.
  4. Enforce one-run/concurrency limits and reject duplicate launches exactly as current locks do.
  5. On Web restart, reconcile active run metadata with actual worker state so a stale running status cannot persist indefinitely.
  6. Add cancellation, failed-worker, restart-recovery, and successful completion tests. Verify that worker exit returns its memory to the OS.

Expected files

  • shared/qdrant_store.py
  • job-admin/backend/backend_app/embedding_service.py
  • job-admin/backend/backend_app/matching_service.py
  • job-admin/backend/backend_app/runtime_state.py
  • job-admin/backend/tag_sync.py
  • job-admin/backend/portrait_builder/*
  • job-admin/backend/backend_app/normalization_*
  • New worker entry points and focused tests.
  • Platform systemd manifest/runbook files if a managed worker unit is added.

Constraints

  • Do not delete embedding_cache.json until a verified rebuild path and backup exist. It is ignored runtime data, not recoverable from Git.
  • Do not recreate or delete Qdrant collections during normal Web startup.
  • Do not expose Qdrant publicly.
  • Do not change match score thresholds based only on approximate observations; use recorded local-vs-Qdrant comparison fixtures.
  • Preserve existing progress, pause/cancel, resume, and error response contracts.
  • Update /home/ubuntu/workspace/platform/projects/zhitu-job-match.json and /home/ubuntu/workspace/platform/docs/projects/zhitu-job-match.md if service topology changes.

Phase 3 acceptance criteria if resumed

  • No active-profile full embedding cache is loaded in the Admin Web process.
  • Fixed-query Qdrant results meet documented order/score tolerances and match endpoints preserve their response contracts.
  • Heavy workflows execute outside the Web PID and remain observable/cancellable.
  • After a heavy worker exits, Admin Web RSS returns close to its pre-run value.
  • Idle/warm Admin RSS target after Phase 2: at most 110 MB, VmSwap: 0.
  • Unchanged startup process high-water target: at most 150 MB.
  • Qdrant outage behavior is deterministic and covered by tests.
  • Public Admin and Student health checks remain successful.
  • Platform registry/runbook and project documentation match the final topology.

Phase 3 verification commands if resumed

Run all Phase 2 checks plus:

curl --noproxy '*' -fsS http://127.0.0.1:6333/healthz
curl --noproxy '*' -fsS http://127.0.0.1:3061/api/admin/vector-store
curl --noproxy '*' -kfsS https://zhitu-admin.6767.chat/api/health
curl --noproxy '*' -kfsS https://zhitu.6767.chat/api/health
cd /home/ubuntu/workspace/platform
node hooker.mjs test projects zhitu-job-match
node hooker.mjs doctor projects --no-report
node hooker.mjs doctor host zhitu-qdrant --no-report

Also run a representative match parity suite, one successful heavy worker run, one cancellation, and one forced worker failure. Report Web and worker PIDs and their individual high-water marks.

Final Reporting Template For Each New Window

The completion response and commit notes must include:

  1. Starting commit and resulting commit.
  2. Purpose and exact scope implemented.
  3. Files and runtime ownership changed.
  4. Before/after startup time, RSS, high-water mark, cgroup peak, and swap.
  5. Endpoint and workflow timings.
  6. Tests and smoke checks run, including failures or skipped checks.
  7. Remaining risks and the next phase's prerequisites.
  8. Any backup paths, migrations, new services, or rollback commands.