Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 39 additions & 22 deletions .dev-loop/INGEST_REPORT.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,52 @@
# Knowledge flush — 1 insight
# Knowledge flush — 4 insight(s)

Source: RNR-3440 (사내 잠재매물 주간 추출 스크립트 메모리 피크 저감). Candidate:
"QueryPie 프록시 경유로 대용량 결과를 스트리밍할 때 server-side named cursor 대신
일반 커서 + `fetchmany` + openpyxl `write_only`."
Cross-Check: 인사이트 4건 각각을 독립 general-purpose 서브에이전트 4개가 1차 소스(공식 문서·소스코드·이슈트래커) 대조로 교차검증 — 2건의 실질 교정 반영(cgroupns는 mount-time 시맨틱으로 메커니즘 수정, kps resourcePath v1alpha1은 현재도 기본값), 캐비앗 2건 추가(vLLM 필드 개명, DeepSeek 별도 CoT 예산).

## Decision Log

- **의도**: `~/.dev-loop/queue`에 쌓인 ★ Insight 후보 4건을 knowledge-flush 파이프라인(리서치 검증 → 기존 레이어 중복확인 → 라우팅 → wiki-ingest)으로 정제해 위키에 반영. 사용자가 명시 지시한 단일 배치 PR.
- **배제한 대안**: ① 기존 페이지에 병합 — 4건 모두 기존 페이지의 트리거와 불일치(아래 Existing-layer check)라 신규 페이지로. ② insight 2·3을 `reliability` 카테고리에 편입 — reliability는 전송 계층 실패(타임아웃/재시도) 소유라 시맨틱 응답 검증과 안 맞아 신규 카테고리 `backend/common/llm`으로(라우팅 근거는 Routing decision). ③ 수확된 원문 그대로 반영 — 리서치에서 메커니즘 교정 2건이 나와 교정본으로 반영.
- **리뷰어가 볼 곳**: 신규 4페이지의 `confidence`/`sources`가 실제 근거와 맞는지, `backend/common/llm` 신규 카테고리 판단, 그리고 기존 5개 페이지의 frontmatter `related:` 한 줄 수정이 과하지 않은지.
- [추정] 이 레포의 PR 리뷰는 dev-loop 규약상 레포 오너(choiyounggi) 단독 리뷰로 충분하다고 판단 — 팀 표준 리뷰어 3인은 개인 레포 collaborator가 아닐 수 있어 gh가 거부하면 미지정으로 진행.

## Verified best-practice

**Claim 1 — psycopg2 server-side (named) cursor requires a transaction; fails under autocommit.**
- Source: psycopg2 usage docs — `https://github.com/psycopg/psycopg2/blob/master/doc/src/usage.rst` (via Context7). Quote: "Named cursors are typically created 'WITHOUT HOLD', meaning they exist only within the current transaction. Attempting to fetch from them after a commit or in autocommit mode raises an exception."
- Matches my live repro (`can't use a named cursor outside of transactions`). → **verified**
**1. Container metrics empty while kubelet targets report up (OrbStack embedded k8s)**
Claim: when cAdvisor emits no `container_*` series, scrape kubelet `/metrics/resource` (kps values `kubelet.serviceMonitor.resource: true` + `resourcePath: "/metrics/resource"`), and expect bundled mixin dashboards to stay empty because of their `image!=""` filter.
Sources checked: Kubernetes resource-metrics-pipeline docs (fetched); kubelet source `pkg/kubelet/metrics/collectors/resource_metrics.go` (fetched — confirms the exact two series); kube-prometheus-stack `values.yaml` on main (fetched — `resource: false`, `resourcePath: "/metrics/resource/v1alpha1"` is STILL the default today, with the in-file "renamed in k8s 1.18" comment, so the override is required, not historical); kubernetes-mixin `dashboards/resources/queries/pod.libsonnet` + `rules/apps.libsonnet` (fetched — `image!=""` in queries and recording rules); orbstack/orbstack#2217 (corroborates the symptom class on OrbStack).
Verification outcome: 3 of 4 sub-claims confirmed against primary sources; the machine_*-only cAdvisor observation itself is a field measurement (2026-07-15, `kubectl get --raw .../metrics/cadvisor` → 0 `container_` series). **Confidence: verified** (field-only detail labeled as such in the page body). One correction folded in: the chart's v1alpha1 default is current, not merely historical.

**Claim 2 — a client-side (default) cursor pulls the whole result set to the client on execute; `fetchmany` only caps the Python-list explosion.**
- Source: psycopg2 cursor/usage docs + FAQ (named-cursor advantage = "data is fetched in chunks … minimal client memory"). By contrast the default cursor buffers the full result in libpq. → **verified**
**2. OpenAI-compatible completion validation (reasoning models, finish_reason=length, empty content)**
Claim: gate on `finish_reason == "length"` and blank `content` before using LLM output; use the reasoning field to disambiguate "reasoning consumed the budget".
Sources checked: OpenAI openapi.yaml (fetched — `length` = truncated by requested token limit, verbatim); OpenAI reasoning guide (fetched — reasoning can consume the whole output budget before any visible output); LiteLLM reasoning_content docs (fetched — normalizes to `message.reasoning_content`); vLLM reasoning outputs docs (fetched — field RENAMED `reasoning_content` → `reasoning`).
Verification outcome: core semantics verified; two provider-dependent caveats surfaced by research and added to the page: (a) vLLM's field rename — the page tells readers to check both spellings; (b) DeepSeek first-party API gives CoT a separate budget, so the empty-content failure mode is OpenAI-o-series/vLLM-style budgeting, not universal — added as an edge case. The exact observed combination (HTTP 200 + `length` + 0-char content + 8,173-char reasoning_content) is the session's own LiteLLM measurement (2026-07-28). **Confidence: field-tested.**

**Claim 3 — openpyxl `write_only` gives near-constant memory (<10 MB); one save only; lxml is for serialization speed, not the memory saving.**
- Source: openpyxl Optimised Modes — `https://openpyxl.readthedocs.io/en/stable/optimized.html` (via WebSearch). "keeping memory usage under 10Mb"; "A write-only workbook can only be saved once"; "make sure you have lxml installed" for large dumps (speed).
- This **corrects** the raw candidate's "lxml unnecessary" → precise form: unnecessary *for the memory win*, recommended *for large-dump speed*. Confirmed by my server test (write-only worked with lxml absent). → **verified**
**3. Code defaults naming gateway-resolved model aliases (re-verify at review/merge)**
Claim: defaults resolved by an external serving catalog can die between PR verification and merge while code/tests/builds stay green; re-query the live catalog (`GET /v1/models`) at review/merge time and fail-fast-validate load-bearing defaults at startup.
Sources checked: OpenAI models-list API reference (fetched — lists currently available model IDs); LiteLLM model_discovery docs (fetched — `/v1/models` reflects configured `model_list`; live upstream checking is opt-in, which strengthens the page's "confirm with one real call" row); vLLM quickstart (fetched — `/v1/models` on the OpenAI-compatible server); GitHub merge-queue docs (fetched — the staleness rationale: validation done earlier must be redone against current state); fail-fast startup-validation references (confirmed via search).
Verification outcome: the endpoint claim fully verified; the practice itself is a field lesson (PR review 2026-07-28: default summarization alias removed from the LiteLLM catalog between PR verification and review → `/chat/completions` 400 while the PR's e2e evidence was genuinely true at write time) supported by general drift/staleness references, not a documented named practice. **Confidence: field-tested.**

**Claim 4 — QueryPie blocks `BEGIN`, so server-side cursor is impossible there.**
- Environment-specific, no external source. Live repro in gui context: `autocommit=False` + named cursor → `[ENGINE] No permission to execute BEGIN statement`. → **field-tested**. Generalized in the page to "a read-only access-control proxy that blocks transaction control", with QueryPie as the concrete example (not a product-specific page).
- Memory figure 838 MB → 38 MB (300k synthetic rows) is my RNR-3440 measurement (`ru_maxrss`, separate processes).
**4. Reading other pods' cgroup v2 stats from inside a container**
Claim: mounting host `/sys/fs/cgroup` is insufficient — you need the host cgroup namespace (`docker --cgroupns=host`; k8s privileged or hostPID + `nsenter -t 1 -C`).
Sources checked: cgroup_namespaces(7) man page (fetched); docker run reference (fetched — `--cgroupns=host`; private is the cgroup v2 default); nsenter(1) (fetched — `-C/--cgroup`); kubernetes/kubernetes#103363 (fetched — no first-class pod hostCgroup option; nsenter workaround as used by Cilium).
Verification outcome: directive verified against docs + the session's own A/B reproduction (2026-07-22 OrbStack: identical `-v` mount, `kubepods` absent without / fully present with `--cgroupns=host`). Research CORRECTED the harvested mechanism: the cgroupfs view is fixed at MOUNT time by the mounting process's cgroup namespace (the runtime creates the container's cgroup mount inside the private ns) — not dynamically filtered per reader; the page states the corrected mechanism. Research bonus added: `nsdelegate` write-restriction edge case (files visible, writes EPERM — a distinct failure mode). **Confidence: field-tested.**

## Existing-layer check

- Pages read: `databases/index.md`, `databases/query-optimization/keyset-pagination.md`, `backend/python/index.md`.
- Overlap: keyset-pagination is the nearest neighbor (both handle large result sets) but a **distinct** topic — pagination splits the read into many bounded queries; this page streams a *single* query's result in chunks. Not a duplicate → new page + **bidirectional `related` link** added to both.
- backend/python has no DB-cursor page; the psycopg2/openpyxl specifics live as concrete examples inside the databases page rather than a separate python page (no duplication).
- Conflicts: none found.
Read: root `INDEX.md`; domain indexes for `infrastructure` and `backend`; candidate-overlap pages in full — `infrastructure/observability/logs-metrics-signals.md`, `infrastructure/containers/resource-limits-and-probes.md`, `backend/common/reliability/timeouts-and-retries.md`, `infrastructure/config/environment-config.md`, `qa/process/release-gates.md` (+ skimmed `qa/process/post-release-verification.md`). Repo-wide grep for `reasoning_content|cadvisor|cgroup|/metrics/resource|v1/models|finish_reason` found no page owning any of these topics (hits were incidental substrings).

- **Overlaps found:** none that owns the new triggers. `logs-metrics-signals` owns instrumenting a service (emitting signals), not collecting cluster metrics; `resource-limits-and-probes` owns manifests/limits (it *consumes* `container_memory_working_set_bytes`, hence a related-link, not a merge); `timeouts-and-retries` owns transport-level outbound failures, not semantic response validation; `environment-config` owns config shape/startup validation (its "no dev-friendly defaults" rule is adjacent to insight 3, but the trigger — external catalog drift at review time — is new); `release-gates` owns the release checklist process.
- **Merged vs created:** 0 merged, 4 new pages created (all four triggers are new).
- **Conflicts flagged:** none — no existing directive contradicts the new pages.
- **Related links added (both ways):** `missing-container-metrics` ↔ `logs-metrics-signals`, ↔ `resource-limits-and-probes`; `host-cgroup-visibility` ↔ `resource-limits-and-probes`, ↔ `missing-container-metrics`; `completion-response-validation` ↔ `gateway-model-alias-defaults`, ↔ `timeouts-and-retries`; `gateway-model-alias-defaults` ↔ `environment-config`, ↔ `release-gates`.

## Routing decision

- Target: **`databases/query-optimization/streaming-large-result-sets.md`** (new page).
- Category `query-optimization` fits (memory-bounding how a query's result is pulled into the app is query-execution optimization); no new category needed.
- Registered in `databases/index.md` (query-optimization section) and appended to `log.md`.
| Insight | Target | Page |
|---------|--------|------|
| 1 (kubelet metrics) | `infrastructure/observability` (existing category) | `missing-container-metrics.md` — new page, new trigger |
| 4 (cgroup namespace) | `infrastructure/containers` (existing category) | `host-cgroup-visibility.md` — new page, new trigger |
| 2 (completion validation) | `backend/common/llm` (**new category**) | `completion-response-validation.md` |
| 3 (catalog-resolved defaults) | `backend/common/llm` (**new category**) | `gateway-model-alias-defaults.md` |

**New category justification (`backend/common/llm`):** both insights concern consuming OpenAI-compatible LLM gateways from server-side code. Existing backend/common categories don't cover the trigger: `api-design` owns designing *your own* API; `reliability` owns transport-level outbound failure handling (timeouts/retries/backoff), while these pages gate on *semantic* response validity (finish_reason/reasoning fields) and external-catalog identifier drift — HTTP-success-but-unusable-output is outside its tables. LLM-gateway consumption is a recurring concern with its own vocabulary, so a dedicated category beats stretching `reliability`. Insight 3 generalizes beyond LLMs (any externally-resolved alias); that generality is stated in the page's "When this applies" while the page stays anchored to its concrete, evidenced instance. Plumbing updated: `backend/index.md` (new `### llm` section + concern list), root `INDEX.md` backend route line, `infrastructure/index.md` (two new load-when rows), `log.md` (two ingest entries).
2 changes: 1 addition & 1 deletion INDEX.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ follow the cross-pointers in their index or take the next matching seeded domain
| Domain | Status | Route here when |
|--------|--------|-----------------|
| [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior |
| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, LLM/gateway completions) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) |
| [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility |
| [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting) |
| [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) |
Expand Down
2 changes: 2 additions & 0 deletions log.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,5 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
## [2026-07-12] revise | security/secrets-in-code +1 edge case: third-party HTTP client (httpx/requests) logs the full request URL — including a query-param API key — at INFO, so root/DEBUG logging leaks it; keep the client logger above INFO. Found when a standalone sync process set logging.basicConfig(INFO) and httpx wrote the data.go.kr serviceKey to the log file. last_verified bumped to 2026-07-12.
## [2026-07-13] ingest | databases +1 (query-optimization): streaming-large-result-sets — memory-bounded export of a huge single-query result. Client-side cursor pulls the whole set to libpq on execute (fetchmany caps only the Python-list explosion); only a server-side/named cursor truly streams but needs a transaction, so it fails under autocommit or a proxy that blocks BEGIN → fall back to client-side fetchmany + disk spool + openpyxl write_only (measured 300k rows 838MB→38MB). Derived from RNR-3440 (potential-listing weekly extract memory peak); QueryPie BEGIN-block generalized to "read-only access proxy", field-tested. Sources: psycopg2 usage/cursor docs (named-cursor WITHOUT HOLD + autocommit exception), openpyxl optimized-modes (write-only near-constant memory, lxml=speed-not-memory).
## [2026-07-23] ingest | databases +2: schema-design/online-schema-changes (ACCESS EXCLUSIVE lock avoidance — non-volatile default fast path, ADD CONSTRAINT NOT VALID + VALIDATE at SHARE UPDATE EXCLUSIVE, CHECK-NOT-NULL trick, CREATE INDEX CONCURRENTLY, expand-and-contract to decouple DB migration from app deploy, lock_timeout for lock-queue pile-up) + operations/autovacuum-and-wraparound (NEW category operations: per-table scale_factor/cost_limit tuning for hot tables, age(datfrozenxid)/relfrozenxid + n_dead_tup monitoring, wraparound read-only cliff and superuser VACUUM recovery, VACUUM FULL vs pg_repack). Derived from the Hatchet "Postgres survival guide"; both cross-checked against PostgreSQL official docs (sql-altertable, routine-vacuuming).
## [2026-07-28] ingest | backend +2 (new category common/llm): completion-response-validation (gate on finish_reason=length + blank content; reasoning_content/reasoning disambiguation, provider budget caveats — DeepSeek separate CoT budget, vLLM field rename) + gateway-model-alias-defaults (re-query the live catalog `GET /v1/models` at review/merge time; startup fail-fast for load-bearing defaults). Derived from a meeting-summary LiteLLM integration (measured 200/length/empty-content/8,173-char reasoning_content; default alias dead by review time). finish_reason semantics + /v1/models verified against OpenAI spec/LiteLLM/vLLM docs; both pages field-tested.
## [2026-07-28] ingest | infrastructure +2: observability/missing-container-metrics (verified — container_* absent while kubelet targets up; kubelet /metrics/resource scrape, kps resourcePath default STILL v1alpha1 per chart values.yaml, kubernetes-mixin image!="" filter excludes resource-endpoint series) + containers/host-cgroup-visibility (field-tested — mounted /sys/fs/cgroup shows namespaced view; docker --cgroupns=host / hostPID+nsenter -t 1 -C; mount-time cgroupns semantics per cgroup_namespaces(7), nsdelegate write caveat). Both from mac-server observability work (OrbStack embedded k8s measurements 07-15/07-22).
Loading
Loading