코어: preflight 스냅샷과 UTF-8 휴리스틱 통합

AGENTS.md

@@ -4,7 +4,7 @@
 
 Token Pilot is evolving from a Spring AI usage-tracking starter into a framework-independent Java LLM control and accounting core with optional framework and observability adapters.
 
-Current truth: post-call usage normalization, cost calculation, ledger events, Micrometer publishing, Clock-based monthly budget windows, pure budget decisions, typed missing-pricing policies, pricing snapshots, legacy provider-boundary BLOCK enforcement, Spring AI integration, and starter autoconfiguration are implemented. Candidate-aware preflight admission, context admission, atomic reservation, and estimate/actual cost reconciliation are 30-day MVP targets, not current capabilities.
+Current truth: post-call usage normalization, cost calculation, ledger events, Micrometer publishing, Clock-based monthly budget windows, pure budget decisions, typed missing-pricing policies, pricing snapshots, framework-independent token count results, a UTF-8 byte heuristic estimator, and a preflight cost-bound projection are implemented. Candidate-aware request production, context admission, atomic reservation, and estimate/actual cost reconciliation are 30-day MVP targets, not current capabilities.
 
 Distribution direction: publish a framework-independent core and an optional Spring AI convenience starter from the same repository and release train. The existing starter artifact is `token-pilot-starter`; `token-pilot-spring-ai-starter` is only a target name until a compatibility ADR and module change land.
 
@@ -71,7 +71,7 @@ Token Pilot의 제품 포지션은 framework-independent Java LLM control and ac
 
 | Module | Status | Notes |
 | --- | --- | --- |
-| `token-pilot-core` | Basic implementation complete | Domain records, pricing, calculator, registry, ledger manager, pricing snapshots, and missing-pricing evaluator |
+| `token-pilot-core` | Basic implementation complete | Domain records, pricing, calculator, registry, ledger manager, pricing snapshots, token count results, UTF-8 byte heuristic estimation, and a preflight cost-bound projection; versioned model snapshots and context admission are still required before reservation |
 | `token-pilot-spring-ai` | Basic implementation complete | Spring AI 2.0.0 `UsageExtractor`, `LedgerAdvisor`, pricing snapshot resolution, response usage recording, reconciliation decisions, and legacy provider-boundary BLOCK enforcement |
 | `token-pilot-micrometer` | Basic implementation complete | `MetricsOptions`, tag whitelist, and metric metadata exist; metric ownership must be narrowed |
 | `token-pilot-budget` | Basic non-atomic implementation | Typed monthly keys, Clock/ZoneId windows, and pure status/admission decisions implemented; needs candidate estimation, reservation, idempotency, and reconciliation |
@@ -335,6 +335,10 @@ The active checklist is in `docs/30_DAY_MVP_REPORT.md`; detailed long-term works
 - Sample app E2E uses a fake Spring AI `ChatModel`; real provider API behavior is not yet verified.
 - `token-pilot-spring-ai-starter` does not exist in the current build; never use it as an install instruction until implemented and published.
 - Maven Central release consumption must be re-verified for both core and starter paths before announcing `0.1.0`.
+- Preflight cost bounds must use one immutable pricing snapshot from calculation through reservation and reconciliation; resolving a mutable registry again by model/policy identifiers can mix prices from different requests.
+- A preflight cost bound is not context admission evidence. The REQUEST token result and reserved output must be checked against the versioned model context window before a provider call or reservation is authorized.
+- `DefaultPreflightCostEstimator` currently relies on the existing exclusive `TokenType` pricing shape; richer pricing combinations and finite/unbounded policy capabilities must be owned by a validated pricing policy snapshot rather than caller-supplied flags.
+- The UTF-8 byte heuristic estimator returns `TEXT_ONLY` and uses `BYTE_LEVEL_BPE_UTF8` only as a byte-level safety-basis identifier; it is not an exact BPE implementation and must not be used as full-request admission evidence.
 
 ## Verification
 
@@ -396,6 +400,19 @@ Stage and deploy a Central release:
 
 ## Update History
 
+### 2026-08-10
+
+- Added the UTF-8 byte heuristic estimator and made preflight cost calculation consume the exact immutable pricing snapshot carried by its context instead of re-resolving a mutable registry.
+- Kept arithmetic failures inside the typed preflight unavailable-result contract for pricing inputs whose decimal scale cannot be represented by the calculation.
+
+### 2026-08-09
+
+- Clarified that the preflight cost-bound projection is not context admission or atomic reservation, and documented the immutable pricing-snapshot requirement for later integration.
+
+### 2026-08-08
+
+- Added framework-independent token count result contracts and conservative preflight cost bounds that preserve immutable pricing metadata, use exact decimal arithmetic, and return typed unavailable outcomes instead of zero-price fallbacks.
+
 ### 2026-08-04
 
 - Added typed missing-pricing policies, immutable pricing snapshots, core rate validation/reconciliation decisions, and Spring AI pre-call pricing resolution; `FAIL_CLOSED` rejects missing plans/rates before provider invocation and `FAIL_OPEN` preserves `UNPRICED`.

build.gradle

@@ -657,11 +657,9 @@ package compatibility;
 
 import io.tokenpilot.core.TokenEstimator;
 import io.tokenpilot.core.domain.Cost;
-import io.tokenpilot.core.domain.TokenCountAccuracy;
 import io.tokenpilot.core.domain.TokenCountResult;
 import io.tokenpilot.core.domain.TokenCountScope;
-import io.tokenpilot.core.domain.TokenEstimatorDescriptor;
-import io.tokenpilot.core.domain.TokenizationBasis;
+import io.tokenpilot.core.internal.LedgerComponents;
 import java.math.BigDecimal;
 import java.util.Currency;
 
@@ -675,17 +673,13 @@ public final class CoreConsumer {
             throw new IllegalStateException("Token Pilot core consumer verification failed");
         }
 
-        TokenCountResult expected = TokenCountResult.counted(
-                0L,
-                0L,
-                TokenCountAccuracy.EXACT,
-                TokenCountScope.TEXT_ONLY,
-                new TokenEstimatorDescriptor("consumer-estimator", "1"),
-                new TokenizationBasis("consumer-basis")
-        );
-        TokenEstimator estimator = text -> expected;
-        TokenCountResult actual = estimator.estimate("");
-        if (!actual.isCounted() || !actual.isExact() || actual.tokens().orElseThrow() != 0L) {
+        TokenEstimator estimator = LedgerComponents.utf8ByteHeuristicTokenEstimator();
+        TokenCountResult actual = estimator.estimate("hello");
+        if (!actual.isCounted()
+                || actual.isExact()
+                || actual.tokens().orElseThrow() != 2L
+                || actual.safeUpperBoundTokens().orElseThrow() != 5L
+                || actual.scope() != TokenCountScope.TEXT_ONLY) {
             throw new IllegalStateException("Token Pilot estimator consumer verification failed");
         }
         System.out.println("token-pilot-core Java 25 consumer OK");

token-pilot-core/src/main/java/io/tokenpilot/core/PreflightCostEstimator.java

@@ -0,0 +1,40 @@
+package io.tokenpilot.core;
+
+import io.tokenpilot.core.domain.PreflightCostResult;
+import io.tokenpilot.core.domain.PreflightPricingContext;
+import io.tokenpilot.core.domain.TokenCountResult;
+
+/**
+ * 호출 전에 계산한 REQUEST 범위의 토큰 상한을 금액 상한으로 변환하는 Core 계약입니다.
+ *
+ * <p>이 계약은 비용을 계산할 뿐 context window 적합성이나 provider 호출 허가를
+ * 판정하지 않습니다. 호출자는 요청 토큰 결과가 모델의 context admission을 통과했고,
+ * {@link PreflightPricingContext}가 계산에 사용할 하나의 불변 pricing snapshot을
+ * 보관하는지 먼저 보장해야 합니다. 계산기는 다른 snapshot을 registry에서 다시
+ * 조회하지 않습니다.</p>
+ *
+ * <p>계산 가능한 경우에도 예약 근거로 사용할 값은
+ * {@link PreflightCostResult.Bounded#safeUpperBoundCost()}뿐입니다.
+ * {@code estimatedCost}는 관찰과 표시를 위한 값입니다.</p>
+ */
+public interface PreflightCostEstimator {
+
+    /**
+     * atomic reservation에 사용할 호출 전 비용 상한을 계산합니다.
+     * 입력이 REQUEST 범위가 아니거나 tokenizer 기준이 맞지 않거나 가격 snapshot이
+     * 없거나 문맥에 snapshot이 없으면 숫자 비용 대신 제한된 unavailable 결과를 반환해야 합니다.
+     *
+     * @param pricingContext canonical model, pricing policy, catalog version과
+     *                       검증된 pricing 조건을 담은 계산 문맥
+     * @param requestInput 실제 전송 요청 전체를 계산한 REQUEST 범위의 token 결과
+     * @param reservedOutputTokens 호출 전에 확보할 최대 출력 token 수
+     * @return 계산 가능한 비용 상한 또는 그 사유를 담은 unavailable 결과
+     * @throws NullPointerException 필수 인자가 {@code null}인 경우
+     * @throws IllegalArgumentException reservedOutputTokens가 음수인 경우
+     */
+    PreflightCostResult estimate(
+            PreflightPricingContext pricingContext,
+            TokenCountResult requestInput,
+            long reservedOutputTokens
+    );
+}

token-pilot-core/src/main/java/io/tokenpilot/core/domain/PreflightCostResult.java

@@ -0,0 +1,152 @@
+package io.tokenpilot.core.domain;
+
+import java.util.Currency;
+import java.util.Objects;
+
+/**
+ * 호출 전 비용 계산 결과입니다. 숫자 상한이 있는 결과와 계산 불가 결과를
+ * sealed hierarchy로 분리해 0원과 pricing 부재를 혼동하지 않습니다.
+ */
+public sealed interface PreflightCostResult
+        permits PreflightCostResult.Bounded, PreflightCostResult.Unavailable {
+
+    /**
+     * 하나의 불변 pricing snapshot으로 계산한 유한 비용 상한입니다.
+     * 내부 계산에서는 반올림하지 않으며, context admission을 통과했다는 의미는 아닙니다.
+     * 예약 계층은 반드시 {@code safeUpperBoundCost}를 사용해야 합니다.
+     *
+     * @param estimatedCost 관찰과 표시를 위한 예상 비용
+     * @param safeUpperBoundCost 예약에 사용할 보수적인 비용 상한
+     * @param inputEstimatedTokens 정보 제공용 입력 token 계산값
+     * @param inputSafeUpperBoundTokens 보수적인 입력 token 안전 상한
+     * @param reservedOutputTokens 예약한 최대 출력 token 수
+     * @param canonicalModelId canonical model 식별자
+     * @param pricingPolicyId 불변 pricing policy 식별자
+     * @param catalogVersion 계산에 사용한 catalog 버전
+     * @param pricingSnapshot 계산·예약·정산에 전달할 정확한 가격 snapshot
+     * @param estimatorDescriptor token estimator 식별 정보
+     * @param tokenizationBasis 검증된 tokenizer 호환성 기준
+     */
+    record Bounded(
+            Cost estimatedCost,
+            Cost safeUpperBoundCost,
+            long inputEstimatedTokens,
+            long inputSafeUpperBoundTokens,
+            long reservedOutputTokens,
+            String canonicalModelId,
+            String pricingPolicyId,
+            String catalogVersion,
+            PricingSnapshot pricingSnapshot,
+            TokenEstimatorDescriptor estimatorDescriptor,
+            TokenizationBasis tokenizationBasis
+    ) implements PreflightCostResult {
+
+        /**
+         * 비용, token 관계와 재현성 metadata를 검증합니다.
+         */
+        public Bounded {
+            estimatedCost = Objects.requireNonNull(estimatedCost, "estimatedCost must not be null");
+            safeUpperBoundCost = Objects.requireNonNull(
+                    safeUpperBoundCost,
+                    "safeUpperBoundCost must not be null"
+            );
+            validateTokens(inputEstimatedTokens, inputSafeUpperBoundTokens, reservedOutputTokens);
+            canonicalModelId = requireText(canonicalModelId, "canonicalModelId");
+            pricingPolicyId = requireText(pricingPolicyId, "pricingPolicyId");
+            catalogVersion = requireText(catalogVersion, "catalogVersion");
+            pricingSnapshot = Objects.requireNonNull(pricingSnapshot, "pricingSnapshot must not be null");
+            if (!pricingSnapshot.modelId().equals(canonicalModelId)
+                    || !pricingSnapshot.pricingPolicyId().equals(pricingPolicyId)
+                    || !pricingSnapshot.catalogVersion().equals(catalogVersion)) {
+                throw new IllegalArgumentException("pricingSnapshot identity must match result metadata");
+            }
+            estimatorDescriptor = Objects.requireNonNull(
+                    estimatorDescriptor,
+                    "estimatorDescriptor must not be null"
+            );
+            tokenizationBasis = Objects.requireNonNull(tokenizationBasis, "tokenizationBasis must not be null");
+
+            if (!estimatedCost.currency().equals(safeUpperBoundCost.currency())) {
+                throw new IllegalArgumentException("estimated and safe upper bound currencies must match");
+            }
+            if (!pricingSnapshot.currency().equals(safeUpperBoundCost.currency())) {
+                throw new IllegalArgumentException("pricingSnapshot currency must match cost currency");
+            }
+            if (safeUpperBoundCost.compareTo(estimatedCost) < 0) {
+                throw new IllegalArgumentException("safeUpperBoundCost must be greater than or equal to estimatedCost");
+            }
+        }
+
+        /**
+         * 계산된 비용의 통화를 반환합니다.
+         *
+         * @return estimated cost와 safe upper bound cost가 공유하는 통화
+         */
+        public Currency currency() {
+            return safeUpperBoundCost.currency();
+        }
+    }
+
+    /**
+     * fail-closed 경계에서 숫자 비용으로 취급할 수 없는 결과입니다.
+     * 이 결과는 0원이나 임의의 기본 단가를 의미하지 않으며, provider 호출과 예약을
+     * 계속할지 여부는 각 control 계층이 사유를 확인해 결정해야 합니다.
+     *
+     * @param reason 숫자 비용을 생성하지 못한 제한된 사유
+     * @param canonicalModelId canonical model 식별자
+     * @param pricingPolicyId 참조한 pricing policy 식별자
+     * @param catalogVersion 참조한 catalog 버전
+     * @param currency 모델이 요구한 비용 통화
+     * @param reservedOutputTokens 요청한 최대 출력 token 수
+     * @param estimatorDescriptor token estimator 식별 정보
+     * @param tokenizationBasis token 결과의 tokenizer 호환성 기준
+     */
+    record Unavailable(
+            PreflightCostUnavailableReason reason,
+            String canonicalModelId,
+            String pricingPolicyId,
+            String catalogVersion,
+            Currency currency,
+            long reservedOutputTokens,
+            TokenEstimatorDescriptor estimatorDescriptor,
+            TokenizationBasis tokenizationBasis
+    ) implements PreflightCostResult {
+
+        /**
+         * Unavailable 사유와 진단 metadata를 검증합니다.
+         */
+        public Unavailable {
+            reason = Objects.requireNonNull(reason, "reason must not be null");
+            canonicalModelId = requireText(canonicalModelId, "canonicalModelId");
+            pricingPolicyId = requireText(pricingPolicyId, "pricingPolicyId");
+            catalogVersion = requireText(catalogVersion, "catalogVersion");
+            currency = Objects.requireNonNull(currency, "currency must not be null");
+            if (reservedOutputTokens < 0) {
+                throw new IllegalArgumentException("reservedOutputTokens must be non-negative");
+            }
+            estimatorDescriptor = Objects.requireNonNull(
+                    estimatorDescriptor,
+                    "estimatorDescriptor must not be null"
+            );
+            tokenizationBasis = Objects.requireNonNull(tokenizationBasis, "tokenizationBasis must not be null");
+        }
+    }
+
+    private static void validateTokens(long estimated, long safeUpperBound, long reservedOutput) {
+        if (estimated < 0 || safeUpperBound < 0 || reservedOutput < 0) {
+            throw new IllegalArgumentException("token values must be non-negative");
+        }
+        if (safeUpperBound < estimated) {
+            throw new IllegalArgumentException(
+                    "inputSafeUpperBoundTokens must be greater than or equal to inputEstimatedTokens"
+            );
+        }
+    }
+
+    private static String requireText(String value, String name) {
+        if (value == null || value.isBlank()) {
+            throw new IllegalArgumentException(name + " must not be blank");
+        }
+        return value;
+    }
+}

token-pilot-core/src/main/java/io/tokenpilot/core/domain/PreflightCostUnavailableReason.java

@@ -0,0 +1,25 @@
+package io.tokenpilot.core.domain;
+
+/**
+ * 호출 전 보수적 비용 상한을 숫자로 확정할 수 없는 제한된 사유입니다.
+ */
+public enum PreflightCostUnavailableReason {
+    /** token 계산 결과를 사용할 수 없는 상태입니다. */
+    COUNT_UNAVAILABLE,
+    /** 전체 REQUEST 범위를 포함하지 않은 token 결과입니다. */
+    INCOMPLETE_SCOPE,
+    /** estimator와 model의 tokenizer 기준이 호환되지 않습니다. */
+    INCOMPATIBLE_TOKENIZER,
+    /** 참조한 pricing policy snapshot을 찾지 못했습니다. */
+    PRICING_NOT_FOUND,
+    /** 비용 계산에 필요한 기본 입력/출력 단가가 누락됐습니다. */
+    INCOMPLETE_PRICING,
+    /** pricing policy가 호출 전에 유한한 상한을 제공할 수 없습니다. */
+    UNBOUNDED_PRICING,
+    /** model과 pricing snapshot의 통화가 다릅니다. */
+    CURRENCY_MISMATCH,
+    /** canonical model, policy 또는 catalog 식별자가 snapshot과 다릅니다. */
+    PRICING_SNAPSHOT_MISMATCH,
+    /** 가격 정밀도가 계산 범위를 벗어나 정형 비용 결과를 만들 수 없습니다. */
+    ARITHMETIC_FAILURE
+}