feat: add estimate-tokens command for offline LLM token & cost estimation

[malteos]

Summary

Adds commonlid estimate-tokens — offline token & cost estimation for LLM
evaluation runs, so we can budget a full-benchmark run (e.g. gpt-5 on the
full commonlid dataset) before spending anything. Makes no API calls.

The command reconstructs the exact DSPy prompt the evaluator would send per
sample (system instruction + per-sample text, via ChatAdapter.format on the
same signature run uses), counts tokens with the model's own tokenizer, and
prices the result via LiteLLM.

What it does

• Model-agnostic tokenization — LiteLLM's token_counter uses tiktoken
  for OpenAI/Azure models and a Hugging Face AutoTokenizer for
  huggingface/* models (install transformers for the latter).
• Accounts for the system prompt as a per-request overhead measured once.
• Reasoning tokens are an explicit --reasoning-tokens per-sample
  assumption (default 0, billed as completion tokens) — true usage can't be
  known before a run, so the output labels the assumption used.
• Pricing via LiteLLM's model map. Unknown models report token counts with
  cost n/a rather than a misleading $0.
• Graceful fallbacks — unresolved tokenizer → heuristic count + a
  pip install transformers note; never crashes an estimate.
• --limit N measures a subset and projects to the full dataset; --json
  for machine-readable output. Accepts the same dspy:<name> spec as run.

Example

commonlid estimate-tokens \
  --model dspy:openai/gpt-5 \
  --dataset commonlid \
  --reasoning-tokens 300

gpt-5 on the full commonlid dataset (373,230 samples)

Measured by this command (offline, full dataset):

	Value
Samples	373,230
Per-request overhead	179 tokens (system prompt + scaffolding)
Input tokens	111.25M (298.1 mean/sample)
Output tokens	115.70M (10 completion + 300 reasoning per sample)
Input cost	$139.06
Output cost	$1,157.01
Total cost	≈ $1,296

Implied gpt-5 pricing from LiteLLM: ~$1.25/M input, ~$10/M output.

Cost is dominated by the reasoning-token assumption — input tokens (111M,
~$139) are fixed, while the output side scales with --reasoning-tokens:

--reasoning-tokens	Output tokens	Total cost
0	~3.7M	≈ $176
300	~115.7M	≈ $1,296
1000	~377M	≈ $3,900

For a task as short as a one-word language ID, real reasoning usage is likely
toward the low end; dial it in with reasoning_effort on the actual run.

Files

• src/commonlid/evaluation/token_cost.py — new: estimate() + TokenCostEstimate.
• src/commonlid/cli.py — new @app.command("estimate-tokens").
• tests/unit/test_token_cost.py — new: 9 unit tests (arithmetic, reasoning
  roll-up, dspy: stripping, unknown-model & zero-price handling, missing
  tokenizer fallback, --limit extrapolation, and a real ChatAdapter shape
  guard).
• README.md, docs/contributing/adding_a_model.md — usage docs.

Test plan

• make lint — clean
• make typecheck — clean
• make test — 258 passed, coverage 94.7%
• Live smoke on commonlid_nano (offline): text output, --reasoning-tokens,
  --limit extrapolation, --json, and HF n/a-pricing paths.
• Full-dataset estimate on commonlid (table above).

Notes / out of scope

• Wiring reasoning_effort into DSPyLLMModel for the real run is a separate
  change; this estimator only assumes a reasoning-token budget.
• No new required dependencies: OpenAI/Azure (incl. gpt-5) tokenization uses
  tiktoken, already present via LiteLLM. HF-model tokenization is best-effort
  pending transformers.