Skip to content

Reconcile Hugging Face router metadata - #1841

Open
DanielButler1 wants to merge 3 commits into
mainfrom
audit/huggingface-20260824
Open

Reconcile Hugging Face router metadata#1841
DanielButler1 wants to merge 3 commits into
mainfrom
audit/huggingface-20260824

Conversation

@DanielButler1

@DanielButler1 DanielButler1 commented Aug 24, 2026

Copy link
Copy Markdown
Member

Summary

  • reconcile the complete live Hugging Face OpenAI-compatible Router catalogue while keeping every Phaseo route disabled
  • distinguish Router-mediated external providers from Hugging Face's separate hf-inference serverless service and dedicated Inference Endpoints
  • add exact case-sensitive Router IDs, missing canonical model/publisher records, modality/context/feature metadata, and an itemized audit ledger
  • add verified token pricing only where exactly one live provider has complete published input/output rates; explicitly categorize multi-provider and incomplete-price routes
  • preserve imported fixed-rate history where automatic provider selection makes a single current price unsafe

Reconciliation

  • requested snapshot: 129 chat models
  • live Router at final verification: 130 chat models (one entry appeared during the audit window)
  • 130/130 live chat entries catalogued and itemized
  • 0 missing, 0 stale ledger entries, 0 unaccounted
  • 134 total Hugging Face records: 130 live Router chat entries plus four separately verified existing non-chat/non-v1 records
  • 24 single-provider chat routes have complete published pricing and are catalogued; 106 are categorized as provider-selection-dependent or price-incomplete
  • hf-inference remains separate; its 14 documented task families are recorded in the audit ledger and are not mislabeled as Router chat models

Reproducibility

  • node scripts/audit-huggingface-router.mjs
  • itemized snapshot: packages/data/catalog/src/data/api_providers/huggingface/router-audit-2026-08-24.json

Validation

  • node scripts/audit-huggingface-router.mjs — 130 live, 130 catalogued, 130 ledger, no missing/stale entries
  • pnpm validate:data
  • pnpm validate:pricing
  • pnpm validate:gateway
  • git diff --check

Created with Codex

@vercel

vercel Bot commented Aug 24, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
phaseo Ready Ready Preview Aug 27, 2026 8:20am

Request Review

@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 21 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 71868400-ed21-4843-95a1-4c35ac9c5265

📥 Commits

Reviewing files that changed from the base of the PR and between de5d08c and 7d97354.

📒 Files selected for processing (70)
  • .changeset/audit-huggingface-router.md
  • packages/data/catalog/src/data/api_providers/huggingface/models.json
  • packages/data/catalog/src/data/api_providers/huggingface/router-audit-2026-08-24.json
  • packages/data/catalog/src/data/models/aisingapore/gemma-sea-lion-v4-27b-it/model.json
  • packages/data/catalog/src/data/models/aisingapore/qwen-sea-lion-v4-32b-it/model.json
  • packages/data/catalog/src/data/models/baidu/ernie-4.5-vl-424b-a47b-base-pt/model.json
  • packages/data/catalog/src/data/models/cohere/aya-expanse-32b/model.json
  • packages/data/catalog/src/data/models/cohere/aya-vision-32b/model.json
  • packages/data/catalog/src/data/models/cohere/c4ai-command-a-03-2025/model.json
  • packages/data/catalog/src/data/models/cohere/c4ai-command-r-08-2024/model.json
  • packages/data/catalog/src/data/models/cohere/c4ai-command-r7b-12-2024/model.json
  • packages/data/catalog/src/data/models/cohere/c4ai-command-r7b-arabic-02-2025/model.json
  • packages/data/catalog/src/data/models/cohere/command-a-reasoning-08-2025/model.json
  • packages/data/catalog/src/data/models/cohere/command-a-translate-08-2025/model.json
  • packages/data/catalog/src/data/models/cohere/command-a-vision-07-2025/model.json
  • packages/data/catalog/src/data/models/deepseek/deepseek-r1-distill-llama-8b/model.json
  • packages/data/catalog/src/data/models/deepseek/deepseek-r1-distill-qwen-14b/model.json
  • packages/data/catalog/src/data/models/deepseek/deepseek-r1-distill-qwen-7b/model.json
  • packages/data/catalog/src/data/models/nvidia/nvidia-nemotron-3-ultra-550b-a55b-bf16/model.json
  • packages/data/catalog/src/data/models/nvidia/nvidia-nemotron-3-ultra-550b-a55b-nvfp4/model.json
  • packages/data/catalog/src/data/models/nvidia/nvidia-nemotron-3.5-lightning-30b-a3b-bf16/model.json
  • packages/data/catalog/src/data/models/pearl-ai/gemma-4-31b-it-pearl/model.json
  • packages/data/catalog/src/data/models/prism-ml/ternary-bonsai-27b-awq-4bit/model.json
  • packages/data/catalog/src/data/models/prism-ml/ternary-bonsai-27b-gguf/model.json
  • packages/data/catalog/src/data/models/prism-ml/ternary-bonsai-27b/model.json
  • packages/data/catalog/src/data/models/qwen/qwen2.5-coder-3b-instruct/model.json
  • packages/data/catalog/src/data/models/qwen/qwen2.5-coder-7b-instruct/model.json
  • packages/data/catalog/src/data/models/sao10k/l3-8b-lunaris-v1/model.json
  • packages/data/catalog/src/data/models/sao10k/l3-8b-lunaris/model.json
  • packages/data/catalog/src/data/models/sao10k/l3-8b-stheno-v3.2/model.json
  • packages/data/catalog/src/data/models/sao10k/l31-70b-euryale-v2.2/model.json
  • packages/data/catalog/src/data/models/swiss-ai/apertus-70b-instruct-2509/model.json
  • packages/data/catalog/src/data/models/swiss-ai/apertus-8b-instruct-2509/model.json
  • packages/data/catalog/src/data/models/swiss-ai/apertus-v1.5-70b/model.json
  • packages/data/catalog/src/data/models/z-ai/glm-4-32b-0414/model.json
  • packages/data/catalog/src/data/models/z-ai/glm-4.5v-fp8/model.json
  • packages/data/catalog/src/data/models/z-ai/glm-4.6-fp8/model.json
  • packages/data/catalog/src/data/models/z-ai/glm-4.6v-fp8/model.json
  • packages/data/catalog/src/data/organisations/pearl-ai/organisation.json
  • packages/data/catalog/src/data/organisations/prism-ml/organisation.json
  • packages/data/catalog/src/data/organisations/sao10k/organisation.json
  • packages/data/catalog/src/data/organisations/swiss-ai/organisation.json
  • packages/data/catalog/src/data/pricing/huggingface/baidu-ernie-4.5-vl-424b-a47b-base-pt/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/deepseek-ai-deepseek-r1-distill-qwen-7b/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/deepseek-ai-deepseek-r1/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/deepseek-ai-deepseek-v3.2-exp/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/google-gemma-3n-e4b-it/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/inclusionai-ling-2.6-1t/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/inclusionai-ling-3.0-flash/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/meta-llama-llama-4-maverick-17b-128e-instruct-fp8/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/minimaxai-minimax-m1-80k/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/moonshotai-kimi-k2-instruct-0905/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/moonshotai-kimi-k2-instruct/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/moonshotai-kimi-k3/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/nvidia-nvidia-nemotron-3-ultra-550b-a55b-bf16/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/openai-gpt-oss-120b/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/openai-gpt-oss-20b/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/pearl-ai-gemma-4-31b-it-pearl/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/prism-ml-ternary-bonsai-27b-awq-4bit/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/prism-ml-ternary-bonsai-27b-gguf/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/qwen-qwen2.5-coder-3b-instruct/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/qwen-qwen3-30b-a3b/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/speakleash-bielik-11b-v3.0-instruct/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/swiss-ai-apertus-70b-instruct-2509/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/swiss-ai-apertus-8b-instruct-2509/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/swiss-ai-apertus-v1.5-70b/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/tencent-hy3/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/xiaomimimo-mimo-v2.5-pro/text.generate/pricing.json
  • packages/data/catalog/src/data/pricing/huggingface/zai-org-autoglm-phone-9b-multilingual/text.generate/pricing.json
  • scripts/audit-huggingface-router.mjs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f89ab50ee

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +3 to +4
"api_model_id": "inclusionAI/Ling-2.6-1T",
"provider_api_model_id": "huggingface:inclusionAI/Ling-2.6-1T",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep canonical API model IDs stable

When this catalogue is imported, provider_api_model_id is the route’s persistent key and api_model_id participates in pricing and public-model lookups, while the case-sensitive upstream identifier already has a dedicated home in provider_model_slug. Replacing the lower-case IDs here creates new route identities for 30 Hugging Face records, removes the old identities during synchronization, and violates the catalogue’s lower-case slug contract; keep the stable api_model_id/provider_api_model_id values and apply Hub casing only to provider_model_slug.

AGENTS.md reference: packages/data/catalog/AGENTS.md:L9-L9

Useful? React with 👍 / 👎.

"prompt_training_source_url": null,
"zero_data_retention": "unknown",
"residency_source_url": "https://huggingface.co/docs/inference-providers/security",
"prompt_training_policy": "no_training",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use a supported prompt-training policy value

no_training is not a valid catalogue policy value: the shared type and gateway/UI normalizers recognize no_train, may_train, opt_out_available, enterprise_no_train, or unknown. Consequently this newly verified policy is persisted as an unsupported raw value and normalized back to unknown by consumers, so Hugging Face will not appear under the intended no-training policy; use the schema-supported value appropriate to the external-provider boundary.

AGENTS.md reference: packages/data/catalog/AGENTS.md:L8-L10

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant