Reuse resolved Hub commits throughout loading

[Wauplin]

Related but not dependent on vllm-project/vllm#49990.

Summary

Propagate a Hub commit as soon as a top-level loader resolves one, then use that immutable revision for the remainder of the same load.

• AutoTokenizer seeds _commit_hash from its resolved/passed model config before loading tokenizer metadata.
• Auto and concrete processor, image, video, and feature-extractor paths carry the resolved commit through sequential file lookups and subcomponents.
• Tokenizer and processor repository listings use the resolved commit.
• cached_files uses _commit_hash for cache misses as well as cache hits, so a missing local file cannot fall back to resolving the original mutable revision again.
• Specialized Bark and Ernie processor paths follow the same invariant.

Motivation

Transformers already records positive and negative Hub cache entries under an immutable commit. However, several loading paths either acquire _commit_hash too late or fail to pass it to later lookups. This causes repeated HEAD requests against main, including for optional files whose absence is already cached for the resolved commit.

This change keeps freshness scoped to a top-level call:

AutoTokenizer.from_pretrained("org/model", revision="main")
AutoTokenizer.from_pretrained("org/model", revision="main")

This still resolves the model config/ref twice. Within each invocation, subsequent same-repository lookups use that invocation's commit. There is no TTL or process-wide mutable-ref cache.

AI assistance

This draft was prepared with AI assistance. The submitting human should review and understand every changed line before marking it ready for review.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, bark, ernie4_5_vl_moe

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 30272340679:1
Result: failure | Jobs: 16 | Tests: 163,419 | Failures: 290 | Duration: 15h 17m

[Wauplin]

closing in favor of #47611