Model: add Tencent Hy 4 (hy_v4) preview architecture support - #28127
Conversation
Adds support for the Tencent Hy 4 model (Hugging Face architecture HYV4ForCausalLM, GGUF arch hy_v4): Add HF -> GGUF conversion script (conversion/hy_v4.py) and wire it into the conversion registry Register hy_v4 GGUF constants, arch enum, and writer support Implement the hy-v4 model graph, hparams, vocab and context changes Register the new arch in llama-arch and models registry Extend arch tests to cover hy_v4 Assisted by Claude Opus 5
This comment was marked as off-topic.
This comment was marked as off-topic.
|
I did some tests of this PR with a 765GB Q8_0 quant (CPU expert offloading):
Some output from the tests: |
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
|
@fairydreaming The hash value may have some problems, I will check it. |
@Little0o0 I found the problem, it was simply the order of hashes in both files, it must be the same. Fixed it. |
|
Take a look at #27970 |
|
Please rebase and adapt to |
|
@Green-Sky Since this model does Perplexity after the changes: |
|
Just a maintainer's heads up; we have two models about to be merged fighting for the same |
|
@Little0o0 now that the model is merged I think it would be good idea to reconvert/re-quantize the GGUFs you had uploaded in https://huggingface.co/AngelSlim/Hy4-preview-GGUF so that they work with mainline llama.cpp and update your README. Good job with the PR! |
|
now STQ1_0 when? :D |
Fixed that for you. Edit: Seriously though, I want this too, but getting reviewers willing to own quants is hard... |
…rg#28127) * model: add Tencent Hy 4 (hy_v4) preview architecture support Adds support for the Tencent Hy 4 model (Hugging Face architecture HYV4ForCausalLM, GGUF arch hy_v4): Add HF -> GGUF conversion script (conversion/hy_v4.py) and wire it into the conversion registry Register hy_v4 GGUF constants, arch enum, and writer support Implement the hy-v4 model graph, hparams, vocab and context changes Register the new arch in llama-arch and models registry Extend arch tests to cover hy_v4 Assisted by Claude Opus 5 * Update convert_hf_to_gguf_update.py Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * Update conversion/base.py Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> * convert : move hy_v4 entry to the same place as in convert_hf_to_gguf_update.py * model : apply changes related to n_ff_exp becoming per-layer in Hy4-preview * n_layer_all --------- Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
|
I'm not sure to understand if STQ1_0 is ok ot not !?? |
|
@AlphaMo99 STQ1_0 is still waiting for merging, referring #22836 . This PR is for the Hy4 preview only. |
Thanks for your contribution and helps @fairydreaming ! I will update the GGUF and README soon. |
Overview
Adds support for the Tencent Hy 4 model (Hugging Face architecture HYV4ForCausalLM, GGUF arch hy_v4)
This PR adds architecture support for the Tencent Hy4 preview model (Hugging Face architecture
HYV4ForCausalLM, GGUF archhy_v4). The model is published at https://huggingface.co/tencent/Hy4-preview. Some GGUF files with patches were published on https://huggingface.co/AngelSlim/Hy4-preview-GGUF , receiving 80K+ downloads in 2 days.This PR is developed from official patch and referring branch by @fairydreaming .
Hy 4 is a DeepSeek-V3-style MLA + MoE model with several notable extensions that this implementation supports:
hc_multparallel residual streams mixed per-layer viahc_attn_fn/hc_ffn_fn(pre/post) and a globalhc_head_fnbefore the final norm. The HC pre/post reduce runs in fp32 to match the reference's numerical behaviour over the full stack of layers.self_attn.linear_gate) is applied to the decompressed attention output, beforeo_proj.learnable_sink_parampassed into the attention sink term.Changes by area:
src/models/hy-v4.cppllama_model_hy_v4with iHC, gated MLA, learnable sink, MoE, optional DSA indexersrc/llama-arch.cpp,src/llama-arch.h,src/llama-model.cpp,src/models/models.hLLM_ARCH_HY_V4src/llama-hparams.h,src/llama-context.cpp,src/llama-graph.cppsrc/llama-vocab.cpp,src/llama-vocab.hLLAMA_VOCAB_PRE_TYPE_HY_V4tokenizer pre-typesrc/llama-model-saver.cppgguf-py/gguf/constants.py,gguf-py/gguf/gguf_writer.pyMODEL_ARCH.HY_V4, tensor names, metadata keysconversion/hy_v4.py,conversion/__init__.py,conversion/base.pyHYV4Model(extendsDeepseekV2Model), registered forHYV4ForCausalLMconvert_hf_to_gguf_update.pyhy_v4entry (BPE tokenizer,tencent/Hy4-preview)tests/test-llama-archs.cppAdditional information
Testing
python convert_hf_to_gguf.py <Hy4 checkpoint> --outfile hy4.ggufconverts cleanly./tests/test-llama-archspasses for the newhy_v4entryNot supported / intentionally dropped
num_nextn_predict_layers(themodel.mtp_layers.*weights) are dropped.Requirements