Skip to content

Model: add Tencent Hy 4 (hy_v4) preview architecture support - #28127

Merged
fairydreaming merged 7 commits into
ggml-org:masterfrom
Little0o0:hy4-preview
Sep 4, 2026
Merged

Model: add Tencent Hy 4 (hy_v4) preview architecture support#28127
fairydreaming merged 7 commits into
ggml-org:masterfrom
Little0o0:hy4-preview

Conversation

@Little0o0

@Little0o0 Little0o0 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Overview

Adds support for the Tencent Hy 4 model (Hugging Face architecture HYV4ForCausalLM, GGUF arch hy_v4)

This PR adds architecture support for the Tencent Hy4 preview model (Hugging Face architecture HYV4ForCausalLM, GGUF arch hy_v4). The model is published at https://huggingface.co/tencent/Hy4-preview. Some GGUF files with patches were published on https://huggingface.co/AngelSlim/Hy4-preview-GGUF , receiving 80K+ downloads in 2 days.

This PR is developed from official patch and referring branch by @fairydreaming .

Hy 4 is a DeepSeek-V3-style MLA + MoE model with several notable extensions that this implementation supports:

  • Independent Hyper-Connections (iHC)hc_mult parallel residual streams mixed per-layer via hc_attn_fn / hc_ffn_fn (pre/post) and a global hc_head_fn before the final norm. The HC pre/post reduce runs in fp32 to match the reference's numerical behaviour over the full stack of layers.
  • Gated MLA — an element-wise sigmoid gate (self_attn.linear_gate) is applied to the decompressed attention output, before o_proj.
  • Learnable sink — per-head learnable_sink_param passed into the attention sink term.

Changes by area:

Area File(s) Notes
C++ model graph src/models/hy-v4.cpp New llama_model_hy_v4 with iHC, gated MLA, learnable sink, MoE, optional DSA indexer
Arch registration src/llama-arch.cpp, src/llama-arch.h, src/llama-model.cpp, src/models/models.h Wire up LLM_ARCH_HY_V4
Hyper-parameters src/llama-hparams.h, src/llama-context.cpp, src/llama-graph.cpp HC / indexer / MoE params and graph routing
Vocab src/llama-vocab.cpp, src/llama-vocab.h LLAMA_VOCAB_PRE_TYPE_HY_V4 tokenizer pre-type
GGUF writer src/llama-model-saver.cpp Emit hy_v4 tensors / metadata
GGUF constants gguf-py/gguf/constants.py, gguf-py/gguf/gguf_writer.py MODEL_ARCH.HY_V4, tensor names, metadata keys
HF to GGUF conversion conversion/hy_v4.py, conversion/__init__.py, conversion/base.py HYV4Model (extends DeepseekV2Model), registered for HYV4ForCausalLM
Model list convert_hf_to_gguf_update.py Add hy_v4 entry (BPE tokenizer, tencent/Hy4-preview)
Tests tests/test-llama-archs.cpp Extend the architecture unit-test suite with hy_v4

Additional information

Testing

  • python convert_hf_to_gguf.py <Hy4 checkpoint> --outfile hy4.gguf converts cleanly
  • ./tests/test-llama-archs passes for the new hy_v4 entry
  • Generated logits / sampled output match the reference implementation within tolerance

Not supported / intentionally dropped

  • MTP / speculative decodingnum_nextn_predict_layers (the model.mtp_layers.* weights) are dropped.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: AI usage disclosure: YES. This implementation is developed from the official patch and real human fork, highly assisted by Claude Opus 5, working under the direction of @Little0o0, who commissioned, reviewed, tested the resulting builds on real hardware. If this PR cannot be merged under policy, it can still be a reference for the Hy4_v4 implementation.

Adds support for the Tencent Hy 4 model (Hugging Face architecture
HYV4ForCausalLM, GGUF arch hy_v4):

Add HF -> GGUF conversion script (conversion/hy_v4.py) and wire it into the conversion registry
Register hy_v4 GGUF constants, arch enum, and writer support
Implement the hy-v4 model graph, hparams, vocab and context changes
Register the new arch in llama-arch and models registry
Extend arch tests to cover hy_v4

Assisted by Claude Opus 5
@github-actions github-actions Bot added model Model specific testing Everything test related conversion labels Aug 31, 2026
@ggml-gh-bot

This comment was marked as off-topic.

Comment thread convert_hf_to_gguf_update.py Outdated
@fairydreaming

fairydreaming commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

I did some tests of this PR with a 765GB Q8_0 quant (CPU expert offloading):

  1. Model conversion - works OK
  2. Inference with some simple prompt - works OK
  3. Calculated perplexity on wiki.test.raw with 8k ubatch: Final estimate: PPL = 2.5185 +/- 0.01150
  4. Ran few lineage-bench quizzes with 4-node, 16-node and 64-node lineage graphs: all 100% passed
  5. Ran basic hello world app creation/compilation/run test in PI coding agent - works OK
  6. Ran llama-batched-bench with 8k ubatch to check the performance (Epyc 9374F + RTX PRO 6000 Max-Q).

Some output from the tests:

$ ./bin/llama-completion -m ~/ggufs/Hy4-preview-Q8_0.gguf -f ~/projects/prompts/prompt-hy4.txt -fit off -fa 1 -c 8192 -cmoe --jinja -no-cnv
0.00.219.591 I llama_completion: llama backend init
0.00.219.600 I llama_completion: load the model and apply lora adapter, if any
0.00.483.862 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.522.296 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance
0.53.516.786 I cmn          init: llama threadpool init, n_threads = 32
0.53.775.319 I 
0.53.775.410 I system_info: n_threads = 32 (n_threads_batch = 32) / 64 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.53.775.411 I 
0.53.776.717 I sampler seed: 3981846486
0.53.776.726 I sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 64
	top_k = -1, top_p = 1.000, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.900
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
0.53.776.732 I sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> ?top-k -> ?typical -> ?top-p -> min-p -> ?xtc -> temp-ext -> dist 
0.53.776.733 I generate: n_ctx = 8192, n_batch = 2048, n_predict = -1, n_keep = 0
0.53.776.733 I 
systemYou are a helpful assistantreasoning_effort:highuserRepeat this text: "The different accidents of life are not so changeable as the feelings of human nature. I had worked hard for nearly two years, for the sole purpose of infusing life into an inanimate body. For this I had deprived myself of rest and health. I had desired it with an ardour that far exceeded moderation; but now that I had finished, the beauty of the dream vanished, and breathless horror and disgust filled my heart."assistant<think:opensource>We need answer user: "Repeat this text: ..." They provided a text. Need repeat it exactly. We should just output the text. Need ensure no extra? The instruction: Repeat this text: "..." They likely want the text repeated. We can quote or just repeat. We should not add analysis. Need be careful to reproduce exactly.

Text: "The different accidents of life are not so changeable as the feelings of human nature. I had worked hard for nearly two years, for the sole purpose of infusing life into an inanimate body. For this I had deprived myself of rest and health. I had desired it with an ardour that far exceeded moderation; but now that I had finished, the beauty of the dream vanished, and breathless horror and disgust filled my heart."

Need verify punctuation: The user included quotation marks around the whole text in their message? They wrote: Repeat this text: "The different accidents ... heart." So the text inside quotes is without surrounding quotes? They ask repeat this text: "..." Typically we repeat the text, maybe including or excluding quotes? Better to repeat the text as given, including the quotation marks? They said Repeat this text: "The different...". Usually the quoted material is the text to repeat. If we output exactly the content inside the quotes, without the outer quotes, that's safest? But "Repeat this text: "..." could mean repeat exactly including the quote marks? Let's examine. They wrote: Repeat this text: "The different accidents of life are not so changeable as the feelings of human nature. I had worked hard for nearly two years, for the sole purpose of infusing life into an inanimate body. For this I had deprived myself of rest and health. I had desired it with an ardour that far exceeded moderation; but now that I had finished, the beauty of the dream vanished, and breathless horror and disgust filled my heart."

The quotation marks enclose the text. The actual text to repeat is likely the sentence(s) inside the quotation marks. If I include the quotes in my response, I'd be repeating the text plus the quote characters, which are not part of the text? Ambiguous. Many similar tasks expect the assistant to output the text verbatim, often without the surrounding quotation marks if the user used them to denote the text. But to be safe, I could output the text exactly as it appears between the quotes, without adding quotes, because they said repeat this text: "..." The colon followed by quote. If I repeat the quoted string including quotes, it would be: "The different accidents..." with quotes. But that adds quote characters that weren't necessarily part of the text. Let's see typical behavior: If a user says Repeat this text: "Hello world", the assistant should reply: Hello world, or "Hello world"? Usually they'd reply: Hello world. But if they want exact repetition, maybe including quote? The prompt: 'Repeat this text: "The different accidents ... heart."' The text is inside quotes. The phrase "this text" refers to the quoted passage. I'd output the passage without surrounding quotation marks. However, it's from Frankenstein. Let's check exact wording: "The different accidents of life are not so changeable as the feelings of human nature. I had worked hard for nearly two years, for the sole purpose of infusing life into an inanimate body. For this I had deprived myself of rest and health. I had desired it with an ardour that far exceeded moderation; but now that I had finished, the beauty of the dream vanished, and breathless horror and disgust filled my heart." That's correct. Note "ardour" British spelling. "nearly two years". "inanimate body." "rest and health." semicolon after moderation; "but now that I had finished, the beauty of the dream vanished, and breathless horror and disgust filled my heart." End period inside quote? In user message, the closing quote is after the period: heart." So period inside quotes. The text itself ends with a period. If we output without surrounding quotes, end with period.

Should I add any lead-in? Maybe just the repeated text. No extra. Ensure no trailing spaces.

Let's consider if there is any policy issue: It's a literary excerpt, public domain. Repeating is fine. No need for commentary. Just output text.

I will output exactly the text (without surrounding quotation marks) to match "repeat this text". But maybe they expect the exact text including the surrounding quotes? Let's decide. The user wrote: Repeat this text: "The different accidents ... heart." If I respond:

The different accidents of life are not so changeable as the feelings of human nature. I had worked hard for nearly two years, for the sole purpose of infusing life into an inanimate body. For this I had deprived myself of rest and health. I had desired it with an ardour that far exceeded moderation; but now that I had finished, the beauty of the dream vanished, and breathless horror and disgust filled my heart.

That is the text. If they wanted quotes, they'd see no quotes and maybe think missing. But they used quotes to delimit. Usually in these tasks (like from a benchmark), they check if the model output matches the text inside quotes, not including quotes. So output without quotes.

Let's produce final.</think:opensource>The different accidents of life are not so changeable as the feelings of human nature. I had worked hard for nearly two years, for the sole purpose of infusing life into an inanimate body. For this I had deprived myself of rest and health. I had desired it with an ardour that far exceeded moderation; but now that I had finished, the beauty of the dream vanished, and breathless horror and disgust filled my heart. [end of text]


3.35.003.563 I common_perf_print:    sampling time =     244.58 ms
3.35.003.564 I common_perf_print:    samplers time =     154.49 ms /  1320 tokens
3.35.003.568 I common_perf_print:        load time =   53362.86 ms
3.35.003.569 I common_perf_print: prompt eval time =   21672.67 ms /   119 tokens (  182.12 ms per token,     5.49 tokens per second)
3.35.003.570 I common_perf_print:        eval time =  139278.99 ms /  1200 runs   (  116.07 ms per token,     8.62 tokens per second)
3.35.003.570 I common_perf_print:       total time =  161228.26 ms /  1319 tokens
3.35.003.571 I common_perf_print: unaccounted time =      32.01 ms /   0.0 %      (total - sampling - prompt eval - eval) / (total)
3.35.003.571 I common_perf_print:    graphs reused =       1194
$ ./bin/llama-perplexity -m ~/ggufs/Hy4-preview-Q8_0.gguf -f ../../perplexity/wikitext-2-raw/wiki.test.raw -c 8192 -b 8192 -ub 8192 -fit off -fa 1 -cmoe
0.00.478.615 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.517.697 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance
0.54.187.559 I cmn          init: llama threadpool init, n_threads = 32
0.54.515.723 I 
0.54.515.830 I system_info: n_threads = 32 (n_threads_batch = 32) / 64 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.54.515.846 I perplexity: tokenizing the input ..
0.55.041.904 I perplexity: tokenization took 526.049 ms
0.55.042.055 I perplexity: calculating perplexity over 36 chunks, n_ctx=8192, batch_size=8192, n_seq=1
1.38.306.197 I perplexity: 43.26 seconds per pass - ETA 25.95 minutes
[1]1.7425,[2]1.6476,[3]1.9372,[4]2.1449,[5]1.9992,[6]2.1228,[7]2.2332,[8]2.2559,[9]2.1857,[10]2.2031,[11]2.3601,[12]2.4474,[13]2.4591,[14]2.4343,[15]2.3398,[16]2.4262,[17]2.4096,[18]2.3522,[19]2.3528,[20]2.3724,[21]2.3703,[22]2.3898,[23]2.3971,[24]2.4610,[25]2.4788,[26]2.4545,[27]2.4704,[28]2.4914,[29]2.5111,[30]2.5212,[31]2.5268,[32]2.5488,[33]2.5593,[34]2.5820,[35]2.5523,[36]2.5185,
26.10.821.453 I Final estimate: PPL = 2.5185 +/- 0.01150
$ for length in 4 16 64; do ./lineage_bench.py -s -l $length -n 1 -r 43|./run_openrouter.py --url http://192.168.18.7:8080/v1/chat/completions -t 8 -m "tencent/Hy4-preview" -r --temp 0.9 --top-p 1.0 -o "results/Hy4-preview_$length"|tee "results/Hy4-preview_$length.csv"; done|./compute_metrics.py
100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [17:55<00:00, 268.89s/it]
Successfully generated 4 of 4 quiz solutions.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [13:54<00:00, 208.67s/it]
Successfully generated 4 of 4 quiz solutions.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [56:45<00:00, 851.48s/it]
|   Nr | model_name          |   lineage |   lineage-4 |   lineage-16 |   lineage-64 |
|-----:|:--------------------|----------:|------------:|-------------:|-------------:|
|    1 | tencent/Hy4-preview |     1.000 |       1.000 |        1.000 |        1.000 |
$ ./bin/llama-batched-bench -m ~/ggufs/Hy4-preview-Q8_0.gguf -b 8192 -ub 8192 -npl 1 -npp 8192,16384,32768,65536,131072,262144 -ntg 128 -c $((262144+8192)) -fa 1 --no-repack -cmoe
0.00.456.117 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.493.011 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance

llama_batched_bench: n_kv_max = 270336, n_batch = 8192, n_ubatch = 8192, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  8192 |    128 |    1 |   8320 |   40.932 |   200.14 |   15.088 |     8.48 |   56.019 |   148.52 |
| 16384 |    128 |    1 |  16512 |   86.613 |   189.16 |   15.187 |     8.43 |  101.800 |   162.20 |
| 32768 |    128 |    1 |  32896 |  193.332 |   169.49 |   15.501 |     8.26 |  208.833 |   157.52 |
| 65536 |    128 |    1 |  65664 |  467.644 |   140.14 |   16.139 |     7.93 |  483.783 |   135.73 |
|131072 |    128 |    1 | 131200 | 1269.178 |   103.27 |   17.476 |     7.32 | 1286.654 |   101.97 |
...aborted manually...

Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
Comment thread conversion/base.py Outdated
Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
@Little0o0

Copy link
Copy Markdown
Contributor Author

@fairydreaming The hash value may have some problems, I will check it.

@fairydreaming

Copy link
Copy Markdown
Contributor

@fairydreaming The hash value may have some problems, I will check it.

@Little0o0 I found the problem, it was simply the order of hashes in both files, it must be the same. Fixed it.

@Green-Sky

Copy link
Copy Markdown
Collaborator

Take a look at #27970

@CISC

CISC commented Sep 3, 2026

Copy link
Copy Markdown
Member

Please rebase and adapt to n_ff_exp changes.

@fairydreaming

Copy link
Copy Markdown
Contributor

@Green-Sky Since this model does build_attn(inp_attn_dsa, ...) and sparse attention changes are already there I think it will benefit from it automatically.
@CISC Done.

Perplexity after the changes: Final estimate: PPL = 2.5182 +/- 0.01150

Comment thread src/models/hy-v4.cpp Outdated
@CISC

CISC commented Sep 4, 2026

Copy link
Copy Markdown
Member

Just a maintainer's heads up; we have two models about to be merged fighting for the same llama_vocab_pre_type, one of them will need to be rebased after the other's merge.

@fairydreaming
fairydreaming merged commit 49c0dc8 into ggml-org:master Sep 4, 2026
27 of 30 checks passed
@fairydreaming

Copy link
Copy Markdown
Contributor

@Little0o0 now that the model is merged I think it would be good idea to reconvert/re-quantize the GGUFs you had uploaded in https://huggingface.co/AngelSlim/Hy4-preview-GGUF so that they work with mainline llama.cpp and update your README.

Good job with the PR!

@Green-Sky

Copy link
Copy Markdown
Collaborator

now STQ1_0 when? :D

@CISC

CISC commented Sep 4, 2026

Copy link
Copy Markdown
Member

now STQ1_0 wen? :D

Fixed that for you.

Edit: Seriously though, I want this too, but getting reviewers willing to own quants is hard...

fewtarius pushed a commit to fewtarius/CachyLLama that referenced this pull request Sep 5, 2026
…rg#28127)

* model: add Tencent Hy 4 (hy_v4) preview architecture support

Adds support for the Tencent Hy 4 model (Hugging Face architecture
HYV4ForCausalLM, GGUF arch hy_v4):

Add HF -> GGUF conversion script (conversion/hy_v4.py) and wire it into the conversion registry
Register hy_v4 GGUF constants, arch enum, and writer support
Implement the hy-v4 model graph, hparams, vocab and context changes
Register the new arch in llama-arch and models registry
Extend arch tests to cover hy_v4

Assisted by Claude Opus 5

* Update convert_hf_to_gguf_update.py

Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>

* Update conversion/base.py

Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>

* convert : move hy_v4 entry to the same place as in convert_hf_to_gguf_update.py

* model : apply changes related to n_ff_exp becoming per-layer in Hy4-preview

* n_layer_all

---------

Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com>
Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
@AlphaMo99

Copy link
Copy Markdown

I'm not sure to understand if STQ1_0 is ok ot not !??

[35203] 0.00.855.587 I srv    load_model: loading model 'AngelSlim/Hy4-preview-GGUF:STQ1_0'
[35203] 0.00.855.615 I srv    load_model: local path '/usr/share/llama/gguf/models--AngelSlim--Hy4-preview-GGUF/snapshots/779242edccdedc2109a0b36b164263a88f015bfa/Hy4-preview-STQ1_0.gguf'
[35203] 0.00.881.039 E gguf_init_from_reader: tensor 'blk.2.ffn_gate_exps.weight' has invalid ggml type 43. should be in [0, 43)
[35203] 0.00.881.045 E gguf_init_from_reader: failed to read tensor info
[35203] 0.00.883.225 E llama_model_load: error loading model: llama_model_loader: failed to load model from /usr/share/llama/gguf/models--AngelSlim--Hy4-preview-GGUF/snapshots/779242edccdedc2109a0b36b164263a88f015bfa/Hy4-preview-STQ1_0.gguf

@Little0o0

Little0o0 commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

@AlphaMo99 STQ1_0 is still waiting for merging, referring #22836 . This PR is for the Hy4 preview only.

@Little0o0

Copy link
Copy Markdown
Contributor Author

@Little0o0既然模型已经合并,我认为最好重新转换/重新量化你在https://huggingface.co/AngelSlim/Hy4-preview-GGUF上传的 GGUF 文件,以便它们能与主线 llama.cpp 一起使用,并更新你的 README 文件。

公关做得好!

Thanks for your contribution and helps @fairydreaming ! I will update the GGUF and README soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conversion model Model specific testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants