Question about MoBA RULER eval

[martyLY]

最近在测试NSA的效果，发现NSA在多跳任务，类似：RULER的Variable Tracking(VT)或者Counting Star上跌得很多；
所以想请教下MoBA的 128K RULER eval，尤其是Variable Tracking(VT)任务是怎样的setting配置？
论文里提到是62.5%的压缩率，是因为测多针或者多跳，把压缩率放宽了吗？

[whitelez]

Greetings!
The MoBA configuration we used during our evaluations is consistent across both 128K RULER and 1M NIAH benchmarks, as documented in our report:

• MoBA Block size: 4K tokens
• MoBA topK parameter: 12
• Full attention in the last 3 layers

This configuration effectively provides full attention equivalence for contexts up to 48K tokens (4K × 12 = 48K).

Regarding the 62.5% sparsity ratio observed during 128K evaluation, this wasn't deliberately relaxed for multi-hop or multi-query tasks. This ratio is a natural consequence of evaluating our 1M-trained model on 128K contexts. We maintained consistent hyperparameters across all training stages and didn't train a separate 8B model specifically for 128K evaluation.

One of MoBA's key design principles is that it approximates full attention at shorter context lengths. Specifically, when the context length is less than chunk×topK (in this case 4K×12=48K), MoBA becomes mathematically equivalent to full attention.

[martyLY]

Thanks for your reply, because paper did not test complex tasks (multi-hop or multi-query tasks) under 1M length, the RULER includes variable tracking(VT) task, so we pay special attention to its compression ratio(62.5%), there are still some questions:

• 1M is continued training from a certain length, such as 128K?
• What compression ratio can be maintained at 128K length?
• And have you tested multi-hop tasks at this compression ratio?

[1180301012]

最近在测试NSA的效果，发现NSA在多跳任务，类似：RULER的Variable Tracking(VT)或者Counting Star上跌得很多； 所以想请教下MoBA的 128K RULER eval，尤其是Variable Tracking(VT)任务是怎样的setting配置？ 论文里提到是62.5%的压缩率，是因为测多针或者多跳，把压缩率放宽了吗？

你好，我们有在做一些moba的复现工作，尝试了很多配置，发现在ruler上也会出现同样的问题；想问一下nsa的话，大概会在ruler掉多少精度呢？

[wxthu]

最近在测试NSA的效果，发现NSA在多跳任务，类似：RULER的Variable Tracking(VT)或者Counting Star上跌得很多； 所以想请教下MoBA的 128K RULER eval，尤其是Variable Tracking(VT)任务是怎样的setting配置？ 论文里提到是62.5%的压缩率，是因为测多针或者多跳，把压缩率放宽了吗？

Tracking

hello, 我最近也准备测试NSA的效果，但是似乎并未 开源完整的DeepSeekMoE NSA版本的model，想问一下是如何解决的