Issues

• The evaluation relies entirely on LLM-as-judge scoring without any human annotation or preference validation, making it unclear how well the measured metrics align with real user perception.

  Generative-Engine-Marketing/GEM-Bench#10 · Filter by author Qingbolan opened on Nov 14, 2025
  1

• Results are only presented with niche LLM of doubao-1.5

  llm-eval-more-modelsrebuttle
  Generative-Engine-Marketing/GEM-Bench#9 · Filter by author Qingbolan opened on Nov 13, 2025
  4

• For Figure 2, are the shown weaknesses of Ad-Chat due to inherent limitations of prompt-based insertion, or mainly due to the specific prompt used? Were alternative prompt designs evaluated?

  adchat-prompt-design-analysisrebuttle
  Generative-Engine-Marketing/GEM-Bench#8 · Filter by author Qingbolan opened on Nov 13, 2025
  2

• How sensitive are the results to the number of ads (k > 1) inserted into each response?

  ads-k-sensitivity-studyrebuttle
  Generative-Engine-Marketing/GEM-Bench#7 · Filter by author Qingbolan opened on Nov 13, 2025
  2

• The chatbot datasets contain only 10 and 100 queries, respectively, which may limit statistical robustness and generalizability to diverse domains.

  expand-chatbot-dataset-queriesrebuttle
  Generative-Engine-Marketing/GEM-Bench#6 · Filter by author Qingbolan opened on Nov 13, 2025
  2