On the Limits of LLM-as-Judge for Scientific Novelty Assessment
LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, a benchmark built from recent arXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They are author-anchored...
| Authors | Soumitra Sinhahajari, Navonil Majumder, Soujanya Poria |
|---|---|
| Primary category | cs.DL |
| Published | 10 June 2026 |
| Last updated at source | 10 June 2026 |
Connected in the knowledge graph
- declare-lab/rq-benchDataset · cited by
Models and datasets referencing this paper
- declare-lab/rq-benchDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.