JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
Reliable evaluation is essential for the development of vision-language models (VLMs). However, Japanese VQA benchmarks have undergone far less iterative refinement than their English counterparts. As a result, many existing benchmarks contain issues such as ambiguous questions, incorrect answers, and instances that can be solved without visual grounding, undermining evaluation reliability and leading to misleading conclusions in model comparisons. To address these limitations, we introduce JAMMEval, a refined collection of Japanese benchmarks for reliable VLM evaluation. It is constructed by systematically refining seven existing Japanese benchmark datasets through two rounds of human an...
| Authors | Issa Sugiura, Koki Maeda, Shuhei Kurita et al. |
|---|---|
| Primary category | cs.CV |
| Published | 1 April 2026 |
| Last updated at source | 4 April 2026 |
Connected in the knowledge graph
- llm-jp/JAMMEvalDataset · cited by
- llm-jp/llm-jp-4-vl-9b-betaModel · cited by
- llm-jp/llm-jp-4-vl-9bModel · cited by
Models and datasets referencing this paper
- llm-jp/llm-jp-4-vl-9b-betaModel
- llm-jp/llm-jp-4-vl-9bModel
- llm-jp/JAMMEvalDataset
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 17 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.