afia-lubaina/medgemma-usmle-blindspots
MedGemma vs. Gemma 3 on USMLE Questions: High Confidence, Even When No Answer Is Correct This repository contains the evaluation questions, model responses, and analysis from a zero-shot comparison of MedGemma 4B (google/medgemma-4b-it) and its general-purpose counterpart, Gemma 3 4B (google/gemma-3-4b-it), on USMLE-style multiple-choice questions. Key finding: Medical specialization improves accuracy by 18 percentage points, but does not improve confidence calibration. Both… See the full description on the dataset page: https://huggingface.co/datasets/afia-lubaina/medgemma-usmle-blindspots.
| Author / organization | afia-lubaina |
|---|---|
| Downloads | 0 (as reported by Hugging Face) |
| Likes | 0 |
| Created | 21 October 2026 |
| Last updated at source | 21 October 2026 |
Recent changes
- 21 Oct 2026 Description: MedGemma vs Gemma 3 on USMLE questions: confident even when no answer is right This repository holds the questions, model outputs and analysis for a zero-shot evaluation of MedGemma 4B (google/medgemma-4b-it) against its general-purpose bas → MedGemma vs. Gemma 3 on USMLE Questions: High Confidence, Even When No Answer Is Correct This repository contains the evaluation questions, model responses, and analysis from a zero-shot comparison of MedGemma 4B (google/medgemma-4b-it) and
- 21 Oct 2026 Description: none → MedGemma vs Gemma 3 on USMLE questions: confident even when no answer is right This repository holds the questions, model outputs and analysis for a zero-shot evaluation of MedGemma 4B (google/medgemma-4b-it) against its general-purpose bas
- 21 Oct 2026 Topics added medical, evaluation, calibration, usmle, medgemma
Changes detected by Silytics between refreshes of the source.
Silytics shows public metadata from Hugging Face and links to the original. Information may be incomplete or out of date. Last refreshed 21 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.