Who Do Perspectivist Models Predict Well? Annotator Agreement, Sparsity, and Performance
Successful completion of perspectivist natural language processing (NLP) annotation tasks requires algorithms to be aligned across diverse perspectives in order to predict how different humans will rate text. This work studies how the performance of such algorithms varies for annotators who behave differently than others, either by providing annotations that diverge from others' responses or by providing fewer annotations altogether. To aid with characterizing model performance, we curate the largest collection to date of perspectivist NLP datasets with ordinal annotations. After evaluating multiple classes of perspectivist algorithms across each of the datasets, we model the relationship...
| Authors | London Lowmanstone, Vaagishwaran Sivakumar, Anwesha Samaddar et al. |
|---|---|
| Primary category | cs.CL |
| Published | 15 October 2026 |
| Last updated at source | 15 October 2026 |
Popularity, last 6 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 24 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.