Szczegóły publikacji
Opis bibliograficzny
Describing subjective experiment consistency by $p$-value P-P plot / Jakub NAWAŁA, Lucjan JANOWSKI, Bogdan ĆMIEL, Krzysztof RUSEK // W: MM '20 [Dokument elektroniczny] : proceedings of the 28th ACM international conference on Multimedia : Seattle, October, 2020. — Wersja do Windows. — Dane tekstowe. — New York : Association for Computing Machinery, [2020]. — e-ISBN: 978-1-4503-7988-5. — S. 852–861. — Wymagania systemowe: Adobe Reader. — Bibliogr. s. 860–861, Abstr.
Autorzy (4)
Słowa kluczowe
Dane bibliometryczne
| ID BaDAP | 130739 |
|---|---|
| Data dodania do BaDAP | 2020-10-21 |
| Tekst źródłowy | URL |
| DOI | 10.1145/3394171.3413749 |
| Rok publikacji | 2020 |
| Typ publikacji | materiały konferencyjne (aut.) |
| Otwarty dostęp |  |
| Wydawca | Association for Computing Machinery (ACM) |
| Konferencja | ACM Multimedia 2020 |
Abstract
There are phenomena that cannot be measured without subjective testing. However, subjective testing is a complex issue with many influencing factors. These interplay to yield either precise or incorrect results. Researchers require a tool to classify results of subjective experiment as either consistent or inconsistent. This is necessary in order to decide whether to treat the gathered scores as quality ground truth data. Knowing if subjective scores can be trusted is key to drawing valid conclusions and building functional tools based on those scores (e.g., algorithms assessing the perceived quality of multimedia materials). We provide a tool to classify subjective experiment (and all its results) as either consistent or inconsistent. Additionally, the tool identifies stimuli having irregular score distribution. The approach is based on treating subjective scores as a random variable coming from the discrete Generalized Score Distribution (GSD). The GSD, in combination with a bootstrapped G-test of goodness-of-fit, allows to construct p-value P–P plot that visualizes experiment's consistency. The tool safeguards researchers from using inconsistent subjective data. In this way, it makes sure that conclusions they draw and tools they build are more precise and trustworthy. The proposed approach works in line with expectations drawn solely on experiment design descriptions of 21 real-life multimedia quality subjective experiments.
