NDLI: Crowdsourcing subjective image quality evaluation

Content Provider	IEEE Xplore Digital Library
Author	Ribeiro, F. Florencio, D. Nascimento, V.
Copyright Year	2011
Description	Author affiliation: Microsoft Research, One Microsoft Way, Redmond, WA, 98052 (Florencio, D.) \|\| Electronic Systems Engineering Department, Universidade de São Paulo, Brazil (Ribeiro, F.; Nascimento, V.)
Abstract	Subjective tests are generally regarded as the most reliable and definitive methods for assessing image quality. Nevertheless, laboratory studies are time consuming and expensive. Thus, researchers often choose to run informal studies or use objective quality measures, producing results which may not correlate well with human perception. In this paper we propose a cost-effective and convenient subjective quality measure called crowdMOS, obtained by having internet workers participate in MOS (mean opinion score) subjective quality studies. Since these workers cannot be supervised, we propose methods for detecting and discarding inaccurate or malicious scores. To facilitate this process, we offer an open source set of tools for Amazon Mechanical Turk, which is an internet marketplace for crowdsourcing. These tools completely automate the test design, score retrieval and statistical analysis, abstracting away the technical details of Mechanical Turk and ensuring a user-friendly, affordable and consistent test methodology. We demonstrate crowdMOS using data from the LIVE subjective quality image dataset, showing that it delivers accurate and repeatable results.
Starting Page	3097
Ending Page	3100
File Size	587473
Page Count	4
File Format	PDF
ISBN	9781457713040
ISSN	15224880
e-ISBN	9781457713033
e-ISBN	9781457713026
DOI	10.1109/ICIP.2011.6116320
Language	English
Publisher	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher Date	2011-09-11
Publisher Place	Belgium
Access Restriction	Subscribed
Rights Holder	Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subject Keyword	Reactive power Image quality Laboratories Humans Transform coding Multimedia communication Conferences mechanical turk crowdsourcing subjective quality quality assessment mean opinion score MOS
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

CROWDMOS: An approach for crowdsourcing mean opinion score studies

Subjective quality evaluations using crowdsourcing

Assessing internet video quality using crowdsourcing

Subject bias: Introducing a theoretical user model

Analysis of Public Image and Video Databases for Quality Assessment

Crowdsourcing based subjective quality assessment of adaptive video streaming

On Designing Paired Comparison Experiments for Subjective Multimedia Quality Assessment

Variable selection for image quality assessment using a Neural Network based approach

A novel methodology for mapping objective video quality metrics to the subjective MOS scale

Crowdsourcing subjective image quality evaluation

Similar Documents

CROWDMOS: An approach for crowdsourcing mean opinion score studies

Subjective quality evaluations using crowdsourcing

Assessing internet video quality using crowdsourcing

Subject bias: Introducing a theoretical user model

Analysis of Public Image and Video Databases for Quality Assessment

Crowdsourcing based subjective quality assessment of adaptive video streaming

On Designing Paired Comparison Experiments for Subjective Multimedia Quality Assessment

Variable selection for image quality assessment using a Neural Network based approach

A novel methodology for mapping objective video quality metrics to the subjective MOS scale

Crowdsourcing subjective image quality evaluation