NDLI: Reliable Multiple-choice Iterative Algorithm for Crowdsourcing Systems

Please wait, while we are loading the content...

Bounds Implied by Drift with Applications

Online Auctions in IaaS Clouds: Welfare and Profit Maximization with Server Costs

Whittle Index Approach to Size-aware Scheduling with Time-varying Channels

Transient and Steady-state Regime of a Family of List-based Cache Replacement Algorithms

Online Convex Optimization Using Predictions

Bandits with Budgets: Regret Lower Bounds and Optimal Algorithms

DiskAccel: Accelerating Disk-Based Experiments by Representative Sampling

Power of d Choices for Large-Scale Bin Packing: A Loss Model

Impact of Fairness and Heterogeneity on Delays in Large-scale Content Delivery Networks

Investigating the Potential of the Inter-IXP Multigraph for the Provisioning of Guaranteed End-to-End Services

Distributed Proportional Fair Load Balancing in Heterogenous Systems

60 GHz Indoor Networking through Flexible Beams: A Link-Level Profiling

Memory Row Reuse Distance and its Role in Optimizing Application Performance

Reliable Multiple-choice Iterative Algorithm for Crowdsourcing Systems

Social Network Monetization via Sponsored Viral Marketing

Combining Phase Identification and Statistic Modeling for Automated Parallel Benchmark Generation

Computable Bounds in Fork-Join Queueing Systems

Fisher Information-based Experiment Design for Network Tomography

MaxWeight Scheduling: Asymptotic Behavior of Unscaled Queue-Differentials in Heavy Traffic

Multi-Resource Fairness: Objectives, Algorithms and Performance

Sponsored Data Plan: A Two-Class Service Model in Wireless Data Networks

Smartphone Energy Drain in the Wild: Analysis and Implications

An Online Learning Approach to Improving the Quality of Crowd-Sourcing

Spy vs. Spy: Rumor Source Obfuscation

Reducing Latency via Redundant Requests: Exact Analysis

When Virtual Meets Physical at the Edge: A Field Study on Datacenters' Virtual Traffic

Kraken: Towards Elastic Performance Guarantees in Multi-tenant Data Centers

Exchange of Services in Networks: Competition, Cooperation, and Fairness

Queue-Proportional Rate Allocation with Per-Link Information in Multihop Networks

Newer Is Sometimes Better: An Evaluation of NFSv4.1

Learning to Rank: Regret Lower Bounds and Efficient Algorithms

Greedy-Bayes for Targeted News Dissemination

Joint Cyber and Physical Attacks on Power Grids: Graph Theoretical Approaches for Information Recovery

Hyper-Compact Virtual Estimators for Big Network Data Based on Register Sharing

Latency in Software Defined Networks: Measurements and Mitigation Techniques

Resource Allocation and Rate Gains in Practical Full-Duplex Systems

A Large-Scale Study of Flash Memory Failures in the Field

Tree-structured Bloom Filters for Joint Optimization of False Positive Probability and Transmission Bandwidth

Scheduling Storms and Streams in the Cloud

Localized Epidemic Detection in Networks with Overwhelming Noise

Pricing Bilateral Electricity Trade between Smart Grids and Hybrid Green Datacenters

Detecting Sponsored Recommendations

Understanding Parallel Performance Under Interferences in Multi-tenant Clouds

Clustering and Inference From Pairwise Comparisons

Deterministic Near-Optimal P2P Streaming

Flexible Transformations For Learning Big Data

Energy Coupon: A Mean Field Game Perspective on Demand Response in Smart Grids

DeltaTree: A Locality-aware Concurrent Search Tree

Detecting and Localizing End-to-End Performance Degradation for Cellular Data Services

Speeding up Glauber Dynamics for Random Generation of Independent Sets

Online Electricity Cost Saving Algorithms for Co-Location Data Centers

Web Transparency for Complex Targeting: Algorithms, Limits, and Tradeoffs

Lagrangian-based Online Stochastic Bin Packing

Towards Cooling Internet-Scale Distributed Networks on the Cheap

A Simple Model to Quantify the Impact of Memory Latency and Bandwidth on Performance

Performance Characterization of Hyperscale Applicationson on NVMe SSDs

Reliable Multiple-choice Iterative Algorithm for Crowdsourcing Systems

Content Provider	ACM Digital Library
Author	Jung, Kyomin Lee, Hyunmin Lee, Donghyeon Kim, Joonyoung
Abstract	The appearance of web-based crowdsourcing systems gives a promising solution to exploiting the wisdom of crowds efficiently in a short time with a relatively low budget. Despite their efficiency, crowdsourcing systems have an inherent problem in that responses from workers can be unreliable since workers are low-paid and have low responsibility. Although simple majority voting can be a solution, various research studies have sought to aggregate noisy responses to obtain greater reliability in results through effective techniques such as Expectation-Maximization (EM) based algorithms. While EM-based algorithms get the limelight in crowdsourcing systems due to their useful inference techniques, Karger et al. made a significant breakthrough by proposing a novel iterative algorithm based on the idea of low-rank matrix approximations and the message passing technique. They showed that the performance of their iterative algorithm is order-optimal, which outperforms majority voting and EM-based algorithms. However, their algorithm is not always applicable in practice since it can only be applied to binary-choice questions. Recently, they devised an inference algorithm for multi-class labeling, which splits each task into a bunch of binary-choice questions and exploits their existing algorithm. However, it has difficulty in combining into real crowdsourcing systems since it overexploits redundancy in that each split question should be queried in multiple times to obtain reliable results. In this paper, we design an iterative algorithm to infer true answers for multiple-choice questions, which can be directly applied to real crowdsourcing systems. Our algorithm can also be applicable to short-answer questions as well. We analyze the performance of our algorithm, and prove that the error bound decays exponentially. Through extensive experiments, we verify that our algorithm outperforms majority voting and EM-based algorithm in accuracy.
Starting Page	205
Ending Page	216
Page Count	12
File Format	PDF
ISBN	9781450334860
DOI	10.1145/2745844.2745871
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2015-06-15
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Crowdsourcing Multiple-choice Resource allocation Iterative learning
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in