NDLI: PPLib: Toward the Automated Generation of Crowd Computing Programs Using Process Recombination and Auto-Experimentation

Content Provider	ACM Digital Library
Author	Boer, Patrick M. De Bernstein, Abraham
Copyright Year	2016
Abstract	Crowdsourcing is increasingly being adopted to solve simple tasks such as image labeling and object tagging, as well as more complex tasks, where crowd workers collaborate in processes with interdependent steps. For the whole range of complexity, research has yielded numerous patterns for coordinating crowd workers in order to optimize crowd accuracy, efficiency, and cost. Process designers, however, often don't know which pattern to apply to a problem at hand when designing new applications for crowdsourcing. In this article, we propose to solve this problem by systematically exploring the design space of complex crowdsourced tasks via automated recombination and auto-experimentation for an issue at hand. Specifically, we propose an approach to finding the optimal process for a given problem by defining the deep structure of the problem in terms of its abstract operators, generating all possible alternatives via the (re)combination of the abstract deep structure with concrete implementations from a Process Repository, and then establishing the best alternative via auto-experimentation. To evaluate our approach, we implemented PPLib (pronounced “People Lib”), a program library that allows for the automated recombination of known processes stored in an easily extensible Process Repository. We evaluated our work by generating and running a plethora of process candidates in two scenarios on Amazon's Mechanical Turk followed by a meta-evaluation, where we looked at the differences between the two evaluations. Our first scenario addressed the problem of text translation, where our automatic recombination produced multiple processes whose performance almost matched the benchmark established by an expert translation. In our second evaluation, we focused on text shortening; we automatically generated 41 crowd process candidates, among them variations of the well-established Find-Fix-Verify process. While Find-Fix-Verify performed well in this setting, our recombination engine produced five processes that repeatedly yielded better results. We close the article by comparing the two settings where the Recombinator was used, and empirically show that the individual processes performed differently in the two settings, which led us to contend that there is no unifying formula, hence emphasizing the necessity for recombination.
Starting Page	1
Ending Page	20
Page Count	20
File Format	PDF
ISSN	21576904
e-ISSN	21576912
DOI	10.1145/2897367
Volume Number	7
Issue Number	4
Journal	ACM Transactions on Intelligent Systems and Technology (TIST)
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2016-04-25
Publisher Place	New York
Access Restriction	One Nation One Subscription (ONOS)
Subject Keyword	Human computation algorithms
Content Type	Text
Resource Type	Article
Subject	Artificial Intelligence Theoretical Computer Science

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

PPLib: towards systematic crowd process design using recombination and auto-experimentation

Rapid Low-Cost Virtual Human Bootstrapping via the Crowd

A research-led and industry-oriented MSc program in natural computation

Natural and Remote Sensing Image Segmentation Using Memetic Computing

Discovering Unique, Low-Energy Transition States Using Evolutionary Molecular Memetic Computing

Experience teaching a graduate level course in evolutionary computation

An Introduction to Evolutionary Computation in Finance

Crowd-Powered Systems

An Introduction to the Special Issue on Participatory Sensing and Crowd Intelligence

PPLib: Toward the Automated Generation of Crowd Computing Programs Using Process Recombination and Auto-Experimentation

Similar Documents

PPLib: towards systematic crowd process design using recombination and auto-experimentation

Rapid Low-Cost Virtual Human Bootstrapping via the Crowd

A research-led and industry-oriented MSc program in natural computation

Natural and Remote Sensing Image Segmentation Using Memetic Computing

Discovering Unique, Low-Energy Transition States Using Evolutionary Molecular Memetic Computing

Experience teaching a graduate level course in evolutionary computation

An Introduction to Evolutionary Computation in Finance

Crowd-Powered Systems

An Introduction to the Special Issue on Participatory Sensing and Crowd Intelligence

PPLib: Toward the Automated Generation of Crowd Computing Programs Using Process Recombination and Auto-Experimentation