NDLI: Information retrieval evaluation with humans in the loop

Please wait, while we are loading the content...

Information retrieval evaluation with humans in the loop

Workshop on searching for fun 2014

Finding information about mental health in microblogging platforms: a case study of depression

Simulated work tasks: the case of professional users

X-REC: cross-category entity recommendation

Web search for instruction-related information: Why? Where? How?

Seeking answers, making sense, changing lifestyles: cognitive models of human-information interaction

Searching as learning (SAL) workshop 2014

Usability and perception of young users and adults on targeted web search engines

Visual information search based on knowledge schema modeling

Interactive summarization of social media

Searcher heterogeneity in collaborative information seeking within the context of work tasks

Engaging users with situational recommendations: challenges and results

Books' interest grading and dwell time in metadata in selecting fiction

Shedding light on a living lab: the CLEF NEWSREEL open recommendation platform

YASFIIRE: yet another system for IIR evaluation

Primary school children's single topic search over multiple search sessions: does search evolve?

Relating user interaction to experience during festivals

What does time constraint mean to information searchers?

Mobile tourist guides: bridging the gap between automation and users retaining control of their itineraries

Measuring and improving data quality of media collections for professional tasks

A user defined taxonomy of factors that divide online information retrieval sessions

Multilingual interface preferences

Develop, implement, and improve a web session detection model

Characterizing relevance with eye-tracking measures

Users' criteria of video digital libraries from a public affairs context

Personal information management and social networks re-finding on Twitter

How concept maps change if a user does search or not?

Modeling decision points in user search behavior

Too little time?: time constraints and time pressure in information search

Are episodic context features helpful for refinding tasks?: lessons learnt from a case study with lifelogs

Evaluating a tool for the exploratory analysis of usability information using a cognitive walkthrough method

Combining document retrieval with knowledge graphs for exploratory search

E-mail categorization using partially related training examples

Modeling the interactive patent retrieval process: an adaptation of Marchionini's information seeking model

Designing autotelic searching experience for casual-leisure by using the user's context

Investigating the potential impact of non-personalized recommendations in the OPAC: Amazon vs. WorldCat.org

Categorising search sessions: some insights from human judgments

Investigation of information behavior in Wikipedia articles

Social priors to estimate relevance of a resource

Health information seeking using smartphones among low SES hispanic adults in the U.S.A.

Adaptive search systems for web archive research

On choosing an effective automatic evaluation metric for microblog summarisation

Contextual modeling content-based approaches for new-item recommendation

Diversifying contextual suggestions from location-based social networks

Users' reading habits in online news portals

Exploring knowledge graphs for exploratory search

Does the perceived usefulness of search facets vary by task type?

From multistage information-seeking models to multistage search systems

A review of users' search contexts for lifelogging system design

Stuck in traffic: how temporal delays affect search behaviour

Classifying the influence of negative affect expressed by the financial media on investor behavior

The effect of cognitive abilities on information search for tasks of varying levels of complexity

Curiosity driven search: when is relevance irrelevant?

Investigating people: a qualitative analysis of the search behaviours of open-source intelligence analysts

User intent behind medical queries: an evaluation of entity mapping approaches with metamap and freebase

Itinerary recommenders: how do users customize their routes and what can we learn from them?

Tweets I've seen: analysing factors influencing re-finding frustration on Twitter

A qualitative exploration of secondary assessor relevance judging behavior

Using artefacts to investigate children's information seeking experiences

Time well spent

A social bookmarking system to support cluster driven archival arrangement

Browsing patterns in retrieved documents

Using sensor graphs to stimulate recall in retrospective think-aloud protocols

Information retrieval evaluation with humans in the loop

Content Provider	ACM Digital Library
Author	Kazai, Gabriella
Abstract	The evaluation and tuning of information retrieval (IR) systems based on the Cranfield paradigm requires purpose built test collections, which include sets of human contributed relevance labels, indicating the relevance of search results to a set of user queries. Traditional methods of collecting relevance labels rely on a fixed group of hired expert judges, who are trained to interpret user queries as accurately as possible and label documents accordingly. Human judges and the obtained relevance labels thus provide a critical link within the Cranfield style IR evaluation framework, where disagreement among judges and the impact of variable judgment sets on the final outcome of an evaluation is a well studied issue. There is also reported evidence that experiment outcomes can be affected by changes to the judging guidelines or changes in the judge population. Recently, the growing volume and diversity of the topics and documents to be judged is driving the increased adoption of crowdsourcing methods in IR evaluation, offering a viable alternative that scales with modest costs. In this model, relevance judgments are distributed online over a large population of humans, a crowd, facilitated, for example, by a crowdsourcing platform, such as Amazon's Mechanical Turk or Clickworker. Such platforms allow millions of anonymous crowd workers to be hired temporarily for micro-payments to complete so-called human intelligence tasks (HITs), such as labeling images or documents. Studies have shown that workers come from diverse backgrounds, work in a variety of different environments, and have different motivations. For example, users may turn to crowdsourcing as a way to make a living, to serve an altruistic or social purpose or simply to fill their time. They may become loyal crowd workers on one or more platforms, or they may leave after their first couple of encounters. Clearly, such a model is in stark contrast to the highly controlled methods that characterize the work of trained judges. For example, in a micro-task based crowdsourcing setup, worker training is usually minimal or non-existent. Furthermore, it is widely reported that labels provided by crowd workers can vary in quality, leading to noisy labels. Crowdsourcing can also suffer from undesirable worker behaviour and practices, e.g., dishonest behaviour or lack of expertise, that result in low quality contributions. While a range of quality assurance and control techniques have now been developed to reduce noise during or after task completion, little is known about the workers themselves and possible relationships between workers' characteristics, behaviour and the quality of their work. In this talk, I will review the findings of recent research that examines and compares trained judges and crowd workers hired to complete relevance assessment tasks of varying difficulty. The investigations include a range of aspects from how HIT design, judging instructions, worker demographics and characteristics may impact work quality. The main focus of the talk will be on experiments aimed to uncover characteristics of the crowd by monitoring their behaviour during different relevance assessment tasks, and compare them to professional judges' behaviour on the same tasks. Throughout the talk I will highlight challenges of quality assurance and control in crowdsourcing and propose a possible direction for solving the issue without relying on gold standard data sets, which are expensive to create and have limited application.
Starting Page	1
Ending Page	1
Page Count	1
File Format	PDF
ISBN	9781450329767
DOI	10.1145/2637002.2637003
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2014-08-26
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Crowdsourcing Worker characteristics and behaviour Ir evaluation
Content Type	Text
Resource Type	Article

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in

Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems

Worker types and personality traits in crowdsourcing relevance labels

Reflections on information retrieval evaluation (1999)

Repeatable and reliable search system evaluation using crowdsourcing

A Blueprint of IR Evaluation Integrating Task and User Characteristics: Test Collection and Evaluation Metrics

Evaluation criteria for information retrieval systems.

The TREC-like evaluation of music IR systems

Exploiting Disambiguation and Discrimination in Information Retrieval Systems

Crowdsourcing for information retrieval: introduction to the special issue

Information retrieval evaluation with humans in the loop

Similar Documents

Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems

Worker types and personality traits in crowdsourcing relevance labels

Reflections on information retrieval evaluation (1999)

Repeatable and reliable search system evaluation using crowdsourcing

A Blueprint of IR Evaluation Integrating Task and User Characteristics: Test Collection and Evaluation Metrics

Evaluation criteria for information retrieval systems.

The TREC-like evaluation of music IR systems

Exploiting Disambiguation and Discrimination in Information Retrieval Systems

Crowdsourcing for information retrieval: introduction to the special issue

Information retrieval evaluation with humans in the loop