NDLI: Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me

Please wait, while we are loading the content...

Forward to the Past: Notes towards a Pre-history of Web Search

Mail Search: It's Getting Personal!

Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me

Learning to Rank Using Localized Geometric Mean Metrics

Variational Deep Semantic Hashing for Text Documents

Engaged or Frustrated?: Disambiguating Emotional State in Search

Multi-site User Behavior Modeling and Its Application in Video Recommendation

Deep Semantic Hashing with Generative Adversarial Networks

Understanding and Modeling Success in Email Search

Personalized Key Frame Recommendation

Accounting for the Correspondence in Commented Data

Retrieval Consistency in the Presence of Query Variations

Efficient Cost-Aware Cascade Ranking in Multi-Stage Retrieval

Intent-Aware Semantic Query Annotation

IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models

Content Recommendation for Viral Social Influence

Classification by Retrieval: Binarizing Data and Classifiers

Learning a Hierarchical Embedding Model for Personalized Product Search

Joint Learning of Response Ranking and Next Utterance Suggestion in Human-Computer Conversation System

Predicting Which Topics You Will Join in the Future on Social Media

Word-Entity Duet Representations for Document Ranking

On the Reusability of "Living Labs" Test Collections: A Case Study of Real-Time Summarization

Experiments with Convolutional Neural Network Models for Answer Selection

BioNex: A System For Biomedical News Event Exploration

MAPping the probability of start-up success

Making Ads More Relevant Innovations in Digital Advertising

Cross-Lingual Information Retrieve in Sogou Search

Managing Tail Latencies in Large Scale IR Systems

Statistical Significance Testing in Information Retrieval: Theory and Practice

Axiomatic Thinking for Information Retrieval: And Related Tasks

Meta-evaluation of Online and Offline Web Search Evaluation Metrics

End-to-End Neural Ad-hoc Ranking with Kernel Pooling

ICE: Item Concept Embedding via Textual Information

A Study of Snippet Length and Informativeness: Behaviour, Performance and User Experience

Item Silk Road: Recommending Items from Information Domains to Social Users

Characterizing and Predicting Enterprise Email Reply Behavior

Investigating Examination Behavior of Image Search Users

Personalized Itinerary Recommendation with Queuing Time Awareness

Jointly Learning Word Embeddings and Latent Topics

Comparing In Situ and Multidimensional Relevance Judgments

Computational Social Indicators: A Case Study of Chinese University Ranking

Efficient & Effective Selective Query Rewriting with Efficiency Predictions

Personalized PageRank in Uncertain Graphs with Mutually Exclusive Edges

Exploiting Food Choice Biases for Healthier Recipe Recommendation

BitFunnel: Revisiting Signatures for Search

Exploring User-Specific Information in Music Retrieval

Learning to Rank Question Answer Pairs with Holographic Dual LSTM Architecture

What Are You Known For?: Learning User Topical Profiles with Implicit and Explicit Footprints

Dynamic Factual Summaries for Entity Cards

Automatically Extracting High-Quality Negative Examples for Answer Selection in Question Answering

Luandri: A Clean Lua Interface to the Indri Search Engine

RankEval: An Evaluation and Analysis Framework for Learning-to-Rank Solutions

Search Without a Query: Powering Job Recommendations via Search Index at LinkedIn

Traditional IR Meets Ontology Engineering in Search for Data

Find Shoes Like These

Health Misinformation in Search and Social Media

Candidate Selection for Large Scale Personalized Search and Recommender Systems

Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2017)

The Probability that Your Hypothesis Is Correct, Credible Intervals, and Effect Sizes for IR Evaluation

Neural Ranking Models with Weak Supervision

Leveraging Contextual Sentence Relations for Extractive Summarization Using a Neural Attention Model

Improving Exploratory Search Experience through Hierarchical Knowledge Graphs

Cross-Domain Recommendation via Clustering on Multi-Layer Graphs

ReAct: Online Multimodal Embedding for Recency-Aware Spatiotemporal Activity Modeling

Extracting Hierarchies of Search Tasks & Subtasks via a Bayesian Nonparametric Approach

Attentive Collaborative Filtering: Multimedia Recommendation with Item- and Component-Level Attention

On the Power Laws of Language: Word Frequency Distributions

Online In-Situ Interleaved Evaluation of Real-Time Push Notification Systems

Information Retrieval Meets Game Theory: The Ranking Competition Between Documents? Authors

Relevance-based Word Embedding

Adapting Markov Decision Process for Search Result Diversification

Embedding Factorization Models for Jointly Recommending Items and User Generated Lists

Efficient Data Structures for Massive N-Gram Datasets

The Utility and Privacy Effects of a Click

Incomplete Follow-up Question Resolution using Retrieval based Sequence to Sequence Learning

Hierarchical Community-Level Information Diffusion Modeling in Social Networks

MEmbER: Max-Margin Based Embeddings for Entity Retrieval

Leveraging Cross-Network Information for Graph Sparsification in Influence Maximization

Finally, a Downloadable Test Collection of Tweets

ReviewMiner: An Aspect-based Review Analytics System

Spotify: Music Access At Scale

Semantic Query Understanding

Machine Learning Powered A/B Testing

Relevance Judgments: Preferences, Scores and Ties

A/B Testing at Scale: Accelerating Software Innovation

First International Workshop on Conversational Approaches to Information Retrieval (CAIR'17)

Can Deep Effectiveness Metrics Be Evaluated Using Shallow Judgment Pools?

Stacking Bagged and Boosted Forests for Effective Automated Classification

Searching on the Go: The Effects of Fragmented Attention on Mobile Web Search Tasks

Optimizing Trade-offs Among Stakeholders in Real-Time Bidding by Incorporating Multimedia Metrics

EntiTables: Smart Assistance for Entity-Focused Tables

Using Information Scent to Understand Mobile and Desktop Web Search Behavior

Neural Rating Regression with Abstractive Tips Generation for Recommendation

Evaluating Web Search with a Bejeweled Player Model

On Application of Learning to Rank for E-Commerce Search

Learning to Diversify Search Results via Subtopic Attention

Faster BlockMax WAND with Variable-sized Blocks

Privacy through Solidarity: A User-Utility-Preserving Framework to Counter Profiling

Modelling Information Needs in Collaborative Search Conversations

Combining Top-N Recommenders with Metasearch Algorithms

Cookpad Image Dataset: An Image Collection as Infrastructure for Food Research

Nordlys: A Toolkit for Entity-Oriented and Semantic Search

Structuring the Unstructured: From Startup to Making Sense of eBay's Huge eCommerce Inventory

Twicalli: An Earthquake Detection System Based on Citizen Sensors Used for Emergency Response in Chile

Naver Search: Deep Learning Powered Search Portal for Intelligent Information Provision

Searchbots: Using Chatbots in Collaborative Information-seeking Tasks

Probabilistic Topic Models for Text Data Retrieval and Analysis

SIGIR 2017 Workshop on eCommerce (ECOM17)

Deep Learning for Extreme Multi-label Text Classification

User Interaction Sequences for Search Satisfaction Prediction

A Probabilistic Reformulation of Memory-Based Collaborative Filtering: Implications on Popularity Biases

Deep Character-Level Click-Through Rate Prediction for Sponsored Search

Neural Factorization Machines for Sparse Predictive Analytics

Evaluating Mobile Search with Height-Biased Gain

Retrieval Algorithms Optimized for Human Learning

LoSHa: A General Framework for Scalable Locality Sensitive Hashing

Counter Deanonymization Query: H-index Based k-Anonymization Privacy Protection for Social Networks

SogouT-16: A New Web Corpus to Embrace IR Research

Teaching the Information Retrieval Process Using a Web-Based Environment and Game Mechanics

Examining Information on Social Media: Topic Modelling, Trend Prediction and Community Classification

Neural Networks for Information Retrieval

The First Workshop on Knowledge Graphs and Semantics for Text Retrieval and Analysis (KG4IR)

SPOT: Selecting occuPations frOm Trajectories

A Test Collection for Evaluating Retrieval of Studies for Inclusion in Systematic Reviews

Smart Media Generation System for Broadcasting Contents

Dynamic Personalized Ranking of Facets for Exploratory Search

Building Test Collections: An Interactive Guide for Students and Others Without Their Own Evaluation Conference Series

The Lucene for Information Access and Retrieval Research (LIARR) Workshop at SIGIR 2017

Personalized Query Suggestion Diversification

One Million Posts: A Data Set of German Online Discussions

EvALL: Open Access Evaluation for Information Access Systems

Multi-dimensional Formula Feature Modeling for Mathematical Information Retrieval

From Design to Analysis: Conducting Controlled Laboratory Experiments with Users

SIGIR 2017 Workshop on Neural Information Retrieval (Neu-IR'17)

Weighted Domain Translation for Online News Comments Emotion Tagging

KASANDR: A Large-Scale Dataset with Implicit Feedback for Recommendation

TOTEM: Personal Tweets Summarization on Mobile Devices

Deep Collaborative Filtering Approaches for Context-Aware Venue Recommendation

SIGIR 2017 Tutorial on Health Search (HS2017): A Full-day from Consumers to Clinicians

SIGIR 2017 Workshop on Open Knowledge Base and Question Answering (OKBQA2017)

From Footprint to Friendship: Modeling User Followership in Mobile Social Networks from Check-in Data

A Collection for Detecting Triggers of Sentiment Spikes

LSTM vs. BM25 for Open-domain QA: A Hands-on Comparison of Effectiveness and Efficiency

Utilizing Online Social Media for Disaster Relief: Practical Challenges in Retrieval

Video Question Answering via Attribute-Augmented Attention Network Learning

Anserini: Enabling the Use of Lucene for Information Retrieval Research

Proactive Information Retrieval via Screen Surveillance

Learning Max-Margin GeoSocial Multimedia Network Representations for Point-of-Interest Suggestion

A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms

ASTERIX: Ambiguity and Missing Element-Aware XML Keyword Search Engine

Learning To Rank Resources

A Large-Scale Query Spelling Correction Corpus

Visual Pool: A Tool to Visualize and Interact with the Pooling Method

DeepStyle: Learning User Preferences for Visual Recommendation

DBpedia-Entity v2: A Test Collection for Entity Search

Event Detection on Curated Tweet Streams

Target Type Identification for Entity-Bearing Queries

A Cross-Platform Collection for Contextual Suggestion

A Task-oriented Search Engine for Evidence-based Medicine

Query Expansion for Email Search

RELink: A Research Framework and Test Collection for Entity-Relationship Retrieval

Social Media Image Recognition for Food Trend Analysis

Generating Clinical Queries from Patient Narratives: A Comparison between Machines and Humans

Computing Web-scale Topic Models using an Asynchronous Parameter Server

Centered kNN Graph for Semi-Supervised Learning

Event Recommendation based on Graph Random Walking and History Preference Reranking

An Extended Relevance Model for Session Search

An Enhanced Approach to Query Performance Prediction Using Reference Lists

Autonomous Crowdsourcing through Human-Machine Collaborative Learning

Game State Retrieval with Keyword Queries

Improving Search Engines via Large-Scale Physiological Sensing

A Poisson Regression Method for Top-N Recommendation

A Hierarchical Multimodal Attention-based Neural Network for Image Captioning

Unifying Multi-Source Social Media Data for Personalized Travel Route Planning

LiveMaps: Converting Map Images into Interactive Maps

Sub-corpora Impact on System Effectiveness

Automatic and Semi-Automatic Document Selection for Technology-Assisted Review

Recommending Complementary Products in E-Commerce Push Notifications with a Mixture Model Approach

Neural Network based Reinforcement Learning for Real-time Pushing on Text Stream

Joint Latent Subspace Learning and Regression for Cross-Modal Retrieval

Label Aggregation for Crowdsourcing with Bi-Layer Clustering

Effective Music Feature NCP: Enhancing Cover Song Recognition with Music Transcription

Deep Multimodal Embedding Model for Fine-grained Sketch-based Image Retrieval

Cross-Device User Linking: URL, Session, Visiting Time, and Device-log Embedding

Mailbox-Based vs. Log-Based Query Completion for Mail Search

The Impact of Linkage Methods in Hierarchical Clustering for Active Learning to Rank

Reinforcement Learning to Rank with Markov Decision Process

A Comparative Live Evaluation of Multileaving Methods on a Commercial cQA Search

Support for Interactive Identification of Mentioned Entities in Conversational Speech

AutoSVD++: An Efficient Hybrid Collaborative Filtering Model via Contractive Auto-encoders

Unsupervised Query-Focused Multi-Document Summarization using the Cross Entropy Method

Translation of Natural Language Query Into Keyword Query Using a RNN Encoder-Decoder

Predictive Network Representation Learning for Link Prediction

Sentence-level Sentiment Classification with Weak Supervision

Predicting Session Length in Media Streaming

Exploring the Query Halo Effect in Site Search: Leading People to Longer Queries

Top-N Recommendation with High-Dimensional Side Information via Locality Preserving Projection

Applying Information Extraction for Patent Structure Analysis

Enhancing Recurrent Neural Networks with Positional Attention for Question Answering

Event Early Embedding: Predicting Event Volume Dynamics at Early Stage

POI Popularity Prediction via Hierarchical Fusion of Multiple Social Clues

Multitask Learning for Fine-Grained Twitter Sentiment Analysis

Evolution of Information Needs based on Life Event Experiences with Topic Transition

Distribution-oriented Aesthetics Assessment for Image Search

On the Benefit of Incorporating External Features in a Neural Architecture for Answer Sentence Selection

Personalized Response Generation via Domain adaptation

An Accurate, Efficient, and Scalable Approach to Channel Matching in Smart TVs

Top-K Influential Nodes in Social Networks: A Game Perspective

Online Learning to Rank for Cross-Language Information Retrieval

Mining Business Opportunities from Location-based Social Networks

On Including the User Dynamic in Learning to Rank

Document Expansion Using External Collections

Detecting Positive Medical History Mentions

Emotional Social Signals for Search Ranking

Skill Translation Models in Expert Finding

A Metric for Sentence Ordering Assessment Based on Topic-Comment Structure

Gaussian Embeddings for Collaborative Filtering

Detecting Controversies in Online News Media

Generating and Personalizing Bundle Recommendations on Steam

X-DART: Blending Dropout and Pruning for Efficient Learning to Rank

Non-negative Matrix Factorization Meets Word Embedding

Term Proximity Constraints for Pseudo-Relevance Feedback

Gauging the Quality of Relevance Assessments using Inter-Rater Agreement

Neural Citation Network for Context-Aware Citation Recommendation

A Study of SVM Kernel Functions for Sensitivity Classification Ensembles with POS Sequences

Entity Set Expansion via Knowledge Graphs

Word Embedding Causes Topic Shifting; Exploit Global Context!

Building Your Own Reading List Anytime via Embedding Relevance, Quality, Timeliness and Diversity

ENCORE: External Neural Constraints Regularized Distant Supervision for Relation Extraction

Information Retrieval Model using Generalized Pareto Distribution and Its Application to Instance Search

Predicting Information Seeking Intentions from Search Behaviors

But Is It Statistically Significant?: Statistical Significance in IR Research, 1995-2014

Generalized Mixed Effect Models for Personalizing Job Search

Contextualizing Citations for Scientific Summarization using Word Embeddings and Domain Knowledge

Crowdsourced App Review Manipulation

CitySearcher: A City Search Engine For Interests

Cross-Language Question Re-Ranking

Open Relation Extraction for Support Passage Retrieval: Merit and Open Issues

Generating Query Suggestions to Support Task-Based Search

De-duping URLs with Sequence-to-Sequence Neural Networks

Graph Summarization for Entity Relatedness Visualization

Layout and Semantics: Combining Representations for Mathematical Formula Search

Understanding and Predicting Usefulness Judgment in Web Search

Open Source Repository Recommendation in Social Coding

Venue Appropriateness Prediction for Personalized Context-Aware Venue Suggestion

Building Bridges across Social Platforms: Answering Twitter Questions with Yahoo! Answers

Large-Scale Goodness Polarity Lexicons for Community Question Answering

A Neural Language Model for Query Auto-Completion

Social Media Advertisement Outreach: Learning the Role of Aesthetics

Seasonal Web Search Query Selection for Influenza-Like Illness (ILI) Estimation

Improving Retrieval Performance for Verbose Queries via Axiomatic Analysis of Term Discrimination Heuristic

Timestamping Entities using Contextual Information

On the Use of an Intermediate Class in Boolean Crowdsourced Relevance Annotations for Learning to Rank Comments

Embedding-based Query Expansion for Weighted Sequential Dependence Retrieval Model

Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me

Content Provider	ACM Digital Library
Author	Cormack, Gordon V. Grossman, Maura R.
Abstract	Technology-assisted review ("TAR") systems seek to achieve "total recall"; that is, to approach, as nearly as possible, the ideal of 100% recall and 100% precision, while minimizing human review effort. The literature reports that TAR methods using relevance feedback can achieve considerably greater than the 65% recall and 65% precision reported by Voorhees as the "practical upper bound on retrieval performance. since that is the level at which humans agree with one another" (Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, 2000). This work argues that in order to build - as well as to, evaluate - TAR systems that approach 100% recall and 100% precision, it is necessary to model human assessment, not as absolute ground truth, but as an indirect indicator of the amorphous property known as "relevance." The choice of model impacts both the evaluation of system effectiveness, as well as the simulation of relevance feedback. Models are presented that better fit available data than the infallible ground-truth model. These models suggest ways to improve TAR-system effectiveness so that hybrid human-computer systems can improve on both the accuracy and efficiency of human review alone. This hypothesis is tested by simulating TAR using two datasets: the TREC 4 AdHoc collection, and a dataset consisting of 401,960 email messages that were manually reviewed and classified by a single individual, Roger, in his official capacity as Senior State Records Archivist. The results using the TREC 4 data show that TAR achieves higher recall and higher precision than the assessments by either of two independent NIST assessors, and blind adjudication of the email dataset, conducted by Roger, more than two years after his original review, shows that he could have achieved the same recall and better precision, while reviewing substantially fewer than 401,960 emails, had he employed TAR in place of exhaustive manual review.
Starting Page	5
Ending Page	14
Page Count	10
File Format	PDF
ISBN	9781450350228
DOI	10.1145/3077136.3080812
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2017-08-07
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Continuous active learning Total recall High recall Relevance assessment
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in