NDLI: Splitting Water: Precision and Anti-Precision to Reduce Pool Bias

Please wait, while we are loading the content...

Salton Award Lecture: People, Interacting with Information

Exploring Session Context using Distributed Representations of Queries and Reformulations

Dynamic Query Modeling for Related Content Finding

Optimal Aggregation Policy for Reducing Tail Latency of Web Search

Summarizing Contrastive Themes via Hierarchical Non-Parametric Processes

Analyzing User's Sequential Behavior in Query Auto-Completion via Markov Processes

A Random Walk Model for Optimization of Search Impact in Web Frontier Ranking

How many results per page?: A Study of SERP Size, Search Behavior and User Experience

Multiple Social Network Learning and Its Application in Volunteerism Tendency Prediction

Relevance Scores for Triples from Type-Like Relations

Bayesian Ranker Comparison Based on Historical User Interactions

WEMAREC: Accurate and Scalable Recommendation through Weighted and Ensemble Matrix Approximation

An Efficient and Scalable MetaFeature-based Document Classification Approach based on Massively Parallel Computing

Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings

Retrieval of Relevant Opinion Sentences for New Products

Learning to Extract Local Events from the Web

Optimised Scheduling of Online Experiments

Inferring Searcher Attention by Jointly Modeling User Interactions and Content Salience

Leveraging Procedural Knowledge for Task-oriented Search

Towards a Game-Theoretic Framework for Information Retrieval

Representative & Informative Query Selection for Learning to Rank using Submodular Functions

Learning to Reweight Terms with Distributed Representations

On the Relation Between Assessor's Agreement and Accuracy in Gamified Relevance Assessment

An Entity Class-Dependent Discriminative Mixture Model for Cumulative Citation Recommendation

Islands in the Stream: A Study of Item Recommendation within an Enterprise Social Stream

Information Retrieval as Card Playing: A Formal Model for Optimizing Interactive Retrieval Interface

Using Sensor Metadata Streams to Identify Topics of Local Events in the City

DINFRA: A One Stop Shop for Computing Multilingual Semantic Relatedness

Promoting User Engagement and Learning in Amorphous Search Tasks

From Web Search Relevance to Vertical Search Relevance

Practical Lessons for Gathering Quality Labels at Scale

Building and Using Models of Information Seeking, Search and Retrieval: Full Day Tutorial

Web Question Answering: Beyond Factoids: SIGIR 2015 Workshop

An Eye-Tracking Study of Query Reformulation

Image-Based Recommendations on Styles and Substitutes

QuickScorer: A Fast Algorithm to Rank Documents with Additive Ensembles of Regression Trees

Splitting Water: Precision and Anti-Precision to Reduce Pool Bias

Learning by Example: Training Users with High-quality Query Suggestions

A Similarity Measure for Weaving Patterns in Textiles

Influence of Vertical Result in Web Search Examination

HSpam14: A Collection of 14 Million Tweets for Hashtag-Oriented Spam Research

Fielded Sequential Dependence Model for Ad-Hoc Entity Retrieval in the Web of Data

Incorporating Non-sequential Behavior into Click Models

Effective Latent Models for Binary Feedback in Recommender Systems

Listwise Collaborative Filtering

Learning to Rank Short Text Pairs with Convolutional Deep Neural Networks

Learning Hierarchical Representation Model for NextBasket Recommendation

Rank-GeoFM: A Ranking based Geographical Factorization Method for Point of Interest Recommendation

Predicting Search Satisfaction Metrics with Interleaved Comparisons

Different Users, Different Opinions: Predicting Search Satisfaction with Mouse Movement Information

Personalizing Search on Shared Devices

Impact of Surrogate Assessments on High-Recall Retrieval

A Probabilistic Model for Information Retrieval Based on Maximum Value Distribution

Assessor Differences and User Preferences in Tweet Timeline Generation

Scientific Information Understanding via Open Educational Resources (OER)

Evaluating Streams of Evolving News Events

From Queries to Cards: Re-ranking Proactive Card Recommendations Based on Reactive Search History

StarSum: A Simple Star Graph for Multi-document Summarization

VenueMusic: A Venue-Aware Music Recommender System

Cross-Platform Question Routing for Better Question Answering

Finding Money in the Haystack: Information Retrieval at Bloomberg

Incremental Sampling of Query Logs

Advanced Click Models and their Applications to IR: SIGIR 2015 Tutorial

Graph Search and Beyond: SIGIR 2015 Workshop Summary

Differences in the Use of Search Assistance for Tasks of Varying Complexity

Semi-supervised Hashing with Semantic Confidence for Large Scale Visual Search

High Quality Graph-Based Similarity Search

Learning Maximal Marginal Relevance Model via Directly Optimizing Diversity Evaluation Measures

adaQAC: Adaptive Query Auto-Completion via Implicit Negative Feedback

Local Ranking Problem on the BrowseGraph

Unconscious Physiological Effects of Search Latency on Users and Their Click Behaviour

Uncovering Crowdsourced Manipulation of Online Reviews

Mining, Ranking and Recommending Entity Aspects

Untangling Result List Refinement and Ranking Quality: a Framework for Evaluation and Prediction

Personalized Recommendation via Parameter-Free Contextual Bandits

BROOF: Exploiting Out-of-Bag Errors, Boosting and Random Forests for Effective Automated Classification

Context- and Content-aware Embeddings for Query Rewriting in Sponsored Search

Parametric and Non-parametric User-aware Sentiment Topic Models

GeoSoCa: Exploiting Geographical, Social and Categorical Correlations for Point-of-Interest Recommendations

Sequential Testing for Early Stopping of Online Experiments

Predicting Search Intent Based on Pre-Search Context

Leveraging User Reviews to Improve Accuracy for Mobile App Retrieval

The Benefits of Magnitude Estimation Relevance Assessments for Information Retrieval Evaluation

Non-Compositional Term Dependence for Information Retrieval

User Variability and IR System Evaluation

In Situ Insights

When Relevance Judgement is Happening?: An EEG-based Study

Shiny on Your Crazy Diagonal

Time Pressure in Information Search

If SIGIR had an Academic Track, What Would Be In It?

Where to Go on Your Next Trip?: Optimizing Travel Destinations Based on User Preferences

An Introduction to Click Models for Web Search: SIGIR 2015 Tutorial

SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR)

Search Engine Evaluation based on Search Engine Switching Prediction

CricketLinking: Linking Event Mentions from Cricket Match Reports to Ball Entities in Commentaries

Controversy Detection and Stance Analysis

WeChat Search & Headline: Sogou Joins Force with Tencent on Mobile Search

Bringing Order to the Job Market: Efficient Job Offer Categorization in E-Recruitment

IR Evaluation: Modeling User Behavior for Measuring Effectiveness

SIGIR 2015 Workshop on Temporal, Social and Spatially-aware Information Access (#TAIA2015)

Time-Aware Authorship Attribution for Short Text Streams

An Aspect-driven Social Media Explorer

Using Contextual Information to Understand Searching and Browsing Behavior

Structure, Personalization, Scale: A Deep Dive into LinkedIn Search

Information Retrieval with Verbose Queries

NeuroIR 2015: Neuro-Physiological Methods in IR Research

A Priori Relevance Based On Quality and Diversity Of Social Signals

ERICA: Expert Guidance in Validating Crowd Answers

Transfer Learning for Information Retrieval

Location in Search

Revisiting the Foundations of IR: Timeless, Yet Timely

SPS'15: 2015 International Workshop on Social Personalization & Search

Document Comprehensiveness and User Preferences in Novelty Search Tasks

Large-scale Image Retrieval using Neural Net Descriptors

Enhancing Mathematics Information Retrieval

Challenges and Opportunities in Online Evaluation of Search Engines

IR Evaluation: Designing an End-to-End Offline Evaluation Pipeline

Privacy-Preserving IR 2015: When Information Retrieval Meets Privacy and Security

Cost-Aware Result Caching for Meta-Search Engines

Galean: Visualization of Geolocated News Events from Social Media

Improving Search using Proximity-Based Statistics

Lower Search Cost

Music Retrieval and Recommendation: A Tutorial Overview

From Unlabelled Tweets to Twitter-specific Opinion Words

SciNet: Interactive Intent Modeling for Information Discovery

Spoken Conversational Search: Information Retrieval over a Speech-only Communication Channel

Exploiting Wikipedia for Information Retrieval Tasks

The Best Published Result is Random: Sequential Testing and its Effect on Reported Effectiveness

Linse: A Distributional Semantics Entity Search Engine

Finding Answers in Web Search

Load-sensitive CPU Power Management for Web Search Engines

Online News Tracking for Ad-Hoc Queries

Retrieval from Noisy E-Discovery Corpus in the Absence of Training Data

DUMPLING: A Novel Dynamic Search Engine

Opinion Spammer Detection in Web Forum

Multi-Faceted Recall of Continuous Active Learning for Technology-Assisted Review

Time Pressure and System Delays in Information Search

How Random Decisions Affect Selective Distributed Search

Comparing Approaches for Query Autocompletion

Sign-Aware Periodicity Metrics of User Engagement for Online Search Quality Evaluation

Modelling Term Dependence with Copulas

Modeling Website Topic Cohesion at Scale to Improve Webpage Classification

Topic-centric Classification of Twitter User's Political Orientation

Word Embedding based Generalized Language Model for Information Retrieval

A Head-Weighted Gap-Sensitive Correlation Coefficient

On Term Selection Techniques for Patent Prior Art Search

Automatic Feature Generation on Heterogeneous Graph for Music Recommendation

Differences in Eye-Tracking Measures Between Visits and Revisits to Relevant and Irrelevant Web Pages

Reducing Hubness: A Cause of Vulnerability in Recommender Systems

Modularity-Based Query Clustering for Identifying Users Sharing a Common Condition

Understanding Temporal Query Intent

On the Reusability of Open Test Collections

Towards Vandalism Detection in Knowledge Bases: Corpus Construction and Analysis

About the 'Compromised Information Need' and Optimal Interaction as Quality Measure for Search Interfaces

I See You: Person-of-Interest Search in Social Networks

Towards Quantifying the Impact of Non-Uniform Information Access in Collaborative Information Retrieval

Features of Disagreement Between Retrieval Effectiveness Measures

Subsequence Search in Event-Interval Sequences

Searcher in a Strange Land: Understanding Web Search from Familiar and Unfamiliar Locations

Evaluating Retrieval Models through Histogram Analysis

Inter-Category Variation in Location Search

Reachability based Ranking in Interactive Image Retrieval

Modeling Multi-query Retrieval Tasks Using Density Matrix Transformation

Predicting User Behavior in Display Advertising via Dynamic Collective Matrix Factorization

Zero-shot Image Tagging by Hierarchical Semantic Embedding

Using Term Location Information to Enhance Probabilistic Information Retrieval

Learning Context-aware Latent Representations for Context-aware Collaborative Filtering

Exploiting User and Business Attributes for Personalized Business Recommendation

Speeding up Document Ranking with Rank-based Features

Mining Measured Information from Text

An Initial Investigation into Fixed and Adaptive Stopping Strategies

Regularised Cross-Modal Hashing

Adapted B-CUBED Metrics to Unbalanced Datasets

A Time-aware Random Walk Model for Finding Important Documents in Web Archives

A Test Collection for Spoken Gujarati Queries

Discovering Experts across Multiple Domains

Using Key Concepts in a Translation Model for Retrieval

On the Cost of Phrase-Based Ranking

Location-Aware Model for News Events in Social Media

Exploring Opportunities to Facilitate Serendipity in Search

Combining Orthogonal Information in Large-Scale Cross-Language Information Retrieval

Tailoring Music Recommendations to Users by Considering Diversity, Mainstreaminess, and Novelty

Challenges of Mathematical Information Retrievalin the NTCIR-11 Math Wikipedia Task

Probabilistic Multileave for Online Retrieval Evaluation

Twitter Sentiment Analysis with Deep Convolutional Neural Networks

Anchoring and Adjustment in Relevance Estimation

Cognitive Activity during Web Search

Personalized Semantic Ranking for Collaborative Recommendation

Active Learning for Entity Filtering in Microblog Streams

Relevance-aware Filtering of Tuples Sorted by an Attribute Value via Direct Optimization of Search Quality Metrics

Multi-source Information Fusion for Personalized Restaurant Recommendation

Joint Matrix Factorization and Manifold-Ranking for Topic-Focused Multi-Document Summarization

Towards Understanding the Impact of Length in Web Search Result Summaries over a Speech-only Communication Channel

Early Detection of Topical Expertise in Community Question Answering

LBMCH: Learning Bridging Mapping for Cross-modal Hashing

Gibberish, Assistant, or Master?: Using Tweets Linking to News for Extractive Single-Document Summarization

Context-aware Point-of-Interest Recommendation Using Tensor Factorization with Social Regularization

Adaptive User Engagement Evaluation via Multi-task Learning

Compact Snippet Caching for Flash-based Search Engines

When Personalization Meets Conformity: Collective Similarity based Multi-Domain Recommendation

Sub-document Timestamping of Web Documents

Splitting Water: Precision and Anti-Precision to Reduce Pool Bias

Content Provider	ACM Digital Library
Author	Hanbury, Allan Lupu, Mihai Lipani, Aldo
Abstract	For many tasks in evaluation campaigns, especially those modeling narrow domain-specific challenges, lack of participation leads to a potential pooling bias due to the scarce number of pooled runs. It is well known that the reliability of a test collection is proportional to the number of topics and relevance assessments provided for each topic, but also to same extent to the diversity in participation in the challenges. Hence, in this paper we present a new perspective in reducing the pool bias by studying the effect of merging an unpooled run with the pooled runs. We also introduce an indicator used by the bias correction method to decide whether the correction needs to be applied or not. This indicator gives strong clues about the potential of a "good" run tested on an "unfriendly" test collection (i.e. a collection where the pool was contributed to by runs very different from the one at hand). We demonstrate the correctness of our method on a set of fifteen test collections from the Text REtrieval Conference (TREC). We observe a reduction in system ranking error and absolute score difference error.
Starting Page	103
Ending Page	112
Page Count	10
File Format	PDF
ISBN	9781450336215
DOI	10.1145/2766462.2767749
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2015-08-09
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Measurement Evaluation Bias Experimentation Pool Test collection Trec Performance
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in