NDLI: Quantifying test collection quality based on the consistency of relevance judgements

Please wait, while we are loading the content...

Future of the web and search

Beyond search: statistical topic models for text analysis

Modeling and analysis of cross-session search tasks

People searching for people: analysis of a people search engine log

Bagging gradient-boosted trees for high precision, low variance ranking models

SCENE: a scalable two-stage personalized news recommendation system

CRTER: using cross terms to enhance probabilistic information retrieval

Who should share what?: item-level social influence prediction for users and posts ranking

A site oriented method for segmenting web pages

Social context summarization

Collaborative competitive filtering: learning recommender using context of user choice

Why searchers switch: understanding and predicting engine switching rationales

CrowdLogging: distributed, private, and anonymous search logging

Predicting web searcher satisfaction with existing community-based answers

Authorship classification: a discriminative syntactic tree mining approach

Hypergeometric language models for republished article finding

Integrating hierarchical feature selection and classifier training for multi-label image annotation

Temporal index sharding for space-time efficiency in archive search

Incremental diversification for very large sets: a streaming-based approach

Handling data sparsity in collaborative filtering using emotion and semantic based features

ILDA: interdependent LDA model for learning latent aspects and their ratings from online product reviews

Multimedia answering: enriching text QA with media information

Enhanced results for web search

Ranking related news predictions

Query suggestions in the absence of query logs

Improved video categorization from text metadata and user comments

Document clustering with universum

System effectiveness, user models, and user utility: a conceptual framework for investigation

Cross-language web page classification via dual knowledge transfer using nonnegative matrix tri-factorization

Posting list intersection on multicore architectures

Utilizing marginal net utility for recommendation in e-commerce

Evaluating diversified search results using per-intent graded relevance

Parallel learning to rank for information retrieval

ClusteringWiki: personalized and collaborative clustering of search results

Machine learning for information retrieval

Persistence in the ephemeral: utilizing repeat behaviors for multi-session personalized search

Sensor-aided mobile information management and retrieval

The economics in interactive information retrieval

Learning search tasks in queries and web pages via graph regularization

Learning to rank for freshness and relevance

Inferring and using location metadata to personalize web search

A boosting approach to improving pseudo-relevance feedback

Mining tags using social endorsement networks

Composite hashing with multiple information sources

Probabilistic factor models for web site recommendation

CLR: a collaborative location recommendation framework based on co-clustering

Find it if you can: a game for modeling different types of web search success using interaction data

Out of sight, not out of mind: on the effect of social and physical detachment on information need

Competition-based user expertise score estimation

On theme location discovery for travelogue services

Estimation methods for ranking recent information

Efficient manifold ranking for image retrieval

Inverted indexes for phrases and strings

Intent-aware search result diversification

Fast context-aware recommendations with factorization machines

Clickthrough-based latent semantic models for web search

Enhancing multi-label music genre classification through ensemble techniques

Summarizing the differences in multilingual news

Collective entity linking in web text: a graph-based method

Synthesizing high utility suggestions for rare web search queries

Multifaceted toponym recognition for streaming news

Identifying points of interest by self-tuning clustering

Evaluating the synergic effect of collaboration in information seeking

No free lunch: brute force vs. locality-sensitive hashing for cross-lingual pairwise similarity

Timestamp-based result cache invalidation for web search engines

Recommending ephemeral items at web scale

Evaluating multi-query sessions

Learning features through feedback for blog distillation

OrientSTS: spatio-temporal sequence searching in flickr

Enhancing web search by mining search and browse logs

Search engines that learn online

Predicting eBay listing conversion

Seeding simulated queries with user-study data for personal search evaluation

Intentions and attention in exploratory health search

A cascade ranking model for efficient ranked retrieval

Active learning to maximize accuracy vs. effort in interactive information retrieval

Enhancing ad-hoc relevance weighting using probability density estimation

Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking

Detecting outlier sections in us congressional legislation

Efficiently collecting relevance information from clickthroughs for web retrieval system evaluation

Functional matrix factorizations for cold-start recommendation

Measuring improvement in user search performance resulting from optimal search tips

Scalable multi-dimensional user intent identification using tree structured distributions

Learning online discussion structures by conditional random fields

Effective sentiment stream analysis with self-augmenting training and demand-driven projection

Query by document via a decomposition-based two-level retrieval approach

Mining weakly labeled web facial images for search-based face annotation

Faster temporal range queries over versioned text

Parameterized concept weighting in verbose queries

Filtering semi-structured documents based on faceted feedback

Regularized latent semantic indexing

Picasso - to sing, you must close your eyes and draw

Evolutionary timeline summarization: a balanced optimization framework via iterative substitution

From one tree to a forest: a unified solution for structured web data extraction

Post-ranking query suggestion by diversifying search results

Enriching document representation via translation for improved monolingual information retrieval

Cluster-based fusion of retrieved lists

Repeatable and reliable search system evaluation using crowdsourcing

An event-centric model for multilingual document similarity

Energy-price-driven query processing in multi-center web search engines

A unified framework for recommendations based on quaternary semantic analysis

Quantifying test collection quality based on the consistency of relevance judgements

Time-based relevance models

A toolkit for knowledge base population

A new look at old tricks: the fertile roots of current research

Query expansion based on a semantic graph model

A large scale machine learning system for recommending heterogeneous content in social networks

Understanding re-finding behavior in naturalistic email interaction logs

User behavior in zero-recall ecommerce queries

Relevant knowledge helps in choosing right teacher: active query selection for ranking adaptation

DOM based content extraction via text density

Unsupervised query segmentation using clickthrough for information retrieval

Exploiting geographical influence for collaborative point-of-interest recommendation

ViewSer: enabling large-scale remote user studies of web search examination and interaction

Social annotation in query expansion: a machine learning approach

Mining topics on participations for community discovery

Indexing strategies for graceful degradation of search quality

UPS: efficient privacy protection in personalized web search

Learning relevance from heterogeneous social network and its application in online targeting

Improving local search ranking through external logs

Automatic boolean query suggestion for professional search

A novel corpus-based stemming algorithm using co-occurrence statistics

Faster top-k document retrieval using block-max indexes

Associative tag recommendation exploiting multiple textual features

Pseudo test collections for learning web search ranking functions

Improved query performance prediction using standard deviation

iMecho: a context-aware desktop search system

Crowdsourcing for information retrieval: principles, methods, and applications

Descriptive modelling of text classification and its integration with other IR tasks

Review of MSR-Bing web scale speller challenge

Learning to rank using query-level regression

Visualizing and querying semantic social networks

Practical online retrieval evaluation

Efficient and effective solutions for search engines

Elsevier SIGIR 2011 application challenge abstract

Diversifying product search results

What-you-retrieve-is-what-you-see: a preliminary cyber-physical search engine

Web retrieval: the role of users

Modeling document scores for distributed information retrieval

Ad hoc IR: not much room for improvement

QuickView: advanced search of tweets

Information organization and retrieval with collaboratively generated content

Improving query and result list adaptation in personalized multilingual information retrieval

Image annotation based on recommendation model

Personalized video: leanback online video consumption

Using k-Top retrieved web snippets to date temporalimplicit queries based on web content analysis

Utilizing minimal relevance feedback for ad hoc retrieval

GreenMeter: a tool for assessing the quality and recommending tags for web 2.0 applications

Domain-specific information retrieval using rcommenders

Sense discrimination for physics retrieval

JuSe: a picture dictionary query system for children

Understanding and using contextual information in recommender systems

When documents are very long, BM25 fails!

CrowdTracker: enabling community-based real-time web monitoring

Multidimensional search result diversification: diverse search results for diverse users

Location and timeliness of information sources during news events

The Meta-Dex Suite: generating and analyzing indexes and meta-indexes

What deliberately degrading search quality tells us about discount functions

Tulsa: web search for writing assistance

Collective topic modeling for heterogeneous networks

The TREC files: the (ground) truth is out there

Graph-cut based tag enrichment

A tool for comparative IR evaluation on component level

Personalized social query expansion using social bookmarking systems

What are the real differences of children's and adults' web search

Cognitive coordinating behaviors in multitasking web search

Optimizing multimodal reranking for web image search

Multi-layer graph-based semi-supervised learning for large-scale image datasets using mapreduce

Tackling class imbalance and data scarcity in literature-based gene function annotation

Bootstrapping subjectivity detection

The effects of choice in routing relevance judgments

Statistical feature extraction for cross-language web content quality assessment

Exploiting endorsement information and social influence for item recommendation

Modeling subset distributions for verbose queries

Domain expert topic familiarity and search behavior

Sample selection for dictionary-based corpus compression

Evaluating medical information retrieval

Region-based landmark discovery by crowdsourcing geo-referenced photos

Towards effective short text deep classification

Temporal latent semantic analysis for collaboratively generated content: preliminary results

Self-adjusting hybrid recommenders based on social network analysis

BlogCast effect on information diffusion in a blogosphere

Product comparison using comparative relations

Collaborative cyberporn filtering with collective intelligence

Do IR models satisfy the TDC retrieval constraint

On diversifying and personalizing web search

Semantic tag recommendation using concept model

Recommending interesting activity-related local entities

Cross-corpus relevance projection

Location disambiguation for geo-tagged images

Towards an indexing method to speed-up music retrieval

An investigation of decompounding for cross-language patent search

Detecting seasonal queries by time-series analysis

Learning to rank under tight budget constraints

A novel hybrid index structure for efficient text retrieval

A weighted curve fitting method for result merging in federated search

Effect of different docid orderings on dynamic pruning retrieval strategies

Time-based query performance predictors

Search task difficulty: the expected vs. the reflected

On the suitability of diversity metrics for learning-to-rank for diversity

How diverse are web search results?

Analysis of an expert search query log

A model for expert finding in social networks

Transductive learning over automatically detected themes for multi-document summarization

Rating-based collaborative filtering combined with additional regularization

Words-of-interest selection based on temporal motion coherence for video retrieval

Aggregating multiple opinion evidence in proximity-based opinion retrieval

Enhancing mobile search using web search log data

Award prediction with temporal citation network analysis

Rating prediction using feature words extracted from customer reviews

Ranking tags in resource collections

Identifying similar people in professional social networks with discriminative probabilistic models

Intent-oriented diversity in recommender systems

Disambiguating biomedical acronyms using EMIM

Best document selection based on approximate utility optimization

Forecasting counts of user visits for online display advertising with probabilistic latent class models

Knowledge effects on document selection in search results pages

Learning to rank from a noisy crowd

How to count thumb-ups and thumb-downs?: an information retrieval approach to user-rating based ranking of items

Predicting users' domain knowledge from search behaviors

The interactive PRP for diversifying document rankings

Detecting success in mobile search from interaction

Measuring assessor accuracy: a comparison of nist assessors and user study participants

A bipartite graph based social network splicing method for person name disambiguation

Link formation analysis in microblogs

Evolution of web search results within years

Decayed DivRank: capturing relevance, diversity and prestige in information networks

Multi-objective optimization in learning to rank

A large-scale study of the effect of training set characteristics over learning-to-rank algorithms

Exploring term temporality for pseudo-relevance feedback

MSSF: a multi-document summarization framework based on submodularity

SEJoin: an optimized algorithm towards efficient approximate string searches

Bag-of-visual-words vs global image descriptors on two-stage multimodal retrieval

Query term ranking based on search results overlap

Tossing coins to trim long queries

A comparison of time-aware ranking methods

Learning for graphs with annotated edges

Formulating effective questions for community-based question answering

Quantifying test collection quality based on the consistency of relevance judgements

Content Provider	ACM Digital Library
Author	Sanderson, Mark Scholer, Falk Turpin, Andrew
Abstract	Relevance assessments are a key component for test collection-based evaluation of information retrieval systems. This paper reports on a feature of such collections that is used as a form of ground truth data to allow analysis of human assessment error. A wide range of test collections are retrospectively examined to determine how accurately assessors judge the relevance of documents. Our results demonstrate a high level of inconsistency across the collections studied. The level of irregularity is shown to vary across topics, with some showing a very high level of assessment error. We investigate possible influences on the error, and demonstrate that inconsistency in judging increases with time. While the level of detail in a topic specification does not appear to influence the errors that assessors make, judgements are significantly affected by the decisions made on previously seen similar documents. Assessors also display an assessment inertia. Alternate approaches to generating relevance judgements appear to reduce errors. A further investigation of the way that retrieval systems are ranked using sets of relevance judgements produced early and late in the judgement process reveals a consistent influence measured across the majority of examined test collections. We conclude that there is a clear value in examining, even inserting, ground truth data in test collections, and propose ways to help minimise the sources of inconsistency when creating future test collections.
Starting Page	1063
Ending Page	1072
Page Count	10
File Format	PDF
ISBN	9781450307574
DOI	10.1145/2009916.2010057
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2011-07-24
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Information retrieval evaluation Search engines
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in