NDLI: Ranking from pairs and triplets: information quality, evaluation methods and query complexity

Please wait, while we are loading the content...

Crowdsourcing for search and data mining

Crowdsourcing 101: putting the WSDM of crowds to work for you

Exploiting statistical and relational information on the web and in social media

Mining billion-node graphs: patterns, generators and tools

Who uses web search for what: and how

#TwitterSearch: a comparison of microblog search and web search

A comparative analysis of cascade measures for novelty and diversity

Bing dialog model: intent, knowledge and user interaction

We feel fine and searching the emotional web

Efficient indexing of repeated n-grams

Understanding temporal query dynamics

Dynamic relationship and event discovery

Dynamic ranked retrieval

Recommender systems with social regularization

Topical semantics of twitter links

Transient crowd discovery on the real-time social web

User modeling for web applications

Introduction to display advertising: a half-day tutorial

Web retrieval: the role of users

Personalizing web search using long term browsing history

Identifying topical authorities in microblogs

Understanding and predicting personal navigation

On the selection of tags for tag clouds

Batch query processing for web search engines

Patterns of temporal variation in online media

Joint training for open-domain extraction on the web: exploiting overlap when supervision is limited

Optimizing two-dimensional search results presentation

Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms

Evaluating the visual quality of web pages using a computational aesthetic approach

Adaptive bootstrapping of recommender systems using decision trees

Correcting for missing data in information cascades

Quality-biased ranking of web documents

Detecting duplicate web documents using clickthrough data

Using graded-relevance metrics for evaluating community QA answer selection

Scalable knowledge harvesting with high precision and high recall

Result enrichment in commerce search using browse trails

Efficient online ad serving in a display advertising exchange

Clustering product features for opinion mining

Predicting future reviews: sentiment analysis models for collaborative filtering

Everyone's an influencer: quantifying influence on twitter

Ranking from pairs and triplets: information quality, evaluation methods and query complexity

KMV-peer: a robust and adaptive peer-selection algorithm

Mining social images with distance metric learning for automated image tagging

Mining named entities with temporally correlated bursts from multilingual web news streams

Identifying task-based sessions in search engine query logs

Trend analysis model: trend consists of temporal words, topics, and timestamps

Materializing multi-relational databases from the web using taxonomic queries

Learning similarity function for rare queries

What blogs tell us about websites: a demographics study

A two-view learning approach for image tag ranking

Cross lingual text classification by mining multilingual topics from wikipedia

Supervised random walks: predicting and recommending links in social networks

Multidimensional mining of large-scale search logs: a topic-concept cube approach

OOLAM: an opinion oriented link analysis model for influence persona discovery

Optimizing merchant revenue with rebates

eBay: an E-commerce marketplace as a complex network

Searchable web sites recommendation

A framework for quantitative analysis of cascades on networks

Inferring search behaviors using partially observable markov model with duration (POMD)

Scalable clustering of news search results

Learning website hierarchies for keyword enrichment in contextual advertising

Large-scale hierarchical text classification without labelled data

Action prediction and identification from mining temporal user behaviors

Low-order tensor decompositions for social tagging recommendation

Collective extraction from heterogeneous web lists

Shopping for products you don't know you need

CMAP: effective fusion of quality and relevance for multi-criteria recommendation

On composition of a federated web search result page: using online users to provide pairwise preference for heterogeneous verticals

CoBayes: bayesian knowledge corroboration with assessors of unknown areas of expertise

Stochastic query covering

Multi-dimensional search result diversification

Learning to re-rank web search results with multiple pairwise features

Improving social bookmark search using personalised latent variable language models

The tube over time: characterizing popularity growth of youtube videos

Strength of social influence in trust networks in product review sites

Citation recommendation without author supervision

A combined topical/non-topical approach to identifying web sites for children

Query suggestion for E-commerce sites

Bid generation for advanced match in sponsored search

An algorithmic treatment of strong queries

Let web spammers expose themselves

Enhanced email spam filtering through combining similarity graphs

Efficient entity resolution for large heterogeneous information spaces

LambdaMerge: merging the results of query reformulations

Web-scale table census and classification

Normalizing web product attributes and discovering domain ontology with minimal effort

A probabilistic approach for learning folksonomies from structured data

Aspect and sentiment unification model for online review analysis

Linking online news and social media

Searching patterns for relation extraction over the web: rediscovering the pattern-relation duality

Document assignment in multi-site search engines

Ranking from pairs and triplets: information quality, evaluation methods and query complexity

Content Provider	ACM Digital Library
Author	Radinsky, Kira Ailon, Nir
Abstract	Obtaining judgments from human raters is a vital part in the design of search engines' evaluation. Today, a discrepancy exists between judgment acquisition from raters (training phase) and use of the responses for retrieval evaluation (evaluation phase). This discrepancy is due to the inconsistency between the representation of the information in both phases. During training, raters are requested to provide a relevance score for an individual result in the context of a query, whereas the evaluation is performed on ordered lists of search results, with the results' relative position (compared to other results) taken into account. As an alternative to the practice of learning to rank using relevance judgments for individual search results, more and more focus has recently been diverted to the theory and practice of learning from answers to combinatorial questions about sets of search results. That is, users, during training, are asked to rank small sets (typically pairs). Human rater responses to questions about the relevance of individual results are first compared to their responses to questions about the relevance of pairs of results. We empirically show that neither type of response can be deduced from the other, and that the added context created when results are shown together changes the raters' evaluation process. Since pairwise judgments are directly related to ranking, we conclude they are more accurate for that purpose. We go beyond pairs to show that triplets do not contain significantly more information than pairs for the purpose of measuring statistical preference. These two results establish good stability properties of pairwise comparisons for the purpose of learning to rank. We further analyze different scenarios, in which results of varying quality are added as "decoys". A recurring source of worry in papers focusing on pairwise comparison is the quadratic number of pairs in a set of results. Which preferences do we choose to solicit from paid raters? Can we provably eliminate a quadratic cost? We employ results from statistical learning theory to show that the quadratic cost can be provably eliminated in certain cases. More precisely, we show that in order to obtain a ranking in which each element is an average of O(n/C) positions away from its position in the optimal ranking, one needs to sample $O(nC^{2})$ pairs uniformly at random, for any C > 0. We also present an active learning algorithm which samples the pairs adaptively, and conjecture that it provides additional improvement.
Starting Page	105
Ending Page	114
Page Count	10
File Format	PDF QT / MOV MP4
ISBN	9781450304931
DOI	10.1145/1935826.1935850
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2011-02-09
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Relevance feedback Ranking from pairs Ranking evaluation
Content Type	Audio Video Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in