NDLI: Hypothesis testing for the risk-sensitive evaluation of retrieval systems

Please wait, while we are loading the content...

Putting searchers into search

Modelling interaction with economic models of search

Temporal feedback for tweet search with non-parametric density estimation

Gaussian process factorization machines for context-aware recommendations

Context-aware web search abandonment prediction

Circumlocution in diagnostic medical queries

Supervised hashing with latent factor models

Hierarchical multi-label classification of social text streams

Predictive parallelization: taming tail latencies in web search

Learning for search result diversification

Seeking simplicity in search user interfaces

Who is the barbecue king of texas?: a geo-spatial approach to finding local experts on twitter

Entity query feature expansion using knowledge base links

Discriminative coupled dictionary hashing for fast cross-media retrieval

Predicting term-relevance from brain signals

Learning user reformulation behavior for query auto-completion

On measuring social friend interest similarities in recommender systems

User-driven system-mediated collaborative information retrieval

The data revolution: how companies are transforming with big data

Learning similarity functions for topic detection in online reputation monitoring

Using information scent and need for cognition to understand online search behavior

Win-win search: dual-agent stochastic game in session search

Coarse-to-fine review selection via supervised joint aspect and sentiment model

New and improved: modeling versions to improve app recommendation

Query expansion for mixed-script information retrieval

The knowing camera 2: recognizing and annotating places-of-interest in smartphone photos

Time-critical search

CTSUM: extracting more certain summaries for news articles

CiteSight: supporting contextual citation recommendation using differential search

Search result diversification via data fusion

Relevation!: An open source system for information retrieval relevance assessment

Cluster links prediction for literature based discovery using latent structure and semantic features

Choices and constraints: research goals and approaches in information retrieval (part 1)

ERD'14: entity recognition and disambiguation challenge

Query-performance prediction: setting the expectations straight

Fine-grained location extraction from tweets with temporal awareness

Addressing cold start in recommender systems: a semi-supervised co-training algorithm

Impact of response latency on user behavior in web search

Interactions between health searchers and search engines

Preference preserving hashing for efficient recommendation

An adaptive teleportation random walk model for learning social tag relevance

Skewed partial bitvectors for list intersection

Fusion helps diversification

Your neighbors affect your ratings: on geographical neighborhood influence to rating prediction

QUADS: question answering for decision support

Active hashing with joint data example and tag selection

Multidimensional relevance modeling via psychometrics and crowdsourcing

A two-dimensional click model for query auto-completion

IMRank: influence maximization via finding self-consistent ranking

SearchPanel: framing complex search needs

Predicting trending messages and diffusion participants in microblogging network

Discrimination between tasks with user activity patterns during information search

Injecting user models and time into precision via Markov chains

Cross-domain and cross-category emotion tagging for comments of online news

Bundle recommendation in ecommerce

Retrieval of similar chess positions

Click-through-based cross-view learning for image search

Learning temporal-dependent ranking models

Continuous word embeddings for detecting local text reuses at the semantic level

Cross-language context-aware citation recommendation in scientific articles

Hashtag recommendation for hyperlinked tweets

WenZher: comprehensive vertical search for healthcare domain

Graph-based large scale RDF data compression

Choices and constraints: research goals and approaches in information retrieval (part 2)

SIGIR 2014 workshop on gathering efficient assessments of relevance (GEAR)

Hypothesis testing for the risk-sensitive evaluation of retrieval systems

Collaborative personalized Twitter search with topic-language models

Explicit factor models for explainable recommendation based on phrase-level sentiment analysis

Towards better measurement of attention and satisfaction in mobile search

Evaluation of machine-learning protocols for technology-assisted review in electronic discovery

Load balancing for partition-based similarity search

Predicting the popularity of web 2.0 items based on user comments

Partitioned Elias-Fano indexes

Utilizing relevance feedback in fusion-based retrieval

Processing spatial keyword query as a top-k aggregation query

Topic labeled text classification: a weakly supervised approach

Latent semantic sparse hashing for cross-modal similarity search

Cohort modeling for enhanced personalized search

Leveraging knowledge across media for spammer detection in microblogging

Investigating users' query formulations for cognitive search intents

Searching, browsing, and clicking in a search session: changes in user behavior by task and over time

Economically-efficient sentiment stream analysis

Does product recommendation meet its waterloo in unexplored categories?: no, price comes to help

A mathematics retrieval system for formulae in layout presentations

Learning to personalize trending image search suggestion

Web page segmentation with structured prediction and its application in web page classification

Personalized document re-ranking based on Bayesian probabilistic matrix factorization

STICS: searching with strings, things, and cats

Entity-based retrieval

Scalability and efficiency challenges in large-scale web search engines

MedIR14: medical information retrieval workshop

Modeling action-level satisfaction for search task satisfaction prediction

ReQ-ReC: high recall retrieval with query pooling and interactive classification

Estimating global statistics for unstructured P2P search in the presence of adversarial peers

Recommending social media content to community owners

Principled dictionary pruning for low-memory corpus compression

A simple term frequency transformation model for effective pseudo relevance feedback

Characterizing multi-click search behavior and the risks and opportunities of changing results during use

PRISM: concept-preserving social image search results summarization

Query log driven web search results clustering

Using the cross-entropy method to re-rank search results

VIRLab: a web-based virtual lab for learning and studying information retrieval models

Improving offline and online web search evaluation by modelling the user behaviour

Statistical significance testing in information retrieval: theory and practice

Privacy-preserving IR: when information retrieval meets privacy and security

Computing and applying topic-level user interactions in microblog recommendation

ServiceXplorer: a similarity-based web service search engine

Modelling of terms across scripts through autoencoders

Speech search: techniques and tools for spoken content retrieval

SIGIR 2014 workshop on semantic matching in information retrieval

Towards context-aware search with right click

Real-time visualization and targeting of online visitors

A tag-based personalized item recommendation system using tensor modeling and topic model approaches

Axiomatic analysis and optimization of information retrieval models

SoMeRA 2014: social media retrieval and analysis workshop

Rendering expressions to improve accuracy of relevance assessment for math search

CharBoxes: a system for automatic discovery of character infoboxes from books

Novelty and diversity enhancement and evaluation in recommender systems and information retrieval

A general account of effectiveness metrics for information tasks: retrieval, filtering, and clustering

SIGIR 2014 workshop on temporal, social and spatially-aware information access (#TAIA2014)

Exploring recommendations in internet of things

ADAM: a system for jointly providing ir and database queries in large-scale multimedia retrieval

Enrichment of user profiles across multiple online social networks for volunteerism matching for social enterprise

Dynamic information retrieval modeling

Sig-SR: SimRank search over singular graphs

NicePic!: a system for extracting attractive photos from flickr streams

The retrievability of documents

Old dogs are great at new tricks: column stores for ir prototyping

A perspective-aware approach to search: visualizing perspectives in news search results

The role of network distance in linkedin people search

FitYou: integrating health profiles to real-time contextual suggestion

Latent community discovery through enterprise user search query modeling

Semantic full-text search with broccoli

Examining collaborative query reformulation: a case of travel information searching

Just-for-me: an adaptive personalization system for location-aware social music recommendation

Influential nodes selection: a data reconstruction perspective

A novel system for the semi automatic annotation of event images

A fusion approach to cluster labeling

An interactive interface for visualizing events on Twitter

Evaluating the effort involved in relevance assessments for images

ExperTime: tracking expertise over time

Diversifying query suggestions based on query documents

Comparing client and server dwell time estimates for click-level satisfaction prediction

Score-safe term-dependency processing with hybrid indexes

Co-training on authorship attribution with very fewlabeled examples: methods vs. views

Probabilistic text modeling with orthogonalized topics

Evaluating non-deterministic retrieval systems

Extending test collection pools without manual runs

The search duel: a response to a strong ranker

Modeling the evolution of product entities

Predicting bursts and popularity of hashtags in real-time

Probabilistic ensemble learning for vietnamese word segmentation

Improving unsupervised query segmentation using parts-of-speech sequence information

Building a query log via crowdsourcing

Web search without 'stupid' results

Discovering real-world use cases for a multimodal math search interface

Improving search personalisation with dynamic group formation

Detection of abnormal profiles on group attacks in recommender systems

On run diversity in Evaluation as a Service

Evaluating answer passages using summarization measures

Analyzing bias in CQA-based expert finding test sets

Understanding negation and family history to improve clinical information retrieval

Modeling dual role preferences for trust-aware recommendation

Mouse movement during relevance judging: implications for determining user attention

PAAP: prefetch-aware admission policies for query results cache in web search engines

User geospatial context for music recommendation in microblogs

Compositional data analysis (CoDA) approaches to distance in information retrieval

Group latent factor model for recommendation with multiple user behaviors

Hashing with List-Wise learning to rank

A burstiness-aware approach for document dating

An analysis of query difficulty for information retrieval in the medical domain

Mobile query reformulations

On peculiarities of positional effects in sponsored search

A collective topic model for milestone paper discovery

Document summarization based on word associations

Do users rate or review?: boost phrase-level sentiment labeling with review-level sentiment classification

Random subspace for binary codes learning in large scale image retrieval

Incorporating query-specific feedback into learning-to-rank models

Large-scale author verification: temporal and topical influences

Evaluating mobile web search performance by taking good abandonment into account

Assessing the reliability and reusability of an E-discovery privilege test collection

Modeling evolution of a social network using temporalgraph kernels

On user interactions with query auto-completion

Re-ranking approach to classification in large-scale power-law distributed category systems

Enhancing personalization via search activity attribution

A syntax-aware re-ranker for microblog retrieval

Weighted aspect-based collaborative filtering

Evaluating intuitiveness of vertical-aware click models

Recipient recommendation in enterprises using communication graphs and email content

Analyzing the content emphasis of web search engines

Effects of task and domain on searcher attention

Learning sufficient queries for entity filtering

PatentLine: analyzing technology evolution on multi-view patent graphs

Query performance prediction for entity retrieval

Second order probabilistic models for within-document novelty detection in academic articles

Modeling the dynamics of personal expertise

An annotation similarity model in passage ranking for historical fact validation

To hint or not: exploring the effectiveness of search hints for complex informational tasks

The effect of sampling strategy on inferred measures

Cache-conscious runtime optimization for ranking ensembles

Bridging temporal context gaps using time-aware re-contextualization

An enhanced context-sensitive proximity model for probabilistic information retrieval

On the information difference between standard retrieval models

A POMDP model for content-free document re-ranking

Using score differences for search result diversification

TREC: topic engineering exercise

How k-12 students search for learning?: analysis of an educational search engine log

The correlation between cluster hypothesis tests and the effectiveness of cluster-based retrieval

The effect of expanding relevance judgements with duplicates

On correlation of absence time and search effectiveness

Necessary and frequent terms in queries

Extracting topics based on authors, recipients and content in microblogs

Exploiting Twitter and Wikipedia for the annotation of event images

Learning to translate queries for CLIR

Predicting query performance in microblog retrieval

An event extraction model based on timeline and user analysis in Latent Dirichlet allocation

What makes data robust: a data analysis in learning to rank

Learning to bridge colloquial and formal language applied to linking and search of E-Commerce data

Uncovering the unarchived web

Inferring topic-dependent influence roles of Twitter users

Reputation analysis with a ranked sentiment-lexicon

On predicting religion labels in microblogging networks

Efficiently identify local frequent keyword co-occurrence patterns in geo-tagged Twitter stream

Item group based pairwise preference learning for personalized ranking

Where not to go?: detecting road hazards using twitter

Enhancing sketch-based sport video retrieval by suggesting relevant motion paths

Dynamic location models

Wikipedia-based query performance prediction

A revisit to social network-based recommender systems

Hypothesis testing for the risk-sensitive evaluation of retrieval systems

Content Provider	ACM Digital Library
Author	Dinçer, B. Taner Ounis, Iadh Macdonald, Craig
Abstract	The aim of risk-sensitive evaluation is to measure when a given information retrieval (IR) system does not perform worse than a corresponding baseline system for any topic. This paper argues that risk-sensitive evaluation is akin to the underlying methodology of the Student's t test for matched pairs. Hence, we introduce a risk-reward tradeoff measure TRisk that generalises the existing URisk measure (as used in the TREC 2013 Web track's risk-sensitive task) while being theoretically grounded in statistical hypothesis testing and easily interpretable. In particular, we show that TRisk is a linear transformation of the t statistic, which is the test statistic used in the Student's t test. This inherent relationship between TRisk and the t statistic, turns risk-sensitive evaluation from a descriptive analysis to a fully-fledged inferential analysis. Specifically, we demonstrate using past TREC data, that by using the inferential analysis techniques introduced in this paper, we can (1) decide whether an observed level of risk for an IR system is statistically significant, and thereby infer whether the system exhibits a real risk, and (2) determine the topics that individually lead to a significant level of risk. Indeed, we show that the latter permits a state-of-the-art learning to rank algorithm (LambdaMART) to focus on those topics in order to learn effective yet risk-averse ranking systems.
Starting Page	23
Ending Page	32
Page Count	10
File Format	PDF
ISBN	9781450322577
DOI	10.1145/2600428.2609625
Language	English
Publisher	Association for Computing Machinery (ACM)
Publisher Date	2014-07-03
Publisher Place	New York
Access Restriction	Subscribed
Subject Keyword	Student's t test Risk-sensitive evaluation
Content Type	Text
Resource Type	Article

Central Library (ISO-9001:2015 Certified)
Indian Institute of Technology Kharagpur
Kharagpur, West Bengal, India | PIN - 721302

See location in the Map
03222 282435
Mail: support@ndl.gov.in

Sl.	Authority	Responsibilities	Communication Details
1	Ministry of Education (GoI), Department of Higher Education	Sanctioning Authority	https://www.education.gov.in/ict-initiatives
2	Indian Institute of Technology Kharagpur	Host Institute of the Project: The host institute of the project is responsible for providing infrastructure support and hosting the project	https://www.iitkgp.ac.in
3	National Digital Library of India Office, Indian Institute of Technology Kharagpur	The administrative and infrastructural headquarters of the project	Dr. B. Sutradhar bsutra@ndl.gov.in
4	Project PI / Joint PI	Principal Investigator and Joint Principal Investigators of the project	Dr. B. Sutradhar bsutra@ndl.gov.in Prof. Saswat Chakrabarti will be added soon
5	Website/Portal (Helpdesk)	Queries regarding NDLI and its services	support@ndl.gov.in
6	Contents and Copyright Issues	Queries related to content curation and copyright issues	content@ndl.gov.in
7	National Digital Library of India Club (NDLI Club)	Queries related to NDLI Club formation, support, user awareness program, seminar/symposium, collaboration, social media, promotion, and outreach	clubsupport@ndl.gov.in
8	Digital Preservation Centre (DPC)	Assistance with digitizing and archiving copyright-free printed books	dpc@ndl.gov.in
9	IDR Setup or Support	Queries related to establishment and support of Institutional Digital Repository (IDR) and IDR workshops	idr@ndl.gov.in