Loading...
Please wait, while we are loading the content...
Similar Documents
Measuring the Reusability of Test Collections
| Content Provider | CiteSeerX |
|---|---|
| Author | Josifovski, Vanja Gabrilovich, Evgeniy Carterette, Ben Metzler, Donald |
| Abstract | While test collection construction is a time-consuming and expensive process, the true cost is amortized by reusing the collection over hundreds or thousands of experiments. Some of these experiments may involve systems that retrieve documents not judged during the initial construction phase, and some of these systems may be “hard ” to evaluate: depending on which judgments are missing and which judged documents were retrieved, the experimenter’s confidence in an evaluation could potentially be very low. We propose two methods for quantifying the reusability of a test collection for evaluating new systems. The proposed methods provide simple yet highly effective tests for determining whether an existing set of judgments is useful for evaluating a new system. Empirical evaluations using TREC datasets confirm the usefulness of our proposed reusability measures. In particular, we show that our methods can reliably estimate confidence intervals that are indicative of collection reusability. |
| File Format | |
| Access Restriction | Open |
| Subject Keyword | Collection Reusability Initial Construction Phase Reusability Measure Trec Datasets Collection Construction Empirical Evaluation Effective Test Expensive Process True Cost Experimenter Confidence Test Collection Confidence Interval New System |
| Content Type | Text |
