| Abstract |
The Cranfield projects began in 1958 -- fifty years ago. They have of course been extraordinarily influential, forming a view of information retrieval as an experimental science, which in some fashion persists to this day. Although the Cranfield tradition has had its ups and downs - the main down being in the late eighties, when it showed signs of being moribund - it revived and regained its status in the nineties with the start of TREC, and remains an extremely influential world-view of the field of search (as it is now commonly known) as we approach the second decade of the $21^{st}$ century. At the same time, the field itself has become vastly better known, and IR systems vastly more widely used, as a result of the spread of the world wide web and the development of the web search engine. But the Cranfield/TREC view of IR has always been partial and biased. In particular, it has very greatly encouraged and promoted research into models, methods and algorithms appropriate to what is seen as the core search functionality of IR systems, at the expense of interaction, cognition, and the task-and user-context of search. It is the case that the basic Cranfield formalisation of the search task itself (receive a query, supply a result set) ignores such aspects; but this is not in itself the heart of the problem. The central problem is that these wider aspects are resistant to being studied experimentally in a laboratory setting. The essence of a laboratory experiment is abstraction. The principle is that we abstract, from the complexities and messiness of the real world, some small set of variables representing some limited aspects of the world. The abstraction is defined by both the choice of which aspects to include and which to leave out, and the nature of the variables themselves, whose laboratory representation may be a simplification of their counterparts in the real world. The laboratory, and the abstraction of the laboratory experiment, is a fundamental and extraordinarily useful device in the scientific method. Theories and models that can be defined in the context of laboratory abstractions, and tested rigorously in that context, have provided us with vast insights and hugely extended understanding of phenomena in the real world - and continue to do so. However, it is also the case that some phenomena in the real world are more resistant than others to such abstraction. The study of the phenomena of force, mass, motion and so on, that gave us such insights into basic physics, were much easier to abstract in this way than the phenomena associated with life. During the $20^{th}$ century, but not before, we began to be able to study at least some aspects of life in the laboratory. But even within physics, much of our knowledge and understanding came from observational astronomy, with chance events that we happened to observe in the real universe, over the several millennia since Babylon, playing a huge role. Typically, we cannot expect to understand everything we need to know about a set of phenomena from laboratory abstractions. Even more, we cannot expect to do so from a single laboratory abstraction. We enlist the help of laboratory abstractions where and when we can; ideally we should seek several different ways to abstract the world for laboratory investigation, as well as seeking insights more directly from the real world. In some respects, the Cranfield tradition has become relatively rich. Although some parts of the abstraction that it represents are common to many laboratory experiments in IR, there has been a range of different approaches and ideas within it - as represented by the TREC tracks and other experiments outside TREC. These experiments have included serious forays into the more difficult aspects that I mentioned above, such as interaction, cognition, and the task-and user-context of search. Some of the difficulties thus revealed are obvious enough - difficulties of scale and reproducibility among them. However, some seem to be more subtle. One difficulty might be described thus. The main focus of all Cranfield/TREC work is system effectiveness - the object of virtually all experiments in this tradition is to evaluate the effectiveness of a method or component, or to compare multiple methods/components for effectiveness, or to optimise a set of parameters for effectiveness, and so on. But this object does not seem to fit well with, say, work on interaction, which tends to be much more interested in questions like 'If I provide this UI feature or device, how do users perceive and use it?' - with no necessity to attach value-judgments to the possible answers. Perhaps, indeed, this focus is itself a limitation, and an unnecessary one, of the Cranfield/TREC tradition. If we take the view that we would like to see IR as (at least in some sense) an experimental science, then it is worth enquiring more deeply into the role of experiment in science. A very cavalier account of how science works is as follows. We gather data about a class of phenomena; we try to formulate models or theories to describe or account for the phenomena; we push the models or theories to give us hypotheses, predictions about other things that we have not yet observed; and we then try to observe those new things, in order to test hypotheses. The role of experiments is sometimes in the initial data-gathering, and often after the formation of theories in straightforward measurement; but the main fundamental scientific contribution of experiments is in the testing of hypotheses. If we think of a typical TREC-style IR experiment as a test of a hypothesis, then it seems that the only form of hypothesis which we ever test is 'if we apply this model, search will be more effective.' This feels like a very limited class of hypotheses. I would like to suggest that we should be trying to open out the possibilities. We should be looking for ways in which our models or theories might be made to give us other kinds of hypotheses, which we could hope to test in experiments. I don't think such a change in the paradigm associated with Cranfield and TREC would be easy, and I'm not at all clear myself on how to go about it. But if it were possible, it could have the effect of breaking down, at least to some extent, the present barrier between what is seen as the Cranfield tradition on the one hand, and all those domains mentioned above, which are also of fundamental interest to IR, on the other. |
