There are actually two questions relevant to the discussion,

1. How should a baseline be chosen,

A baseline is to help the experimenter to understand what worked, and why it worked. So I believe all baselines should be customized to the particular task under study. E.g. all ad hoc retrieval models should be compared with BM25, Dirichlet LM; all phrase models should be compared with dependency model; all pseudo relevance feedback models should be compared with Rocchio and relevance model.

Now come the second question,
2. what if the guideline is not followed, or how to ensure that standard.

Well, it’s really a moral question, with practical means to ensure the quality of papers.

Generally it’s the authors’ responsibility to make an effort to create a reasonable baseline, and it’s the reviewers’ responsibility to assess whether that result is believable, and whether it will work for the state of the art.

But there will always be cases where such guidelines are not followed, the paper *can* still be accepted if its contribution is believed to be significant by the reviewer. However, I believe the proportion of such work in conferences should be kept low.

I tend to view TREC as an “evaluation platform” for researchers not as a competition. However, I agree that a global perspective on the progress over the years is needed – maybe NIST could play a role in this…

This is the really important question that arises out of the paper. What is a reasonable baseline? Similarly, how much effort should researchers have to put into improving their baseline systems, before they implement and test their innovation?

The argument we make in the paper is to note that a method which improves some vanilla system might not improve, or indeed might even harm, a state-of-the-art one. In the paper, we do a proof-of-concept experiment to demonstrate this, based on toggling options for Indri. Therefore, if the experimenter’s claim is that they are presenting a method that improves retrieval performance over what is currently known, then they have an obligation to test it on state-of-the-art systems. Or, alternatively (and this I think is the really superior method), they need to test their method on a number of different baselines, to demonstrate how generally beneficial it is; but here too, their mix of systems must include ones that implement state-of-the-art techniques. These baselines should, for instance, include methods such as query expansion and proximity operators, which have been shown to lead to improved effectiveness, at least on some collections. It seems that instead what people tend to do is to take something like plain BM25 as their baseline.

It could also be noted that while the best TREC runs may be subject to extensive manual tuning, they do face several disadvantages compared to subsequent experimental runs. First, TREC systems don’t have access to the topics in advance, whereas subsequent runs do. (Yes, I know that direct use should not be made of this advantage, but I believe that it does infer an indirect advantage). Second, TREC systems don’t have the knowledge of which methods have actually worked on a particular collection, which subsequent systems do have.

All of that said, I’m prepared to grant that the best TREC system may not always be an appropriate baseline. But if so, shouldn’t the best (or at least an improved) previously published method be the correct baseline choice? The fact that neither baselines nor “improved” scores trend up over time indicates that this is not the experimental approach that has been adopted — and that whatever approach it is that has been adopted, that approach is not leading to an improvement in effectiveness.

We certainly take the results observed here as an argument for using EvaluatIR or a similar service that centrally records and reports retrieval results. We face the practical issue now of whether and how to incorporate results reported in previous publications, for which we don’t have per-topic scores, let alone the runs themselves.

For the TREC runs, no, it is not possible to download them from EvaluatIR; they can only be obtained from NIST based upon signing an agreement regarding their use. We have also, at NIST’s request, disabled serving out per-topic scores for TREC systems on EvaluatIR, although calculations based on these scores (such as significance tests) are available, and per-topic scores are still shown on graphs. The issue is with permissions of use; we’d have to individually ask the groups that submitting to TREC whether we can make their runs available, and we haven’t gotten around to doing that.

That said, we do use EvaluatIR ourselves extensively simply as a convenient way of looking up the (per-collection) scores achieved on different collections by TREC systems.

Yes, we were careful to note when researchers were using variant collections, and we recorded them separately. One of the amusing second-order findings of the survey was the enormous number of different variant collections that were in fact used. In the 106 papers that met our survey criteria, 83 different variant TREC collections were used — almost as many collections as papers! We discuss this in the second-last paragraph of Section 3; see also Figure 2. Most of the variants were slicing and dicing the TIPSTER corpus and associated topic sets; in some cases, to increase the number of topics (which required excluding parts of the document corpus that were not shared between topic sets); in other cases, to reduce the document corpus size to make computationally intensive methods practical.