The sampling procedure used to estimate completeness of the filtering step is described in some detail in Paragraphs 11 through 13, Exhibit C, of the defendant's affidavit (physical page 86 of the PDF). The full collection is made up of 19.5 million documents (I'll round figures for simplicity). The keyword filter extracted 3.9 million of them into the document review system; de-duplication reduced this further to 2.5 million unique documents. The question then is, what proportion of the responsive documents made it into the filtered set; or, conversely, what proportion were left in the 16.6 million excluded documents?

The defendants performed three separate and independent (and partially redundant) samples to answer this question: one of the full collection, one of the filtered-in set, and one of the filtered-out set. The defendant's affidavit provides confidence intervals, but to begin with, let's just work with point estimates of the quantities of interest. The sample results were as follows:

Immediately, you'll see we have a problem. The estimate of the filtered-in set exceeds that of the entire collection, let alone of the collection minus the filtered-out set. In fact, the latter (212,000 222,000) is well outside the 99% confidence interval on the filtered-in set yield of [348,126; 464,887]. And that's understating matters: the filtered-out and collection estimates are undeduplicated, whereas the filtered-in estimate is deduplicated (with a reduction to around 65% of the undeduplicated size).

And indeed the working of the affidavit here is incorrect. I cite verbatim from Paragraph 11 of Exhibit C:

A random sample with a confidence level of 95% and estimation interval of 2.377% consisting of 1,689 documents was drawn from the 2.5+ million documents published to Axcelerate. This sample was reviewed to obtain a baseline relevance rate for the document population created by keyword culling. 273 documents were identified as relevant in the sample, indicating with 95% confidence that the percentage of relevant documents in the population is between 184,268 and 229,162 or stated in percentages, between 14.41% and 17.91%.

The interval of "between 184,268 and 229,162" is coherent with the other estimates in the above table, but it is _not_ "between 14.41% and 17.91%" of "2.5+ million documents", as a few seconds with a calculator will confirm (rather, it is between 7.37% and 9.17%). The 14.41% to 17.91% interval is correct given the stated sample output, but doesn't work with the other estimates.

I'm not sure what the mistake is that has been made here (perhaps confusing figures from pre-deduplication and post-deduplication?), but let's ignore the sample from the filtered-in set, and look just at the other two samples in the above table. The point estimates are round 370,000 responsive documents in the full collection, and 158,000 148,000 responsive documents in the filtered-out set. This means that roughly 42% 40% of the responsive documents are excluded from the set sent to document review, or in other words that even a flawless review process can achieve a maximum recall of 58% 60%. (And that's further assuming that the collection itself has not excluded responsive material.)

It's not my intention to comment on whether the defendant's use of keyword pre-filtering is appropriate and proportionate for the particular circumstances of this particular case. But these figures do illustrate the likelihood that keyword pre-filtering will exclude a large volume of responsive data, and often (where defendants are not so thorough as the current ones have been in their sampling and validation protocol) without anyone being aware of it. The problems artificially imposed by vendors charging by volume aside, I venture to suggest that keyword pre-filtering does not represent best production practice.

And finally, to return to any earlier point, it's really time that practitioners and counsel stop producing the plethora of different estimates that we find in this case, and started sampling for, estimating, and reporting the prime quantity of interest: namely recall.

Stratified sampling improves estimate accuracy in two ways. First, if the characteristic of interest (in e-discovery, the proportion of responsive documents) differs systematically between sub-populations, then accuracy is gained by estimating the characteristic for each sub-population separately, and then aggregating it for the population as a whole. The increase in accuracy comes because we substitute lower within-sub-population for higher between-sub-population sampling error.

Say, for example, we're estimating the average weight of a mixed herd of great danes and chihuahuas. In stratified sampling, we sample from the great danes and from the chihuahuas separately, and only have to worry about whether the sampled great danes are representative of all great danes, and the sampled chihuahuas of all chihuahuas. In simple random sampling, we sample dogs indifferently, and then have to worry about whether the sampled dogs are representative of dogs in general. Happening unawares to sample more great danes than chihuahuas will hurt our estimate a lot; happening by chance to sample more heavy chihuahuas than light chihuahuas will hurt our estimate only a little.

Stratified sampling offers an additional advantage if sub-populations have differing homogeneities. The more heterogeneous a sub-population, the less accurate a sample-based estimate. We can achieve greater accuracy for the same sample budget by sampling the more heterogeneous sub-population at a higher rate than the more homogeneous sub-population. If our chihuahuas all have similar weights, while the great danes are more variable, then we're better off devoting more samples to the great danes, and fewer to the chihuahuas.

How does all of this apply to production evaluation in e-discovery? Here, we have two clearly defined sub-populations: the (candidate) production and the null set. These sub-populations differ significantly in the characteristic of interest, namely the proportion of responsive documents: the production is likely to be dense with them, the null set vary sparse. The null set is also far more homogeneous, being overwhelmingly non-responsive, and so we can sample it more lightly and assign the saved annotations to the more heterogeneous production. (*)

We can go further and divide the collection into more strata than just the produced and null sets. Most contemporary predictive coding systems are able to rank documents by probability of responsiveness, with the production being determined by truncating the ranking at some cutoff. The documents in the null set just below this cutoff are more likely to be responsive than the documents further down. Therefore, it makes sense to divide the null set into two (or more) strata based on this ranking, and sample the upper null set more densely than the lower null set---to look, in essence, for responsive documents missed by the production where we are more likely to find them.

Finally, we require a method of estimation, not just of point estimates, but of confidence intervals. For simple random sampling, the simple binomial confidence interval is used, on the proportion of sampled responsive documents that happen to fall in the production. Interval estimation for stratified sampling is more complex, which has perhaps deterred practitioners to date. However, a method of estimating confidence intervals on recall from stratified sampling is described and validated in my recently published ACMTOIS article, Approximate Recall Confidence Intervals (January 2013, Volume 31, Issue 1, pages 2:1--33) (free version in arXiv). In a later posting, I will provided worked examples with the two sampling and interval methods, and illustrate the significant cost savings that stratified sampling offers.

(*) In practice, predicting the optimal sample allocation between null set and production set is more complicated than just observing which has the higher prevalence, since the production set yield occurs twice in the recall formula (one as numerator, one as denominator), and the null set yield is a divisor, not an added term, in the formula. We will examine this issue in more detail in a later post.

What difference does collection prevalence make to the choice between multimodal and computer-driven review? Well, text classification relies on having both positive and negative examples. The less balanced the example set is, the more examples you require to achieve a given level of effectiveness in the classifier. Indeed, a rough rule of thumb is that cost of training scales more with the number of examples in total, at least where examples are selected by pure random sampling.

The need for positive examples creates a boot-strapping problem for a machine-driven classification approach when collection prevalence is low. If responsive documents are rare, then you'll need a large initial random sample (and therefore extensive review time) to locate them. It would likely be much faster to find an adequate number of relevant documents by a human-directed (keyword or concept) search. Moreover, learning is likely to be slower the fewer relevant documents there are in the initial sample set, even with active learning; again, it might be more efficient to have the human actively help locate relevant documents in the early stages of the search, and delay moving into computer-driven mode until the review process is well under-way.

How serious is the effect of positive-example seed-set deprivation on the efficiency of text classification? The impact on seed-set creation is easy to calculate; it's a simple question of sampling. Let's say that a minimum of ten responsive documents are required to get the text classifier going; if one in a thousand documents in the collection is responsive, then on average a sample of ten-thousand documents is required to make an adequate seed set (more if you want a reasonable degree of confidence that at least ten positive examples will be found). With a low enough prevalence, a randomly-sampled seed set may simply be a non-starter.

Say, though, we have a more intermediate case, where prevalence, while low, is still sufficient to give a handful of responsive documents in a moderately-sized sample; Ralph's own example produces 10 relevant documents in a sample of 2,401, for a sample prevalence of just under half a percent. The effect of such a low-prevalence seed set on speed of learning is complex, particularly when active learning is being used, as is the case with most classification systems these days. (In active learning, the computer selects for coding at each training iteration those documents that it is least certain about. This is generally much more efficient than selecting documents at random, but still requires an initial seed set to get started.) Active learning should in principle compensate for low prevalence, by exploring ambiguous regions of the document space. How strong this effect will be in practice is an empirical question.

What, then, is the argument in favour of solely computer-driven review, even in the face of low-prevalence tasks (apart from the fact that it is much simpler to build a user interface for batch annotation than for active search)? The main argument is the assertion that relying on human judgment to select seed documents will bias the results towards the human's initial conception of relevance; that responsive documents similar to those found in the initial keyword search are more likely to appear in the final production than responsive documents different from the seed set. Certainly, human choice in creating the seed set will have some impact on final classifier output. On the other hand, in the subsequent training iterations, active learning will select documents that are unlike the initial seed set, and so over time should drive the classifier away from dependence on the human's initial judgment. Again, the strength of this effect is an empirical question.

So, we have two empirical questions about the comparative impact of random and human-created seed sets for active learning on low-prevalence topics. Is active learning on a seed set with only a handful of responsive documents able to close the effectiveness gap with a higher-prevalence seed set populated by human search, and if so, how quickly? And how strongly does a human-selected seed set bias final classification results towards the sub-topic represented in that seed set?

The confidence interval approach directly answers this question of failure probability. When we say that "recall is greater than 60% with 95% confidence", we are expressing the probability that a false positive will occur, if we accept the production. Here, the 60% recall is the lower limit on an acceptable quality of production. The unstated third term in this equation is the sample size; but it is the sample size that allows us to make these statements about confidence.

Conversely, if the production fails -- if the lower bound is less that our threshold (here, 60%) -- we can look at the upper bound on recall, and check that it is below the threshold of a system that should reliably pass. If, for instance, we had a confidence interval on recall of [55%, 95%], then we have a serious problem: we can't pass the production (lower-bound recall is only 55%), but we can't be confident that the production is not actually good (upper-bound recall is 95%).

We'd prefer not to realize that we've failed a good system after validation is complete, of course. Therefore, we plan the sample size of our validation protocol in advance, to reduce the chance of failing a good production to an acceptable level. (And note that the lower bound of a "good" production must be higher than the upper bound of a "bad" production; otherwise, the probability of rejecting a good system is always one minus the probability of accepting a bad one.) Planning the protocol to pass good productions requires similar reasoning about sample sizes and failure probabilities to that of calculating the confidence interval after the production is sampled.

The problem with using a point estimate alone is that it gives us none of this information about confidence and failure probabilities. It fails to do so because it does not consider the size of the sample we have used. A point estimate of 75% recall means quite a different thing on a sample of 100 documents than it does on a sample of 10,000.

One motive for the preference for point intervals is the perception that confidence intervals make the protocol unnecessarily more difficult to satisfy. This objection is poorly founded in theory, in that one can generally state a confidence interval requirement that is as loose or strict as a point estimate one. The objection perhaps has some psychological basis, as it may seem (to the statistically naive) that a 95% confidence threshold of 60% is looser than a point estimate threshold of 75% recall. But that psychological judgment is achieved only by ignoring what the different bounds are telling us.

Let's take Chicago--Los Angeles non-stop round-trip as a representative in-continent flight, and Frankfurt--Beijing non-stop round-trip as a representative out-of-continent flight. According to the emissions calculator provided by the International Civil Aviation Organization, the former trip generates around 480kg of CO2, the latter around 1,200kg. (We're being conservative here and assuming everyone flies economy, and everyone can make a non-stop trip. You can add roughly 200 kilos of CO2 per stop). Therefore we have (480 * 300 + 1200 * 300) = 504 metric tons of CO2 gas.

So how much is 504 metric tons of CO2? The US EPA provides a handy calculator of greenhouse gas equivalences; according to this, 504 metric tons of CO2 is equivalent to the annual emissions of around 100 US passenger vehicles; the energy use of 44 US households; 5 acres of deforestation; and so forth.

What could be done to cut these emissions? The simplest answer would be -- cancel SIGIR (though in practice this would probably lead to an increase in compensatory air travel for meetings that take place at SIGIR). A less drastic solution would be to move towards a model in which conference attendance is not compulsory for publication; something like the journalized conference form of the Proceedings of the VLDB Endowment. Journalization would reduce conference attendance (and resulting travel) to those who actually want to attend the conference (as well as removing a financial hurdle to publication for those who have limited finance). Additionally, if a series of three conferences were associated, on equal footing, with the attached journal (perhaps SIGIR plus CIKM plus one other), and if one such conference was held in each region per year, distance flown by attendees would likely drop.

As a not-for-profit organization, SIGIR is not subject to the profit-maximizing imperatives of business entities. The SIGIR community should be taking the lead in implementing practical solutions to reduce the environmental impact of its activities. Restructuring the way SIGIR and associated conferences work can help to achieve this.

Generally in statistics when we think about such tests, we look for something more than satisfiability. Rather, we think about how confident we are that what we consider a sufficiently complete production would fail the protocol -- what is known as the power of the test. Power stands in tension to the strictness of the test, the confidence required for passing the threshold. The stricter our test, the less likely bad systems are to pass, but the more likely good systems are to fail.

Let's put this in concrete terms. To make things simpler, we'll work initially in terms of what Herb Roitblat calls elusion: the proportion of non-produced documents that are relevant. (This can be directly mapped to recall, but converting backwards and forwards is going to confuse us.) Our QA protocol might then be stated in such terms:

A production only passes QA if we have 95% confidence that elusion is no more than 1%.

Once we add to this a sample size, the protocol can be stated in terms of a minimum number of relevant documents that will fail the production if they are found in the sample. So, for instance, if we have a sample of 1000 documents, then the above protocol will fail if 5 or more relevant documents are found in the sample; the upper-tailed exact 95% confidence interval on elusion with a sample of 5 relevant out of 1000 is 1.05%, whereas for 4 relevant it is 0.91%. (For those who are savvy with the R statistical language, we calculate the interval using binom.test(5, 1000, alternative="less"), and look at the result marked "95% confidence interval". The threshold value -- here 5 -- is calculated using qbinom(1 - 0.95, 1000, 0.01); that is, the 5th percentile of the Binom(1000, 0.01) sampling distribution.)

So now we need to decide what actual sample size is appropriate for our protocol. This decision hinges on two questions. The first question is how confident we want to be that a good production would pass our protocol, what is known as the "power" of the test. A power of 80% is a common choice. The second question is what we would consider a "good" production.

It might seem at first blush that the answer to the question "what is a good production?" is "a good production is one with elusion of no more than 1%", which is what our protocol is testing for. But in fact a production that has the threshold elusion of the protocol only has a 5% chance or less of passing that protocol; this is what gives us the 95% confidence that a passing production has threshold elusion or higher. (If in fact you want to define 50% production as good, then you need to rethink your protocol, and set a lower rejection threshold.)

More generally, for a given sample size, we can state the probability that a production of a given level of performance will pass our protocol. For instance, if our sample size is 1000, and the production has actual elusion 0.5% (half the threshold amount), then it still has a 56% chance of failing our protocol. (In R, 1 - pbinom(5 - 1, 1000, 0.005), where 5 is threshold failure sample positives calculated earlier). In fact, the maximum elusion that would have at least an 80% chance of passing our protocol is 0.23%, under a quarter of the threshold amount (in R, 1 - qbeta(0.8, 1000 - (5 - 1) + 1, 5 - 1), exploiting the relationship between the beta and binomial distributions).

What does this all mean for recall? Well, given a retrieved and an unretrieved size, and assuming all retrieved documents have been verified relevant (and all known relevant documents are retrieved), then a recall value can be mapped into an elusion value, and used in the above calculations. Let's therefore restate our QA protocol in terms of recall:

A production only passes QA if we have 95% confidence that recall is no less than 50%.

We also need to specify what constitutes a good production, one that we want to be 80% confident will pass our protocol. Let's specify that a production with actual recall of 75% is good. Then we can ask, what sample size is required to satisfy our evaluation criteria for a given retrieval size? The following graph gives the answer:

Sample size required for production size

... but it's a bit hard to read. Let's show the same thing in a log-log graph (note the axes carefully):

Sample size required for production size (log-log)

The line is approximately straight in the log-log graph because there's an inverse relationship between retrieval and sample size (doubling the retrieval size roughly halves the sample size required), at least while retrieval size is a small fraction of collection size. In fact, for this particular collection size, and threshold and "good" recall, the approximate relationship is captured by the formula:

(Don't pick sample sizes for actual protocols using this approximation, however: our test statistic -- relevant documents sampled -- is discrete, and inference from discrete statistics can be very sensitive to apparently minor differences in sample size.)

From the figure we can see that, if we are going to validate our production with a post-production sample of 2,400 from the unretrieved set, then there's no point performing the validation unless our retrieval has found at least 2,700-odd relevant documents, because otherwise even a good production can fail the validation. And, if we set the upper bound on the number of documents we're prepared to look at during validation to 10,000, then productions of less than 640 documents essentially cannot be validated, unless we're prepared to loosen our validation requirements. If, for instance, we set the threshold recall at 30%, and require only 90% confidence in beating this threshold (while retaining 75% recall as our definition of an "good" retrieval), then required sample size falls from around 10,000 to below 2,000. But we'd want to be sure that we're willing to accept the reduced confidence and lower threshold.

In summary, the feasibility of using sampling-based methods to validate the recall of an e-discovery production is highly sensitive to production size. For small productions, strict protocols will require an impractically large sample size. Protocol parameters should be determined with an eye to expected collection yield and production size, and where expectation or (production-independent) evidence suggests relevant documents are rare, alternative levels or methods of production validation must be considered.

Such figures might cause us to question the usefulness of quality assurance sampling. How reasonable is it to include assurance requirements that force us to review tens of thousands of documents to verify that we've only found a highly relevant handful because highly relevant documents are rare? One way to address the question of effort required is to set up a simple protocol, and see how big a sample is necessary for this protocol to be satisfiable. Let's say the requirement is that we have 95% confidence in a lower-bound of 50% recall. And, to keep things simple, let's say that we'll select a QA sample large enough to meet this requirement, provided none of the documents in it prove relevant (at whatever threshold of relevance we're testing). (This is a primitive form of [edit: power] analysis, adequate only for the sake of our current discussion. To do things properly, we need also to consider the likelihood of a sample appear for an hypothesized level of "acceptable" system performance. Don't choose sample sizes for QA of actual productions based only on the discussion here!)

With these assumptions, we quickly realize that there are two factors that determine the required size of the QA sample: the size of the null set, and the size of the production (assuming that the production consists of all and only manually-verified relevant documents). Let $r$ be the size of the production, and $N$ the size of the null set. The sample size $n$ required for an all-negative sample to satisfy our protocol is:

Setting the null size set at 700,000, as for Ralph's production, this equates to the following figure:

Sample size for production size

Required sample size drops precipitously with increased production size; so precipitously that it is difficult to read off corresponding values. Plotting on a log-log graph makes the correspondence clearer; the relationship becomes a straight line:

Sample size for production size (log-log)

While a production of 20 documents requires a sample size around 100000 for our QA protocol to be satisfiable, a sample size of 1000 documents is sufficient once the production exceeds 2000 documents. The size of the null set also effects minimum protocol-satisfiable sample size; where $r \ll N$ , doubling the null set size roughly doubles the sample size required. Nevertheless, if the production is large enough, then the our QA protocol is satisfiable with a feasibly-sized sample.

In short, the informativeness of a QA sample depends upon the characteristics of topic and collection. If relevant documents are rare, then even an effective production will be small; meanwhile, sampling can set only so low an upper bound on the proportion of relevant documents in the null set; and if the collection itself is large, then the plausible number of missed relevant documents will overwhelm the number actually found. If relevant documents are less rare, though, and if the collection size has been kept under control, then productions will be larger, plausibly missed documents will be fewer, and QA sampling is able to confirm a satisfactory level of recall.

In his production, Ralph is separately coding relevant and highly relevant documents. Let's look at the case of estimating the number of highly relevant documents that may have been missed by the production; the numbers involved are smaller, and more extreme cases of the sampling and estimation procedure are tested.

Ralph's production of relevant and highly relevant documents can be divided into three main stages:

1. An initial simple random sample of 1507 documents from the full collection of 699,082 documents, none of which sampled documents were found to be highly relevant.
2. Several iterations of search, predictive coding, and review, culminating in a production list of 659 relevant documents, of which 18 were highly relevant. All of these 659 documents have been reviewed by Ralph and confirmed relevant. This leaves 699,082 - 659 = 698,423 documents in the null set. Some of the null set have also been reviewed by Ralph and found irrelevant, but that information is ignored for the next step.
3. A final random sample of 1065 from the null set, of which 0 were found to be highly relevant. We'll refer to this as the quality assurance (QA) sample.

The question now is, how many highly relevant documents may Ralph have missed in the null set, and thus have failed to produced? Ralph observes that since the QA sample produced no highly relevant documents, our best estimate is that no highly relevant documents have been missed; if we add in the evidence of the initial sample, which also found no highly relevant documents, and of the production process, which only found 18 highly relevant documents, then it seems likely that very few, if any, have been missed.

At this point, though, Gordon observes that if we calculate a confidence interval on the number of highly relevant documents, we get rather a different story. Gordon calculate a two-tailed 95% confidence interval, but since we're looking for a probabilistic upper bound (and the lower bound is evidently going to be 0), I'll use a one-tailed upper confidence interval. Using the exact (Clopper-Pearson) interval on a proportion, the one-tailed 95% confidence upper bound on the proportion of highly relevant documents, based upon finding 0 in a sample of 1065, is 0.281%. Given there are 698,423 documents in the null set, this amounts to 1956 highly relevant documents. Thus, based on this final sample alone, the most we can say with 95% confidence is that we've found 18 / (18 + 1956) = 1% of the highly relevant documents. This is not, on the surface, an encouraging finding.

How do we manage to conjure the ghosts of 1956 highly relevant documents potentially lurking in the null set from a sample containing 0 such documents? We work backwards as follows. Assume that 0.281% of the null set was in fact highly relevant; that is, that 99.719% of the null set is not highly relevant. What is the probability of sampling 1065 documents from a set in which 99.719% are not highly relevant and having no highly relevant documents in the sample? It is 0.99719 ^ 1065 = 5%. Thus, our upper bound on the 95% interval is the proportion relevant for which our observed sample result (or fewer -- but you can't have fewer than 0) has a 5% chance of occurring.

At this stage, however, as Ralph observes, something seems to be off in our reasoning. We did an initial random sample of 1507 documents, of which none were highly relevant. The 95% upper bound on that sample 0.198%, or 1388 highly relevant documents. We then did a production and found 18 highly relevant documents. Now we're estimating an upper bound of 1956 highly relevant documents. Why have we gone backwards?

The issue is that one cannot in general simply add up the evidence from the two samples, especially when the first sample has been used to help form the production that the second sample is evaluating. Care needs to be taken in assessment protocols to make sure that sample evidence does not become "polluted" by being entangled in process being measured; this is expressed in the maxim "separate training and testing data".

In fact, for this particular protocol, we can more or less add the two samples together, provided we assume that the production has done at least as well as random (and so has avoided "pushing" highly relevant documents out of the production into the null set) -- and, given that 2.7% of the production is highly relevant, this is a reasonable enough assumption. (And the production is too small to have much impact in any case). So we can add the 1507 to the 1065 sample to derive a sample size of 2572, and a 95% confidence upper bound of 813 highly relevant documents left in the null set. (We're violating the strict assumptions of the exact binomial interval here, but the answer is a very good approximation.) But even with this reduced upper bound on the highly-relevant document missed, we've still only got a 95% lower bound of 18 / (18 + 813) = 2% recall.

What about the fact that an expert searcher (Ralph) and a reputable predictive coding system (Kroll OnTrack's inView) have made several iterations over the document collection, and that they've only been able to find 18 highly relevant documents? Does this influence our confidence? Put another way, had the production been performed instead by randomly picking 659 documents, and a QA sample on the null set produced the same result of no highly relevant documents, would we assign the same probabilistic upper-bound estimate? The answer to the latter two questions are, respectively, "no", and "yes": no, the fact that a careful production been performed doesn't change our estimate; and yes, we'd give the same upper bound (for the same sample evidence) even if the production had been made at random.

It is not that our beliefs (prior to the sample) about the thoroughness of the production are irrational or have no evidentiary basis. Subjectively, they have a reasonable foundation (though we'd be wise not to place too much trust in our own judgment). Nor is it merely that we don't want to allow our estimate to depend upon assumptions about the quality of the process being estimated (though this is certainly part of the issue). The central problem is that we don't know how objectively to quantify and justify the probabilities associated with our prior beliefs and the evidence they are based on. We can't model the process by which the production has found the highly relevant documents it has, and so we can't say how likely it is that it has missed others. Random sampling, however, follows a simple selection process that we can model and reason, probabilistically, about. Chance is predictable; choice (by machine or human) is not: that is the foundational insight of modern statistical science.

Nevertheless, the self-imposed amnesia involved in the final QA sample emphasizes that such a sample does not, by itself, constitute an adequate assurance of the quality of a production. (Put another way, the production could be quite inept, and there still be a good chance that the ineptitutde would be missed by the QA sample, if relevant documents are rare enough.) Rather, such QA sampling needs to take place as part of a proven production protocol, one which incorporates various assessment checks, even if their evidence cannot formally be combined into a single confidence interval. The development of a standard for such protocols, as recently proposed by Jason Baron through an ANSI working group, can guide practitioners towards best practice, and alleviate them from the responsibility of having to roll their own processes.

I've written some research work on more advanced topics in confidence intervals, but I thought it might be useful to write some more introductory material as well. I originally intended to write a series of blog posts giving a brief tutorial on sampling and estimation, but the brief tutorial worked out to be around 5,000 words, so I've made it into a separate document: A tutorial on interval estimation for a proportion, with particular reference to e-discovery. The tutorial aims to give an understanding of the workings behind confidence intervals, while avoiding as much math as possible. (If you want a still more high-level discussion of sampling, estimation, and intervals, then I recommend Venkat Rangan's post on Predictive Coding -- Measurement Challenges.) The tutorial is marked as Version 0.1; I'd be very grateful for any corrections, comments, or suggestions for improvement, and will work them in to later versions.

There's a slightly tricky question about sampling rates and gold standard reliability to consider also. (Readers not interested in statistical niceties can skip this and the next paragraph.) The sample for our experiment was designed to produce an even balance of officially relevant and officially irrelevant documents, since this provided the greatest statistical power for the comparison we were making (in that case using Cohen's $\kappa$ ). As it happens, this sampling also increases the concentration of appealed documents. So directly comparing results on only the experimental sample would be unfair to the TREC assessors. Instead, we perform a sample-based extrapolation from the experimental assessments to the sample drawn for assessment at TREC (though not to the full corpus, since this would increase estimate variance without fundamentally changing estimate accuracy).

A second nicety is in the handling of what I call the "bottom stratum", of documents retrieved by no TREC participant. This stratum was very lightly sampled from the TREC to the experimental sample (as it was from the corpus into the TREC sample), so (in extrapolation) each document here has a big impact upon comparisons of effectiveness. At the same time, no team had an incentive to (and no team did) appeal assessments of irrelevance in this stratum, so any false negatives (actually relevant documents assessed as irrelevant) will have been missed. Thus, including the bottom stratum potentially biases the comparison in favour of the TREC assessors. I therefore report results both with and without the bottom stratum considered.

The statistical prologemena done with, here are our results. The original experiment involved three treatments: a first batch with the topic statement only; a second batch with the detailed criteria of relevance; and then both batches jointly re-assessed by both assessors in a (mostly successful) attempt to reach agreement on relevance. Below, we report mutual $F_1$ scores with the official qrels for each batch. The TREC assessments were not (and cannot) be divided into the same treatments, so only the one $F_1$ score is reported for all three treatments. (Put another way, each batch provides an estimate on the full evaluation sample under different treatments). The TREC assessors worked independently to the topic authority's detailed guidelines; this therefore is the fairest comparison with our experimental assessors. I also show the unextrapolated $F_1$ score of the strongest participant team for this topic (the industry team included in Grossman and Cormack's analysis).

Here then, finally, are the results:

We can see that, if the bottom stratum is excluded, both our assessors (Exp-A and Exp-B) outperform the TREC assessor. For each of the two batches, Exp-B found 2 officially irrelevant documents to be relevant, so if this stratum is included in the comparison, Exp-B's score is depressed to below that of the TREC assessor (note the strong sampling effect in this stratum), though assessor Exp-A's and the joint-review score is unaffected. It was noted above that including the bottom stratum is potentially biased in favour of the TREC assessors, though it should be said that for the four documents in question, our assessors jointly concluded that they weren't in fact relevant (and, having viewed them myself, I concur).

I've not checked the above results for statistical significance. Given the sampling complexities involved, that would be a tricky thing to do. My suspicion is that the differences without the bottom stratum may be significant, but with the bottom stratum likely are not, due to the much greater sampling variability in the latter case. More to the point, though, this is only a single topic, in an experiment not directly set up to measure the comparison made here (and where even the official assessments might contain some remaining errors or ambiguities); even a finding of statistical significance would not show that our assessor A is "provably more reliable" than the TREC assessors. There are also differences of conditions involved; our assessors worked three-hour shifts, assessing around 60 documents (assessor A) to 30 documents (assessor B) an hour, which are less fatiguing conditions than perhaps are typical in large-scale manual review (though the conditions of the professional TREC review are not specified in the TREC overview document). The conclusion that can be reached, though, is that our assessors were able to achieve reliability (with or without detailed assessment guidelines) that is competitive with that of the professional reviewers -- and also competitive with that of a commercial e-discovery vendor.

Time to meet our crack reviewers:

Intern-reviewer Bryan

Intern-reviewer Marjorie

At the time of the experiment, Marjorie and Bryan were high school seniors working in the E-Discovery lab as interns four mornings a week. (Since then they have become ex-high-school students waiting for their summer holidays to end and their studies at the University of Maryland to commence.) They have no legal training, and no prior e-discovery experience, aside from assessing a few dozen documents for a different TREC topic as part of a trial experiment. They performed assessments on TIFF files, displayed on rather underpowered laptops that couldn't fit the full TIFF onto the screen and tended to crash every now and then. They worked independently and without supervision or correction, though one would be correct to describe them as careful and motivated.

All of this raises the question that is posed in the subject of this post: if (some) high school students are as reliable as (some) legally-trained, professional e-discovery reviewers, then is legal training a practical (as opposed to legal) requirement for reliable first-pass review for responsiveness? Or are care and general reading skills the more important factors?

As it happens, the same question has been addressed in a couple of other recent studies, which reached similar results to our own, and thus add some support to our tentative finding, namely: that legal training does not confer expertise in document review. In A User Study of Relevance Judgments for E-Discovery (Proc. ASIST, 2010), Jianqiang Wang and Dagobert Soergel had four law school and four library and information studies (LIS) students re-assess documents from four TREC topics. In an exit interview, the law students stated that their legal training and experience was helpful in performing their assessments, whereas the LIS students felt such experience would not have helped much. The LIS students appear to have been correct: there was little or no difference between the two assessor groups in reliability or speed. In another study, Legal Discovery: Does Domain Expertise Matter? (Proc. ASIST, 2008), Efthimis N. Efthimiadis and Mary A. Hotchkiss had six groups of MLIS students build a run for the TREC 2007 interactive task. Two of these groups consisted of law librarianship students with JD degrees and professional experience as lawyers or legal searchers; but they achieved no greater reliability than the remaining four groups, who had no legal experience.

Some TREC 2009 topics were initially assessed by law students, other by professional reviewers. The difference in expertise level can be considered here, though with two caveats: first, we're comparing assessors on different topics, and variability in the assessment difficulty of topics appears to be high (see Table 3.5 of our draft survey); and second, we don't know how the professional reviewers conducted their reviews, and there could be differences in process as well as raw expertise. Of the five topics in the TREC 2009 Legal Interactive task that were appealed with sufficient thoroughness for the us to be confident that most errors were found (see my SIRE 2011 paper, Re-examining the Effectiveness of Manual Review, for a fuller discussion), three were initially assessed by professional review teams, two by volunteer law students. The picture of the benefits of expertise is mixed for this dataset. If we consider only the documents actually assessed (without making a variance-increasing extrapolation to the corpus), then the law students seem roughly as reliable as the professional reviewers (Table 4 of Re-examining), though if we do extrapolate to the full population, one of the professional review teams does outperform the two student and other two professional teams in both reliability (ibid., Table 2) and consistency (ibid., Figure 2) -- though perhaps this is explained by good process (and care in recruitment), rather than greater legal training.

Now, even if we take at face value the finding that legal training does little to improve reliability in first-pass document review, this does not mean that there are not systematic differences in the skills of reviewers (some will be more careful and attentive readers than others, for instance), or the accuracy of professional review teams (depending upon the quality of their processes and their care in recruiting said skilled reviewers). I'd not recommend rounding up high school students on summer break to save review costs. And of course legal training and expertise are required to frame, monitor, and interpret the production. Our finding might suggest that first-pass document review is not practice of the law (since it apparently does not employ legal skills), and therefore that using non-lawyers to perform it should not be considered unauthorized practice of the law, but I'll offer a bottle of wine to the lawyer brave enough to argue that in court on the basis of this blog post. Perhaps in the construction of e-discovery test collections, we do not have to insist that all reviewers have legal training, though again this depends upon the acceptability of this to the practicing e-discovery community. The research implications are clearer: we don't know what makes a reliable reviewer; legal training doesn't appear to play a major part, which is a negative outcome; but that in compensation means that findings on the conception and perception of relevance from outside the e-discovery domain can with more confidence be applied within it.