I'd committed a calculation error myself -- the null set was 15.6 million, not 16.6 million, document (corrected in post). This would be undeduplicated, if I read the affidavit correctly. That still means that around 40% of the responsive material is excluded by the keyword filter.

One questions about your numbers - you say that the 16.6 million were sampled. My understanding is that only the 15.5 null set was sampled - the 1.4 million documents which were duplicated out were not sampled. Thus we do not know the responsiveness of the duped out docs. Perhaps they were 100% responsive, perhaps zero!

"The remaining 15,576,529 documents not selected by keywords (referred to as the “null set”)..."

"To obtain the relevance rate of the null set, Biomet reviewed a random sample of 4,146 documents..."

There are widely known methodologies to test and improve search term recall and precision. Used correctly, I think that search terms are still an efficient and defensible means to cull the review universe prior to the application of predictive coding. I don't think you will hear that very much from the vendor side as it doesn't sell per gigabyte predictive coding charges.

In this case, what is not clear is whether any of these techniques were used and there is not any analysis that the search terms' recall was reasonable in proportion to the additional cost necessary to improve the recall.

I'll let Losey speak as to the lower bounds of acceptable recall for either search terms or predictive coding, but my two cents is that the test should be on the basis of proportionality.

Hi! Sorry for taking such a long time to reply: holidays and deadlines intervened.

Consider the following two uneven, incomplete lists:

S:
L:

X_1 is 1, since $|{a} \cap {a}| = 1$ . Therefore A_1 is 1 / 1 = 1. X_2 is also 1, making A_2 be 1 / 2 = 0.5. What though is X_3? We know that "b" is a member of {a, b}, but is "?" a member of {a, c, b}? We don't know. Instead, we give it a putative membership of A_2; that is, of 0.5. Therefore, X_3 is 1 + (1 + 0.5)/2 = 1.75. And A_3 is 1.75 / 3 = 0.583. This then works out to the formula shown in Equation 32 (I hope! -- at least it does every time I recreate the working for this formula).

Thanks. Actually I was able to derive much of the equation. The only part that's unclear to me is how you compute A_l. The last part of equation 32 is ((X_l - X_s)/l + X_s/s). As I understand, it is extrapolating average intersection beyond the last element of the longer list. However I am not able to understand how did you get that. By the way I really enjoyed the paper. It was long but really interesting.

Thanks

Hi! I wouldn't try to derive it from Equation 23. Instead, you should go back to the derivation of Equation 23 from Equation 7 and Equation 9 with X_k / k instead of X_k / d. The expression $((1 - p) / p) \sum_{d=k+1}^\infty p^d (X_k / k)$ goes simply to $X_k/k \cdot p^k$ using the geometric series. The rightmost term of Equation 32 is the generalization of $X_k / k$ for uneven lists. Then the left term is simply summing up the per-ranking weighted overlaps from rank 1 to the depth of the longer list, adjusting for the gap between shorter and longer lists.

Sorry, I don't have my original working in front of me. If this is still unclear, I'll have a go at reworking it over the weekend and put the working up somewhere for you.

William