I'd committed a calculation error myself -- the null set was 15.6 million, not 16.6 million, document (corrected in post). This would be undeduplicated, if I read the affidavit correctly. That still means that around 40% of the responsive material is excluded by the keyword filter.

One questions about your numbers - you say that the 16.6 million were sampled. My understanding is that only the 15.5 null set was sampled - the 1.4 million documents which were duplicated out were not sampled. Thus we do not know the responsiveness of the duped out docs. Perhaps they were 100% responsive, perhaps zero!

"The remaining 15,576,529 documents not selected by keywords (referred to as the “null set”)..."

"To obtain the relevance rate of the null set, Biomet reviewed a random sample of 4,146 documents..."

There are widely known methodologies to test and improve search term recall and precision. Used correctly, I think that search terms are still an efficient and defensible means to cull the review universe prior to the application of predictive coding. I don't think you will hear that very much from the vendor side as it doesn't sell per gigabyte predictive coding charges.

In this case, what is not clear is whether any of these techniques were used and there is not any analysis that the search terms' recall was reasonable in proportion to the additional cost necessary to improve the recall.

I'll let Losey speak as to the lower bounds of acceptable recall for either search terms or predictive coding, but my two cents is that the test should be on the basis of proportionality.