Correctness tests

[amallia]

Please add some correctness tests. One simple idea to do so would be to compute Kendall's Tau between Lucene and the other engines.

@JMMackenzie @elshize any other ideas?

[JMMackenzie]

Kendall's tau has some problems when the lists aren't conjoint, I would suggest computing Rank Biased Overlap (RBO) - see here: http://codalism.com/research/papers/wmz10_tois.pdf

To make the comparison more fair, we could also try to get a CIFF ingestor for each of the systems (https://github.com/osirrc/ciff) - PISA already has this but perhaps Tantivy could try it too?

[fulmicoton]

Adding it for tantivy is a 1 day job. If I follow you correctly, it would help with comparing results by :

• ensuring there are no tokenization discrepancy.
• ensuring the doc ordering is the same.

1. is not really an issue as it is not related with the verification itself. We can give non-ambiguous corpus (lowercase, no punctuation, no apostrophes, no one letter word) and guidelines (no stemming). I prefer keeping natural language there, the reader of the results can have a rough idea of the underlying statistics. (correlation & docfreq), and can craft his own query.

2. For the second part, that would indeed really help. Another way to deal with this is to use a document id and have engine return the list of doc ids. Or (Score, DocIds).

About the metric to use here... You guys are the expert, I am open to any idea, but I am stretched very thin, so I will not be able to help on this.