Fix quantizer for UInt8 type #418
Conversation
| c1 = ((float)(*pX++) - (float)(*pY++)); diff += c1 * c1; | ||
| c1 = ((float)(*pX++) - (float)(*pY++)); diff += c1 * c1; | ||
| a = (*pX); b = (*pY); | ||
| c1 = a + 0.0f - b; |
There was a problem hiding this comment.
Could you share a bit more detail what's the difference in behavior here (floating point quirks?) to me the logic looks the same
There was a problem hiding this comment.
It is the AMD cpu compiling issue. When using the original two lines it will cause illegal instruction segmentation fault.
|
Heya! Just a heads up: we got a quite comprehensive test suite on top of SPTAG and after reviewing this PR, I don't think the changes in add innerproduct PQ/OPQ support are correct. It doesn't seem like simply changing the distance function to IP when building the tables is enough to properly quantize. The resulting tables cannot re-construct the original vectors, and the distance calculations between quantized values do not even have the right relative ordering. Have you guys tested the implementation by building a large index with Product Quantization and IP as a distance? I'd love to hear if I'm doing something wrong here. |
No description provided.