This repository contains the code and input data for the calculation of possible quotations and similar passages within the gretil corpus based on SIF-weighted averages of word vectors as described in this paper. The Sandhi-seperated input data was created with this code. The code of this repository is licensed under the GNU AGPLv3 license.
The etext data has been taken entirely from the GRETIL collection.
The HTML-tables produced by this code can be accessed here (download).
A graphic visualization is available here.
The code is built on fasttext and nmslib. These are needed along with numpy and scipy in order to be able to execute the code. The code for handling/storing the word vectors (fasttext.py) was taken from this repository.
To run the calculations, execute the file 'run.sh' in the code-dir.
Completing the calculations takes about 8 hours on a multicore machine with sufficient memory (please adjust the scripts according to the hardware before running them).
Given the BOW-nature of averaged word vectors, the system can quite reliably detect passages where the word order was changed or different particles have been inserted between the words. It is not yet very good at detecting phrases where a larger part of the vocabulary was exchanged with synonyms. Stemming can certainly help with regard to this problem.