This repository contains utilities for performing Locality Sensitive Hashing in order to identify instances of text reuse in large text collections.
brew install redis)git clone https://gist.github.com/duhaime/eea7a6bfb499904031736f72989a3d1e
cd eea7a6bfb499904031736f72989a3d1e
pip install -r requirements.txtAfter installing the Python dependencies, update config.json to identify the path to your infiles. If your infiles don’t contain XML, set “xml_tag” to false.
Then you can process your texts by running:
python minhash.pyOnce the minhashing completes, you can find all matches in your corpus by running:
python find_all_matches.pyOnce you’ve found all matches, you can create an outfile that represents all the matches by running:
python get_segments.pyExplicitly state max bytes in RAM
# explicitly state max memory in bytes
redis-cli
CONFIG SET maxmemory newlimit-in-bytes
# NB: On a machine with 16GB RAM, an appropriate value would be:
CONFIG SET maxmemory 2000000000If you’re using Linux, you should also disable Transparent Huge Pages
never > /sys/kernel/mm/transparent_hugepage/enabled