block by duhaime eea7a6bfb499904031736f72989a3d1e

minhash

LSH Text

This repository contains utilities for performing Locality Sensitive Hashing in order to identify instances of text reuse in large text collections.

Dependencies

Quickstart

git clone https://gist.github.com/duhaime/eea7a6bfb499904031736f72989a3d1e
cd eea7a6bfb499904031736f72989a3d1e
pip install -r requirements.txt

After installing the Python dependencies, update config.json to identify the path to your infiles. If your infiles don’t contain XML, set “xml_tag” to false.

Then you can process your texts by running:

python minhash.py

Once the minhashing completes, you can find all matches in your corpus by running:

python find_all_matches.py

Once you’ve found all matches, you can create an outfile that represents all the matches by running:

python get_segments.py

Configure RAM Usage

Explicitly state max bytes in RAM

# explicitly state max memory in bytes
redis-cli
CONFIG SET maxmemory newlimit-in-bytes

# NB: On a machine with 16GB RAM, an appropriate value would be:
CONFIG SET maxmemory 2000000000

If you’re using Linux, you should also disable Transparent Huge Pages

never > /sys/kernel/mm/transparent_hugepage/enabled

config.json

find_all_matches.py

get_segments.py

minhash.py

requirements.txt