Content Deduplication with Granularity Tweak Based on Base and Deviation for Large Text Dataset

Table 1

Documents and bag of words.

D. no	Documents
D1	John has some cats
D2	Cats eat a fish
D3	Shipment of gold damaged in a fire
D4	Shipment of gold arrived in a truck
D5	Romeo Juliet
D6	Romeo died by dagger
D7	I eat a fish
D8	All that glitters is not gold
D9	Money makes many

Bag of words {“arrived: T1,” “cats: T2,” “dagger: T3,” “damaged: T4,” “died: T5,” “eat: T6,” “fishT7,” “glitters: T8,” “gold: T9,” “John: T10,” “Juliet: T11,” “makes: T12,” “money: T13,” “Romeo: T14,” and “shipment: T15,” “truck: T16”}.