Analysis on Complete Works of Swami Vivekananda

Fri, 08 Dec 2023 00:00:00 +0000

Overview

Calculating the exact count of unique items in a large data stream typically requires maintaining an in-memory hash table of every distinct entry seen so far. As the corpus grows into millions of tokens, memory consumption scales linearly ($O(N)$), creating bottlenecks in streaming and distributed pipelines.

The Flajolet-Martin (FM) algorithm solves this by probabilistically estimating stream cardinality using a fixed, minimal amount of memory ($O(\log N)$ bits). This project applies the Flajolet-Martin algorithm to estimate the unique vocabulary size across the nine volumes of the Complete Works of Swami Vivekananda, comparing the probabilistic estimate against exact set cardinality.

The Flajolet-Martin Algorithm

The algorithm estimates distinct elements by tracking the distribution of trailing zeros in hashed bit representations:

Hash Mapping: Each token $w$ in the text stream is hashed to a uniform 32-bit or 64-bit integer $h(w)$.
Trailing Zeros ($\rho$): For each hash value, the algorithm determines $\rho(h(w))$—the number of trailing zeros before the first 1-bit. A hash ending in $k$ zeros occurs with probability $2^{-(k+1)}$.
Maximum Tracking ($R$): The algorithm maintains $R = \max(\rho(h(w)))$ across all processed tokens. If we observe a sequence ending in $R$ trailing zeros, we can infer that roughly $2^R$ unique elements have passed through the stream.
Correction Factor: The raw estimate is adjusted using the Flajolet-Martin correction factor $\phi \approx 0.77351$:

$$\text{Estimated Cardinality} \approx \frac{2^R}{\phi}$$

Variance Reduction (Stochastic Averaging)

A single hash function has significant statistical variance. To produce a stable estimate:

The stream is evaluated across $m$ independent hash functions.
The hash functions are divided into $g$ distinct groups.
The arithmetic mean of estimates is computed within each group.
The overall result is obtained by taking the median of group averages, filtering out extreme outlier hash values.

Processing Pipeline

The text processing pipeline handles the multi-volume corpus in Python:

Text Extraction & Cleaning: Reads raw volume texts, stripping metadata, page numbers, and punctuation while normalizing text to lowercase tokens.
Token Streaming: Simulates a continuous token stream rather than holding the entire preprocessed vocabulary in memory.
Multi-Hash Simulation: Generates independent hash functions using linear congruential generators:

$$h_i(x) = (a_i \cdot x + b_i) \pmod p$$

Benchmarking: Compares the FM probabilistic cardinality against standard Python set() counting to evaluate error bounds across different numbers of hash buckets.

Key Insights