NSEScraper: An Open-Source Scraper for nseindia.com

Sat, 03 Aug 2024 17:37:11 +0000

Overview

The National Stock Exchange of India (NSE) website provides live market feeds, but extracting clean intraday price histories for quantitative analysis or algorithmic backtesting often requires navigating session headers, cookie rotation, and nested JSON endpoints.

nsescraper is a lightweight, open-source Python library that queries NSE endpoints directly and returns cleaned, timestamped pandas DataFrames for any listed index or equity. It supports standard 1-minute OHLC bars, custom multi-minute intervals, and second-by-second tick data.

Installation

Install the package directly from PyPI using pip:

1

pip install nsescraper

Dependencies: Python 3.8+ and pandas.

Features

Real-time Intraday Feeds: Extract the current trading day’s prices for indices (Nifty 50, Bank Nifty, Midcap, Sectoral indices) and all NSE-listed stocks.
1-Minute OHLC Candles: Formats timestamped Open, High, Low, and Close bars across the full market session.
1-Second Tick Data: Capture granular tick-level price changes throughout active market hours.
Custom Candlestick Resampling: Parameterize candle intervals to custom durations (e.g., 5-minute, 15-minute, or 60-minute bars).
Direct pandas Integration: Every function returns a standard pandas DataFrame ready for technical analysis, plotting, or CSV export.

Usage & Examples

Import the package functions:

1

from nsescraper import intraday_index, intraday_stock, historical_index, historical_stock

1. 1-Minute OHLC Candlestick Data

To fetch the current day’s 1-minute candles for an index:

1
2


df_nifty = intraday_index("nifty 50")
print(df_nifty.head())

1
2
3
4
5
6
7
8


+-----+---------------------+---------+---------+---------+---------+
| | DATETIME | open | high | low | close |
+=====+=====================+=========+=========+=========+=========+
| 0 | 2024-08-02 09:00:00 | 25010.9 | 25085.6 | 24777.0 | 24945.8 |
| 1 | 2024-08-02 09:01:00 | 24946.0 | 24950.0 | 24826.8 | 24830.0 |
| 2 | 2024-08-02 09:02:00 | 24816.5 | 24816.5 | 24784.0 | 24792.3 |
| 3 | 2024-08-02 09:03:00 | 24791.5 | 24791.5 | 24777.6 | 24789.8 |
+-----+---------------------+---------+---------+---------+---------+

To fetch 1-minute candles for an individual stock:

1
2


df_stock = intraday_stock("britannia")
print(df_stock.head())

1
2
3
4
5
6
7


+-----+---------------------+---------+---------+---------+---------+
| | DATETIME | open | high | low | close |
+=====+=====================+=========+=========+=========+=========+
| 0 | 2024-08-02 09:15:00 | 5645.9 | 5695.8 | 5645.9 | 5695.8 |
| 1 | 2024-08-02 09:16:00 | 5707.8 | 5732.95 | 5706.7 | 5729.85 |
| 2 | 2024-08-02 09:17:00 | 5715.0 | 5737.9 | 5709.15 | 5729.85 |
+-----+---------------------+---------+---------+---------+---------+

2. High-Frequency 1-Second Tick Data

Pass tick=True to retrieve tick-by-tick market logs:

1
2
3
4
5
6


# Tick data for index
df_midcap_ticks = intraday_index("nifty midcap 100", tick=True)

# Tick data for equity
df_reddy_ticks = intraday_stock("dr reddy", tick=True)
print(df_reddy_ticks.head())

1
2
3
4
5
6
7
8


+------+---------------------+---------+
| | DATETIME | Tick |
+======+=====================+=========+
| 0 | 2024-08-02 09:15:24 | 6891.20 |
| 1 | 2024-08-02 09:15:30 | 6886.10 |
| 2 | 2024-08-02 09:15:37 | 6900.05 |
| 3 | 2024-08-02 09:15:44 | 6900.60 |
+------+---------------------+---------+

3. Custom Timeframe Candlesticks

Use the candlestick argument to specify an arbitrary minute interval (e.g., 10-minute candles):

1
2


df_energy_10m = intraday_index("nifty energy", candlestick=10)
print(df_energy_10m.head())

1
2
3
4
5
6
7


+----+---------------------+---------+---------+---------+---------+
| | DATETIME | open | high | low | close |
+====+=====================+=========+=========+=========+=========+
| 0 | 2024-08-02 09:00:00 | 44954.4 | 44954.4 | 44481.8 | 44481.8 |
| 1 | 2024-08-02 09:10:00 | 44481.8 | 44627.7 | 44258.1 | 44625.6 |
| 2 | 2024-08-02 09:20:00 | 44618.6 | 44684.2 | 44537.8 | 44551.7 |
+----+---------------------+---------+---------+---------+---------+

4. Historical Data Retrieval

To fetch longer-term end-of-day price histories:

1
2


df_hist_nifty = historical_index("NIFTY 50")
df_hist_abb = historical_stock("ABB")

Analysis on Complete Works of Swami Vivekananda

Fri, 08 Dec 2023 00:00:00 +0000

GitHub Repository

Overview

Calculating the exact count of unique items in a large data stream typically requires maintaining an in-memory hash table of every distinct entry seen so far. As the corpus grows into millions of tokens, memory consumption scales linearly ($O(N)$), creating bottlenecks in streaming and distributed pipelines.

The Flajolet-Martin (FM) algorithm solves this by probabilistically estimating stream cardinality using a fixed, minimal amount of memory ($O(\log N)$ bits). This project applies the Flajolet-Martin algorithm to estimate the unique vocabulary size across the nine volumes of the Complete Works of Swami Vivekananda, comparing the probabilistic estimate against exact set cardinality.

The Flajolet-Martin Algorithm

The algorithm estimates distinct elements by tracking the distribution of trailing zeros in hashed bit representations:

Hash Mapping: Each token $w$ in the text stream is hashed to a uniform 32-bit or 64-bit integer $h(w)$.
Trailing Zeros ($\rho$): For each hash value, the algorithm determines $\rho(h(w))$—the number of trailing zeros before the first 1-bit. A hash ending in $k$ zeros occurs with probability $2^{-(k+1)}$.
Maximum Tracking ($R$): The algorithm maintains $R = \max(\rho(h(w)))$ across all processed tokens. If we observe a sequence ending in $R$ trailing zeros, we can infer that roughly $2^R$ unique elements have passed through the stream.
Correction Factor: The raw estimate is adjusted using the Flajolet-Martin correction factor $\phi \approx 0.77351$:

$$\text{Estimated Cardinality} \approx \frac{2^R}{\phi}$$

Variance Reduction (Stochastic Averaging)

A single hash function has significant statistical variance. To produce a stable estimate:

The stream is evaluated across $m$ independent hash functions.
The hash functions are divided into $g$ distinct groups.
The arithmetic mean of estimates is computed within each group.
The overall result is obtained by taking the median of group averages, filtering out extreme outlier hash values.

Processing Pipeline

The text processing pipeline handles the multi-volume corpus in Python:

Text Extraction & Cleaning: Reads raw volume texts, stripping metadata, page numbers, and punctuation while normalizing text to lowercase tokens.
Token Streaming: Simulates a continuous token stream rather than holding the entire preprocessed vocabulary in memory.
Multi-Hash Simulation: Generates independent hash functions using linear congruential generators:

$$h_i(x) = (a_i \cdot x + b_i) \pmod p$$

Benchmarking: Compares the FM probabilistic cardinality against standard Python set() counting to evaluate error bounds across different numbers of hash buckets.

Key Insights

Memory Efficiency: While exact counting requires storing hundreds of thousands of unique string tokens in memory, the Flajolet-Martin register array requires only a few bytes of storage.
Accuracy Trade-off: Increasing the number of hash functions significantly tightens the standard error, bringing the estimated vocabulary count within 5-10% of the true distinct word count.

Python on Ujjwal Chowdhury

NSEScraper: An Open-Source Scraper for nseindia.com

Overview

Installation

Features

Usage & Examples

1. 1-Minute OHLC Candlestick Data

2. High-Frequency 1-Second Tick Data

3. Custom Timeframe Candlesticks

4. Historical Data Retrieval

Analysis on Complete Works of Swami Vivekananda

Overview

The Flajolet-Martin Algorithm

Variance Reduction (Stochastic Averaging)

Processing Pipeline

Key Insights