Keyword Extraction & Benchmarking Suite

A multi-language research repository for comparing and benchmarking keyword, reference, and entity extraction methods alongside experimental C/C++20 algorithms.

The repository is cleanly separated into three layers:

1. cpp/ — High-performance C++20 libraries and tools, divided into two decoupled components:
   • cpp/md_to_yaml/: Standalone Markdown $\to$ Document AST $\to$ YAML parser and validator (md4c + yaml-cpp).
   • cpp/keyword_extractor/: Standalone tokenization, statistical scoring (TF-IDF, BM25), and graph-based ranking (TextRank, PositionRank) engine with pybind11 bindings (keyword_extractor_cpp) and zero dependency on Markdown/YAML libraries.
2. python/ — Installable analytics Python package (pyproject.toml) for multi-method evaluation, Spearman rank correlation, consensus set overlap, report generation, and visualization.
3. notebooks/ — Pure Jupyter notebooks for side-by-side comparative benchmarking and single-method exploration.

---

Scoring Algorithms

1. TF-IDF (Term Frequency–Inverse Document Frequency)

Each token's score is its normalized relative frequency within the document:

$$\text{tf}(t) = \frac{\text{count}(t)}{|D|}$$

2. BM25 (Best Matching 25)

Non-linear term frequency saturation component that prevents high-frequency terms from dominating linearly:

$$\text{score}(t) = \frac{\text{count}(t) \cdot (k_1 + 1)}{\text{count}(t) + k_1}$$

3. TextRank

Graph-based unsupervised ranking algorithm adapted from PageRank over a sliding-window co-occurrence graph (default window: 5 tokens) with uniform teleportation bias:

$$S(v_i) = (1 - d) \cdot \frac{1}{N} + d \sum_{v_j \in \text{In}(v_i)} \frac{w_{ji}}{\sum_{v_k} w_{jk}} S(v_j)$$

4. PositionRank

Position-aware variant of TextRank that biases the teleportation vector toward terms appearing earlier in the document:

$$b(v_i) = \frac{1}{\text{pos}(v_i) + 1}$$

---

Project Structure

SpectralTDAKeywordIndexer/
├── cpp/                                  # C/C++20 core libraries, CLI, and Python bindings
│   ├── md_to_yaml/                       # Standalone Markdown -> Document -> YAML parser
│   │   ├── include/                      # document.h, reader.h, writer.h, md_reader.h, yaml_writer.h
│   │   └── src/                          # document.cpp, md_reader.cpp, yaml_writer.cpp, main.cpp
│   └── keyword_extractor/                # Standalone Keyword Extraction & Scoring Engine
│       ├── include/keyword_extractor/    # tokenizer.h, statistical.h, graph_rank.h
│       ├── src/                          # tokenizer.cpp, statistical.cpp, graph_rank.cpp
│       └── bindings/pybind_module.cpp    # pybind11 module (`keyword_extractor_cpp`)
├── python/                               # Installable Python analytics package
│   ├── pyproject.toml
│   └── analytics/
│       ├── config.py                     # Method registry (METHODS), palette, and paths
│       ├── loader.py                     # Document/YAML text loader & C++ evaluation bridge
│       ├── overlap.py                    # Consensus sets, unique terms, Spearman correlations
│       ├── report.py                     # Markdown report & console summary formatters
│       ├── cli.py                        # CLI entry point (`python -m analytics.cli`)
│       └── charts/                       # Modular plotting functions (bar, radar, scatter, heatmap, graph, exploration)
├── notebooks/                            # Interactive Jupyter notebooks
│   ├── analysis.ipynb                    # Composite 4-method comparative analysis
│   ├── tfidf_exploration.ipynb
│   ├── bm25_exploration.ipynb
│   ├── textrank_exploration.ipynb
│   └── positionrank_exploration.ipynb
├── data/
│   ├── input/                            # Source .md documents
│   └── output/                           # Generated YAML, reports, charts/, notebooks/*.html
└── Makefile

---

Build & Run

Build C++ binaries, Python bindings, and parse Markdown to YAML

make all

Bootstrap Python virtual environment and install analytics package

make venv

Run the full comparative analysis pipeline

make analysis

Builds keyword_extractor_cpp.so, runs all 4 scoring methods on the input document, saves charts to data/output/charts/, and generates data/output/01-hamming_metric_analysis_report.md.

Execute all notebooks headlessly to HTML

make notebooks

Executes all notebooks in notebooks/ and writes HTML reports to data/output/notebooks/.

Launch interactive Jupyter Notebook server

make notebook

Clean targets

make clean        # Remove build/ directory
make clean-py     # Remove Python caches and compiled .so extension
make clean-data   # Remove generated YAML, reports, charts, and HTML notebooks
make clean-all    # All of the above

---

Python API Usage

High-level analytics package

from analytics.config import TOP_N, DEFAULT_DOC_PATH
from analytics.loader import process_document
from analytics.overlap import compute_overlap, spearman_table

doc_name, df_all, df_tf, df_bm, df_tr, df_pr, G = process_document(DEFAULT_DOC_PATH)
overlap = compute_overlap(df_tf, df_bm, df_tr, df_pr, TOP_N)
correlations = spearman_table(df_all)

Low-level C++ keyword_extractor_cpp bindings

from analytics import keyword_extractor_cpp as cpp

tokens = cpp.extract_tokens_from_file("data/input/01-hamming_metric.md")
tfidf  = cpp.calculate_tf_idf(tokens)
bm25   = cpp.calculate_bm25(tokens, k1=1.5, b=0.75)
tr     = cpp.textrank(tokens)
pr     = cpp.positionrank(tokens)
edges  = cpp.get_cooccurrence_edges(tokens)