A modular, microservice-based AI application for automated HSN code classification using RAG. Features plug-and-play support for cloud APIs, custom endpoints, and local agent providers.
NexusHSN-Classifier leverages advanced AI and a Retrieval-Augmented Generation (RAG) pipeline to correctly classify products according to the Harmonized System (HS) Nomenclature and the General Rules of Interpretation (GRI).
- AI-Powered Classification: Uses advanced LLMs (like Gemma 4) to reason through complex product descriptions.
- RAG Architecture: Searches across 2,200+ HS heading chunks from 97 chapters using BM25 and LLM-reranking.
- Microservice Design: Clean, modular structure makes it easy to maintain and scale.
- Agent Provider Pattern: Switch seamlessly between OpenRouter, custom API endpoints, and local script agents via environment variables.
- Real-Time UI: Includes a premium, glassmorphism frontend that streams the agent's decision-making process in real time using SSE.
- Runs Out of the Box: The corpus and a prebuilt index cache ship with the repository — no scraping or index build step before the first query.
The system is divided into several clear components:
- Frontend: HTML/CSS/JS interface using Server-Sent Events (SSE) for real-time progress updates.
- Backend API: FastAPI server handling requests, routing, and SSE streaming.
- Retrieval Pipeline:
indexer.pyandretriever.pybuild and query a local BM25 index over the HS chapters. - Agent Providers:
providers/package abstracts the LLM interactions. It supports OpenRouter out of the box, as well as custom webhooks and local agents. - GRI Agent Workflow:
classifier.pyandreranker.pyuse the configured agent to filter candidates and execute a 7-step GRI methodology.
Retrieval runs entirely over the markdown corpus in data/ — there is no external database and no network call during retrieval.
- Source: the WCO Harmonized System Explanatory Notes, 7th Edition (2022), converted to markdown. Each chapter file carries YAML frontmatter (
chapter,title,section,section_title,volume,source). - Corpus: 97 chapters plus
chapter_00_GIR.md(the General Rules of Interpretation), alongsideheading_index.md(659 heading lookups),chapter_routing.md(100 keyword→chapter routing entries),cross_reference_map.md,classification-core.mdandcustom-classification.md. - Index:
indexer.pysplits each chapter at heading boundaries (XX.XX) and emits typed chunks —heading,chapter_notes,generalandfull. The build yields 2,278 chunks, of which 2,183 are heading chunks, across 97 chapters, cached tobackend/.index_cache.json(committed, so first run is instant).POST /rebuild-indexforces a fresh build.
classifier.py runs seven steps and streams each one to the browser over SSE:
| # | Step | What happens |
|---|---|---|
| 1 | Keyword Routing | The product description is matched against chapter_routing.md keywords to pick candidate chapters. |
| 2 | BM25 Retrieval | Okapi BM25 scores every chunk. Routed chapters get a 1.5× boost and heading chunks a 1.1× boost. Top 40 survive. |
| 3 | LLM Reranking | reranker.py sends the top 25 candidates to the LLM for a 0–10 relevance score, then blends it with BM25 (0.3 × normalised BM25 + 0.7 × LLM score). Top 10 survive. If the LLM call fails, it falls back to BM25 order. |
| 4 | Context Assembly | The GIR text, the chapter notes for the surviving chapters, and the 6 top heading chunks are concatenated into one prompt. |
| 5 | GRI Classification | One LLM call (temperature 0.15, 8192 max tokens) applies GRI 1–6 in order and returns a strict JSON verdict. |
| 6 | Confidence Scoring | Deterministic, code-side calibration: the HSN code format is regex-validated, and confidence is multiplied down for a malformed code (×0.7), missing verbatim quotes (×0.85) or missing GRI steps (×0.8). The HIGH/MEDIUM/LOW label is then recomputed at the 0.8 / 0.5 thresholds. |
| 7 | Output Generation | The result is rendered as a formal Classification Opinion (falling back to a code-built template if the model did not supply one). |
Note on retrieval: this is a lexical RAG pipeline. BM25 is implemented from scratch in
retriever.py(IDF, tokenisation and scoring withk1=1.5,b=0.75) — there is no embedding model, no vector store and no third-party search dependency. Semantic judgement is supplied by the LLM reranker in step 3 rather than by dense vectors.
HSN-Classifier-LLM/
├── .env # Configuration variables (git-ignored)
├── .env.example # Template for environment variables
├── requirements.txt # Python dependencies
├── start.bat # Windows launcher script
├── README.md # This file
├── backend/
│ ├── server.py # FastAPI application entry point
│ ├── config.py # Configuration loader (dotenv)
│ ├── classifier.py # Main GRI classification workflow
│ ├── reranker.py # LLM-based reranking logic
│ ├── retriever.py # BM25 search logic
│ ├── indexer.py # Chunking and caching logic
│ ├── models.py # Pydantic models
│ ├── .index_cache.json # Prebuilt search index (committed)
│ └── providers/ # AI Agent Providers
│ ├── __init__.py
│ ├── base.py # AgentProvider abstract base class
│ ├── factory.py # Factory to get the correct provider
│ ├── openrouter.py # OpenRouter API implementation
│ ├── custom_api.py # Custom API implementation
│ └── local_agent.py# Local script execution implementation
├── frontend/
│ ├── index.html # Main web UI
│ ├── style.css # Styling (Glassmorphism)
│ └── app.js # Client-side logic & SSE handling
└── data/ # HS knowledge base
├── chapters/ # Chapter markdown files (00_GIR + 01–97)
├── chapter_routing.md # Keyword → chapter routing table
├── heading_index.md # Heading → chapter lookup
├── cross_reference_map.md
├── classification-core.md
└── custom-classification.md
- Backend: Python 3.10+, FastAPI, Uvicorn, HTTPX, Pydantic, python-dotenv
- Frontend: HTML5, Vanilla JS, CSS3
- AI/RAG: Okapi BM25 (implemented in-repo, no search dependency) + LLM reranking, OpenRouter API (Gemma 4 31B by default)
- Clone the repository and enter it:
git clone https://github.com/GODOSTROYER/HSN-Classifier-LLM.git cd HSN-Classifier-LLM - Install the required dependencies:
The five runtime dependencies are
pip install -r requirements.txt
fastapi,uvicorn[standard],httpx,pydanticandpython-dotenv. There is no GPU requirement — inference runs on whichever provider you configure, not on your machine.
Copy the example environment file and configure it:
cp .env.example .envEdit .env to include your API keys and configuration preferences. .env lives at the repository root and is git-ignored — never commit it.
| Variable | Used when | Purpose |
|---|---|---|
AGENT_PROVIDER |
always | openrouter (default), custom or local. |
OPENROUTER_API_KEY |
openrouter |
Your OpenRouter key. |
OPENROUTER_BASE_URL |
openrouter |
Chat-completions endpoint. |
MODEL_ID |
openrouter |
Model slug. Defaults to google/gemma-4-31b-it:free. |
CUSTOM_AGENT_API_URL |
custom |
Your OpenAI-compatible endpoint. |
CUSTOM_AGENT_API_KEY |
custom |
Optional bearer token. |
LOCAL_AGENT_PATH |
local |
Path to the executable agent script. |
HOST / PORT |
always | Bind address. Defaults to 127.0.0.1:8899. |
Retrieval tuning (BM25_TOP_K, RERANK_TOP_K, FINAL_CONTEXT_CHUNKS, CHAPTER_BOOST_FACTOR) lives in backend/config.py rather than the environment.
You can run the project using the provided batch script or via the command line.
Option 1: Using the Batch Script (Windows)
start.batOption 2: Using the Command Line
cd backend
python server.pyThen open your browser and navigate to http://127.0.0.1:8899.
Run the server from inside
backend/— the modules import each other by flat name (from config import ...), so launching from the repository root will fail on import.
On startup the server loads the index cache and prints the chunk and heading counts; the frontend reads the same numbers from /health to populate its stats bar.
The application uses the AGENT_PROVIDER environment variable to determine which AI agent to use. Modify your .env file to switch providers:
AGENT_PROVIDER=openrouter
OPENROUTER_API_KEY=your_api_key_here
MODEL_ID=google/gemma-4-31b-it:freeAGENT_PROVIDER=custom
CUSTOM_AGENT_API_URL=http://localhost:8000/api/agent
CUSTOM_AGENT_API_KEY=optional_api_keyAGENT_PROVIDER=local
LOCAL_AGENT_PATH=./agents/local-agent.pyThe application uses an Adapter/Provider pattern, making it extremely easy to add new agents.
- Create a new file in
backend/providers/(e.g.,openai_provider.py). - Implement the
AgentProviderinterface frombase.py:from .base import AgentProvider class OpenAIProvider(AgentProvider): async def generate_response(self, system_prompt: str, user_prompt: str, ...) -> str: # Implementation here pass async def generate_streaming_response(self, system_prompt: str, user_prompt: str, ...) -> AsyncGenerator[str, None]: # Implementation here pass
- Register your provider in
backend/providers/factory.py:elif provider_name == "openai": from .openai_provider import OpenAIProvider return OpenAIProvider()
- Update your
.envfile to use the new provider:AGENT_PROVIDER=openai.
You can bypass the frontend and interact directly with the FastAPI backend.
| Method | Route | Purpose |
|---|---|---|
GET |
/ |
Serves the web UI. |
GET |
/health |
Health check plus index stats. |
GET |
/stats |
Index statistics only. |
POST |
/classify |
Runs the workflow; returns an SSE stream. |
POST |
/rebuild-index |
Deletes the cache and rebuilds the index from data/. |
Classify a Product:
curl -X POST http://127.0.0.1:8899/classify \
-H "Content-Type: application/json" \
-d '{"product_description": "Live cattle for breeding purposes"}'Note: This endpoint returns a Server-Sent Events (SSE) stream. Each event carries the step number, name, status, a human-readable detail line and a progress_pct; the stream ends with a complete event holding the full ClassificationResult, then [DONE]. product_description must be 3–2000 characters.
Check Server Health:
curl http://127.0.0.1:8899/healthRebuild the index after editing data/:
curl -X POST http://127.0.0.1:8899/rebuild-index- Advisory, not authoritative. The output is a drafting aid. Every Classification Opinion should be reviewed by a qualified customs professional before it is relied on — the confidence score is a heuristic, not a guarantee, and anything below 0.7 is explicitly flagged for manual review.
- Lexical retrieval. BM25 matches words, not meaning. A product described in vocabulary that does not appear in the Explanatory Notes may never surface the right heading as a candidate, and step 5 can only classify into what steps 1–4 retrieved.
- Corpus scope. The knowledge base is the HS Explanatory Notes (7th Edition, 2022). It is not updated automatically, and it does not include Indian tariff schedules, duty rates, notifications, CAAR rulings or case law — so the model's 8-digit national-level suggestions go beyond what the corpus can support.
- No benchmark. This repository does not ship an evaluation set or accuracy figures; none are claimed.
- Single-shot reasoning. Each classification is one LLM call with no self-verification pass, and there is no caching of previous verdicts.
- Provider-dependent quality. Results vary with
MODEL_ID. Free-tier models are rate-limited and may return malformed JSON, in which case the run is reported with LOW confidence. - Not hardened for production. CORS is wide open, there is no authentication or rate limiting, and state is in-process — the server is built to run locally.
- Fork the repository.
- Create a feature branch (
git checkout -b feature/amazing-feature). - Commit your changes (
git commit -m 'Add amazing feature'). - Push to the branch (
git push origin feature/amazing-feature). - Open a Pull Request.
Please ensure your code adheres to clean code principles and does not hardcode configuration variables.
Made by Arnav Bule.
- Portfolio — arnavbule.in
- GitHub — @GODOSTROYER
This project is licensed under the MIT License. See the LICENSE file for details.
The HS Explanatory Notes text in data/ is the copyright of the World Customs Organization and is included here for reference use only.