繁體中文 | English
A config-driven NLP annotation platform with built-in dataset analytics, designed for academic research labs.
Launch annotation tasks through simple config files — no custom code required. Built-in dataset statistics eliminate post-hoc analysis scripts.
Existing annotation platforms such as Label Studio are powerful but come with significant friction for academic research teams:
- Complex setup: Deploying Label Studio requires configuring a dedicated server, which is time-consuming and demands engineering effort beyond the scope of most research teams.
- Fragmented workflows: Task configuration, labeling, and dataset analysis are often handled by separate tools or ad-hoc scripts, forcing researchers to repeatedly build one-off systems from scratch.
- No dataset quality visibility: Existing tools provide no built-in dataset statistics, forcing researchers to write analysis scripts after each labeling round.
Label Suite aims to eliminate these pain points by providing a lightweight, config-driven annotation platform that any NLP research team can launch with minimal setup.
Standalone HTML version: docs/site-map/site-map.en.html — open it locally in a browser to see hover descriptions for each page.
- Config-driven Task Launch: Define NLP annotation tasks through simple YAML/JSON config files — no custom code required. Supports Single Sentence, Sentence Pairs, Sequence Labeling, and Generative Labeling.
- Dry Run / Official Run Mechanism: Validate labeling interfaces and configurations before formal data collection, with strict data isolation between modes.
- Built-in Dataset Analytics: Automatically computes and surfaces #Sentence, #Token, and #Label statistics in real time for quality monitoring.
- High Usability UI: Intuitive labeling interface designed for non-engineering annotators.
-
Config-Driven and General-Purpose Launch annotation tasks for diverse NLP task types through a simple configuration file — no custom code required for each new task.
-
Config-Driven Task Workflow Turns task setup, dry-run validation, official labeling, and dataset analysis into one repeatable workflow for academic NLP labs.
-
Built-in Dataset Analytics Eliminates the need for post-hoc analysis scripts by automatically computing and surfacing dataset statistics within the portal.
-
Integrated Annotation Workflow Combines task configuration, data labeling, and dataset analysis in a single platform, replacing fragmented multi-tool pipelines.
-
Low Entry Barrier Designed for researchers and annotators without deep engineering backgrounds — spin up a labeling server in minutes, not days.
-
Publicly Viewable Research Preview The source code is publicly viewable for academic evaluation and citation; usage rights are governed by the repository's LICENSE notice.
This project is positioned as a Demo Paper, with its core value in:
- Lowering the barrier for NLP research teams to set up annotation environments.
- Providing a reusable annotation toolkit that addresses the practical inefficiency of ad-hoc workflows in academic labs.
| Layer | Technology |
|---|---|
| Frontend | React + TypeScript + Vite |
| Backend | FastAPI (Python) |
| Database | SQLite (quick start) / PostgreSQL (production) |
| Cache / Queue | Redis |
| Async Tasks | Celery |
| Testing | Playwright (E2E) + pytest |
Note: This tech stack reflects the current design decision; implementation is tracked in Phase 3.
SQLite quick-start warning: The default SQLite tier is intended for single-user local demos and evaluation only. It does not support concurrent writes and is not recommended for multi-user production deployments. Set
DATABASE_URL=postgresql+asyncpg://...to switch to the production-grade PostgreSQL tier (see ADR-024).
| Feature | Label Studio | Label Suite |
|---|---|---|
| Easy setup (no server config) | ✗ | ✓ |
| Config-driven task definition | Partial | ✓ |
| Built-in dataset statistics | ✗ | ✓ |
| Dry Run / Official Run isolation | ✗ | ✓ |
| Designed for NLP research teams | ✓ | ✓ |
| Open source | ✓ | ✗ (Research Preview — viewable, all rights reserved) |
%%{init: {
"theme": "base",
"themeVariables": {
"cScale0": "#3B82F6",
"cScale1": "#8B5CF6",
"cScale2": "#10B981",
"cScale3": "#F59E0B",
"gridColor": "#E5E7EB",
"todayLineColor": "#EF4444",
"fontSize": "14px"
}
}}%%
gantt
title Label Suite — Research Roadmap
dateFormat YYYY-MM-DD
axisFormat %b %Y
section Phase 1 · Problem Definition
Tool survey (Label Studio) :p1a, 2026-04-01, 2026-06-01
UX interview & pain-point questionnaire :p1d, 2026-05-01, 2026-07-01
Academic paper survey (Related Work) :p1e, 2026-05-01, 2026-08-01
Define system contributions :p1b, 2026-06-01, 2026-07-01
Study Demo Paper examples from target venue :p1c, 2026-07-01, 2026-08-01
section Phase 2 · System Design
Core module planning :p2a, 2026-08-01, 2026-09-01
General-purpose task template design :p2b, 2026-09-01, 2026-11-01
Tech stack documentation :p2c, 2026-10-01, 2026-11-01
Dataset analytics module design :p2d, 2026-11-01, 2026-12-01
Preliminary Related Work draft :p2e, 2026-10-01, 2026-12-01
section Phase 3 · Development & Validation
Project infrastructure & CI :p3a, 2026-12-01, 2027-02-01
Backend — FastAPI + DB + Celery :p3b, 2027-02-01, 2027-07-01
Frontend — React annotation UI :p3c, 2027-04-01, 2027-09-01
Dataset analytics & export features :p3d, 2027-07-01, 2027-11-01
Domain validation & user feedback :p3e, 2027-09-01, 2027-12-01
Mini user study (SUS questionnaire) :p3g, 2027-11-01, 2028-01-01
Demonstration scenarios & demo video :p3h, 2027-11-01, 2028-02-01
section Phase 4 · Paper & Demo
Paper outline & section drafts :p4a, 2028-01-01, 2028-03-01
Advisor review & revision cycle :p4b, 2028-03-01, 2028-04-01
System demonstration preparation :p4c, 2028-03-01, 2028-04-01
- Survey Label Studio and identify pain points in setup, usability, and dataset analytics
- Conduct UX interviews and distribute a pain-point questionnaire to target users (researchers, annotators)
- Survey related academic papers on annotation platforms to establish positioning for the Related Work section
- Define the system's contribution: clarify how Label Suite is simpler and more usable than Label Studio
- Study Demo Paper examples from target venue proceedings to understand structure, length, and demonstration requirements
- Plan core modules: Task Management, Annotation Tasks, Dataset Analysis
- Design general-purpose task templates — ensure the system supports diverse NLP tasks (Single Sentence, Sentence Pairs, Sequence Labeling, Generative Labeling)
- Document and ratify tech stack decision (FastAPI + React + PostgreSQL + Redis + Celery)
- Design dataset analytics module (#Sentence, #Token, #Label, quality monitoring)
- Draft preliminary Related Work notes; confirm no existing system makes the same contribution claim
- Project infrastructure setup (SDD workflow, CI, AI agents)
- Implement frontend annotation interface and backend logic (leverage AI tools to assist development)
- Implement task member coordination through task detail workflows
- Implement Dry Run / Official Run mechanism with strict data isolation
- Implement built-in dataset analytics (#Sentence, #Token, #Label)
- Validate system on domain-specific NLP tasks (e.g., Chinese medical/healthcare, sentiment & psychological analysis)
- Conduct structured mini user study with lab members (SUS questionnaire); document results as paper evidence
- Define 2–3 demonstration scenarios covering core workflows (task launch via config, dry run validation, dataset analysis)
- Capture system screenshots and record a demo walkthrough video
- Draft paper outline and confirm structure with advisor (Introduction, System Overview, Key Features, Demonstration Scenarios, Related Work, Conclusion)
- Write thesis in English to Demo Paper length and format
- Complete advisor review cycle; address all feedback
- Prepare system demonstration to showcase practical impact
- Chinese Medical & Healthcare NLP
- Sentiment & Psychological Analysis
- General NLP annotation tasks (classification, span labeling, etc.)
Prof. Lung-Hao Lee — Natural Language Processing Lab
- Personal Page: lunghao.weebly.com
Research focus: Chinese NLP, text annotation, and language model evaluation.
Copyright © 2026 Sing-Yi Chen (陳欣怡). All rights reserved.
Label Suite is unpublished research software. No permission is granted to copy, modify, publish, distribute, deploy, or create derivative works without prior written authorization from the copyright holder. See LICENSE for the complete notice and CITATION.cff for citation metadata.