LLMs4OL 2026 — Primitive Ontology Extraction System

Team: XuanJi  |  Challenge variant: Task A  | 

An end-to-end, LLM-driven ontology-learning system for the LLMs4OL 2026 Task A (primitive ontology extraction from text). Given a document, the system extracts a set of primitive-ontology-triples of the form [head, relation, tail], covering instance-of, is-a (subclass-of), and non-taxonomic relations. It is built as a 5-step pipeline on top of the OntoLearner framework.

flowchart TD
    A["Input document<br/>(Title + Content)"] --> B["Step 1: Concept &amp; Type Extraction"]
    B --> C["Step 2: Term Typing<br/>(instance-of)"]
    C --> D["Step 3: Taxonomy Discovery<br/>(is-a / subclass-of)"]
    D --> E["Step 4: Non-Taxonomic<br/>Relation Extraction"]
    B --> F["Few-shot retrieval<br/>(TF-IDF, optional adaptive)"]
    F --> B
    C --> G["Step 5: Deterministic Post-processing<br/>(alias normalization, disjoint rules,<br/>entity alignment, FoodEx2/catalog cleanup)"]
    D --> G
    E --> G
    G --> H["Output: primitive-ontology-triples<br/>[head, relation, tail]"]
    style H fill:#cfe,stroke:#393

Loading

---

1. Environment Setup

• Python: ≥ 3.10 (developed on 3.14)

• Install dependencies:

  pip install -r requirements.txt

  ontolearner is installed from source at the pinned commit (the PyPI release lags at 0.3.0 and lacks the v1.6.0 data_structure API this system depends on). See requirements.txt for details.

• Configure API access by copying the example env file and filling in your key:

  cp .env.example .env
  # then edit .env: set OPENAI_API_KEY, OPENAI_BASE_URL and OPENAI_MODEL

  A single OpenAI-compatible provider is supported (see model_config.py): point OPENAI_BASE_URL at any Chat Completions-compatible endpoint and set OPENAI_MODEL to the model id. The LLM_THINKING toggle controls whether the model emits a chain-of-thought (auto to forward thinking, off to explicitly disable it).

---

2. Running the System

The primary entry point is main.py, which supports five modes. All modes take data paths as arguments (provide your own train/test files).

Mode	Command
Generate submission	python main.py --mode submit --train train.json --test test.json --output submission.json
Evaluate (needs gold labels in test)	python main.py --mode eval --train train.json --test test.json --output predictions.json
Predict single doc	python main.py --mode predict --predict "Title: ...\n\nContent: ..."
Validate a submission file	python main.py --mode validate_submission --submission-file submission.json
Train (build few-shot index)	python main.py --mode train --train train.json

Optional flags: --provider openai, --llm <model_id>, --max-few-shot <N>, --adaptive (TF-IDF adaptive few-shot retrieval), --max-train, --max-test, --max-new-tokens, --context-window.

Auxiliary entry points

• batch_submit.py — parallel batch submission generator (multiprocessing) with resume support and an embedded evaluation report. Useful for running large test sets.

  python batch_submit.py --train train.json --test test.json \
      --output submission.json --report report.json --workers 4

• verify_triples.py — QA tool that checks each predicted triple against the source text and applies keep/fix/delete verdicts to produce a corrected submission + error log CSV.

  python verify_triples.py --source test_input.json --submission submission.json \
      --out-submission corrected.json --out-csv errors.csv --backend llm

---

3. Data Format (LLMs4OL 2026 Task A)

• Input test sample: {"id": <str>, "context": "Title: <title>\n\nContent: <text>"} (gold triples may be absent for the real test set).
• Output submission item: {"id": <str>, "primitive-ontology-triples": [[head, rel, tail], ...]}

sample_data.json (included) shows three labeled examples for reference.

---

4. Module Reference

Script	Purpose
main.py	Primary CLI entry point (train / predict / eval / submit / validate_submission).
orchestrator.py	EndToEndPipeline — orchestrates the 5-step extraction + deterministic post-processing.
step_extractors.py	LLM calling layer and the five extraction steps; few-shot prompt assembly.
data_converter.py	Converts between LLMs4OL JSON triples and OntoLearner OntologyData structures; few-shot document selection.
model_config.py	Centralized OpenAI-compatible provider/model configuration read from .env.
evaluator.py	Local precision/recall/F1 metrics with phrase-level + plural-variant concept matching.
official_metrics.py	Official graph-similarity metrics (edge F1, neighborhood/taxonomy similarity).
submission.py	Builds and validates the official Task A submission payload.
concept_matcher.py	Concept extraction (strips catalog wrappers like 14880 - X (efsa foodex2)).
disjoint_rules.py	Detects disjoint entity pairs (e.g. multicellular vs acellular) for post-processing.
triple_verifier.py	Triple verification primitives: verdict parsing, apply keep/fix/delete, CSV logging.
batch_submit.py	Parallel batch submission runner with resume + report generation.
verify_triples.py	Triple-verification CLI (manual or LLM-judge backends).
test_suite.py	Unit tests for the offline core modules (see below).

---

5. Unit Tests

python test_suite.py

test_suite.py contains 38 unit tests covering the deterministic, API-free core modules: submission payload build/validate, concept matching, disjoint rules, model config, evaluation metrics (including the official graph-similarity example), the LLMs4OL↔OntoLearner data converter, the FiveStepExtractor config wiring, and triple-verifier parsing/IO.

Note on the filename: the test file is named test_suite.py rather than unittest.py. A file literally named unittest.py placed on the run path shadows Python's standard-library unittest module; because the OntoLearner dependency (via torch) performs import unittest internally, the shadow caused a circular import failure. Renaming to test_suite.py resolves the conflict while keeping the same content and intent.

---

6. License

This project is licensed under the GNU General Public License v3.0 or later (GPL-3.0-or-later). The complete license text is included in the LICENSE file shipped with this package.