Violas

In-Memory Vector Group System for Semantic Search.

Violas is an in-memory vector group system for semantic search. Modern vector databases typically store each item as an embedding and retrieve approximate nearest neighbors in the embedding space. This is effective when embedding proximity is enough, but it treats each vector as an independent searchable object and leaves richer semantic information outside the retrieval model.

Violas addresses this gap with VectorGroup, a semantic-first storage abstraction that keeps semantic entities, diverse representations, member embeddings, and object-level dependencies together. On top of this structure, HDMG indexes micro-clusters with heterogeneous traversal edges to support efficient semantic search.

Outline

• Why Violas?
• Typical Cases
• How It Works
• Experiments
• API Overview
• Quick Start
• Installation Options
• Repository Layout
• Reproducing Benchmarks

Why Violas?

From Vector Search to Semantic Search

Existing vector search systems primarily organize and access data through embedding proximity. This design is scalable, but it struggles when the query requires semantic information that cannot be reduced to nearest-neighbor distance alone. Filters, rerankers, and application-side dependency recovery can help, but they still depend on the candidates returned by flat vector search.

Following the paper's semantic-search formulation, Violas extends vector search to support results that:

• stay consistent with the intended semantic entity, such as a class, topic, or document
• cover diverse representations within that entity
• include dependent objects, such as adjacent chunks, temporal neighbors, or cross-modal evidence

The examples below show the practical retrieval failures that motivate these requirements and the corresponding retrieval capabilities Violas supports:

• Semantic-consistent retrieval: retrieve results that stay consistent with the intended semantic entity. See Entity Mismatch.
• Diversity-driven retrieval: cover multiple poses, chunks, or local modes within the same entity. See Diversity Loss.
• Dependency-expanded retrieval: expand a hit to linked context or temporal neighbors. See Dependency Loss.
• Cross-modal retrieval: retrieve paired evidence across modalities. See Cross-Modal Retrieval.

See Vector Group for how Violas stores this structure and HDMG Indexing for how it is indexed efficiently.

Typical Cases

The examples below correspond to the native capabilities evaluated in the paper: semantic-consistent retrieval, diversity-driven retrieval, dependency-expanded retrieval, and cross-modal retrieval. We show them through common flat-retrieval failure modes: entity mismatch, diversity loss, dependency loss, and broken cross-modal retrieval.

Entity Mismatch

Flat nearest-neighbor search can drift into a different semantic entity. Violas routes by semantic entity first, then ranks local members.

Some wrong neighbors look plausible because they share coarse traits with the query, but they still violate the intended entity. More interestingly, some nearest neighbors are embedding-close without a clear category-level or visual relation to the query.

Rhinoceros: large-animal shape is not enough	Anchor: abstract shape should not override entity meaning
Pigeon: fine-grained category identity is not preserved	Stegosaurus: semantic identity should dominate
Ant: entity identity is not preserved	Wristwatch: embedding proximity is not enough

Diversity Loss

Even when the entity is correct, top results can be too redundant. Violas uses representative local regions to expose different poses, viewpoints, chunks, or scene layouts under the same semantic entity.

Bird: one entity, multiple useful views	Airplane: cover flight configurations and scenes
Leopard: cover poses and body layouts	Gramophone: cover structural variations and viewing conditions

Dependency Loss

Some useful answers require linked objects rather than a single nearest chunk. In these OHSUMED cases, the middle segment is used as the query, while the desired answer is the surrounding same-document evidence chain. Flat top-k retrieval often returns isolated snippets that share surface vocabulary but lose the dependency between setup, evidence, and conclusion.

Norfloxacin: trial evidence should be retrieved as a chain

Candidiasis: treatment choices need patient context

Cross-Modal Retrieval

Some retrieval tasks need paired evidence across modalities, such as a caption and its corresponding image. Violas keeps these modality links inside the same retrieval object instead of reconstructing them after a flat vector search.

Sheet cake: text query aligned with image evidence	Motorbike: scene-level text and image agreement
Girl with cat: paired visual and caption evidence	Candle: multimodal evidence keeps context intact

How It Works

Vector Group

VectorGroup is the semantic-first storage abstraction behind Violas. It stores the semantic information required by semantic search together with the objects and embeddings. Each vector group is organized as a three-level structure:

Layer	Role
Group header	Stores the semantic key and group-level semantic vector, giving retrieval an entity-level entry point.
Micro-clusters	Organize member objects that exhibit diverse local representations within the same entity.
Members	Store concrete objects, embeddings, metadata, and object-level dependency relations.

This lets one entity, such as a class, document, event, or multimodal item, be managed as one retrieval object instead of a loose collection of independent embeddings.

Retrieval Modes

The same VectorGroup structure supports multiple retrieval modes by changing the retrieval scope, output granularity, and expansion policy. A query first enters candidate groups through an explicit key or a group-level semantic vector, then selects micro-clusters or members with a mixed semantic-embedding score. The query-controlled parameter beta balances semantic consistency and embedding proximity. When linked objects are required, selected members are expanded through stored relations.

Paper capability	Primary APIs	Typical output
Semantic-consistent retrieval	search_entity(...), search(..., key=...)	Results scoped to the intended entity.
Diversity-driven retrieval	create_cluster(...), search_diverse(...), search_with_representative_rerank(...)	Results composed across local forms of one entity.
Dependency-expanded retrieval	add_relation(...), search_dependency(...), search_with_contextual_vectors(...)	A seed hit plus linked context or evidence.
Cross-modal retrieval	add_pair_relation(...), search_modal(...), search_multimodal(...)	Paired image-text or multi-view evidence.

HDMG Indexing

On top of VectorGroup, Violas builds HDMG (Hierarchical Diversified Micro Cluster Graph), an index for efficient semantic search over vector groups. HDMG indexes micro-clusters because they are the main searchable units and preserve both semantic and embedding coherence. It uses heterogeneous traversal edges:

• embedding edges preserve proximity between micro-clusters
• semantic edges preserve reachability within the same or semantically related vector groups
• query-time navigation follows the tradeoff between semantic consistency and embedding proximity

Experiments

Violas is evaluated on six image and text datasets. Following the paper, we compare Violas with three popular vector databases: Milvus, Qdrant, and Chroma. We also study the performance differences between Violas and Violas without HDMG (w/o HDMG). For system performance, we measure query latency and the operation time of data and index maintenance.

_{Comparison of average Mixed Recall@3, Mixed NDCG@3, and query latency under different β.}

_{Average operation time for data and index maintenance.}

What this shows:

• Violas improves average Mixed Recall@3 and Mixed NDCG@3 by 40.3% and 30.0% over representative vector-search baselines.
• Violas achieves the lowest query latency and maintenance operation time.

The benchmark environment uses an Intel Xeon Platinum 8350C CPU (2.60 GHz), 16 CPU cores, and 64 GB RAM. Image embeddings use CLIP ViT-B/32 and text embeddings use Sentence-Transformers all-MiniLM-L6-v2 in the benchmark pipelines.

More details:

• Benchmark Results
• Benchmark Notes
• Data Format

API Overview

Violas exposes the high-level retrieval capabilities described in the paper, and backs them with concrete system APIs for object lifecycle management, index construction, relation maintenance, query execution, and inspection. The goal is to make the research abstraction usable as an implemented retrieval system, not only as paper pseudocode.

Implemented System Surface

Area	APIs	What it covers
Create / insert	create_group(...), insert(...), insert_object(...), add_vector(...), add_vector_list(...)	Create semantic groups and insert individual or batched objects.
Read / access	get(...), get_all_keys(...), get_group_by_name(...), get_group_by_id(...)	Access stored keys, metadata, groups, and group contents.
Update / move	update(...), update_object(...), assign(...), assign_object(...)	Update object embeddings or metadata and move objects across groups.
Delete	delete(...), delete_object(...)	Remove stored objects by reference.
Index construction	build_index(...), build_rep_index(...), build_single_index(...), set_key_vectors(...), build_hdmg(...), get_last_hdmg_search_stats(...)	Build member indexes, representative indexes, semantic key state, and HDMG.
Relation management	VectorRef, VectorRelation, add_relation(...), remove_relation(...), add_pair_relation(...), add_tree_relation(...), get_relations(...)	Maintain context, temporal, hierarchy, dependency, and multimodal links.
Query execution	search(...), search_entity(...), search_diverse(...), search_dependency(...), search_modal(...), search_hdmg(...)	Run standard, scoped, diverse, relation-aware, multimodal, and HDMG-backed retrieval.
Inspection	get_all_keys(...), get_group_by_name(...), get_statistics(...), analyze_relationships(...)	Inspect stored keys, groups, index state, and relation coverage.

Example: object lifecycle and relation setup:

ref = vm.insert_object("paper-001", query_vec, {"text": "middle segment"})
ctx = vm.insert_object("paper-001", context_vec, {"text": "previous segment"})
vm.add_relation(ref, ctx, relation_type="context")
vm.update(ref, description={"section": "clinical evidence"})

Example: build indexes and run structured retrieval:

vm.set_key_vectors(key_vectors)
vm.build_index()
vm.build_hdmg()

results = vm.search_hdmg(query_vector, query_key_vector=semantic_embedding, top_k=5)
context = vm.search_dependency(query_vector, relation_types=["context"], top_k=5)

For the fuller method list and retrieval patterns, see docs/api.md.

Quick Start

Install the package in editable mode:

python -m venv .venv
source .venv/bin/activate
pip install -e .

If you want the packaged core library from PyPI later, the distribution name is violas.

Run the minimal example. It uses synthetic vectors and does not require any dataset, embedding model, or external vector database:

python examples/minimal_vectormap.py

Minimal API usage:

import numpy as np

from violas import VectorMap

vectors = [np.random.rand(4) for _ in range(5)]
vm = VectorMap()
vm.create_group(
    key="example",
    group_name="demo",
    representative=np.mean(vectors, axis=0),
    rep_description="demo representative",
    vectors=vectors,
    descriptions=[{"text": f"item {i}"} for i in range(5)],
    vector_type="demo",
    group_type="synthetic",
)

query = np.random.rand(4)
results = vm.search_entity(query, key="example", top_k=3)
for rank, result in enumerate(results, start=1):
    text = result.group.descriptions[result.vector_idx]["text"]
    print(f"{rank}. key={result.key} distance={result.distance:.4f} text={text}")

Installation Options

For quick library experiments, install only the minimal dependencies shown in Quick Start. If you only need FAISS-backed local indexing in the core package, install the optional FAISS extra:

pip install -e ".[faiss]"

The full benchmark suite uses additional embedding models and external vector database baselines. For the full benchmark environment:

pip install -r requirements.txt
pip install -e .

The benchmark dependencies include optional-heavy packages such as CLIP, Sentence-Transformers, FAISS, Milvus Lite, Qdrant, and Chroma because the benchmark suite compares Violas with external vector database baselines.

Repository Layout

Violas/
  violas/
    storage/       # VectorMap, VectorGroup, relation helpers
    core/          # feature helpers, recall utilities, baseline indexes
  benchmarks/      # six benchmark pipelines plus diversity cases
  examples/        # small runnable examples
  scripts/         # benchmark wrapper scripts
  docs/            # API notes, data formats, benchmark results, README figures

Reproducing Benchmarks

The shell scripts assume a workspace where Violas/ and dataset/ are siblings. Override the dataset variables if your data lives elsewhere.

bash scripts/run_bench_all.sh
bash scripts/run_bench_vision.sh
bash scripts/run_bench_text.sh
bash scripts/run_diversity_case.sh

Vision dataset variables:

• CALTECH_ROOT
• CUB_ROOT
• COCO_ROOT
• COCO_JSON

Text dataset variables:

• NEWS20_ROOT
• OHSUMED_ROOT
• YAHOO_ROOT

Saved artifacts are written under outputs/<benchmark>/ by default. External vector database baselines are disabled by default for portability. To enable them:

export VIOLAS_ENABLE_EXTERNAL_DBS=1

License

Apache License 2.0 - See LICENSE