Proposal: Include a Hibernate-Based DFS Implementation in the JGit Distribution

[carstenartur]

Hello JGit Community,

JGit already provides the DfsRepository / DfsObjDatabase abstraction for pluggable storage backends. I have built a full database-backed implementation of this API using Hibernate ORM and Hibernate Search, and I would like to discuss including it as an optional module in the JGit distribution.

What This Is

A complete DFS backend that stores Git objects, packs, refs, and reflogs in a relational database:

• HibernateRepository extends DfsRepository
• HibernateObjDatabase extends DfsObjDatabase — stores pack data as BLOBs, keyed by pack name and extension
• HibernateRefDatabase, HibernateReflogWriter, HibernateReflogReader — database-backed ref and reflog storage
• HibernateRepositoryBuilder extends DfsRepositoryBuilder
• JPA entity mappings for GitObjectEntity, GitRefEntity, GitPackEntity, GitReflogEntity, GitCommitIndex, JavaBlobIndex, FilePathHistory

It is actively used in my Sandbox project as an Eclipse plugin — see the sandbox-jgit-storage-hibernate module.

Why Include It in the JGit Distribution

1. Atomicity and Transactional Guarantees
   Database transactions provide true ACID guarantees for pack commits, ref updates, and rollbacks — something inherently difficult to achieve with filesystem-based storage.

2. Packaging and Dependency Management
   Maintaining this implementation externally means dealing with version compatibility against JGit internals, OSGi/p2 packaging challenges, and duplicated build infrastructure. Including it as an optional module in JGit would eliminate this burden and ensure it stays in sync with API changes.

3. A Real-World DFS Implementation Beyond InMemoryRepository
   JGit ships InMemoryRepository as its only DFS implementation. A Hibernate-based module would be the natural persistent counterpart — usable with any JDBC-compatible database (H2 for testing/embedded, PostgreSQL for production) — and would serve as a reference implementation that helps validate and harden the DFS API.

Features This Has Enabled

The database layer has made it possible to build features that would be extremely difficult or impossible with filesystem-based storage:

• ECJ-Based Java Source Tokenizer for Lucene
  An EcjTokenizer that uses Eclipse's own Java compiler scanner to produce lexically correct Java tokens for Lucene indexing. Combined with an EcjTokenFilter that applies CamelCase splitting, string literal stripping, and token-type-aware processing. This gives true language-aware full-text search over Java source code stored in Git — something fundamentally different from plain text search.

• AST-Based Structural Indexing
  A JavaBlobExtractor and JavaFileStrategy that parse Java source files using JDT's ASTParser and extract structural metadata — package names, declared types, methods, fields, supertypes, interfaces, imports — all indexed and queryable via Hibernate Search/Lucene.

• Semantic and Hybrid Code Search
  A SemanticSearchClient that supports natural language queries over the indexed repository content — semantic search (vector-based), hybrid search (full-text + semantic), type search, symbol search, commit message search, and changed-path search. This allows asking questions like "find all implementations of a caching strategy" rather than just grep for a string.

• LLM-Powered Commit Analysis
  A CommitAnalysisJob that feeds commit diffs to an LLM service to generate DSL rules and semantic evaluations of code changes. The database layer makes it practical to store, index, and query both the raw repository data and the AI-generated analysis results together.

• Structured Querying via GitDatabaseQueryService
  SQL/HQL queries over commits, trees, blobs, and refs — e.g., "find all commits by author X touching files in path Y between dates A and B" — without walking the entire object graph.

None of these features require changes to the DFS API itself — they all build on top of the Hibernate storage layer. But they demonstrate why having this implementation inside the JGit distribution (rather than maintained externally) would benefit the broader ecosystem.

References

• JGit Fork (persistence layer work):
  👉 https://github.com/carstenartur/jgit

• Hibernate Storage Module (in Sandbox):
  👉 sandbox-jgit-storage-hibernate

• Sandbox Project (Eclipse product with all integrations):
  👉 https://github.com/carstenartur/sandbox

• Related EGit Discussion — allowing plugins to switch the persistence layer:
  👉 eclipse-egit/egit#145

What I Would Like to Discuss

1. Is there interest in including this as an optional module in the JGit distribution?
2. What concerns are there around additional dependencies (Hibernate ORM, Hibernate Search, Lucene)?
3. Would the team be open to reviewing the implementation as a starting point?
4. Are there DFS API improvements that should be addressed to better support database backends?

Thank you for your time and feedback!

Best regards,
Carsten Hammer
GitHub: @carstenartur

[msohn]

Thanks for this proposal.

1. I am open to include this as an optional module into JGit. We need some commitment that you are willing to maintain the added code in a sustainable manner. A large one-off contribution hoping that existing committers will take over long-term maintenance won't work.
2. As long as the additional dependencies are confined to the new optional module and their license is compatible with JGit's license I have no principal concerns. Of course increasing the set of dependencies adds costs. The surface for vulnerabilities increases, dependencies need to be updated on a regular basis, build and test times will increase etc.
3. I am open to review the implementation if it comes as a series of consumable changes uploaded to GerritHub which add one thing at a time. Reviewing a large contribution coming in a single or few huge changes considerably increases the review effort.
4. I don't know at this moment.

A couple of comments/questions which came to my mind when skimming through your changes in the fork:

• The prefix DFS means distributed file system, a relational database doesn't necessarily fit this. Shawn had implemented a database based storage abstraction in JGit earlier and later deleted it since it had performance and scalability issues especially when trying to horizontally scale JGit in a distributed setup. Did you explore performance and scalability of this implementation ?
• Be prepared that we soon will look into adding support for sha256 for object identifiers since git 3.0 plans to make this the default. So far JGit only has a sha256 implementation (LongObjectId) but that's only used in the LFS modules to identify LFS objects stored in a separate storage. See https://www.kernel.org/pub/software/scm/git/docs/technical/hash-function-transition.html

[carstenartur]

Thank you very much for the detailed reply — this is very helpful.

I want to be transparent about the main limitation on my side: this started as a hobby project, so I cannot realistically commit to the kind of long-term maintenance that would be needed for an upstream JGit module.

So I think the most realistic path for now is to keep this as a separate project built on top of JGit’s extension points, rather than trying to move it into JGit itself.

On the technical side, I also agree that there are still important open questions. I do not yet have the kind of JMH benchmarks or broader performance/scalability evaluation that would be needed to make a strong case here, especially given the concerns you mentioned about earlier database-backed approaches and horizontal scaling. I would be willing to implement such benchmarks, but at this point I cannot claim that this has been evaluated far enough.

I also take your point about SHA-256 support. That is something I have not considered deeply enough yet, and I agree that any serious persistence backend discussion should avoid building in assumptions that would make that transition difficult later.

Another practical issue is reviewability: I agree that anything like this would need to be split into small, reviewable pieces, but I also have to admit that finding a good decomposition here is itself not trivial. Given that, together with the maintenance and performance questions, I think it is probably best not to pursue upstream integration further at this stage.

For now I will continue treating it as an external project and try to keep the design reasonably aligned with JGit conventions, so that the door is not completely closed if the situation changes in the future.

Thanks again for the thoughtful response.

[carstenartur]

A small update since my previous reply:

I have extracted the work into a standalone project built on top of JGit’s existing DFS and Reftable extension points:

jgit-storage-hibernate

https://github.com/carstenartur/jgit-storage-hibernate

The project is intentionally maintained outside JGit for now. It provides a Hibernate-backed DfsRepository implementation for packs, refs, Reftables and queryable reflogs, together with versioned Flyway migrations for H2 and PostgreSQL.

The database backend uses explicit transaction boundaries for repository storage operations. Pack components are initially stored as uncommitted and remain invisible to readers until the complete pack is published. Publishing new pack components and deleting replaced packs happens transactionally, and the Reftable-based ref database advertises atomic ref transactions. This does not currently mean that an arbitrary application entity update, Git object insertion, ref update and search indexing automatically participate in one ambient transaction; those boundaries are documented separately.

The project now also demonstrates why a database-backed repository can be useful beyond changing the physical storage location.

The optional Search module maintains a relational and Hibernate Search/Lucene projection of commit history. Revision walking, first-parent tree comparison, changed-path extraction and text indexing are performed when commits are indexed or during an explicit rebuild, rather than being repeated for every query. This enables, for example:

• compound queries such as “which changes did Alice make below services/payments/fraud/ during Q1?”;
• full-text search across commit messages, actual changed paths and selected changed-file contents.

Git and JGit do not normally provide a general full-text history index of this kind. The relevant distinction is therefore not merely that an application avoids writing a few JGit loops: indexing creates a durable, read-optimized data structure with different ingestion and query-time characteristics.

The Java Analysis module derives an additional binding-aware semantic model that is not present in Git itself. One documented and tested example answers which code locations used a logical Java type in each analyzed version, including after the type moved to another package. Results include source paths, lines, relation types and binding quality.

The public use cases are backed by executable integration tests, including PostgreSQL/Testcontainers tests, and compatibility is currently checked against JGit 7.5, 7.6 and 7.7.

The broader performance and horizontal-scalability questions raised above are still valid. The project now contains JMH benchmarks, but I would not claim that this constitutes a complete distributed scalability evaluation. I am continuing to treat that as an area requiring measured evidence rather than assumptions.

Feedback on the storage boundaries, query model, benchmarks or useful DFS extension points would be very welcome.

[carstenartur]

• The prefix DFS means distributed file system, a relational database doesn't necessarily fit this. Shawn had implemented a database based storage abstraction in JGit earlier and later deleted it since it had performance and scalability issues especially when trying to horizontally scale JGit in a distributed setup. Did you explore performance and scalability of this implementation ?

A further update on the performance and scalability question:

Since my previous reply, the standalone project has gained a reproducible JMH/Testcontainers benchmark suite covering public JGit object, ref, ReceivePack and UploadPack operations. The same workloads are executed against FileRepository, HSQLDB, PostgreSQL and PostgreSQL with HikariCP.

In the latest 0.1.17 merge-gate run, the protocol point estimates were:

• initial push, 24 commits: filesystem 163.9 ms, PostgreSQL 196.8 ms;
• incremental push, four commits: filesystem 23.1 ms, PostgreSQL 45.1 ms;
• initial clone: filesystem 182.7 ms, PostgreSQL 162.0 ms;
• incremental fetch, four commits: filesystem 30.9 ms, PostgreSQL 25.2 ms.

The confidence intervals are still wide, especially for push operations, so I would not claim that either backend is generally faster. The results do show that the database backend is competitive for clone and fetch in this controlled serial setup, while durable publication and transaction overhead remain clearly visible on small write operations.

Large pack extensions are staged through temporary files and persisted in bounded 1 MiB chunks instead of being held as one complete byte array. Dedicated capacity tests cover 1, 16 and 128 MiB payloads. A focused 12 MiB PostgreSQL publication benchmark currently takes about 519–522 ms; enabling JDBC batching did not produce a latency difference outside the measurement uncertainty.

There is also a PostgreSQL correctness test using two independent Hibernate SessionFactory instances. Two concurrent ref updates with the same expected old object ID result in exactly one successful update and one lock failure, with the winning ref and reflog visible from a fresh persistence context.

However, I would still not describe this as a completed horizontal-scalability evaluation. The current suite does not yet measure concurrent reader/writer throughput across multiple JVMs or server nodes, p50/p95/p99 latency under contention, database saturation, repository metadata growth, or operation over a production network. Repository instances also retain local JGit caches, so cross-process writes require the established refresh or reopen boundary before an immediately authoritative view is needed.

The most accurate summary is therefore: serial performance, bounded large-pack behavior and a specific multi-instance conflict scenario have now been measured and tested; horizontal throughput scaling remains an open benchmark area rather than an established claim.

SHA-256 also remains explicit future work. Some current schema fields still assume 40-character SHA-1 object IDs, so I would not claim compatibility until those assumptions have been removed and covered by migrations and tests.

https://carstenartur.github.io/jgit-storage-hibernate/dev/bench/