Add support for lexical dictionary format for custom vocabulary

[grhoten]

There are times that you need to amend or completely replace the lexical dictionaries derived from Wikidata. Nebojša created a document called Evaluation of Lexicon Format Alternatives that covers investigating this topic. This feature enhancement will cover the discussion and progress on such an effort.

The current top contender is DMLex.

The second option is OntoLex-Lemon, but that seems more geared towards ontologies and knowledge graphs. That's probably more like entirety of Wikidata, instead of the lexeme part of Wikidata.

The other options listed in the document struggle with containing sufficient structure to disambiguate words that differ in the meaning. The current file format read by buildDictionary is best at compressing a post validated lexical structure. It's not a good format for humans to edit.

The new file format to support should have the following characteristics:

• Have an acceptable license that is compatible with the Unicode license.
• Reference a main label. This is typically a lemma.
• Represent a part of speech.
• Label multiple surface forms under the lemma with grammemes that allow each state to be uniquely identified (when possible).
• Allow phonetic information to be available so that articles or particles can be properly added.
• Be computer parseable and validated. The group is leaning towards JSON because Wikidata already uses a JSON parser. XML is acceptable if necessary. DMLex supports both.
• Ideally have other repositories of lexical data to choose from that are compatible with the Unicode license.
• Ideally is an open standard, but a custom format is possible.

In order to use such data, it's likely that the existing dictionary-parser will be modified to support the additional format. This additional file format can be used to amend, preempt, or replace the data from Wikidata as necessary. The proposal is not to remove support for the existing lst/xml file formats. The proposal is to convert this new custom lexical dictionary support into the existing file formats that buildDictionary can parse. This is already mildly done with the supplemental data for specific languages. The data structures will likely have to be shared between the 2 parsers in the dictionary-parser tool.

[nciric]

Another point of consideration wrt format and infrastructure is avoiding additional 3rd party dependencies for json/xml parsing and validation. DMLex uses already present deps, so I am ok with it.

[nciric]

TL;DR;

Distributing Data:
Both formats allow you to publish lexicographic data files with a header containing the Unicode License v3 text.

Distributing Software/Tools:
Both formats allow you to write parser/converter code in C++, Python, Rust, or Java and release it under the Unicode License.

Embedding Schemas:
Schemas and RDF ontologies from both bodies are royalty-free to consume and validate against in your software distributions.

When we talk about licensing we mean:

1. Using actual formats for data representation
2. Using tools to manipulate json/xml serialized data
3. Using schemas for validation of serialized data (both standards provide them)

should be compatible with Unicode license. My research shows:

When you create a dataset formatted in XML, JSON, or RDF conforming to a standard data model (DMLex or OntoLex), the copyright in the content and compilation of data belongs entirely to you (the author/publisher). Referencing vocabulary terms, XML tags, or JSON property names defined by an open standard does not grant the standard organization copyright over your data, nor does it impose copyleft obligations.

OASIS IPR Policy (Data Implementations):

• Under the OASIS IPR Policy Section 10, OASIS deliverables grant a perpetual, worldwide, non-exclusive, royalty-free license to implement conforming products and create conforming data without licensing fees or copyleft terms.

W3C Community Final Specification Agreement (FSA):

• Under the W3C Community Final Specification Agreement, contributors grant a royalty-free, perpetual patent and copyright license to create implementations and data instances conforming to the Community Report.