Skip to content

Add pc tags data extracted from the vanilla server jar (1.13-26.1) - #1308

Open
Pix3lPirat3 wants to merge 1 commit into
PrismarineJS:masterfrom
Pix3lPirat3:feat/pc-tags
Open

Pix3lPirat3 wants to merge 1 commit into
PrismarineJS:masterfrom
Pix3lPirat3:feat/pc-tags

Conversation

@Pix3lPirat3

Copy link
Copy Markdown
Contributor

Add tools/js/extractPcTags.js, which reads data/minecraft/tags from the official server jar (sha1-verified from the Mojang manifest) and writes data/pc//tags.json with #tag references resolved, everything sorted and registries namespaced. Generate it for every pc release from 1.13 to 26.1; identical versions share a directory through dataPaths.

Implements the tags direction from #412 and answers the backport question (PrismarineJS/minecraft-data-generator#56, #1080): tags are plain datapack JSON inside every server jar since 1.13, so they need no generator and no mappings. tools/js/extractPcTags.js (dependency-free) reads them from the official jar - sha1-verified download from the Mojang manifest, cached in the OS temp dir — resolves #tag references, and writes data/pc/<version>/tags.json; identical versions share a directory through dataPaths, as other data does. node extractPcTags.js --all regenerates everything in about two minutes with cached jars.

Data:

27 files covering all 46 pc releases (and pre/rc aliases) from 1.13 to 26.1. Weekly snapshots are excluded because their blocks.json aliases unrelated versions. Base datapack only. Registry keys, tag keys and members are namespaced and sorted; empty registries are omitted; every registry in the jar is emitted (block, item, fluid, entity_type from 1.13, growing to twenty on 26.1 including worldgen/*).

Format follows #1194's tags_schema.json, included verbatim (by @VasilisDragon) with its one-line test.js allowlist change so CI validates the data; #1194 remains the home for the audit and schema tests, and I'll rebase onto it if it lands first. Note #1080's file uses bare registry keys in registry order and does not pass that audit; pc/1.21.8/tags.json here has identical tag and member sets to it (sorted), so it supersedes #1080.

Verification: all 46 versions pass the schema and #1194's auditTagDocument in official mode with block/item/entity_type/biome/enchantment members from each version's own data - no dangling members. npm test passes. 1.16 and 1.16.1 alias pc/1.16-rc1 because the rc precedes the release and is identical; say if a release directory is preferred.

What consumers get: mineable/* and needs_*_tool from 1.17, incorrect_for_*_tool from 1.20.5 - the data prismarine-block needs to compute dig time without materials.json (#987, #1307). Follow-ups: PrismarineJS/node-minecraft-data#457 to expose mcData.tags, then a prismarine-block digTime() that reads tags with materials.json as the pre-1.17 fallback.

Add tools/js/extractPcTags.js, which reads data/minecraft/tags from the official server jar (sha1-verified from the Mojang manifest) and writes data/pc/<version>/tags.json with #tag references resolved, everything sorted and registries namespaced. Generate it for every pc release from 1.13 to 26.1; identical versions share a directory through dataPaths.

Copy link
Copy Markdown
Contributor

I checked all 46 mapped versions against the official server-jar contents using an independent extractor; the tag sets and members match. --all reproduces the committed data, and both npm test and the tag audit/schema tests from #1194 pass. The 1.16-rc1 alias also looks fine: its tag data matches both 1.16 and 1.16.1.

@rom1504 rom1504 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Astra agent review — AI-generated, not manually written by the maintainer.

At 2eb69db, this looks ready for maintainer merge review. The producer reads the official SHA1-verified server jars, resolves nested tag references, and keeps the namespaced registry/tag/member contract across the 46 release mappings. I reviewed the complete extractor and aliases earlier today; VasilisDragon’s independent exact-head comparison reports matching official contents and reproducible --all output. I did not repeat those 46 downloads in this publication pass.

Coordinate the shared schema/audit work with #1194. The optional Node wrapper PR #457 should document mcData.tags['minecraft:block'], matching these namespaced keys, before consumers rely on it. These extracted facts can land independently of a later dig-time redesign.

Skills used: prismarine-review checked the current revision and discussion; prismarine-protocol-data-review checked the version-selected producer/consumer contract; prismarine-architecture-review checked package ownership and integration scope.

@rom1504 rom1504 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Astra agent review — AI-generated, not manually written by the maintainer.

Re-reviewed 2eb69db9d6bf6176846dd20c1386979165b78f36 following the maintainer's request to check the schema, extraction documentation, maintained tests and PrismarineJS usefulness of the output.

This supersedes my earlier ready-to-merge assessment: #1308 (review). I would hold this PR for the changes below. My earlier review put too much weight on reproducible extraction and schema-valid generated files, without requiring the complete supported data contract, durable documentation and integrated regression coverage.

The shared tags foundation is useful, and flattening references/normalizing names already goes beyond a literal jar dump. Before merging, please:

  1. Define and enforce the registry scope useful to PrismarineJS, with a documented purpose for each included category.
  2. Add repository documentation explaining usefulness, format, regeneration, validation and consumer/release limitations.
  3. Integrate the relevant semantic/schema tests currently in #1194 into the normal repository test path.
  4. Add focused tests of the actual extractor and reject missing required references/cycles rather than silently producing empty results.

Four inline threads give the specific locations, evidence and bounded next steps. These do not require a complete downstream dig-time redesign, a new testing framework or a live-server suite for this data extractor. Coordinate #1194 rather than maintaining competing validators.

Validation for this pass: re-read the current source/diff and all existing discussion; checked the companion test PR remains unmerged; exercised schema acceptance and the production extractor function with controlled inputs. I did not repeat all official-jar downloads or claim a fresh full repository test run in this pass.

Skills used: prismarine-review checked current-head feedback, duplicate findings and testing conventions; prismarine-protocol-data-review checked schema, versioned members, extraction and release boundaries; prismarine-architecture-review checked useful shared-package scope and its documented consumer contract.

Comment thread schemas/tags_schema.json
Comment on lines +17 to +20
"patternProperties": {
"^[a-z0-9_.-]+:[a-z0-9/._-]+$": {
"$ref": "#/$defs/tagMap",
"description": "Any additional namespaced registry the generator emits, including path-shaped registries such as minecraft:worldgen/biome."

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Astra agent review — AI-generated, not manually written by the maintainer.

Please define the supported PrismarineJS registry scope explicitly instead of accepting every registry the extractor happens to find. This top-level patternProperties means additionalProperties: false still permits any syntactically namespaced registry. I confirmed that an extra example:unrelated_registry with arbitrary namespaced tags/members passes this schema. The committed 26.1 file includes 20 registries, including dialog, timeline, villager_trade and several world-generation registries, without documenting a consumer contract for each.

The maintainer specifically wants useful PrismarineJS data rather than an automatically expanding dump of Java output. Block tags have a concrete consumer in prismarine-block#127. Please document the purpose of each additional registry retained, define the agreed registry list in the schema, and make extraction filter to that scope. Test that an unsupported registry is rejected and not emitted. New registry categories should require a deliberate contract change.

This does not require enumerating every tag or every version's members in JSON Schema, nor forbidding custom member namespaces indiscriminately. Keep namespaced tag/member names extensible within the supported registries; use version-aware semantic validation for the checked-in vanilla data. The current normalization and reference flattening are useful and should be retained.

Skills used: prismarine-architecture-review checked the consumer-facing data boundary; prismarine-protocol-data-review checked what the schema actually accepts.

Comment thread tools/js/extractPcTags.js
Comment on lines +4 to +9
// Usage: node extractPcTags.js <version> [<version> ...]
// node extractPcTags.js --all (every pc release in dataPaths from 1.13 on)
//
// Output follows schemas/tags_schema.json: namespaced registry keys, namespaced tag keys,
// resolved namespaced members, everything sorted, empty registries omitted. Versions whose
// output is identical to the previous version reuse its directory in dataPaths, like other data.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Astra agent review — AI-generated, not manually written by the maintainer.

Please add repository documentation, for example doc/tags.md, and link it from the appropriate data/extraction documentation. These usage comments and the PR description are a useful start, but the checked-in documentation does not explain why consumers need this dataset or its supported contract.

The guide should cover:

  • The retained registries and concrete uses. Explain the block-tag use in dig-time/tool selection, including why a single material classification is insufficient, and justify other categories included.
  • A small JSON example and lookup example with namespaced keys; explain resolved references, sorting/deduplication, valid empty member lists and version availability.
  • Commands runnable from the repository root for one version and --all, jar provenance/SHA1 checks, MCDATA_JAR_CACHE, generated files/dataPaths aliases, and the validation command.
  • Scope: Java vanilla base-datapack defaults, not the negotiated tags of a live/custom server; snapshot exclusions and the Node wrapper/release prerequisite for mcData.tags.

Distinguish what is available in the data repository from what a released consumer can currently load. This documentation should remain useful after the PR conversation is no longer being read.

Skills used: prismarine-architecture-review checked the public contract and its purpose; prismarine-protocol-data-review checked reproducibility and the data → wrapper → consumer boundary.

Comment thread tools/js/test/test.js
}

const data = ['attributes', 'biomes', 'commands', 'instruments', 'items', 'materials', 'blocks', 'blockCollisionShapes', 'recipes', 'windows', 'entities', 'protocol', 'version', 'effects', 'enchantments', 'language', 'foods', 'particles', 'blockLoot', 'entityLoot', 'mapIcons', 'tints', 'blockMappings', 'sounds', 'blockStates']
const data = ['attributes', 'biomes', 'commands', 'instruments', 'items', 'materials', 'blocks', 'blockCollisionShapes', 'recipes', 'windows', 'entities', 'protocol', 'version', 'effects', 'enchantments', 'language', 'foods', 'particles', 'blockLoot', 'entityLoot', 'mapIcons', 'tints', 'blockMappings', 'sounds', 'blockStates', 'tags']

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Astra agent review — AI-generated, not manually written by the maintainer.

Adding tags to the existing schema/version iterator is the right integration point, but it only validates the shape of the committed output. The semantic audit and schema-negative tests described in this PR are still in unmerged #1194; they are not included in this branch's normal test command.

Please land/rebase on the relevant #1194 tests or coordinate their inclusion here, adapting them to the agreed registry scope rather than duplicating a second audit. Retain normal Mocha/CI discovery and verify:

  • Every mapped version selects an existing tags file, and members resolve against that version's corresponding registry data where modeled, including versions sharing an alias.
  • Malformed keys, duplicates, unresolved references, missing required registries and newly disallowed registry categories are rejected; valid empty member lists remain accepted.
  • Deterministic ordering and dataPaths/file consistency are maintained. For any retained category without an existing member table, document the validation boundary rather than treating syntax validation as proof that its members exist.

A focused schema probe accepts minecraft:this_block_does_not_exist as a block-tag member; membership belongs in the semantic audit, not a hardcoded schema enum. The earlier 3,048 passing cases were mostly the existing suite plus 27 generated-file schema cases, not extractor regression coverage. The independent 46-version comparison in the discussion is useful evidence, but it does not install those checks in CI.

Skills used: prismarine-review applied the repository testing gate; prismarine-protocol-data-review checked per-version references and schema-versus-semantic coverage.

Comment thread tools/js/extractPcTags.js
Comment on lines +105 to +109
if (stack.includes(tag)) return []
const members = new Set()
for (const value of tags[tag] || []) {
const id = typeof value === 'string' ? value : value.id
if (id.startsWith('#')) resolve(id.slice(1).replace(/^minecraft:/, ''), [...stack, tag]).forEach(x => members.add(x))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Astra agent review — AI-generated, not manually written by the maintainer.

Please exercise the actual extractor/resolver in focused offline regression tests, including invalid reference graphs. Currently tags[tag] || [] treats a missing required #tag as an empty tag, and the cycle branch returns []. Both can produce schema-valid output while silently losing membership.

I ran this exact extractTags function in isolation with the existing archive-reader interface: a tag with values: ["#minecraft:absent"] becomes an empty member list; an a → b → a cycle produces empty lists for both tags, without an error. This is a malformed-input/future-regeneration case, not evidence that the committed vanilla files contain such errors.

Please distinguish a valid empty tag from an unresolved required reference or cycle. Fail extraction with the registry/tag context for required missing references and cycles, while explicitly handling optional references according to their documented semantics. Add positive controls for nested references, duplicate resolution and empty tags, plus negative controls for these cases. Cover the pre-1.21 plural/current singular directory normalization, supported-registry filtering and deterministic ordering as well.

A small exported pure extraction function with the CLI behind a main guard would let the existing Mocha suite test the production path using controlled archive entries; no new test framework or network-heavy all-version extraction is required for those unit tests.

Skills used: prismarine-protocol-data-review traced regeneration/reference semantics; prismarine-review distinguished meaningful production-function tests from merely validating generated output.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants