diff --git a/docs/bsa-inspection.md b/docs/bsa-inspection.md index 24cde0b..5f73ded 100644 --- a/docs/bsa-inspection.md +++ b/docs/bsa-inspection.md @@ -17,3 +17,23 @@ Limits: 4,294,967,295 input bytes, 64 MiB metadata, 100,000 folders, 500,000 fil This report is **source evidence only**. It does not change the ZIP/7z/RAR ledger or automatically produce a recipe. To establish an upstream source chain, separately verify the outer download archive, its exact BSA member hash and the desired internal member hash. Compare the installed file's complete size/SHA-256; a similar name or image is insufficient. A reviewed recipe can use the explicit [`extract-bsa104` action](loadstone-format.md#outputs-and-actions) for [bounded nested-BSA reconstruction](native-reconstruction.md#nested-bsa-104-files). Transformed-file reproduction, rights review and real installation acceptance remain separate work. No game or mod data is bundled by this command. The JavaScript implementation and synthetic fixtures are original. Format semantics were checked against Wabbajack's pinned [104/105 reader](https://github.com/wabbajack-tools/wabbajack/blob/4.2.3.0/Wabbajack.Compression.BSA/TES5Archive/Reader.cs), [folder records](https://github.com/wabbajack-tools/wabbajack/blob/4.2.3.0/Wabbajack.Compression.BSA/TES5Archive/FolderRecord.cs) and [file records](https://github.com/wabbajack-tools/wabbajack/blob/4.2.3.0/Wabbajack.Compression.BSA/TES5Archive/FileRecord.cs). Loadstone deliberately supports a narrower format subset. + +## Nested source mapping + +The separate creator command extends a private [source ledger](installed-inventory.md) with exact nested matches: + +```powershell +node scripts/map-bsa-sources.mjs private-mapping.json new-private-bsa-mapping.json --mod "Example container mod" --7zip C:\Tools\7-Zip\7z.exe +``` + +Repeat `--mod` for other exact inventoried mod names. The selected mods must contain inventoried `.bsa` files and have original archive candidates in the ledger. Disabled mods can supply containers without being enabled. The optional `--7zip` is required for 7z/RAR candidates; ZIP comparison uses the built-in reader. No tool or original archive is downloaded by this command. + +The mapper independently rehashes each selected original archive and validates its complete structure. It hashes candidate BSA members and compares their full size/SHA-256 with the installed BSA inventory. Only then does it inspect the matching local BSA, rechecking the complete container fingerprint and hashing its internal contents. The local file is a read-only witness of the verified original member; nothing is staged or extracted. It compares inner content with all managed mod-file fingerprints in the accepted ledger, including disabled mods, and excludes local `meta.ini`. It does not rehash every loose installed target during this pass, nor map game/root/profile/overwrite records. + +Each matched file receives a separate `bsaSourceMatches` array. A reference records `archiveSha256`, `container: { member, bytes, sha256 }`, the inner `member` and the number of byte-equivalent inner members (`equivalentMembers`). The representative is the lexicographically first normalized inner path per container and content fingerprint. The original archive's fingerprint and available provider hints appear in `bsaMapping.sources`; provider hints still require review. These references supply the fields for a reviewed `extract-bsa104` action once a recipe source ID is assigned. They do not generate or approve a recipe. + +Existing `sourceMatches`, the direct `mapping` report, mod states and other inventory records remain intact. `bsaMapping` separately reports nested matches, newly matched files and the combined unique file totals. Re-running the command replaces the nested extension from fresh checks without multiplying old references. A fresh direct archive-mapping pass clears the old nested extension, which must then be recomputed. Failed archives or containers produce unresolved issues; no partial matches from a failed inspection are accepted. + +Limits: 64 selected containers, 20 GB total container bytes, 20 GB combined successfully inspected logical content, 1,000 candidate archive paths and 500,000 nested match references, in addition to the existing archive/BSA reader limits. ZIP and per-member 7z/RAR comparison retain their 1 GB member limit; eligible 7z archives can use the existing bounded single-pass reader for larger members. Other BSA versions and arbitrary nesting remain unsupported. + +The command locks the input ledger, referenced inventory and optional source hints, checks them again before writing, refuses existing report paths and keeps reports outside installation/download roots. Ctrl+C cancels without a final report. Reports contain private paths and installed-content names: keep them outside public source. Matching bytes establish source evidence only; source rights, typed transformations, recipe validation and actual reconstruction/gameplay acceptance remain required. diff --git a/docs/installed-inventory.md b/docs/installed-inventory.md index f07a1cf..b87c74d 100644 --- a/docs/installed-inventory.md +++ b/docs/installed-inventory.md @@ -84,7 +84,7 @@ For tests, `scripts/prepare-sevenzip.ps1 -Destination ` d 7-Zip is a separate upstream tool with LGPL, BSD and unRAR terms; a future bundled distribution needs its own notices/source compliance review. See the [official release](https://github.com/ip7z/7zip/releases/tag/26.04) and [license](https://www.7-zip.org/license.txt). -For a separate read-only inventory of files inside a local Fallout 3/New Vegas BSA, see [BSA 104 inspection](bsa-inspection.md). This hashes the internal members without adding nested-container matches to the ledger. Reviewed source chains can be declared explicitly for [native BSA reconstruction](native-reconstruction.md#nested-bsa-104-files). +For a separate read-only inventory of files inside a local Fallout 3/New Vegas BSA, see [BSA 104 inspection](bsa-inspection.md). To connect selected installed BSAs to original archive candidates and compare their contents with the ledger, use [nested BSA source mapping](bsa-inspection.md#nested-source-mapping). Reviewed source chains can then be declared explicitly for [native BSA reconstruction](native-reconstruction.md#nested-bsa-104-files). ## Native `.loadstone` format diff --git a/electron/archive-mapping.cjs b/electron/archive-mapping.cjs index 77c9a58..d5943cf 100644 --- a/electron/archive-mapping.cjs +++ b/electron/archive-mapping.cjs @@ -103,6 +103,8 @@ async function mapArchiveSources(ledger, options = {}) { throw new Error('Use a supported source ledger.'); const result = structuredClone(ledger), archiveGroups = new Map(); + // A fresh direct pass does not revalidate previously recorded nested chains. + delete result.bsaMapping; let fileCount = 0; for (const mod of result.mods) { if ( @@ -122,6 +124,7 @@ async function mapArchiveSources(ledger, options = {}) { ) throw new Error('Invalid installed file fingerprint.'); file.sourceMatches = []; + delete file.bsaSourceMatches; } if (mod.separator || !mod.installedFiles.length) continue; for (const candidate of mod.archiveCandidates) { @@ -240,4 +243,8 @@ async function mapArchiveSources(ledger, options = {}) { ]; return result; } -module.exports = { mapArchiveSources, mapZipSources: mapArchiveSources }; +module.exports = { + mapArchiveSources, + mapZipSources: mapArchiveSources, + inspectZipMembers, +}; diff --git a/electron/bsa-mapping.cjs b/electron/bsa-mapping.cjs new file mode 100644 index 0000000..f88fdda --- /dev/null +++ b/electron/bsa-mapping.cjs @@ -0,0 +1,347 @@ +const fs = require('node:fs'); +const path = require('node:path'); +const { relativeFile } = require('./manifest.cjs'); +const { ordinaryFile, sameFile } = require('./native-package.cjs'); +const { archiveKind } = require('./archive-kind.cjs'); +const { inspectZipMembers } = require('./archive-mapping.cjs'); +const { inspectSevenZipMembers } = require('./sevenzip.cjs'); +const { inspectBsa104, limits } = require('./bsa104.cjs'); +const { recordArchiveMatch } = require('./archive-matches.cjs'); + +const fingerprint = (v) => + v && + Number.isSafeInteger(v.bytes) && + v.bytes >= 0 && + typeof v.sha256 === 'string' && + /^[a-f0-9]{64}$/.test(v.sha256); +const safeName = (name) => + typeof name === 'string' && + relativeFile(name) && + !name.includes('\\') && + name === name.normalize('NFC'); +const identity = (v) => `${v.bytes}:${v.sha256}`; +function demand(ok, message) { + if (!ok) throw new Error(`BSA mapping: ${message}`); +} + +// Installed containers are read-only witnesses. Rehashing their original +// archive members establishes the source chain without temporary extraction. +async function mapBsaSources(ledger, options = {}) { + options.signal?.throwIfAborted(); + demand( + ledger?.kind === 'loadstone-source-ledger' && + ledger.schemaVersion === 1 && + Array.isArray(ledger.mods) && + ledger.mods.length <= 10000, + 'use a supported source ledger.', + ); + demand( + Array.isArray(options.containerMods) && + options.containerMods.length > 0 && + options.containerMods.length <= limits.containers && + options.containerMods.every((v) => safeName(v) && !v.includes('/')) && + new Set(options.containerMods.map((v) => v.toLowerCase())).size === + options.containerMods.length, + 'select distinct inventoried container mods.', + ); + demand( + typeof options.modsFolder === 'string' && + path.isAbsolute(options.modsFolder), + 'select the inventoried mods directory.', + ); + const root = path.resolve(options.modsFolder), + rootStat = fs.lstatSync(root); + const guardRoot = () => { + const stat = fs.lstatSync(root); + demand( + stat.isDirectory() && + !stat.isSymbolicLink() && + stat.ino === rootStat.ino && + stat.dev === rootStat.dev && + fs.realpathSync(root).toLowerCase() === root.toLowerCase(), + 'mods directory changed or is redirected.', + ); + }; + guardRoot(); + const result = structuredClone(ledger), + names = new Set(), + files = [], + containers = [], + archives = new Map(), + wanted = new Map(); + for (const mod of result.mods) { + demand( + safeName(mod.name) && + !mod.name.includes('/') && + !names.has(mod.name.toLowerCase()) && + Array.isArray(mod.installedFiles) && + Array.isArray(mod.archiveCandidates) && + mod.archiveCandidates.length <= 100000, + 'invalid mod records.', + ); + names.add(mod.name.toLowerCase()); + const paths = new Set(); + for (const file of mod.installedFiles) { + demand( + safeName(file.path) && + !paths.has(file.path.toLowerCase()) && + fingerprint(file) && + (file.sourceMatches === undefined || + Array.isArray(file.sourceMatches)), + 'invalid installed file records.', + ); + paths.add(file.path.toLowerCase()); + file.bsaSourceMatches = []; + files.push(file); + demand(files.length <= 1000000, 'too many installed files.'); + if ( + mod.separator || + mod.unmanaged || + file.path.toLowerCase() === 'meta.ini' + ) + continue; + const matches = wanted.get(identity(file)) || []; + matches.push(file); + wanted.set(identity(file), matches); + } + if (!options.containerMods.includes(mod.name)) continue; + demand(!mod.separator && !mod.unmanaged, 'container mod must be managed.'); + const selected = mod.installedFiles.filter((v) => /\.bsa$/i.test(v.path)); + demand(selected.length > 0, 'selected mod has no inventoried BSA files.'); + for (const file of selected) { + demand( + file.bytes >= 36 && file.bytes <= limits.archiveBytes, + 'container exceeds reader limits.', + ); + containers.push({ + mod: mod.name, + record: file, + file: path.join(root, mod.name, file.path), + chains: [], + }); + } + for (const candidate of mod.archiveCandidates) { + demand( + fingerprint(candidate) && + typeof candidate.archive === 'string' && + path.isAbsolute(candidate.archive) && + !/[\x00-\x1f]/.test(candidate.archive), + 'invalid archive candidate.', + ); + if (candidate.metadataConflict) continue; + const key = path.resolve(candidate.archive).toLowerCase(), + group = archives.get(key) || { candidate, selected: new Set() }; + demand( + identity(group.candidate) === identity(candidate), + 'conflicting archive fingerprints.', + ); + for (const file of selected) group.selected.add(identity(file)); + archives.set(key, group); + demand( + archives.size <= 1000, + 'too many candidate archives for BSA mapping.', + ); + } + } + demand( + options.containerMods.every((name) => + result.mods.some((v) => v.name === name), + ), + 'container mod names must exactly match the ledger.', + ); + demand( + containers.length <= limits.containers && + containers.reduce((n, v) => n + v.record.bytes, 0) <= limits.stagingBytes, + 'selected containers exceed the inspection budget.', + ); + const inspectedArchives = [], + inspectedContainers = [], + issues = [], + sources = new Map(); + for (const { candidate, selected } of archives.values()) { + options.signal?.throwIfAborted(); + try { + const kind = archiveKind(candidate.archive); + demand( + kind === 'zip' || options.sevenZip, + 'select a trusted archive tool for 7z/RAR comparison.', + ); + const before = ordinaryFile(candidate.archive, 50_000_000_000); + const matched = + kind === 'zip' + ? await inspectZipMembers(candidate, selected, options.signal) + : await inspectSevenZipMembers( + candidate, + selected, + options.sevenZip, + options.signal, + ); + demand( + sameFile(before, ordinaryFile(candidate.archive, 50_000_000_000)), + 'source archive changed.', + ); + const source = { + archive: candidate.archive, + bytes: candidate.bytes, + sha256: candidate.sha256, + ...(candidate.nexus ? { nexus: candidate.nexus } : {}), + }; + for (const item of containers) { + if (!selected.has(identity(item.record))) continue; + const match = matched.get(identity(item.record)); + if (!match) continue; + // Content-addressed equivalent archive copies share one source chain. + if ( + !item.chains.some( + (v) => + v.archiveSha256 === candidate.sha256 && + v.container.member === match.member, + ) + ) + item.chains.push({ + archiveSha256: candidate.sha256, + container: { + member: match.member, + bytes: item.record.bytes, + sha256: item.record.sha256, + }, + }); + } + inspectedArchives.push({ + archive: candidate.archive, + sha256: candidate.sha256, + }); + if (matched.size) sources.set(candidate.sha256, source); + } catch { + options.signal?.throwIfAborted(); + issues.push({ + archive: candidate.archive, + reason: + 'Original archive could not be verified safely; no nested matches accepted from it.', + }); + } + options.progress?.({ + phase: 'bsa-archives', + inspected: inspectedArchives.length, + issues: issues.length, + }); + } + let logicalBytes = 0, + references = 0; + for (const item of containers) { + options.signal?.throwIfAborted(); + if (!item.chains.length) { + issues.push({ + mod: item.mod, + path: item.record.path, + reason: 'No exact BSA member found in verified original candidates.', + }); + continue; + } + try { + guardRoot(); + const report = await inspectBsa104(item.file, { + expected: item.record, + signal: options.signal, + }); + guardRoot(); + demand( + logicalBytes + report.totalBytes <= limits.totalBytes, + 'combined BSA content exceeds the inspection budget.', + ); + const matched = new Map(); + for (const member of report.members) + if (wanted.has(identity(member))) + recordArchiveMatch(matched, identity(member), member.path); + const proposals = []; + for (const [key, match] of matched) + for (const file of wanted.get(key)) + for (const chain of item.chains) { + const value = { + ...chain, + member: match.member, + equivalentMembers: match.equivalentMembers, + }; + if ( + !file.bsaSourceMatches.some( + (v) => + v.archiveSha256 === value.archiveSha256 && + v.container.member === value.container.member && + v.member === value.member, + ) + ) + proposals.push({ file, value }); + demand( + references + proposals.length <= 500000, + 'nested match-reference limit exceeded.', + ); + } + demand( + references + proposals.length <= 500000, + 'nested match-reference limit exceeded.', + ); + for (const { file, value } of proposals) + file.bsaSourceMatches.push(value); + references += proposals.length; + logicalBytes += report.totalBytes; + inspectedContainers.push({ + mod: item.mod, + path: item.record.path, + bytes: item.record.bytes, + sha256: item.record.sha256, + members: report.members.length, + totalBytes: report.totalBytes, + matchedContent: matched.size, + }); + } catch { + options.signal?.throwIfAborted(); + issues.push({ + mod: item.mod, + path: item.record.path, + reason: + 'Installed BSA could not be verified safely within inspection limits; no matches accepted from it.', + }); + } + options.progress?.({ + phase: 'bsa-containers', + inspected: inspectedContainers.length, + issues: issues.length, + }); + } + options.signal?.throwIfAborted(); + guardRoot(); + result.bsaMapping = { + kind: 'archive-bsa104-member-sha256', + memberSelection: 'lexicographic-representative-per-content-per-container', + containerMods: [...options.containerMods], + ...(options.sevenZip ? { sevenZipVersion: options.sevenZip.version } : {}), + sources: [...sources.values()], + inspectedArchives, + inspectedContainers, + issues, + existingArchiveMatchedInstalledFiles: files.filter( + (v) => v.sourceMatches?.length, + ).length, + nestedMatchedInstalledFiles: files.filter((v) => v.bsaSourceMatches.length) + .length, + newlyMatchedInstalledFiles: files.filter( + (v) => !v.sourceMatches?.length && v.bsaSourceMatches.length, + ).length, + combinedMatchedInstalledFiles: files.filter( + (v) => v.sourceMatches?.length || v.bsaSourceMatches.length, + ).length, + combinedUnmatchedInstalledFiles: files.filter( + (v) => !v.sourceMatches?.length && !v.bsaSourceMatches.length, + ).length, + }; + result.installable = false; + result.publicationApproved = false; + result.blockers = [ + ...new Set([ + ...(result.blockers || []), + 'Nested BSA matches prove exact content only; review source declarations, reconstruction, credits and permissions before creating a recipe.', + ]), + ]; + return result; +} +module.exports = { mapBsaSources }; diff --git a/scripts/map-bsa-sources.mjs b/scripts/map-bsa-sources.mjs new file mode 100644 index 0000000..9affa7f --- /dev/null +++ b/scripts/map-bsa-sources.mjs @@ -0,0 +1,123 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import crypto from 'node:crypto'; +import { createRequire } from 'node:module'; +const require = createRequire(import.meta.url); +const { mapBsaSources } = require('../electron/bsa-mapping.cjs'); +const { createSevenZip } = require('../electron/sevenzip.cjs'); +const { ordinaryFile, sameFile } = require('../electron/native-package.cjs'); +const { canonical, contains } = require('../electron/live-catalog.cjs'); +const { readSourceHints } = require('../electron/source-hints.cjs'); +const [input, output, ...args] = process.argv.slice(2); +const containerMods = []; +let sevenZipPath; +for (let i = 0; i < args.length; i++) { + if (args[i] === '--mod' && args[i + 1] && !args[i + 1].startsWith('--')) + containerMods.push(args[++i]); + else if ( + args[i] === '--7zip' && + !sevenZipPath && + args[i + 1] && + !args[i + 1].startsWith('--') + ) + sevenZipPath = args[++i]; + else throw new Error('Unsupported or incomplete BSA mapping option.'); +} +if (!input || !output || !containerMods.length) + throw new Error( + 'Usage: map-bsa-sources.mjs --mod "Container mod" [--mod "Another mod"] [--7zip absolute-path-to-7z.exe]', + ); +function read(file) { + const before = ordinaryFile(file, 128_000_000), + content = fs.readFileSync(file); + if (!sameFile(before, ordinaryFile(file, 128_000_000))) + throw new Error('Mapping input changed while reading.'); + return { + value: JSON.parse( + new TextDecoder('utf-8', { fatal: true }).decode(content), + ), + sha256: crypto.createHash('sha256').update(content).digest('hex'), + }; +} +const ledgerInput = read(input), + ledger = ledgerInput.value; +if (typeof ledger.inventory?.file !== 'string') + throw new Error('Use a ledger with its inventory reference.'); +const inventoryInput = read(ledger.inventory.file); +if (inventoryInput.sha256 !== ledger.inventory.sha256) + throw new Error('Referenced inventory has changed.'); +const verifyHints = () => { + if ( + ledger.sourceHints && + readSourceHints(ledger.sourceHints.file).reference.sha256 !== + ledger.sourceHints.sha256 + ) + throw new Error('Referenced source hints have changed.'); +}; +verifyHints(); +const source = inventoryInput.value.source, + target = path.resolve(output); +const protectTarget = () => { + if (fs.existsSync(target)) + throw new Error('Choose a new mapping report path.'); + for (const root of [ + source.root, + source.gameFolder, + source.modsFolder, + source.profilesFolder, + source.profileFolder, + source.overwriteFolder, + ...ledger.mods.flatMap((mod) => + mod.archiveCandidates.map((v) => path.dirname(v.archive)), + ), + ].filter(Boolean)) + if (contains(canonical(root), canonical(target))) + throw new Error( + 'Keep mappings outside source installations and downloads.', + ); +}; +protectTarget(); +const controller = new AbortController(); +process.once('SIGINT', () => controller.abort()); +const mapped = await mapBsaSources(ledger, { + containerMods, + modsFolder: source.modsFolder, + signal: controller.signal, + sevenZip: sevenZipPath ? await createSevenZip(sevenZipPath) : undefined, + progress: (value) => console.log(JSON.stringify(value)), +}); +if ( + read(input).sha256 !== ledgerInput.sha256 || + read(ledger.inventory.file).sha256 !== inventoryInput.sha256 +) + throw new Error('Referenced ledger or inventory changed during mapping.'); +verifyHints(); +mapped.bsaMapping.inputLedger = { + file: path.resolve(input), + sha256: ledgerInput.sha256, +}; +const encoded = Buffer.from(JSON.stringify(mapped, null, 2) + '\n'); +if (encoded.length > 128_000_000) + throw new Error('Mapping report exceeds its size limit.'); +controller.signal.throwIfAborted(); +protectTarget(); +fs.mkdirSync(path.dirname(target), { recursive: true }); +protectTarget(); +fs.writeFileSync(target, encoded, { flag: 'wx' }); +const { sources, inspectedArchives, inspectedContainers, issues, ...summary } = + mapped.bsaMapping; +console.log( + JSON.stringify( + { + ...summary, + sources: sources.length, + inspectedArchives: inspectedArchives.length, + inspectedContainers: inspectedContainers.length, + issues: issues.length, + installable: false, + publicationApproved: false, + }, + null, + 2, + ), +); diff --git a/tests/bsa-mapping.test.mjs b/tests/bsa-mapping.test.mjs new file mode 100644 index 0000000..256cda1 --- /dev/null +++ b/tests/bsa-mapping.test.mjs @@ -0,0 +1,355 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import { spawnSync } from 'node:child_process'; +import { createRequire } from 'node:module'; +import { fileURLToPath } from 'node:url'; +import { bsaFixture } from './bsa-fixture.mjs'; +import { packNativeRecipe } from '../scripts/native-package.mjs'; +import { + createArchiveFixture, + fingerprint, +} from './native-archive-fixture.mjs'; +const require = createRequire(import.meta.url); +const { mapBsaSources } = require('../electron/bsa-mapping.cjs'); +const { createSevenZip } = require('../electron/sevenzip.cjs'); +const { mapArchiveSources } = require('../electron/archive-mapping.cjs'); +const { materializeNative } = require('../electron/native-materializer.cjs'); +const sourceRoot = fileURLToPath(new URL('..', import.meta.url)); + +async function fixture(kind = 'zip-windows', mutate, extraFiles = []) { + const encoded = bsaFixture( + [ + { path: 'Meshes/z.nif', data: 'same content', compressed: false }, + { + path: 'Meshes/a.nif', + data: 'same content', + alias: 0, + compressed: false, + }, + { + path: 'Textures/test.dds', + data: 'compressed content', + compressed: true, + }, + { path: 'empty.bin', data: '' }, + ], + 0x107, + ); + mutate?.(encoded); + const f = await createArchiveFixture(kind, { + extraFiles: [['Data/Example.bsa', encoded.bytes], ...extraFiles], + }); + f.modsFolder = path.join(f.root, 'installed-mods'); + fs.mkdirSync(path.join(f.modsFolder, 'Container'), { recursive: true }); + f.container = path.join(f.modsFolder, 'Container', 'Example.bsa'); + fs.writeFileSync(f.container, encoded.bytes); + const file = (name, data) => ({ + path: name, + ...fingerprint(Buffer.from(data)), + sourceMatches: [], + }); + const candidate = { + archive: f.archive, + ...fingerprint(fs.readFileSync(f.archive)), + }; + f.ledger = { + kind: 'loadstone-source-ledger', + schemaVersion: 1, + mapping: { kind: 'archive-member-sha256', matchedInstalledFiles: 2 }, + mods: [ + { + name: 'Container', + enabled: false, + installedFiles: [ + { + path: 'Example.bsa', + ...fingerprint(encoded.bytes), + sourceMatches: [ + { + archiveSha256: candidate.sha256, + member: 'Data/Example.bsa', + equivalentMembers: 1, + }, + ], + }, + ], + archiveCandidates: [candidate], + }, + { + name: 'Generated', + enabled: true, + installedFiles: [ + file('renamed.nif', 'same content'), + file('another.nif', 'same content'), + file('test.dds', 'compressed content'), + file('empty.bin', ''), + file('changed.nif', 'locally changed'), + file('meta.ini', 'same content'), + ], + archiveCandidates: [], + }, + ], + overwrite: [file('unmapped.nif', 'same content')], + installation: { gameFiles: [file('untouched.nif', 'same content')] }, + blockers: ['Source permissions remain unreviewed.'], + installable: false, + publicationApproved: false, + }; + f.ledger.mods[1].installedFiles[0].sourceMatches = [ + { + archiveSha256: 'a'.repeat(64), + member: 'prior.nif', + equivalentMembers: 1, + }, + ]; + f.nativeOptions = f.options; + f.options = { + modsFolder: f.modsFolder, + containerMods: ['Container'], + sevenZip: + kind === 'zip-windows' + ? undefined + : await createSevenZip(process.env.LOADSTONE_TEST_7ZIP), + }; + return f; +} + +for (const kind of ['zip-windows', '7z', 'rar']) + test( + `BSA mapping verifies the full ${kind} chain and preserves prior evidence and disabled state`, + { + skip: kind !== 'zip-windows' && !process.env.LOADSTONE_TEST_7ZIP, + }, + async () => { + const f = await fixture(kind), + original = structuredClone(f.ledger), + archiveBefore = fs.readFileSync(f.archive), + containerBefore = fs.readFileSync(f.container); + const mapped = await mapBsaSources(f.ledger, f.options); + assert.deepEqual(f.ledger, original); + assert.deepEqual(mapped.mapping, original.mapping); + assert.deepEqual(mapped.overwrite, original.overwrite); + assert.deepEqual(mapped.installation, original.installation); + assert.equal(mapped.mods[0].enabled, false); + for (let i = 0; i < 6; i++) + assert.deepEqual( + mapped.mods[1].installedFiles[i].sourceMatches, + original.mods[1].installedFiles[i].sourceMatches, + ); + assert.equal(mapped.bsaMapping.inspectedArchives.length, 1); + assert.equal(mapped.bsaMapping.inspectedContainers.length, 1); + assert.equal(mapped.bsaMapping.issues.length, 0); + assert.equal(mapped.bsaMapping.existingArchiveMatchedInstalledFiles, 2); + assert.equal(mapped.bsaMapping.nestedMatchedInstalledFiles, 4); + assert.equal(mapped.bsaMapping.newlyMatchedInstalledFiles, 3); + assert.equal(mapped.bsaMapping.combinedMatchedInstalledFiles, 5); + assert.equal(mapped.bsaMapping.combinedUnmatchedInstalledFiles, 2); + assert.deepEqual(mapped.mods[1].installedFiles[0].bsaSourceMatches, [ + { + archiveSha256: f.ledger.mods[0].archiveCandidates[0].sha256, + container: { + member: 'Data/Example.bsa', + ...fingerprint(containerBefore), + }, + member: 'Meshes/a.nif', + equivalentMembers: 2, + }, + ]); + assert.equal(mapped.mods[1].installedFiles[4].bsaSourceMatches.length, 0); + assert.equal(mapped.mods[1].installedFiles[5].bsaSourceMatches.length, 0); + assert.equal(mapped.installable, false); + assert.equal(mapped.publicationApproved, false); + assert.deepEqual(fs.readFileSync(f.archive), archiveBefore); + assert.deepEqual(fs.readFileSync(f.container), containerBefore); + // Recompute the nested extension without multiplying old references. + const rerun = await mapBsaSources(mapped, f.options); + assert.deepEqual(rerun, mapped); + }, + ); + +test('BSA mapping rejects changed originals, changed local witnesses and unsupported containers without partial matches', async (t) => { + for (const change of [ + 'archive', + 'container', + 'unsupported', + 'unsafe-outer', + 'metadata-conflict', + 'missing', + ]) + await t.test(change, async () => { + const f = await fixture( + 'zip-windows', + change === 'unsupported' + ? (v) => v.bytes.writeUInt32LE(105, 4) + : undefined, + change === 'unsafe-outer' + ? [['redirect', Buffer.from('elsewhere'), 0o120777]] + : [], + ); + // The reusable archive fixture keeps only pair values; add an unsafe ZIP directly. + if (change === 'unsafe-outer') { + const { storedZip } = await import('./native-archive-fixture.mjs'); + fs.writeFileSync( + f.archive, + storedZip([ + ['Data/Example.bsa', fs.readFileSync(f.container)], + ['redirect', 'elsewhere', 0o120777], + ]), + ); + Object.assign( + f.ledger.mods[0].archiveCandidates[0], + fingerprint(fs.readFileSync(f.archive)), + ); + } + if (change === 'archive') fs.appendFileSync(f.archive, 'changed'); + if (change === 'container') fs.appendFileSync(f.container, 'changed'); + if (change === 'metadata-conflict') + f.ledger.mods[0].archiveCandidates[0].metadataConflict = true; + if (change === 'missing') fs.unlinkSync(f.container); + const mapped = await mapBsaSources(f.ledger, f.options); + assert.equal(mapped.bsaMapping.nestedMatchedInstalledFiles, 0); + assert.equal(mapped.bsaMapping.inspectedContainers.length, 0); + assert.ok(mapped.bsaMapping.issues.length > 0); + assert.equal(mapped.bsaMapping.existingArchiveMatchedInstalledFiles, 2); + }); +}); + +test('BSA mapper bounds selections, rejects invalid ledger paths and fingerprints, and propagates cancellation', async () => { + const f = await fixture(); + for (const containerMods of [ + [], + ['container'], + ['Container', 'Container'], + ['../Container'], + Array(65).fill('Container'), + ]) + await assert.rejects( + mapBsaSources(f.ledger, { ...f.options, containerMods }), + /BSA mapping/, + ); + for (const mutate of [ + (v) => { + v.mods[0].name = '../Container'; + }, + (v) => { + v.mods[0].installedFiles[0].path = '../escape.bsa'; + }, + (v) => { + v.mods[0].installedFiles[0].bytes = 0xffffffff + 1; + }, + (v) => { + v.mods[0].installedFiles[0].sha256 = 'bad'; + }, + (v) => { + v.mods[0].archiveCandidates[0].archive = 'relative.zip'; + }, + (v) => { + v.mods.push(structuredClone(v.mods[0])); + }, + ]) { + const changed = structuredClone(f.ledger); + mutate(changed); + await assert.rejects(mapBsaSources(changed, f.options), /BSA mapping/); + } + const controller = new AbortController(); + await assert.rejects( + mapBsaSources(f.ledger, { + ...f.options, + signal: controller.signal, + progress: () => controller.abort(), + }), + { name: 'AbortError' }, + ); + assert.equal(f.ledger.mods[1].installedFiles[0].bsaSourceMatches, undefined); +}); + +test('mapped nested references reconstruct exact outputs through the native recipe engine', async () => { + const f = await fixture(), + mapped = await mapBsaSources(f.ledger, f.options); + const selected = mapped.mods[1].installedFiles.filter( + (v) => v.bsaSourceMatches.length, + ); + for (const file of selected) { + const match = file.bsaSourceMatches[0]; + assert.equal( + match.archiveSha256, + f.recipe.sources.find((v) => v.id === 'original-archive').sha256, + ); + f.recipe.outputs.push({ + path: `mods/Generated/${file.path}`, + bytes: file.bytes, + sha256: file.sha256, + action: { + type: 'extract-bsa104', + source: 'original-archive', + container: match.container, + member: match.member, + }, + }); + } + const packageFile = path.join(f.packages, 'mapped.loadstone'); + await packNativeRecipe(f.recipe, f.payloads, packageFile); + const result = await materializeNative( + packageFile, + f.destination, + f.nativeOptions, + ); + assert.equal(result.status, 'verified'); + for (const file of selected) + assert.deepEqual( + fingerprint( + fs.readFileSync(path.join(f.destination, 'mods/Generated', file.path)), + ), + { bytes: file.bytes, sha256: file.sha256 }, + ); + const direct = await mapArchiveSources(mapped); + assert.equal(direct.bsaMapping, undefined); + assert.ok( + direct.mods.every((mod) => + mod.installedFiles.every((file) => file.bsaSourceMatches === undefined), + ), + ); + assert.equal(mapped.bsaMapping.nestedMatchedInstalledFiles, 4); +}); + +test('BSA mapping CLI fingerprints its input and preserves reports and source directories', async () => { + const f = await fixture(), + inventory = path.join(f.root, 'inventory.json'), + input = path.join(f.root, 'ledger.json'), + output = path.join(f.root, 'reports', 'mapping.json'); + fs.writeFileSync( + inventory, + JSON.stringify({ source: { modsFolder: f.modsFolder } }), + ); + f.ledger.inventory = { + file: inventory, + sha256: fingerprint(fs.readFileSync(inventory)).sha256, + }; + fs.writeFileSync(input, JSON.stringify(f.ledger)); + const run = (target) => + spawnSync( + process.execPath, + ['scripts/map-bsa-sources.mjs', input, target, '--mod', 'Container'], + { cwd: sourceRoot, encoding: 'utf8', timeout: 60000, windowsHide: true }, + ); + const result = run(output); + assert.equal(result.status, 0, result.stderr); + const before = fs.readFileSync(output), + mapped = JSON.parse(before); + assert.equal( + mapped.bsaMapping.inputLedger.sha256, + fingerprint(fs.readFileSync(input)).sha256, + ); + assert.equal(mapped.bsaMapping.nestedMatchedInstalledFiles, 4); + assert.notEqual(run(output).status, 0); + assert.deepEqual(fs.readFileSync(output), before); + const protectedOutput = path.join(f.modsFolder, 'forbidden.json'); + assert.notEqual(run(protectedOutput).status, 0); + assert.equal(fs.existsSync(protectedOutput), false); + const changed = path.join(f.root, 'changed.json'); + fs.appendFileSync(inventory, '\n'); + assert.notEqual(run(changed).status, 0); + assert.equal(fs.existsSync(changed), false); +});