diff --git a/docs/installed-inventory.md b/docs/installed-inventory.md index 366eaef..c334dfc 100644 --- a/docs/installed-inventory.md +++ b/docs/installed-inventory.md @@ -62,6 +62,8 @@ node scripts/map-zip-sources.mjs private-ledger.json new-private-mapping.json The member mapper verifies each candidate ZIP against its locked archive hash, then streams member hashes and compares them with installed size/SHA-256 pairs. It records exact member paths, retains matching archive candidates and leaves locally changed files unresolved. It does not extract files or execute content. MO2 `meta.ini` is excluded because it needs privacy-aware regeneration. Each ZIP is accepted as a whole: unsafe paths, duplicate/colliding names, links, changed input or limit violations discard its potential matches. A cancellation aborts the pass. +Upstream ZIPs may use Windows backslashes in their internal names. These separators are normalized to `/` before path validation, collision checks and member selection; the original archive is never rewritten. Slash/backslash aliases, case collisions, traversal, absolute paths and file/directory overlaps still reject the entire archive. Mapped member paths use canonical forward slashes and can be passed directly to native reconstruction. `.loadstone` package entries and recipe paths remain strict and do not accept backslashes. + Limits per archive: 50 GB compressed input, 500,000 entries, 100 GB declared uncompressed content, 1 GB per hashed member and 20 GB of hashed member data. The report accepts at most 500,000 match references. Exact byte matches still require source/rights review, profile and root reconstruction, and actual installation acceptance. When multiple members of one archive have identical size and SHA-256, the mapper keeps the lexicographically first normalized member path as a deterministic representative and records `equivalentMembers`, the count of verified members with those exact bytes. The mapping declares `memberSelection: "lexicographic-representative-per-content"`. This compact representation replaces the earlier 128-equivalent-member rejection and prevents the installed-file count from multiplying by the number of identical copies. Every selected member is still hashed; complete archive validation and stream checks must succeed before any matches are accepted. Different archives remain distinct candidates. A representative proves a byte-equivalent source, not which archive member the original installation used. Recipes still require review of the source and exact selected member. diff --git a/docs/native-reconstruction.md b/docs/native-reconstruction.md index ea201b1..0571ac1 100644 --- a/docs/native-reconstruction.md +++ b/docs/native-reconstruction.md @@ -43,6 +43,8 @@ In the desktop workflow only, an archive map value may instead be `null` to requ All requirements and archive hashes are checked before creating the installation directory. ZIP, 7z, RAR4 and RAR5 inputs are recognized by their signatures, allowing mislabeled original filenames without renaming them; signatures never replace hash or structural validation. Archive paths, links, duplicate/colliding names, encrypted/unsupported entries, required members and declared sizes are checked in advance. Available destination space must cover expected outputs, recipe metadata and a 64 MB reserve. Archive inventories are limited to 500,000 entries and 100 GB declared uncompressed content; local archives are bounded to 50 GB. +For upstream ZIP sources, backslashes are normalized to forward slashes before structural checks and during extraction, matching the source mapper's member paths. Mixed-separator aliases, case collisions and file/directory overlaps are rejected before creating a destination. Recipes still use canonical `/` member paths, and native `.loadstone` package entries retain strict naming. Original archives are read unchanged; this compatibility rule does not permit arbitrary path cleanup or remove traversal checks. + 7z/RAR reconstruction uses the same validated adapter as [source comparison](installed-inventory.md). The external tool runs without a shell and emits file content to stdout; it never receives output paths. Loadstone waits for disk writes before reading more data, checks exit status and input identity, and closes the child before reporting failure or cancellation. Tool diagnostics do not enter build history. 7-Zip 26.04 with `zlib.crc32` uses one pass for eligible 7z archives with per-member CRCs and at most 20 GB total content. Every boundary, including unselected members, is CRC-checked. Up to 100,000 selected members are supported in that pass. Other versions, RAR and ineligible 7z archives read each selected member with literal selectors: at most 10,000 members and 20 GB selected content per archive. This fallback can be slow for solid archives. Single-pass extraction has a five-minute limit; fallback reads have a ten-minute limit per member. These are experimental creator limits, not general archive support guarantees. Destinations cannot overlap selected game inputs, source archive/package folders, the selected tool directory, the application, system folders or additional protected roots supplied by the caller. Redirected directories are rejected; existing destinations require a separate bound recovery review in the desktop builder. New directory creation is exclusive; output files use exclusive creation. The materializer never modifies source game files or downloads, executes archive content or writes save paths. diff --git a/electron/archive-mapping.cjs b/electron/archive-mapping.cjs index ecd492b..77c9a58 100644 --- a/electron/archive-mapping.cjs +++ b/electron/archive-mapping.cjs @@ -26,7 +26,8 @@ async function inspectZipMembers(candidate, wanted, signal) { lazyEntries: true, autoClose: false, validateEntrySizes: true, - strictFileNames: true, + // Normalize upstream Windows separators before the path/collision checks. + strictFileNames: false, }); const paths = new Map(), matches = new Map(); diff --git a/electron/native-materializer.cjs b/electron/native-materializer.cjs index fd2aca0..91b5d71 100644 --- a/electron/native-materializer.cjs +++ b/electron/native-materializer.cjs @@ -267,6 +267,7 @@ async function materializeNative(file, destination, options = {}) { if (output.action.type === 'extract' || input.package) { const group = groups.get(input.file) || { file: input.file, + package: input.package === true, snapshot: input.snapshot, candidate: input.candidate, inspectedArchive: input.inspectedArchive, @@ -302,7 +303,9 @@ async function materializeNative(file, destination, options = {}) { const zip = await yauzl.openPromise(group.file, { lazyEntries: true, autoClose: false, - strictFileNames: true, + // Only upstream archives may use Windows separators; native packages + // retain their canonical names. zipPath and collision checks run next. + strictFileNames: group.package, validateEntrySizes: true, }); const entries = new Map(), @@ -616,7 +619,7 @@ async function materializeNative(file, destination, options = {}) { const zip = await yauzl.openPromise(group.file, { lazyEntries: true, autoClose: false, - strictFileNames: true, + strictFileNames: group.package, validateEntrySizes: true, }); const produced = new Set(); diff --git a/tests/native-archive-fixture.mjs b/tests/native-archive-fixture.mjs index 89c752a..6f0d458 100644 --- a/tests/native-archive-fixture.mjs +++ b/tests/native-archive-fixture.mjs @@ -15,6 +15,47 @@ const uint32 = (value) => { buffer.writeUInt32LE(value); return buffer; }; +// Minimal stored ZIP encoder for synthetic raw names that normal writers +// sanitize. Local and central names agree; payloads are never executable. +export function storedZip(files) { + const local = [], + central = []; + let offset = 0; + for (const [name, data, mode = 0o100644] of files) { + const encoded = Buffer.from(name), + content = Buffer.from(data); + const header = Buffer.alloc(30), + directory = Buffer.alloc(46); + header.writeUInt32LE(0x04034b50, 0); + header.writeUInt16LE(20, 4); + header.writeUInt16LE(0x800, 6); + header.writeUInt32LE(crc32(content), 14); + header.writeUInt32LE(content.length, 18); + header.writeUInt32LE(content.length, 22); + header.writeUInt16LE(encoded.length, 26); + directory.writeUInt32LE(0x02014b50, 0); + directory.writeUInt16LE(0x314, 4); + directory.writeUInt16LE(20, 6); + directory.writeUInt16LE(0x800, 8); + directory.writeUInt32LE(crc32(content), 16); + directory.writeUInt32LE(content.length, 20); + directory.writeUInt32LE(content.length, 24); + directory.writeUInt16LE(encoded.length, 28); + directory.writeUInt32LE((mode * 65536) >>> 0, 38); + directory.writeUInt32LE(offset, 42); + local.push(header, encoded, content); + central.push(directory, encoded); + offset += header.length + encoded.length + content.length; + } + const table = Buffer.concat(central), + end = Buffer.alloc(22); + end.writeUInt32LE(0x06054b50, 0); + end.writeUInt16LE(files.length, 8); + end.writeUInt16LE(files.length, 10); + end.writeUInt32LE(table.length, 12); + end.writeUInt32LE(offset, 16); + return Buffer.concat([...local, table, end]); +} const vint = (number) => { const bytes = []; do { @@ -81,7 +122,10 @@ export async function createArchiveFixture( ['Data/unused.bin', Buffer.alloc(500000, 88)], ...extraFiles, ]); - f.archive = path.join(f.packages, `source.${kind === '7z' ? '7z' : 'rar'}`); + f.archive = path.join( + f.packages, + `source.${kind === '7z' ? '7z' : kind === 'zip-windows' ? 'zip' : 'rar'}`, + ); if (kind === '7z') { const input = path.join(f.root, 'archive-input'); for (const [name, data] of allFiles) { @@ -101,6 +145,13 @@ export async function createArchiveFixture( ], { cwd: input, windowsHide: true, stdio: 'ignore' }, ); + } else if (kind === 'zip-windows') { + fs.writeFileSync( + f.archive, + storedZip( + [...allFiles].map(([name, data]) => [name.replaceAll('/', '\\'), data]), + ), + ); } else fs.writeFileSync(f.archive, storedRar5(allFiles)); f.recipe.sources.push({ id: 'original-archive', diff --git a/tests/native-archives.test.mjs b/tests/native-archives.test.mjs index 4a5b320..bd06f65 100644 --- a/tests/native-archives.test.mjs +++ b/tests/native-archives.test.mjs @@ -75,138 +75,148 @@ test( }, ); -test( - 'real Electron acquires a reviewed archive, displays progress and reconstructs exact outputs through the bridge', - options, - async () => { - const f = await createArchiveFixture('rar5', { timestamps: true }); - fs.writeFileSync( - f.inputsFile, - JSON.stringify({ - games: { fnv: f.game }, - archives: { 'original-archive': null }, - }), - ); - const env = { ...process.env, LOADSTONE_DATA_DIR: f.state }; - delete env.ELECTRON_RUN_AS_NODE; - const app = await electron.launch({ - executablePath: process.env.LOADSTONE_TEST_EXE || require('electron'), - args: process.env.LOADSTONE_TEST_EXE ? [] : [path.resolve('.')], - env, - }); - try { - const page = await app.firstWindow(); - await page - .getByRole('button', { name: 'Explore the demo', exact: true }) - .click(); - await page.getByRole('button', { name: 'Settings', exact: true }).click(); - await page.evaluate(() => - window.loadstone.settings({ dummyData: false }), +for (const kind of ['rar5', 'zip-windows']) + test( + `real Electron acquires a reviewed ${kind} archive, displays progress and reconstructs exact outputs through the bridge`, + options, + async () => { + const f = await createArchiveFixture(kind, { timestamps: true }); + fs.writeFileSync( + f.inputsFile, + JSON.stringify({ + games: { fnv: f.game }, + archives: { 'original-archive': null }, + }), ); - await app.evaluate( - ({ dialog }, fixture) => { - const fs = process.getBuiltinModule('fs'); - const https = process.getBuiltinModule('https'); - const { Readable } = process.getBuiltinModule('stream'); - const { EventEmitter } = process.getBuiltinModule('events'); - const files = [ - fixture.packageFile, - fixture.inputsFile, - fixture.parent, - fixture.executable, - ]; - dialog.showOpenDialog = async () => ({ - canceled: false, - filePaths: [files.shift()], - }); - https.get = (url, options, callback) => { - if ( - url.href !== 'https://example.org/original' || - options.headers.apikey - ) - throw new Error('Unexpected fixture network request'); - const bytes = fs.readFileSync(fixture.archive); - const stream = Readable.from( - (async function* () { - yield bytes.subarray(0, 64000); - await new Promise((resolve) => setTimeout(resolve, 1700)); - yield bytes.subarray(64000, 128000); - await new Promise((resolve) => setTimeout(resolve, 1700)); - yield bytes.subarray(128000); - })(), - ); - stream.statusCode = 200; - stream.headers = {}; - const request = new EventEmitter(); - request.setTimeout = () => {}; - request.destroy = (error) => { - stream.destroy(error); - request.emit('error', error); + const env = { ...process.env, LOADSTONE_DATA_DIR: f.state }; + delete env.ELECTRON_RUN_AS_NODE; + const app = await electron.launch({ + executablePath: process.env.LOADSTONE_TEST_EXE || require('electron'), + args: process.env.LOADSTONE_TEST_EXE ? [] : [path.resolve('.')], + env, + }); + try { + const page = await app.firstWindow(); + await page + .getByRole('button', { name: 'Explore the demo', exact: true }) + .click(); + await page + .getByRole('button', { name: 'Settings', exact: true }) + .click(); + await page.evaluate(() => + window.loadstone.settings({ dummyData: false }), + ); + await app.evaluate( + ({ dialog }, fixture) => { + const fs = process.getBuiltinModule('fs'); + const https = process.getBuiltinModule('https'); + const { Readable } = process.getBuiltinModule('stream'); + const { EventEmitter } = process.getBuiltinModule('events'); + const files = [ + fixture.packageFile, + fixture.inputsFile, + fixture.parent, + fixture.executable, + ]; + dialog.showOpenDialog = async () => ({ + canceled: false, + filePaths: [files.shift()], + }); + https.get = (url, options, callback) => { + if ( + url.href !== 'https://example.org/original' || + options.headers.apikey + ) + throw new Error('Unexpected fixture network request'); + const bytes = fs.readFileSync(fixture.archive); + const stream = Readable.from( + (async function* () { + yield bytes.subarray(0, 64000); + await new Promise((resolve) => setTimeout(resolve, 1700)); + yield bytes.subarray(64000, 128000); + await new Promise((resolve) => setTimeout(resolve, 1700)); + yield bytes.subarray(128000); + })(), + ); + stream.statusCode = 200; + stream.headers = {}; + const request = new EventEmitter(); + request.setTimeout = () => {}; + request.destroy = (error) => { + stream.destroy(error); + request.emit('error', error); + }; + options.signal.addEventListener( + 'abort', + () => request.destroy(new Error('cancelled')), + { once: true }, + ); + queueMicrotask(() => callback(stream)); + return request; }; - options.signal.addEventListener( - 'abort', - () => request.destroy(new Error('cancelled')), - { once: true }, - ); - queueMicrotask(() => callback(stream)); - return request; - }; - }, - { ...f, executable }, - ); - await page - .getByRole('button', { name: 'Choose local build inputs', exact: true }) - .click(); - const review = page.getByRole('region', { name: 'Review native build' }); - await review.waitFor(); - assert.match(await review.textContent(), /1\s+archive to acquire/); - assert.match(await review.textContent(), /example.org/); - await review - .getByRole('button', { name: 'Build candidate', exact: true }) - .click(); - await page - .getByText('Acquiring archive files', { exact: true }) - .waitFor(); - fs.mkdirSync('test-results', { recursive: true }); - await page - .getByRole('region', { name: 'Native candidate builds' }) - .screenshot({ path: 'test-results/native-download-progress.png' }); - await page - .getByText('Files verified · Launch untested', { exact: true }) - .waitFor(); - const jobs = await page.evaluate(() => window.loadstone.nativeBuilds()); - assert.equal(jobs[0].status, 'verified'); - for (const output of f.recipe.outputs) - assert.equal( - Math.round( - fs.statSync(path.join(jobs[0].destination, output.path)).mtimeMs, - ), - output.mtimeMs, + }, + { ...f, executable }, ); - for (const output of f.recipe.outputs) + await page + .getByRole('button', { + name: 'Choose local build inputs', + exact: true, + }) + .click(); + const review = page.getByRole('region', { + name: 'Review native build', + }); + await review.waitFor(); + assert.match(await review.textContent(), /1\s+archive to acquire/); + assert.match(await review.textContent(), /example.org/); + await review + .getByRole('button', { name: 'Build candidate', exact: true }) + .click(); + await page + .getByText('Acquiring archive files', { exact: true }) + .waitFor(); + fs.mkdirSync('test-results', { recursive: true }); + await page + .getByRole('region', { name: 'Native candidate builds' }) + .screenshot({ + path: `test-results/native-download-progress-${kind}.png`, + }); + await page + .getByText('Files verified · Launch untested', { exact: true }) + .waitFor(); + const jobs = await page.evaluate(() => window.loadstone.nativeBuilds()); + assert.equal(jobs[0].status, 'verified'); + for (const output of f.recipe.outputs) + assert.equal( + Math.round( + fs.statSync(path.join(jobs[0].destination, output.path)).mtimeMs, + ), + output.mtimeMs, + ); + for (const output of f.recipe.outputs) + assert.equal( + fingerprint( + fs.readFileSync(path.join(jobs[0].destination, output.path)), + ).sha256, + output.sha256, + ); + const history = fs.readFileSync( + path.join(f.state, 'native-builds.json'), + 'utf8', + ); + assert.equal(history.includes('example.org'), false); + assert.equal(history.includes('native-downloads'), false); assert.equal( - fingerprint( - fs.readFileSync(path.join(jobs[0].destination, output.path)), - ).sha256, - output.sha256, + await page.evaluate( + async () => (await window.loadstone.liveJobs()).length, + ), + 0, ); - const history = fs.readFileSync( - path.join(f.state, 'native-builds.json'), - 'utf8', - ); - assert.equal(history.includes('example.org'), false); - assert.equal(history.includes('native-downloads'), false); - assert.equal( - await page.evaluate( - async () => (await window.loadstone.liveJobs()).length, - ), - 0, - ); - } finally { - await app.close(); - } - }, -); + } finally { + await app.close(); + } + }, + ); test( 'native CLI reconstructs a RAR candidate using an explicit operator-selected archive tool', diff --git a/tests/zip-paths.test.mjs b/tests/zip-paths.test.mjs new file mode 100644 index 0000000..f1de39d --- /dev/null +++ b/tests/zip-paths.test.mjs @@ -0,0 +1,162 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import { createRequire } from 'node:module'; +import { + createArchiveFixture, + fingerprint, + storedZip, +} from './native-archive-fixture.mjs'; +import { packNativeRecipe } from '../scripts/native-package.mjs'; +const require = createRequire(import.meta.url); +const { mapArchiveSources } = require('../electron/archive-mapping.cjs'); +const { materializeNative } = require('../electron/native-materializer.cjs'); +const { inspectNativePackage } = require('../electron/native-package.cjs'); +const { + canonicalJson, + validateRecipe, +} = require('../electron/native-format.cjs'); + +function ledger(f) { + return { + kind: 'loadstone-source-ledger', + schemaVersion: 1, + mods: [ + { + name: 'Example', + enabled: true, + installedFiles: [...f.files].map(([name, data]) => ({ + path: name, + ...fingerprint(data), + })), + archiveCandidates: [ + { archive: f.archive, ...fingerprint(fs.readFileSync(f.archive)) }, + ], + }, + ], + }; +} +async function replaceArchive(f, entries) { + fs.writeFileSync(f.archive, storedZip(entries)); + Object.assign( + f.recipe.sources.find((s) => s.id === 'original-archive'), + fingerprint(fs.readFileSync(f.archive)), + ); + f.packageFile = path.join(f.packages, 'reviewed.loadstone'); + await packNativeRecipe(f.recipe, f.payloads, f.packageFile); +} + +test('Windows ZIP names map to canonical members and reconstruct exact files without altering the archive', async () => { + const f = await createArchiveFixture('zip-windows'); + await replaceArchive(f, [ + ['Data\\', Buffer.alloc(0), 0o040755], + ...[...f.files].map(([name, data], i) => [ + i % 2 ? name.replaceAll('/', '\\') : name, + data, + ]), + ]); + const before = fs.readFileSync(f.archive), + beforeTime = fs.statSync(f.archive).mtimeMs; + const mapped = await mapArchiveSources(ledger(f)); + assert.equal(mapped.mapping.issues.length, 0); + assert.equal(mapped.mapping.matchedInstalledFiles, f.files.size); + for (const file of mapped.mods[0].installedFiles) { + const match = file.sourceMatches[0]; + assert.equal(match.member, file.path); + assert.equal(match.member.includes('\\'), false); + for (const output of f.recipe.outputs.filter( + (v) => v.action.member === file.path, + )) + output.action.member = match.member; + } + f.packageFile = path.join(f.packages, 'mapped.loadstone'); + await packNativeRecipe(f.recipe, f.payloads, f.packageFile); + const result = await materializeNative( + f.packageFile, + f.destination, + f.options, + ); + assert.equal(result.status, 'verified'); + for (const output of f.recipe.outputs) + assert.equal( + fingerprint(fs.readFileSync(path.join(f.destination, output.path))) + .sha256, + output.sha256, + ); + assert.deepEqual(fs.readFileSync(f.archive), before); + assert.equal(fs.statSync(f.archive).mtimeMs, beforeTime); +}); + +test('ZIP separator normalization rejects traversal, aliases, links and file/directory overlap before destination writes', async () => { + const badGroups = [ + ...[ + '..\\outside', + 'Data\\..\\outside', + '\\absolute', + 'C:\\absolute', + '\\\\server\\share', + 'Data\\\\double', + 'Data\\.\\relative', + 'Data\\nul.txt', + 'Data\\trailing.', + 'Data\\file:stream', + 'Data\\e\u0301.txt', + ].map((name) => [[name, Buffer.from('bad')]]), + [['Data\\A[1].esp', Buffer.from('alias')]], + [['data\\a[1].ESP', Buffer.from('case alias')]], + [['Data', Buffer.from('file conflicts with children')]], + [ + ['Data\\Alias', Buffer.from('file')], + ['Data/Alias/Child', Buffer.from('child')], + ], + [ + ['Data\\Alias\\Child', Buffer.from('child')], + ['Data/Alias', Buffer.from('file')], + ], + [ + ['Data\\Alias\\', Buffer.alloc(0), 0o040755], + ['Data/Alias', Buffer.from('file')], + ], + [['Data\\redirect', Buffer.from('elsewhere'), 0o120777]], + ]; + for (const extras of badGroups) { + const f = await createArchiveFixture('zip-windows'); + await replaceArchive(f, [...f.files, ...extras]); + const before = fingerprint(fs.readFileSync(f.archive)); + const mapped = await mapArchiveSources(ledger(f)); + assert.equal(mapped.mapping.inspected.length, 0, extras[0][0]); + assert.equal(mapped.mapping.issues.length, 1, extras[0][0]); + assert.equal(mapped.mapping.matchedInstalledFiles, 0, extras[0][0]); + await assert.rejects( + materializeNative(f.packageFile, f.destination, f.options), + ); + assert.equal(fs.existsSync(f.destination), false, extras[0][0]); + assert.deepEqual(fingerprint(fs.readFileSync(f.archive)), before); + } +}); + +test('native package and recipe names remain strict when upstream ZIP separators are normalized', async () => { + const f = await createArchiveFixture('zip-windows'); + const invalid = structuredClone(f.recipe); + invalid.outputs.find((v) => v.action.type === 'extract').action.member = + 'Data\\00-empty.txt'; + assert.throws(() => validateRecipe(invalid), /safe member path/); + const packageFile = path.join(f.packages, 'backslash-payload.loadstone'); + const entries = [ + ['recipe.json', Buffer.from(canonicalJson(f.recipe))], + ...f.recipe.sources + .filter((s) => s.type === 'payload') + .map((s) => [ + `payload\\${s.sha256}`, + fs.readFileSync(path.join(f.payloads, s.sha256)), + ]), + ]; + fs.writeFileSync(packageFile, storedZip(entries)); + await assert.rejects(inspectNativePackage(packageFile), /invalid characters/); + await assert.rejects( + materializeNative(packageFile, f.destination, f.options), + /invalid characters/, + ); + assert.equal(fs.existsSync(f.destination), false); +});