Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/installed-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ node scripts/map-zip-sources.mjs private-ledger.json new-private-mapping.json

The member mapper verifies each candidate ZIP against its locked archive hash, then streams member hashes and compares them with installed size/SHA-256 pairs. It records exact member paths, retains matching archive candidates and leaves locally changed files unresolved. It does not extract files or execute content. MO2 `meta.ini` is excluded because it needs privacy-aware regeneration. Each ZIP is accepted as a whole: unsafe paths, duplicate/colliding names, links, changed input or limit violations discard its potential matches. A cancellation aborts the pass.

Upstream ZIPs may use Windows backslashes in their internal names. These separators are normalized to `/` before path validation, collision checks and member selection; the original archive is never rewritten. Slash/backslash aliases, case collisions, traversal, absolute paths and file/directory overlaps still reject the entire archive. Mapped member paths use canonical forward slashes and can be passed directly to native reconstruction. `.loadstone` package entries and recipe paths remain strict and do not accept backslashes.

Limits per archive: 50 GB compressed input, 500,000 entries, 100 GB declared uncompressed content, 1 GB per hashed member and 20 GB of hashed member data. The report accepts at most 500,000 match references. Exact byte matches still require source/rights review, profile and root reconstruction, and actual installation acceptance.

When multiple members of one archive have identical size and SHA-256, the mapper keeps the lexicographically first normalized member path as a deterministic representative and records `equivalentMembers`, the count of verified members with those exact bytes. The mapping declares `memberSelection: "lexicographic-representative-per-content"`. This compact representation replaces the earlier 128-equivalent-member rejection and prevents the installed-file count from multiplying by the number of identical copies. Every selected member is still hashed; complete archive validation and stream checks must succeed before any matches are accepted. Different archives remain distinct candidates. A representative proves a byte-equivalent source, not which archive member the original installation used. Recipes still require review of the source and exact selected member.
Expand Down
2 changes: 2 additions & 0 deletions docs/native-reconstruction.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ In the desktop workflow only, an archive map value may instead be `null` to requ

All requirements and archive hashes are checked before creating the installation directory. ZIP, 7z, RAR4 and RAR5 inputs are recognized by their signatures, allowing mislabeled original filenames without renaming them; signatures never replace hash or structural validation. Archive paths, links, duplicate/colliding names, encrypted/unsupported entries, required members and declared sizes are checked in advance. Available destination space must cover expected outputs, recipe metadata and a 64 MB reserve. Archive inventories are limited to 500,000 entries and 100 GB declared uncompressed content; local archives are bounded to 50 GB.

For upstream ZIP sources, backslashes are normalized to forward slashes before structural checks and during extraction, matching the source mapper's member paths. Mixed-separator aliases, case collisions and file/directory overlaps are rejected before creating a destination. Recipes still use canonical `/` member paths, and native `.loadstone` package entries retain strict naming. Original archives are read unchanged; this compatibility rule does not permit arbitrary path cleanup or remove traversal checks.

7z/RAR reconstruction uses the same validated adapter as [source comparison](installed-inventory.md). The external tool runs without a shell and emits file content to stdout; it never receives output paths. Loadstone waits for disk writes before reading more data, checks exit status and input identity, and closes the child before reporting failure or cancellation. Tool diagnostics do not enter build history. 7-Zip 26.04 with `zlib.crc32` uses one pass for eligible 7z archives with per-member CRCs and at most 20 GB total content. Every boundary, including unselected members, is CRC-checked. Up to 100,000 selected members are supported in that pass. Other versions, RAR and ineligible 7z archives read each selected member with literal selectors: at most 10,000 members and 20 GB selected content per archive. This fallback can be slow for solid archives. Single-pass extraction has a five-minute limit; fallback reads have a ten-minute limit per member. These are experimental creator limits, not general archive support guarantees.

Destinations cannot overlap selected game inputs, source archive/package folders, the selected tool directory, the application, system folders or additional protected roots supplied by the caller. Redirected directories are rejected; existing destinations require a separate bound recovery review in the desktop builder. New directory creation is exclusive; output files use exclusive creation. The materializer never modifies source game files or downloads, executes archive content or writes save paths.
Expand Down
3 changes: 2 additions & 1 deletion electron/archive-mapping.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ async function inspectZipMembers(candidate, wanted, signal) {
lazyEntries: true,
autoClose: false,
validateEntrySizes: true,
strictFileNames: true,
// Normalize upstream Windows separators before the path/collision checks.
strictFileNames: false,
});
const paths = new Map(),
matches = new Map();
Expand Down
7 changes: 5 additions & 2 deletions electron/native-materializer.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,7 @@ async function materializeNative(file, destination, options = {}) {
if (output.action.type === 'extract' || input.package) {
const group = groups.get(input.file) || {
file: input.file,
package: input.package === true,
snapshot: input.snapshot,
candidate: input.candidate,
inspectedArchive: input.inspectedArchive,
Expand Down Expand Up @@ -302,7 +303,9 @@ async function materializeNative(file, destination, options = {}) {
const zip = await yauzl.openPromise(group.file, {
lazyEntries: true,
autoClose: false,
strictFileNames: true,
// Only upstream archives may use Windows separators; native packages
// retain their canonical names. zipPath and collision checks run next.
strictFileNames: group.package,
validateEntrySizes: true,
});
const entries = new Map(),
Expand Down Expand Up @@ -616,7 +619,7 @@ async function materializeNative(file, destination, options = {}) {
const zip = await yauzl.openPromise(group.file, {
lazyEntries: true,
autoClose: false,
strictFileNames: true,
strictFileNames: group.package,
validateEntrySizes: true,
});
const produced = new Set();
Expand Down
53 changes: 52 additions & 1 deletion tests/native-archive-fixture.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,47 @@ const uint32 = (value) => {
buffer.writeUInt32LE(value);
return buffer;
};
// Minimal stored ZIP encoder for synthetic raw names that normal writers
// sanitize. Local and central names agree; payloads are never executable.
export function storedZip(files) {
const local = [],
central = [];
let offset = 0;
for (const [name, data, mode = 0o100644] of files) {
const encoded = Buffer.from(name),
content = Buffer.from(data);
const header = Buffer.alloc(30),
directory = Buffer.alloc(46);
header.writeUInt32LE(0x04034b50, 0);
header.writeUInt16LE(20, 4);
header.writeUInt16LE(0x800, 6);
header.writeUInt32LE(crc32(content), 14);
header.writeUInt32LE(content.length, 18);
header.writeUInt32LE(content.length, 22);
header.writeUInt16LE(encoded.length, 26);
directory.writeUInt32LE(0x02014b50, 0);
directory.writeUInt16LE(0x314, 4);
directory.writeUInt16LE(20, 6);
directory.writeUInt16LE(0x800, 8);
directory.writeUInt32LE(crc32(content), 16);
directory.writeUInt32LE(content.length, 20);
directory.writeUInt32LE(content.length, 24);
directory.writeUInt16LE(encoded.length, 28);
directory.writeUInt32LE((mode * 65536) >>> 0, 38);
directory.writeUInt32LE(offset, 42);
local.push(header, encoded, content);
central.push(directory, encoded);
offset += header.length + encoded.length + content.length;
}
const table = Buffer.concat(central),
end = Buffer.alloc(22);
end.writeUInt32LE(0x06054b50, 0);
end.writeUInt16LE(files.length, 8);
end.writeUInt16LE(files.length, 10);
end.writeUInt32LE(table.length, 12);
end.writeUInt32LE(offset, 16);
return Buffer.concat([...local, table, end]);
}
const vint = (number) => {
const bytes = [];
do {
Expand Down Expand Up @@ -81,7 +122,10 @@ export async function createArchiveFixture(
['Data/unused.bin', Buffer.alloc(500000, 88)],
...extraFiles,
]);
f.archive = path.join(f.packages, `source.${kind === '7z' ? '7z' : 'rar'}`);
f.archive = path.join(
f.packages,
`source.${kind === '7z' ? '7z' : kind === 'zip-windows' ? 'zip' : 'rar'}`,
);
if (kind === '7z') {
const input = path.join(f.root, 'archive-input');
for (const [name, data] of allFiles) {
Expand All @@ -101,6 +145,13 @@ export async function createArchiveFixture(
],
{ cwd: input, windowsHide: true, stdio: 'ignore' },
);
} else if (kind === 'zip-windows') {
fs.writeFileSync(
f.archive,
storedZip(
[...allFiles].map(([name, data]) => [name.replaceAll('/', '\\'), data]),
),
);
} else fs.writeFileSync(f.archive, storedRar5(allFiles));
f.recipe.sources.push({
id: 'original-archive',
Expand Down
Loading
Loading