Reconstruct native outputs from nested BSA 104 containers

.github/workflows/windows.yml

@@ -38,4 +38,4 @@ jobs:
           node scripts/steam-depot.mjs --verify $kit.FullName
           if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
           $env:LOADSTONE_TEST_EXE = Join-Path $kit.FullName 'content/Loadstone.exe'
-          node --test tests/electron.test.mjs tests/live.test.mjs tests/nexus.test.mjs tests/onboarding.test.mjs tests/close-guard.test.mjs tests/native-preview.test.mjs tests/native-builder.test.mjs tests/native-archives.test.mjs tests/native-library-electron.test.mjs tests/native-guide-electron.test.mjs tests/native-repair-electron.test.mjs tests/native-update-electron.test.mjs tests/native-manual-electron.test.mjs
+          node --test tests/electron.test.mjs tests/live.test.mjs tests/nexus.test.mjs tests/onboarding.test.mjs tests/close-guard.test.mjs tests/native-preview.test.mjs tests/native-builder.test.mjs tests/native-archives.test.mjs tests/native-bsa-electron.test.mjs tests/native-library-electron.test.mjs tests/native-guide-electron.test.mjs tests/native-repair-electron.test.mjs tests/native-update-electron.test.mjs tests/native-manual-electron.test.mjs

README.md

@@ -61,6 +61,8 @@ The [real TTW acceptance record](docs/native-steam-acceptance.md) now includes d
 
 [Manual Nexus downloads](docs/native-manual-downloads.md) open each declared file page in the default browser and verify the downloaded original before selection, with byte progress and cancellation. Browser mode tracks pending files and requires all Nexus selections before installation review; existing cache files can be explicitly checked too. Automatic Nexus downloads still require fresh Premium access. Real account/website acceptance and public API registration remain open.
 
+Recipes can also [reconstruct exact loose files from BSA 104 containers](docs/native-reconstruction.md#nested-bsa-104-files) inside original ZIP/7z/RAR downloads. Each archive, container and inner file has an explicit fingerprint; temporary staging, cancellation, resume and repair preserve the existing installation checks. Automatic nested-source mapping and generated-content reproduction remain unfinished.
+
 ## Persistence and desktop behavior
 
 **My library → Repair native setup** now offers [same-version native repair](docs/native-repair.md). It keeps matching files, restores missing/damaged recipe outputs, retains replaced originals and preserves existing profiles/configuration by default. A separate option restores recipe configuration too. Interrupted repairs require fresh review and keep library launch disabled. [Install another version](docs/native-updates.md) creates a separate setup with fresh recipe settings, preserving the original playthrough. Real gameplay acceptance remains open.

docs/bsa-inspection.md

@@ -14,6 +14,6 @@ This initial reader accepts the version-104 layout used by Fallout 3/New Vegas:
 
 Limits: 4,294,967,295 input bytes, 64 MiB metadata, 100,000 folders, 500,000 files, 1 GB per uncompressed member, 20 GB total logical member content (including shared copies) and five minutes per inspection. Hashing streams have bounded buffers; declared sizes, zlib input consumption and output counts are verified. The reader uses the same open file handle throughout and checks file identity afterward.
 
-This report is **source evidence only**. It does not change the ZIP/7z/RAR ledger or add a native recipe operation. To establish an upstream source chain, separately verify the outer download archive, its exact BSA member hash and the desired internal member hash. Compare the installed file's complete size/SHA-256; a similar name or image is insufficient. Native nested-BSA reconstruction, transformed-file reproduction, rights review and real installation acceptance remain separate work. No game or mod data is bundled by this command.
+This report is **source evidence only**. It does not change the ZIP/7z/RAR ledger or automatically produce a recipe. To establish an upstream source chain, separately verify the outer download archive, its exact BSA member hash and the desired internal member hash. Compare the installed file's complete size/SHA-256; a similar name or image is insufficient. A reviewed recipe can use the explicit [`extract-bsa104` action](loadstone-format.md#outputs-and-actions) for [bounded nested-BSA reconstruction](native-reconstruction.md#nested-bsa-104-files). Transformed-file reproduction, rights review and real installation acceptance remain separate work. No game or mod data is bundled by this command.
 
 The JavaScript implementation and synthetic fixtures are original. Format semantics were checked against Wabbajack's pinned [104/105 reader](https://github.com/wabbajack-tools/wabbajack/blob/4.2.3.0/Wabbajack.Compression.BSA/TES5Archive/Reader.cs), [folder records](https://github.com/wabbajack-tools/wabbajack/blob/4.2.3.0/Wabbajack.Compression.BSA/TES5Archive/FolderRecord.cs) and [file records](https://github.com/wabbajack-tools/wabbajack/blob/4.2.3.0/Wabbajack.Compression.BSA/TES5Archive/FileRecord.cs). Loadstone deliberately supports a narrower format subset.

docs/installed-inventory.md

@@ -84,7 +84,7 @@ For tests, `scripts/prepare-sevenzip.ps1 -Destination <new-private-directory>` d
 
 7-Zip is a separate upstream tool with LGPL, BSD and unRAR terms; a future bundled distribution needs its own notices/source compliance review. See the [official release](https://github.com/ip7z/7zip/releases/tag/26.04) and [license](https://www.7-zip.org/license.txt).
 
-For a separate read-only inventory of files inside a local Fallout 3/New Vegas BSA, see [BSA 104 inspection](bsa-inspection.md). This hashes the internal members but does not yet add nested-container matches to the ledger or native reconstruction.
+For a separate read-only inventory of files inside a local Fallout 3/New Vegas BSA, see [BSA 104 inspection](bsa-inspection.md). This hashes the internal members without adding nested-container matches to the ledger. Reviewed source chains can be declared explicitly for [native BSA reconstruction](native-reconstruction.md#nested-bsa-104-files).
 
 ## Native `.loadstone` format
 

docs/loadstone-format.md

@@ -80,7 +80,8 @@ Copy/extract outputs are `{path, bytes, sha256, action}`, optionally with `mtime
 Full recipe verification and build recovery require both the declared bytes/hash and timestamp. A timestamp-only change becomes a recovery/repair conflict: preserve the old file before replacing it, and retain it during removal. Repair's existing default of preserving personal configuration still applies; explicitly restoring configuration is needed to recover its recipe metadata. Normal launch retains its existing executable/profile safety checks and does not perform full recipe verification. Omission preserves legacy behavior and canonical recipe identity. Adding the field creates a new recipe fingerprint; earlier readers reject the unsupported field rather than silently ignoring it.
 
 - `{type: "copy", source: "source-id"}` copies a `game-file`, `prerequisite-file` or `payload`. Expected output bytes/hash must equal the input.
-- `{type: "extract", source: "archive-id", member: "Data/Example.esp"}` selects one safe member of a locked Nexus/HTTPS archive. The output hash is the hash of the extracted member. Nested archives, FOMOD execution, binary patches and arbitrary text substitution are not implemented by this version.
+- `{type: "extract", source: "archive-id", member: "Data/Example.esp"}` selects one safe member of a locked Nexus/HTTPS archive. The output hash is the hash of the extracted member. It never recursively interprets extracted bytes.
+- `{type: "extract-bsa104", source: "archive-id", container: {member: "Data/Example.bsa", bytes: 12345, sha256: "<64 lowercase hex characters>"}, member: "meshes/example.nif"}` explicitly selects one internal member of a BSA-104 container within a locked Nexus/HTTPS ZIP, 7z or RAR download. The container has its own exact size/hash; the output locks the inner file's exact uncompressed size/hash. Both member paths follow the canonical recipe path rules. A recipe may use at most 64 distinct containers totaling 20 GB of staging data; each container is 36–4,294,967,295 bytes and each inner output is at most 1 GB. Conflicting declarations for the same container are rejected. Only the [documented BSA-104 subset](bsa-inspection.md) is accepted. This is exactly one declared container level; direct owned-game BSA inputs, BSA writing, BA2, BSA 103/105, arbitrary nested archives, FOMOD execution, binary patches and arbitrary text substitution remain unsupported.
 - `{type: "mo2-config", gameEdition: "Steam", executables: [...]}` generates only `ModOrganizer.ini` from bounded, relative executable declarations and recipe game/launch fields. `gameEdition` accepts `Steam` or `GOG` matching the target game requirement; executable entries may declare `steamAppId` matching the supported target game and its Steam requirement. Both fields are optional for legacy recipes, whose exact generated bytes remain unchanged. It has no source or static output fingerprint. The materializer computes the exact destination-specific bytes/hash before writing and records them in the receipt after verification. An empty, fingerprinted `portable.txt` output is required. See the [complete generator contract](native-mo2-configuration.md), including serialization, size bounds and unsupported settings.
 
 Outputs must include `ModOrganizer.exe`, `ModOrganizer.ini`, the declared profile's `modlist.txt` and `plugins.txt`, and the target game executable beneath `launch.gameRoot`. Mod priorities, disabled entries and plugin order are carried by the exact profile files. Where the manager/game derives order from modification times, declare the corresponding winning plugin timestamps too. Neither order nor timestamps are inferred from source order. Final-file verification checks each static or derived output fingerprint after reconstruction. Save paths may never be claimed as owned recipe outputs.
@@ -98,6 +99,6 @@ The compiler validates and hashes all bundled bytes, creates the output exclusiv
 
 ## Next implementation gates
 
-An [experimental local materializer](native-reconstruction.md) reconstructs copy and ZIP/7z/RAR-extract recipes in a new isolated directory and verifies every output hash. It is separate from the desktop queue and does not establish playability.
+The [experimental native materializer](native-reconstruction.md) reconstructs copy, ZIP/7z/RAR-extract and explicitly declared BSA-104 recipes in an isolated directory and verifies every output hash. The desktop guide and creator tools use this materializer; reconstruction does not establish playability.
 
 Complete installed-file/source mapping; add bounded reconstruction for edited/generated files; verify real provider downloads and extend supported source flows; extend durable creator jobs with partial-build recovery and playable-library integration. Preserve the existing launch/shortcut/removal ownership checks. Prove actual clean installation and gameplay before publishing the three featured recipes or assigning 1.0.

docs/native-reconstruction.md

@@ -55,6 +55,16 @@ Cancellation or failure retains the candidate directory and receipt. A hard proc
 
 These checks supplement standard Windows access controls; they are not a sandbox for an actively hostile process racing filesystem changes. Keep source files and the new destination settled during reconstruction.
 
+## Nested BSA 104 files
+
+An explicit `extract-bsa104` action can reconstruct loose files from a BSA container inside an original ZIP, 7z or RAR source. The recipe independently locks the outer archive, its BSA member and the final inner file. The installer validates the complete outer archive structure, stages only the declared containers, verifies their full hashes, and checks every BSA member and selected output before creating or changing installation files. Ordinary `extract` actions keep their direct-member meaning.
+
+Staging uses one new directory beneath the operating system's temporary directory and numbered files, never archive-supplied filesystem paths. Up to 64 containers and 20 GB of staging bytes are accepted per recipe. The temporary volume needs the declared container bytes plus a 64 MB reserve; destination space is checked separately. Each container follows the [BSA reader's format and decompression limits](bsa-inspection.md). A compressed or shared member is streamed through an awaited consumer, then the installation writer verifies its exact output bytes/hash and applies any declared timestamp. No archive code runs.
+
+Handled success, cancellation and failure close the readers/writers and remove the operation's ordinary staging files. Cleanup checks directory/file identity, performs no recursive deletion and preserves changed or unfamiliar staging contents. A hard process termination can leave temporary containers behind; restarting builds new staging after a fresh source/recovery review and does not blindly purge earlier temporary folders. Failed installation outputs and user data retain the existing receipt/recovery behavior. Resume reuses verified outputs; repair preserves conflicts and personal configuration through the same review path as other actions.
+
+Synthetic tests cover ZIP, solid 7z and RAR5 outer archives, mixed direct/nested outputs, compressed/empty/shared BSA members, exact timestamps, invalid containers and inner members, unsafe trailing archive entries, resource limits, failed staging writes, cancellation, resume and repair. Actual Electron tests exercise reviewed ZIP and RAR5 reconstruction through the bridge and are also run against the staged executable. The CLI inventory remains evidence only; automatic nested-source ledger mapping is separate work.
+
 ## Acceptance remaining
 
 Complete real provider-download acceptance, conversion, profile/tool configuration reconstruction, real version-update and maintenance acceptance. Creator reconstruction is exercised through real Electron with original synthetic ZIP, solid 7z and stored RAR5 inputs, including cancellation, failed writes and tool replacement. The RAR5 fixture is generated from the [published format](https://www.rarlab.com/technote.htm), without third-party archive contents or encoder code. Clean-machine installation and gameplay acceptance remain unfinished. The separate package inspection card stays read-only.

electron/bsa-limits.cjs

@@ -0,0 +1,11 @@
+module.exports = Object.freeze({
+  archiveBytes: 0xffffffff,
+  metadataBytes: 64 * 1024 * 1024,
+  folders: 100000,
+  files: 500000,
+  memberBytes: 1_000_000_000,
+  totalBytes: 20_000_000_000,
+  milliseconds: 300000,
+  containers: 64,
+  stagingBytes: 20_000_000_000,
+});

electron/bsa104.cjs

@@ -6,15 +6,7 @@ const { createInflate } = require('node:zlib');
 const { ordinaryFile, sameFile } = require('./native-package.cjs');
 const { relativeFile } = require('./manifest.cjs');
 
-const limits = Object.freeze({
-  archiveBytes: 0xffffffff,
-  metadataBytes: 64 * 1024 * 1024,
-  folders: 100000,
-  files: 500000,
-  memberBytes: 1_000_000_000,
-  totalBytes: 20_000_000_000,
-  milliseconds: 300000,
-});
+const limits = require('./bsa-limits.cjs');
 function demand(ok, message) {
   if (!ok) throw new Error(`BSA inspection: ${message}`);
 }
@@ -33,8 +25,9 @@ function normalizedName(bytes, folder = false) {
 }
 
 // Original, deliberately limited BSA 104 reader. See docs/bsa-inspection.md
-// for format references and unsupported variants. No payload is written to disk.
-async function inspectBsa104(file, options = {}) {
+// for format references and unsupported variants. Filesystem destinations belong
+// to the trusted consumer, never to paths supplied by the BSA.
+async function readBsa104(file, options = {}, extraction) {
   const signal = options.signal
     ? AbortSignal.any([
         options.signal,
@@ -79,7 +72,13 @@ async function inspectBsa104(file, options = {}) {
     );
     return bytes;
   }
-  async function digestRange(offset, storedBytes, bytes, compressed = false) {
+  async function digestRange(
+    offset,
+    storedBytes,
+    bytes,
+    compressed = false,
+    consume,
+  ) {
     signal.throwIfAborted();
     const digest = crypto.createHash('sha256');
     let count = 0;
@@ -98,7 +97,11 @@ async function inspectBsa104(file, options = {}) {
               new Error('BSA inspection: output exceeds declared size.'),
             );
           digest.update(chunk);
-          callback();
+          if (consume)
+            Promise.resolve()
+              .then(() => consume(chunk))
+              .then(() => callback(), callback);
+          else callback();
         },
       });
       if (compressed) {
@@ -268,19 +271,41 @@ async function inspectBsa104(file, options = {}) {
     }
     const hashed = new Map(),
       members = [];
+    if (extraction) {
+      const indexed = new Map(entries.map((entry) => [entry.path, entry]));
+      for (const [name, expectedMember] of extraction.selected)
+        demand(
+          indexed.get(name)?.bytes === expectedMember.bytes,
+          'selected member is missing or differs in size.',
+        );
+    }
     for (const entry of entries) {
       signal.throwIfAborted();
+      const selected = extraction?.selected.get(entry.path);
+      if (extraction && !selected) continue;
+      const publicEntry = { name: entry.path, bytes: entry.bytes };
       const key = `${entry.offset}:${entry.storedBytes}:${entry.compressed}`;
-      let memberHash = hashed.get(key);
+      let memberHash = extraction ? undefined : hashed.get(key);
       if (!memberHash) {
         memberHash = await digestRange(
           entry.start,
           entry.length,
           entry.bytes,
           entry.compressed,
+          extraction
+            ? (chunk) => extraction.consumer.consume(publicEntry, chunk)
+            : undefined,
         );
         hashed.set(key, memberHash);
       }
+      if (extraction) {
+        demand(
+          memberHash === selected.sha256,
+          'selected member differs from expected hash.',
+        );
+        signal.throwIfAborted();
+        await extraction.consumer.finish(publicEntry);
+      }
       members.push({
         path: entry.path,
         bytes: entry.bytes,
@@ -310,4 +335,33 @@ async function inspectBsa104(file, options = {}) {
     fs.closeSync(fd);
   }
 }
-module.exports = { inspectBsa104, limits };
+async function inspectBsa104(file, options = {}) {
+  return readBsa104(file, options);
+}
+async function extractBsa104(file, expected, wanted, consumer, signal) {
+  demand(
+    expected &&
+      wanted instanceof Map &&
+      wanted.size > 0 &&
+      wanted.size <= limits.files &&
+      typeof consumer?.consume === 'function' &&
+      typeof consumer?.finish === 'function',
+    'invalid extraction request.',
+  );
+  const selected = new Map();
+  for (const [name, value] of wanted) {
+    demand(
+      relativeFile(name) &&
+        !name.includes('\\') &&
+        Number.isSafeInteger(value?.bytes) &&
+        value.bytes >= 0 &&
+        value.bytes <= limits.memberBytes &&
+        typeof value.sha256 === 'string' &&
+        /^[a-f0-9]{64}$/.test(value.sha256),
+      'invalid selected member fingerprint.',
+    );
+    selected.set(name, { bytes: value.bytes, sha256: value.sha256 });
+  }
+  await readBsa104(file, { expected, signal }, { selected, consumer });
+}
+module.exports = { inspectBsa104, extractBsa104, limits };

electron/native-archive-path.cjs

@@ -0,0 +1,17 @@
+const { relativeFile } = require('./manifest.cjs');
+function zipPath(entry) {
+  const directory = entry.fileName.endsWith('/'),
+    name = directory ? entry.fileName.slice(0, -1) : entry.fileName;
+  const type = (entry.externalFileAttributes >>> 16) & 0o170000;
+  if (
+    !relativeFile(name) ||
+    name.includes('\\') ||
+    name !== name.normalize('NFC') ||
+    (type && type !== (directory ? 0o040000 : 0o100000)) ||
+    entry.generalPurposeBitFlag & 1 ||
+    ![0, 8].includes(entry.compressionMethod)
+  )
+    throw new Error('Unsafe or unsupported ZIP entry.');
+  return { name, directory };
+}
+module.exports = { zipPath };